compile_incremental overwrites

[daviehh]

If run in separate commands, the latter compile_incremental will overwrite the sysimg
e.g.

1. run julia without the -J /path/to/sysimg switch, do compile_incremental(:Plots)
2. add the -J switch, go to the pkg prompt ] do st:

(PackagesPrecompile) pkg> st
Project PackagesPrecompile v0.0.0
Status ~/.julia/packages/PackageCompiler/oT98U/packages/Project.toml
[d38c429a] Contour v0.5.1
[5789e2e9] FileIO v1.0.5
[53c48c17] FixedPointNumbers v0.5.3
[28b8d3ca] GR v0.38.1
[6218d12a] ImageMagick v0.7.1
[916415d5] Images v0.17.2
[682c06a0] JSON v0.20.0
[b964fa9f] LaTeXStrings v1.0.3
[442fdcdd] Measures v0.3.0
[77ba4419] NaNMath v0.3.2
[9b87118b] PackageCompiler v0.6.3
[c5e800db] PlotReferenceImages v0.0.0 #master (/~https://github.com/JuliaPlots/PlotReferenceImages.jl.git)
[ccf2f8ad] PlotThemes v0.3.0
[995b91a9] PlotUtils v0.5.5+ #master (/~https://github.com/JuliaPlots/PlotUtils.jl.git)
[91a5bcdd] Plots v0.23.1
[dca85d43] QuartzImageIO v0.5.2
[ce6b1742] RDatasets v0.6.1
[3cdcf5f2] RecipesBase v0.6.0+ #master (/~https://github.com/JuliaPlots/RecipesBase.jl.git)
[189a3867] Reexport v0.2.0
[ae029012] Requires v0.5.2
[992d4aef] Showoff v0.2.1
[90137ffa] StaticArrays v0.10.3
[2913bbd2] StatsBase v0.29.0
[f3b207a7] StatsPlots v0.10.2
[b8865327] UnicodePlots v1.1.0
[34922c18] VisualRegressionTests v0.3.0
[2a0f44e3] Base64
[ade2ca70] Dates
[37e2e46d] LinearAlgebra
[44cfe95a] Pkg
[de0858da] Printf
[3fa0cd96] REPL
[9a3f8284] Random
[2f01184e] SparseArrays
[10745b16] Statistics
[8dfed614] Test
[cf7118a7] UUIDs

3. return to julia without the -J switch, compile another package, e.g. compile_incremental(:Makie), exit, add the -J switch, and pkg>st returns

(PackagesPrecompile) pkg> st
Project PackagesPrecompile v0.0.0
Status ~/.julia/packages/PackageCompiler/oT98U/packages/Project.toml
[537997a7] AbstractPlotting v0.9.6
[5789e2e9] FileIO v1.0.5
[f7f18e0c] GLFW v2.3.0
[e9467ef8] GLMakie v0.0.5
[a09fc81d] ImageCore v0.7.4
[6218d12a] ImageMagick v0.7.1
[02fcd773] ImageTransformations v0.7.2
[66fc600b] ModernGL v1.0.0
[9b87118b] PackageCompiler v0.6.3
[dca85d43] QuartzImageIO v0.5.2
[44cfe95a] Pkg

Looks like the precompiled Plots is overwritten. Is this the expected behavior/is there an option to add to existing sysimg without overwriting? Thanks!

[asinghvi17]

This is expected as far as I am aware, since PackageCompiler will compile the system image into its directory. One thing you can do is refer to the following code:

PackageCompiler.jl/src/incremental.jl

Lines 114 to 145 in 54c0c12

	"""
	compile_incremental(
	toml_path::String, snoopfile::String;
	force = false, precompile_file = nothing, verbose = true,
	debug = false, cc_flags = nothing
	)
	Extract all calls from `snoopfile` and ahead of time compiles them
	incrementally into the current system image.
	`force = true` will replace the old system image with the new one.
	The argument `toml_path` should contain a project file of the packages that `snoopfile` explicitly uses.
	Implicitly used packages & modules don't need to be contained!
	To compile just a single package, see the simpler version `compile_incremental(package::Symbol)`:
	"""
	function compile_incremental(
	toml_path::Union{String, Nothing}, precompiles::String;
	force = false, verbose = true,
	debug = false, cc_flags = nothing
	)
	systemp = sysimg_folder("sys.a")
	sysout = sysimg_folder("sys.$(Libdl.dlext)")
	code = PrecompileCommand(precompiles)
	run_julia(
	code, O = 3, output_o = systemp, g = 1,
	track_allocation = "none", startup_file = "no", code_coverage = "none"
	)
	build_shared(sysout, systemp, false, sysimg_folder(), verbose, "3", debug, system_compiler, cc_flags)
	curr_syso = current_systemimage()
	force && cp(sysout, curr_syso, force = true)
	return sysout, curr_syso
	end

and change it manually (essentially, replicate the compile_incremental method with whatever output directory you want).

A PR could be made to amend sysimg_folder

PackageCompiler.jl/src/PackageCompiler.jl

Lines 62 to 66 in 8987c26

	function sysimg_folder(files...)
	base_path = normpath(abspath(joinpath(@__DIR__, "..", "sysimg")))
	isdir(base_path) || mkpath(base_path)
	normpath(abspath(joinpath(base_path, files...)))
	end

perhaps with a keyword argument to change the parent directory?