Add a benchmark for compilation times

[rikhuijzer]

Want meten is weten (because measuring is knowning). I've tried to do something fancy like rikhuijzer/PrecompileMacro.jl#1, but that will take some (a lot of) time to make. So, as a first step, this PR just adds a separate benchmark for compilation time. I think that I've missed a lot of entry points, but I hope that once we are able to pin down what spots are compiled exactly, then we can start making progress on improving the performance.

[github-actions]

Try this Pull Request!

Open Julia and type:

julia> import Pkg
julia> Pkg.activate(temp=true)
julia> Pkg.add(url="/~https://github.com/rikhuijzer/Pluto.jl", rev="rh/precompile-macro")
julia> using Pluto

[rikhuijzer]

I've added precompile directives to the other entrypoints inside SessionActions. I wasn't able to use PrecompileMacro.jl because it gave very weird errors. Adding the precompile directives show some improvements on the allocations in the logs:

5×3 DataFrame
 Row │ Operation                Allocations (MB)  Time (seconds)
     │ String                   Float64           Float64
─────┼───────────────────────────────────────────────────────────
   1 │ using Pluto                          76.5             0.8
   2 │ Pluto.ServerSession()                13.2             0.2
   3 │ SessionActions.open                 897.4            10.4
   4 │ SessionActions.shutdown              18.8             0.3
   5 │ SessionActions.new                   16.9             0.4

Don't take these times too seriously. This run was on a particularly slow runner. Look at allocations.

I guess this provides a good idea of the tradeoff between running time and adding these directives. In this case, the 3 extra directives save 0.4 seconds in total or so on my pc.

[fonsp]

I made JuliaPackaging/RelocatableFolders.jl#15 about this. Did you switch git branches?

[rikhuijzer]

This is how the output looks now:

 ─────────────────────────────────────────────────────────────────────
                                           Time          Allocations
                                     ───────────────   ───────────────
           Total measured:                 740s            14.0GiB

 Section                     ncalls     time    %tot     alloc    %tot
 ─────────────────────────────────────────────────────────────────────
 import Pluto                     1    459ms    0.1%   66.0MiB    0.5%
 compiletimes.jl                  1    24.7s    3.4%   2.10GiB   15.3%
   SessionActions.open            1    12.5s    1.7%   1.14GiB    8.3%
   SessionActions.shutdown        1    563ms    0.1%   71.5MiB    0.5%
 Events.jl                        1    4.58s    0.6%    405MiB    2.9%
 WorkspaceManager.jl              1    93.3s   12.7%   1.01GiB    7.4%
 packages/Basic.jl                1     148s   20.1%    878MiB    6.2%
 Bonds.jl                         1    27.0s    3.7%    284MiB    2.0%
 RichOutput.jl                    1    45.1s    6.1%   1.31GiB    9.5%
 React.jl                         1     116s   15.8%   2.91GiB   21.2%
 Dynamic.jl                       1    31.5s    4.3%    799MiB    5.7%
 MacroAnalysis.jl                 1    98.9s   13.4%    825MiB    5.9%
 Logging.jl                       1    15.1s    2.0%   52.8MiB    0.4%
 webserver.jl                     1    20.6s    2.8%   1.52GiB   11.1%
 Notebook.jl                      1    87.5s   11.9%    568MiB    4.0%
 Configuration.jl                 1    8.27s    1.1%    402MiB    2.9%
 ReloadFromFile.jl                1    4.92s    0.7%    100MiB    0.7%
 packages/PkgCompat.jl            1    398ms    0.1%   33.1MiB    0.2%
 ExpressionExplorer.jl            1    5.28s    0.7%    317MiB    2.3%
 MethodSignatures.jl              1    132ms    0.0%   15.7MiB    0.1%
 MoreAnalysis.jl                  1    925ms    0.1%   79.9MiB    0.6%
 Analysis.jl                      1   94.6ms    0.0%   4.12MiB    0.0%
 webserver_utils.jl               1    818ms    0.1%   43.1MiB    0.3%
 data structures.jl               1    713ms    0.1%   68.3MiB    0.5%
 DependencyCache.jl               1    355ms    0.0%   27.4MiB    0.2%
 Throttled.jl                     1    841ms    0.1%   2.14MiB    0.0%
 cell_disabling.jl                1    536ms    0.1%   26.9MiB    0.2%
 ───────────────────────────────────────────────────────────────────── 

The compiletimes.jl runs first and gives an estimate of how long compilation takes.

[rikhuijzer]

@fonsp Can you review this?

[fonsp]

I took the sessionactions precompile statements out, let's do that in a new PR (also so that we can use these timing to see the difference!!)

[fonsp]

Thanks!! Looks great! Seeing allocations seems very useful in CI

[fonsp]

precompile statements in #1977