Merge pull request #115 from mukel/a2p/espresso-jshell-improvements

Update espresso-jshell demo.

espresso-jshell/README.md

@@ -1,51 +1,58 @@
 # GraalVM Demos: Native jshell and Espresso
 
-This demo showcases the integration between [GraalVM Native Image](https://www.graalvm.org/reference-manual/native-image/) and [Java-on-Truffle (Espresso)](https://www.graalvm.org/reference-manual/java-on-truffle/).
-You will build a native executable of `jshell`, that executes dynamically generated bytecode on Espresso.
-This hybrid mode achieves instant startup, beating the vanilla `jshell` in both: time to the first interaction and time to evaluate a simple expression.
+This demo showcases the integration between [GraalVM Native Image](https://www.graalvm.org/reference-manual/native-image/) and [Java-on-Truffle (Espresso)](https://www.graalvm.org/reference-manual/java-on-truffle/).  
+It builds a native image of `jshell`, that executes the dynamically generated bytecodes on Espresso. This hybrid mode achieves instant startup, beating the vanilla `jshell` in both: time to the first interaction and time to evaluate a simple expression.
+
+JShell is a Java read-eval-print loop tool first introduced in Java 9, this demo also allows running `jshell` on Java 8.  
+For further discussions and questions please join our `#espresso` channel on the [GraalVM Slack Community](https://graalvm.slack.com/).
 
-JShell is a Java read-eval-print loop tool first introduced in the JDK 9, but this demo also shows how to run `jshell` on Java 8.
 
 ## Prerequisites
 
-- GraalVM for Java 11 or higher
+- GraalVM for Java 11, 17 or higher
 - Native Image support
 - Java-on-Truffle (Espresso) support
 
-You can download the latest GraalVM [here](https://www.graalvm.org/downloads/).
-Having [installed GraalVM](https://www.graalvm.org/docs/getting-started/#install-graalvm), you can add Native Image and Java on Truffle (Espresso) support easily.
-GraalVM comes with `gu` which is a command line utility to install and manage additional functionalities, and to install Native Image and Espresso, run this single command:
+Download the latest GraalVM [here](https://www.graalvm.org/downloads/).  
+Having [GraalVM installed](https://www.graalvm.org/docs/getting-started/#install-graalvm), install the _Native Image_ and _Java on Truffle_ (Espresso) components.  
+GraalVM bundles [`gu`](https://www.graalvm.org/reference-manual/graalvm-updater/), a command line utility to install and manage additional functionalities/components; to install the _Native Image_ and _Java on Truffle_ (Espresso) components, run the following command:
 
 ```bash
 <graalvm>/bin/gu install native-image espresso
 ```
 
 ## How to Build
-You should set the `GRAALVM_HOME` variable to the GraalVM home:
+Set the `GRAALVM_HOME` environment variable to the GraalVM home:
 ```bash
 export GRAALVM_HOME="/path/to/graalvm"
 ```
 
-Then exetute the `build-espresso-jshell.sh` script:
+Then execute the `build-espresso-jshell.sh` script:
 ```bash
 ./build-espresso-jshell.sh
 ```
 
 It generates a native executable: `espresso-jshell` in the working directory.
-This native executable, `espresso-jshell [options...]`, by default runs on Java 11 with instant startup.
+This native executable, `espresso-jshell [options...]`, runs with almost-instant startup.
 
-Launch `espresso-jshell`, execute some Java code and see the output immediately.
+Launch `./espresso-jshell -Dorg.graalvm.home="$GRAALVM_HOME"`, execute some Java code and see the output immediately.
 To exit the shell, type `/exit`.
 
-## Running jshell for Java 8
+## Running `jshell` on Java 8
+
+`jshell` was first introduced in Java 9, but thanks to Java's excellent backwards compatibility, it's possible to run `jshell` on a Java 8 environment.
 
 ```bash
 export GRAALVM_HOME="/path/to/graalvm"
 export JDK8_HOME="/path/to/jdk8"
-sh jshell8.sh [options...]
+./jshell8.sh [options...]
 ```
 
-You can run the following snippets to convince yourself that it runs on the Java 8 mode:
+It may seem that `espresso-jshell` is running Java 11 in this mode, and it is.  
+`jshell` (the frontend) is compiled by native-image with Java 11 (or 17), but the Java compiler (`javac` has a Java API used by `jshell`) is fully backwards compatible with Java 8 e.g. `javac -source 8 -target 8 -bootclasspath JAVA8_BOOT_CLASSPATH`.  
+The generated bytecodes are then executed in Espresso (the backend) which runs a Java 8 guest JVM.
+
+You can run the following snippet to convince yourself that it indeed, runs on Java 8:
 ```java
 System.getProperty("java.vm.version");
 
@@ -58,10 +65,28 @@ Object.class.getModule();
 Class.class.getDeclaredMethod("getModule");
 ```
 
-Pass system properties to `espresso` with `-R-Dkey=value`.
-Pass polyglot options to `espresso` with `-Rjava.InlineFieldAccessors -Rengine.Compilation=false`.
+Pass system properties to Espresso with `-R-Dkey=value`.  
+Pass polyglot options to Espresso with `-Rjava.InlineFieldAccessors -Rengine.Compilation=false`.
+
+## How it works?
+For hybrid projects like this one, a clear boundary between host and guest code is a requirement.  
+`jshell` has two main components: the frontend, which includes the console interface and the Java compiler; and the backend, referred as the "execution engine", where the dynamically generated bytecodes are executed.
+
+`jshell` allows to implement custom backends, see [jdk.jshell.spi.ExecutionControl](https://docs.oracle.com/en/java/javase/11/docs/api/jdk.jshell/jdk/jshell/spi/ExecutionControl.html). This is a very clean boundary since there are no complex objects crossing it, only primitives, strings, arrays and a few exceptions. It also provides a few implementations e.g. [JdiDefaultExecutionControl](https://docs.oracle.com/en/java/javase/11/docs/api/jdk.jshell/jdk/jshell/execution/JdiDefaultExecutionControl.html) which spawns another process and communicates with it through JDI, [LocalExecutionControl](https://docs.oracle.com/en/java/javase/11/docs/api/jdk.jshell/jdk/jshell/execution/LocalExecutionControl.html) which runs in the same VM as `jshell`.
+
+To communicate between host and guest worlds, an [adapter](/~https://github.com/mukel/graalvm-demos/blob/master/espresso-jshell/src/main/java/com/oracle/truffle/espresso/jshell/EspressoExecutionControl.java) was implemented in host Java that, via interop, forwards all methods calls to an Espresso guest object, taking care of converting all the arguments and return values, translating guest exceptions to host exceptions...  
+On the Espresso side, a guest [LocalExecutionControl](https://docs.oracle.com/en/java/javase/11/docs/api/jdk.jshell/jdk/jshell/execution/LocalExecutionControl.html) executes all the methods calls forwarded by the host adapter; any other guest `ExecutionControl` implementation could be used.
+
+<p align="center">
+    <img src="./diagram.svg">
+</p>
+
+`espresso-jshell` runs partially on a guest `LocalExecutionControl` instance, so it behaves similar to `jshell -execution local` and shall not be compared with the default `jshell` execution mode e.g. class redefinition is not supported in this mode, `jshell -execution local` does not support it either. It does not support other execution engines other than `-execution espresso`, which is our host-to-guest adapter, see [EspressoLocalExecutionControlProvider](/~https://github.com/graalvm/graalvm-demos/blob/master/espresso-jshell/src/main/java/com/oracle/truffle/espresso/jshell/EspressoLocalExecutionControlProvider.java).
+
+Since the Java compiler is part of the host code compiled by `native-image`, dynamically loaded annotation processors are not supported. Annotation processors must be compiled-in AOT by `native-image`. But this limitation could be lifted: since the Java compiler provides an interface to implement annotation processors, it could be possible to run the Java compiler on the host and the dynamically loaded annotation processors on Espresso using the same idea we just described.
 
-## Details
-`espresso-jshell` behaves similar to `jshell -execution local` and should not be directly compared with the default `jshell` execution engine, e.g., class redefinition is not supported (`jshell -execution local` does not support it either).
-It does not  support execution engines other than `-execution espresso`.
-It is not fully standalone, it needs the _jars/jmods_ to compile and for Espresso to run.
+## Distribution
+`espresso-jshell` is not fully standalone, it doesn't bundle _jars/jmods_ nor the core Java native libraries. `jshell` also needs a `java.home` for the host Java compiler.
+Specifying a GraalVM home is the easiest way for Espresso and `jshell` to find all these dependencies e.g. `./espresso-jshell -Dorg.graalvm.home="$GRAALVM_HOME"`.  
+`espresso-jshell` doesn't require a full-blown GraalVM distribution to run. It can run with a minimal/jlink-ed Java home that includes the Espresso home (`$GRAALVM_HOME/languages/java`), mimicking the same folder structure as GraalVM.  
+I successfully ran `espresso-jshell` on a minimal Java home (11) with only `java.base` + Espresso home, weighting just ~26MB uncompressed, ~9.8MB zipped; in addition to the `espresso-jshell` executable.

espresso-jshell/build-espresso-jshell.sh

@@ -38,11 +38,12 @@ echo "7/7 Building espresso-jshell native image, this may take a few minutes..."
 "$GRAALVM_HOME/bin/native-image" \
   -H:+AllowJRTFileSystem \
   -H:Name=espresso-jshell \
-  --initialize-at-build-time=com.sun.tools.doclint,'jdk.jshell.Snippet$SubKind' \
+  --initialize-at-build-time=com.sun.tools.doclint,'jdk.jshell.Snippet$SubKind','com.sun.tools.javac.parser.Tokens$TokenKind' \
+  --initialize-at-run-time=com.sun.tools.javac.file.Locations \
+  --report-unsupported-elements-at-runtime \
   -H:ConfigurationFileDirectories=. \
   --language:java \
   -J-Djdk.image.use.jvm.map=false \
-  -J-Dorg.graalvm.launcher.home="$GRAALVM_HOME" \
   -J--patch-module -Jjdk.jshell=./jdk.jshell-patch `# patch for Java 8, not required for 11` \
   -H:+ReportExceptionStackTraces \
   $@ \

espresso-jshell/jshell8.sh

@@ -24,6 +24,7 @@ fi
 JAVAC_BOOT_CLASSPATH="$JDK8_JRE/lib/resources.jar:$JDK8_JRE/lib/rt.jar:$JDK8_JRE/lib/sunrsasign.jar:$JDK8_JRE/lib/jsse.jar:$JDK8_JRE/lib/jce.jar:$JDK8_JRE/lib/charsets.jar:$JDK8_JRE/lib/jfr.jar:$JDK8_JRE/classes"
 
 ./espresso-jshell \
+    -Dorg.graalvm.home="$GRAALVM_HOME" \
     -Rjava.JavaHome="$JDK8_HOME" \
     -Rjava.BootClasspathAppend="./jshell8.jar" \
     -C-source -C8 \