[jsscripting] GraalVM's optimised compiler or find another solution

[florian-h05]

Coming from https://community.openhab.org/t/blockly-rules-after-upgrade-to-oh-4-0-1-very-slow/148503/43?u=florian-h05, where users reported bad performance of JS Scripting/GraalJS.

As @wirthi, a GraalVM developer, explained, this is because we do not run GraalJS on GraalVM, you should at least use its optimised compiler.
GraalVM provided an example of how to run it on a stock JDK (https://www.graalvm.org/latest/reference-manual/js/RunOnJDK/), and even a Maven example: /~https://github.com/graalvm/graal-js-jdk11-maven-demo.

Regarding size limitations of the add-on: JS Scripting already is a pretty large add-on compared to the average, but if we can improve performance, I see no problem adding more MBs.

FYI our JavaScript Scripting add-on bundle is here: /~https://github.com/openhab/openhab-addons/tree/main/bundles/org.openhab.automation.jsscripting, there you can also find the Maven POM.

I've created this issue to discuss the inclusion of the GraalVM optimised compiler and focus on the technical aspects, for which IMO GitHub is better suited than our community forum.

@wirthi and @digitaldan, I'm neither a GraalVM expert nor a Maven expert, would be great if you both could have a look and check whether it is possible to include the compiler from the add-on, or this would be something to be done from openhab-distro side.

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/blockly-rules-after-upgrade-to-oh-4-0-1-very-slow/148503/55

[florian-h05]

Thinking a bit of it, I have changed the title of the issue to also discuss other solutions here. The community discussion IMO is getting to off-topic with some people not contributing to the technical solution of the problem but instead being very rude.

[stefan-hoehn]

Here are some tests I did today. Note that my dev and prod system are not equal.

The prod system has hundreds of items and things, while the TEST system is minimal.
So, yes, we are comparing apple with pies...but still this is the best I can do.

Full Disclore: I used a very simple rule for the test intentionally.

Let's start to measure the 32 Bit debian on a 64 bit Arm RP4

• Without caching consistently around 20 secs on manual run (which saves the rule first and then run. Consecutive runs ARE fast!)
• With caching on : around 15-17 seconds (so it seems that caching does have an effect)

Let's start to measure the 64 Bit debian on a 64 bit Arm RP4

Here is the comprehensive test on my dev system which has a 64 Bit ARM PI4 and the Debian is also on ARM64.

I installed two different JDKs

.../17.0.7-graalce/bin/java
...java-17-openjdk-arm64/bin/java (Zulu)

The results are

• In general it takes around 15 only when the first rule is run

openjdk 17.0.7 WITH caching:
time: ~ 1-2 sec

I then used a more complex rule and the result is more or less the same.

openjdk 17.0.7 Without caching:
time: ~ 5 sec

-> So, it seems that if we assume that the openHAB setup itself does not have an impact, that JDK 64 Bit is MUCH FASTER
-> And Caching does in fact help a lot here!

So, how about GraalVM, will it be the holy Grail?

The results are

graalvm-ce 17.0.7 WITHOUT caching:
time: ~ 2 sec

graalvm-ce 17.0.7 WITH caching:
time: ~ 2 sec

-> So, it seems that if we assume that the openHAB setup itself does not have an impact, that graalVM 64 Bit is also much faster
-> Caching doesn't have an effect anymore as the GraalVM seems to have a boost that caching will not improve.

So, my take is

• In general a 64 bit VM is much faster and from my perspective already fast enough
• Caching does help again here and the time goes down to a pretty nice latency.
• Using graalVM does in fact speed up the same like openJDK but compared to the cached variant it is not in any way faster.
  • note, if you go for graalvm, only choose the community edition as the other one has to be licenced with Oracle, AFAIK.

My take is that I go with the standard Java Zulu 64 bit implementation based on openhabian 64 bit as, in particular with caching on, it is more than fast enough even in development.

I would be vary happy if someone would be able to verify the above results by doing the same or similar test on two different machines with the setup I described above.

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/blockly-rules-after-upgrade-to-oh-4-0-1-very-slow-on-the-first-run/148503/73

[wirthi]

Hi @stefan-hoehn

thanks for doing those measurements. I am not sure I interpret the results correctly (e.g. what 32bit and 64bit versions you compare on what machines, and whether you use OpenJDK OR Zulu). I am missing a lot of context here, basically your whole setup (what exactly do you mean by caching, what do you cache? how do you call into the script (likely with javax.script.ScriptEngine?) Do you execute the same script repeatedly, or just one script once?).

In general, I strongly believe that a benchmark that runs for 2 seconds only is way to short in a Java environment. You will have a huge impact by startup and warmup effects.

Christian

[rkoshak]

In general, I strongly believe that a benchmark that runs for 2 seconds only is way to short in a Java environment. You will have a huge impact by startup and warmup effects.

The whole point is to measure those startup warmup effects. Subsequent runs of the rule are fast and if one were to run multiple times it would obscure what we are trying to measure in the first place. It's that first run after being loaded where the Helper Library is injected that is the problem that we are attempting to solve.

We are trying to eliminate the 20-30 second delay between editing a rule and that first run of the rule. So we are only measuring that first run of the rule.

what exactly do you mean by caching, what do you cache?

Caching refers to the helper library. The first time the script is run /~https://github.com/openhab/openhab-js gets loaded into the script. This library converts the native Java API to openHAB into JavaScript to allow for a smoother and more language pure environment. It is the loading of that library that is causing the 20-30 second delay. Subsequent runs reuses the script and that penalty is no longer an issue.

As for the details on how that caching is done I can't say.

In general, I strongly believe that a benchmark that runs for 2 seconds only is way to short in a Java environment. You will have a huge impact by startup and warmup effects.

What is being shown is that by using GraalVM it brings that first run of the rule from 20 seconds down to 2 seconds.

If I put @stefan-hoehn's result in a table (Raspberry Pi 4):

JRE	32/64	helper library caching	first run of the rule latency in seconds
OpenJDK	32 bit	OFF	20 s
OpenJDK	32 bit	ON	15-17 s
OpenJDK	64 bit	OFF	5 s
OpenJDK	64 bit	ON	1-2 s
GraalVM	64 bit	OFF	2 s
GraalVM	64 bit	ON	2 s

[florian-h05]

Hi @wirthi,

if you have any questions about the implementation of the JS Scripting add-on, feel free to ask @digitaldan and me!

When running a script using the JS Scripting add-on, a new OpenhabGraalJSScriptEngine is created.

This is the place where a javax.script.ScriptEngine (or more exactly it's implementation com.oracle.truffle.js.scriptengine.GraalJSScriptEngine) is created to run the script.
All GraalJSScriptEngines share a single org.graalvm.polyglot.Engine.

In the next step, our helper library called openhab-js is injected to allow easy access to the openHAB Java APIs. It takes care of type conversions between Java and JS and provides easy access to openHAB.
We currently have two ways of injecting it:

• Object.assign(this, require('openhab')} (we call it caching disabled)
• Loading the library into a org.graalvm.polyglot.Source, which enables together with the shared Engine code-caching across multiple contexts (see https://www.graalvm.org/22.0/reference-manual/embed-languages/#code-caching-across-multiple-contexts) (we call it caching enabled)

To inject the library, the add-on bundle ships two different webpack bundled versions of the library: One that is compatible with require and one that exports the library's namespaces onto the global object.

The whole point is to measure those startup warmup effects. Subsequent runs of the rule are fast and if one were to run multiple times it would obscure what we are trying to measure in the first place. It's that first run after being loaded where the Helper Library is injected that is the problem that we are attempting to solve.

Exactly to the point!

[florian-h05]

For reference: „Caching“ of the openhab-js has been introduced with #14135.
Community discussion on compilation times: https://community.openhab.org/t/oh3-to-use-or-not-to-use-js-ecmascript-2021/141911/40?u=florian-h05

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/blocky-js-unusable-showstopper-delay-on-1st-run-please-advice-for-alternative-solution/151787/7

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/yet-another-js-scripting-first-run-delay-test-pi5/152774/2

[egoist6]

Just wanted to mention another test I did:
On my new non-openhabian production system the delay of a first run of a rule is consistently less than a second across 5 tests:

==> /var/log/openhab/events.log <==
2024-01-07 11:56:07.386 [INFO ] [openhab.event.RuleUpdatedEvent      ] - Rule 'TestRule' has been updated.
==> /var/log/openhab/openhab.log <==
2024-01-07 11:56:08.278 [INFO ] [nhab.automation.script.ui.TestRule] - Info: Rule TestRule gestartet.

Config:
NO openhabian - pure manual installation:

• openhab 4.1.0 Release Build
• GraalVM 17
• Frontail, Samba und a few more
• Hardware: Pi5-8GB/bookworm, USB-SSD
• EXTRA_JAVA_OPTS="-Xms800m -Xmx1500m"

[masipila]

I recently updated from openHAB 3.3 to 4.1, I'm running Raspberry Pi 4. Some of my Javascript Rules were very slow after the update (tens of seconds, when they were completing in a matter of seconds on OH3.3). This was also after they had been cached.

It turned out to be the 32-bit vs. 64-bit thing mentioned here. My OH3.3 was running on 32-bit version of the openHABian SD image (as was recommended back in the days when I installed OH3). Obviously it was still 32-bit version after I had updated OH to 4.1.

Based on the information in this thread I re-installed a second SD card with 64-bit edition of openHABian and now the same Javascript Rules are performing on the expected level again.

Do you guys agree that this documentation is outdated and should be updated:
https://www.openhab.org/docs/installation/openhabian.html#_64-bit

Cheers,
Markus

[rkoshak]

Do you guys agree that this documentation is outdated and should be updated:
https://www.openhab.org/docs/installation/openhabian.html#_64-bit

I think we agree here but I'm not sure that the maintainer of openHABian would agree. I think opening an issue for discussion is warranted as I think we have enough evidence here to show that using 64-bit is worth a recommendation. But there is a balance. If you are on an RPi 3, merely switching to 64-bit is likely to cause you to run out of RAM while running OH and 64-bits does make even the 4 GB on an RPi 4 a little tight. So it can't just be a simple "always use 64-bit".

[egoist6]

So it can't just be a simple "always use 64-bit".
True. And it can't just be a simple "always use 32-bit", the mantra of yesterday. (and I do noticed the „discussion“ to get at least a statement regarding 64-bit into openhabian documentation)
This mantra might have been correct in the old days where Pis just had 1 GB of RAM.
Additionally the forum is getting crowded with posts of problems where CPU load is too high, memory consumption too high, various (micro)-services not running properly, etc. Most of them are related to insufficient hardware, like Pi3, 1GB, which was ok for OH3, but not anymore for OH4.
I think it is time to provide a clear message and clear minimum system requirement recommendation for OH4, such as Pi4, 4GB, 64bit PiOS. To me it is more than urgent to stop fiddling with poor, old hardware.
World is evolving and we are running into danger to prevent improvements if we still stick with the old mantra and potentially scaring away new users who wonder, why OH does not officially recommend 64-bit, does not support SSDs and does not support bookworm.

[stefan-hoehn]

All that I can say that after migration to 64bit on my RP4 compilation vastly improved on my production system.

@mstormi would you agree to add a recommendations towards 64bit in some way and some of the ideas that @egoist6 recommends?

[rkoshak]

why OH does not officially recommend 64-bit

It's no panacea and comes with it's own issues even on machines with 4GB RAM like an RPi 4.

does not support SSDs

It's technically challenging and no one has volunteered to implement and support it.

and does not support bookworm.

Bookworm breaks a lot of stuff that openHABian relied upon to work. Support for bookworm is still in progress but it's way more than just copy the old stuff onto the new OS. Whole parts of openHABian are needing to be reimplemented.

[mstormi]

Well it's true that things have changed since those docs were written and I'll change them into a more balanced statement.

This statement, however

I think it is time to provide a clear message and clear minimum system requirement recommendation for OH4, such as Pi4, 4GB, 64bit PiOS. To me it is more than urgent to stop fiddling with poor, old hardware.

is dead wrong, clearly not balanced and, sorry for the pun, too egoistic as to become part of any recommendation.

openHABian is for everyone and in particular for all those that don't care about hardware but just want a working base system to focus on the application level, OH that is.
Including people that have been running OH for years, and including those that don't want to buy new hardware but run on what they have available.

And holding off not to comment on the openHABian bashing.
@egoist6 move that sort of content off Github, that does not belong to the forum and even less so here.

[egoist6]

I followed the discussion and the status of bookworm support of openahbian. Again, I think it is not a good idea to wait until sometimes in the future homegear and InfluxData runs on bookworm which is currently one of the reason why bookworm is not supported by openhab(ian).
Why not clearly communicating, that if you require a specific set of apps you need to stick with bullseye otherwise bookworm is recommended - it is all about communication.
We are focusing too much on openhabian and its limitations (no bookworm, no clear recommendation for 64bit, no ssd), instead of telling people that openhab runs on state of the art hardware. Agreed, with the „downside“ of having to install openhab manually. But that isn‘t too complex at all.

[egoist6]

@mstormi
is dead wrong, clearly not balanced and, sorry for the pun, too egoistic as to become part of any recommendation.
That simply is your opinion. That doesn‘t necessarily mean that your opinion is correct. But your attitude across all posts is that YOU are the only owner of truth. Simply said, no you are not. And this is by far not just my opinion as you might have already noticed by the various responses you have received.

[rkoshak]

instead of telling people that openhab runs on state of the art hardware. Agreed, with the „downside“ of having to install openhab manually. But that isn‘t too complex at all.

There is a huge overlap between those users who would find it challenging to install all that stuff manually and users of openHABian. And it's those less technically capable users who are the audience of openHABian. Unfortunately, these are also the same users who will be using Blockly where 64-bit is needed to get acceptable performance on the first run of a rule. So it's a challenging situation.

These are the users least able to adjust to changes. These are the users least able to do stuff manually. And not all of these users are new users and legacy support is a worthy consideration.

Also, don't confuse recommendations on the openHABian docs with general recommendations. As soon as you enter "having to install openhab manually" you are out of scope for openHABian. You are now in scope of whatever OS you are installing on (e.g. https://www.openhab.org/docs/installation/linux.html) and the Prerequisites: https://www.openhab.org/docs/installation/#prerequisites.

There we find the following:

The 32-bit version of the JVM is recommended on ARM platforms such as the Raspberry Pi. The 32-bit JVM performs better on the ARM platform. Some add-ons use libraries that do not work with a 64-bit JVM on ARM.

So do we just throw some bindings under the bus? Does anyone know which bindings they are?

If we "clearly recommend only 64-bit" we are clearly recommending a configuration that is known not to work for some officially supported add-ons. If we continue to clearly recommend 32-bit for ARM, we suffer performance problems for JS Scripting. Which set of users do we lead down the wrong path?

Unless something changes, I don't think we can make a clear recommendation either way. And to make it even more complicated the information the first time user is going to need to make a good decision is likely to be information they won't have until they've already installed and worked with OH for a bit.

So I propose the following:

1. On the openHABian page recommend if running an ARM with 4 GB RAM or more, recommend 64-bit, perhaps with the caveat that some bindings are known not to work (it would be great to have a list) but that JS Scripting and Blockly are known to be slow on 32-bit.
2. On the prerequisites page recommend if running on an ARM with 4 GB RAM or more, recommend 64-bit with the same caveats. We could possibly provide a bit more detail here as to why than in the openHABian case.

Unless and until all bindings work in 64-bit on ARM we cannot just make a blanket "use 64-bit" recommendation. Unless and until we completely drop support for RPi 3s we cannot just make a blanket "use 64-bit" recommendation. Not until both of these occur can we switch to a 64-bit first approach.

[egoist6]

Good point Rich.
Does anyone know which bindings they are? Apart from this discussion and whether to install openhabian or openhab manually, shouldn’t we document this so that it can become part of the user’s decision making process? On the other hand, are there maybe already bindings which require 64bit?
I am happy to create additional content to the download page e.g. as a matrix like

• Pi3: openhabian (32bit, bullseye, SD)
• Pi4 or 5:
  • if the following bindings required: openhabian (32bit, bullseye, SD)
  • otherwise: openhabian (64bit, bullseye, SD) or manual installation (64bit, bookworm, SSD or SD)
• 64bit is recommended, if
  • you plan to create and test continuously rules (e.g. you are new to openhab and need to build up your rules from scratch and need to test your rules every time you change your rules)
  • you want to use blockly
  • …
    there are probably more key factors which a user need to consider.

[masipila]

@egoist6 : The performance issue with Javascript on openHAB 4 is not limited to the first run of the script / rule. Some of my more performance expensive scripts (which involve quite heavy calculations) were running almost a minute on a 32-bit openHABian even after the first run. Execution time on 64-bit openHABian is a couple of seconds, and most of this time is spent in calling influxDB API on another server.

[egoist6]

Incredible. Wasn’t aware of this huge discrepancy. Thanks for letting us know.

[stefan-hoehn]

If we "clearly recommend only 64-bit" we are clearly recommending a configuration that is known not to work for some officially supported add-ons.

That was totally new to me actually. Do you know any particular add-on that is known not to be working and maybe even why?

[J-N-K]

Which add-ons are 32bit only?

[jlaur]

Which add-ons are 32bit only?

Perhaps @wborn can remember? That sentence was added in openhab/openhab-docs#1067 (more than four years ago)

[wborn]

The Telldus binding mentions using 32-bit: /~https://github.com/openhab/openhab-addons/tree/main/bundles/org.openhab.binding.tellstick#telldus-core-bridge

Some dependencies like Netty and Google/AWS SDKs and anything using JNA also use native libraries. It is probably less of an issue nowadays because running a 64-bit ARM OS is more common nowdays compared to 4 years ago.

[J-N-K]

Netty provides AARCH_64, X86_64 and RISCV64 bundles for netty-transport-native-epoll and LINUX_AARCH_64_FEDORA, LINUX_X86_64_FEDORA, LINUX_X86_64, OSX_AARCH_64 and OSX_X86_64 bundles for tcnative.

[jlaur]

The Telldus binding mentions using 32-bit: /~https://github.com/openhab/openhab-addons/tree/main/bundles/org.openhab.binding.tellstick#telldus-core-bridge

@dabj123 - are you running 32 bit or 64 bit with the Tellstick binding? I found this almost six years old post where it seemed like 64 bit was working, but broke 32 bit support: https://community.openhab.org/t/oh2-tellstick-and-64-bit-debian/42469/17. However, that was with JNA > 4.1.0, and we are at 4.5.2 now (and passed 4.1 many years ago, #5335):

openhab-addons/bundles/org.openhab.binding.tellstick/pom.xml

Lines 23 to 29 in 51109f4

	<!-- JNA -->
	<dependency>
	<groupId>net.java.dev.jna</groupId>
	<artifactId>jna</artifactId>
	<version>4.5.2</version>
	<scope>compile</scope>
	</dependency>

[florian-h05]

As we decided to drop 32-bit support for openHAB 5, the slow performance issue on 32-but is gone, and as we have added pre-compilation of all scrips at startup, the delay when running a script the first time is gone.

I will therefore close this issue now as the issue that caused its creation will be fixed soon.
It would still be nice to get the Graal compiler work on a stock JVM with the JS Scripting add-on, but I neither see a need for this nor have any idea how/if this can be done in our OSGi environment.