feat(ses): Shim compatible with Hermes compiler

[leotm]

Description

• Refs: SES lockdown Hermes compat tracking issue #1891
  • SES shim compatible with Hermes compiler
  • Fixes: Support Windows in Hermes scripts #2474

Add lockdown shim compatible with Hermes
Make current shim compatible with Hermes compiler
for building React Native (RN) prod apps with SES

Generating the release AAB (Android App Bundle)
i.e. npx react-native build-android --mode=release via RN CLI
calls Gradle's bundleRelease task under the hood (bundle build task on release variant)
which calls RNGP (React Native Gradle Plugin) React task createBundleReleaseJsAndAssets and fails
after Metro finishes writing the release bundle (bundle, sourcemaps, assets)

Gradle emits these Hermes errors

• async functions are unsupported
  • Feature request: Improved error async arrow functions are unsupported facebook/hermes#1395
• async generators are unsupported

(at runtime we can see both are SyntaxErrors)
Resulting in vague java.lang.StackOverflowError (no error message)

The try/catch approach testing language ft support via a new fn works
since RNGP no longer emits the Hermes errors in the task after Metro bundles
and we're conditionally using parts of SES compatible with the Hermes engine

The initial approach involved building a new shim via an env var
which involved a lot of duplicates /src files

Nb: clean before bundling to see changes reflected (avoid cache)
i.e. ./gradlew clean :app:bundleRelease

Nb: async function* a() {}; alone won't emit an error
but using/referencing it and beyond const b = a; will

Nb: eventually we hit RNGP BundleHermesCTask.kt > run > detectedHermesCommand > detectOSAwareHermesCommand from PathUtils.kt, which calls the Hermes compiler default command hermesc - the path of the binary file, to create the optimised bytecode bundle to load/exec at runtime

TODO

• use standard async functions without arrows (module-load.js)
• remove async generators from whitelist (permits.js)
• remove async generators from intrinsics (get-anonymous-intrinsics-hermes.js)
• remove repairing async generators during taming (tame-function-constructors.js)
• remove async generator tests (ses, harden, tame-function-unit)
• ensure 379 ses tests passing
• test new shim in React Native repro
  • test android bundle via gradle: ./gradlew :app:createBundleReleaseJsAndAssets
  • test android build/runtime via gradle: ./gradlew :app:installRelease -PreactNativeArchitectures=arm64-v8a
  • test android/ios build/runtime via RN CLI: yarn <android/ios> --mode release
• add new entry points to hook into get-anonymous-intrinsics-hermes.js
  • original: bundle.js > index.js > lockdown-shim.js > lockdown.js
  • hermes: bundle.js > index-hermes.js > lockdown-shim-hermes.js > lockdown-hermes.js
• support env var to bundle new ses/lockdown shim bundle/terse variants (bundle.js)
• bundle new shims in prepare npm lifecycle script (package.json)
• add new shim exports entry points (package.json)
• conditionally get intrinsics (get-anonymous-intrinsics.js)
• conditionally repair async generators (tame-function-constructors.js)
• conditionally add async generators to whitelist (permits.js)
• revert new publishing changes
• revert new entry points
• revert new bundling with env var changes
• ensure 379 ses tests still passing
• restore original tests
• suggested change
• revert permits.js changes
• asyncTrampoline non-arrow async function
• console info msg x2
• improved catch block
  • do nothing only on the SyntaxError
  • otherwise re-throw
• add to commons.js and use
  • AsyncGeneratorFunctionInstance
  • AsyncArrowFunctionInstance?
• CI (new PR, stacked)

Follow-up, CI testing options discussed

• ubuntu: RN app + SES, cd android && ./gradlew :app:bundleRelease
  • add Java CI docs if included
• react-native-community/docker-android is available to use
• CI(macos): RN app test + SES, Xcode release
  • not needed, Xcode does not emit Hermes errors like Gradle
• test262:hermes script (like test262:xs)
• Hermes maintains test262 tests (testsuite.py, testsuite_skiplist.py)
• (eshost) > Hermes CLI v0.12.0 (few years old, newer being discussed)
• or build and run Hermes (and/or Static Hermes) as a standalone compiler and VM
  • add docs or at least link to Hermes docs

Security Considerations

Does this change introduce new assumptions or dependencies that, if violated, could introduce security vulnerabilities? How does this PR change the boundaries between mutually-suspicious components? What new authorities are introduced by this change, perhaps by new API calls?

Scaling Considerations

Does this change require or encourage significant increase in consumption of CPU cycles, RAM, on-chain storage, message exchanges, or other scarce resources? If so, can that be prevented or mitigated?

Documentation Considerations

Give our docs folks some hints about what needs to be described to downstream users. Backwards compatibility: what happens to existing data or deployments when this code is shipped? Do we need to instruct users to do something to upgrade their saved data? If there is no upgrade path possible, how bad will that be for users?

Testing Considerations

Every PR should of course come with tests of its own functionality. What additional tests are still needed beyond those unit tests? How does this affect CI, other test automation, or the testnet?

Compatibility Considerations

Does this change break any prior usage patterns? Does this change allow usage patterns to evolve?

Upgrade Considerations

What aspects of this PR are relevant to upgrading live production systems, and how should they be addressed?

Include *BREAKING*: in the commit message with migration instructions for any breaking change.

Update NEWS.md for user-facing changes.

Delete guidance from pull request description before merge (including this!)

[leotm]

ready for review ^ tackling the testing strategy separately in a follow-up PR, to keep this change small

[kriskowal]

This is indeed far more surgical than I expected.

I only request that we capture the async generator function instance in a single try{eval}catch in commons.js so we can just use if blocks to decide whether to include async generators in the various points you’re currently using try/catch.

I would like to make sure @erights looks at these changes as well.

[kriskowal]

ready for review ^ tackling the testing strategy separately in a follow-up PR, to keep this change small

Are you familiar with stacked PRs? You can propose the test changes in a PR based on this branch so they can be reviewed together. I would hesitate to approve code that doesn’t come with tests in the same merge, though I’m fine with reviewing them separately. Much depends on your workflow.

One workflow that I like is individually reviewable commits in a single PR. That does require more commit grooming, and it looks like you intend to squash the 21 commits here when you merge.

Another workflow is to create “stacked” PRs for each review artifact, then merge them top to bottom, so that ultimately the feature and its tests arrive in master with a single merge commit.

[erights]

Glad to see this!

I would like to make sure @erights looks at these changes as well.

Hopefully tomorrow.

[leotm]

What is the status of this? Is it still needed?

yes this is still needed for CI feedback keeping SES compatible with Hermes
after rebasing off master i've added 7b9eca2 for now (tbd)
otherwise 5 failing tests left to resolve and #2334 (comment) (tbd) before ready for final re-review/merge
(unless any further comments on the open threads)

I plan to add draft release notes to NEWS.md separately after the last 2 issues are addressed (removeUnpermittedIntrinsics and assertDirectEvalAvailable) which close #1891 and enable LavaMoat/LavaMoat#1438

[SMotaal]

LGTM 👍