wasm2c support: compile everything into a single C file #11213

kripken · 2020-05-21T23:57:32Z

This new option makes us run wabt's wasm2c, combine the output
with an emscripten main() file + runtime in C, and emits a final
single C file of all that stuff. If that is built to a native executable it
does the same thing as the wasm(+js) would have done.

This has several potentially interesting use cases:

If we can build the emsdk tools (llvm, clang, binaryen, etc.) to C files,
we can distribute those to users as an alternative for people that
can't use the emsdk builds for some reason. For example if binaryen
upgrades to c++20 but someone is on a linux distro that is too old for
our standard builds, and doesn't have a c++20 compiler, then giving
them a C file may help them.
A C build is also easier to generate. We have infrastructure for
building on the 3 major OSes but it takes a bunch of work. A C build
could be generated on a developer's machine very easily and run
on all targets. This is the same as making a build that runs in node.js,
which also would have those benefits. A C build has different
tradeoffs, the main benefit being that it may be faster and in particular
faster to start up (in theory node with wasm code caching could be
similar, but that doesn't work atm - hopefully it will be possible eventually).
C builds are also interesting for sandboxing untrusted C code in
a simple way.

Note that this PR uses an emscripten build of wasm2c, running on
node, instead of us needing to package normal binaries of wabt. That was
super-easy to do, and maybe we can consider it for binaryen or llvm
eventually (but for performance reasons, a wasm2c build may be
preferable there). It's an example of how the second item in that
bullet list can be helpful!

sbc100 · 2020-05-23T20:21:13Z

tools/shared.py

@@ -2781,6 +2781,108 @@ def run_binaryen_command(tool, infile, outfile=None, args=[], debug=False, stdou
  def run_wasm_opt(*args, **kwargs):
    return Building.run_binaryen_command('wasm-opt', *args, **kwargs)

+  @staticmethod
+  def do_wasm2c(infile):


I don't like adding stuff to shared.py like this because:

(a) this is not shared.. it has exactly one call site
(b) shared.py get included absolutely everywhere.. we should try to keep it minimal.

I know that we are long way from (b) and we have history of throwing stuff in shared.py .. but I don't think thats good reason to make it worse. I still like the idea of wasm2c.py which exactly one function in it.

Yeah, I think I see what you mean. I moved it to a separate file now, let me know what you think.

kripken · 2020-05-25T01:04:30Z

tools/wasm2c/__init__.py

@@ -0,0 +1,111 @@
+# Copyright 2020 The Emscripten Authors.  All rights reserved.


I don't care much about the location of the new file, but a benefit to wasm2c/__init__.py is that it won't mess up autocomplete (which having both wasm2c.py and wasm2c/*.c would do).

It think just having tools/wasm2c.py is fine.

I don't think we should worry too much about auto-complete when naming files.

Ok, renamed as you requested.

kripken · 2020-06-05T15:42:52Z

ping @sbc100

sbc100 · 2020-06-05T18:53:28Z

tests/other/wasm2c/my-code.c

+
+extern int (*Z_do_bad_thingZ_ii)(int);
+
+extern int (*Z_twiceZ_ii)(int);


I find it clearer to just use C ABI for these kinds of tests to avoid the name mangling.

That's name mangling that wasm2c does. cc @binji

The Z_ prefix might make it look too much like C++ mangling, which can be confusing. I wonder if it would be worth considering W2C_ or such?

sbc100 · 2020-06-05T18:54:12Z

tests/other/wasm2c/unsafe-library.c

+#include <stdio.h>
+
+EMSCRIPTEN_KEEPALIVE
+int twice(int x) {


This file is a called ".c" but the name mangling looks like its being compiled as C++?

tests/other/wasm2c/unsafe-library.c

sbc100 · 2020-06-05T18:55:32Z

tests/runner.py

@@ -1231,6 +1235,9 @@ def do_run(self, src, expected_output, args=[], output_nicerizer=None,
      if len(wasm_engines) == 0:
        logger.warning('no wasm engine was found to run the standalone part of this test')
      js_engines += wasm_engines
+      if self.get_setting('WASM2C'):
+        # the "engine" to run wasm2c builds is clang that compiles the c
+        js_engines += [[CLANG_CC]]


This seems odd. Maybe "js_engines" needs to be renamed?

Good idea, I'll look into that as a followup.

sbc100 · 2020-06-05T18:56:29Z

tests/test_core.py

@@ -345,7 +392,6 @@ def get_bullet_library(self, use_cmake):
                            configure_args=configure_args,
                            cache_name_extra=configure_commands[0])

-  @also_with_standalone_wasm


What delete this?

test_hello_world is extremely basic, and we test far more interesting things, so it's redundant.

also, we use test_hello_world in a few places as a sanity check. it can be confusing to have complicated things like standalone happen there.

I like the idea of being able to test simple things... now if I want to test the most basic standalone program I need to look for a more complex test?

What do you think about make also_with_standalone_wasm expand to @parameterized and turn the tests into two different tests?

Then test_hello_world_standalone can fail separately to test_hello_world_standalone (I recently made the parameterized thing handle empty string as the name of the variant correctly).

In any case this change is unrelated to wasm2c and doesn't really belong in this PR.

Ok, I restored that test now.

We could also use parametrized, yeah. I can look into that.

sbc100 · 2020-06-05T18:59:00Z

tools/wasm2c/os.c

+  return 0;
+});
+
+IMPORT_IMPL(u32, Z_wasi_snapshot_preview1Z_fd_readZ_iiiii, (u32 fd, u32 iov, u32 iovcnt, u32 pnum), {


Is this name mangling part of the way wasm2c works?

tools/wasm2c/reactor.c

sbc100 · 2020-06-05T22:30:03Z

tests/test_core.py

@@ -345,7 +392,6 @@ def get_bullet_library(self, use_cmake):
                            configure_args=configure_args,
                            cache_name_extra=configure_commands[0])

-  @also_with_standalone_wasm


I like the idea of being able to test simple things... now if I want to test the most basic standalone program I need to look for a more complex test?

What do you think about make also_with_standalone_wasm expand to @parameterized and turn the tests into two different tests?

Then test_hello_world_standalone can fail separately to test_hello_world_standalone (I recently made the parameterized thing handle empty string as the name of the variant correctly).

In any case this change is unrelated to wasm2c and doesn't really belong in this PR.

kripken and others added 30 commits May 20, 2020 11:28

main

da18e44

start

c208a6d

more [ci skip]

802b60c

wip [ci skip]

2daf554

fixes [ci skip]

7c0c920

working [ci skip]

38bc2b5

test passes [ci skip]

8114412

another test

8bdf87e

another [ci skip]

cfbc317

another test [ci skip]

e8ad395

more

efd17ee

works with js [ci skip]

5d8fc96

fixes [ci skip]

e555a04

benches [ci skip]

6fcb454

fix [ci skip]

fa81d58

fix [ci skip]

7761591

temp [ci skip]

f04b0e3

fixes [ci skip]

9b7f095

fix

eab9809

cleanup [ci skip]

3639b36

autogen invokes [ci skip]

dd0bdb3

npm

20eda6f

cleanup

67c68f4

more

3f0d8dc

better

6e5bbfc

fixes

c445c0e

nicer

f450a40

nicer

649d7dd

fix suffix [ci skip]

2e449e5

splitting

30c1754

kripken added 4 commits May 23, 2020 10:33

more [ci skip]

56ffcf3

Merge remote-tracking branch 'origin/master' into wasm2c

8d65523

flake8

d490cf9

flake8

73924dd

sbc100 reviewed May 23, 2020

View reviewed changes

kripken added 2 commits May 24, 2020 14:10

separate out

cd1bfbf

flake8

83502a5

kripken commented May 25, 2020

View reviewed changes

kripken added 2 commits May 25, 2020 16:13

rename

5f3abb9

Merge remote-tracking branch 'origin/master' into wasm2c

29fd4fd

kripken requested a review from sbc100 May 28, 2020 20:11

kripken added 6 commits May 28, 2020 13:14

Merge remote-tracking branch 'origin/master' into wasm2c

6532c11

simplify

33f804a

Merge remote-tracking branch 'origin/master' into wasm2c

d0b9a91

Merge remote-tracking branch 'origin/master' into wasm2c

b1f5fa9

fix after merge

4028cab

fix

fe05819

sbc100 approved these changes Jun 5, 2020

View reviewed changes

kripken added 2 commits June 5, 2020 14:41

cleanup

5f99dc5

Merge remote-tracking branch 'origin/master' into wasm2c

b07152d

sbc100 approved these changes Jun 5, 2020

View reviewed changes

restore test

8dbc4d4

kripken merged commit fb72b87 into master Jun 5, 2020

kripken deleted the wasm2c branch June 5, 2020 23:36

kripken mentioned this pull request Jun 16, 2020

Should getenv() map to process.env in node.js? #11422

Closed

This was referenced Jul 28, 2020

WasmBoxC: Simple Easy and Fast VM-less Sandboxing guevara/read-it-later#6937

Open

WasmBoxC: Simple Easy and Fast VM-less Sandboxing guevara/read-it-later#6938

Open

WasmBoxC: Simple Easy and Fast VM-less Sandboxing guevara/read-it-later#6939

Open

juj mentioned this pull request May 15, 2023

Should wasm2c be a devDependency instead? #19367

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

wasm2c support: compile everything into a single C file #11213

wasm2c support: compile everything into a single C file #11213

kripken commented May 21, 2020

sbc100 May 23, 2020

kripken May 25, 2020

kripken May 25, 2020

sbc100 May 25, 2020

kripken May 25, 2020

kripken commented Jun 5, 2020

sbc100 Jun 5, 2020

kripken Jun 5, 2020 •

edited

Loading

sbc100 Jun 5, 2020

sbc100 Jun 5, 2020

kripken Jun 5, 2020

sbc100 Jun 5, 2020

kripken Jun 5, 2020 •

edited

Loading

sbc100 Jun 5, 2020

kripken Jun 5, 2020

sbc100 Jun 5, 2020

kripken Jun 5, 2020

sbc100 Jun 5, 2020

		@@ -0,0 +1,111 @@
		# Copyright 2020 The Emscripten Authors. All rights reserved.


		extern int (*Z_do_bad_thingZ_ii)(int);

		extern int (*Z_twiceZ_ii)(int);

wasm2c support: compile everything into a single C file #11213

wasm2c support: compile everything into a single C file #11213

Conversation

kripken commented May 21, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kripken commented Jun 5, 2020

Choose a reason for hiding this comment

kripken Jun 5, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kripken Jun 5, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kripken Jun 5, 2020 •

edited

Loading

kripken Jun 5, 2020 •

edited

Loading