Support building stand-alone programs

[lukego]

Add support for building stand-alone programs based on Snabb Switch.

This is (proposed to be) the new standard way to write programs based on Snabb Switch. You build your application as a single executable (instead of a separate script). This executable will be portable to all recent Linux distributions (see #359). Each program will have its own command-line syntax (based on unix-style getopt) and documentation (man page).

The default build will create one executable that can run all programs (like BusyBox). You can run a specific application with either:

./snabb myprogram args...

or equivalently:

ln -s snabb myprogram
./myprogram

and you can distribute as easily as:

scp snabb somehost:myprogram

The initial programs will be based on the designs/ scripts:

• snsh (snabb shell): The new name for the traditional ./snabb and #!snabb interface.
• packetblaster: The new name for loadgen.
• ... everything else in designs/ with a suitably catchy stand-alone name :-)

The current draft pull request extends the build with:

LINK      snabb
BINARY    980K snabb
PROGRAM   bin/snabb
PROGRAM   bin/packetblaster
PROGRAM   bin/snsh

and running a program looks like this:

$ sudo bin/snsh
Usage: snsh [OPTION]... [SCRIPT] [PARAMETER]...

Snabb Shell: Load the Snabb Switch core and execute Lua source code.

Execute SCRIPT if specified and then exit.

  -i,        --interactive   Start an interactive Read-Eval-Print Loop.
  -e EXPR,   --eval EXPR     Evaluate the Lua expression EXPR.
  -l MODULE, --load MODULE   Load (require) the Lua module MODULE.
  -t MODULE, --test MODULE   Test (selftest) the Lua module MODULE.
  -d,        --debug         Enable additional debugging checks.
  -j CMD,    --jit CMD       Control LuaJIT behavior. Available commands:
               -jv=FILE, --jit v=FILE
                 Write verbose JIT trace output to FILE.
               -jdump=OPTS[,FILE] --jit dump=OPTS[,FILE]
                 Output JIT traces, optionally to a file.
               -jp=OPTS[,FILE] --jit p=OPTS[,FILE]
                 Profile execution with low-overhead sampling.
             See luajit documentation for more information:
               http://luajit.org/running.html
  -P PATH,   --package-path PATH
             Use PATH as the Lua 'package.path'.
  -h,        --help          Print this usage message.

or:

$ sudo bin/packetblaster
Usage: packetblaster <PCAP-FILE> <PCI>...

Transmit packets from PCAP-FILE continuously to one or more network
adapters. The PCI arguments are Lua pattern strings that are used to
match the network adapters to use.

Examples:
  packetblaster myfile.cap 0000:01:00.0
  packetblaster myfile.cap 01:00

See mailing list thread.

[lukego]

I have pushed substantial updates and I hope to merge this onto master soon. (The changes touch so many files that it is a challenge to keep this branch compatible with master. I did a large and necessary merge from master today.)

Please review @eugeneia and anybody else who is willing and interested! Here is what is new today:

The whole designs/ directory is gone. The scripts have all been refactored into modules in programs/. (This includes the Neutron shell scripts!) See commit 49cadd6.

Makefile now automatically embeds files named *.inc into the snabb binary. (The Neutron shell scripts are bundled this way and so are the READMEs containing usage information.) See 3b8d459.

NFV related files have been refactored to live under programs/snabbnfv/ instead of more general locations (lib/nfv/ and test_fixtures/).

The vhost-user client app has been removed. This was bit-rotten and its C declarations of Linux ioctl data structures clash with ljsyscall. The app should be revived on top of ljsyscall. See commit e49a292.

Documentation structure is modified. As before, the public API code (core lib apps) is documented for Lua developers. However, the application code (program.nfv program.snsh ...) is now documented for end-users, in the style of man pages. This is proposed as policy: the programs are for non-programmer end-users while the APIs are for programmers writing programs. This implies that we should revise the program syntax (e.g. NFV config file) to be less Lua-centric (mea culpa). (I may have also lost some useful documentation e.g. from the NFV modules by not finding a suitable place to put it under this model.)

Question and comments welcome! (we can also throw this whole branch away if it is really a bad idea.)

[plajjan]

I quite like the idea of being able to compile and ship a contained binary that performs a specific duty. It obviously removes a lot of complexity for end users and seeing how Snabbswitch is more of a framework for developers, I think this is a great move. I haven't read through all of the code but did have a peek at the packetblaster app. What I do wonder is if we sacrifice flexibility through this? Can I still specify an app network the classic way?

I'm still not sure how I will deploy Snabb in real production environments and how static/dynamic those environments will be, but let's say I have some NC/YANG integration for Snabb so that I can describe a Snabb design, complete with links, apps and configuration for those apps. Would I be able to deploy a Snabb-instance and configure a app network through YANG just the way I want it or would I need to build a specific binary for that and map some form of input to that? I think YANG is easier to map to the config language (albeit I agree that it's a tad Lua-centric) than to pure command line arguments and having a compile step in between app network config and deployment is obviously rather cumbersome.

[lukego]

Thank you for the great question!

You should now be able to decide how generic your binary is. The extremes are packetblaster that does one narrow task and snsh that is the same full-blown Lua scripting interface that we have always had. You could ship snsh and do the rest with scripts if you want to. You could also create something medium-generic e.g. a new front-end that reads a YANG-based config file and translates to Lua internally. (That is kind-of how the snabbnfv command works except with YANG.)

You can still run your old scripts/designs with snsh <script> or snabb snsh <script> so there is no loss of functionality in this new style. The command-line syntax is not backwards compatible though. (Maybe we should deprecate the snabb <script> syntax instead of removing it? I'm not sure.)

[plajjan]

Ok, cool. Then I'm all for!

Again, I haven't looked much at the code but from a high level functional perspective I like it a lot!