App Testing

Test Driven App Development

The holochain system includes a testing harness that allows you to test your application functions. In addition we have built integration that simultaneously runs a set of tests in separate virtual nodes. We strongly recommend using both these tools for application development.

Test Files

A test file is essentially a collection of function calls into your application with specified inputs and expected outputs, or error values. A test file consists of an array of json test objects of the form:

[{
    Convey: string      // a human readable description of the test's intent
    Zome:   string      // the zome in which to find the function (not necessary if test is in a zome test directory)
    FnName: string      // the application function to call
    Input:  string      // the function's input
    Output: string      // the expected output to match against (full match)
    Err:    string      // the expected error to match against
    Regexp: string      // the expected out to match again (regular expression)
    Time:   int         // delay in millis before running this test (useful for multi-node testing)
    Raw:    bool        // set to true if we should ignore FnName and just call input as raw code in the Ribosome, useful for testing helper functions and validation functions, or doing more complex testing
 },
 {
   ...
 }
]

Integrating tests into your holochain application

A holochain application directory (with the test directory expanded) should look something like this:

• my_app
  • dna
  • ui
  • test
    • singleNodeTest.1.json
    • singleNodeTest.2.json
    • scenario.myScenario.1
      • role1.json
      • role2.json
      • ...
      • roleN.json
      • _config.json
    • scenario.myScenario.2
      • etc...

Single-node tests

Assuming your current working directory is my_app you can run all the single-node tests (test/*.json) with this command:

hcdev test

This command loads a new copy of my_app into the ~/.holochaindev directory, and then finds all the .json extension files in the test directories and runs them in a random order. Not that this doesn't run the test files in the scenario directories, only the top-level test files.

You can run a single test file with: hcdev test <test-name> (note you don't need the .json) extension.

Muilt-node tests (scenarios)

Multi-node tests are can be though of as a scenario with a number of roles, where each role represents one node in the multi-node test, and is defined by a test file in the scenario sub-directory.

To run a mulit-node scenario test, use this command:

hcdev scenario <your-scenario-directory-name-here>

Video showing tests being run (doesn't open new tab, maybe ctrl-click or w/e)

Each test/scenario sub-directory should contain one test file for each role required to model the test. The scenario command requires the name of the scenario directory [my_app/test/<scenarioName>] as a parameter. Test filenames are automatically discovered.

Tests pass or fail on the basis of the content of messages passed between roles/nodes on the network. In order to test the content of messages passed between roles, it is necessary for tests to account for the amount of time it takes for messages to travel between nodes on the network. This is achieved with the Time parameter of the test object. If roles 1 and 2 are called back and forth respectively, then when forth sends a message, back should wait at least 50ms for checking to see if there are messages that contain the expected content. If the message has not yet arrived, then the test will fail.

You can add a _config.json file to the scenario directory of the following format:

{
    "GossipInterval":500,
    "Duration":3,
    "Clone":[{"Role":"sender","Number":20}]
}

This config file allows you:

• set the gossip interval for your test
• a minimum duration that all the nodes should be kept running for the test to succeed, i.e. so other nodes that still have tests pending can gossip with them.
• a list of roles for which you want multiple clones to run at the same time, plus the count of clones of that role.

In the chat sample application you will find a backnforth scenario with two roles defined: person1 & person2. Check out these examples to see how things work.

Notes on the Time parameter:

• if Time is null or 0, tests are executed in the order discovered in the test file
• all other values of the Time parameter are used in multi-node testing to allow sufficient space for gossip operations between nodes on the DHT.
• to test a sequence of messages sent and received between holochain nodes, between 50 to 200 millis is required for message arrival.
  • role1.json >> "send message, Time = 0"
  • role2.json >> "check for message, Time = 100"
• the 50 to 200 millis number works well within the test harness network. Tests crossing external networks may require much more time for messages to be delivered.

TODO Note, it might make sense to implement both an automatic backnforth test generator, and also an asynchronous method of testing for the test message arrival (e.g. a message id)