Running Unit Tests in CodeSandbox

[gautamarora]

After some early conversations with @CompuIves in Discord, I'm starting this issue to begin discussions for adding support to run unit tests in CodeSandbox.

What i was thinking was a way to use codesandbox as a tool to support "learn by doing" environment where others can learn JS/React or any other topic by writing code to pass failing tests. There are other coding environments that allow this like tddbin and there are curriculums like nodeschool, js-assesment, es6-katas that exist; my proposal is to provide support for creating such a setup in codesandbox.

I was trying out something simple that would work for me given the current codesandbox setup, and created this: https://codesandbox.io/s/716lx85wk6

Baking in something like this that is native to the codesandbox site would make it really fun for folks to create and complete coding workshops.

Happy to hear thoughts from everyone and work together on it if this is interesting.

[CompuIves]

Nice! I was thinking that we could detect all files that have a .test.js or .spec.js. We could run these files using jest-matchers, which works in the browser.

Then we can show all results in a tab besides Console (under the preview).

What do you think?

[gautamarora]

@CompuIves thats a really good idea for native integration.

In my earlier Test component approach (sandbox here), the tests would need to be written in a new style but in the approach you suggest, we would be able to have tests written and viewed in a more real-world setup.

I was quite easily able to switch using expect from chai to jest-matchers - thanks for that tip!

However, the next hurdle would be to run jest in the browser, something that is still being discussed and not resolved (here, here and here). I tried to stub the features we would need from the jest runner over to something simple and created this sandbox as a demonstration of the end result:
https://codesandbox.io/s/2jxjmoj1lp

In order to see the output in console (as in screenshot above) currently, the preview has to be switched to current module view, which I think can be worked out in the unit test integration feature.

Instead of using Console, we can move over the results to a dedicated Tests tab.

Thoughts?

[CompuIves]

Wow, this is really great @gautamarora. I completely agree with the approach. Let's start with a simple version, like your runner.js, and put the results in a separate Tests tab. I think a good benchmark would be to get the tests in the redux example running.

In order to see the output in console (as in screenshot above) currently, the preview has to be switched to current module view, which I think can be worked out in the unit test integration feature.

Yes. We could first evaluate the sandbox itself, then evaluate all test files by finding modules with the name (or maybe do these two tasks concurrently?). Then aggregate the results and send them to the tab. We communicate with messages between te preview and the editor, using codesandbox-api. An example is here: sending and receiving.

What do you think of the approach?

[gautamarora]

@CompuIves agree with everything above 👍

Exciting updates...

I forked the redux example you shared, imported the tester/runner.js into the test files and the tests in actions and reducers just worked!
Check it out here: https://codesandbox.io/s/wn6o8mmx8l

Only the tests in components do not work, i'll look into why later today.

As next steps I will be figuring out the actual integration of the tester into codesandbox-client code & sandbox processes.

[gautamarora]

Needed to add jest-mock to runner. Tests in components pass as well now!

[CompuIves]

Wow!! This is really cool. So the next thing would be integrated evaluation, right?

For that we'd need to evaluate the code with globals describe, it, test and expect set I reckon. We have manager.evaluateModule /~https://github.com/CompuIves/codesandbox-client/blob/master/packages/app/src/sandbox/eval/manager.js#L113 that we could use. But we should add functionality to specify globals. I can do this, so you can specify an object like this:

{
  it,
  expect,
  // etc
}

We could start evaluating all tests module after this: /~https://github.com/CompuIves/codesandbox-client/blob/master/packages/app/src/sandbox/compile.js#L188.

Then let the test runner send the messages which the DevTools can respond to. So the first thing we'd need to do is move runner to a file in the sandbox. Something like sandbox/tests/runner.js.

Would you be interested in opening a PR that we can work on 😄 ?

[gautamarora]

yes, next step is integrated evaluation. Will get started on a PR for us to work on 👍

[gillchristian]

Wow this is really cool! 💪 🚀

I've been working on a couple of projects lately that involved a lot of helpers and utils so I wanted to test them but at the same time I did not want to leave CodeSandbox.

I'd like to help (if needed of course). I'm not really familiar with CodeSandbox internals but this could be a great chance to get into it 😉

[DisasterMan78]

This looks absolutely fantastic.
It would be amazing if the feature could be developed to work with other testing libraries added as dependencies. I came across codesandbox today and am seriously impressed, but had hoped to create a reduced test example to share and get some help with testing issues I'm having - converting my tests would be impractical, and not very helpful for getting help with my actual dev environment.

I'm relying on chai, sinon, enzyme and babel-plugin-rewire, which would seem to complicate things enormously.

Would this even be possible in browser?

I appreciate this may be a long time coming, if ever, but wanted to add my congratulations and encouragement, nonetheless.

[gautamarora]

Thanks for the votes of excitement @gillchristian & @DisasterMan78. I am glad that you find this early work exciting and in a direction that will be useful for you in your projects.

@gillchristian - I'll take a shot at starting the native integration for unit testing in code sandbox based on the notes from Ives above, and create a PR to collaborate on next week. I'm sure there will be enough work in this PR for both of us to learn and contribute to CodeSandbox!

@DisasterMan78 - I do think it will be technically feasible to extend this feature to work with set of other testing libraries in the future 👍

[marcinczenko]

I am getting an error when trying to run tests in components:

Warning: React.createElement: type is invalid -- expected a string (for built-in components) or a class/function (for composite components) but got: undefined. You likely forgot to export your component from the file it's defined in.

[marcinczenko]

Great work guys!

I like the solution for its simplicity, but I am also wondering if the tests cannot be run more "natively".
For me the most important value of the CodeSandbox is that each sandbox follows the create-react-app pattern, which means you can really develop production code using CodeSandbox. Running tests should IMHO be handled in the same idiomatic way without having to resort to some "hacks".

I do not yet know the architecture of CodeSandbox, but I would imagine that there is something spinning a create-react-app-based build process dedicated to each sandbox (maybe this is not the case, just guessing for the moment). The proposed solution has one disadvantage: now when you export the app, your testing code is different from the one you would normally write. I mean, in the normal flow, your would not have to do:

import { describe, it, expect, jestMock } from "../../tester/runner";

or anything like that, not to mention you should not need the ./../tester/runner in the first place.

Also, when you start doing snapshot testing, I am not sure if you can handle that on purely on the client side in an elegant way.

I hope to find some time to dig into the code, so that I will be better informed and will not ask potentially silly questions. For now, I am just wondering. I am also willing to help, if time allows.

[CompuIves]

I completely agree @marcinczenko, for that reason we want to build the test functionality right into CodeSandbox, so you don't have to do:

import { describe, it, expect, jestMock } from "../../tester/runner";

etc. The given demo by @gautamarora was more of a way to show that it's possible, and as a pointer on how to implement it natively on CodeSandbox 😄

[marcinczenko]

That's great. As I said I still miss basic understanding of how CodeSandbox works. That's why I thought that it is obvious that you can implement it natively and it looked obvious to me that it would be completely different from the proposed solution, which in turn is why I do not see in which does it shows that something obviously possible is possible when what is obviously possible would have nothing in common with proposed solution 😄😄😄😄😄.

But seriously, from what learned so far, transpiration is happening on the client side - I thought initially that you run webpack on the server. Your approach to transpilation and packaging are impressive!

Is there anywhere a branch on which you work on this, so that I could take a look? I could not find any fork at @gautamarora, maybe I just do not see it.

[gautamarora]

@marcinczenko - TBH i have been trying to wrap my head around how such an integration would look and the demo above was me trying to understand how things could fit together. My very first implementation of trying to implement testing in CodeSandbox was very very non-native as you can see here; then the the idea of native integration was suggested by Ives and I did this where the main thing I was trying to figure out was if we can run jest tests in the browser since there is no official support for that yet and once that demo worked, I tried to apply it to the redux todo app here. Hopefully you can see that this was me trying to do a feasibility study of the feature :)

A PR for this will be coming this week (though it will not be complete, but show the early approach of native integration)

[gautamarora]

I am getting an error when trying to run tests in components

@marcinczenko yes, the browser preview UI currently shows the warnings, but if you open up the Console and scroll to top then you should see the test passing (let me know if you dont). Since these were warnings and not errors, I did not dig deeper into them yet but will certainly review them when we are working on native integration.

[gillchristian]

@gautamarora this is an awesome work you are doing 💪

I like the approach you are taking to support something like a "jest lite" (exposing describe, it, expect, jestMock) on CodeSandbox. If more advanced features are required the user can eject and keep working with the full fledged Jest locally. And of course we (I say we as if I were doing something already haha) could iterate on enabling more features.

[gillchristian]

Btw, @kentcdodds sorry to ping you but I believe you would like this 😉

[marcinczenko]

@gillchristian:

If more advanced features are required the user can eject and keep working with the full fledged Jest locally.

From my current perspective (I am trying to convince people locally to use CodeSanbox for actual project and production code), the value of CodeSandbox will be much lower if you cannot continuously switch between local installation and the Sandbox. I am sure we will get there and of course, having subset of Jest temporarily is ok, but I hope CodeSandbox will be able to provide experience that is not only identical, but in many ways even better than local installation. For me this project shows that a lot more is achievable than I expected.

@gautamarora Super. Thanks a lot for sharing this. This is great start!

[marcinczenko]

@gillchristian Hmm...I learned about CodeSandbox from an interview with @kentcdodds on EggHead :).

[gillchristian]

the value of CodeSandbox will be much lower if you cannot continuously switch between local installation and the Sandbox

Totally agree with that. But is better to start with something and then as I said, iterate on it and to aim for something really ambitious and the not be able to complete it.

For me this project shows that a lot more is achievable than I expected.

Yes, indeed!

[kentcdodds]

Yes I am excited by this! The other day I built something at a meetup with some folks in codesandbox and they wanted to learn about testing, so I built this: https://codesandbox.io/s/l5no8yznmq

It would be super to have a "Jest Lite" built-in.

There's been a lot of great discussion here. Let's not forget that CRA supports finding test files in a __tests__ directory as well 👍

Also, in the future if snapshots could be supported that would be amazing :)

[marcinczenko]

@gautamarora Perhaps something that can be helpful. There is something called jest-editor-support, jest-editor-support/src. This package is used as a dependency in vscode-jest which is a jest extension to Visual Studio Code. I was working on creating JestProcess abstractions (my pull request is still waiting to be merged), so I kind of know how it works. Out of the box we can run all the tests having feedback on the results, we can easily start jest in the watch-mode, and we can even update snapshots. Perhaps we have to look at the jest to see if there is a way to feed it with our tests instead of watching local file system. I feel this should be possible. What do you think? I can take a look if this is something you did not try yet...

BTW: I also hope this would also allow us to use standard ways of configuring jest, including watching the __tests__ directory ;).

[gautamarora]

@marcinczenko that sounds like a very interesting idea, and yeah if we can feed it test results differently it could work. I havent tried it (or knew of it), so please go ahead and try it out 👍

[gautamarora]

@kentcdodds - the tic tac toe example is really cool. Getting to learn a lot about Code Sandbox, Jest and CRA :)

[CompuIves]

That would be super @marcinczenko! We already do the watching, so would be really cool to feed the results in Jest for every update.

[gautamarora]

@CompuIves @gillchristian @DisasterMan78 @marcinczenko - based on the pointers and discussions above, we have finally have a PR to collaborate on #445

Here is a screenshot of the project being scanned for tests and selected, being transpiled and executed.

Though we need some jest variables (it, expect etc) to be added to globals before the tests actually start getting executed 😝

[gautamarora]

Latest screenshot with tests being run and sent to the Test Manager...

[lbogdan]

@gautamarora Just wanted to let you know you can access that PR at http://pr445.cs.lbogdan.tk/s/new (replace new with your sandbox id, it also updates as you add commits).

[marcinczenko]

@gautamarora Nice! Thanks to this pull request it is faster for me to learn where things are happening ;).

@CompuIves I will try to experiment with native integration with jest via jest-editor-support. Is it ok that I create a separate issue for that to keep the relevant discussion (if any) easier to track?

[gautamarora]

@lbogdan - wow! 😮 Thanks for that tip!

Demo for native integration using jest lite is here: http://pr445.cs.lbogdan.tk/s/4jw49300kx
Test output will be in console. Give it a spin everyone and let me know how it goes!

[gautamarora]

@marcinczenko - 🙇 i'm glad its helpful! many thanks to @CompuIves for pointing me to the right places 😄

i would agree on a separate issue for the native integration with jest-editor-support experiment.

[gautamarora]

Happy to report that all tests pass for React Redux TodoMVC App using the native integration!
http://pr445.cs.lbogdan.tk/s/github/reactjs/redux/tree/master/examples/todomvc

[gautamarora]

Update: Support for describe blocks, formatted test summary (like jest cli)

[gautamarora]

Updated Demo URL: http://pr458.cs.lbogdan.tk/s/github/reactjs/redux/tree/master/examples/todomvc

[gautamarora]

early release of jest unit testing in codesandbox has landed! https://codesandbox.io/s/github/reactjs/redux/tree/master/examples/todomvc

@gillchristian @kentcdodds @DisasterMan78 @marcinczenko - give it a spin if you can. want to resolve any major bugs before next codesandbox version release.

[kentcdodds]

Cool! It's a great first-step!

I think the most important next step would be to improve the error displayed.

I think it would also be great to get a dedicated UI for tests so it shows the error messages and you can focus on the tests.

This is cool. Thanks for working on it @gautamarora!

[marcinczenko]

@gautamarora I will. I've just created an issue #462 where I will start adding Developer Documentation. I will put some initial docs on how Unit Tests are currently implemented (I hope @gautamarora will able to correct me where I am wrong).

[gautamarora]

@kentcdodds - totally agree. Dedicated UI is coming real soon 😄

@marcinczenko - 😮 thats awesome!! so exciting.

[gillchristian]

It's already merged 😱 Really nice work @gautamarora !!

Please let me know if you want some help in the UI part. I was thinking we could toggle the runner so test aren't running all the time, and maybe some way of manually triggering the run, and Kent's suggestion of a dedicated UI would be awesome too.

[gautamarora]

@gillchristian - thank you! everyone was way too encouraging, had to give it a shot!! 🙇

For the UI, having a toggle to make it run/stop is a good idea vs tests running all the time (even though they are super fast to run!). @CompuIves is working on the dedicated UI in the redesign branch. I am excited to see what he has in the works!!

As next steps on the testing side, I want to take a look into jest-circus and see if we can incorporate that as it will give us support for 1) async running for tests 2) beforeEach, afterEach etc

[CompuIves]

@cpojer would certainly be interested in this!

[cpojer]

Woah, this is cool.

[gautamarora]

A fully baked Jest Integration shipped with CodeSandbox 2.5 yesterday!

@CompuIves added a shiny new UI (he really knows how to interfacivize things!), jest-circus to make tests run async and jest-snapshot for snapshot testing.

I take great pleasure in closing this issue as we have taken an exploration to its end goal and shipped the feature! I will create a new "Jest in CodeSandbox 3.0" issue where we can discuss the evolution of this integration and features that should be part of the next release.

[apieceofbart]

@gautamarora Is there a way to add enzyme to testing? I was trying to use create-react-app but it need to setup the adapter for react 16 and it looks like it's not being picked up.

[CompuIves]

Hey @apieceofbart, do you have an example sandbox I can test?

[gautamarora]

@apieceofbart - checkout this sandbox as an example for adding enzyme for testing https://codesandbox.io/s/009pr18q10
specifically, take a look at the setupTests.js

Let me know if that doesn't work.

[unleashit]

@gautamarora just tried your link in both Chrome and Firefox and for me it's stuck forever on "Running Test Suites...". In my own sandbox, the problem happens as soon as I add setupTests.js. When I take it away, tests work again except its impossible to import from Enzyme. Maybe a recent update to Codesandbox broke things?

[philiphendry]

This is such a nice feature and it seems to be working well but the editor is showing lint errors of 'describe is not defined' (among others). I'm afraid this is far from my expertise but with a little searching I thought adding "env": { jest: true } would do the trick and given I couldn't see a way to add .eslintrc I put it in package.json. However the errors are still there as shown in the screenshot below. Am I doing something wrong or is this currently unsupported?

[gautamarora]

@philiphendry - i can take a look and see if there is a way around, could you give some more details here: #937