Support mstest and jUnit in the JobSender

[markwilkie]

Edit 06 Feb 2019:
Work is to support more test platforms in the JobSender.

---

There has been quite a bit of discussion around the fact that xunit is no longer being actively supported. Further, mstest adoption is growing.

What's the right guidance for moving forward? This guidance likely will have implications regarding new feature support for mstest in additional to xunit.

[tmat]

Can you point to the discussion?

[markwilkie]

To clarify - xunit active support is in regards to the xunit runner specifically. (at least in this context)

[Chrisboh]

@markwilkie do you know where this landed?

[markwilkie]

nowhere yet - that's part of the work.

[jonfortescue]

I believe the new work here is going to be to adjust the job sender to support a broader set of test platforms (possibly be test platform agnostic?) and allow jUnit and mstest as well. @alexperovich can provide more context, but for now I'm renaming this.

[alexperovich]

Yea, the work here is to just change the Helix SDK to use dotnet vstest to run the tests, rather than the xunit console runner. That is the supported way to run xunit going forward and it will work for any test framework that can be used in the .NET SDK.

[tmat]

@alexperovich I don't think we should use dotnet vstest until we can use it locally as well. Unfortunately, vstest uses different settings when running xunit tests than xunit.console.runner, so we could get into situation when test pass locally but fail on Helix. That's not good.

We can't use dotnet vstest right now since it has bugs like microsoft/vstest#1764 and microsoft/vstest#1907.

[alexperovich]

That makes sense. In that case I'd rather not implement a bunch of throwaway work for junit/mstest and just wait for vstest to get fixed. I had to write annoying hacks with dotnet exec to get the xunit console runner to load the proper assemblies, and if something changes in the way tests are supposed to run I would have to fix it manually.

[jonfortescue]

I agree 1000% with @alexperovich.

[tmat]

@alexperovich Curious about the hacks.... Why not use the same target as Arcade SDK uses?

[alexperovich]

The helix execution doesn't have any msbuild context at all. It knows what test dll it should be testing, so it can find the deps and runtimeconfig files, but dotnet exec is an undocumented implementation detail of the cli. This logic can break at any time if the cli changes things. dotnet vstest is a documented command that will change less and hopefully in non-breaking ways.

[tmat]

It knows what test dll it should be testing

That should be entirely sufficient.

The target inputs are following:
/~https://github.com/dotnet/arcade/blob/master/src/Microsoft.DotNet.Arcade.Sdk/tools/Tests.targets#L47-L58

--depsfile and --runtimeconfig look very much documented to me:

>dotnet exec

Usage: dotnet [host-options] [path-to-application]

path-to-application:
  The path to an application .dll file to execute.

host-options:
  --depsfile <path>                   Path to <application>.deps.json file
  --runtimeconfig <path>              Path to <application>.runtimeconfig.json file

[tmat]

Never mind, you mean the entire exec command. Interesting. Anyways, doesn't matter much. If we dropped it we would break our own tests, so we won't drop it before we have a replacement. In any case, I don't see why Helix should be doing something else.

[alexperovich]

This work is finished.