How this open source test framework evolves with .NET

Re-evaluating and overhauling a software project's design are crucial steps to keep up as circumstances change.

A software project's design is a consequence of the time it was written. As circumstances change, it's wise to take a step back and consider whether old ideas still make for a good design. If not, you risk missing out on enhancements, simplifications, new degrees of freedom, or even a project's very survival.

This is relevant advice for .NET developers whose dependencies are subject to constant updates or are preparing for .NET 5. The Fixie project confronted this reality as we flexed to outside circumstances during the early adoption phase of .NET Core. Fixie is an open source .NET test framework similar to NUnit and xUnit with an emphasis on developer ergonomics and customization. It was developed before .NET Core and has gone through a few major design overhauls in response to platform updates. The problem: Reliable assembly loading

A .NET test project tends to feel a lot like a library: a bunch of classes with no visible entry point. The assumption is that a test runner, like Fixie's Visual Studio Test Explorer plugin, will load your test assembly, use reflection to find all the tests within it, and invoke the tests to collect results. Unlike a regular library, test projects share some similarities with regular console applications:

  1. The test project's dependencies should be naturally loadable, as with any executable, from their own build output folder.

  2. When running multiple test projects, the loaded assemblies for test project A should be separate from the loaded assemblies for test project B.

  3. When the system under test relies on an App.config file, it should use the one local to the test project while the tests are running.

I'll call these behaviors the Big Three. The Big Three are so natural that you rarely find a need to even say them. A test project should resemble a console executable: It should be able to have dependencies, it should not conflict with the assemblies loaded for another project, and each project should respect its own dedicated config file. We take this all for granted. The sky is blue, water is wet, and the Big Three must be honored as tests run.

Fixie v1: Designing for the Big Three

The Big Three pose a huge problem for .NET test frameworks: the primary running process, such as Visual Studio Test Explorer, is nowhere near the test project's build output folder. The most natural attempt to load a test project and run it will fail all of the Big Three.

Early alpha builds of Fixie were naive about assembly loading: The test runner .exe would load a test project and run simple tests, but it would fail as soon as a test tried to run code in another assembly—like the application being tested. By default, it searched for assemblies near the test runner, nowhere near the test project's build output folder.

Once we resolved that, using that test runner to run more than one test project would result in conflicts at runtime, such as when each test project referenced different versions of the same library. And when we resolved that, the test runner would fail to look in the right config files, mistakenly thinking the test runner's config file was the one to use.

In the days of the regular old .NET Framework, the solution to the Big Three came in the form of AppDomains. AppDomains are a fairly old and now-deprecated technology. Fixie v1 was developed when this was the primary solution, with no deprecation in sight, to the Big Three. Under those circumstances, using AppDomains to solve the Big Three was the ideal design, though it was a bit frustrating to work with them. In short, they let a single test runner carve out little pockets of loaded assemblies with rigid communication boundaries between them.

fixie-design-diagram-v1-cropped.jpg The Test Explorer plugin and its own dependencies (like Mono.Cecil) live in one AppDomain. The test assembly and its dependencies live in a second AppDomain. A painful serialization boundary allows requests to cross the chasm with no risk of mixing the loaded assemblies.

AppDomains let you identify each test project's build output folder as the home of that test project's config file and dependencies. You could successfully load a test project's folder into the test runner process, call into it, and get test results while meeting the Big Three requirements.

And then .NET Core came along. Suddenly, AppDomains were an old and deprecated concept that simply would not continue to exist in the .NET Core world.

Circumstances had changed with a vengeance.