Hi Ilya,
Great question! First the guiding principle is that shared test fixtures first value robustness/correctness, and second they value sharing as much as possible. Another principle is that tests explicitly declare the shared test fixtures that they need, and the test framework ensures that they get exactly that list of test fixtures set up for the context. This means, for example, that if a test does not have a shared test fixture "declared" but that fixture is setup and running because the previous test required it, then it will be torn down.
Once a fixture needs to be torn down, it needs to be torn down with all the other active fixture in LIFO order in order to ensure that the fixture restoration happens robustly. Otherwise, fixture state can be incorrectly restored. An example of this is shown in the "A House of Order" section of this blog post.
So thus we want to provide the ability to shared fixtures across test files, folders, packages, and indeed entire suites to allow setup that is very expensive to be optimized. However, incorrect software is worse than inefficient software, so we must respect precisely the fixtures that each test states they require, and we must respect that the teardown execute in the reverse order of all shared fixture setup. This means that there are times when fixtures are setup and torn down only to be setup once again. For your example, assuming these three test classes have just one test method each, then you have
- test1 {sf1, sf2}
- test2 {sf2}
- test3 {sf3}
and if you run them in that order then you will
- setup sf1 & sf2 for test1
- You'll need to tear down sf1 for test 2, but since sf2 was setup last and we need to teardown in the reverse order of setup, we will need to teardown sf2 in order to be able to teardown sf1. Then we need to setup sf2 again
- For test3 we will teardown sf2 and setup sf3.
If the test were run in this order:
- test2
- test1
- test3
then the result would be:
- setup sf2 for test2
- setup sf1 for test1 (which now has both sf1 and two running as it requires)
- teardown sf1, then sf2 for test3 which specifies neither, and setup sf3 (and subsequently tear it down when done.
You can see with this approach that the order of the elements in the Test array matters a whole lot, and you can imagine someone doing the following if they want to be really inefficient:
suite = suite(randperm(length(suite)));
This is inefficient because this splits all the test elements, including the various methods of each test class and distributes them randomly around. The result will be lots of expensive setup and teardown code executing. While expensive, however, this approach even in the randperm case is still robust and each test element is provide with precisely the correct environment it requires. We have considered providing a feature to "reorder" the suite to be as efficient as we can make it. Does that sound interesting to you? In your example such a suite reordering would have preferred choosing test2->test1->test3 over test1-test2-test3.
