Cranking on the Test Engine

I recently "finished" an inventory of the testing tools and test suites available here at Mozilla. I put quotes around the word "finished" because, of course, it is not done. In a sense, it never will be. What I have done is that really turned one crank on an engine. Hopefully, the engine will be started.

I view this as an engine because there is a cycle that needs to occur. (I may actually have my engine terminolgy wrong here. I do not actually know engines as anything other than an abstraction. Apologies in advance.) Also, I am not an artist. I do words, not pictures.


Spark - identify and ignite existing tests. This will lead to...

Combustion - tests are executed and results recorded.

There has been a lot of random firing in the past. There are a bunch of tests. But when people look to run tests, their choice of tests has been close to random. I believe this to be so because if one wanted to pick tests by another method, there is no way to do so. There is no information available to provide any other criteria. There is a very rough correspondence between modules and tests, but modules are amorphously defined and the results are usually passed on in the form of a message that "I ran tests in X and they worked." Not very specific.

Right now, combustion is starting to happen. Litmus tests get executed and there is information at the end that can be pointed to. There are suites of reftests and mochitests scripts that are being run after a build and reported in tinderbox. But there are a lot of tests that are not in these harnesses yet. There are a lot of tests that are hard to analyze and hard to understand. It is hard to tell how useful they are.

Expansion - results of testing are analyzed.

It is usually in this stage that a determination is made about whether something can be shipped, or if the quality of the product is deemed good enough for some purpose. This stage involves an evaluation about what the testing meant. This leads to the next stage.

Compression - functionality is identified for which new tests are needed.

Did the testing that was done test the features that we care about? Usually, enough testing is done so that we are confident of the new features and of drastically changed functionality. But we all know there are side-effects and side-effects to side-effects and I do not think anybody feels we are testing as much of the functionality as we should. Thinking about what new tests are needed pushes the piston back down to a firing position.

Injection - define more tests of the functionality.

And then this brings us back to Spark.

I suggest this could be thought of as a cycle, and a cycle that needs to continue to go around. It is not something to do until we are "done".