A Standalone Test Product

What does one need to do to help test Firefox. It turns out that to help very much, it takes a fair amount of effort. Just downloading the latest browser and "doing stuff" with it does provide testing, but it is not very complete. That kind of testing tends to be shallow at best.

Firefox has great automated testing tools built into it, and so one can build a copy of Firefox using '--enable-tests' as a configuration flag. One gets a bunch of standalone executable tests, some xpcshell tests, the mochitest functionality and reftest. There is a lot there to work with.

However, building Firefox can be a pretty hefty requirement. Is it really necessary? One benefit of automated testing is that the more of it that gets done, the more valuable it can be. There are lots of OS versions and machine configurations to worry about. MoCo can either try to have a copy of all those OSes and all those machines, or it can allow the community to meaningfully assist with testing without jumping through all the extra hoops needed to be able to build the browser. After all, the software required to just run tests is much simpler.

In pursuit of this, I am creating a standalone product which can be a test vehicle for testing Firefox. After all, if Firefox is changing every day, should one have to refetch all those tests? One would hope that when Firefox is changing a lot, the tests are not changing, so that the tests can provide confidence for the changes being made.

So far, I have a standalone product that I can use, on my Mac OS X system, to run reftests. There are issues with some of the other technologies that I will try to address. For now, I have taken the last 4 nightly builds and used the same test product on them and got the same result.

- REFTEST UNEXPECTED FAIL: file:///.../testing/reftest/layout/bidi/bidi-000.html
- REFTEST UNEXPECTED FAIL: file:///.../testing/reftest/layout/bidi/bidi-001.html
- REFTEST UNEXPECTED FAIL: file:///.../testing/reftest/layout/bidi/bidi-001-j.html
- REFTEST UNEXPECTED FAIL: file:///.../testing/reftest/layout/bidi/mixedChartype-02.html
- REFTEST UNEXPECTED FAIL: file:///.../testing/reftest/layout/bidi/mixedChartype-02-j.html
- REFTEST UNEXPECTED FAIL: file:///.../testing/reftest/layout/line-breaking/currency-2.html

There may already be bugs on these test failures. It is hard to tell. I cannot seem to find the obvious way to locate bugs filed against automated tests.

The test product is based on the contents of the "_tests" directory as built with the '--enable-tests' flag, with some minor changes. Also, the contents of the directory are actually files, not links back into a source tree. The basic directory structure right now looks as so:

tests\
    Minefield.app/Contents/MacOS/chrome/reftest.jar
    Minefield.app/Contents/MacOS/chrome/reftest.manifest
    Minefield.app/Contents/MacOS/components/reftest-cmdline.js
    testing\
        mochitest\
        reftest\
    xpcshell-simple\

There are issues, already, that make it a little difficult to just grab this test product and put it up in a shareable location for other Mac OS X users. For one thing, script bits need to be put inside the application bundle of the app being tested. For reftest, it is only three files that need to be put inside the ".app/Contents" directory, but it is awkward to do manually.

Looking at mochitests, I can already see problems. The tests all assume that they can do something like "../../../dist/bin/testExecutable". There is no reason they need to assume that, but they all do. Does the xpcshell executable that runs the tests need to be the same build as the Firefox browser itself? I would say no, but that is the way it has always been done. Until issues like these are teased apart, there will have to be lots of workarounds.

If anyone has any suggestions, I am open to them. If MoCo wants to make automated testing available to a much larger group of people, without massively increasing their app download times, it seems that a standalone test product would be one way to do it.