A Way Around my Gordian Knot with Reftests

I have been trying to figure out a way for reftests to be able to do image comparisons. I have been hampered by both my lack of experience with writing "Mozilla-ish" JavaScript and by the fact that reftest was not designed to do this. Or perhaps, reftest was designed not to do this. Indeed, the thing that makes reftest different is that it does not do exact comparisons with a hard-coded "golden" page and so it is more flexible and does not cause unnecessary failures when unimportant changes are made to, for example, layout.

But there is a call for exact rendering comparisons.

For example, if a certain Unicode value in an html page is supposed to render an Urdu character, the only way to test for this is to have a human look at it and see whether it is right or not. Similarly, whether a given MathML leads to correctly rendered expressions is, at this point, only checkable by a human loading a page and looking at it. I call this the "browsers and eyeballs" test harness.

But the people blocking my changes for reftest are better at doing that than I am at figuring out what would be acceptable and would also do the job, so I have figured out a way to use the current, checked-in version of reftest to do exact visual comparisons. Rather than go into the code, I will demonstrate.

First, I created a file, u1.txt, that contained a list of URLs that I pulled from Ian Hickson's "HTML 4.0 Test Suite" at http://hixie.ch/tests/html40/test41-1a.html . Following the chain that starts at that page leads one to 16 URLs.

Then I run:

% perl runRef -generate -src u1.txt -app /Users/ray/mozilla/dist/MinefieldDebug.app/Contents/MacOS/firefox -profile /tmp/12qp5rt8.mozilla_20070630_153321_PDT > u2.txt

The profile is nothing special. It is just dynamically generated so that the browser can run in an automation-friendly manner. Then u2.txt will contain something like:

URL: http://hixie.ch/tests/html40/test41-1a.html
DGT: 9e747d8a7b957f5126f07135b75f9564405da513
URI: data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAyAAAAPoCAYAAAAmy5qxAAAgAElEQVR4nOzdf1zV9 (and so on ...)

URL: http://hixie.ch/tests/html40/test41-1b.html
DGT: 1bb9c5fba9fd37d6afdced89817f0a61104b3ced
URI: data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAyAAAAPoCAYAAAAmy5qxAAAgAElEQVR4nOzdf1zV9d3/8QfXgA4 (and so on ...)

URL: http://hixie.ch/tests/html40/test41-2.shtml
DGT: 0f3e0c63e174afff4c61924384cbe2dc78722447
URI: data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAyAAAAPoCAYAAAAmy5qxAAAgAElEQVR4nOzdf1zV9d3/8QfXAA8aB (and so on ...)

URL: http://hixie.ch/tests/html40/test41-3.html
DGT: ad782efed8ab827701fb6740f2089d897f8499bd
URI: data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAyAAAAPoCAYAAAAmy5qxAAAgAElEQVR4nOzdf1zV9f (and so on ...)

Then when I can do:

% <re-fetch and build Firefox here>
% perl runRef -test -src u2.txt -app /Users/ray/mozilla/dist/MinefieldDebug.app/Contents/MacOS/firefox -profile /tmp/12qp5rt8.mozilla_20070630_153321_PDT
URL: http://hixie.ch/tests/html40/test41-1a.html -> Ok
URL: http://hixie.ch/tests/html40/test41-1b.html -> Ok
URL: http://hixie.ch/tests/html40/test41-2.shtml -> Ok
URL: http://hixie.ch/tests/html40/test41-3.html -> Ok
...

This will verify that the html being received from the URL is exactly as is expected (the DGT marker above is for an SHA1 digest of the source) and the new copy of Firefox displays exactly the same images as the old copy.

Are there obvious issues raised by doing this kind of testing? Yes. And there are ways around them. There are some tools needed to support a workflow that uses this test tool. And I could try to guess what the objections will be now but, then again, "vita brevis, programmi longa".

This should definitely not be seen as an attempt to replace testing tools like mochitest or reftest. On the other hand, I could spend a few days and find 50,000 static URLs and be able to say whether any given check-in causes any difference in the display of any of those pages. How long would it take to find the 5000 or so important differences in those pages and write reftests for them? I do not know, but I am not going to spend a few years trying to find out.