But there is a call for exact rendering comparisons.
For example, if a certain Unicode value in an html page is supposed to render an Urdu character, the only way to test for this is to have a human look at it and see whether it is right or not. Similarly, whether a given MathML leads to correctly rendered expressions is, at this point, only checkable by a human loading a page and looking at it. I call this the "browsers and eyeballs" test harness.
But the people blocking my changes for reftest are better at doing that than I am at figuring out what would be acceptable and would also do the job, so I have figured out a way to use the current, checked-in version of reftest to do exact visual comparisons. Rather than go into the code, I will demonstrate.
First, I created a file, u1.txt, that contained a list of URLs that I pulled from Ian Hickson's "HTML 4.0 Test Suite" at http://hixie.ch/tests/html40/test41-1a.html . Following the chain that starts at that page leads one to 16 URLs.
Then I run:
% perl runRef -generate -src u1.txt -app /Users/ray/mozilla/dist/MinefieldDebug.app/Contents/MacOS/firefox -profile /tmp/12qp5rt8.mozilla_20070630_153321_PDT > u2.txt
The profile is nothing special. It is just dynamically generated so that the browser can run in an automation-friendly manner. Then u2.txt will contain something like:
URI: data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAyAAAAPoCAYAAAAmy5qxAAAgAElEQVR4nOzdf1zV9 (and so on ...)
URI: data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAyAAAAPoCAYAAAAmy5qxAAAgAElEQVR4nOzdf1zV9d3/8QfXgA4 (and so on ...)
URI: data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAyAAAAPoCAYAAAAmy5qxAAAgAElEQVR4nOzdf1zV9d3/8QfXAA8aB (and so on ...)
URI: data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAyAAAAPoCAYAAAAmy5qxAAAgAElEQVR4nOzdf1zV9f (and so on ...)
Then when I can do:
% <re-fetch and build Firefox here>
% perl runRef -test -src u2.txt -app /Users/ray/mozilla/dist/MinefieldDebug.app/Contents/MacOS/firefox -profile /tmp/12qp5rt8.mozilla_20070630_153321_PDT
URL: http://hixie.ch/tests/html40/test41-1a.html -> Ok
URL: http://hixie.ch/tests/html40/test41-1b.html -> Ok
URL: http://hixie.ch/tests/html40/test41-2.shtml -> Ok
URL: http://hixie.ch/tests/html40/test41-3.html -> Ok
This will verify that the html being received from the URL is exactly as is expected (the DGT marker above is for an SHA1 digest of the source) and the new copy of Firefox displays exactly the same images as the old copy.
Are there obvious issues raised by doing this kind of testing? Yes. And there are ways around them. There are some tools needed to support a workflow that uses this test tool. And I could try to guess what the objections will be now but, then again, "vita brevis, programmi longa".
This should definitely not be seen as an attempt to replace testing tools like mochitest or reftest. On the other hand, I could spend a few days and find 50,000 static URLs and be able to say whether any given check-in causes any difference in the display of any of those pages. How long would it take to find the 5000 or so important differences in those pages and write reftests for them? I do not know, but I am not going to spend a few years trying to find out.