automating leaks testing

I have managed to get some ideas I had to work in code. It required a couple of things.

First, why. The leak-gauge.pl script by dbaron is obviously very useful for tracking leaks in Firefox. I was just reminded of this reading this blog post by Steve England.

But reading Steve's post, it looks as though he did the following:

1) invoke leaks-gauge.pl on some URL
2) do a bunch of stuff on the resulting page
3) quit the browser
4) look at the results
5) perhaps file a bug
6) lather, rinse and report

If one wants to automate this testing, there are issues from some of this.

"invoke leaks-gauge.pl on some URL"

Right now, one uses a web page to tell leak-gauge.pl what to do. It is a manual process, hence this is not automatable.. I modified the leaks-gauge.pl script so that you give it an application executable, a profile directory, and a URL and it launches the browser for you. Another script goes through a list of apps and a list of URLs and calls the modified leaks-gauge.pl with the Cartesian product of these. This could all be done smarter than I am doing it.

What URLs can I run it on? Not wanting to answer that right now, I just looked at my copy of the trunk and used all the html files inside the layout/reftests directory. Right now this is 1081 URLs. I used 5 different applications, those being nightlies from 2007/01/30, 2007/03/24, 2007/05/23, 2007/06/15, and 2007/07/13.

"do a bunch of stuff"

Steve was testing extensions. Indeed, if one wants to test the memory footprint of extensions, one probably has to interact with the UI to cause the extension to do its stuff. I have no solutions to this quandary. I can say that it would be useful to use leak-gauge.pl just for single-page loads and this is automatable now.

"quit the browser"

This is not hard, but it is a bother. There are lots of ways to do it, but many of them make testing hard for various reasons. My solution to this was to create a very small extension, which will be up on AMO very soon. All it does is wait for a page to load, any page, and quit the browser. So, if one invokes firefox from the command-line and passes in a URL, then Firefox gets launched, it loads the page and then it automatically quits.

"look at the results"

So, I have a script that takes the output from the test run and outputs a couple of pages. Of course, anybody would know I will say this should be in a database, since I think everything should be in a database, but that is for later.

For full results, see http://www.wykiwyk.com/mozilla/leakTesting/testlog_20070718_byApp.html and http://www.wykiwyk.com/mozilla/leakTesting/testlog_20070718_byURL.html.

The summary results are that 2007/01/30 leaked some on a lot of pages, 2007/03/24 was slightly worse, and 2007/05/23, 2007/06/15, and 2007/07/13 did not leak at all on these files. Yeah for Cycle Collection! :-)

And now, there is a Firefox extension for which someone is actually paying me, so I should get back to it.