The wrong way of allowing users to make informed decisions · 2011-04-13 15:15 by Wladimir Palant
One point I probably didn’t stress often enough in my previous blog posts on AMO‘s add-on performance measurements is how poorly percentages reflect the real effect of add-ons. At least I see no indication that the people responsible for this campaign intend to change their mind and provide users with more accurate information. Please allow me to explain once again why these numbers are misleading (bug 648742 reworded).
At first it looks like a good idea — the actual slowdown numbers may vary depending on user’s hardware so why not put them in proportion to Firefox startup times? However, the assumption here is that Firefox startup times and slowdowns used by extensions are roughly proportional. So far it doesn’t look like it, even AMO’s measurement data shows Firefox startup times varying wildly depending on operating system (same hardware) while slowdown caused by add-ons stays roughly the same (right now it is hard to tell however, it is difficult to find add-ons where the results aren’t affected by any measurement bugs).
What’s more important, Firefox startup times in these tests are extremely good, on average 0.5 seconds. According to the graphic published by AMO’s Justin Scott the real-world Firefox start-up times are on average beyond 2.5 seconds however — even without add-ons. I am pretty certain that this is not because consumer hardware is typically 5 times slower than the Mac Minis that Mozilla is testing on. The main contributing factors for the difference are IMHO cold startup vs. warm startup (in AMO’s measurements most Firefox components are already in the operating system’s cache which allows for faster startup — this usually won’t be the case when end-users start Firefox) and profile state (the measurements are performed with a clean profile that has no history, no cookies, no bookmarks — end-users’ profiles will have a significant amount of data in them which will slow down Firefox startup). While both have significant impact on Firefox startup times, they usually have fairly little effect on delays caused by add-ons.
So, if your Firefox startup time is 2.5 seconds and you install Firebug (currently listed at the top of the “slow add-ons” list with 74% slowdown) — how much slower should you expect your Firefox startup to get? If your answer is “somewhat between 1.5 and 2 seconds” then you most likely guessed wrong (like most end-users who were helpfully “informed” by this recent AMO campaign). What you should really expect is more between 0.3 and 0.5 seconds, pretty close to the value that AMO actually measured. Which still isn’t great but far less scary than what is implied by AMO’s numbers — if you are a web developer and you only have a few add-ons installed then you probably shouldn’t think twice before installing Firebug.
Commenting is closed for this article.