Adblock Plus user survey results [Part 0] · 2011-10-10 10:43 by Wladimir Palant

Please note that this will be the only post to be syndicated to Planet Mozilla, the other parts of this series will appear in my blog only — I can imagine that most readers of Planet have very little interest in what I am writing here. If you are interested in the full analysis of the results, please visit my blog in a few days.

At the moment we have 1360 responses to the user survey which is more than enough to start analyzing the results. There will be some more responses before the survey ends but we don’t expect to get much more than 1500 responses and the overall results shouldn’t change significantly.

Why did we do it?

The normal process of getting user feedback in the Adblock Plus project involves the Future development forum. Many of the topics there I’ve created myself, others were created in other forum parts and moved to discuss the usefulness or details of the proposal. This approach works well and the feedback gathered by it is generally very useful.

Certain topics make it very obvious however that this feedback isn’t very representative for the Adblock Plus community as the whole. Particularly questions on the general project direction tend to yield very radicalized opinions. A user survey was brought up several times as a possibility to get a better idea of what our user base. In Adblock Plus 1.3.10 this survey was finally implemented.

What questions did we hope to answer?

Unfortunately, we couldn’t ask everything in this survey — the longer the survey, the fewer people will respond. So the mail topics were:

In the end the survey still became longer than I hoped: 10 questions on three pages. Originally only the questions on the first page (country, age, Firefox experience) were optional but after some consideration I made all of them optional — if somebody really doesn’t have an opinion on a topic then clicking “No opinion” for each answer option is just tiresome. Interestingly, almost all people answered all questions in full nevertheless.

Who was “invited” to the survey?

Obviously, it makes fairly little sense to ask every single Adblock Plus user to complete the survey — we would simply be unable to process the number of responses. Instead the code in Adblock Plus 1.3.10 decided on a number of conditions on whether the user should be invited:

AMO shows around 14 million active daily users for Adblock Plus which should translate into roughly 20 million actual users. This means that we asked something like 20 million * 0.90 * 0.80 * 0.65 * 0.95 * 0.005 = 44,460 users to participate in the survey. The invitation would show up 20 seconds after starting the browser:

User survey invitation

Given that this isn’t a modal window and that it closes immediately if the user changes focus we didn’t expect a very high participation rate. However, the current participation rate of 3-4% is still quite a bit lower than we expected. Luckily, without having any experience in this area we aimed for a very high number of responses.

Who did respond?

As mentioned already, we’ve got 1360 completed surveys so far. Most respondents used the survey in English, followed by German and then Russian:

Submissions by language

Interestingly, for English this matches the expected number exactly: if you only look at English, German and Russian locales among the Adblock Plus users then 52% of the users have the English browser locale (with German getting 30% and Russian 18%). It looks like the Russian users were somewhat more likely to complete the survey while the Germany users were slightly less likely to do that. Which is more likely a statement about the translation quality than the cultural differences.

This is only the start of it of course, I will take a closer look at the demographics of the survey respondents in the next part of the series.

The technical details

For the survey platform we looked at a few but all of them had very strict limits on the number of respondents and we weren’t sure that we would stay within those. In the end I decided to install LimeSurvey on this server instead of using an external provider. Configuring the survey was rather complicated but for that we got full control and didn’t need to send the data to some third-party server that might have a privacy policy that’s significantly different from ours. E.g. while the survey system does record IP addresses an automated script runs daily and replaces them with the corresponding country (GeoIP country database is being used).

Tags:

Comment [8]

  1. 13xforever · 2011-10-10 11:24 · #

    Well, I qualify for this survey (I think; I only have AdBlock visible in Addon Bar), but I never got the invitation.

    Reply from Wladimir Palant:

    “From the users that passed the other tests we selected 0.5% randomly”

  2. tomh · 2011-10-10 13:00 · #

    Looks interesting. I must say that “the current participation rate of 3-4%” sounds like a fairly decent response rate to me, given how many times I refuse to do surveys ;-)

  3. Stephan Sokolow · 2011-10-11 01:08 · #

    I worry about the kind of selection biases that might be introduced by requiring that the icon be visible.

    Compared to Chrome, it takes more effort to lay out a clean-feeling interface when keeping icons you rarely use… partly because Chrome lets you mimic the Windows system tray and hide the icons you rarely use without getting rid of them.

    As a result, you get examples like my system where the icon’s gone and I access ABP’s UI either through the Add-on Manager or the “Block Image…” context menu entry and nowhere else.

    Reply from Wladimir Palant:

    I am fairly confident that only a small minority bothers reordering icons. Even the people asking in the forum rarely know about this possibility – and they are the more experienced ones compared to the entire user base. But we have selection biases no matter what, there are several indications that our survey respondents are more experienced than our average user, I will mention that in the follow-up blog posts.

  4. Olivier · 2011-10-18 12:22 · #

    I’m sure I’m not the only enthusiastic and grateful user who just damn wishes s/he was included in the survey. =)
    I realize that the very fact that I wasn’t asked to participate directly follows from your desire to get a true sample of the user population. The way you built the poll is particularly interesting and the mark of a rigorous approach.

  5. rob · 2011-10-25 18:47 · #

    we have about the same response rate (much lower overall # user base) for the survey’s we’ve been conducting. will try to share our results and any similarities or differences we see in users reasons for using these categories of tools.

    great that you’re doing this!
    rob

  6. Arthur Carter · 2011-10-31 16:24 · #

    I too never received an invitation to participate in your survey. If I had participated, I would have tried to ask the following question: Do you have or know of a way to block ads in emails, particularly dynamic ads with irritating blinking messages?

    Reply from Wladimir Palant:

    Ads in emails? What ads? I use Thunderbird with Adblock Plus and don’t allow emails to load anything – this is good enough to see no ads. Many web interfaces for emails have this functionality as well (Gmail for sure).

  7. B.J. · 2011-11-07 02:36 · #

    When will the full analysis be posted? It’s been more than “a few days”.

    Reply from Wladimir Palant:

    Sorry, other things kept me occupied and analyzing these results isn’t quite easy. I’ve posted the statistics for the first page now.

  8. Link · 2011-11-17 20:45 · #

    Why only a survey for the firefox market ?
    What about chrome ?

    Reply from Wladimir Palant:

    Do you think that the answers would have been significantly different? Other than to the “improvements” question of course – but I don’t need to ask that question, I know everything that’s wrong with the Chrome version.

Commenting is closed for this article.