[Done] Protecting subscription downloads

Various discussions related to Adblock Plus development
Wladimir Palant

[Done] Protecting subscription downloads

Post by Wladimir Palant »

Apparently, corrupted subscription downloads aren't too uncommon - be it because of bad proxies, firewalls or whatever. Adblock Plus should make sure the download doesn't get altered - invalid downloads have to be rejected.

What I want to implement: if Adblock Plus finds a line like "! Hash: 123456" in a subscription, the hash value should be an MD5 hash of the subscription without this line and with all line breaks converted to Unix format. If hash validation fails the download will be considered as failed.

I think this is the best we can do short of serving subscriptions from HTTPS - the user should not get corrupted filters. Of course the hash line should be placed at the beginning of the file, so that it gets through even if the file is cut off.
Last edited by Wladimir Palant on Thu Oct 30, 2008 11:03 am, edited 1 time in total.
User avatar
chewey
Posts: 501
Joined: Wed Jun 14, 2006 10:34 pm
Location: somewhere in Europe

Re: Protecting subscription downloads

Post by chewey »

Very good idea indeed. And easily integratable with my existing list generation automation :-)

Some sort of validator (via a web form?) might be useful though.
Dr. Evil
Posts: 194
Joined: Fri Sep 08, 2006 3:51 pm

Post by Dr. Evil »

hm... why not just check whether the Content-Length header and the length of the downloaded string* match?
* (converted back to a byte stream probably, but I think that's necessary for md5 as well)
chewey wrote:And easily integratable with my existing list generation automation :-)
What are you using for that? (I use my Adblock Plus Filter Uploader extension, another extension that adds rudimentary synchronization and a php script on the server)
Wladimir Palant

Post by Wladimir Palant »

Dr. Evil wrote:hm... why not just check whether the Content-Length header and the length of the downloaded string* match?
Because I don't trust proxy servers. I already found out that some of them messed up the Expires header which caused Adblock Plus to download subscriptions hourly. It would be easy for them to corrupt the download but "fix up" Content-Length header - or simply remove it.

PS: In some cases a firewall "censored" the contents of the download - I doubt that it changed the file length when it did that.
Last edited by Wladimir Palant on Wed Jul 02, 2008 3:22 pm, edited 1 time in total.
User avatar
chewey
Posts: 501
Joined: Wed Jun 14, 2006 10:34 pm
Location: somewhere in Europe

Post by chewey »

Dr. Evil wrote:
chewey wrote:And easily integratable with my existing list generation automation :-)
What are you using for that?
A homebrew shell script, nothing really woth sharing because it is very specific to my situation.

When I have made changes in my "list development" SeaMonkey profile, I just type adftp in a shell.

This automatically extracts a properly formatted file from the patterns.ini, adds a time stamp, makes
a copy to my "filter list history", gzips the final adblock.txt and pushes it to my web server.

I'm thinking about including some kind of comment management as well, so I can archive reasons
for the changes - I don't do that as of yet.
Wladimir Palant

Post by Wladimir Palant »

chewey, will a Perl script to insert the MD5 hash do it for you? I'll have to write it anyway, for EasyList.
User avatar
chewey
Posts: 501
Joined: Wed Jun 14, 2006 10:34 pm
Location: somewhere in Europe

Post by chewey »

Wladimir Palant wrote:chewey, will a Perl script to insert the MD5 hash do it for you?
I would've gone for an extension of my multiple hacky shell pipes. ;-)

But it might be a good idea to rewrite my ugly shell stuff in Perl anyway, so yeah, sounds useful.
Wladimir Palant

Post by Wladimir Palant »

Well, you can run Perl from a shell script if you don't want to rewrite everything ;)
Dr. Evil
Posts: 194
Joined: Fri Sep 08, 2006 3:51 pm

Post by Dr. Evil »

I don't know if this is a good idea or not, but I thought I'd throw it in here...
The last bytes of a gzip file contain a crc32 hash (of the uncompressed data) and the file size. Firefox doesn't care about this when decoding, but if you stripped the "Content-Encoding" header from the channel and did the decoding yourself (or rather pass it to nsIStreamConverterService yourself), you could enforce these fields to match.
Wladimir Palant

Post by Wladimir Palant »

I don't think decoding gzip data myself is a good idea. And I doubt it is possible to verify the checksum after the data was already uncompressed. But maybe Firefox can be changed to make the checksum available even if it doesn't consider it...
Dr. Evil
Posts: 194
Joined: Fri Sep 08, 2006 3:51 pm

Post by Dr. Evil »

Wladimir Palant wrote:I don't think decoding gzip data myself is a good idea.
It's not that much work. I'm doing the opposite (encoding) in the Filter Uploader.
But maybe Firefox can be changed to make the checksum available even if it doesn't consider it...
I wouldn't know any other way besides a faked http header. And that doesn't sound very clean.
Wladimir Palant

Post by Wladimir Palant »

It is really not about the amount of work - with so many quirks around HTTP and broken server implementations I trust the browser (which had decades of development put into it) with things like this much more than I trust myself.
Wladimir Palant

Post by Wladimir Palant »

Done: http://hg.mozdev.org/adblockplus/rev/5fbd5e590515

There is also a reference script to add a checksum to a filter subscription: http://hg.mozdev.org/adblockplus/file/t ... hecksum.pl

I also want to make "Export filters" insert a checksum automatically. This checksum shouldn't be considered for "Import filters" (people who simply back up their filters might also change the file manually) but will be considered if the file is uploaded as a filter subscription.

Edit: All downloads from easylist.adblockplus.org now get the checksum added automatically.

Edit2: "Export filters" now inserts a checksum - http://hg.mozdev.org/adblockplus/rev/0ca1488c074a
User avatar
rick752
Posts: 2709
Joined: Fri Jun 09, 2006 7:59 pm
Location: New York USA
Contact:

Post by rick752 »

Cool!

Hopes this resolves those corrupt subscription downloads now :D
Ares2
Posts: 1275
Joined: Fri Feb 15, 2008 12:47 pm

Post by Ares2 »

Just wanted to say addChecksum.pl is working fine here. :D

It doesn't matter if I remove the space before 'Checksum', does it?
Locked