Firefox 2 Phishing Protection Effectiveness Testing
Note: this blog entry is being updated regularly as more information and analysis comes to hand, so don't forget to check back.
Hot off the presses, and very limited in information at the moment, is a report claiming that Firefox beats IE7 in the phishing protection stakes.
PDF report here (note to blog.washingtonpost.com - please don't use spaces in your blog URLs and learn how to make your links shorter):
http://blog.washingtonpost.com/securityfix/Firefox%202%20Phishing%20Protection%20Effectiveness%20Testing.pdf
We have already been down this path with IE7 as reported here:
http://msmvps.com/blogs/spywaresucks/archive/2006/09/28/144948.aspx
I sure hope that the PDF hosted by blog.washingtonpost.com is not the sum total of information that is going to be provided about the tests completed for Firefox - the 3Sharp report contained 37 pages of what they did, how they did it, why they did it, and a comprehensive breakdown of the results. How can we compare such detailed information to the 3 pages of summary provided so far about the Firefox tests?
There simply isn't enough information to be able to make a judgment call on the Firefox report, but on reading what little information has been released the following jumps out at me. More info is coming out, and this blog is being updated as the results are analysed. My thoughts so far...
Firefox tests: Apparently only went head to head with IE7
3Sharp tests: IE7 went up against Firefox, EarthLink, eBay, GeoTrust, Google using Firefox, McAfee, Netcraft, and Netscape.
Firefox tests: Only one source of phishing URLs was tested. Test phishing URLs were received from PhishTank via their public XML feed of valid phishing URLs. Is this the same PhishTank that already provides a plug-in service for Firefox users? Were the URLS provided by the service already provided to Firefox via the phishing protection service for Firefox users? Did the testers ensure that only URLs that had not already been provided to Firefox users via PhishTank were used? Can they guarantee that there was no way the Firefox filter could have had an unfair advantage over IE7 considering the pre-existing service being provided to Firefox users by PhishTank?
PhishTank's service has only been around for a couple of months, and the Phishtank plug-in for Firefox was released to the public on 3 November. Testing for the Firefox report continued until 6 November so there was only a few days of "public" overlap. Data for PhishTank is gathered from subscribers and "external feeds" - I have not been able to find out what those external feeds are.
Ok, so PhishTank provides a plugin for Firefox (albeit for only 3 days).
PhishTank users are highly likely to be Firefox users.
Firefox includes the Google Phishing Service.
What is to stop those who are submitting to PhishTank also submitting to Google at the same time considering they are likely to be using Firefox? After all, it is a Firefox plugin? The ties are too close. PhishTank needs to prove that the URLs that were used for the tests had not also been submitted to Google's service by PhishTank users.
3Sharp tests: The known phishing URLs were not taken from any feeds from known third-party data providers or end users to the Microsoft Phishing Filter Service in IE7 - in short, if a service provided phish URL data to IE7 they were excluded from the test to remove any chance of advantage. Known good URLS were pulled from a feed of randomly selected traffic-weighted URLs provided by Microsoft and were independent of, and confirmed not to be included in, the Microsoft Phishing Filter system (they are not in the Phishing Filter white list).
Firefox tests: A list of URLs that were used "will be posted soon". (Update: now posted)
http://www.mozilla.org/security/phishing-test-results.html
3Sharp tests: The URLs were included in the report.
I am going to have a closer look at how the results differed between IE7 and FF when Firefox used Google, and IE7 Autocheck was on. My goal will be to discover how many times either browser failed when the other was successful. If one browser fails when another succeeds, we must ask ourselves why? The only problem is, how do we take into consideration the fact that the provider of phishing information has a pre-existing relationship with Firefox? Do Google (who provide FF's online phishing service) and Phishtank who provided the URLs for the test, share data?
Results to come later - this will take a little time.
Firefox: The feed was downloaded once per hour, and any new phishing URLs found were added to a testing database, and tested within 15 minutes of download.
3Sharp: The products were tested using 100 known phishing URLs (which had to be tested within 48 hours of collection) and 500 known good URLs. A failure by a Web browser to detect a phishing URL that was reported up to 48 hours earlier can be considered to be a more dangerous failure than failing to detect a URL that was only reported within the previous hour and 15 minutes.
Firefox: The available reporting fields were • Not Blocked - the page loaded normally without notification to the user. • Blocked - the page was blocked by a warning indicating that the current page was a suspected web forgery. • Warned (IE 7 only) - This would warn the users of a suspicious page, but would not block or prevent the page from loading.
When scoring there was no differentiation between blocking and warning, despite the URLs being tested being known phishing sites.
3Sharp: Tested catch rate (block, warn and allow) *and* false positive rate. When scoring results, a false block on a good site was scored as twice as bad as a false warning. Allowing a good site had zero value.
Firefox specifically excluded false positives from their testing regime because they "test for false positives through other mechanisms". Sorry, but this is not good enough. If a phishing filter triggers false positives too often, then the users are going to eventually ignore the warnings:
http://www.mozilla.org/security/phishing-test.html
Trackbacks to this entry led me to this report, which also stresses the importance of capturing and considering false positives in anti-phishing filters - I hope that when the Firefox results are finally released in full that such information is made available:
http://lorrie.cranor.org/pubs/toolbars.pdf
Firefox: So far only information about percentage blocked is available.
3Sharp: Report contains blocked, not blocked, warn and false positive percentages.
In short, Mozilla are going to have to provide a LOT more information for us to be able to make a fair comparison between the Firefox tests and the 3Sharp tests carried out a while back.
iSECPartners audit report:
http://www.mozilla.org/security/iSECPartners_Phishing.pdf