http://blogs.pcmag.com/securitywatch/2010/11/dirty_apps_and_dirty_marketing.php
2010-11-8
Is Google Chrome the most vulnerable application of the year? That's the implication of a new report from Bit9, a company that specializes in application whitelisting solutions. Too bad the report doesn't actually show what it implies.
Bit9 looked at legitimate, non-malicious applications which had at least one vulnerability reported between January 1, 2010 through October 21, 2010 which had a severity rating of high (between 7.0-10.0) on the Common Vulnerability Scoring System (CVSS) registered in the NIST (National Institute of Standards and Technology, an agency of the U.S. Dept. of Commerce) database. Here is the list of the most vulnerable, as tallied by Bit9:
- Google Chrome (76 reported vulnerabilities)
- Apple Safari (60)
- Microsoft Office (57)
- Adobe Reader and Acrobat (54)
- Mozilla Firefox (51)
- Sun Java Development Kit (36)
- Adobe Shockwave Player (35)
- Microsoft Internet Explorer (32)
- RealNetworks RealPlayer (14)
- Apple WebKit (9)
- Adobe Flash Player (8)
- Apple QuickTime (6) and Opera (6) - TIE
What does this actually demonstrate? Not a whole lot. In fact, if anything it shines a light on weaknesses in the CVSS scoring system. CVSS is a standardized system for scoring vulnerabilities so that they can be compared for severity. But the score doesn't reflect what the system doesn't take into account. I don't want to be too disparaging of CVSS, because I like the basic idea, but the Bit9 report is a perfect example of the limits of the CVSS system.
The very first Chrome vulnerability listed in the Bit9 report was CVE-2010-4042. It's described by Google as "Stale elements in an element map," rated "high" (below the top level of "critical" and was discovered by the well-known Michal Zalewski of Google. Click here for the CVSS scoring on it.
Coincidentally, Zalewski blogged on the subject of the limits of scoring based on numbers of vulnerabilities earlier this year. He pointed out that some vendors (like Google) openly report vulnerabilities discovered in-house as they fix them, and CVE-2010-4042 is a perfect example. Others (like Microsoft) choose to bury vulnerabilities discovered in-house in service packs and cumulative updates and who knows where else. That's a criticism of Microsoft's reporting techniques, not their commitment to security which has been top-notch of late. But it just goes to show that the number of reported vulnerabilities is not a good proxy for the security of a program.
Lots of security companies engage in cheap-shot (惡意中傷、攻訐)marketing like this and I've learned to overlook most of it. But in this case the result is so far from the truth that it cries out for clarification（澄清）. In fact, the outrage came so loud and fast that the next day Bit9's CTO Harry Sverdlove blogged to discuss some of the severe limitations in their own report methodology. Sverdlove repeats the points above about the diligence with which different companies approach vulnerabilities.
Many have made the same points while criticizing the Bit9 report and there are more:Marc Maiffret of eEye Digital Security points out that the number of vulnerabilities in an app is not the same as the number of exploits or the prevalence of any exploits in the wild. This may be due to the "flying under the radar"（避人耳目） effect, but the fact remains that users of many "dirty" apps are not going to get attacked because nobody is attacking them.
I'll add one more reason: The CVSS severity scores for almost all the Chrome vulnerabilities are overstated（言過其實） and Bit9 was exaggerating（誇大） by relying on them. Chrome, as I have written numerous times in this blog, uses a sandbox architecture which limits the ability of exploit code to accomplish any privileged operations. This is why Google rated CVE-2010-4042 as "high" and not "critical": any exploits of it would run in the sandbox. The CVSS scoring system makes no allowance（未作修正，未留餘地） for sandboxes as mitigations to lower the score. It's probably just a case of CVSS being behind the times（過時）, as standards often are.
It's a real shame, as I said, because I like CVSS and it bothers me to see it abused. In fact, I also like Bit9 and their products and whitelisting in general (and, I must disclose, I have done work for Lumension, a competitor to Bit9). I hope we all learn some lessons from this.