Zoips, thanks, that helped a good bit. Even so, 1400 rows takes about 8-12 seconds to sort. The Perl code that generates these reports takes a fraction of a second to sort and print out (including all formatting) that many rows.
There is a browser war going on, but it’s not the front end, it’s the rendering engines. The contenders are really Microsoft (Silverlight and .NET), Adobe (Flash), and WebKit (JavaScript/Ajax).
In a few years, most of the “webpages” will be displayed on handheld devices that don’t even seem to be running a browser (much like the iTunes store). The idea that the contest is between IE and FireFox is much like thinking that the battle for supremacy in the American luxury car market is between Lincoln and Cadillac.
Right now, Microsoft is having difficulties getting its act together. .NET was never really a rendering engine technology and Silverlight has the same problems as Flash: You need a proprietary client in order to see its content. That may be possible when you are only talking about x86 based chips and only two major operating systems with one having 90% of the market. When you have thousands of devices, each with its own chipset and maybe its own OS, you simply cannot depend upon proprietary clients.
Apple/Google are banking on WebKit and AJAX. WebKit is already in Mac OS X, the iPhone, and is on Android. Google is also banking on it for all of its stuff. KDE has now adopted WebKit over its own KHTML (which WebKit was based), and the GTK+ project is also adopting it too. That puts WebKit firmly into Linux which means it’ll be in all Linux based devices like setup boxes, phones, and PDAs.
I am not too sure what Microsoft is going to do. Next year will be very tumultuous for them. As the price of hardware drops, it gets harder and harder to convince PC makers to buy Windows licenses on all PCs they manufacture. Linux desktops are becoming more consumer oriented, and you can’t beat the cost of a Linux license. Combine that with OpenOffice, and you can offer a PC for about $100 less than a competitor. When you’re talking about $200 PCs, that’s quite a bit of money.
Microsoft also hasn’t done very well with Windows Mobile. Windows has been difficult to scale down to handheld devices, and they’ve pretty much lost the Setup box market. Scientific Atlanta, which has a near monopoly on the market, has gone solidly behind Linux. The iPhone, which hasn’t even been out for the whole year yet, has outsold all Windows Mobile smartphones combined.
Microsoft isn’t going to go away, and I am sure they have plenty of things they’re working on. They’re a smart bunch of people, and I’m sure they see the same things I mentioned. Bill Gates has always said that Microsoft is just an innovation away from irrelevancy which is why he always fought so hard for Windows’s monopoly. My guess is that the OM license price for Vista Home Basic will drop to less than a dollar by the end of next year. That will encourage PC manufactures to stick to Windows over Linux. I suspect that a Microsoft Home Basic Edition is also in the works too and will have a similar OM license price.
The big question is how Microsoft will handle the non-Desktop market. Right now, they’re way behind, but Microsoft has been in this position before with Netscape, and came out on top stronger than ever before. Can Microsoft do that again?
This is a fun excersize, but I dispute the usefulness of the results to the real world.
What do we use Javascript for in our web applications? Making 3D cubes? Doing crypto work? No. We use it for the following tasks that are not tested in the “benchmark”:
- Ansyncronous and syncronous HTML GET calls.
- DOM calls to the page structure for getElementByID() and setting style properties
- DOM updates like new Image().
- Responding to events like onClick() and onLoad()
The string benchmarks are the most disparate, and other than the examples I mentioned, they are where most of the processing time will be spent. Regular expressions (input validation) and validating input are the only really useful javascript metrics that I see here.
@David: I don’t think you really understand what rendering engines are. please read this article http://en.wikipedia.org/wiki/Layout_engine before commenting on something you’re not competent about.
@Jeff: thank you for the interesting analysis.
I think Tufte might like the first chart and the second… you are looking at two distinct angles on the same data set.
The first is the relative performance of each benchmark with the suite by each browser—you get to see where each team needs to spend more time to maximize performance… the Webkit team would do well to work on “access” and “bittops” while any effort “controlflow” would yield only a marginal impact on improving the overall score.
The second chart does a pretty darn good job at comparing which browser does the best/worst—the relative performance of a benchmark between browsers.
Nice work Jeff! I’d love to the first chart for my browser when I run SunSpider!
@Chris Chubb
Actually, people do crypto in JavaScript for real. Here’s an example: http://www.clipperz.com/. Many of the other benchmark test cases come from real code that people reported as slow, either in Safari or other browsers, or from real JS libraries. And yes, as JavaScript gets faster, it will become more commonplace to use it for games and 3D code.
You are right that on many web sites, DOM/layout performance matters more than core JS. Maybe we need better DOM benchmarks, too. But it is useful to be able to analyze performance of different engine components separately.
Please take into account how long it takes to load the browser…
A cheap browser like Konqueror can load and display a page while firefox is still opening and loading as an app…
Jeff,
Little off the topic question. Can you tell me how you generate those graphs? Any specific tool or macros etc. I am a big fan of how you present information in your blog and would love some tips.
The second chart is horrible. I don’t think Tufte would approve. It makes it kind of hard to compare the different browsers.
CanFDG’ssuperlonglinkspleasebefixed?Thanks.
prepare to get slashdotted 
@Aditya Banerjee: did you try Firefox with a clean profile? It’s entirely possible that errant extensions that you don’t have in your Flock install would be causing a 10% drop.
While I commend you for taking the time to do these tests, the graphs at the top are a poor way to present the data. I don’t think people really care to compare how long one benchmark compares against another, they care how the various browsers compare.
I suggest you swap the browser and test data sources in your bar graphs so that people can more easily compare how well browsers fared against each other.
Cheers
i’m looking forward to better js debugging tool. as of now, it’s seriously lacking.
Although Safari is based on KHTML could you run some benchmarks
with Konqueror to see how well it compares against its Apple raised cousin?
@Mat Hall: Opera 9.25 and 9.50 use different scripting engines, and the results are much better for 9.50:
Linear_b is used in Opera versions up to and including 9.25: http://en.wikipedia.org/wiki/Linear_b_(script_engine)
Futhark is used in Opera 9.50: http://en.wikipedia.org/wiki/Futhark_(script_engine)
Why hasn’t anyone complained about IE7 having one of it’s tests thrown out? Obviously it’s a test that the other browsers can handle and should not be thrown out for IE. Has anyone actually looked into why IE7 does so bad on the string test?
Scatter shot comments follow:
As someone who has done a lot of javascript, I agree that the string result for IE is likely very typical.
And with the bitops, I’m surprised that the other browsers were able to do so much better than Firefox since bitops in JavaScript are inherently inefficient.
And I agree with some other posters: the second graph is pretty but remarkably bad, making it very easy to compare the runtime of different benchmarks (apples and oranges–useless!) as totaled across the browsers (double useless!) and very difficult to compare the browsers to each other.
Leaving off the strings (unfairly, since one does an awful lot of string manipulation in many apps), Firefox is about 30% slower than IE. That really ain’t much, and as soon as you do some string work that takes almost 9 times as long in IE, that 30% vanishes fast.
Most important, Firefox has the most complete and correct implementation of JavaScript of the browsers, and although I’d like it to be faster, on balance I’ll take the power of the most complete JavaScript implementation over the 30% speed.
@jin, Firebug?
JavaScript is also good for artificial intelligence.
@Robin My tests were not under a controlled scenario as I have noted in my blog post. Also, Mike Shaver (http://abaditya.wordpress.com/2007/12/20/firefox-3-beta-2-javascript-benchmark-plus-why-flock-is-faster-than-firefox-2/#comments) has pointed out in my blog comments that the Flock JS engine is not any different from that of Firefox 2. In fact I tried out the tests on another PC (Athlon 64 3200+, 1 GB RAM) both in the safe mode and with extensions enabled for Firefox 2 and Flock, and found the results to be similar (FF2 was slightly faster this time). I have updated my post (http://abaditya.wordpress.com/2007/12/20/firefox-3-beta-2-javascript-benchmark-plus-why-flock-is-faster-than-firefox-2/) with this note.