32 sec and 40 sec in Firefox 3 beta 2 on an old Celeron 2.2 GHz with 1.24gb RAM, vs 60-65 sec in Firefox 2.0.0.11 on the same box.
@Maciej, Robert:
“charCodeAt(i)” != “[i]”
“charAt(i)” == “[i]”
The second implies string alloc where the first returns an integer. I expect it’ll be a wash.
Regards
Thanks for pointing that out. However, I do think it will make some difference in IE, because string[i] returns undefined, so they’re not actually executing the intended benchmark and probably something weird and bogus is happening. I will publish 0.9.1 soon with some fixes to the tests.
Since people have asked, here’s what the test categories are about:
3d: Pure JavaScript computations of the kind you might use to do 3d rendering, but without the rendering. This ends up mostly hitting floating point math and array access.
access: Array, object property and variable access.
bitops: Bitwise operations, these can be useful for various things including games, mathematical computations, and various kinds of encoding/decoding. It’s also the only kind of math in JavaScript that is done as integer, not floating point.
controlflow: Control flow constructs (looping, recursion, conditionals). Right now it mostly covers recursion, as the others are pretty well covered by other tests.
crypto: Real cryptography code, mostly covers bitwise operations and string operations.
date: Performance of JavaScript’s “date” objects.
math: Various mathematical type computations.
regexp: Regular expressions. Pretty self-explanatory.
string: String processing, including code to generate a giant “tagcloud”, extracting compressed JS code, etc.
What’s not in the test: It’s true that for a lot of JavaScript code on web sites, the bottleneck is more in DOM operations and layout than JS per se. There are interesting benchmarks available for that sort of thing, but we wanted to keep this one focused on core JS processing, which does matter for many operations on the web. Bottom line page load time benchmarks are also interesting, but do not substitute for this kind of testing. In many cases, the performance of JS hits you when doing various operations on a Web 2.0/AJAX type site after the initial page load. In any case, we may come out with other benchmarks in the future.
The second chart is horrible. I don’t think Tufte would approve. It makes it kind of hard to compare the different browsers
Which is exactly the point of the FIRST graph, directly above it. Sheesh.
I think you missed the point, Jeff. The information shown might be correct, but the style of graph is inappropriate. A regular bar graph (different browsers side by side) would be better than a stacked bar. Nobody really cares what the total effect of running a string process in all of the browsers sequentially would be, because nobody would do it.
Otherwise, interesting post.
Rob
I don’t like the way you’ve stacked your graphs. Makes it really hard to compare visually the different individual tests. And does that axis jump from 6 to 16 seconds? nice!
“2) It is useful to know what general class of activities in JavaScript are slow, and that’s what the bottom graph shows – string operations and bitwise operations tend to be the slowest activities in JavaScript. Just scan the relative colors to see which browser was good (or bad) at a particular class of operations in JavaScript.”
I don’t think this benchmark is suitable for figuring out what classes of activities in JavaScript are slower or faster than others. I tried to roughly balance the individual tests, but some categories have more tests than others, by design. So, for example, you can’t conclude that “regexp” stuff is faster than “access” stuff; there are more separate “access” subtests, by design, since object access is somewhat more common than regexp processing.
I have use Array.join when concatenating strings. I’ve used more than one platform that exhibits the IE style slowdown when repeatedly concatenating strings with operator+, whereas Array.join has always performed well enough, with fairly linear performance degradation (although it is not the fastest in some cases).
@eNtropicChild
I with you on the syntax front… arr.push wins over str+=otherStr for me
I did a quick test in IE… concatenating a 7 char string 20000 times:
with string concatenation (+=) :1340ms
with Array.push/Array.join :62ms
From your article: “The browser market is more competitive than it has been in years, with Opera 9.5, Firefox 3, Safari 3, and IE 8 all vying for the coveted default browser position.”
I just wonder how come you\re saying IE8 is competing to anything while it’s not even out? Actually, it’s not finished, i haven’t even seen any betas yet. I’d argue that as long as it’s not an actual product in peoples hands, it’s not competing with anything.
I love this blog but you’re saying some “odd” stuff lately. I hope it was just a typo.
Why hate on the font thing? What do fonts have to do with performance? It’s a good article except for that comment.
I think you missed the point, Jeff. The information shown might be correct, but the style of graph is inappropriate.
OK, fair enough. Valid points, all. I changed the chart.
- The top graph and bottom graph are mirrors. So if you don’t like the bottom graph, look at the top one.
They are both terrible graphs, but at least the top one gives you a visual indication of the overall performance of each browser in the total set of tests.
The bottom graph may as well be a table of numbers with '70s-ish colors splattered on it. The visual cues are at best misleading, and generally don’t help a bit. After studying the numbers a while you can figure out what it’s trying to tell you, but it misleads you in almost every way possible (for instance, the numbers on the bottom of the graph don’t help because the browsers are stacked one after another with no relation to time). My initial impression of the first bar on the graph was that opera was the slowest and firefox the fastest on the string test, in opposition to the first graph. Only be reading the numbers and taking in the rest of the graph did I realize that the position of the browsers was the same across bars and had nothing to do with the result of the test in question.
It would have been better if the bar’s full length was that of the longest time and the browsers ordered from fastest to slowest, essentially merging the more standard display of 4 bars in a group (one next to the other) into 1 multi-colored bar that still had a relationship to the scale at the bottom of the graph, rather than stacking them end to end in the same order on each line.
bah, figures you’d change it while I was typing (I was slow). Too bad I don’t really find much of interest in the actual benchmarks, though, other than the usual cautions that come along for the ride when someone releases a benchmark for their own products. At least, for once, Apple actually gives you the tools to run the benchmarks for yourself. They used to be pretty terrible about posting benchmark results with little or no information to make them repeatable.
I just did the test in Konqueror. Too bad I can’t actually compare them to anything, since my Firefox just complains that some script is taking too long to execute and then it wants to kill them. Is that how it’s supposed to work?
Maybe it’s because I’ve been reading Edward Tufte lately, but I don’t think the second graph makes much sense at all. Browsers are stacked together, yet add up to a total time? Just in case I wanted to see what would happen if I gave the same task to all browsers and see their total? And the scale jumps from 6s to 16s, corrupting the data view - if a comparative graphic is not going to use a consistent scale, the data would be better presented as a table.
At minimum I would stack the bars for each browser. As it data displays now, I don’t know what I’m supposed to get out of it.
Slashdotted… 
You’ve misspelled “Xaprb” in the test name.
@Francis Fish
Are you talking about the GWT (Google Web Toolkit)?
We are in the process of building an app using the GWT, and it is going well. Basically, Google provides several widgets for Java to replace the standard swing components… they provide a pretty complete set of controls, and as long as you use those controls in your app, you can convert the project to javascript.
Interesting…
I just ran it on a few machines here and was intrigued by the results. When I get more time I’ll do a write-up on my blog, but until then…
Out of a Sun v250 (UltraSPARC w/Solaris), HP server (P-III w/ Win2k3), and an Apple Powerbook (PowerPC), and a whitebox x64 running Solaris (all very different boxes indeed!):
- Safari on the Mac
- Firefox on x64 Solaris
- Mozilla on SPARC Solaris
- Mozilla on x64 Solaris
- Firefox on SPARC Solaris (threw errors during the test)
- Firefox on Windows (threw errors during the test)
- IE on Windows