What I'd like to see are benchmarks that involve visiting some sample of sites out of the Alexa top 100 and performing standard tasks (reading Gmail, finding a certain thing on Amazon.com, etc.) I know that's dependent on things like network conditions, so you'd have to run it a lot of times to get meaningful results, but it'd still provide a more meaningful result than a lot of synthetic benchmarks.
I agree that major improvements are needed. SunSpider, Octane, and Kraken are all pure JavaScript benchmarks, very microbenchmarky (look at bitwise-and in the SunSpider suite). Oort Online is a benchmark of WebGL performance—WebGL is definitely worth benchmarking, but how common is it in people's daily browsing? HTML5Test tests whether a bunch of "HTML5" features are sorta present, not whether they actually work or anything—it encourages broken implementations that support just enough to pass the test.
The state of browser benchmarking is pretty bad. Right now the common suites of benchmarks overwhelmingly focus on what is easy to measure, not what reflects real-world perception of browser performance.
It feels to me like current benchmarks all measure execution time, not user time. Let's say you go from a reasonably sized web page to another, and then hit the back button. How long to load the previous page? Right click on a link and select open in new tab. How long does it take to open a new tab? It's definitely part of the user's perception of the browser's speed, but I don't think stuff like SunSpider and Octane even tries to measure it.
Heavily related to this, one key insight that I think has been neglected is that JavaScript benchmarks do a bad job of measuring cold code. Most code on the Web is cold, because much Web browsing consists of fresh page loads. Heavily optimizing JITs are exactly what you don't want for cold code: time spent compiling is time you could have been spending running the code. But benchmarks consist of running the same code repeatedly in a long-running loop, which encourages development of sophisticated, highly-optimizing JITs that spend a lot of time in up-front compilation, which end up being totally pointless for JS whose sole purpose is to run once to place an ad somewhere on the page.
It's easy to forget that JavaScript is still primarily a scripting language used to perform high-level application tasks. It's awesome that JS has gone so far low-level, and we should continue to push it as far as it can, but I think the bread and butter that users care about is not so much heavy number crunching but rather quick cold code that needs to run once or twice and get out of the way.
> Heavily optimizing JITs are exactly what you don't want for cold code: time spent compiling is time you could have been spending running the code.
I agree with your overall point of cold cold performance being paramount. But I don't think this follows. A heavily optimizing JIT can have very good cold-code performance if the JIT is especially fast and/or it makes especially good decisions about when to JIT.
The important trait is short break-even time: how long after JIT-ting some code that you break even on the investment of JITting time.
When all I knew was the JVM I assumed that JIT-compiling inherently had a long warm-up / break-even time. Now that I've experienced LuaJIT I know this isn't true. I haven't benchmarked the major JIT-ting JS engines but they seem pretty good at this too?
By "a heavily optimizing JIT" I mean "the specific mode of a larger JIT-enabled engine that compiles slowly but emits good code". So IonMonkey, Crankshaft, SpeculativeJIT/FTL are all JITs that live inside a larger engine that switches between them as needed. In JS land we usually call the individual modes "JITs" even though they may (or may not) share a bunch of code.
I don't know if it's a coincidence that around the time that paper was released Microsoft were considerably behind the curve in terms of js engine performance. The paper also misses the fact that excellent hot-code performance opens up new frontiers for the web like games, photo editing, and similar compute-intensive applications that are not well represented by their sample of popular sites since those are, by definition, limited by what was performant in extant browsers.
Well, yes and no. Pure JavaScript benchmarks (SunSpider, Octane, Kraken) are all more or less microbenchmarks, at least to the degree that they're testing only one specific kernel of an application (even if Octane and Kraken are justifable since they're pulling largish kernels from real code). A pure JS "cold" benchmark would be an even poorer microbenchmark, since it carries a large presumption that the cold JS is the bottleneck. In real JS code of the stuff you're talking about, you might be able to easily get measurable performance gains by, say, improving FFI between JS and DOM code, or improving the DOM code itself.
A better benchmark would be measuring page load times (or energy usage) or dynamic operations that included such cold JS code, since it would illustrate where performance improvements would be most beneficial.
Have you actually looked at some of these benchmarks? For example, Octane contains "Code loading", sunspider has "code decompression" (not true code loading, but at least it does de-minification).
Of course it's important, and all borwsers are developed with this in mind - but it's not possible to create reproducable test suite (with a score given) that will test actual sites - sites change
> I think the bread and butter that users care about is not so much heavy number crunching but rather quick cold code that needs to run once or twice and get out of the way.
I care about HTML and CSS with as little JavaScript as possible. The best code to optimize is the one isn't there in first place.
Unfortunately the way it measures JS perf is questionable- it essentially records all operations during page load then replays them for the benchmark. This has the effect of unrolling all of the loops and disabling many of the optimizations that would have been applied to the original code.
Good point. I have a surface and want to use IE 11 more because its touch screen friendly, however the back button performance is slow sometimes. Can be very frustrating, so I typically use Chrome. Hopefully Edge fixes that.
I hate the HTML5Test (and alternative) sites with a passion for the exact reason you say.
HTML5Test lists iOS 8 as having IndexedDB support but it's so broken it's unusable on that platform. Even Mozilla completely disabled it in their LocalForage library due to it's issues.
HTML5test is OK but caniuse.com shows more but people will complain about every browser test site since I started doing web dev 11 years ago.
Here's the thing. Edge is IE without the legacy baggage and that's all it is. No different. It's still behind every other browser in its ability to comply with standards and APIs. Despite Microsoft's claim that it will be an "evergreen" browser, that it will update more often, we have not seen this so the proof needs to be done.
The thing I'm happiest about is the user-agent change so poorly written JavaScript / CSS that was written for webkit instead of standards will now magically start working.
They do talk about the chicken and egg situation, where top sites will optimize for engines, then engines optimize for sites... i.e. it highly depends on the sites - of course all browsers test on top sites and optimize for them. But then the cycle repeats.
