Recent test results announced by SGI are touting the Altix UV 1000 as the world’s fastest Java application server. While I love the Altix UV and think it’s a cracking bit of kit, this result highlights how useless benchmarks are.

SGI claim:

• SGI Altix UV 1000, with 256 cores and 128 JVMs, beat ScaleMP, its nearest competitor, by 82 percent in Java performance with throughput of 12,665,917 BOPS using Oracle JRockit.
• SGI Altix UV 1000, with 256 cores, beat Fujitsu/Sun, its nearest competitor, by 60 percent with a single Oracle Java HotSpot JVM performance of 2,818,350 BOPS.

As Darth Vader says – “Impressive. Most impressive.” – but lets look at those figures in more detail. A 256 core Altix UV 1000 running 128 JVMs? Who on earth would buy a massive single system image machine, with massive shared memory performance, and then carve it up with hundreds of JVMs – which are pegged to cores?

This is nonsense. Far better to pick a smaller machine – say, an Altix UV 100 – which would much more realistically be used for this sort of task. Or an Altix XE cluster, which would give both good parallelism and also high availability.

Joerg Mollenkamp has even more details, comparing these results to those of a Sun T5440. The price difference is extreme, leading to some embarrassing price/performance comparisons, which also highlights how meaningless these sort of benchmark results really are.

There’s no doubt those are some cracking results from the Altix UV 1000. But they’re meaningless, pie in the sky figures, and I can safely say anyone who ordered one of those to run Java apps on would be laughed at, and then fired on the spot.

A far more meaningful result I’d like to see from SGI would be pitching the Altix UV where it really needs to go – into the corporate data centre. Let’s see some data warehouse figures using Oracle and Sybase for big data workloads that can take advantage of all that fast, shared memory.

SGI need to move out of the pure HPC play for their big kit, and add in more sales to big business. (They’ve need to do this since the first big Origins came out, but that’s a rant for another day). This means playing to the kits strength, and posting JVM benchmarks like this accomplishes nothing apart from opening them up to some – very valid – criticism.

SGI have just posted some pretty decent SPEC benchmarks, and it’s clear that they’re aiming straight for the datacentre, calling out IBM, HP and Sun’s big SSI (Single System Image) machines. The benchmarks posted are for the Altix 4700, and the cynic in me says this is a bit of “rah rah rah” flag waving exercise before we see a Xeon version rolled out.

From the press release:

* SGI is more than five times faster than its next closest SSI competitor in the SPECfp_rate_base2006 floating point performance benchmark. Altix 4700 proved 5.7 times faster than the Sun SPARC Enterprise 9000, its next closest SSI competitor; IBM and HP products trailed further behind.

* SGI performed over four times better in the SPECint_rate_base2006 integer performance benchmark. Altix 4700 proved 4.3 times faster than the Sun product, its next closest SSI competitor.

* SGI outperforms IBM by almost three times in Java performance with SPECjbb2005 benchmark. Altix 4700 is the leader in Java performance as measured by SPECjbb2005, outperforming IBM by 2.8 times.

* SGI has five times higher aggregate memory bandwidth than its next closest competitors. Altix 4700 has the highest aggregate memory bandwidth in the world, five times higher than its next closest competitors, NEC and IBM.

The arguments over how unrealistic and artificial the SPEC benchmarks are will rage forever, but these are some pretty impressive numbers and some pretty bold claims from SGI. You can read the full press release here.

I suppose I should be resigned to the whole benchmark silliness, where vendors highly tune their own systems and then run favourable benchmark tools – that have no real-world relevance – and compare the results against dissimilar and un-tuned gear from their rivals.

It’s the biggest ego match in the industry, and it’s all pretty sad. Are people really naive enough to buy kit based on these benchmarks, instead of trying out different vendors’ solutions in-house using their own real world workloads?

Regardless, I find the latest press release from Sun touting their ‘HPC leadership’ a bit much.

In it, Sun cites SGI beating FLUENT benchmarks. Yet, the press release doesn’t mention numbers at all. The link to the actual FLUENT benchmarks shows no values at all for the X2250 cluster Sun used, and the main Sun benchmarks website makes no mention of these ‘HPC dominating’ benchmark figures either. Surely if you’re going to trumpet the results, then you should also be making the actual figures available?

bmseer (a blogger from Sun who I have a huge respect for, and who regularly picks apart the outrageous benchmark figures from IBM) should be hanging his head in shame right about now ;-)

Regardless, the Sun benchmark site is well worth a visit at http://www.sun.com/benchmarks/

Silicon Graphics could take a leaf out of Sun’s book here and be far more pro-active and upfront with their current systems’ performance figures.

Meanwhile I live in hope for the day when we have a benchmark that measures the enormity of a vendor’s lies in their press releases. Like the Top 500, I’m sure IBM would be way out in front.