Recent test results announced by SGI are touting the Altix UV 1000 as the world’s fastest Java application server. While I love the Altix UV and think it’s a cracking bit of kit, this result highlights how useless benchmarks are.

SGI claim:

• SGI Altix UV 1000, with 256 cores and 128 JVMs, beat ScaleMP, its nearest competitor, by 82 percent in Java performance with throughput of 12,665,917 BOPS using Oracle JRockit.
• SGI Altix UV 1000, with 256 cores, beat Fujitsu/Sun, its nearest competitor, by 60 percent with a single Oracle Java HotSpot JVM performance of 2,818,350 BOPS.

As Darth Vader says – “Impressive. Most impressive.” – but lets look at those figures in more detail. A 256 core Altix UV 1000 running 128 JVMs? Who on earth would buy a massive single system image machine, with massive shared memory performance, and then carve it up with hundreds of JVMs – which are pegged to cores?

This is nonsense. Far better to pick a smaller machine – say, an Altix UV 100 – which would much more realistically be used for this sort of task. Or an Altix XE cluster, which would give both good parallelism and also high availability.

Joerg Mollenkamp has even more details, comparing these results to those of a Sun T5440. The price difference is extreme, leading to some embarrassing price/performance comparisons, which also highlights how meaningless these sort of benchmark results really are.

There’s no doubt those are some cracking results from the Altix UV 1000. But they’re meaningless, pie in the sky figures, and I can safely say anyone who ordered one of those to run Java apps on would be laughed at, and then fired on the spot.

A far more meaningful result I’d like to see from SGI would be pitching the Altix UV where it really needs to go – into the corporate data centre. Let’s see some data warehouse figures using Oracle and Sybase for big data workloads that can take advantage of all that fast, shared memory.

SGI need to move out of the pure HPC play for their big kit, and add in more sales to big business. (They’ve need to do this since the first big Origins came out, but that’s a rant for another day). This means playing to the kits strength, and posting JVM benchmarks like this accomplishes nothing apart from opening them up to some – very valid – criticism.

McObject are one of those database vendors who you don’t normally hear of, but who are really pushing the boundaries of what can be done with your data.

Their product, extremeDB-64, is written to take advantage of large memory systems by pegging the entire dataset in physical RAM. The advantages are pretty obvious – as are the downsides as well. The McObject guys have really thought about the problems, though, and extremeDB-64 is an impressive database solution.

What’s more impressive is McObject’s recent benchmark and scalability testing, where they test a 1.17 Terabyte, 15.54 billion row in-memory database on a 160-core SGI Altix 4700 server. They measured transaction throughput of up to 87.78 million query transactions per second, which is the sort of uber data-warehousing capability I know a number of businesses would love to get their hands on.

The benchmark white paper is available as a free download – head on over to this page to enter your details.

I suppose I should be resigned to the whole benchmark silliness, where vendors highly tune their own systems and then run favourable benchmark tools – that have no real-world relevance – and compare the results against dissimilar and un-tuned gear from their rivals.

It’s the biggest ego match in the industry, and it’s all pretty sad. Are people really naive enough to buy kit based on these benchmarks, instead of trying out different vendors’ solutions in-house using their own real world workloads?

Regardless, I find the latest press release from Sun touting their ‘HPC leadership’ a bit much.

In it, Sun cites SGI beating FLUENT benchmarks. Yet, the press release doesn’t mention numbers at all. The link to the actual FLUENT benchmarks shows no values at all for the X2250 cluster Sun used, and the main Sun benchmarks website makes no mention of these ‘HPC dominating’ benchmark figures either. Surely if you’re going to trumpet the results, then you should also be making the actual figures available?

bmseer (a blogger from Sun who I have a huge respect for, and who regularly picks apart the outrageous benchmark figures from IBM) should be hanging his head in shame right about now ;-)

Regardless, the Sun benchmark site is well worth a visit at http://www.sun.com/benchmarks/

Silicon Graphics could take a leaf out of Sun’s book here and be far more pro-active and upfront with their current systems’ performance figures.

Meanwhile I live in hope for the day when we have a benchmark that measures the enormity of a vendor’s lies in their press releases. Like the Top 500, I’m sure IBM would be way out in front.