AMD Claims of Intel Benchmarks 'Not Ethical' Scrutinized

Update ribbon (small)

8:45 pm March 2, 2007 - After BetaNews' publication late this afternoon of our original story on ZDNet blogger George Ou calling into question the validity of recent AMD benchmark claims -- after AMD had called Intel's claims into question -- AMD changed its mind and decided to comment in full. Spokesperson John Taylor's response appears in full after our original story:


5:33 pm March 2, 2007 - After a press conference on Wednesday in which AMD Executive Vice President Henri Richard told reporters he's tired of being pushed around by Intel with regard to the accuracy of its recent performance superiority claims, ZDNet blogger George Ou turned the interrogation light back onto AMD. As Ou confirmed, data AMD used to debunk some of Intel's performance claims had since been superseded by new data that substantiates Intel.

"I think we've been too quiet," Richard told reporters, "and I think part of that is because we're trained to be very honest, grounded in reality, truthful with our benchmarks, and I'm sick and tired of being pushed around by a competitor that doesn't respect the rules of fair and open competition."

Later, in a video interview with ZDNet reporter David Berlind, Richard expanded on those comments, elevating them into a serious charge: "I think using five-year-old benchmarks that nobody uses any more to pretend to have a better performance, is not ethical."

But one of the slides from a presentation Berlind gave during the conference cited official SPECint_rate2006 results dated last January 29, showing the best peak performance of systems tested for the Standard Performance Evaluation Corporation using comparable dual-core Opteron and Xeon processors. In that slide - which AMD also provided to BetaNews last February 7 - an AMD Opteron 2220 SE-based server outperformed an Intel Xeon 5160-based system by a narrow margin of three percentage points, in a test in whose category Intel had earlier claimed superiority. Intel's numbers, cited by AMD, show SPECint_rate2000 results for systems with the same two processors, in which the Xeon system performed 57% better than the Opteron system. AMD has pointed out Intel's numbers come from an outdated benchmark.


A composite of two slides from an AMD presentation last February 7, showing SPECint_rate performance figures for Intel Xeon 5160 and AMD Opteron 2220 SE processors, as cited by both companies. In the original presentation, AMD has one slide with the right-hand "reality" side partly obscured; in the second slide where "reality" is revealed, the "perception" side is obscured.

BetaNews cited these figures in a February 7 story that also examined the efficacy of AMD's claim. The numbers AMD chose to cite, we confirmed then, represented peak performance of the two systems tested, whereas the base performance figures for the same two systems showed the Xeon server with 6% better base performance.

The day after BetaNews' initial report, Dell submitted a new test result to SPEC, for a Precision 690 server with a Xeon 5160 processor clocked at 3 GHz. The new figures give the 5160 a best peak performance score of 54.9 - 6% better than AMD's best peak performance for Opteron 2220 SE - and a best base performance score of 53.2 - 17.6% better than the AMD.

In a scathing blog post this morning, George Ou invoked the dreaded "H" word in questioning whether AMD should have known the SPEC figures were updated during its press conference on February 28, and pulled the slide from its deck - on the same day Richard planned to accuse Intel of being "not ethical."

"It is a well-known fact that all companies hardware or software will present their best foot forward and favor the benchmarks that makes them look good," Ou wrote. "But leaving out your competitors up-to-date data to make it look like you're ahead in the same benchmark is very disturbing and there is a world of difference between cherry picking benchmarks versus cherry picking data. The only thing worse than that is hypocrisy."

AMD thought long and hard today about how to respond to Ou's claims, before electing to decline comment to BetaNews.

In an interview last month with AMD's senior client performance analyst Mark Welker - who incidentally is a member of SPEC committees - he acknowledged that both AMD and Intel may tend to highlight the benchmarks which most favor their processors at any one point in time. But he also suggested a remedy to the confusion to which such habits can give rise.

"From a testing standpoint, we need to look more at what a real user does," Welker told BetaNews. "There are users out there who do one thing at a time; there are also crazy users out there who have fifteen, twenty things open at once, [all of it] legitimate...We're trying to go out, talk to people and the press, survey people, and try to get end-user scenarios so that we can ask the end users, what would make sense to them?

"I want a real-world-style benchmark that...won't be a CPU-specific benchmark," Welker continued. "I can bet you dollars to doughnuts that I can write one that makes mine oh-so-much better than Intel's, and Intel can write one that looks better than us. It's going to happen. But I think if we write stuff that's for systems and stuff that will generally benefit the end user, then if the benchmark shows up that we might be weak in some spots but strong in others, that gives us something to work on, and we can evolve our processors and our systems, because we're even more of a systems house than we used to be. We can evolve our systems to help the end user more, to go out there and say, 'This is what's going to make a difference to you,' because it's not just about frequency any more. It's about the whole system."


Tonight, AMD spokesperson John Taylor provided the following response to BetaNews, which we present in its entirety:

I’d say that Mark Welker’s comments in your story are right on – at the end of the day what really matters is how a customer’s real world workloads perform on a system. That’s the discussion AMD will continue to have with customers and where AMD Opteron will continue to shine.

As for responding to Mr. Ou’s blog, we clearly showed the date of the benchmark and link to the Web site we were pulling from for this very reason – benchmarks can get updated ahead of the corporate legal review process. This was a matter of weeks/days, not months/years as in the Intel presentation from last week. That same Intel presentation made some comparisons to single-core AMD Opteron processor performance from several years ago, and less than half the time made comparisons to current-generation highest-performing AMD Opteron processors.

We also clearly marked the names and model numbers of the processor models we were comparing.

The immediately previous slide that Mario Rivas presented – that Mr. Ou does not show but BetaNews thankfully does – spoke to Intel portraying a 57% advantage over AMD Opteron-based systems. We were addressing the misleading perception of gross performance advantages that are at the center of Intel’s months-long marketing campaign which reached new lows during a presentation to Wall Street analysts last week.

Even if Intel eked out a slightly higher score in recent days, the basic point remains unchanged; roughly 50 percentage points of Intel’s claims get lopped off. And certainly, we may again top the new Intel scores in the weeks/months to come. There is a significant difference between Intel endeavoring to foster perceptions of a 57% performance advantage, and going back and forth on a 2-3% difference. And AMD Opteron remains the standard-bearer for performance-per-watt, an important direct comparison that Intel continues to avoid.

There may continue to be updates of benchmarks scores that amount to back-and-forth until the introduction of AMD “Barcelona” server processors. “Barcelona” is the most significant architectural enhancement from AMD since the introduction of the AMD64 architecture and AMD Opteron processors in April 2003.

22 Responses to AMD Claims of Intel Benchmarks 'Not Ethical' Scrutinized

© 1998-2024 BetaNews, Inc. All Rights Reserved. Privacy Policy - Cookie Policy.