New algorithm for determining overclocking in charts

[David (PassMark)]

After our study of the 'Distribution of benchmark results for Intel i5 CPUs' we decided that we could improve the accuracy of the charts by better filtering of the overclocked machines.

What we were doing up until today (2/Feb/2012) was including a CPU in the standard benchmark chart if the measured base clock speed of the CPU was within 15% of the factory specified speed. Anything over 15% got moved to the overclocking chart.

While it could be argued that the 15% level wasn't strict enough, this worked pretty well in the past. A 3.0Ghz CPU found to be running at less than 3.3Ghz at base speed got put into the standard chart, while anything at 3.3Ghz and above was classified as overclocked. But with the introduction of the Turbo Boost feature in Intel i5 and i7 CPUs new ways of overclocking are now available. It is now possible (and even common) to overclock just the turbo speed of the CPU, leaving the base speed unchanged, or only slightly changed.

The result was that CPUs that had overclocked turbo multipliers were slipping through the overclocking detection algorithm. The main noticeable consequence of this was that the 'K' series CPUs (e.g. Core i5-2500K, i7-2600K, i7-2700K, etc..) were ranking higher than they should have been in the standard charts.

From today we are adding new data into the database on factory and measured turbo clock speeds and tweaking the algorithm to trigger the overclocking flag based on either overclocking of the base speed, or overclocking of the turbo speed. We are also going to change the criteria for what is overclocked from 15% of the clock speed to 90Mhz difference from the factory speeds. This will mean even lightly overclocked CPUs are now going to moved into the overclocked chart. Hopefully this won't result in too many false positives.

The result will be a bunch of CPUs with high benchmark results being moved from the standard chart to the overclocking chart.

We think the 'K' series CPUs will still rank better than their almost identical non K counterparts for other reasons however. But this should narrow the gap and improve the accuracy of the charts.

[David (PassMark)]

This change has just gone live. You might need to force a refresh in your browser to see the new results.

Here are some examples of the impact. The old values were what was in the charts yesterday. The new values is what we have today (3/Feb/2012).

CPU - Old / New CPUMark Rating
Intel 2600K - 10074 / 9106
Intel 2600 - 8964 / 8912

Intel 2500K - 7991 / 6739
Intel 2500 - 6649 / 6614

AMD FX-8150 - 8369 / 8226
AMD FX-6100 - 5714 / 5556
AMD 1090T - 6069 / 5977

So there was a gap of 1110 in the CPUmark score between the 2600K and 2600. After this change in overclocking detection the gap is now 194 (2%). Which is much more reasonable considering these are effectively the same CPU in their stock state.

[David (PassMark)]

Also here is an interesting break down of what we have measured for the turbo speed of an i5-2500K.

Note that for the i5-2500K, the factory turbo speed is 3.7Ghz. Which makes our cut off point 3.79Ghz. The red dotted line show the break point where we have classified a CPU as being overclocked based on its turbo clock speed.



Based on this criteria 49% of baselines people have sent in for the i5-2500K are overclocked.

By comparison, for the non 'K' chip, the i5-2500, only 19% were found to be overclocked.

[David (PassMark)]

The other change today was a change to the information displayed on the overclocking chart.
http://www.cpubenchmark.net/overclocked_cpus.html

Previously we displayed the average percentage increase in the clock speed for each type of CPU.

This calculation became a lot harder to do as both the base clock speed (without turbo active) and the turbo clock speed can be separately overclocked. Also many of the older CPUs don't support turbo mode. There was also a problem that we don't have the turbo mode value for even all of the new CPUs, as the older releases of PerformanceTest never collected this value.

The simplest solution in the end was to stop reporting clock speed increases, and instead report performance increases.

So now for each CPU in the Overclocked chart you can mouse over the row in the chart to see the average percentage increase in performance you get by overclocking that model of CPU.

Here are some examples from popular CPUs from the OC Chart.

Intel Core i7-2700K @ 3.50GHz, 18% performance increase
Intel Core i7-2600K @ 3.40GHz, 24% performance increase

AMD FX-8150 Eight-Core, 18% increase
AMD FX-8120 Eight-Core, 25% increase

AMD Phenom II X6 1100T, 16% increase
AMD Phenom II X6 1090T, 18% increase
AMD Phenom II X6 1075T, 23% increase
AMD Phenom II X6 1055T, 24% increase

As a general rule if you start with a CPU with a low clock speed, you can overclock it more than a CPU with a higher initial clock speed.

So if you are an overlocker looking for the best value, then it makes sense to buy the cheaper CPU, lower clocked, CPUs and overclock them harder.

This is really just statistical proof of what overclocker have known for years. Cheap CPUs have more overclocking headroom.

[David (PassMark)]

A new patch release of PerformanceTest V7 was made public today, build 1026.
http://www.passmark.com/products/pt_history.htm

Changes include better turbo mode clock speed detection on the newer AMD chips. Which should in turn allow better detection of overclocking and thus more charts accurate benchmark charts (over time).

[gmortimer]

Just one thing is puzzling about this - non K-version (desktop) and non XM-version (mobile) processors are now appearing in the overclocked chart. As I understand it, these cannot have their turbo multipliers changed, and are difficult to overclock in any other way. I must have missed something...

[David (PassMark)]

Non K chips can still be overclocked. But only to a much more limited extent.

[rsn10100]

I'm curious to know more about how Passmark creates it's benchmark score. For one, how does integrated graphics affect CPU score or does the CPU score include any integrated graphics benchmarks at all?

Another thing I noticed while searching around is that the CPU list includes only an overall score that some might view as 'synthetic'. Honestly, I see no other way of obtaining an accurate benchmark of a CPU then to test all of it's capabilities, but since not all applications use all of these capabilities would it not be better to list benchmarks of individual features?

I believe that showing results of individual tests might help individuals choose the best upgrade path for them. Benchmarks of things such as standard x86, 64-bit instructions, MMX, SSE, SSE2, ..., AVX, integrated GPU, etc. can give much more relevant information to the user instead of having to rely on an overall score where the calculation of such a score could be flawed. It would also help identify where overclocking is least/most beneficial and be a good tie breaker for closely matched devices.

Does this make any sense?

[David (PassMark)]

> how does integrated graphics affect CPU score.

It doesn't significant effect the CPU result.

> Does this make any sense?

No. Not for 99% of users.
Even people with university degrees in computer science are going to struggle to describe what MMX and AVX are. Then to know which applications make use of these CPU features is nearly impossible.

I interview a lot of programmers applying for jobs. Most can't describe the real difference between 32bit and 64bit. So the general population has no chance.

So the breakdown you are suggesting would not be more relevant.

What would be more relevant would be a benchmark of World of Warcraft for people who play World of Warcraft. And a benchmark of Photoshop for Photoshop users. etc...
But making it more relevant to some users makes it much less relevant to others.

Thus the need for a simple broad based CPU benchmark that has some relevance to every computer user.

[rsn10100]

No. Not for 99% of users.
Even people with university degrees in computer science are going to struggle to describe what MMX and AVX are. Then to know which applications make use of these CPU features is nearly impossible.

I don't doubt that they'd struggle, but I do think you are underestimating them with MMX. I don't have any degree in computer science, but I could explain the technologies.

I interview a lot of programmers applying for jobs. Most can't describe the real difference between 32bit and 64bit. So the general population has no chance.

Wow... Maybe they didn't start programming when they were 12 like real programmers?
I don't doubt that many programmers don't know what's really under the hood, but I think you are underestimating a lot of other people here.

So the breakdown you are suggesting would not be more relevant.

So I'm basically too geek to be relevant. Sorry no offense, but it all just sounds like a bunch of marketing fud to me. I guess I'm just that 1% that looks at numbers like everybody else looks at shiny objects. I would rather see factual scores of something that's real though rather then a score calculated with magic (although I do love the magic )... But you're right, it would only be useful to that 1% and it is a lot of work and a lot of data to collect.