New algorithm for determining overclocking in charts

[David (PassMark)]

After our study of the 'Distribution of benchmark results for Intel i5 CPUs (http://www.passmark.com/forum/showthread.php?t=3502)' we decided that we could improve the accuracy of the charts (http://www.cpubenchmark.net/high_end_cpus.html) by better filtering of the overclocked machines.

What we were doing up until today (2/Feb/2012) was including a CPU in the standard benchmark chart if the measured base clock speed of the CPU was within 15% of the factory specified speed. Anything over 15% got moved to the overclocking chart (http://www.cpubenchmark.net/overclocked_cpus.html).

While it could be argued that the 15% level wasn't strict enough, this worked pretty well in the past. A 3.0Ghz CPU found to be running at less than 3.3Ghz at base speed got put into the standard chart, while anything at 3.3Ghz and above was classified as overclocked. But with the introduction of the Turbo Boost feature in Intel i5 and i7 CPUs new ways of overclocking are now available. It is now possible (and even common) to overclock just the turbo speed of the CPU, leaving the base speed unchanged, or only slightly changed.

The result was that CPUs that had overclocked turbo multipliers were slipping through the overclocking detection algorithm. The main noticeable consequence of this was that the 'K' series CPUs (e.g. Core i5-2500K, i7-2600K, i7-2700K, etc..) were ranking higher than they should have been in the standard charts.

From today we are adding new data into the database on factory and measured turbo clock speeds and tweaking the algorithm to trigger the overclocking flag based on either overclocking of the base speed, or overclocking of the turbo speed. We are also going to change the criteria for what is overclocked from 15% of the clock speed to 90Mhz difference from the factory speeds. This will mean even lightly overclocked CPUs are now going to moved into the overclocked chart. Hopefully this won't result in too many false positives.

The result will be a bunch of CPUs with high benchmark results being moved from the standard chart to the overclocking chart.

We think the 'K' series CPUs will still rank better than their almost identical non K counterparts for other reasons (http://www.passmark.com/forum/showthread.php?t=3502) however. But this should narrow the gap and improve the accuracy of the charts.

[David (PassMark)]

This change has just gone live. You might need to force a refresh in your browser to see the new results.

Here are some examples of the impact. The old values were what was in the charts yesterday. The new values is what we have today (3/Feb/2012).

CPU - Old / New CPUMark Rating
Intel 2600K - 10074 / 9106
Intel 2600 - 8964 / 8912

Intel 2500K - 7991 / 6739
Intel 2500 - 6649 / 6614

AMD FX-8150 - 8369 / 8226
AMD FX-6100 - 5714 / 5556
AMD 1090T - 6069 / 5977

So there was a gap of 1110 in the CPUmark score between the 2600K and 2600. After this change in overclocking detection the gap is now 194 (2%). Which is much more reasonable considering these are effectively the same CPU in their stock state.

[David (PassMark)]

Also here is an interesting break down of what we have measured for the turbo speed of an i5-2500K.

Note that for the i5-2500K, the factory turbo speed is 3.7Ghz. Which makes our cut off point 3.79Ghz. The red dotted line show the break point where we have classified a CPU as being overclocked based on its turbo clock speed.

http://www.passmark.com/images/forumimages/Intel-i5-2500K-turbo-speed-distribution.png

Based on this criteria 49% of baselines people have sent in for the i5-2500K are overclocked.

By comparison, for the non 'K' chip, the i5-2500, only 19% were found to be overclocked.

[David (PassMark)]

The other change today was a change to the information displayed on the overclocking chart.
http://www.cpubenchmark.net/overclocked_cpus.html

Previously we displayed the average percentage increase in the clock speed for each type of CPU.

This calculation became a lot harder to do as both the base clock speed (without turbo active) and the turbo clock speed can be separately overclocked. Also many of the older CPUs don't support turbo mode. There was also a problem that we don't have the turbo mode value for even all of the new CPUs, as the older releases of PerformanceTest never collected this value.

The simplest solution in the end was to stop reporting clock speed increases, and instead report performance increases.

So now for each CPU in the Overclocked chart you can mouse over the row in the chart to see the average percentage increase in performance you get by overclocking that model of CPU.

Here are some examples from popular CPUs from the OC Chart.

Intel Core i7-2700K @ 3.50GHz, 18% performance increase
Intel Core i7-2600K @ 3.40GHz, 24% performance increase

AMD FX-8150 Eight-Core, 18% increase
AMD FX-8120 Eight-Core, 25% increase

AMD Phenom II X6 1100T, 16% increase
AMD Phenom II X6 1090T, 18% increase
AMD Phenom II X6 1075T, 23% increase
AMD Phenom II X6 1055T, 24% increase

As a general rule if you start with a CPU with a low clock speed, you can overclock it more than a CPU with a higher initial clock speed.

So if you are an overlocker looking for the best value, then it makes sense to buy the cheaper CPU, lower clocked, CPUs and overclock them harder.

This is really just statistical proof of what overclocker have known for years. Cheap CPUs have more overclocking headroom.

[David (PassMark)]

A new patch release of PerformanceTest V7 was made public today, build 1026.
http://www.passmark.com/products/pt_history.htm

Changes include better turbo mode clock speed detection on the newer AMD chips. Which should in turn allow better detection of overclocking and thus more charts accurate benchmark charts (over time).

[gmortimer]

Just one thing is puzzling about this - non K-version (desktop) and non XM-version (mobile) processors are now appearing in the overclocked chart. As I understand it, these cannot have their turbo multipliers changed, and are difficult to overclock in any other way. I must have missed something...

[David (PassMark)]

Non K chips can still be overclocked. But only to a much more limited extent.

[rsn10100]

I'm curious to know more about how Passmark creates it's benchmark score. For one, how does integrated graphics affect CPU score or does the CPU score include any integrated graphics benchmarks at all?

Another thing I noticed while searching around is that the CPU list includes only an overall score that some might view as 'synthetic'. Honestly, I see no other way of obtaining an accurate benchmark of a CPU then to test all of it's capabilities, but since not all applications use all of these capabilities would it not be better to list benchmarks of individual features?

I believe that showing results of individual tests might help individuals choose the best upgrade path for them. Benchmarks of things such as standard x86, 64-bit instructions, MMX, SSE, SSE2, ..., AVX, integrated GPU, etc. can give much more relevant information to the user instead of having to rely on an overall score where the calculation of such a score could be flawed. It would also help identify where overclocking is least/most beneficial and be a good tie breaker for closely matched devices.

Does this make any sense?

[David (PassMark)]

> how does integrated graphics affect CPU score.

It doesn't significant effect the CPU result.

> Does this make any sense?

No. Not for 99% of users.
Even people with university degrees in computer science are going to struggle to describe what MMX and AVX are. Then to know which applications make use of these CPU features is nearly impossible.

I interview a lot of programmers applying for jobs. Most can't describe the real difference between 32bit and 64bit. So the general population has no chance.

So the breakdown you are suggesting would not be more relevant.

What would be more relevant would be a benchmark of World of Warcraft for people who play World of Warcraft. And a benchmark of Photoshop for Photoshop users. etc...
But making it more relevant to some users makes it much less relevant to others.

Thus the need for a simple broad based CPU benchmark that has some relevance to every computer user.

[rsn10100]

No. Not for 99% of users.
Even people with university degrees in computer science are going to struggle to describe what MMX and AVX are. Then to know which applications make use of these CPU features is nearly impossible.I don't doubt that they'd struggle, but I do think you are underestimating them with MMX. I don't have any degree in computer science, but I could explain the technologies.


I interview a lot of programmers applying for jobs. Most can't describe the real difference between 32bit and 64bit. So the general population has no chance.
Wow... Maybe they didn't start programming when they were 12 like real programmers? :p
I don't doubt that many programmers don't know what's really under the hood, but I think you are underestimating a lot of other people here.


So the breakdown you are suggesting would not be more relevant.So I'm basically too geek to be relevant. :confused: Sorry no offense, but it all just sounds like a bunch of marketing fud to me. I guess I'm just that 1% that looks at numbers like everybody else looks at shiny objects. I would rather see factual scores of something that's real though rather then a score calculated with magic (although I do love the magic :D)... But you're right, it would only be useful to that 1% and it is a lot of work and a lot of data to collect.

[David (PassMark)]

You can actually see the individual numbers for the CPU tests by downloading the baselines from within PerformanceTest. The breakdown isn't as detailed as you are suggesting above, but it is more detailed that just a single CPUMark figure.


but I think you are underestimating a lot of other people here.

I disagree. Only assembly language programmers are going to really understand MMX, SSEx, etc.. And even with an understanding of what it is you still can't tell what real applications use which technology.

If we take your argument to the logical conclusion then we would end up with 1000s of benchmarks. e.g. one for 32bit addition of single precision floats, one for 32bit addition 8 bit integers, one for 32bit addition of a deferenced address holding a single word that is in cache, etc... It really has to be brought up to a higher level.

You might also be interested in this short study on the speed and breakdown of CPU instructions (http://www.passmark.com/forum/showthread.php?t=3348).

[rsn10100]

You can actually see the individual numbers for the CPU tests by downloading the baselines from within PerformanceTest. The breakdown isn't as detailed as you are suggesting above, but it is more detailed that just a single CPUMark figure.
This is cool. I never really explored it.


... And even with an understanding of what it is you still can't tell what real applications use which technology.
Probably only 1%...


e.g. one for 32bit addition of single precision floats, one for 32bit addition 8 bit integers, one for 32bit addition of a deferenced address holding a single word that is in cache, etc... It really has to be brought up to a higher level.Yeah, that makes sense. And I guess anyone really in need of those results would know how to write the code to profile it themselves (or could figure it out anyway).


You might also be interested in this short study on the speed and breakdown of CPU instructions (http://www.passmark.com/forum/showthread.php?t=3348).
Now this was interesting. I knew that divisions were slow, but I had no idea that they were that much slower. :o I got a lot more reading to do. ;)

[rdeleonp]

This change has just gone live. You might need to force a refresh in your browser to see the new results.

Here are some examples of the impact. The old values were what was in the charts yesterday. The new values is what we have today (3/Feb/2012).

CPU - Old / New CPUMark Rating
Intel 2600K - 10074 / 9106
Intel 2600 - 8964 / 8912

Intel 2500K - 7991 / 6739
Intel 2500 - 6649 / 6614

AMD FX-8150 - 8369 / 8226
AMD FX-6100 - 5714 / 5556
AMD 1090T - 6069 / 5977

So there was a gap of 1110 in the CPUmark score between the 2600K and 2600. After this change in overclocking detection the gap is now 194 (2%). Which is much more reasonable considering these are effectively the same CPU in their stock state.

Hi.

What could explain the current differences between the i5-2500K scores and the i5-2550K scores?

They have basically the same specs:
http://ark.intel.com/compare/52210,65647
... but the scores are very different:
Intel Core i5-2500K @ 3.30GHz: 6,745
Intel Core i5-2550K @ 3.40GHz: 8,653
Thanks for your time.

[David (PassMark)]

It has been fixed up. Results are now (1/March/2012),

Intel Core i5-2500K @ 3.30GHz: 6,745
Intel Core i5-2550K @ 3.40GHz: 7,167

The i5-2550K was new, so we didn't have the max factory turbo speed set in the database. It is there now. So the split between normally clocked and overclocked should now be done properly for the 2550K.

[rdeleonp]

It has been fixed up. Results are now (1/March/2012),

Intel Core i5-2500K @ 3.30GHz: 6,745
Intel Core i5-2550K @ 3.40GHz: 7,167

The i5-2550K was new, so we didn't have the max factory turbo speed set in the database. It is there now. So the split between normally clocked and overclocked should now be done properly for the 2550K.

Thanks for the info.

Regards.

[a7751218]

PassMark admin: Many Thanks for clarifying this!

I've been using the cpu charts for quite a while now and was always concerned about overclocked cpu's leaking into the normal bechmarks.

Your excellent explanation and change in algorithms allows me to put more trust in your benchmarks now. And I find your new 90mhz threshold to be a fairly wise choice considering how cpu manufacturers release minor upgrades on CPU, i.e. i7 2600 vs i7 2700 :)

However, it does concern me a little that other hardware (motherboard / ram) can have such a difference in cpu clock (~90mhz)?

Also, I hope you guys keep on top of new CPU's, their stock and turbo speeds as new ones are constantly released.

[David (PassMark)]

We aren't publishing graphs for RAM performance nor motherboards, at least for the moment. So CPU overclocking impacting on RAM speeds isn't a big issue for us.

There has been a ridiculous number of CPUs released of the last 24 months. Some are also very rare, engineering samples, private OEM releases, or buggy so the model number doesn't report correctly. So finding the correct specs can be hard.

[rdeleonp]

There has been a ridiculous number of CPUs released of the last 24 months. Some are also very rare, engineering samples, private OEM releases, or buggy so the model number doesn't report correctly. So finding the correct specs can be hard.

Hi.

If you don't mind... can you validate the following scores?

AMD FX-8100 : 6508
AMD FX-6200 : 8084

Thanks for your time.

Regards.

[David (PassMark)]

Both these CPUs are just a few days old. And very rare at the moment.
The FX-8100 result looks about right.
For the FX-6200, the only results we had turned out to be overclocked, and not just by a small amount. Like the i5-2550K above, it was missing the turbo speed in the database. This has now been fixed. The consequence of this however is that we now don't have any standard result for the FX-6200 and it has disappeared from the standard chart.

[David (PassMark)]

After fixing up the above it became obvious that there was also another overclocking issue that impacted the charts with the AMD Llano chips. Details can be found here,
http://www.passmark.com/forum/showthread.php?t=3635
and here
http://www.passmark.com/forum/showthread.php?t=3580

This should be worked-around in PerformanceTest V7.0 build 1028.

In conjunction with the release of the 1028 build we manually adjusted the rankings of the AMD Llano chips in the CPU charts to manually filter out the overclocked results (as the automatic filtering wasn't always working for these chips). This has dropped the AMD Llano's down the charts a bit as of the 23/Mar/2012.

Update: Turns out there might be another AMD Llano issue. This time with a CPU bug (http://www.passmark.com/forum/showthread.php?t=3656).