PerformanceTest for Tender Responses

Why use Performance Test for tenders?

When preparing for tenders, you can utilize data from our benchmark graphs to help ensure that the proposed parts meet the expected level of performance.

PassMark has collected the baselines benchmarks of over a million computers and made them available in our network of industry recognized benchmark sites such as pcbenchmarks.net, cpubenchmark.net, videocardbenchmark.net, harddrivebenchmark.net and more.

How does Passmark generate these benchmarks?

• Benchmark results (“Baselines”) are gathered from users’ submissions to the PassMark web site as well as from internal testing. These results are submitted to the website with the PerformanceTest Software.
• The submissions are divided up into groups according to their hardware component, averaged & then graphed. E.g. all valid Intel 4770K CPU results are averaged together to produce the CPUMark value for the Intel 4770K. You can see distribution graphs in PT, but there are also some examples here.
• PerformanceTest contains many individual sub-tests and tests all the main components of a computer (CPU, video card, disk & RAM). It produces an overall score (the “PassMark rating” and scores for each component, e.g. the “CPUMark”).
• Some invalid submissions are filtered out of the main charts. E.g. overclocked CPUs, duplicate submissions from the same system, etc. This filtered automatically and manually, as time allows.
• Benchmark results from popular components are more accurate. For example the Ryzen 7 1700X has thousands of submissions from different systems. So the average benchmark value has a very low margin for error. However for rare, expensive or very new CPUs the margin for error can be large. An example would be the Xeon Platinum 8180M, which has just 2 submissions.
• Average results are updated on a daily basis, typically around 2am US time.
• At the moment we don’t keep backups past daily results. The thinking behind this is that each new day brings new submissions which improves the averages, making them more accurate. If you wish to refer to historical results, be sure to save a copy of our web page (e.g. create a PDF).

Tender Best Practice

• Cores

  Can your application really use more than 8 cores without encountering a bottleneck elsewhere in the system? As an example, we completed this study and found 12 cores was the optimal number for one of our own applications.

• CPU

  If you are running a single application that can only use 1 CPU core (single threaded app) then the CPUMark value isn’t useful and you need to look at the single threaded CPU results.

• RAM

  Generally speaking, the requirement for fast RAM is often over estimated. Large CPU caches in modern CPUs mean RAM speeds only have a relatively small impact on overall performance (there are a few exceptions where some application can really make use of a lot of fast RAM).

• Performance

  Generally speaking, the performance improvement from a fast hard drive is underestimated. People focus too much on the CPUs and not enough on disk and internet performance.

• Multiple CPU

  A single fast CPU is often better than dual CPUs. This avoid NUMA memory effects. Same for video cards. One fast card is much better than two slower ones nearly all of the time.

• Laptops

  If buying laptops consider what the performance is like on both mains power and on battery power. Also does it throttle when hot?

• Always test with the same version of the software

  Don’t compare results from different versions of the software unless there is no other choice. PT does scale results of earlier versions to make an attempt to make them comparable, but it is better to compare results from the same PT software release wherever possible.

Tendering Against Benchmarks

• CPU Mark

  If there is a tender requirement such as, “The machine must have a CPU with CPUMark greater than 8000”. Then don’t tender a machine with a CPUMark of just 8005. The average result will go up and down slightly during the period of the tender. On the closing date of the tender the average result might be 7990 and you’ll be excluded. If the part you are tendering is a popular part, allow yourself at least 2% margin of error. If the part you are tendering is rare or brand new allow yourself at least 5% margin of error (CPUMark of 8400 in the above example).

• Different CPU Marks

  Not all machines with the same CPU will have the same CPUMark. As pointed out above there is a distribution of results. This is due to poor memory setups, bloatware, thermal throttling, overclocking, XMP RAM, and many other reasons. If you are running a tender test the actual machine you are buying at least gets the average result in our benchmark charts. Don’t just take the vendors word for it.

Tender FAQs

• Why do we run ads?

  While we made some money from the ads, the charts are basically a free service to the world. We make no promises about anything being complete, correct or accurate. It is best effort with the time and resources we have available. But if you see something that looks funky, let us know. There are millions of records and we can’t examine all of them by hand, community support is important.

• What if your system isn't in the database?

  If the system you are tendering doesn’t appear in our database at all, feel free to run PerformanceTest on your own systems and submit the results to our database. In some cases however we don’t publish results where we have just a single sample (margin for error is too high). Min sample requirements vary per chart.

• Can you test the system for me?

  Related to the above, we can benchmark machines for you running software than matches your use case (as a paid service) and write up a report.