CPU-Maths - no operations reported in timeout period

[derek@CDI]

We are running BIT Pro V5.3 build 1036 on AMD Geode LX800 embedded platform under XP Pro (PCM-3353 SBC from Advantech) and consistently getting strange error in both math and multimedia CPU tests - "no operations reported in timeout period". I've searched the forum for it and it seems like the test for some reason stops running which is consistent with what we see here. At some point the test window just stops updating and I guess after 8min the error shows up. I noticed that sometimes the test window freezes even before the final occurence but it recovers after some time. The link below will take you to the location of the trace file from the last test run when we enabled only CPU-Maths at 20%. We would really like to know what's causing this error.
Thanks,
Derek

https://filetransfer.gefanuc.com/fs/...8c5a61b27d6dac

[David (PassMark)]

The no operations messages mean that particaular test has crashed or locked up or running extremely extremely slow. The no operations message is generated by a watchdog timer.

As for why the crash occurs, it can be device driver, BIOS or hardware issues.

If the tests restarts by itself, and the operations counter starts increasing again (as is the case here) then is can not have been a crash, as there is no way to recover from a crash. Instead the system is freezing. This can happen for example if you are really low on RAM, and have a really slow hard drive, and the system has to spend long periods swapping data in and out of the swap file.

[derek@CDI]

Let me just clarify my previous post cause I don't think you understood me correctly.
I started out by setting up a standard test with all necessary peripherals included. Everything passed except both CPU tests - Maths and SIMD which registered single error that I mentioned before. In the process of troubleshooting I started turning off different tests and finally ended up just with CPU Math test at 20% load. Every time the error showed up within 30min. CPU usage reported by Performance Monitor bounced between 60 - 80% and memory usage was marginal. I also disabled Paging File to make sure it didn't affect the results as we use IDE Flash Drive in this system.
I was watching the test window and I noticed (as I mentioned before) that it freezes periodically but recovers after a while. It goes on like that until it finally freezes for good. Then after several minutes the error occurs. The unit however, is perfectly responsive and when I eventually stop the test the number of operations in the main window all of a sudden increases. I believe this tells me that the test was still running in the background but results were not being updated. Please take another look at it as we really need to know what's behind this error.
We have been using BurnInTest Pro 5 on various systems for few years now and never seen it before.
Thanks,
Derek

[Ian (PassMark)]

As with David's post, I would first suspect device driver, BIOS or hardware issues.

If you run the CPU test at 100% load, does the problem occur more quickly?

In V5 of BurnInTest, the CPU math test operation count is increased in the CPU math test thread, while the CPU MMX test operation count is increased in the CPU MMX display thread.

The next step would be to determine if the CPU test thread or the CPU test windows update thread has stopped, if you open task manager while running BurnInTest and watch the CPU load. If the CPU test is running, you should see load on all CPU cores - in which case the display thread has stopped. If the CPU test is not running, you should not see load on all CPU cores - in which case the cpu test thread has stopped.

If the CPU test thread has stopped, I would try V6.0 of BurnInTest as you have much more control over which CPU instructions are run from Preferences->CPU. I would then try each CPU instruction grouping seperately to see if the problem occurs on particular types of CPU instructions. V6.0 also has more Activity trace level 2 logging for the CPU test. You can downlaod a trial version of V6.0 here:
http://www.passmark.com/download/bit_download.htm

Regards,
Ian

[derek@CDI]

Time has passed but the problem is still here. Going along with your suggestion we purchased v6 of BIT Pro. Unfortunately the symptoms remained the same. I ran several tests selecting only each group of instructions separately and they all produce the same results. 15-20 min within the test CPU test windows stops updating, CPU usage however, remains unchanged - around 87%. Also, changing CPU load doesn't seem to affect the results either.
I'm including the trace from one of the tests below. Please try to be more specific in your explanation. Thanks.

PassMark BurnInTest Log file - http://www.passmark.com
================================================== ======

Date: 06/04/09 14:44:14

BurnInTest V6.0 Pro 1014
Trace detail level: Activity trace 2

**************
SYSTEM SUMMARY
**************
Windows XP Professional Service Pack 2 build 2600 (32-bit),
1 x Geode(TM) Integrated Processor by AMD PCS [499.9 MHz],
503MB RAM,
AMD Custom Driver For 640x640 Panel,
4GB HDD,


GENERAL
System Name: OEM-AF71FC943EB
System Model: AWRDACPI
Motherboard Name: AMD-GX3
BIOS Manufacturer: Phoenix Technologies, LTD
BIOS Version: 6.00 PG
BIOS Release Date: 08/28/2009

CPU
CPU manufacturer: AuthenticAMD
CPU Type: Geode(TM) Integrated Processor by AMD PCS
CPUID: Family 5, Model A, Stepping 2
Physical CPU's: 1
Cores per CPU: 1
Hyperthreading: Not capable
CPU features: MMX 3DNow!
Clock frequencies:
Measured CPU speed: 499.9 MHz
Cache per CPU package:
L1 Instruction Cache: 2 x 64 KB
L1 Data Cache: 2 x 64 KB
L2 Cache: 2 x 128 KB

MEMORY
Total Physical Memory: 503MB
Available Physical Memory: 380MB
Memory devices:
A0:
- 512MB,
A1:
- 8MB,
A2:
- 8MB,
A3:
- 8MB,

GRAPHICS
AMD Custom Driver For 640x640 Panel
Chip Type: GeodeLX
DAC Type: Internal DAC
Memory: 8MB
Driver provider: Advanced Micro Devices
Driver version: 3.0.2.0
Driver date: 6-19-2009
Monitor 1: 640x640x32 70Hz (Primary monitor)

DISK VOLUMES
C: Local drive, NTFS, (3.83GB total, 2.49GB free)

DISK DRIVES
Disk drive: Model Netlist Flash v2.0 (Size: 3.83GB)

OPTICAL DRIVES

NETWORK
Intel(R) 8255xER PCI Adapter

PORTS
Communications Port: COM2 - RS232 Serial Port (max Baud rate: 115200)
Keyboard Port: PS/2 connector
Mouse Port: PS/2 connector
Standard OpenHCD USB Host Controller
Standard Enhanced PCI to USB Host Controller
******************
DETAILED EVENT LOG
******************
LOG NOTE: 2009-06-04 14:44:14, Save Preferences before: C:\Documents and Settings\User\My Documents\PassMark\BurnInTest\LastUsed.bitcfg, 27824
LOG NOTE: 2009-06-04 14:44:14, General, C:\Documents and Settings\User\My Documents\PassMark\BurnInTest\LastUsed.bitcfg, 27824, 27824
LOG NOTE: 2009-06-04 14:44:14, Save Preferences after: C:\Documents and Settings\User\My Documents\PassMark\BurnInTest\LastUsed.bitcfg, 27824
LOG NOTE: 2009-06-04 14:44:14, Status, PassMark BurnInTest V6.0 Pro 1014
LOG NOTE: 2009-06-04 14:44:15, Status, Main Tests started
LOG NOTE: 2009-06-04 14:44:15, Perform test: CPU at 50%
LOG NOTE: 2009-06-04 14:44:15, CPU, Starting test
LOG NOTE: 2009-06-04 14:44:16, CPU, CPU General test: CPU 0 Cycle 0 Ops 0
LOG NOTE: 2009-06-04 14:52:14, Operation watchdog for 1 is 8419133448 (previous 0)
LOG NOTE: 2009-06-04 15:00:14, Operation watchdog for 1 is 8419133448 (previous 841913344
WARNING: 2009-06-04 15:00:14, CPU, No operations reported in timeout period
LOG NOTE: 2009-06-04 15:00:14, Operation watchdog triggered for 1 is 8419133448 (previous 841913344
LOG NOTE: 2009-06-04 15:08:14, Operation watchdog for 1 is 8419133448 (previous 841913344
LOG NOTE: 2009-06-04 15:14:15, StopTests [2]
LOG NOTE: 2009-06-04 15:14:15, Test run stopping - step 0.0. 0: 0 of 1
LOG NOTE: 2009-06-04 15:14:15, CPU, Stopping test
LOG NOTE: 2009-06-04 15:14:18, Status, Test run stopped

**************
RESULT SUMMARY
**************
Test Start time: Thu Jun 04 14:44:14 2009
Test Stop time: Thu Jun 04 15:14:18 2009
Test Duration: 000h 30m 04s

Test Name Cycles Operations Result Errors Last Error
CPU 52 95.788 Billion FAIL 1 No operations reported in timeout period
TEST RUN FAILED

*******************************************
SERIOUS ERROR SUMMARY FOR THE LAST TEST RUN
*******************************************

-----------------------------------------------------------------------------------------------------
******************
DETAILED EVENT LOG
******************
LOG NOTE: 2009-06-04 15:33:36, StopTests [6]
LOG NOTE: 2009-06-04 15:33:36, Test run stopping - step 0.0. 0: 0 of 0

[David (PassMark)]

We have never done any testing on the Geode CPU and the associated hardware. In principle it should be OK, but clearly something is going wrong. From what you have described the display thread is crashing. So the display stops updating and the watchdog timer triggers some time later. For here we can't tell you the exact cause of this crash. I assume you don't get any pop windows with crash details?

This is not a known issue & no other customer has reported behaviour like this. Do you have a 2nd example of this device on which you can run the test and compare the behaviour. Given that no other customer has reported the issue it might well be hardware related. e.g. a bad CPU, bad RAM or bad Bus.

If these were cheap devices you could ship us one, and we then we could do a much deeper investigation of the problem (and ship it back at the end if required).

[derek@CDI]

I realize that Geode is not a mainstream for the consumer market but in the embedded world it is quite popular. Although as you said the problem may be specific to this particular board and not the CPU/chipset in general. We've built close to 40 units so far based on this hardware in couple different flavors and all of them during testing showed that error.
We appreciate you assistance as resolving this problem is important to us. We are willing to ship the hardware to you if you think it will help. Is there a procedure we need to follow? Is your shipping address the same as listed on the website? Thanks.

[David (PassMark)]

I'll send you an E-Mail to arrange the details.

[WillTA3@CDI]

Hello, My name is Will Anniss and I am the Technology Leader where Derek worked. He was working with you on an issue that we were seeing with an AMD board and the Passmark test. Materials were shipped to you to conduct said tests on the SBC.

Who should I contact to discuss this further?