Failed again

Message boards : Number crunching : Failed again

Author	Message
[B^S] HenryHunter Send message Joined: 27 Dec 08 Posts: 4 Credit: 6,055,773 RAC: 0 Level Scientific publications	Message 13555 - Posted: 15 Nov 2009 \| 0:56:14 UTC
	<core_client_version>6.6.38</core_client_version> <![CDATA[ <message> Unzul�ssige Funktion. (0x1) - exit code 1 (0x1) </message> <stderr_txt> # Using CUDA device 0 # There is 1 device supporting CUDA # Device 0: "GeForce 9800 GT" # Clock rate: 1.65 GHz # Total amount of global memory: 1073741824 bytes # Number of multiprocessors: 14 # Number of cores: 112 MDIO ERROR: cannot open file "restart.coor" # Using CUDA device 0 # There is 1 device supporting CUDA # Device 0: "GeForce 9800 GT" # Clock rate: 1.65 GHz # Total amount of global memory: 1073741824 bytes # Number of multiprocessors: 14 # Number of cores: 112 Cuda error: Kernel [pme_fill_charges_accumulate] failed in file 'fillcharges.cu' in line 73 : unknown error. </stderr_txt> ?
	ID: 13555 \| Rating: 0 \| rate: / Reply Quote

[B^S] HenryHunter Send message Joined: 27 Dec 08 Posts: 4 Credit: 6,055,773 RAC: 0 Level Scientific publications	Message 13558 - Posted: 15 Nov 2009 \| 9:09:01 UTC - in response to Message 13555.
	Another failure: <core_client_version>6.6.38</core_client_version> <![CDATA[ <message> Unzul�ssige Funktion. (0x1) - exit code 1 (0x1) </message> <stderr_txt> # Using CUDA device 0 # There is 1 device supporting CUDA # Device 0: "GeForce 9800 GT" # Clock rate: 1.65 GHz # Total amount of global memory: 1073741824 bytes # Number of multiprocessors: 14 # Number of cores: 112 MDIO ERROR: cannot open file "restart.coor" # Using CUDA device 0 # There is 1 device supporting CUDA # Device 0: "GeForce 9800 GT" # Clock rate: 1.65 GHz # Total amount of global memory: 1073741824 bytes # Number of multiprocessors: 14 # Number of cores: 112 Cuda error: Kernel [pme_fill_charges_accumulate] failed in file 'fillcharges.cu' in line 73 : unknown error. </stderr_txt> ]]>
	ID: 13558 \| Rating: 0 \| rate: / Reply Quote

Damaraland Send message Joined: 7 Nov 09 Posts: 152 Credit: 16,181,924 RAC: 0 Level Scientific publications	Message 13920 - Posted: 14 Dec 2009 \| 9:46:18 UTC - in response to Message 13558.
	I keep having this error on every unit. What should I do? <core_client_version>6.10.18</core_client_version> <![CDATA[ <message> Funci�n incorrecta. (0x1) - exit code 1 (0x1) </message> <stderr_txt> # Using CUDA device 0 # There are 2 devices supporting CUDA # Device 0: "GeForce GTX 260" # Clock rate: 1.41 GHz # Total amount of global memory: 939196416 bytes # Number of multiprocessors: 27 # Number of cores: 216 # Device 1: "GeForce 9800 GT" # Clock rate: 1.37 GHz # Total amount of global memory: 1073545216 bytes # Number of multiprocessors: 14 # Number of cores: 112 MDIO ERROR: cannot open file "restart.coor" </stderr_txt> ]]> <core_client_version>6.10.18</core_client_version> <![CDATA[ <message> Funci�n incorrecta. (0x1) - exit code 1 (0x1) </message> <stderr_txt> # Using CUDA device 0 # There are 2 devices supporting CUDA # Device 0: "GeForce GTX 260" # Clock rate: 1.41 GHz # Total amount of global memory: 939196416 bytes # Number of multiprocessors: 27 # Number of cores: 216 # Device 1: "GeForce 9800 GT" # Clock rate: 1.37 GHz # Total amount of global memory: 1073545216 bytes # Number of multiprocessors: 14 # Number of cores: 112 MDIO ERROR: cannot open file "restart.coor" Cuda error: Kernel [shake_step_1] failed in file 'shake.cu' in line 79 : unspecified launch failure. </stderr_txt> ]]>
	ID: 13920 \| Rating: 0 \| rate: / Reply Quote

Damaraland Send message Joined: 7 Nov 09 Posts: 152 Credit: 16,181,924 RAC: 0 Level Scientific publications	Message 13923 - Posted: 14 Dec 2009 \| 14:48:17 UTC - in response to Message 13920.
	More and more... task 1629422 task 1629408 task 1629383 task 1629337 task 1629322 Tonight I suspend proyect until further news. If admins what me to report any further information, I'll gladly help.
	ID: 13923 \| Rating: 0 \| rate: / Reply Quote

skgiven Volunteer moderator Volunteer tester Send message Joined: 23 Apr 09 Posts: 3968 Credit: 1,995,359,260 RAC: 0 Level Scientific publications	Message 13929 - Posted: 14 Dec 2009 \| 20:41:59 UTC - in response to Message 13923.
	You should do this, Install and run GPU-Z. It is freeware and will allow you to see the temperatures of the GPUS. If they are over 70degrees when running GPUGrid tasks, you may have a heating / ventilation problem. If you do then you could test this by just leaving the door off for a while and try running more tasks and check the temperatures. If this is definitely a problem, either get a couple of extra system fans or manually turn the fan speed up on the card(s). I would highly recommend that you uninstall the 9800GT. These cards use a G92 core and do not handle most of today’s tasks too well, especially hERG tasks! This card is most likely causing ALL your failures. If you do not then the card will eventually give so many failures that you will get no new work. Your GTX260 will be doing at least 3 times the work of the 9800GT. What are the GPU temperatures like with and without the 9800GT installed, Running tasks and not running tasks? What is your PSU?
	ID: 13929 \| Rating: 0 \| rate: / Reply Quote

Richard Haselgrove Send message Joined: 11 Jul 09 Posts: 1620 Credit: 8,893,667,858 RAC: 19,888,672 Level Scientific publications	Message 13934 - Posted: 14 Dec 2009 \| 22:08:55 UTC - in response to Message 13923.
	More and more... task 1629422 task 1629408 task 1629383 task 1629337 task 1629322 Tonight I suspend proyect until further news. If admins what me to report any further information, I'll gladly help. When your card fails task after task in 10 seconds or less, it's likely that its internal state has become corrupted. Do a complete power cycle - power down the host and restart: that should allow most of those tasks (the GIANNI-BIND and the KASHIF-HIVPR, at least) to run properly - even on the 9800GT that SKGiven despises so much :-)
	ID: 13934 \| Rating: 0 \| rate: / Reply Quote

Michael Goetz Send message Joined: 2 Mar 09 Posts: 124 Credit: 60,073,744 RAC: 0 Level Scientific publications	Message 13937 - Posted: 14 Dec 2009 \| 23:56:03 UTC - in response to Message 13929.
	Install and run GPU-Z. It is freeware and will allow you to see the temperatures of the GPUS. If they are over 70degrees when running GPUGrid tasks, you may have a heating / ventilation problem. That's good advice for the CPU. The GPU is a different beast altogether. Newer GPU's appear to be designed to run hotter. A LOT hotter. The latest generation of Nvidia GPUs (260, 280, 285, 295) actually are designed to safely run to -- get this -- just over 100 degrees Celsius. In fact, when the fan is left on automatic, it won't even start to ramp up the fan speed until the temperature exceeds 70. Even then, it's only slightly increasing the fan speed and is clearly not trying to keep temps in the 70s Normal operating temperature under a heavy load seems to be around 80 to 85 degrees -- and that's with the fan still below 60% on automatic control. So don't panic if you see GPU temps in the 70s or 80s. That's normal. On the other hand, if your CPU is running that hot, you're possibly just on the brink of having it fail, depending on the model. ____________ Want to find one of the largest known primes? Try PrimeGrid. Or help cure disease at WCG.
	ID: 13937 \| Rating: 0 \| rate: / Reply Quote

skgiven Volunteer moderator Volunteer tester Send message Joined: 23 Apr 09 Posts: 3968 Credit: 1,995,359,260 RAC: 0 Level Scientific publications	Message 13956 - Posted: 15 Dec 2009 \| 20:26:43 UTC - in response to Message 13937. Last modified: 15 Dec 2009 \| 21:02:12 UTC
	Install and run GPU-Z. It is freeware and will allow you to see the temperatures of the GPUS. If they are over 70degrees when running GPUGrid tasks, you may have a heating / ventilation problem. That's good advice for the CPU. The GPU is a different beast altogether. So don't panic if you see GPU temps in the 70s or 80s. That's normal. OK, GPU-Z measures GPU temperatures. It is not CPU-Z! Although the cards can run hot, it is NOT Good for them to stay at that temperature for extended periods of time and certainly Not Normal! My GTX260-216sp sits at 66 Degrees C when crunching GPUGrid and the fan is at 40%. That is my Normal and the card works 100% for ALL tasks. I would not like to hear the fan at 60% or run the GPU at 80 Degrees C. Pull the 9800GT You are testing to find the problem, so trying things is important!
	ID: 13956 \| Rating: 0 \| rate: / Reply Quote

Damaraland Send message Joined: 7 Nov 09 Posts: 152 Credit: 16,181,924 RAC: 0 Level Scientific publications	Message 14057 - Posted: 25 Dec 2009 \| 15:39:20 UTC - in response to Message 13956. Last modified: 25 Dec 2009 \| 15:40:21 UTC
	Sure it wasn't the heat, because since I thought that it was in a very cold room with the mainboard without case. I had the GT in the same board with the GTX card. I moved the gtx to a brand new board and everything is fine since then... the wonderfull world of computers... Now having problems with Linux drivers with my GTX, but that's another story I'll handle with time and patience...
	ID: 14057 \| Rating: 0 \| rate: / Reply Quote

skgiven Volunteer moderator Volunteer tester Send message Joined: 23 Apr 09 Posts: 3968 Credit: 1,995,359,260 RAC: 0 Level Scientific publications	Message 14062 - Posted: 26 Dec 2009 \| 14:38:05 UTC - in response to Message 14057.
	A pain I’m sure. It ran for some time before it failed! Not sure a 9500 GT is up to the task any more. Well, certainly not that task. Was there a system restart, an update, or did you clear the cache, wipe free space...? Just on the off chance it is something other than Boinc/GPUGrid! I tried my 8800GTS again, but all tasks failed. 4 failed within 3sec (no real problem) but the fifth failed after 13h or so. It is running another task. If it fails I will take it back off GPUGrid, but I might get another GT240 instead. The one I have uses less electric and has successfully finished all 11 tasks so far.
	ID: 14062 \| Rating: 0 \| rate: / Reply Quote

Post to thread

Message boards : Number crunching : Failed again

	About	Science	Volunteers	Performance	Forum	Join us	Donate