With work unit;
e00009a00848-ABOU_rnd_ppod_expand_demos23-0-1-RND9304_0
It happened:
Error while computing 262,081.10

What does this mean and what does it suggest is causing this error?

Result 33000731 (easier that way)

It looks like it's failed and attempted a restart multiple times. The number (262,081.10) is the number of seconds it's wasted doing all that - not a happy bunny.

It's not immediately obvious what ultimately went wrong, but I'll keep looking.

Your GPU (NVIDIA GeForce MX250 (4042MB) driver: 510.73) is unusual, and may be a little short on memory (6 GB is recommended), and I'm not an expert on the Debian drivers.

That's a mobile gpu in a laptop. Not usually recommended to even try running gpu tasks on a laptop.

If you find out what is causing the problem, please let me know about it also.

It looks like overheating and GPU driver kicked out of operation:

It looks like overheating and GPU driver kicked out of operation:

I run laptop GPU's. Take a small air blower to its slots and blow the dust-out. I do it daily.

Result 33000731 (easier that way)

It looks like it's failed and attempted a restart multiple times. The number (262,081.10) is the number of seconds it's wasted doing all that - not a happy bunny.

It's not immediately obvious what ultimately went wrong, but I'll keep looking.

Your GPU (NVIDIA GeForce MX250 (4042MB) driver: 510.73) is unusual, and may be a little short on memory (6 GB is recommended), and I'm not an expert on the Debian drivers.

Result 33000731 (easier that way)

It looks like it's failed and attempted a restart multiple times. The number (262,081.10) is the number of seconds it's wasted doing all that - not a happy bunny.

It's not immediately obvious what ultimately went wrong, but I'll keep looking.

Your GPU (NVIDIA GeForce MX250 (4042MB) driver: 510.73) is unusual, and may be a little short on memory (6 GB is recommended), and I'm not an expert on the Debian drivers.

It is clear that these GPU tasks (MX-series) fail with insufficient GPU memory.
I am unable to find such "requirements" - I can't even find details of applications. Can anyone help ?

I have 2 laptops, the older one running Nvidia GT 1060, and the newer, running RTX 3060. The older laptop, though reaching GPU temperature of ~90C, completes most of Python tasks, though it may take up to 40 hours. The newer laptop, however, fails vast majority of Python tasks though that started quite recently. For example,

29/08/2022 9:32:39 PM | GPUGRID | Computation for task e00008a01599-ABOU_rnd_ppod_expand_demos24_3-0-1-RND7566_0 finished
29/08/2022 9:32:39 PM | GPUGRID | [task] result state=COMPUTE_ERROR for e00008a01599-ABOU_rnd_ppod_expand_demos24_3-0-1-RND7566_0 from CS::app_finished

29/08/2022 9:44:40 PM | GPUGRID | [task] Process for e00014a00897-ABOU_rnd_ppod_expand_demos24_3-0-1-RND0029_3 exited, exit code 195, task state 1
29/08/2022 9:44:40 PM | GPUGRID | [task] task_state=EXITED for e00014a00897-ABOU_rnd_ppod_expand_demos24_3-0-1-RND0029_3 from handle_exited_app
29/08/2022 9:44:40 PM | GPUGRID | [task] result state=COMPUTE_ERROR for e00014a00897-ABOU_rnd_ppod_expand_demos24_3-0-1-RND0029_3 from CS::report_result_error
29/08/2022 9:44:40 PM | GPUGRID | [task] Process for e00014a00897-ABOU_rnd_ppod_expand_demos24_3-0-1-RND0029_3 exited
29/08/2022 9:44:40 PM | GPUGRID | [task] exit code 195 (0xc3): (unknown error)
29/08/2022 9:44:43 PM | GPUGRID | Computation for task e00014a00897-ABOU_rnd_ppod_expand_demos24_3-0-1-RND0029_3 finished
29/08/2022 9:44:43 PM | GPUGRID | [task] result state=COMPUTE_ERROR for e00014a00897-ABOU_rnd_ppod_expand_demos24_3-0-1-RND0029_3 from CS::app_finished

I have 2 laptops, the older one running Nvidia GT 1060, and the newer, running RTX 3060. The older laptop, though reaching GPU temperature of ~90C, completes most of Python tasks, though it may take up to 40 hours. The newer laptop, however, fails vast majority of Python tasks though that started quite recently. For example,

29/08/2022 9:32:39 PM | GPUGRID | Computation for task e00008a01599-ABOU_rnd_ppod_expand_demos24_3-0-1-RND7566_0 finished
29/08/2022 9:32:39 PM | GPUGRID | [task] result state=COMPUTE_ERROR for e00008a01599-ABOU_rnd_ppod_expand_demos24_3-0-1-RND7566_0 from CS::app_finished

or another task, with more logging details:

29/08/2022 9:44:40 PM | GPUGRID | [task] Process for e00014a00897-ABOU_rnd_ppod_expand_demos24_3-0-1-RND0029_3 exited, exit code 195, task state 1
29/08/2022 9:44:40 PM | GPUGRID | [task] task_state=EXITED for e00014a00897-ABOU_rnd_ppod_expand_demos24_3-0-1-RND0029_3 from handle_exited_app
29/08/2022 9:44:40 PM | GPUGRID | [task] result state=COMPUTE_ERROR for e00014a00897-ABOU_rnd_ppod_expand_demos24_3-0-1-RND0029_3 from CS::report_result_error
29/08/2022 9:44:40 PM | GPUGRID | [task] Process for e00014a00897-ABOU_rnd_ppod_expand_demos24_3-0-1-RND0029_3 exited
29/08/2022 9:44:40 PM | GPUGRID | [task] exit code 195 (0xc3): (unknown error)
29/08/2022 9:44:43 PM | GPUGRID | Computation for task e00014a00897-ABOU_rnd_ppod_expand_demos24_3-0-1-RND0029_3 finished
29/08/2022 9:44:43 PM | GPUGRID | [task] result state=COMPUTE_ERROR for e00014a00897-ABOU_rnd_ppod_expand_demos24_3-0-1-RND0029_3 from CS::app_finished

What could be the reason? GPU has enough graphical memory, the laptop has 16 GB - same as the older one. There is enough disk space, and the temperature doesn't rise above 55-60C.

Acemd3 never fail on the newer laptop (at least, i don't remember such failures) and usually finish in under 15 hours. It's only Python, and only recently. Why could that be? What should I enable in the logs to diagnose better?

Thank you so much! Well, i was suspecting the memory, but both laptops have Windows-managed pagefile which is set to 40+GB. Is that enough, or should I increase it even more on the newer system? Besides, if it's Windows-managed, doesn't the pagefile size increase automatically on demand, though it may cause issues to the processes that request memory allocation? Anyway, I set the pagefile to 64GB now, as someone did in a similar situation. I expect it to be enough for the WUs to complete without issues. Thanks again.

The problem with Windows managed pagefile is that it probably doesn't increase its size immediately when the Python application starts loading all its spawned processes.

It probably responds initially to the initiating Boinc wrapper app, but that then loads the Python libraries which have huge memory footprints on Windows.

So the pagefile might not be large enough at the time that python dependent libraries are requesting lots of memory allocation space.

I have been recommending that Windows user just set a custom static sized pagefile of 32GBmin - 64GBmax size and that seems to cover the Python application and tasks and tasks complete successfully.

But with your 64GB current size, you probably have resolved the issue.

I can't speak for Windows behavior, but the last several Linux tasks I processed use a little more than 3GB per task when running 1 task per GPU.

with the help of a cuda mps server, I can push 2 tasks concurrently in a 6GB GTX 1060 card as some of the memory gets shared.

I would say 3GB minimum needed per task. and at least 4GB to be comfortable.

system memory is also quite high. uses about 8GB system memory per task.

but you should monitor it, the project could change the requirements at any time if they want to run larger jobs or change the direction of their research.

The problem with Windows managed pagefile is that it probably doesn't increase its size immediately when the Python application starts loading all its spawned processes.

It probably responds initially to the initiating Boinc wrapper app, but that then loads the Python libraries which have huge memory footprints on Windows.

So the pagefile might not be large enough at the time that python dependent libraries are requesting lots of memory allocation space.

I have been recommending that Windows user just set a custom static sized pagefile of 32GBmin - 64GBmax size and that seems to cover the Python application and tasks and tasks complete successfully.

But with your 64GB current size, you probably have resolved the issue.

Can anyone insight to the reason of these WU crashes?

https://www.gpugrid.net/result.php?resultid=33020923
https://www.gpugrid.net/result.php?resultid=33021416
https://www.gpugrid.net/result.php?resultid=33020419

I couldn't find any understandable clues for myself : /

Can anyone insight to the reason of these WU crashes?

https://www.gpugrid.net/result.php?resultid=33020923
https://www.gpugrid.net/result.php?resultid=33021416
https://www.gpugrid.net/result.php?resultid=33020419

I couldn't find any understandable clues for myself : /

ValueError: Object arrays cannot be loaded when allow_pickle=False

BrokenPipeError: [WinError 232] The pipe is being closed

In at least two of these cases, other people who ran the same WUs also had errors. Likely to just be a problem with the task itself and not your system.

There are more and more problem WUs -_-

https://www.gpugrid.net/result.php?resultid=33022242

I have been seeing a number of the same failures for quite some time as well.

https://www.gpugrid.net/result.php?resultid=33024302
https://www.gpugrid.net/result.php?resultid=33024910
https://www.gpugrid.net/result.php?resultid=33025156

It works out to about a 26% error rate.

There really isn't a good indication of why these are failing but I would expect it's a python error or inherent in the simulations that don't work.

The scientists will need to figure out these issues and let us know.


Jeff

My tasks are still failing with computational errors.
How long before the bas WU get out of the queue?
____________

The WU's that you have processed are failing with a CUDA error. Your GPU driver version 512.15 is somewhat outdated. I would suggest updating to the latest driver version 526.86.

Download the driver directly from NVIDIA https://www.nvidia.com/en-us/drivers/results/194176/

I would suggest you fully deinstall the driver using the DDU utility. https://www.guru3d.com/files-get/display-driver-uninstaller-download,9.html