Advanced search

Message boards : Number crunching : crunching 2 Python tasks concurrently

Author Message
Erich56
Send message
Joined: 1 Jan 15
Posts: 1132
Credit: 10,277,932,676
RAC: 29,168,538
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwat
Message 59199 - Posted: 7 Sep 2022 | 11:40:13 UTC

I remember when way back there were these "short runs" ACEMD tasks, some people, including myself, made some settings in a config_app.xml with the effect that 2 such tasks were processed simultaneously with the same GPU.

Since the current Python tasks are not really that GPU-intensive, it might be interesting to also have 2 of them run simultaneously.
Is anyone here doing this already? Successfully? If yes, what would the entry in the app_config.xml look like?

Or are there reasons why this would not be a good idea?

Ian&Steve C.
Avatar
Send message
Joined: 21 Feb 20
Posts: 1069
Credit: 40,231,533,983
RAC: 432
Level
Trp
Scientific publications
wat
Message 59200 - Posted: 7 Sep 2022 | 12:14:46 UTC - in response to Message 59199.

I ran 3x concurrently on my RTX 3060. was doing each task in about 13.5hrs (effectively 4.5hrs per task) for ones that run full-length. but now that I've added a second GPU, and 3 more tasks, the CPU can't keep up anymore. will update to a 16/32 part soon to see if it helps.

these tasks take about 3.6GB VRAM per task. so to run 2x you'll need 8GB VRAM.

also, each task will use about 7GB of system ram. so if you're running two, you need at minimum 16GB of system RAM.

doing multiples can help increase GPU utilization a bit, but you need to worry about overwhelming the CPU with so many concurrent processes (32x threads per task). and make sure you have enough GPU and system RAM to handle it. looks like your system has enough system ram at 32GB, but your 980Ti with 6GB VRAM might not be enough.

I did try running 2x on a GTX 1060 6GB, and the tasks just barely fit, but that was on linux and I'm not sure if it will work as well on Windows. you could try, just be sure to compare runtimes of the max length tasks to see if switching to 2x is worth it. if it's taking more than 2x as long to process tasks at 2x, then it's not worth it and slower than 1x.
____________

Erich56
Send message
Joined: 1 Jan 15
Posts: 1132
Credit: 10,277,932,676
RAC: 29,168,538
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwat
Message 59201 - Posted: 7 Sep 2022 | 14:03:16 UTC

thanks, Ian&Steve C., for the thorough explanation.

Well, concerning my system with the GTX980ti (6144MB VRAM) inside, I could try. Currently, GPU-Z shows a VRAM usage of some 2.888MB, and this value seems pretty much stable.
What I am more concerned about is the CPU usage. Currently, the Windows task manager shows a total of about 67% CPU usage for one Python task.
So, crunching 2 Pythons would most probably slow down the crunching process quite a bit (the CPU is 6-core/6-HT). I guess that total crunching time for 2 Pythons concurrently would be clearly more than double the time of 1 Python. But, again, one needs to try and see.

The situation would be much better on my other machine with 2 RTX3070 inside (and a 10-core / 10-HT CPU). However, there the problem is that I am running a 32GB Ramdisk (with additional 32GB system RAM). I wanted to try a Python on that system, but I could not even download any. BOINC told me that I need some 7GB more disk space (probably it took into account the 2 GPUs).
So I figured that the only way to circumvent this situation would be to install a second BOINC on the SSD - so far I didn't take my time to do this, according to the instructions I have seen this is not quite so simple; at least nothing which one could to within 5 or 10 minutes.

A new situation will come up when probably next week I will receive a second hand HP workstation with 2 Xeon processors inside (8 cores + 8-HT ea.) and a Quadro P5000 (16GB). However, I am aware that this GPU is not exactly the ideal or fast card for crunching; but I hope that it will still work satisfactorily.

But, to come to an end: I still don't know how I could get a GPU to process more than 1 task. Again I guess it needs specific entries in an app_config.xml. So please let me know :-)

gemini8
Avatar
Send message
Joined: 3 Jul 16
Posts: 31
Credit: 2,212,787,676
RAC: 3,785,098
Level
Phe
Scientific publications
watwat
Message 59202 - Posted: 8 Sep 2022 | 12:34:37 UTC

You need something like this:

<app_config>
<app>
<name>PythonGPU</name>
<max_concurrent>1</max_concurrent>
<gpu_versions>
<gpu_usage>0.5</gpu_usage>
<cpu_usage>2.0</cpu_usage>
</gpu_versions>
</app>
</app_config>

Put it into a file you call app_config.xml, put that into the GPUGrid project folder and re-read config files from the Boinc client, or just quit Boinc and start it again.
As I'm not at home atm I'm not able to have a look at the proper name for the Python GPU tasks. It's something along PythonGPU, but I'm not certain it's this exact spelling.
Boinc will tell you if it's not right, so you can change it afterwards.
I use the configuration with two CPU-threads per work-unit on one of my Ryzens which does other things on the rest of the CPU. Not sure if it would work as well with 0.45 or something like that, so the PythonGPU task would just take whatever CPU-cycles it can get from the sixteen threads the machine has altogether.

I run a MilkyWay task beside two GPUGrid Python tasks as the latter won't really satisfy the 1080.
____________
- - - - - - - - - -
Greetings, Jens

jenesuispasbavard
Avatar
Send message
Joined: 13 Oct 10
Posts: 1
Credit: 5,941,457
RAC: 0
Level
Ser
Scientific publications
watwatwatwatwatwatwat
Message 59296 - Posted: 22 Sep 2022 | 16:03:54 UTC - in response to Message 59202.
Last modified: 22 Sep 2022 | 16:06:31 UTC

You need something like this:
<app_config>
<app>
<name>PythonGPU</name>
<max_concurrent>1</max_concurrent>
<gpu_versions>
<gpu_usage>0.5</gpu_usage>
<cpu_usage>2.0</cpu_usage>
</gpu_versions>
</app>
</app_config>

Put it into a file you call app_config.xml, put that into the GPUGrid project folder and re-read config files from the Boinc client, or just quit Boinc and start it again.
As I'm not at home atm I'm not able to have a look at the proper name for the Python GPU tasks. It's something along PythonGPU, but I'm not certain it's this exact spelling.
Boinc will tell you if it's not right, so you can change it afterwards.
I use the configuration with two CPU-threads per work-unit on one of my Ryzens which does other things on the rest of the CPU. Not sure if it would work as well with 0.45 or something like that, so the PythonGPU task would just take whatever CPU-cycles it can get from the sixteen threads the machine has altogether.

I run a MilkyWay task beside two GPUGrid Python tasks as the latter won't really satisfy the 1080.


How do I run an Einstein@Home / Milkyway GPU task alongside one GPUGrid Python task? GPUGrid's app_config seems to decide only how many simultaneous GPUGrid tasks run.

With GPUGrid's Python tasks, I only get intermittent GPU usage, with constant 40ish% CPU usage - with a Core i7-12700K and RTX 3080 FE. GPU-Z screenshot attached.



I don't think I have enough VRAM to comfortably run 2 GPUGrid Python tasks - with a single task I get up to 4947MB VRAM usage, and the GPU only has 10240MB total. But Milkyway / Einstein tasks only use about 1GB, so I could run one GPUGrid task alongside one Einstein task; just don't know how to...
____________

Keith Myers
Send message
Joined: 13 Dec 17
Posts: 1341
Credit: 7,687,721,308
RAC: 13,341,125
Level
Tyr
Scientific publications
watwatwatwatwat
Message 59298 - Posted: 22 Sep 2022 | 21:15:47 UTC

Set all your projects to 0.5 gpu_usage and then restrict GPUGrid to only one max_concurrent PythonGPU task.

That will only allow one Python task to occupy the gpu along with one of your Einstein or Milkyway tasks at the same time.

gemini8
Avatar
Send message
Joined: 3 Jul 16
Posts: 31
Credit: 2,212,787,676
RAC: 3,785,098
Level
Phe
Scientific publications
watwat
Message 59299 - Posted: 23 Sep 2022 | 5:17:12 UTC

You'll need an app_config.xml in your respective Milkyway or Einstein directories.

<app_config>
<app>
<name>Milkyway</name>
<max_concurrent>1</max_concurrent>
<gpu_versions>
<gpu_usage>0.5</gpu_usage>
<cpu_usage>.04</cpu_usage>
</gpu_versions>
</app>
</app_config>

For Einstein there's a lot of application names.
Use this app_config.xml with 'Milkyway' in the Einstein directory as well, and Einstein will tell you it's wrong and its actual appliction names.
Einstein uses 1.0 for cpu_usage if on Nvidia or AMD GPUs.
____________
- - - - - - - - - -
Greetings, Jens

Asghan
Send message
Joined: 30 Oct 19
Posts: 6
Credit: 405,900
RAC: 0
Level

Scientific publications
wat
Message 59588 - Posted: 22 Nov 2022 | 9:28:36 UTC

The biggest lie for the new python tasks is "0.9 CPU".
My current task, and the one before, were/is using 20 out of my 24 cores on my 5900X...

Vester
Send message
Joined: 22 Nov 16
Posts: 3
Credit: 455,000
RAC: 0
Level

Scientific publications
wat
Message 59595 - Posted: 25 Nov 2022 | 4:03:57 UTC

I errored out on three work units (27347453, 27347587, and 27347506) while trying to run more than one GPU task on my single GPU. I don't know if they failed because of my app_config.xml file or not, but I did not error out on a single GPU task, 27346636. I was also running Rosetta@home concurrently but have suspended R@h while running GPUGrid.

This is my computer.

Keith Myers
Send message
Joined: 13 Dec 17
Posts: 1341
Credit: 7,687,721,308
RAC: 13,341,125
Level
Tyr
Scientific publications
watwatwatwatwat
Message 59596 - Posted: 25 Nov 2022 | 6:08:38 UTC - in response to Message 59595.

The tasks failed because you haven't properly set a large enough paging file for your host in Windows.

This is your error:

[WinError 1455] The paging file is too small for this operation to complete.

There is nothing wrong with your gpu which is more than adequate to run 2X tasks concurrent.

But each task needs about 35GB of virtual memory in the paging file.

See the main thread about the Python tasks.

https://www.gpugrid.net/forum_thread.php?id=5233

If you increase your paging file size, you should have no issues successfully completing the work units.

Vester
Send message
Joined: 22 Nov 16
Posts: 3
Credit: 455,000
RAC: 0
Level

Scientific publications
wat
Message 59597 - Posted: 25 Nov 2022 | 21:47:57 UTC

If you increase your paging file size, you should have no issues successfully completing the work units.

Thank you, Keith, for your patience and help. I should have looked at the stderr output. My NMVe C: drive is only 250GB, so I put a fixed 228000MB paging file in another SSD. (Now I need more RAM to run three or more GPU tasks simultaneously.)

Erich56
Send message
Joined: 1 Jan 15
Posts: 1132
Credit: 10,277,932,676
RAC: 29,168,538
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwat
Message 59600 - Posted: 26 Nov 2022 | 8:01:20 UTC - in response to Message 59597.

Now I need more RAM to run three or more GPU tasks simultaneously.

how powerful is your CPU ?

Vester
Send message
Joined: 22 Nov 16
Posts: 3
Credit: 455,000
RAC: 0
Level

Scientific publications
wat
Message 59601 - Posted: 26 Nov 2022 | 23:09:12 UTC - in response to Message 59600.

how powerful is your CPU ?

Intel i9-10850K @4.9GHz, ten cores and hyperthreaded. It could be a limitation because it is utilized at 100% when running two tasks on the Nvidia Titan X (single GPU).

Erich56
Send message
Joined: 1 Jan 15
Posts: 1132
Credit: 10,277,932,676
RAC: 29,168,538
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwat
Message 59602 - Posted: 28 Nov 2022 | 9:22:32 UTC - in response to Message 59601.
Last modified: 28 Nov 2022 | 9:23:47 UTC

how powerful is your CPU ?

Intel i9-10850K @4.9GHz, ten cores and hyperthreaded. It could be a limitation because it is utilized at 100% when running two tasks on the Nvidia Titan X (single GPU).

well, the CPU is definitely powerful enough to run 2 Pythons concurrently.
And basically more than 2 Pythons cannot be downloaded anyway, unless one "tunes" the "coproc-Info.xml" in the BOINC folder.

Post to thread

Message boards : Number crunching : crunching 2 Python tasks concurrently

//