Advanced search

Message boards : Number crunching : Not Getting GPU Tasks

Author Message
Profile SuperSluether
Send message
Joined: 2 Oct 14
Posts: 5
Credit: 14,878,475
RAC: 0
Level
Pro
Scientific publications
watwatwat
Message 49648 - Posted: 7 Jun 2018 | 19:49:34 UTC

The server status page shows that there are tasks for "Short runs" and "Long runs", but I am not getting either of these tasks on my PC. The event log entries are:

GPUGRID | Sending scheduler request: To fetch work.
GPUGRID | Requesting new tasks for NVIDIA GPU
GPUGRID | Scheduler request completed: got 0 new tasks
GPUGRID | No tasks sent
GPUGRID | Tasks for CPU are available, but your preferences are set to not accept them

I did see another topic stating that CUDA 8.0 is required, but the BOINC logs suggest that I am running 9.1:

CUDA: NVIDIA GPU 0: GeForce GTX 760 (driver version 390.48, CUDA version 9.1, compute capability 3.0, 1992MB, 1545MB available, 2620 GFLOPS peak)

My account preferences are set to accept all applications, with "Use CPU" set to "no".

I am running on Ubuntu 18.04 LTS with Nvidia driver 390.48. Do I need to install a separate CUDA package to get GPU tasks?

Jim1348
Send message
Joined: 28 Jul 12
Posts: 819
Credit: 1,591,285,971
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 49649 - Posted: 7 Jun 2018 | 20:08:38 UTC - in response to Message 49648.
Last modified: 7 Jun 2018 | 20:22:35 UTC

I am running on Ubuntu 18.04 LTS with Nvidia driver 390.48. Do I need to install a separate CUDA package to get GPU tasks?

I have not found it necessary to install a separate CUDA package on Ubuntu 18.04.

EDIT: Let me clarify that. I had to install the "Nvidia driver metapackage from nvidia-driver-390 (proprietary, tested)" from Ubuntu Software & Updates in order to get the OpenCl drivers. It did not used to be the case for Ubuntu 16.04, but that seems to have changed somewhat, and may be necessary for CUDA too.

These preferences work for me:

Resource share 100
Use CPU no
Use ATI GPU no
Use NVIDIA GPU yes
Run test applications? no
Maximum CPU % for graphics 20

Run only the selected applications
ACEMD short runs (2-3 hours on fastest card): no
ACEMD long runs (8-12 hours on fastest GPU): yes
ACEMD Beta: no
Quantum Chemistry (CPU): no
Quantum Chemistry (CPU, beta): no
Python Runtime : no
If no work for selected applications is available, accept work from other applications? yes
Use Graphics Processing Unit (GPU) if available yes
Use Central Processing Unit (CPU) yes


Also, in BOINC Preferences, do you have "Use GPU while computer is in use" enabled?

Profile Retvari Zoltan
Avatar
Send message
Joined: 20 Jan 09
Posts: 2346
Credit: 16,290,365,968
RAC: 6,912,525
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 49650 - Posted: 7 Jun 2018 | 20:25:22 UTC - in response to Message 49648.

The server status page shows that there are tasks for "Short runs" and "Long runs", but I am not getting either of these tasks on my PC.

CUDA: NVIDIA GPU 0: GeForce GTX 760 (driver version 390.48, CUDA version 9.1, compute capability 3.0, 1992MB, 1545MB available, 2620 GFLOPS peak)

I am running on Ubuntu 18.04 LTS with Nvidia driver 390.48.

According to this post, the present Linux app (v9.18) does not support cc3.0 cards (so called shader model 3.0, sm3.0). The older v8.49 app (CUDA6.5) supports cc3.0 cards, but there's only Windows version of this app. So you have to use Windows to make your cc3.0 card work with GPUGrid.

mmonnin
Send message
Joined: 2 Jul 16
Posts: 337
Credit: 7,666,958,683
RAC: 10,044,224
Level
Tyr
Scientific publications
watwatwatwatwat
Message 49651 - Posted: 7 Jun 2018 | 20:25:56 UTC

I had to add the nvidia PPA to get drivers to work with BOINC in 18.04. I don't recall if that was for OpenCL or CUDA. One was missing.

Profile SuperSluether
Send message
Joined: 2 Oct 14
Posts: 5
Credit: 14,878,475
RAC: 0
Level
Pro
Scientific publications
watwatwat
Message 49653 - Posted: 7 Jun 2018 | 20:34:28 UTC - in response to Message 49650.
Last modified: 7 Jun 2018 | 20:34:55 UTC

The older v8.49 app (CUDA6.5) supports cc3.0 cards, but there's only Windows version of this app. So you have to use Windows to make your cc3.0 card work with GPUGrid.


In that case, I will unfortunately be unable to contribute to this project (at least for GPU tasks). I do not have a Windows PC, and I do not plan to use Windows in the forseeable future.

liderbug
Send message
Joined: 29 Jul 16
Posts: 24
Credit: 80,719,054
RAC: 0
Level
Thr
Scientific publications
watwatwat
Message 49676 - Posted: 13 Jun 2018 | 14:05:51 UTC - in response to Message 49653.

I have a like problem: I'm getting CPU tasks but no GPU.

Fedora-25, 7.8.4 x86_64-pc-linux-gnu, GeForce GTX 770

Do I have a configuration problem or as above there is nothing I can do to use my GPU?

Thanks

PappaLitto
Send message
Joined: 21 Mar 16
Posts: 511
Credit: 4,672,242,755
RAC: 0
Level
Arg
Scientific publications
watwatwatwatwatwatwatwat
Message 49677 - Posted: 13 Jun 2018 | 17:49:26 UTC

Currently there are no more "unsent" GPU work units.

Zalster
Avatar
Send message
Joined: 26 Feb 14
Posts: 211
Credit: 4,496,324,562
RAC: 0
Level
Arg
Scientific publications
watwatwatwatwatwatwatwat
Message 49708 - Posted: 22 Jun 2018 | 5:37:00 UTC

Looks like the GPU well has run dry...
____________

liderbug
Send message
Joined: 29 Jul 16
Posts: 24
Credit: 80,719,054
RAC: 0
Level
Thr
Scientific publications
watwatwat
Message 49735 - Posted: 28 Jun 2018 | 12:40:47 UTC - in response to Message 49708.

so, this is really the CPUgrid.net ?????

Betting Slip
Send message
Joined: 5 Jan 09
Posts: 670
Credit: 2,498,095,550
RAC: 0
Level
Phe
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 49736 - Posted: 28 Jun 2018 | 13:53:00 UTC - in response to Message 49735.

so, this is really the CPUgrid.net ?????


Some new GPU WU's will be generated from returned results...keep trying.
____________
Radio Caroline, the world's most famous offshore pirate radio station.
Great music since April 1964. Support Radio Caroline Team -
Radio Caroline

tullio
Send message
Joined: 8 May 18
Posts: 190
Credit: 104,426,808
RAC: 0
Level
Cys
Scientific publications
wat
Message 49738 - Posted: 28 Jun 2018 | 17:05:05 UTC

I am still getting both CPU and GPU tasks on my main Linux box. A Linux laptop, with no GPU board, is getting CPU tasks.
Tullio
____________

WPrion
Send message
Joined: 30 Apr 13
Posts: 96
Credit: 2,770,334,111
RAC: 21,246,051
Level
Phe
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 49749 - Posted: 30 Jun 2018 | 10:48:52 UTC - in response to Message 49736.

so, this is really the CPUgrid.net ?????


They were going to update the name but couldn't because the recaptcha is still out of date. /snark

Win

Erich56
Send message
Joined: 1 Jan 15
Posts: 1132
Credit: 10,363,897,676
RAC: 29,196,793
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwat
Message 50382 - Posted: 3 Sep 2018 | 16:42:35 UTC - in response to Message 49735.

so, this is really the CPUgrid.net ?????

good question -
my impression within the last weeks has been the GPUGRID staff is now a lot more concentrating on the CPU tasks (Quantum Chemistry), resulting in more frequent periods of time where GPU tasks are NOT available.
Although QC is a minority program only, not being available for Windows users.

In a way this is really too bad for all the crunchers who for one reason or the other want to stick with Windows: CPU jobs cannot be processed, and GPU jobs are not available :-(

Of course, everyone who wants go keep his/her GPU busy can choose from a variety of backup projects;
however, it should be clear to GPUGRID that there may be cases where crunchers do not return to GPUGRID once they satisfactorily participate in another project - particularly if there is a much better and more reliable availability of work.

Erich56
Send message
Joined: 1 Jan 15
Posts: 1132
Credit: 10,363,897,676
RAC: 29,196,793
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwat
Message 50414 - Posted: 6 Sep 2018 | 8:01:37 UTC

somehow I would have expected some kind of statement from the GPUGRID people in reply to my posting above. Well, nothing so far :-(

A look at the current Server Status page shows interesting figures:

GPU tasks - unsent: 0; in progress: 1.761; users in last 24hrs: 924

CPU tasks - unsent: 44.723; in progress: 848; users in last 24hrs: 76

this alone shows the huge imbalance. 924 GPU users have no news tasks available, and 76 CPU users have 44.723 available.
What sense does this make?

There are about 12x more GPU users, not getting new taks. Whereas only 76 users are swamped with 44.723 tasks.

Where is the logic behind this?

[VENETO] boboviz
Send message
Joined: 10 Sep 10
Posts: 158
Credit: 388,132
RAC: 0
Level

Scientific publications
wat
Message 50456 - Posted: 10 Sep 2018 | 7:53:13 UTC - in response to Message 50382.

Of course, everyone who wants go keep his/her GPU busy can choose from a variety of backup projects;
however, it should be clear to GPUGRID that there may be cases where crunchers do not return to GPUGRID once they satisfactorily participate in another project - particularly if there is a much better and more reliable availability of work.


I hope all this work on cpu helps the advancement of science on gpugrid. :-)
It's not so clear what we are crunching on our cpu....

Stefan
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 5 Mar 13
Posts: 348
Credit: 0
RAC: 0
Level

Scientific publications
wat
Message 50457 - Posted: 10 Sep 2018 | 7:57:17 UTC - in response to Message 50456.
Last modified: 10 Sep 2018 | 7:57:36 UTC

http://www.gpugrid.net/forum_thread.php?id=4707&nowrap=true#49595

What is not clear about what we are crunching on CPU? If you have any doubts or questions I'll be glad to answer them.

tullio
Send message
Joined: 8 May 18
Posts: 190
Credit: 104,426,808
RAC: 0
Level
Cys
Scientific publications
wat
Message 50458 - Posted: 10 Sep 2018 | 10:22:34 UTC - in response to Message 50457.
Last modified: 10 Sep 2018 | 10:23:24 UTC

After reading what is printed in the output directory of stderr.txt I would welcome a short explanation of what the program is doing. I have taken a structural chemistry course in my university years, talked with Enrico Clementi while working at Mondadori Scientific Publishing and followed some lectures on Density Functional Theory while at Trieste Area Research Park.
Tullio

Jim1348
Send message
Joined: 28 Jul 12
Posts: 819
Credit: 1,591,285,971
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 50459 - Posted: 10 Sep 2018 | 12:52:49 UTC - in response to Message 50458.

The overview that Stefan gave is good enough for me. I expect we will have to await the details until the publication(s). There should be a series of them.

Erich56
Send message
Joined: 1 Jan 15
Posts: 1132
Credit: 10,363,897,676
RAC: 29,196,793
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwat
Message 50460 - Posted: 10 Sep 2018 | 14:28:27 UTC - in response to Message 50414.

somehow I would have expected some kind of statement from the GPUGRID people in reply to my posting above. Well, nothing so far :-(

A look at the current Server Status page shows interesting figures:

GPU tasks - unsent: 0; in progress: 1.761; users in last 24hrs: 924

CPU tasks - unsent: 44.723; in progress: 848; users in last 24hrs: 76

this alone shows the huge imbalance. 924 GPU users have no news tasks available, and 76 CPU users have 44.723 available.
What sense does this make?

There are about 12x more GPU users, not getting new taks. Whereas only 76 users are swamped with 44.723 tasks.

Where is the logic behind this?


too bad that there still is no reaction or reply from GPUGRID side to my above posting.

It can be reduced to the main question: is there a continuing, long-term tendency towards CPU tasks, with a decreasing focus on GPU tasks?
(I do not remember any siginificant period of time within the past 2 1/2 years where no new GPU tasks were available as it was the case within the past weeks).

I guess I am not the only one who would be interested in getting some kind of information.

[VENETO] boboviz
Send message
Joined: 10 Sep 10
Posts: 158
Credit: 388,132
RAC: 0
Level

Scientific publications
wat
Message 50461 - Posted: 10 Sep 2018 | 15:36:25 UTC - in response to Message 50459.

The overview that Stefan gave is good enough for me. I expect we will have to await the details until the publication(s). There should be a series of them.


This is my doubt. I had already read the Stefan post and i understood the "tecnical part" (and it's very interesting), but i don't know what we are crunching for: Hiv? Cancer??? Just a curiosity
I'll wait publications :-)
And i'll wait for windows client :-))

tullio
Send message
Joined: 8 May 18
Posts: 190
Credit: 104,426,808
RAC: 0
Level
Cys
Scientific publications
wat
Message 50462 - Posted: 10 Sep 2018 | 16:33:23 UTC - in response to Message 50461.

Where can I find Stefan's post?
Tullio

Erich56
Send message
Joined: 1 Jan 15
Posts: 1132
Credit: 10,363,897,676
RAC: 29,196,793
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwat
Message 50463 - Posted: 10 Sep 2018 | 16:47:59 UTC - in response to Message 50462.

Where can I find Stefan's post?

I think this is the one:
http://www.gpugrid.net/forum_thread.php?id=4707&nowrap=true#49595

tullio
Send message
Joined: 8 May 18
Posts: 190
Credit: 104,426,808
RAC: 0
Level
Cys
Scientific publications
wat
Message 50466 - Posted: 11 Sep 2018 | 6:56:26 UTC - in response to Message 50463.
Last modified: 11 Sep 2018 | 7:01:46 UTC

Thanks Erich56. Let me see if I have understood. We calculate the energy level of some molecules using Quantum Mechanical methods such as Density Functional Theory, then our results are fed to some neural networks which will perform the same calculations much faster. Is this right?
Tullio

Jim1348
Send message
Joined: 28 Jul 12
Posts: 819
Credit: 1,591,285,971
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 50467 - Posted: 11 Sep 2018 | 16:32:14 UTC - in response to Message 50466.

We calculate the energy level of some molecules using Quantum Mechanical methods such as Density Functional Theory, then our results are fed to some neural networks which will perform the same calculations much faster. Is this right?

That seems to be the case. Interestingly, it appears that Cosmology (a VirtualBox project) also uses BOINC CPUs to train their neural network. I don't know if they perform their final calculations on GPUs or not, but they go much faster as a result of the training.
https://en.wikipedia.org/wiki/Cosmology@Home

tullio
Send message
Joined: 8 May 18
Posts: 190
Credit: 104,426,808
RAC: 0
Level
Cys
Scientific publications
wat
Message 50468 - Posted: 11 Sep 2018 | 16:45:59 UTC - in response to Message 50467.
Last modified: 11 Sep 2018 | 16:47:44 UTC

Recently the scientists at Berkeley SETI Research but also at SETI Institute,two different institutions, have used neural networks to analyze data from the Green Bank radio telescope, financed by Breakthrough Listen, to search for Fast Radio Bursts in the data (400 TB) accumulated at Green Bank and found many emission from a repeating source, the only repeated Fast Radio Burst source discovered so far.They had not been found by traditional methods. The origin of the FRB is still a mystery.
Tullio

Jim1348
Send message
Joined: 28 Jul 12
Posts: 819
Credit: 1,591,285,971
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 50469 - Posted: 11 Sep 2018 | 17:08:24 UTC - in response to Message 50468.

Recently the scientists at Berkeley SETI Research but also at SETI Institute,two different institutions, have used neural networks to analyze data from the Green Bank radio telescope, financed by Breakthrough Listen, to search for Fast Radio Bursts in the data (400 TB) accumulated at Green Bank and found many emission from a repeating source, the only repeated Fast Radio Burst source discovered so far.They had not been found by traditional methods. The origin of the FRB is still a mystery.
Tullio

That is VERY interesting. I have been hoping for some time that all the software expertise and radio time accumulated by SETI could be used for some more useful scientific purpose than LGMs. (Not that they don't exist, but not likely to be found with the hardware we presently have I think.)

tullio
Send message
Joined: 8 May 18
Posts: 190
Credit: 104,426,808
RAC: 0
Level
Cys
Scientific publications
wat
Message 50470 - Posted: 11 Sep 2018 | 17:18:15 UTC - in response to Message 50469.

I am running SETI@home because there are many users who know a lot about computers and GPU boards. Some have put eight of them in a home made supercomputer, contrarily to the SETI original idea which was to exploit "unused cycles". Some of them are also my friends.
Tullio

Erich56
Send message
Joined: 1 Jan 15
Posts: 1132
Credit: 10,363,897,676
RAC: 29,196,793
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwat
Message 50487 - Posted: 13 Sep 2018 | 5:21:35 UTC - in response to Message 50470.
Last modified: 13 Sep 2018 | 5:21:57 UTC

... Some have put eight of them in a home made supercomputer ...

I did not exactly assemble a "supercomputer", but I use altogether 8 PCs and notebooks for crunching BOINC projects (normally 24/7).

Erich56
Send message
Joined: 1 Jan 15
Posts: 1132
Credit: 10,363,897,676
RAC: 29,196,793
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwat
Message 50689 - Posted: 14 Oct 2018 | 7:16:11 UTC

GPU tasks queue drying out again :-(

On the other hand, almost 170.000 CPU tasks for not more than 60 users.

BelgianEnthousiast
Send message
Joined: 7 Apr 15
Posts: 33
Credit: 1,201,157,375
RAC: 0
Level
Met
Scientific publications
watwatwatwatwatwatwatwat
Message 50693 - Posted: 15 Oct 2018 | 16:21:13 UTC

Whole weekend again without WU's while an overload of CPU tasks are waiting to be crunched by a only a small packet of people.

We/I continuously bump into breaks where no WU's are available on GPU while an abundance of Quantum Chemistry CPU WU's are endlessly queuing.

As Erich66 pointed out already a number of times, I do not see any statement from the GPUGrid team.

To our disappointment.

Can anyone have the courtesy and take 5 minutes of time to respond please ?

Will QC CPU based WU's overtake GPU WU's in importance over time ?
Is it the purpose to abandon GPU based WU's ?

What is the strategy ?

Many thanks in advance for a prompt reply.

Profile Retvari Zoltan
Avatar
Send message
Joined: 20 Jan 09
Posts: 2346
Credit: 16,290,365,968
RAC: 6,912,525
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 50695 - Posted: 15 Oct 2018 | 21:18:47 UTC - in response to Message 50693.

We/I continuously bump into breaks where no WU's are available on GPU while an abundance of Quantum Chemistry CPU WU's are endlessly queuing.

As Erich66 pointed out already a number of times, I do not see any statement from the GPUGrid team.

To our disappointment.
This project was running like that for years, even before the QC CPU tasks. I think it will be the same for a couple of years.

Will QC CPU based WU's overtake GPU WU's in importance over time ?
Is it the purpose to abandon GPU based WU's ?
No. (That was answered by the staff before)

mmonnin
Send message
Joined: 2 Jul 16
Posts: 337
Credit: 7,666,958,683
RAC: 10,044,224
Level
Tyr
Scientific publications
watwatwatwatwat
Message 50697 - Posted: 16 Oct 2018 | 12:49:36 UTC

I'm just glad there is pretty steady work during the week. There were times when GPUs could not be kept busy. Even if GPUGrid is not your main project it's good practice to have a backup project at 0% resource share.

BelgianEnthousiast
Send message
Joined: 7 Apr 15
Posts: 33
Credit: 1,201,157,375
RAC: 0
Level
Met
Scientific publications
watwatwatwatwatwatwatwat
Message 50730 - Posted: 21 Oct 2018 | 10:59:05 UTC - in response to Message 50695.

Hi Zoltan,

Thanks for the reply, but I was hoping to rather get a statement from the GPUGrid team itself.

It may have been like that for years, agreed (and I confirm), but it doensn't mean that it should be like that :-)

Maybe it would be possbile to create big enough batches with maybe a little longer deadlines as to have sufficient WU's to also cover weekends and/or holiday periods.

There are some algorithms which enable you to predict future workload (I'm working in the contact centre industry and workforce load predictability is a big topic there).

And yes, I have Einstein and MilkyWay as backup projects, but it's a shame that I have to spend time on other projects while there are still uncountable protein interactions to be calculated.

So happy to have the CPU/GPU question solved, but still on my hunger about the WU scheduling approach.

Cheers & have a nice weekend !

BelgianEnthousiast
Send message
Joined: 7 Apr 15
Posts: 33
Credit: 1,201,157,375
RAC: 0
Level
Met
Scientific publications
watwatwatwatwatwatwatwat
Message 51437 - Posted: 6 Feb 2019 | 11:27:48 UTC

It seems like we're again hitting a low in supply on GPU based WU's.
End of december and january (apart from the last week and a half) were
horrendous for GPU based WU's.

As we discussed and pointed out earlier in this post, there is an abundance
of QC WU's which is only being crunched by 1000 at a time while GPU is
crunched by 3000 simultaneously and hitting continuously a state of depletion
of GPU WU's.

Can some resources working on the QC WU's be redirected to GPU WU's please
so there are ample available and we can keep crunching without interruption ?

Current server statistics : 0 short runs, 0 long runs, 0 ACEMD, 148428 QC
Current crunching Wu's in progress : 730 short runs, 3035 long runs, 1250 QC.

Meaning that GPU based crunching has 4 times as many WU's being crunched compared to QC, while QC has 148428 WU's in queue and GPU WU's totally empty.

This has been the case for a while now (over half a year).
Where is the logic in this ?

Please reconsider your strategy and put more focus on GPU WU's...
And no, it's not my/our purpose to start crunching backup projects (although I do) and I have my personal reasons for that.

We would greatly appreciate to take our question into consideration :-)

Cheers,

BE.

BelgianEnthousiast
Send message
Joined: 7 Apr 15
Posts: 33
Credit: 1,201,157,375
RAC: 0
Level
Met
Scientific publications
watwatwatwatwatwatwatwat
Message 51438 - Posted: 6 Feb 2019 | 11:30:32 UTC - in response to Message 51437.

Or even better : develop QC WU's for GPU ;-)

Post to thread

Message boards : Number crunching : Not Getting GPU Tasks

//