I would like to suggest a 1 WU per GPU policy for this project in order to increase its efficient use of resources and speed up results.

This post is not about fine tuning the project to any particular user or group of users benefit although, ultimately I believe it will benefit everyone. It is about fine tuning the project for the projects benefit. Also, while there are many things that could be done server side to improve efficiency for the purpose of this discussion I would like you to concentrate on this one only.

As things stand we have hosts caching WU's for crunching hours in the future while we have other hosts that can't get any work standing idle. This doesn't make a lot of sense to me. In addition, when a handful of WU's become available they are often quickly gobbled up by hosts that already have one running thus adding it to their cache and not allowing idle hosts the chance to download it because of the vagaries of the BOINC backoff. You only get a WU if it's available when BOINC asks for it.

All this is akin to having 2,000 workers and 1,500 jobs instead of giving 1 job each to 1,500 and having 500 waiting for work, this project is giving 2 jobs to each of 750 and having 1,250 waiting for work. Doesn't make for speedy turnaround of work.

The above is only an example of what is taking place and the numbers are not definitive.

Apart from the unequal distribution of work, the system actually gets worse because a completed unit of work in most cases actually generates another WU thus supplying more work but it can't do that while it is sitting in a hosts cache so again less work available for idle hosts.

There is another problem, as we all know, there are hosts that continually "timeout" and the WU is resent after 5 days, there are other hosts which continually "error out" after long periods of time. These hosts can hold up the progression of 2 WU's at a time under the present policy. However, they would only halt the progression of 1 WU at a time and thus reducing their impact on the project if a 1 WU per GPU policy were implemented.

BOINC when downloaded and installed has cache size set by default and will download 2 WU's even if the card on that host is a slow one. Now if that card only got one it may complete within 24hrs thus speeding the project and earning a bonus however while downloading 2 WU's it will do neither and a lot of users will not even be aware it can be changed or the implications. In addition, there will be plenty of users running multiple projects on BOINC and have a slow card. They want a high cache on some projects but not GPUGrid. They can't because these settings are global and apply to all running projects.

I'm sure you're thinking that GPUGrid can't keep enough work in the queue to keep us all active now so why should we be concerned with speed and efficiency.
Firstly, I would suspect the scientist want their results back as quickly as possible so they can analyze the results and maybe issue more work on the back of the results. There is also another consideration, Stefan has mentioned collaboration with others which if fruitfull could lead to more work and boost demand on GPUGrid. Scientists at GPUGrid may have cause in the future to release a huge amount of WU's and need it to be running at MAX capacity.

In any case the time to fix a leaky roof is when the sun is shining not when it begins to rain and the roof is certainly leaking on this project, leaking resources.

I hope I have made my case clearly.

For people that are like TL:DR, Set in BOINC 0 days works and 0 additional days work. If you crunch 2 WU at a time per GPU, it's probably not the best time to do that, especially because the WU have such high utilization now. We just want to have more people crunching the WUs in parallel.

I also agree that the scientists should implement, at least temporarily, a 1 WU per GPU policy and only until it either errors out, finishes or times out does it get another. This would dramatically speed up work flow

You've made it perfectly clear.
I agree with you, but this reduction of maximum workunits per GPU is effective only when there is a shortage of workunits (which is quite frequent lately). So perhaps this setting should be different for every host, based on the turnaround time of that host. Every host should start with 1 workunit per GPU, but if the turnaround time is below 48 hours, the maximum is set to 2. When the turnaround time is above 48 hours, the maximum is set to 1.
The drawbacks of 1 workunit per GPU are:
1. You can't run two workunits on a single GPU simultaneously (the hosts configured to do so will have an idle GPU, so they have to be reconfigured)
2. It will reduce the RAC of every host, so it will reduce the overall RAC of GPUGrid, because the host won't receive a new GPUGrid workunit until the finished one is uploaded. Hosts without backup projects will be idle during the upload/download period, while hosts with backup projects will process (at least) one workunit from the backup project between GPUGrid workunits, as it will force the host to get work from the backup project(s) until the finished GPUGrid workunit is uploaded. If the backup project has long workunits, or the host has a slow internet connection (mainly hosts outside Europe) this reduction could be significant for these hosts (and for the whole project also). This would make some volunteers to disable their backup project(s), or manually manage them (which is pretty inconvenient).
However we should try it to know exactly its impact.

I actually think it will be just as effective when there are lots of WU's to be had because it's likely at least one or more types of work will have a higher priority setting than others so you don't want them cached in order to get them completed as fast as possible.

Thanks for your support Retvari and Pappa

I dont really second that as reducing the number of WU per GPU will lead to a poor utilization of capable Hardware (gtx 980, 980ti, 1070 and up) because of a bad CPU/GPU ratio. The 1 WU/GPU thing would only work well for mid range Maxwell Cards and most of the Kepler and Fermi Cards. But it will surely reduce the efficiency of the fast machines.

Well, I guess it is all about tasks. Let's also consider some other ideas to fill the queue, especially for the short runs. Just my two Cents.
____________
I would love to see HCF1 protein folding and interaction simulations to help my little boy... someday.

the basic idea is not a bad one from various viewpoints; however, my reservations are mainly because of these facts brought up by Zoltan:

... If the backup project has long workunits, or the host has a slow internet connection (mainly hosts outside Europe) this reduction could be significant for these hosts (and for the whole project also). This would make some volunteers to disable their backup project(s), or manually manage them (which is pretty inconvenient).

So perhaps this setting should be different for every host, based on the turnaround time of that host. Every host should start with 1 workunit per GPU, but if the turnaround time is below 48 hours, the maximum is set to 2. When the turnaround time is above 48 hours, the maximum is set to 1.

Thanks Betting Slip for the thread :) Very nicely put.

Hm, the points made by Zoltan are quite valid I think. It's a risk if people have backup projects with long WUs.

It could make sense if it was as simple as switching a button, so when there are fewer WUs than GPUs it forces 1 WU per GPU, otherwise it allows 2. I don't know how feasible it would be to implement such a feature.

Now that I think about it, on the short queue some people might even want to run 2 WU per GPU if they indeed fit. So it would be better if it was enforced on specific queues, not whole project.

So the question becomes quantifying the damage. We could maybe do a short trial and see what the overall effect is.

May I put in one more remark.. the short run queue is intended for the slower cards where it doesn't make sense to run 2 concurrent jobs anyway. Whereas the fast Maxwell and Pascal cards need 2 tasks to have a good utilization otherwise they might be bottlenecked by the CPU.

So if at all I would apply the 1 Job/GPU rule to the short runs. As a positive side effect, the crunchers with high-end hardware will leave out the short runs to get more GPU utilization by multi-tasking. Which leaves more short runs to the crunchers who really need them.

____________
I would love to see HCF1 protein folding and interaction simulations to help my little boy... someday.

I would think it would be " Max WU in progress GPU"

Host = 1 machine

Would ask/PM Richard Hazelgrove or Jacob Klein https://gpugrid.net/show_user.php?userid=8048

I am not exactly a Boinc expert either ... but can't the Boinc client check the estimated GFLOPS capability of the GPU used and send to the server?

https://boinc.berkeley.edu/dev/forum_thread.php?id=10716

If less than 2000 GFLOPS I would send only one short run per GPU. Otherwise grant 1-2 long runs. IF that is at all possible to configure.
____________
I would love to see HCF1 protein folding and interaction simulations to help my little boy... someday.

I would think it would be " Max WU in progress GPU"

Host = 1 machine

Would ask/PM Richard Hazelgrove or Jacob Klein https://gpugrid.net/show_user.php?userid=8048

If less than 2000 GFLOPS I would send only one short run per GPU. Otherwise grant 1-2 long runs. IF that is at all possible to configure.

Sorry for spelling your name wrong Richard was going to go for the "s" but went with "z" instead. Does explane why I couldn't find you when doing a search, that should have given me a clue but no...

Thanks for helping.

Sorry for spelling your name wrong Richard was going to go for the "s" but went with "z" instead. Does explane why I couldn't find you when doing a search, that should have given me a clue but no...

Thanks for helping.

Will any server policy implemented include a fix for multiple architecture GPUs running simultaneously?
For months ACEMD (CUDA 8.0) Pascal and (CUDA 6.5) Kelper / Maxwell app unable to work together on the same host.

I am running Kepler and Maxwell together so it must just be that they are different versions of CUDA, so I don't expect this to ever be fixed. But I could be wrong.

At present there are not enough tasks to go around & most/all WU's have high GPU utilization so the project doesn't benefit (overall) from even high end GPU's running 2tasks at a time.

At present there are not enough tasks to go around & most/all WU's have high GPU utilization so the project doesn't benefit (overall) from even high end GPU's running 2tasks at a time.

I think though that Betting Slip meant his suggestion in a different way: not talking about 2 tasks running at a time, but rather one task running, with another task downloading already (long time) before the running task gets finished.

However, I am asking whether the policy suggested by him has meanwhile been implemented:
On one of my PCs, when the BOINC manager tried an Update on GPUGRID tasks (i.e. trying to download the next WU) while one WU was in progress, it said "the computer has reached a limit on tasks in progress".

No, it has not been implemented evidenced by one of your computers with a single 750ti having 2 long WU's. How does that even begin to make any sense? http://www.gpugrid.net/results.php?hostid=372115

I'm not holding my breath on it being implemented either.

Frankly, I don't think it's a good idea that crunchers are not starting to bite (not to say: attack) each other because there are WUs in waiting position at the host of a given cruncher, while the GPU of another cruncher is running idle.
The policy in place so far allows for this, and it's up to GPUGRID to either continue it or to change it.

However, the situation of idle GPUs would of course not happen if there were enough WUs available all the time.
So, crunchers should not blame other crunchers for GPUs in idle status.

Do you mind aborting at least one of those WUs? My two 1070s and 980ti are idling, and I'm sure many others are like me.

Frankly, I don't think it's a good idea that crunchers are not starting to bite (not to say: attack) each other because there are WUs in waiting position at the host of a given cruncher, while the GPU of another cruncher is running idle.
The policy in place so far allows for this, and it's up to GPUGRID to either continue it or to change it.

However, the situation of idle GPUs would of course not happen if there were enough WUs available all the time.
So, crunchers should not blame other crunchers for GPUs in idle status.

Zoltan you are very right, there is a risk, but I have been fast enough to claim every aborted task so far. And JoergF, there's no sense in completely withdrawing GPUs from GPUGrid, just have a backup with less priority so it automatically switches to that other work load if there is no work in GPUGrid. This is the only way I can keep my house warm with this intermittent work, and with no babysitting.

Yes --- Leave GPUGrid attached!

I recommend setting GPUGrid at resource share 100 or higher, and attach to other GPU projects like SETI or Einstein or Asteroids, at 0 resource share, as backup projects, meaning they only get work when your non-backup projects don't have work.

Resource Share info:
http://boinc.berkeley.edu/wiki/Preferences#List_of_project_preferences

Honestly, all this griping about "no work" is a bit sickening. You all sure do like to micromanage :) Just attach to multiple projects, set your resource shares correctly, and let BOINC do its job to keep your resources busy!

Somehow I manged to not get a single WU from the 600 new WUs released even though I manually updated all my my computers during the time they were still being claimed.

Somehow I manged to not get a single WU from the 600 new WUs released even though I manually updated all my my computers during the time they were still being claimed.

Has the 1 WU per GPU rule been enforced? It seems so. I have just purchased a new 1080ti and have very poor utilization (~70%) with only one 1WU (although the client config is set to 2WU in parallel). My 6700K (not OC yet) isn't fast enough to feed the 1080ti with data, so this is the outcome. My old 1080 had >90% with 2WU even under Windows 10.

What a bother.

Any suggestion ... aside from switching to Folding@Home ... or cooling the 6700K with LN2 and massively OC it?
____________
I would love to see HCF1 protein folding and interaction simulations to help my little boy... someday.

Well.. at present there are not many WU in the Queue anyway... :-(
____________
I would love to see HCF1 protein folding and interaction simulations to help my little boy... someday.

Has the 1 WU per GPU rule been enforced? It seems so. I have just purchased a new 1080ti and have very poor utilization (~70%) with only one 1WU (although the client config is set to 2WU in parallel). My 6700K (not OC yet) isn't fast enough to feed the 1080ti with data, so this is the outcome. My old 1080 had >90% with 2WU even under Windows 10.

I beg to differ. My app_config.xml is the same as always... but Boinc seems to ignore it.

<app_config>
<app>
<name>acemdlong</name>
<max_concurrent>2</max_concurrent>
<cpu_usage>1.0</cpu_usage>
<gpu_usage>0.5</gpu_usage>
</app>
</app_config>


____________
I would love to see HCF1 protein folding and interaction simulations to help my little boy... someday.

Have you tried disabling Hyper Threading (HT)? I've been experimenting and have seen slightly higher utilization when HT is off on my i7-7700k. CPU usage jumps from 13% to 25% per WU, and GPU-Z shows a small gain in usage.

I wish there was a way to force the WU to use a full physical core without turning HT off, but I haven't been able to figure it out. I tried setting <cpu_usage>2.0</cpu_usage> with HT on, but that didn't help, still 13% usage.

My comments apply to Windows 10, cpu usage in Linux seems less, I'm not sure this would help there.

I've been experimenting and have seen slightly higher utilization when HT is off on my i7-7700k.

[when HT is off] CPU usage jumps from 13% to 25% per WU

GPU-Z shows a small gain in usage [when HT is off].

I wish there was a way to force the WU to use a full physical core without turning HT off, but I haven't been able to figure it out. I tried setting <cpu_usage>2.0</cpu_usage> with HT on, but that didn't help, still 13% usage.

My comments apply to Windows 10, cpu usage in Linux seems less, I'm not sure this would help there.

less CPU tasks = higher GPU usage

less CPU tasks = higher GPU usage

To clarify, are you saying that the only reason that disabling HT is showing an improvement is because it's reducing the number of CPU tasks from other projects?

That would imply that the increase in CPU usage from 12.5% (HT) to 25% (HT off) does not in itself improve WU return times.

For example, if no other projects were running, I would still think that more CPU usage is better regardless, but perhaps it is not that simple.

Using SWAN_SYNC (in combination with no CPU tasks)...

Retvari Zoltan wrote:

Using SWAN_SYNC (in combination with no CPU tasks)...

..or just increase priority of GPU tasks.
I wrote earlier in this message how to automate the process of increasing priority of GPU tasks.
I see no reason to abandon computing CPU tasks, if there is a quite simple way to minimize their effect(or other programs, if it's not a dedicated host for computing) on GPU tasks.

Retvari Zoltan wrote:

Using SWAN_SYNC (in combination with no CPU tasks)...

..or just increase priority of GPU tasks.
I wrote earlier in this message how to automate the process of increasing priority of GPU tasks.
I see no reason to abandon computing CPU tasks, if there is a quite simple way to minimize their effect(or other programs, if it's not a dedicated host for computing) on GPU tasks.

Are you seeing a speedup by hacking the priorities with Process Hacker? And, if so, isn't the real reason that you see it is because you're bumping priorities to be higher than the (Normal priority = 8) tasks that Windows uses for all your other non-BOINC processes?

BOINC is meant to run tasks without interfering with the normal operations of a device. So, I think our defaults work well to do that, and I think your hacking may work well to achieve more GPU throughput at the expense of potentially slowing down normal operations of the device which now run at a priority lower than your hacked processes.

Thanks for the reply. I'm curious how much of a speedup your process hacking actually gets?

well ..... back to my question and the topic ... is the 1 Work Unit Per GPU rule now set? If so, may I protest. That measure has an enormous impact on my GPU utilization.

____________
I would love to see HCF1 protein folding and interaction simulations to help my little boy... someday.

No, I don't think it is set. What evidence do you have? I have a PC with 2 GPUs and 4 GPUGrid GPU tasks.

is the 1 Work Unit Per GPU rule now set?

Thanks for the reply. I'm curious how much of a speedup your process hacking actually gets?

Thanks Retvari.

You must understand that your goals of "maximized GPU performance" are radically different than my goal of "help out lots of projects with science, on my regular PC". I'm attached to all projects, do work for about 12 of them, I happily run lots of CPU tasks alongside the GPU tasks, and I use Windows 10 (Insider - Fast Ring - I love to beta test). I do not care at all about credit, but I do care about utility of helping the projects.

I adjust my settings to allow a little bit of a performance boost for GPUGrid.
My settings are:
- Use app_config to budget 0.500 CPU per GPUGrid task
- Main PC: Set BOINC to use (n-1) % CPUs
- Unattended PCs: Set BOINC to use 100% CPUs

One of the main reasons I use (n-1) there, also, is because there has always been a Windows 10 shell priority bug where right-clicking taskbar icons gets a delayed response - sometimes several seconds! That piece of shell code must be running at IDLE priority, and stalls if using 100% CPUs! Using (n-1) alleviates it.

Kind regards,
Jacob

I can understand the different goals and mindsets with all the projects out there. I'd say my goals are more in line with Retvari's; fastest GPU WU return times possible. With this in mind, I've detached from all CPU projects and have seen improvements. The faster/ more GPU's you have in your system the more this makes sense. An improvement of only 1% on the return times from my pair of GTX 1080's easily provides more RAC than my i7-7700k could running CPU tasks alone.

Regardless of your goals, Aleksey's advice of running "Process Hacker" can be extremely beneficial for those of us running tasks on a non-dedicated machine. For example, I run Plex media server on the same machine, and my WUs would get decimated whenever Plex started transcoding video. The Plex transcoder ran at "Normal" priority an WUs at "Below Normal". With "Process Hacker" I was able to permanently adjust Plex transcoder to a lower priority and WUs to a higher priority. Now the WU's hardly slow at all during transcoding and I've seen no performance reduction in Plex. Thank you!

I would encourage you to do some testing, using the following BOINC client options (which you'd use in cc_config.xml), especially <process_priority_special>, which might allow you to do what you want without hacking.

https://boinc.berkeley.edu/wiki/Client_configuration

<no_priority_change>0|1</no_priority_change>

If 1, don't change priority of applications (run them at same priority as client).
NB: This option can, if activated, impact system responsiveness for the user. Default, all CPU science apps run at lowest (idle) priority Nice 15.

<process_priority>N</process_priority>
<process_priority_special>N</process_priority_special>

The OS process priority at which tasks are run. Values are 0 (lowest priority, the default), 1 (below normal), 2 (normal), 3 (above normal), 4 (high) and 5 (real-time - not recommended). 'special' process priority is used for coprocessor (GPU) applications, wrapper applications, and non-compute-intensive applications, 'process priority' for all others. The two options can be used independently. New in 7.6.14

Try them and let us know :)

I reproduced your problem - GPUGrid GPU tasks aren't honoring that <process_priority_special>. Other projects' GPU tasks do honor it.

I guess that's another bug to add to GPUGrids list of brokenness. Dangit.

I reproduced your problem - GPUGrid GPU tasks aren't honoring that <process_priority_special>. Other projects' GPU tasks do honor it.

I guess that's another bug to add to GPUGrids list of brokenness. Dangit.

Edit: Double post :(

I reproduced your problem - GPUGrid GPU tasks aren't honoring that <process_priority_special>. Other projects' GPU tasks do honor it.

I guess that's another bug to add to GPUGrids list of brokenness. Dangit.

Would swan_sync play a role with this priority option?

Would swan_sync play a role with this priority option?

I reproduced your problem - GPUGrid GPU tasks aren't honoring that <process_priority_special>. Other projects' GPU tasks do honor it.

I guess that's another bug to add to GPUGrids list of brokenness. Dangit.

Well, it should be fixed. If the user is going to use Swan_Sync via manual system variable, they can use manual cc_config to control the priority (if the app isn't rude like it is currently).

Fixable. Some staff required.