Problem: Remaining (Estimated) time is unusually high; duration correction factor unusually large

Basically, recently I've noticed that my GPUGrid tasks are saying they're going to take a long long time (140 hours), when they normally complete in about 1-6 hours.

I've tracked it down to the duration correction factor. This is usually between 1 and 2 on my system, but something is making it unusually large, right now it is above 9.

Is the problem in:
- the new beta acemd application?
- the plan classes for that application?
- the MJHARVEY TEST 10/12/13 tasks I've recently processed?
- something else?

Is there any way I can further diagnose this issue?
We need to identify the problem, and correct it if possible.

Thanks,
Jacob

There was at least 1 other user that reported it also, in the beta news thread.

All I can do is report the problem, with as much info as I can provide, like a good tester should. I wish it was properly investigated and fixed by the administrators.

There was at least 1 other user that reported it also, in the beta news thread.

All I can do is report the problem, with as much info as I can provide, like a good tester should. I wish it was properly investigated and fixed by the administrators.

Until you get more tasks that throw it out of balance.
Problem remains to be solved at the server.

I see this too that the remaining time is not correct. I saw it first after we got a lot of MJH beta tests to optimize the app and then "normal" WU's again.
I thought it was not a big deal as the WU crunch at good speed. However looking a bit to the system, it could be that as BOINC says it need 4 hours more to finish, it will not request new work. In reality these 4 hours mean 20 minutes, so it could ask for new work. This could explain why I did not get now work last afternoon, but when I hit the "update" button I got a WU.
If this is true than it would be nice if it can be solved at server side.
____________
Greetings from TJ

... And now my Duration Correction Factor is back up to 12!

I think the MJHARVEY beta tasks are throwing it off (by possibly having bad estimates).
Is there a way to conclusively prove this?

Admins:
Is there a way to fix this server-side so it stops happening?
It may even be related to the recent EXIT_TIME_LIMIT_EXCEEDED errors people are getting!

I'm fairly certain DCF is very relevant.

The BOINC client "remembers" how much time a task was estimated to have taken, when it received said task. Then, when the task is done, it makes comparison to the actual time versus the estimated time. It stores that off as a client-side "factor" for the project (viewable in the project properties, called Duration Correction Factor DCF), so that in the future, when it gets a task from that project, it can "dynamically adjust" the Remaining time to better reflect the estimate.

The problem that we're seeing is that DCF is way larger than expected, all of the sudden. And I believe the root cause is that recent tasks might have been sent with very incorrect estimated times (or maybe very incorrect estimated sizes of FLOPS or FPOPS).

Can you please check into the estimates of recent tasks?
Are the beta ones or the "_TEST" ones possibly very incorrect?

:) Right now my DCF (which is usually between 1.0 and 1.6) is at 29.88, and a long-task (which usually completes in 8-16 hours) is currently estimated at 281 hours (~12 days!)

Note: Projects can also choose to "not use" DCF, especially when they have various apps where estimates could be way off, per app. World Community Grid, for instance, has disabled DCF usage. I personally believe GPUGrid should keep using it.

I'm not certain why it is "losing its marbles" lol, but I don't think it's related to a "no progress" task.

Instead, I think it gets "adjusted" when a task is completed, and it compares the "estimated time before starting the task" versus "actual time after completing a task". The estimated task size is very important in that calculation, and it feels like some tasks went through with incorrect estimated sizes.

I like DCF for several reasons. It accounts for "how BOINC normally runs, possibly fully loaded with CPU tasks", it accounts for computer-specific hardware scenarios that the project doesn't know about, maybe the user plays games while GPU crunching some of the time... all sorts of reasons to use it. I just wish it was app-based and not project-based. But, for GPUGrid, I think it's useful to still use, since you have been able to make accurate estimates for all your apps, in the past, and you seem to have always used the same method of making those estimates regardless of app.

Regarding how it works... Once the WU is in progress, I believe DCF is still used even until the task finishes. (Not all tasks report linear progress like GPUGrid). So, for instance, here where the DCF is very skewed, the "remaining" time may start at 281 hours, but each clock-second that happens, the remaining time decreases 30 seconds. Once tasks complete, then the DCF gets adjusted for future tasks.

Again, I believe the real issue is that something is wrong in the estimates of recent tasks.

Have you investigated whether
<rsc_fpops_est>
... was and is being set correctly for all recent tasks?
If it's incorrect, it'll be the cause of DCF being improperly adjusted, resulting in bad estimated times people are seeing.

Also, have you instigated whether
<rsc_fpops_bound>
... was and is being set correctly for all recent tasks?
If it's incorrect, it'll be the cause of the "exceeded maximum time" EXIT_TIME_LIMIT_EXCEEDED errors people are seeing.

My GTX 670 host 132158 has been running through a variety of Beta tasks

I recently finished one of the full-length NOELIA_KLEBEbeta being put through the Beta queue: I'm now working through some more MJHARVEY_TEST.

Runtimes vary, as you would expect: in very round figures, the NOELIA ran for 10 hours, and the test pieces for 100 minutes, 10 minutes, or 1 minute - a ratio of 600::1 between the longest and shortest.

I've checked a few <rsc_fpops_est> and <rsc_fpops_bound> pairs, and every workunit had been identical: The values have been

Yes, a million times over!

Admins:
Regardless of whether we're still sending these tasks out, could we PLEASE make sure the code is correctly setting the values for them? They're causing estimation chaos!

Disclaimer first: I speak as a long-term (interested) volunteer on number of BOINC projects, but I have no first-hand experience of administering a BOINC server. What follows is my personal opinion only, but an opinion informed by seeing similar crises at a number of projects.

My opinion is that the current issues largely arise from the code behind http://boinc.berkeley.edu/trac/wiki/RuntimeEstimation. It appears that this code is equisitely sensitive to outlier values in wu.rsc_fpops_est, and the hypersensitive code has a nasty habit of creeping out of the woodwork and biting sysadmins' ankles at times of stress.

I think there are two reasons for that:

1) it's exactly at times like this - working weekend overtime to chase out a stubborn application bug, with a project-full of volunteers clamouring for new (error-free) work - that harrassed sysadmins are likely to take obvious shortcuts: put the new test app on the server with the same settings as last time, make the tasks shorter so the results come back sooner. I think we can all recognise, and empathise with, that syndrome.

2) BOINC's operations manual, and systems training courses (if they run any), don't emphasise enough: no shortcut through here - here be dragons, and they will demand a sacrifice if you get it wrong. Unless you run a completely sandboxed test server, these beta apps - it's always beta apps - will screw your averages and estimates.

This runtime estimation code has been active in the master code base for BOINC servers for over three years now, and as I said, I've seen problems like this several times now - the most spectacular was at AQUA, where it spilled over into credit awards too.

But to the best of my knowledge, no post-mortem has ever been carried out to find out why things go so badly wrong. In - again - my personal opinion only, this area of code needs to be re-visited, and re-written according to sound engineering - fault tolerant and error tolerant - principles (the way the original BOINC server code was written).

However, I see no sign that the BOINC core programmers have any plans to do this. Some project administrators have looked into the code, and expressed frustration at having to work with it in its current form - Eric Korpela of SETI and Bernd Machenschalk of Einstein come to mind - but I've not heard of any improvements being migrated back into the master codebase.

There is a workshop "for researchers, scientists and developers with significant experience or interest in BOINC" at Inria near Grenoble, at the end of this month. If anybody reading this has any sympathy with my point of view, that workshop might be a suitable opportunity to raise any concerns you might have.

:) I hope we don't have to travel to France for a solution.

Admins:
Can you please solve this issue? We've given tons of information that should point you in the right direction I hope!

Each task is sent with an estimate. You can even view that estimate in the task properties. It can be different amongst tasks within the same app.

We need the admins to perform an investigation, and to fix the problem.

The opposite happens too. I have now a NOELIA_KLEBEbeta-2-3... that has done 9% in 1 hour, but in 5 minutes it will be finished....?
And an HARVEY_TEST that will take about 15 minutes, is expected to run 6h53m52s.

If I have watched correct it all started with the HARVEY_TEST that took around 1 minute, but estimation of 12 hours.
____________
Greetings from TJ

Each task is sent with an estimate. You can even view that estimate in the task properties.

The BOINC manager suspends a WU which has normally estimated run time when it receives a fresh WU which has overestimated run time (my personal high score is 2878(!) hours), which makes my batch programs think that they are stuck (actually they are, it's intentional to give priority to the task with the overestimated run time).
This is really annoying.

The BOINC manager suspends a WU which has normally estimated run time when it receives a fresh WU which has overestimated run time (my personal high score is 2878(!) hours), which makes my batch programs think that they are stuck (actually they are, it's intentional to give priority to the task with the overestimated run time).
This is really annoying.

This is really annoying.

This is really annoying.

I agree, it is annoying.
I reported it as soon as I spotted it, over a week ago.
Hopefully the admins take it a bit more seriously.

EDF policy says we should run the ones with earliest deadlines.

Note: this is how it used to be (as designed by John McLeod). I attempted to improve it, and got it wrong.
[DA]

The problem Retvari Zoltan is annoyed about - BOINC suspending one task and running a different one when high priority is needed - isn't something the project admins can solve (except by getting the estimates right so EDF isn't needed, obviously).

Decisions about which task from the cache to run next are taken locally by the BOINC core client. v6.10.60 is getting quite old now - and yes, this bug has been around that long. It was fixed in v7.0.14:

I've just picked up a new Beta task - 7238202

New (to me) task type MJHARVEY_TESTSIN, and new cuda55 application version 8.06

Task has the same 5 PetaFpops estimate as we're used to seeing, and - what with it being a new app and all - speed estimate was 295 GigaFlops, lower than previously on this host. DCF is still high, so BOINC calculated the runtime as 87 hours. Looks like it's turning into a 100 minute job... (but still leapt into action in High Priority as soon as I let my AV complete the download).

I've just picked up a new Beta task - 7238202

Just a heads-up: estimated times for the current v8.10 (cuda55) Beta are unusually low - it is likely that many runs (especially of the full-length production NOELIA_KLEBEbeta tasks being processed through the Beta queue) will fail with EXIT_TIME_LIMIT_EXCEEDED - after about an hour, on my GTX 670.

If you are a BOINC user experienced and competent enough to edit client_state.xml - and taking all the standard safety warnings as read - you can avoid this by increasing <rsc_fpops_bound> for all GPUGrid Beta tasks. A couple of orders of magnitude should do it, maybe three for luck.

Yup, I just had some tasks fail because of that poor server estimation.
This is getting very ridiculous.

Just a heads-up: estimated times for the current v8.10 (cuda55) Beta are unusually low - it is likely that many runs (especially of the full-length production NOELIA_KLEBEbeta tasks being processed through the Beta queue) will fail with EXIT_TIME_LIMIT_EXCEEDED - after about an hour, on my GTX 670.

If you are a BOINC user experienced and competent enough to edit client_state.xml - and taking all the standard safety warnings as read - you can avoid this by increasing <rsc_fpops_bound> for all GPUGrid Beta tasks. A couple of orders of magnitude should do it, maybe three for luck.

All the WU actualy running in this host has a estimated of +/- 130hrs! (normal time to crunching a WU = 8-9 hrs)

http://www.gpugrid.net/results.php?hostid=157835&offset=0&show_names=0&state=1&appid=

So actualy most of them are crunching in high priority mode.
____________

Please let me know if you start to see this problem on the important acemdshort and acemdlong queues. There's no reason why it should be happening there (any more than usual), but the client is full of surprises.

MJH

8.10 ACEMD beta version (cuda55) 109nx46-NOELIA_KLEBEbeta-0-3-RND7143_2 01:58:09 Reported: Computation error (197,)

8.10 ACEMD beta version (cuda55) 66-MJHARVEY_TEST10-42-50-RND0504_0 01:58:02 Reported: Computation error (197,)

The Noelia was also reported as failed by four other people thus far, whereas the Harvey was completed successfully by someone running ACEMD beta version v8.05 (cuda42)

Sorry. I thought you were asking about the betas. No problems with the longs thus far. All seven that I have received under CUDA 5.5 have been completed successfully.

8.03 Long runs (cuda55) 063ppx239-NOELIA_FRAG063pp-2-4-RND4537_0 20:57:08
8.03 Long runs (cuda55) 041px89-NOELIA_FRAG041p-3-4-RND5262_0 17:41:48
8.03 Long runs (cuda55) 041px89-NOELIA_FRAG041p-2-4-RND5262_0 17:39:49
8.03 Long runs (cuda55) 063ppx290-NOELIA_FRAG063pp-1-4-RND3152_0 20:59:32
8.03 Long runs (cuda55) I35R7-NATHAN_KIDKIXc22_6-8-50-RND8566_0
8.02 Long runs (cuda55) I50R6-NATHAN_KIDKIXc22_6-3-50-RND0333_0 17:48:16
8.00 Long runs (cuda55) I81R8-NATHAN_KIDKIXc22_6-4-50-RND0944_0 17:44:35

All the WU actualy running in this host has a estimated of +/- 130hrs! (normal time to crunching a WU = 8-9 hrs)

http://www.gpugrid.net/results.php?hostid=157835&offset=0&show_names=0&state=1&appid=

So actualy most of them are crunching in high priority mode.

MJH,

Thanks for finding the cause of the problem.
Has the problem been fixed, such that new tasks (including beta!) are sent with appropriate fpops estimates?

Once the problem has been fixed at the server, if a user wants to immediately reset Duration Correction Factor (instead of waiting it to adjust down over the course of a few days/weeks), they could do these steps:

- Exit BOINC, including stopping running tasks
- Open client_state.xml within the data directory
- Search for the <project_name>GPUGRID</project_name> element
- Find the <duration_correction_factor> element within that project element
- Change the line to read: <duration_correction_factor>1</duration_correction_factor>
- Restart BOINC
- Monitor the value in the UI by viewing the GPUGrid project properties, to test, to make sure it hopefully stays between 0.6 and 1.6.

.... I just need to know when the problem has been fixed by the server, so I can begin testing the solution on my client (which has luckily not been full of too many surprises).

Has it been fixed?

This problem should confined to the beta queue and is a side-effect of having issued a series of short running WUs with the same fpops estimate as normal longer-running ones.

Please let me know if you start to see this problem on the important acemdshort and acemdlong queues. There's no reason why it should be happening there (any more than usual), but the client is full of surprises.

MJH

I have tried 3 betas in the long queue

I'm not sure what you mean my that. Those are WUs from the acemdbeta queue, run by the beta application. The acemdlong queue isn't involved.

MJH

The problem is, that if you run a succession of short test units with full-size <rsc_fpops_est> values, then the BOINC server thinks your host is insanely fast

The server doesn't have any part in it - it's the client making that decision.
anyway, the last batch of short WUs have been submitted, with no more to follow. Hopefully the client will be as quick to correct itself back as before.

MJH

In the last month, my two machines seem to have stabilized for errors. Last errors were the server canceled runs Aug 24 and before that Jul 30 - Aug 11 with the NOELIA runs.

Just added the Beta runs to be allowed a couple days ago and last night they started running on one machine. The HARVEY's went through no problem. The NOELIA_KLEBEbeta did error with 197 (0xc5) EXIT_TIME_LIMIT_EXCEEDED after running ~30 of the HARVEY's.

http://www.gpugrid.net/result.php?resultid=7246089

When reviewing all the comments about rsc_fpops_bound, duration_correction_factor, etc., it seems that with the current setup, one minute runs and 8 hour runs should not be mixed under the same app. Since it is our side which is calculating how long it should take, and our side is limited by the app definitions.

<app_version>
<app_name>acemdshort</app_name>
<version_num>800</version_num>
<flops>741352314880.428590</flops> (0.74 TFlops)
<file_name>acemd.800-42.exe</file_name>

<app_version>
<app_name>acemdshort</app_name>
<version_num>802</version_num>
<flops>243678296053.232120</flops> (0.24 TFlops)
<file_name>acemd.802-55.exe</file_name>

<app_version>
<app_name>acemdlong</app_name>
<version_num>803</version_num>
<flops>133475073594.177120</flops> (0.13 TFlops)
<file_name>acemd.800-55.exe</file_name>

<app_version>
<app_name>acemdlong</app_name>
<version_num>803</version_num>
<flops>185603220803.599580</flops> (0.19 TFlops)
<file_name>acemd.800-55.exe</file_name>

<app_version>
<app_name>acemdbeta</app_name>
<version_num>811</version_num>
<flops>69462341903928.672000</flops> (69.46 TFlops)
<file_name>acemd.811-55.exe</file_name>

My two machines also make this a bit more complicated in that they both have a GTX680 and a GTX460 so it seems that the estimated time remaining is driven by the GTX460 speed. That seems to work out ok though. There seems to be no tracking ability local or server for the different cards so this is a moot point.

So if the acemdbeta app could be run with at least different version with identical binary (e.g. 5min_811 and 8hr_811 where 5 minutes and 8 hours expected time), this could at least separate the runs such as HARVEY and KLEBE which have orders of magnitude in different run times.

This may possibly reduce the number of errors due to time outs.

Thanks all for te explanations, the path explained by Jacob aparently fix the times.

Now the question remains, it´s fixed at the server side? can i allow to receive new Beta WU or is beter wait a little more?
____________

Thanks all for te explanations, the path explained by Jacob aparently fix the times.

Now the question remains, it´s fixed at the server side? can i allow to receive new Beta WU or is beter wait a little more?

Richard is right, as usual. I'm going to take my computer off of the beta queue, until a new app and new APRs are in place; too much wasted computing power is occurring.

MJH:
Please let us know when a new beta app (with new APRs) is released on the server.

Thanks all for te explanations, the path explained by Jacob aparently fix the times.

Now the question remains, it´s fixed at the server side? can i allow to receive new Beta WU or is beter wait a little more?

Having consulted the usual oracle on that, we think it would be wise to wait a little longer.

Although changing DCF will change the displayed estimates for runtime, we don't think it affects the underlying calculations for EXIT_TIME_LIMIT_EXCEEDED. And whatever you change, DCF is re-calculated every time a task exits. If you happen to draw another NOELIA_KLEBEbeta, DCF will go right back up through the roof in one jump when it finishes.

The only solution is a new Beta app installation, with a new set of APRs - and that has to be done on the server.

Having consulted the usual oracle on that, we think it would be wise to wait a little longer.

Richard is right, as usual. I'm going to take my computer off of the beta queue, until a new app and new APRs are in place; too much wasted computing power is occurring.

Please let us know when a new beta app (with new APRs) is released on the server.

MJH:

If people get stuck in a "Maximum Time Exceeded" loop, where they cannot complete their tasks, then surely their time will be wasted with the work, no?

Richard knows what needs to be done in order to get people out of that scenario. It deals with APRs on the Application Details page - Average processing rate is way too high for some applications.

As long as the "busted APR" applications aren't used anymore, then I think the problem will go away from a user's perspective, as DCF will eventually work it's way back to 0.6-1.6. Since you say the beta is over, I take it we're done issuing tasks on these "busted APR" applications, right? (ie: no future beta will ever use these applications)

But I'm also hopeful you can do something to prevent the issue from happening in the future - maybe some safeguard that ensures proper fpops estimates. Just a thought.

For your reference, I care more about the "Maximum time exceeded" errors than the "bad estimates" problems. The former cause lost work/time.

But I'm also hopeful you can do something to prevent the issue from happening in the future - maybe some safeguard that ensures proper fpops estimates.

Yes, of course! We've not tried this method of live debugging using short WUs before and weren't expecting this unfortunate side-effect. Next time the fpops estimate will be dialled down appropriately.

MJH

I was just about to suggest that we wait until this urgent fine-tuning of the app was complete, but I see we've reached that point already if you're happy with v8.11

So I guess we're left with some MJHARVEY_TEST tasks with, ahem, 'creative' runtime estimates working their way through the system on the Beta queue, and in addition some of the full-length NOELIA_KLEBEbeta tasks.

Are you able to monitor the numbers of tasks of each type still remaining in the queue?

The best course of action might be to wait until all MJHARVEY_TEST tasks are complete and out of the way - or even help them along, with a 'cancel if unstarted' purge.

Then, once we are sure that no MJHARVEY_TEST tasks are left or in any danger of being re-issued with current estimates, re-deploy the Beta app as version 812 (doesn't have to be any change in the app itself). That could then be left to clean up the remaining NOELIA_KLEBEbeta tasks with a new app_version detail record, which would automatically be populated with more realistic APRs.

There's nothing you can do at the server end to hasten the correction of DCF - that's entirely managed by the client, at our end. Those who feel confident will adjust their own, others will have to wait, but it'll come right in the end.

One other small point while we're here - I noticed that the CUDA 5.5 app_version referenced the v5.5 cudart and cufft DLLs - correct. But it also referenced the equivalent CUDA 4.2 DLLs. Subject to checking at your end, that sounds like duplication, which might add an unnecessary bandwidth overhead as both sets of files are downloaded by new testers. Might be scope for a bit of a saving there.

There's nothing you can do at the server end to hasten the correction of DCF - that's entirely managed by the client, at our end. Those who feel confident will adjust their own, others will have to wait, but it'll come right in the end.

Will disconnecting and re-attaching to the project force a reset?

I just got the following task:
http://www.gpugrid.net/result.php?resultid=7248506

Name 063px55-NOELIA_KLEBEbeta-2-3-RND9896_0
Created 4 Sep 2013 | 17:38:22 UTC
Sent 4 Sep 2013 | 17:41:12 UTC
Application version ACEMD beta version v8.11 (cuda55)

I believe it is a new task on the beta queue, though I'm not sure if this app version has "busted APR" or not.

Can we make sure that any new tasks are not using the "busted APR" app versions?

There's nothing you can do at the server end to hasten the correction of DCF - that's entirely managed by the client, at our end. Those who feel confident will adjust their own, others will have to wait, but it'll come right in the end.

Will disconnecting and re-attaching to the project force a reset?

Richard,

For APR: Does that means that, to effectively solve the APR problem, the solution is to create a new app version for any app version that had tasks sent with bad estimates. Is that correct?

Richard,

For APR: Does that means that, to effectively solve the APR problem, the solution is to create a new app version for any app version that had tasks sent with bad estimates. Is that correct?

The last KLEBE beta WUs are being deleted now.

I did a reset project, but the estimated time for a new WU afterwards is still wrong. One from 1.5 minutes was estimated 1h45m and a SANTI SR 7h40m52s. This will be faster already done 3% in 5m.
____________
Greetings from TJ

I thought the problem had been solved, but now I don't know.
Overnight I had completed the following 5 tasks, and now my Duration Correction Factor is the highest it's ever been, 96. Some new tasks are saying they'll take 848 hours to complete :)

Do any of the following apps have "busted APR" from having tasks with inappropriate <rsc_fpops_est> values?
- ACEMD beta version v8.11 (cuda55)
- ACEMD beta version v8.13 (cuda42)
- ACEMD beta version v8.13 (cuda55)

Can you answer my question about the apps?

Oh. I only know a little, but I'll share what I know. Richard knows a ton about it.

From what I gather... each app-version that a host runs, gets its own "set of statistics". So... If you click my computers, and view the details of RacerX, and then click "Application details", you'll see all the app-versions I've run, including statistics on them.

For the app-versions that were sent out with bad <rsc_fpops_est> values, you'll see that my "Average processing rate" (APR) was very jacked up (normal values for me are about ~350 for short-run queue, ~100 for long-run queue; but if you look at the beta apps on my list there, you'll see values like 4000, 7000, 16000)

If the APR is very high, then the result is that the server thinks the card is like 40-160 times faster than it actually is! So, in a sense, we can use this APR to see if apps had been sent with inappropriate <rsc_fpops_est> values. At the client, when a task with incorrect <rsc_fpops_est> is processed, the clear indication is that Duration Correction Factor gets jacked up well. And then the client time estimates get jacked.

I was hoping that we had fixed all this "bad <rsc_fpops_est> values" business (which I've termed as "busted-APR apps"), but last night it seems one of the tasks messed it up. I noticed it because DCF was wrong, and running tasks said 400+ hours.

The main question I have:
On the beta queue, which app version is the earliest where you believe the <rsc_fpops_est> issue was fully resolved?

I'm pretty sure the answer is not "8.11"; I think the 8.11 task may have been the culprit out of the 5 tasks from a couple posts up.

All of the CRASH tasks are exact copies of SANTI-MAR4222s

MJH

Oh. I only know a little, but I'll share what I know. Richard knows a ton about it.

From what I gather... each app-version that a host runs, gets its own "set of statistics". So... If you click my computers, and view the details of RacerX, and then click "Application details", you'll see all the app-versions I've run, including statistics on them.

For the app-versions that were sent out with bad <rsc_fpops_est> values, you'll see that my "Average processing rate" (APR) was very jacked up (normal values for me are about ~350 for short-run queue, ~100 for long-run queue; but if you look at the beta apps on my list there, you'll see values like 4000, 7000, 16000)

If the APR is very high, then the result is that the server thinks the card is like 40-160 times faster than it actually is! So, in a sense, we can use this APR to see if apps had been sent with inappropriate <rsc_fpops_est> values. At the client, when a task with incorrect <rsc_fpops_est> is processed, the clear indication is that Duration Correction Factor gets jacked up well. And then the client time estimates get jacked.

I was hoping that we had fixed all this "bad <rsc_fpops_est> values" business (which I've termed as "busted-APR apps"), but last night it seems one of the tasks messed it up. I noticed it because DCF was wrong, and running tasks said 400+ hours.

The main question I have:
On the beta queue, which app version is the earliest where you believe the <rsc_fpops_est> issue was fully resolved?

I'm pretty sure the answer is not "8.11"; I think the 8.11 task may have been the culprit out of the 5 tasks from a couple posts up.

Thanks Jacob,

In fact our value for rsc_fpops_est has never changed - it is set the same for every WU we have ever put out on every queue.

Evidently the mechanisms that depend on it being accurate had adjusted to accept as correct our typical WU lengths, but got very confused when I put out a whole batch of very short WUs with the same pops estimate.

MJH

Thank you Matt and Richard.

Not very sure if you still want this info. Maybe you could be more precise:

CUDA: NVIDIA GPU 0: GeForce GTX 260 (driver version 331.65, CUDA version 6.0, compute capability 1.3, 896MB, 818MB available, 912 GFLOPS peak)

ACEMD beta version v8.15 (cuda55)
77-KLAUDE_6429-0-2-RND1641_1 Expected to finish in 22h. 83% processed right so far.


But on the other GPU I have one that is messing quite much with time estimation:
CUDA: NVIDIA GPU 1: GeForce GTX 560 Ti (driver version 331.65, CUDA version 6.0, compute capability 2.1, 1024MB, 847MB available, 1352 GFLOPS peak)

Long runs (8-12 hours on fastest card) v8.14 (cuda55)
Progress 43% 7h so far, remaining 58h!!

I don't really care about estimated times. Just in case this info is of some help.