Message boards : Number crunching : Bad batch of WU's
| Author | Message |
|---|---|
|
flashawk Send message Joined: 18 Jun 12 Posts: 297 Credit: 3,572,627,986 RAC: 0 Level  Scientific publications    | |
|
I just shot up to 21 errors, I was watching a group of WU's startup and all of them pushed my 3 1080's to 2100MHz and they failed. They are failing on everyone's computers, they are Adria's WU's. | |
ID: 49260 | Rating: 0 | rate:
 /   Reply
Quote
| |
|
WPrion Send message Joined: 30 Apr 13 Posts: 96 Credit: 2,686,634,111 RAC: 20,729,906 Level  Scientific publications    | |
|
I just had two Pablo's crash as soon as they started. | |
| ID: 49261 | Rating: 0 | rate:
/ Reply
Quote
| |
|
flashawk Send message Joined: 18 Jun 12 Posts: 297 Credit: 3,572,627,986 RAC: 0 Level Scientific publications | |
|
Strange, both their WU's are crashing. You have 7 from today, I was poking around and everyone is getting errors on long WU's. | |
| ID: 49262 | Rating: 0 | rate:
/ Reply
Quote
| |
 Retvari Zoltan Send message Joined: 20 Jan 09 Posts: 2343 Credit: 16,217,465,968 RAC: 1,257,790 Level  Scientific publications | |
|
All tasks error out immediately with: ERROR: file mdioload.cpp line 81: Unable to read bincoordfile This should be fixed by the staff ASAP. | |
| ID: 49263 | Rating: 0 | rate:
/ Reply
Quote
| |
|
flashawk Send message Joined: 18 Jun 12 Posts: 297 Credit: 3,572,627,986 RAC: 0 Level Scientific publications | |
|
Any word yet? Has anyone gotten any WU's yet and did they fail? Where are the moderators at, I haven't seen any mods sense I've been back. Also, has anyone heard anything about CPDN that crunches for them? Their forum and everything else has been down for almost 2 weeks now, no WU's, not a word. their main page is up but no mention of what happened, I know there's some people here that crunch for them. just wondering if they might have heard something. | |
| ID: 49265 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Richard Haselgrove Send message Joined: 11 Jul 09 Posts: 1620 Credit: 8,866,392,698 RAC: 20,073,067 Level  Scientific publications  | |
|
There was a batch of bad PABLO tasks tasks created between about 17:30 - 18:00 UTC yesterday afternoon. I've watched some crash, and I've aborted some others (after checking that they had failed on other machines first). But there are good tasks created before and after. | |
| ID: 49266 | Rating: 0 | rate:
/ Reply
Quote
| |
|
3de64piB5uZAS6SUNt1GFDU9d...   Send message Joined: 20 Apr 15 Posts: 285 Credit: 1,102,216,607 RAC: 0 Level  Scientific publications | |
|
I had quite a few today as well... scared me to death. Frankly I suspected my 4 month young ASUS ROG gtx1070 of being defective and was (figuratively) about to throw it out of the window... when I stumbled across the same error | |
| ID: 49267 | Rating: 0 | rate:
/ Reply
Quote
| |
|
It seems another bad batch of PABLO_p27_wild_0_sj403_ID has just been released. | |
| ID: 49277 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Richard Haselgrove Send message Joined: 11 Jul 09 Posts: 1620 Credit: 8,866,392,698 RAC: 20,073,067 Level Scientific publications | |
- although as these are output files I guess it might be that this is a symptom of the failure rather than the cause. Yes, those are symptoms, not causes. If you follow through your account (name link at top of page) / computer / workunit / task, you should be able to see something like workunit 13443713 - that one was well worth aborting. And if you look at one of the errored tasks, the real cause: ERROR: file mdioload.cpp line 81: Unable to read bincoordfile That was the earlier batch. Today's are possibly similar, but we need to see one to be sure. Your computers are hidden, and we don't have the 'find task by name' tool here, so we'll have to ask you to look it up for use. Edit - thanks for the heads up, I've got one of those too. WU 13451812 is indeed the same as before, created 16 Apr 2018 | 11:22:43 UTC. That can go in the bit-bucket with the others. | |
| ID: 49278 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Richard Haselgrove Send message Joined: 11 Jul 09 Posts: 1620 Credit: 8,866,392,698 RAC: 20,073,067 Level Scientific publications | |
|
I've just been sent another from today's bad batch e174s49_e48s25p1f219-PABLO_p27_wild_0_sj403_IDP-0-2-RND8766. The files downloaded were 16/04/2018 14:16:44 | GPUGRID | Started download of e174s49_e48s25p1f219-PABLO_p27_wild_0_sj403_IDP-0-LICENSE For comparison, I'm working on an older one, resent this morning but created on 11 April - e82s5_e80s15p1f298-PABLO_p27_W60A_W76A_0_IDP-1-2-RND9196. Those files were called 16/04/2018 09:06:22 | GPUGRID | Started download of e82s5_e80s15p1f298-PABLO_p27_W60A_W76A_0_IDP-1-LICENSE Quite a difference. | |
| ID: 49279 | Rating: 0 | rate:
/ Reply
Quote
| |
|
robertmiles Send message Joined: 16 Apr 09 Posts: 503 Credit: 755,434,080 RAC: 186,180 Level  Scientific publications | |
|
Two of my recent PABLO tasks gave Error while computing, with this error message in the stderr file: | |
| ID: 49280 | Rating: 0 | rate:
/ Reply
Quote
| |
|
flashawk Send message Joined: 18 Jun 12 Posts: 297 Credit: 3,572,627,986 RAC: 0 Level Scientific publications | |
|
I just got 10 more bad Pablo's, that brings me to 34 total. The server is going to give me the boot for to many errors in such a short amount of time, I hope they figure this out soon. | |
| ID: 49281 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Richard Haselgrove Send message Joined: 11 Jul 09 Posts: 1620 Credit: 8,866,392,698 RAC: 20,073,067 Level Scientific publications | |
Richard, do you have any idea what's going on over at CPDN? Everything has been down for 2 weeks or so and I was curious when they might be back up, thanks. I received the same emails as have been quoted on the BOINC message board at CPDN project going offline this afternoon, but I've had no more specific news that that. Better to consolidate all the news that we do get in that thread, I think. | |
| ID: 49283 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Richard Haselgrove Send message Joined: 11 Jul 09 Posts: 1620 Credit: 8,866,392,698 RAC: 20,073,067 Level Scientific publications | |
Could you check if this is due to a missing file that should have been sent with the task? All the failed / failing tasks over the last four days have had the exact string PABLO_p27_wild in their name. I can see that I've completed at least one successfully with that string: also, at least one other with PABLO_p27_O43806_wild I'll go and search the message logs to see what I can find, but I think any completely missing files would show up as a problem at the download stage, and never get as far as attempting to run. I think it's more likely that the contents are badly formatted in some way, and it won't be possible to compare good and bad after the event. Edit - well, e173s16_e149s4p1f23-PABLO_p27_wild_0_sj403_IDP-1-2-RND2043 had file names with the workunit name embedded, like the second example in my comparison example earlier. I think that Pablo, or whoever is submitting the work on Pablo's behalf, might be using the wrong script/template when preparing the workunits. | |
| ID: 49284 | Rating: 0 | rate:
/ Reply
Quote
| |
|
mmonnin Send message Joined: 2 Jul 16 Posts: 337 Credit: 7,635,065,645 RAC: 10,764,900 Level Scientific publications | |
|
2nd batch of bad tasks today. 8 on the 13th and 4 more today. | |
| ID: 49286 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Tuna Ertemalp Send message Joined: 28 Mar 15 Posts: 46 Credit: 1,547,496,701 RAC: 0 Level  Scientific publications | |
|
So, one my one, my seven hosts will be jailed, wasting 12x 1080Ti and 2x TitanX... Something is very non-ideal with that picture. Given that this sort of bad batches are happening with some relevant non-ignorable frequency, there should be a way to unblock machines in bulk that were blocked/limited due to bad batch issues, methinks. | |
| ID: 49288 | Rating: 0 | rate:
/ Reply
Quote
| |
|
flashawk Send message Joined: 18 Jun 12 Posts: 297 Credit: 3,572,627,986 RAC: 0 Level Scientific publications | |
|
Why don't they know what's going on? Don't they monitor their project? | |
| ID: 49289 | Rating: 0 | rate:
/ Reply
Quote
| |
|
robertmiles Send message Joined: 16 Apr 09 Posts: 503 Credit: 755,434,080 RAC: 186,180 Level Scientific publications | |
Could you check if this is due to a missing file that should have been sent with the task? I'd expect the download stage to fail if the file was missing on the server, but only if the name of the file was included on the list of files sent with the task to tell the client what files to download before starting the task. If the name of the file was missing from that list, I'd expect download stage to download all the files on the list, report success for that stage, and the problem to become visible only when the application tries to open the file. | |
| ID: 49290 | Rating: 0 | rate:
/ Reply
Quote
| |
|
flashawk Send message Joined: 18 Jun 12 Posts: 297 Credit: 3,572,627,986 RAC: 0 Level Scientific publications | |
|
Why are these work units still in the que? Is anyone running this program? Where are the moderators at? This place is like an airplane on autopilot, it seems like some of these projects have no more enthusiasm. | |
| ID: 49291 | Rating: 0 | rate:
/ Reply
Quote
| |
|
flashawk Send message Joined: 18 Jun 12 Posts: 297 Credit: 3,572,627,986 RAC: 0 Level Scientific publications | |
|
Double post | |
| ID: 49292 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Betting Slip Send message Joined: 5 Jan 09 Posts: 670 Credit: 2,498,095,550 RAC: 0 Level Scientific publications | |
Why are these work units still in the que? Is anyone running this program? Where are the moderators at? This place is like an airplane on autopilot, it seems like some of these projects have no more enthusiasm. A similar post of mine about this projects participation with its contributors. http://www.gpugrid.net/forum_thread.php?id=4585#47369 another one, http://www.gpugrid.net/forum_thread.php?id=4368&nowrap=true#48039 | |
| ID: 49293 | Rating: 0 | rate:
/ Reply
Quote
| |
|
flashawk Send message Joined: 18 Jun 12 Posts: 297 Credit: 3,572,627,986 RAC: 0 Level Scientific publications | |
|
I'm under the impression they think were all getting paid for crunching, I will never do data mining, ever!! I'll bet a lot of the dedicated crunchers that have been here for a while are now miners for hire. | |
| ID: 49294 | Rating: 0 | rate:
/ Reply
Quote
| |
Richard, thank you for your post. I have received a single further failing task this morning and following the link as you advise it seems to have originated in yesterday's bad batch at 11:17 (UTC). I notice that by following the links for all ten of my failed tasks, seven arising from the 13 April bad batch (around 17:50 UTC) and three from the 16 April bad Batch (around 11:20 UTC), they now all show exactly eight 'Error while Computing' failures so perhaps there is some automatic mechanism whereby they are automatically pulled after eight failures on different computers? I also notice on the Server Status Page that PABLO_p27_wild_0_sj403_ID currently shows 740 successes and a 92.33% error rate which, if my maths is correct, suggests over 9600 failures, potentially resulting in a large number of computers having been temporarily locked out. Frustrating though that is for we donors it doesn't appear to have created a (GPU) processing backlog and perhaps is an indication that the processing resource offered by donors currently far exceeds the requirements of the available work.[/url] | |
| ID: 49295 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Tuna Ertemalp Send message Joined: 28 Mar 15 Posts: 46 Credit: 1,547,496,701 RAC: 0 Level Scientific publications | |
So, one my one, my seven hosts will be jailed, wasting 12x 1080Ti and 2x TitanX... Something is very non-ideal with that picture. Given that this sort of bad batches are happening with some relevant non-ignorable frequency, there should be a way to unblock machines in bulk that were blocked/limited due to bad batch issues, methinks. Case in point: When my hosts are fully utilized, I would see "State: In progress (28)" under my account's Tasks page (I have a custom config file that tells BOINC that GPUGRID tasks are each 1 CPU + 0.5 GPU, and that works well for these cards, I found, so 14 cards = 28 tasks). Last night I saw it at 26, then 22, went to sleep, then this morning at 14, now it is 12. For instance, when one of my single TitanX machines (http://www.gpugrid.net/results.php?hostid=205349) that has NOTHING ELSE going on in BOINC contacts GPUGRID, it gets: 4/17/2018 8:23:54 AM | GPUGRID | Sending scheduler request: To fetch work. 4/17/2018 8:23:54 AM | GPUGRID | Requesting new tasks for CPU and NVIDIA GPU 4/17/2018 8:23:56 AM | GPUGRID | Scheduler request completed: got 0 new tasks 4/17/2018 8:23:56 AM | GPUGRID | No tasks sent Quite ironic when the Server Status says "Tasks ready to send 34,375"... :( | |
| ID: 49299 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Richard Haselgrove Send message Joined: 11 Jul 09 Posts: 1620 Credit: 8,866,392,698 RAC: 20,073,067 Level Scientific publications | |
I notice that by following the links for all ten of my failed tasks, seven arising from the 13 April bad batch (around 17:50 UTC) and three from the 16 April bad Batch (around 11:20 UTC), they now all show exactly eight 'Error while Computing' failures so perhaps there is some automatic mechanism whereby they are automatically pulled after eight failures on different computers? Yes, on the workunit page, you should see a red banner above the task list saying Too many errors (may have bug) Once that appears (at this project, after 8 failures), no more are sent out. | |
| ID: 49300 | Rating: 0 | rate:
/ Reply
Quote
| |
|
flashawk Send message Joined: 18 Jun 12 Posts: 297 Credit: 3,572,627,986 RAC: 0 Level Scientific publications | |
Quite ironic when the Server Status says "Tasks ready to send 34,375"... Those are Quantum Chemistery WU's for your CPU. ____________  | |
| ID: 49301 | Rating: 0 | rate:
/ Reply
Quote
| |
Hi Tuna, The 'Tasks ready to send' figure on the Server Status page is the total of all task types ready to send. There is a table beneath this showing a breakdown of 'Tasks by application' (although for some reason the totals always differ by one). You should see that almost all, if not all, of the unsent tasks are currently Quantum Chemistry tasks which do not run on GPUs or Windows; they run on Linux CPUs only. The Unsent Short runs and Unsent Long Runs figures show the work available for GPUs. I cannot remember the exact wording, but when my own PC was temporarily jailed, requests for new work in the event log then reported the reason that the daily quota (3) had been exceeded - as the log you have posted above does not say this I suspect you might not be locked out and the reason you are not receiving tasks is simply that currently, most of the time, there are no GPU tasks available to send. As I understand it, tasks are only sent in response to requests from the client so it is down a matter of luck as to whether tasks are available when your PC makes its requests. | |
| ID: 49302 | Rating: 0 | rate:
/ Reply
Quote
| |
|
mmonnin Send message Joined: 2 Jul 16 Posts: 337 Credit: 7,635,065,645 RAC: 10,764,900 Level Scientific publications | |
|
I had 8 fail on the 13th at 18:38:35 UTC and received 4 more on the 15th at 6:17:47 UTC. So if they were jailed it was less than 2 days. | |
| ID: 49303 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Tuna Ertemalp Send message Joined: 28 Mar 15 Posts: 46 Credit: 1,547,496,701 RAC: 0 Level Scientific publications | |
|
Ooops. Yup. I didn't scroll down far enough, I guess... :) | |
| ID: 49304 | Rating: 0 | rate:
/ Reply
Quote
| |
I had 8 fail on the 13th at 18:38:35 UTC and received 4 more on the 15th at 6:17:47 UTC. So if they were jailed it was less than 2 days. Yes, as in these circumstances the event log refers to a daily quota being exceeded, I guess the lockout only lasts for a day or the remaining part of a day. | |
| ID: 49307 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Erich56 Send message Joined: 1 Jan 15 Posts: 1132 Credit: 10,256,332,676 RAC: 29,209,412 Level Scientific publications | |
|
Richard wrote on April 16th: All the failed / failing tasks over the last four days have had the exact string These faulty task are still in the queue; I got such one this morning: http://gpugrid.net/result.php?resultid=17472190 again, as before: ERROR: file mdioload.cpp line 81: Unable to read bincoordfile | |
| ID: 49320 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Erich56 Send message Joined: 1 Jan 15 Posts: 1132 Credit: 10,256,332,676 RAC: 29,209,412 Level Scientific publications | |
|
just now, the next faulty PABLO_927_wild task :-((( | |
| ID: 49321 | Rating: 0 | rate:
/ Reply
Quote
| |
Hi Erich, As Richard confirmed earlier in the thread there is a mechanism whereby tasks are automatically withdrawn after eight failures and it seems this is being relied on to remove the faulty batches (for example http://www.gpugrid.net/workunit.php?wuid=13443881). If you look at the history of the Work Unit that you picked up http://gpugrid.net/workunit.php?wuid=13443681 you can see that it originated in the bad batch released on 13 April. After it failed for you, it was reissued to flashawk (who aborted it) and is currently reissued to an anonymous user but still requires three further failures to trigger automatic removal. I guess that because there is currenlty so little GPU work available these remaining bad units are still a significant proportion of available work and agree that it is disappointing, and seems a little disrespectful to donors, that no action has been taken to remove them proactively. | |
| ID: 49323 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Erich56 Send message Joined: 1 Jan 15 Posts: 1132 Credit: 10,256,332,676 RAC: 29,209,412 Level Scientific publications | |
|
@STFC9F22, thank you for your explanations and insights. | |
| ID: 49325 | Rating: 0 | rate:
/ Reply
Quote
| |
|
flashawk Send message Joined: 18 Jun 12 Posts: 297 Credit: 3,572,627,986 RAC: 0 Level Scientific publications | |
Hi Erich, Most of us are and have been aware of this for sometime. The problem is when a computer gets to many errors it's put on a black list for a time, I know right after I get an error I can't download any WU's for a time. We should'nt be taking these hits. It's as though he loaded up these last WU's, packed his bags and left. | |
| ID: 49327 | Rating: 0 | rate:
/ Reply
Quote
| |
|
PappaLitto Send message Joined: 21 Mar 16 Posts: 511 Credit: 4,672,242,755 RAC: 0 Level Scientific publications | |
It's as though he loaded up these last WU's, packed his bags and left. You know he might be on vacation. Scientists have real lives too. | |
| ID: 49328 | Rating: 0 | rate:
/ Reply
Quote
| |
|
robertmiles Send message Joined: 16 Apr 09 Posts: 503 Credit: 755,434,080 RAC: 186,180 Level Scientific publications | |
Hi Erich, When the tasks are withdrawn after eight failures, they should also no longer be counted as failures for the eight computers that ran them. While this is fixed, compute errors in 2015 and earlier years should be removed from the lists of failures for computers, even for workunits that never had a successful task. | |
| ID: 49329 | Rating: 0 | rate:
/ Reply
Quote
| |
|
flashawk Send message Joined: 18 Jun 12 Posts: 297 Credit: 3,572,627,986 RAC: 0 Level Scientific publications | |
|
I have 8 failed WU's from 2013 through 2014 that won't go away, how do I get those removed? I've asked twice and got no response, are their mods here anymore? | |
| ID: 49331 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Erich56 Send message Joined: 1 Jan 15 Posts: 1132 Credit: 10,256,332,676 RAC: 29,209,412 Level Scientific publications | |
|
I just notice that many more of these faulty PABLO_P27_wild tasks are distributed, although they have an error rate of 88% (which means that nearly 9 out of 10 are bad). | |
| ID: 49345 | Rating: 0 | rate:
/ Reply
Quote
| |
|
flashawk Send message Joined: 18 Jun 12 Posts: 297 Credit: 3,572,627,986 RAC: 0 Level Scientific publications | |
I just notice that many more of these faulty PABLO_P27_wild tasks are distributed, although they have an error rate of 88% (which means that nearly 9 out of 10 are bad). I have 4 over 50% right now and they seem fine, they may have been reworked. | |
| ID: 49346 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Erich56 Send message Joined: 1 Jan 15 Posts: 1132 Credit: 10,256,332,676 RAC: 29,209,412 Level Scientific publications | |
...they may have been reworked. maybe so; I got 2 of them within the past 12 hours, and both of them did NOT fail so far :-) | |
| ID: 49347 | Rating: 0 | rate:
/ Reply
Quote
| |
Message boards : Number crunching : Bad batch of WU's
