Hi all,

I have submitted some new work units that will replace some I submitted earlier in the week. The names will be "NATHAN_FAX3". These tasks are in the true spirit of the long queue, and will take about 12+ hours on the fastest cards. Some have already been returned and indeed have been around 13 hours. This is markedly longer than what you have expected traditionally, but we really want the long queue to be for critical tasks, computationally intensive tasks, and the like. I suggest you all take note of how these tasks run on your computers and be mindful of temps and errors as you start to receive them.

I have noticed some crunchers expressing concern/dismay that perhaps they will not be able to get the 24h bonus with such long tasks. We are mindful of that concern, and will keep an eye on this group as an experiment. If we think it is too unfair to people with fast but not the fastest cards, we'll be sure to correct that in future groups. But the less send/recieve we have to do, the better. We are also mindful of the fact that longer tasks might be more susceptible to errors/crashes, and we want to see how this goes. I'll be looking out for the severe error percentage over the next few days for any problems.

Also, a note about tasks beginning with NATHAN_FA... These tasks are unique in that they are quite large simulations, compared to many others we have done in the past which are smaller (bigger biomolecules mean bigger simulations). They not only take longer per step, but require more memory. Cards with lower memory (below 1GB) may suffer additional performance loss. There is nothing we can do about this, unfortunately.

Happy crunching.

Nate

Thanks for the info.

This is markedly longer than what you have expected traditionally, but we really want the long queue to be for critical tasks, computationally intensive tasks, and the like.

but we really want the long queue to be for critical tasks

Regarding the 24 hour limit, I also am concerned about this because I have been experiencing a problem with the connection on upload which is apparently unrelated to either my system or the servers for GPUGRID.

When uploading a completed work unit, the upload invariably fails on one file. If there are successive failures, BOINC will retry, and BOINC adds time to the retry delay each time so that in the worst case, BOINC might not retry for up to 8 hours or more. There is, apparently, nothing that I or GPUGRID can do about this, and if the worst should happen, then it appears that my machine has not returned the work unit within the 24 hour limit - when it has, in fact, completed the work unit and made best attempts to return it.

Personally, I think the 24 hour limit is completely unfair, especially in cases like this where my machine and, indeed, I, have done everything humanly possible to return the work unit.

I understand what you are trying to accomplish by the 24 hour limit, however, I do not agree with the implementation.
____________

Man, I thought I was going crazy when I saw this task running for over 15 hours, before I read this post. I thought my card had downclocked or something, but windows wasn't reporting anything unusual. I just let it run and it finished fine, in about 20 hours on my overclocked GTX570.

I did receive consistent credit with other "NATHAN_FAX" tasks -- the whopper was 114,000 in credit.

If I had a less of a card, the 24 hour bonus would have been in jeopardy. I would prefer if the tasks ran in the 8-12 hour window, but one of these occasionally doesn't matter (as long as it doesn't fail after 95%).

They not only take longer per step, but require more memory. Cards with lower memory (below 1GB) may suffer additional performance loss. There is nothing we can do about this, unfortunately.

Man, I thought I was going crazy when I saw this task running for over 15 hours, before I read this post. I thought my card had downclocked or something, but windows wasn't reporting anything unusual. I just let it run and it finished fine, in about 20 hours on my overclocked GTX570.

I got one of them finshed too. I was only wondering why it gives only 30000 credits for 35000secs (FAX) instead the (CB1) nearly 25000sec for 35000 credits ^^

http://www.gpugrid.net/workunit.php?wuid=3247302

Now i got a second one. i will look how this one is going ^^



I think on a 560TI it would take ~12,5 Hours, the 285 took little bit under 12 Hours.
____________
DSKAG Austria Research Team: http://www.research.dskag.at

Man, I thought I was going crazy when I saw this task running for over 15 hours, before I read this post. I thought my card had downclocked or something, but windows wasn't reporting anything unusual. I just let it run and it finished fine, in about 20 hours on my overclocked GTX570.

I thought about aborting one myself. Luckily when I saw one it only had a hour left so I let it finish. These are running at around 20 hours on a stock clocked GTX470 in Linux. Unfortunately with a 115 GB upload I cannot get them turned in fast enough.

I have my cache set to 0.01 but they still download a few hours before they start so I run out of time.

How much is being used? 1243MB of my GTX470's 1280MB is being used - which explains the higher temps. ~450MB is more common for GPUGrid tasks. I expect W7 may be further restricted by this, as the operating system grabs some GDDR for itself.

How much is being used? 1243MB of my GTX470's 1280MB is being used - which explains the higher temps. ~450MB is more common for GPUGrid tasks. I expect W7 may be further restricted by this, as the operating system grabs some GDDR for itself.

My current memory usage is (running NATAHAN_FAX3 task, and one NATHAN_CB1 task) ~700Mb=23% @71 Celcius &
~610Mb = 20% @65C.

Again, this seems like a curious waste of RAM.

I got one of them finshed too. I was only wondering why it gives only 30000 credits for 35000secs (FAX) instead the (CB1) nearly 25000sec for 35000 credits ^^

http://www.gpugrid.net/workunit.php?wuid=3247302

Now i got a second one. i will look how this one is going ^^



I think on a 560TI it would take ~12,5 Hours, the 285 took little bit under 12 Hours.

I got one of them finshed too. I was only wondering why it gives only 30000 credits for 35000secs (FAX) instead the (CB1) nearly 25000sec for 35000 credits ^^

http://www.gpugrid.net/workunit.php?wuid=3247302

Now i got a second one. i will look how this one is going ^^



I think on a 560TI it would take ~12,5 Hours, the 285 took little bit under 12 Hours.

Well that is significantly better than what my 560ti is doing.

currently running http://www.gpugrid.net/result.php?resultid=5098225

overclocked to 950/1900/2007@ 1.025v and it's showing 13:21:xx time used with 10:40:xx to go. So that is going to be just over the 24hrs, NOT including upload time. how big are the upload files on these anyway?

additional information for those interested. It's a 2gb model currently using 1240 mb VRAM and 263Mb system ram.
4.5Ghz i7-2600k

Simba: Finally, runned a real FAX3 unit now. 30-31Hours runtime (285) with additional 12h uploadtime ^^ So your not alone with the 560TI ^^ (25hours is not bad for this card!) I will not run any FAX3 WU on my 560TI cos it computes to long. I often run the GamingPC over Night for 2 or 3 WUs after i was gaming. But now i dont continue that, cos i could get a FAX3 while i´m in sleep and want to turn the PC off in morning, not waiting aditional 20hours or so ^^

We still get bonustime for report it under 48hours, but not as much as under 24hours.
____________
DSKAG Austria Research Team: http://www.research.dskag.at

Sorry for the delayed response, I have been at a conference for the past two days. Considering the responses and the stats we have gotten, it definitely seems that these are a little too long. There are a lot of people who complete the tasks on time but then have trouble uploading them in time for the bonuses at both the 24 and 48 hr limits (116mb is a large upload size). More importantly, there also seems to be more severe errors than are typical, though it's not clear why that is right now.

Therefore, I am going to modify and rename these tasks. They will be renamed to NATHAN_FAX4 (surprise!), and they will run for ~66% of the time of the FAX3 jobs. My checks here indicate that they will run for 8.3 hours on GTX 580, so remember that they are still not like short tasks. People who are not using the GTX 500 or 400 series cards should consider crunching only on the short queue if you are concerned about runtime, bonuses, failures, overheating, etc. Ultimately, the choice is yours. There is still a 25% bonus for finishing before 48 hours.

Hopefully this hits the sweet spot we and you guys have been looking for. This is a learning moment for us and it has been good to get some data on the behavior of such long tasks.

Nate

Wiyosaya: Definitely an unfortunate situation. I'm not sure what we can do on our end, since we can't change how BOINC works. I don't think we can do anything, but will ask.

Michael Kingsford Gray: We can't choose which computers get which jobs. The reason is that the BOINC system is not set up to allow that, and we can't modify it to do that. We can make different queues, as we already have three (ACEMD2 aka short, ACEMDlong, and ACEMDbeta), but making too many queues makes it confusing for crunchers and also increasingly complicated for us. The only people that could fix it to a better system are the people who maintain the BOINC software.

dskagcommunity et al.: I should clear up confusion about the different "FAX" named tasks. The first round of tasks, titled "NATHAN_FAX-" (with no number after) were supposed to be long like the FAX3 tasks. However, there was a problem with our test software and it underestimated runtimes (but correctly calculated credit). Those have been replaced by the FAX3 tasks. There were "FAX2" tasks, but they were stopped before anyone recieved them because the credit was incorrect by a huge factor (a human mistake, I failed to edit a line in the submission to the server). As you know the "FAX3" tasks are the very long tasks, which have proven to be a bit too long for everyone's taste, so I will modify them to run for shorter times as I explained above. I will rename them to "FAX4" in order to make the distinction clear. Sorry for all the different task names, but I need to be sure I can keep track of the simulations, and what data comes from where.

Personaly i think 66% is a real fair alternative :)
____________
DSKAG Austria Research Team: http://www.research.dskag.at

Hopefully this hits the sweet spot we and you guys have been looking for. This is a learning moment for us and it has been good to get some data on the behavior of such long tasks.

I'm just starting on a FAX4 and while my gpu idled for few seconds I saw in GPU-Z that the memory used was 150mb for Win7.
Now with the card loaded with FAX4 it's at 1232mb so that's a 1082megabytes usage for FAX4. Memory controller load is at 28-29% with 800mhz memory clock (gpu-z reading)

One thing I do NOT like is the gpu load. With other long runs and other projects the gpu load is anything between 94-99%. Now it's at 86-88% ?!
CPU is 2500K@4500Mhz
I have swan_sync=0 and only three cpu tasks running along with gpugrid.
Do I really have to put my cpu at higher clocks or what?
This is the task http://www.gpugrid.net/workunit.php?wuid=3264669
E: Well I bumped the clocks to 4800Mhz and it really didn't affect the gpu load %, still around 87-88%

I'm just starting on a FAX4 and while my gpu idled for few seconds I saw in GPU-Z that the memory used was 150mb for Win7.
Now with the card loaded with FAX4 it's at 1232mb so that's a 1082megabytes usage for FAX4. Memory controller load is at 28-29% with 800mhz memory clock (gpu-z reading)

One thing I do NOT like is the gpu load. With other long runs and other projects the gpu load is anything between 94-99%. Now it's at 86-88% ?!
CPU is 2500K@4500Mhz
I have swan_sync=0 and only three cpu tasks running along with gpugrid.
Do I really have to put my cpu at higher clocks or what?
This is the task http://www.gpugrid.net/workunit.php?wuid=3264669

You seen to have found the happy medium (sweet spot) nate.

I am crunching a FAX4 wu on Win7 i7 990X @ 3876MHz and a GTX570, similar GPU load as Lagittaja 86-88%, memory controller load quite a bit less @ 16% and only using 690MB memory. I have swan_sync=0 and only two cpu tasks running along with gpugrid.

It is looking like approx 13hrs 40mins to complete.

Does the usages seem right to you or is there a way to improve on this?

I'm just starting on a FAX4 and while my gpu idled for few seconds I saw in GPU-Z that the memory used was 150mb for Win7.
Now with the card loaded with FAX4 it's at 1232mb so that's a 1082megabytes usage for FAX4. Memory controller load is at 28-29% with 800mhz memory clock (gpu-z reading)

One thing I do NOT like is the gpu load. With other long runs and other projects the gpu load is anything between 94-99%. Now it's at 86-88% ?!
CPU is 2500K@4500Mhz
I have swan_sync=0 and only three cpu tasks running along with gpugrid.
Do I really have to put my cpu at higher clocks or what?
This is the task http://www.gpugrid.net/workunit.php?wuid=3264669

That's a pretty solid overclock already, so I wouldn't push it unless you really want to. Or if you are using LN2 to cool your cores ;).

These tasks seem to utilize less percent of the GPU than others, and I'm not sure why that is at the moment. I'm not sure if it's an issue with dividing the task between the CPU and GPU or something else (memory management, maybe?). I'll try to get back to you. There may not be anything we or you can do. We'll see.

Thanks Nate :)
those changes should make things a little easier and more consistent.

As for the GPU usage, anything above 90% is good IMHO. 95% would be excellent.

I have no idea about the coding used, but it appears to be adaptable to the workunit on the fly. As in, usage of the GPU can change throughout the run. I too have noticed that usage can change from 80-88% during the run.
Also noticed that the time taken to complete a % can change throughout the run as well.
My last unit took significantly more time to process the last 25% of the unit than it did to do the other 75%(when observed in 25% blocks)

I think 12 hours or so is an optimum run-time. That lets us run machines overnight and complete a task so we can turn them off before we go to work. Also they are not long enough that it is too annoying if a unit fails close to the end. You would have heard my scream from Australia had that unit of mine crashed at 22+ hours of runtime :/

just lost one after 10 hours running
aaarrrrrggggghhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh!
well, no harm done
and it was my falt (messing with overclocking and knowing nothing about that, just going to return it to default settings and leave it like that)
____________

That's around half what I expected (1420MB).
...So is it the operating system, driver, or app?

Spatzthecat: That does seem on the slow side for 570. For these tasks 10-11 hours seems about average (for that card). Windows has slightly lower performance than Linux, so that might explain part of it. Further, looks like you are running on your display card, which will also take away some performance if you are using the system to do other stuff. I'm not sure about other factors.

Lagittaja: Thanks for the info. Still looking into it, but it seems that a script added to the simulation may be the cause. Some simulations don't have them, but this one required it. We try to optimize the scripts as much as possible but they are interpreted at runtime and cause slowdowns. Many functions, but not all, are implemented in C/C++ to help out with speed.

Simba123:

I think 12 hours or so is an optimum run-time. That lets us run machines overnight and complete a task so we can turn them off before we go to work. Also they are not long enough that it is too annoying if a unit fails close to the end.

My stock C2Q/GTX570 finished this FAX4 in about 14.5 hours with SWAN_SYNC=0 and 1 dedicated CPU. Almost 3 times as long to finish as a typical NATHAN_CB1 task for only twice the credit? :-( [ ;-)]
____________

I'm leaving this project. My card could handle those CB1 ones. They took about 12/13 hrs (my limit of computer use) on my 460SE card.

But these FAX things are too long for me and because of the time limitation I also ran out of any bonus points with these wu's. I am not interested in your short stuff.

For the fast cards and 24/7 people, 'good luck' with these FAX batch(es.) Its time to change to a other project. Bye.

I'm leaving this project. My card could handle those CB1 ones. They took about 12/13 hrs (my limit of computer use) on my 460SE card.

But these FAX things are too long for me and because of the time limitation I also ran out of any bonus points with these wu's. I am not interested in your short stuff.

For the fast cards and 24/7 people, 'good luck' with these FAX batch(es.) Its time to change to a other project. Bye.

I'm leaving this project. My card could handle those CB1 ones. They took about 12/13 hrs (my limit of computer use) on my 460SE card.

But these FAX things are too long for me and because of the time limitation I also ran out of any bonus points with these wu's. I am not interested in your short stuff.

For the fast cards and 24/7 people, 'good luck' with these FAX batch(es.) Its time to change to a other project. Bye.

It appears that both the FAX3 and FAX4 tasks have exhausted, potentially rendering nagging gripes moot.

Still getting FAX4's & the occasional FAX3 here.

I'm new and I've got 53 hours to do and it's also not crunching but saying the computer is in use I have an i5 processor and a Cuda GEFORCE 410 mobile and latest drivers, any suggestions?

Bruce

I've gone on to join World Community grid.

I'm new and I've got 53 hours to do and it's also not crunching but saying the computer is in use I have an i5 processor and a Cuda GEFORCE 410 mobile and latest drivers, any suggestions?

I've gone on to join World Community grid.

ouch.
just picked up
http://www.gpugrid.net/result.php?resultid=5123672
and it's showing a time to complete of over 29 hours. :(

I'll give it an hour to see if that time comes down to below 24 hours.
If it doesn't, I'll abort it.

Just finished an FAX4 on a GTX 460 1GB. Took 28 hours, and about 45-mins of pressing the retry button to get the WU to upload completely.

http://www.gpugrid.net/workunit.php?wuid=3278935

____________

Just finished an FAX4 on a GTX 460 1GB. Took 28 hours, and about 45-mins of pressing the retry button to get the WU to upload completely.

http://www.gpugrid.net/workunit.php?wuid=3278935

I'm new and I've got 53 hours to do and it's also not crunching but saying the computer is in use I have an i5 processor and a Cuda GEFORCE 410 mobile and latest drivers, any suggestions?

Your GeForce 410 mobile is too slow for this project.

I've gone on to join World Community grid.

So you figure it out by yourself. At WCG your GPU will be useful.

just completed 2 'Paola' tasks which I have not seen before.
they were biggies!
http://www.gpugrid.net/result.php?resultid=5415785

http://www.gpugrid.net/result.php?resultid=5410160

ran consistenly at 90% GPU load. didn't check memory usage on them though.

this is on a 560ti 2gb @ 925

so current task
http://www.gpugrid.net/result.php?resultid=5418278

is using



currently showing a completion time of 19:04:00

bit longer than I would like, but it may come down in a few hours

Hi all,

I have submitted some new work units that will replace some I submitted earlier in the week. The names will be "NATHAN_FAX3". These tasks are in the true spirit of the long queue, and will take about 12+ hours on the fastest cards. Some have already been returned and indeed have been around 13 hours. This is markedly longer than what you have expected traditionally, but we really want the long queue to be for critical tasks, computationally intensive tasks, and the like. I suggest you all take note of how these tasks run on your computers and be mindful of temps and errors as you start to receive them.

I have noticed some crunchers expressing concern/dismay that perhaps they will not be able to get the 24h bonus with such long tasks. We are mindful of that concern, and will keep an eye on this group as an experiment. If we think it is too unfair to people with fast but not the fastest cards, we'll be sure to correct that in future groups. But the less send/recieve we have to do, the better. We are also mindful of the fact that longer tasks might be more susceptible to errors/crashes, and we want to see how this goes. I'll be looking out for the severe error percentage over the next few days for any problems.

Also, a note about tasks beginning with NATHAN_FA... These tasks are unique in that they are quite large simulations, compared to many others we have done in the past which are smaller (bigger biomolecules mean bigger simulations). They not only take longer per step, but require more memory. Cards with lower memory (below 1GB) may suffer additional performance loss. There is nothing we can do about this, unfortunately.

Happy crunching.

Nate

I really wish you wouldn't suck the last drop of blood out of my cards, they actually do have another purpose!!!

Not everyone can have dedicated machines or the latest and greatest cards for your benefit.

My problem with these units isn't how long they take but the amount of resources they use while running.

Or do you mean reduce resource demands to 66%?


____________
Radio Caroline, the world's most famous offshore pirate radio station.
Great music since April 1964. Support Radio Caroline Team -
Radio Caroline

This area has been a bone of contention for years. It's about choice for the cruncher vs project management and overall project performance for the scientists. GPUGrid is an ever changing project. New faces, new research, new apps, new cards..., new problems and re-emerging old ones.

Firstly, it's fantastic that Nate has managed to utilize larger memory resources; most CC2.0 cards have >1GB GDDR5 memory, just waiting there to be used. In doing so Nate has expanded GPUGrid's research boundaries. This is a very important achievement in itself.

Perhaps in this case some sort of opt out is worth considering?
Crunchers could chose to opt out of a project if it requires more resources (GPU memory) than they have, if not having more memory actually slows performance significantly on such cards (512MB). The alternatives are crunchers choosing which tasks to crunch (a lot of work for the team) or aborting such tasks (bad for research). Of course this impacts on the recognition system too; if you don't crunch for a project you don't get recognition (a badge and links to that research), hence the opt out rather than an opt in suggestion.
It's also worth noting that the GTX460 (and similar CC2.1 cards) are not high end cards, and while they can complete long tasks in a reasonable length of time (<2days), they are perhaps more suited to the normal length tasks.
It might also be worth waiting and seeing how the CUDA4.0 or CUDA4.2 apps perform (if there will be a 15% performance boost for Nate's tasks, and if this will allow them to finish significantly faster, or not).

Thanks for your opinions on such matters,

I'm another that runs "not high end" cards, a GTX460 and a GTX550Ti and much prefer running long tasks.....I can live with the fact my 550 misses out on the 24Hr bonus most of the time, and on only a couple of occasions I have missed out on the full bonus the the 460.... once by a matter of seconds cos of the upload speed :-(

I am quite proud of the fact that my low cost setup has me sitting in the top 150 RAC wise

This area has been a bone of contention for years. It's about choice for the cruncher vs project management and overall project performance for the scientists. GPUGrid is an ever changing project. New faces, new research, new apps, new cards..., new problems and re-emerging old ones.

Firstly, it's fantastic that Nate has managed to utilize larger memory resources; most CC2.0 cards have >1GB GDDR5 memory, just waiting there to be used. In doing so Nate has expanded GPUGrid's research boundaries. This is a very important achievement in itself.

Perhaps in this case some sort of opt out is worth considering?
Crunchers could chose to opt out of a project if it requires more resources (GPU memory) than they have, if not having more memory actually slows performance significantly on such cards (512MB). The alternatives are crunchers choosing which tasks to crunch (a lot of work for the team) or aborting such tasks (bad for research). Of course this impacts on the recognition system too; if you don't crunch for a project you don't get recognition (a badge and links to that research), hence the opt out rather than an opt in suggestion.
It's also worth noting that the GTX460 (and similar CC2.1 cards) are not high end cards, and while they can complete long tasks in a reasonable length of time (<2days), they are perhaps more suited to the normal length tasks.
It might also be worth waiting and seeing how the CUDA4.0 or CUDA4.2 apps perform (if there will be a 15% performance boost for Nate's tasks, and if this will allow them to finish significantly faster, or not).

Thanks for your opinions on such matters,

I don't know if that was meant for my previous post or not... But good explanation of things I was not aware of :)
Still, Long projects regardless of GPUGrid or elsewhere, have always been (in my experience) more resource intensive. That being said, after replacing my GPUGrid system to have both GPU's 570's as opposed to 460 and 570 have made it possible to play World of warcraft while Both crunching long GPUGrid WU's and 12 cores of WCG without issue. Sure it slows the projects down a little but there is not a constantly noticeable "lag" while gaming on my 990X. And no failures of WU's since. This is the reason that I doubt the "lag" in computer use for simple surfing and such.
Just my experience. OBTW, did the same with 570 and 460 (768) minus the 570 project startup fails (not the 460)... Gaming or not.