Advanced search

Message boards : Number crunching : Anyone using Amazon to crunch?

Author Message
Profile MJH
Project administrator
Project developer
Project scientist
Send message
Joined: 12 Nov 07
Posts: 696
Credit: 27,266,655
RAC: 0
Level
Val
Scientific publications
watwat
Message 37852 - Posted: 6 Sep 2014 | 13:01:17 UTC

Hi

Anyone here using Amazon web services / EC2 for crunching (GPUGrid or otherwise)?
Anyone keen to try?

Matt

biodoc
Send message
Joined: 26 Aug 08
Posts: 160
Credit: 1,405,920,847
RAC: 437
Level
Met
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 37856 - Posted: 7 Sep 2014 | 10:11:34 UTC - in response to Message 37852.
Last modified: 7 Sep 2014 | 10:25:55 UTC

I looked into it briefly.

https://aws.amazon.com/ec2/

"AWS Free Tier includes 750 hours of Linux and Windows t2.micro instances each month for one year. To stay within the Free Tier, use only EC2 Micro instances."

"micro instances" don't seem suitable for Boinc.

Have you used this service?

EDIT: From the PrimeGrid forum: http://www.primegrid.com/forum_thread.php?id=3551&nowrap=true#39254

Profile MJH
Project administrator
Project developer
Project scientist
Send message
Joined: 12 Nov 07
Posts: 696
Credit: 27,266,655
RAC: 0
Level
Val
Scientific publications
watwat
Message 37865 - Posted: 9 Sep 2014 | 12:08:25 UTC - in response to Message 37856.

To put it into more quantitative terms, the g2.2 instance-type with a GPU about as fast as a GTX760 and an 8-core 2.6GHz E5-2670 Xeon can cost as little as $1.60/day.

popandbob
Send message
Joined: 18 Jul 07
Posts: 65
Credit: 10,972,900
RAC: 0
Level
Pro
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 37870 - Posted: 10 Sep 2014 | 0:58:32 UTC

For the price of the GPU EC2 for 1 year I could build and power a new system for over a year.... Is it really worth it?

mikey
Send message
Joined: 2 Jan 09
Posts: 278
Credit: 453,901,190
RAC: 472,654
Level
Gln
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwat
Message 37873 - Posted: 10 Sep 2014 | 11:55:33 UTC - in response to Message 37870.

For the price of the GPU EC2 for 1 year I could build and power a new system for over a year.... Is it really worth it?


That sorta depends on your other associated costs, repairs cost time and energy, presumably Amazon would take care of those. Electricity costs different amounts in different areas of the World, again presumably Amazon would take care of this too. OS's cost money, again Amazon's costs. I am NOT saying I am going to jump on it, I won't be, but for some it could be a wash, or pretty close to it, as far as overall costs go. I know some projects use something similar so they don't have to pay for drive replacements, maintenance, electricity, cooling, OS etc.

captainjack
Send message
Joined: 9 May 13
Posts: 157
Credit: 1,174,156,933
RAC: 212,559
Level
Met
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwat
Message 41371 - Posted: 19 Jun 2015 | 21:08:50 UTC

Now that the summer heat is upon us here in the Northern hemisphere, I decided to give Amazon Web Services a try. My air conditioner has a hard time keeping up when running multiple GPU's at full tilt. For the next few months, I plan to rent from AWS and keep the house cool.

biodoc pondered

"micro instances" don't seem suitable for Boinc


A micro instance is a single core virtual server running at a modest speed. I started out with one to get some experience and it fine for CPU tasks (as long as it's free). It is still running and crunching away on some WCG tasks.

It took me several days to figure out how to get the correct NVIDIA drivers installed on a virtual server instance and get them set up where BOINC would recognize them. My first GPU task just finished and validated about an hour ago. Task number 2 is underway now. If anybody wants to know what it takes, let me know and I will try to compile all my notes and post them here.

The cost is a bit different now from when Matt originally asked for interest. Spot prices for a GPU instance are running about 10 cents per hour which would be about $2.40 U.S. per day. The beauty about spot instances is that you only pay the going rate and you can specify the maximum hourly rate that you are willing to pay. If the going hourly rate exceeds your maximum, your instance will be shut down until the going rate gets back below your maximum rate. Also there is a charge for file I/O and for internet I/O. I'm waiting to find out how much it cost to download GPUGRID tasks and upload the results. After a few days, I should have a better idea.

Let me know if you have other questions.

Profile skgiven
Volunteer moderator
Project tester
Volunteer tester
Avatar
Send message
Joined: 23 Apr 09
Posts: 3968
Credit: 1,991,617,060
RAC: 146,649
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 41388 - Posted: 23 Jun 2015 | 15:27:12 UTC - in response to Message 41371.
Last modified: 23 Jun 2015 | 16:37:40 UTC

Instructions to do this might be helpful to some, however the GPU being used is an NVIDIA GRID K520 (4095MB), which is a Kepler GK104. You may be hogging it but it's only bringing in 160K/day, albeit running short tasks.

$2.40 U.S. per day = £1.54.

For comparison, an average quad core computer with a GTX970 might use around about 250W and at £0.15/KW (high in UK) that would be £0.90/day.
Turn off CPU tasks and that would be ~£0.70/day (or £0.47/day at £0.10/KW).
At a price tag of £250 a GTX970 would pay for itself inside a year (TCO), relative to running a K520 for a year.

Running 2 tasks simultaneously a 28500credit/task took 12,662sec on one of my GTX970's (W7). Overall, 2.4 times the performance of the K520 (Linux). So in terms of credit/£ (or $) the K520's performance isn't going to be too attractive, but it could keep your credit ticking over in the hot summer months (if you have those).
____________
FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

captainjack
Send message
Joined: 9 May 13
Posts: 157
Credit: 1,174,156,933
RAC: 212,559
Level
Met
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwat
Message 41399 - Posted: 24 Jun 2015 | 19:20:14 UTC

skgiven said:

it could keep your credit ticking over in the hot summer months (if you have those)


Yes, I live in a part of the world where we absolutely have hot summer months and I am trying to find a way to keep my credit ticking over during those months.

Here's some instructions on getting the GPU instance set up: you have to start with an image that uses HVM (hardware virtualization). I used Ubuntu because I am familiar with it and Linux has the cheapest rental rate. When downloading drivers to install, go to the NVIDIA download site then choose Product Type = GRID, Product Series = GRID Series, and Product = GRID K520.

Then I mostly followed the instructions outlined here
http://tleyden.github.io/blog/2014/10/25/cuda-6-dot-5-on-aws-gpu-instance-running-ubuntu-14-dot-04/. I installed all of his recommendations except for the CUDA installer software.

To get the prices that Matt was referencing, you have to use spot instances. With spot instances, you bid a maximum price that you are willing to pay per hour and you get access to unused hardware until there is a demand for that hardware for dedicated instances. AWS then starts shutting down spot instances starting with the cheapest bids first and the spot price goes up until they can meet demand for dedicated instances.

Here's the problem for volunteer computing, when a spot instance is terminated, all data that was generated during that run is erased. All tasks that are underway when the instance shuts down are then abandoned. You can set up the spot request to automatically spawn a new request when the price gets back down below your bid, but it starts over from the original image (with no data downloaded from the volunteer computing projects).

Most of the recommendations that I have read for dealing with the lost data suggest to take periodic snapshots of the data volume to S3 storage on AWS approximately every hour. Then set up your image to automatically restore from the snapshot if/when the instance starts back up again.

It looks to me like that could be problematic in the world of volunteer computing where tasks are downloaded, crunched and the result is uploaded then the cycle starts over again. Lets say that a task is downloaded and should run for 4.5 hours. At 4 hours a snapshot is taken. At 4.5 hours, the task ends and the result is uploaded. Then another task is downloaded and runs for 15 minutes. The spot instance is terminated by AWS then it starts back up later in the day. If the snapshot is restored, it will restore what it thinks is a partially completed task that in reality has already been completed and uploaded and it won't know about the partially completed task that the project sent to the instance. I would be afraid to guess at what happens next. Will BOINC Manager figure out the task underway has previously been completed, stop work on it and re-download the un-completed task that is now assigned to that instance?

For now, I can't see abandoning 10 hours work on a 12 hour task so I have the spot instance turned off.

If anyone else has any ideas on how to best work around this issue, I would be glad to hear them.

vorksholk
Send message
Joined: 22 Dec 09
Posts: 2
Credit: 31,062,789
RAC: 0
Level
Val
Scientific publications
watwatwatwatwat
Message 41478 - Posted: 4 Jul 2015 | 19:35:18 UTC - in response to Message 41399.

You can set the EBS volume to not terminate when your spot instance is terminated, so you could spin up a new spot instance, attach the old EBS volume, and continue working.

Betting Slip
Send message
Joined: 5 Jan 09
Posts: 669
Credit: 2,498,095,550
RAC: 0
Level
Phe
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 41484 - Posted: 5 Jul 2015 | 8:57:24 UTC

Could someone with a lot more experience than myself tell me which package I would need to run GPUGrid as fast as possible with a reserved instance and the cost?

Profile Acey Pilot
Send message
Joined: 4 Jan 14
Posts: 4
Credit: 1,901,240,470
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwat
Message 42133 - Posted: 10 Nov 2015 | 7:28:09 UTC - in response to Message 37852.

Hi

Anyone here using Amazon web services / EC2 for crunching (GPUGrid or otherwise)?
Anyone keen to try?

Matt


I'm trying now. Got NVIDIA GRID K520 GPU installed, but GPU Grid will not recognize the card. Boinc itself installs and runs great cpu projects.

Profile Acey Pilot
Send message
Joined: 4 Jan 14
Posts: 4
Credit: 1,901,240,470
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwat
Message 42134 - Posted: 10 Nov 2015 | 7:35:27 UTC - in response to Message 41371.

Now that the summer heat is upon us here in the Northern hemisphere, I decided to give Amazon Web Services a try. My air conditioner has a hard time keeping up when running multiple GPU's at full tilt. For the next few months, I plan to rent from AWS and keep the house cool.

biodoc pondered

"micro instances" don't seem suitable for Boinc


A micro instance is a single core virtual server running at a modest speed. I started out with one to get some experience and it fine for CPU tasks (as long as it's free). It is still running and crunching away on some WCG tasks.

It took me several days to figure out how to get the correct NVIDIA drivers installed on a virtual server instance and get them set up where BOINC would recognize them. My first GPU task just finished and validated about an hour ago. Task number 2 is underway now. If anybody wants to know what it takes, let me know and I will try to compile all my notes and post them here.

The cost is a bit different now from when Matt originally asked for interest. Spot prices for a GPU instance are running about 10 cents per hour which would be about $2.40 U.S. per day. The beauty about spot instances is that you only pay the going rate and you can specify the maximum hourly rate that you are willing to pay. If the going hourly rate exceeds your maximum, your instance will be shut down until the going rate gets back below your maximum rate. Also there is a charge for file I/O and for internet I/O. I'm waiting to find out how much it cost to download GPUGRID tasks and upload the results. After a few days, I should have a better idea.

Let me know if you have other questions.

Hi I'm set up on Amazon, but this project will not recognize Grid 520 GPU there. How did you get it working?

captainjack
Send message
Joined: 9 May 13
Posts: 157
Credit: 1,174,156,933
RAC: 212,559
Level
Met
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwat
Message 42140 - Posted: 10 Nov 2015 | 22:28:57 UTC

Hi Acey Pilot,

All of my experience at AWS is using Linux Ubuntu. It looks like you are using Windows so I may not be of much help. If you are using something other than Windows let us know.

You might try this:

When you log in to AWS it takes you to a page for Amazon Web Services. In the top left corner, click on "EC2 Virtual Servers in the Cloud".

In the middle of the page is a blue button for "Launch Instance", click on that.

It should bring up a list of Amazon Machine Images (AMI's) that you can use. On the left side are several tabs. Click on the tab for "AWS Marketplace".

In the upper left corner of the window should be a box where you can enter a search term. The box should say: "Search AWS Marketplace Products". In that box type "nvidiagrid" and hit enter. It should bring up a list of AMI's that already have the GPU drivers already installed.

From there, you should be able to pick an AMI of your choice and build out a GPU cruncher.

Please let us know how it turns out.

If that doesn't get you going, send us some more specifics about your configuration.

Hope that helps.

Profile MJH
Project administrator
Project developer
Project scientist
Send message
Joined: 12 Nov 07
Posts: 696
Credit: 27,266,655
RAC: 0
Level
Val
Scientific publications
watwat
Message 42315 - Posted: 8 Dec 2015 | 13:47:47 UTC - in response to Message 42140.

Hi,

You need to install the nvidia driver before the GPU can be used.
The most cost-effective way to use ec2 for GPUGRID is to run on g2.2xlarge instances and use spot instances. These are almost a tenth the cost of on-demand instances, about 0.07USD/hr.

If there's any interest, I can put together a Marketplace AMI than you can run without having to get your hands dirty.

matt

mhhall
Send message
Joined: 21 Mar 10
Posts: 23
Credit: 861,667,631
RAC: 0
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 42754 - Posted: 3 Feb 2016 | 19:29:54 UTC - in response to Message 42315.

I'd be willing to help test this if you decied to try it.

Post to thread

Message boards : Number crunching : Anyone using Amazon to crunch?