DevOps in Academic Research

https://mattsegal.dev/devops-academic-research.html

41 10 5 years ago

Great stuff, wonder how he convinced the team to drop the "for free" academic HPC/Slurm mess in favor of costly cloud computing. That is my biggest hurdle working at a university right now.

zhdc1• 5 years ago

A lot of companies are donating computing power for COVID-19 research, so it's possible they have a deal of some kind worked out with Amazon.

That is my biggest hurdle working at a university right now.

We took the opposite approach and assembled our own equipment, which we self host and manage. The cost savings we're significant, and what we lose doing admin and maintenance is saved by having the flexibility and freedom to code and run what we want, when we want.

dkdk82832• 5 years ago

Onprem doesn’t get a lot of love these days but it’s significantly cheaper.

pacbard• 5 years ago

wonder how he convinced the team to drop the "for free" academic HPC/Slurm

Probably because they are working with publicly available data so they don't need to worry about HIPAA/restricted data compliance. One of the main reasons for using the HPC/Slurm cluster is that IT (or the data compliance office) guarantees compliance with restricted data.

evv555• 5 years ago

Couldn't IT do this with AWS as well? One of the few advantages of AWS are the compliant and auditable infrastructure patterns they provide.

universa1• 5 years ago

mhm... having skimmed through the article, the only reason (imho) is not wanting to wait for hpc/slurm to schedule your jobs... all the other stuff, could have been done with university resources. They usually offer some shared drive space and are often forthcoming in supporting special needs (at least from my experience).

Having used both AWS Spot Instances for compute jobs, as well as hpc resources (slurm), the amount of work to getting it run is comparable... Except that the aws stuff quickly gets expensive when needing thousands of cpu hours per "experiment", and money is usually tight at universities ;-). Some ansible scripts automate basically everything away for myself. But using the tool you know is probably not always a bad idea.

PeterisP• 5 years ago

I had a case where using the "for free" academic HPC cluster was so inconvenient that we used only in very few cases when we needed very large amounts of compute, and for any smaller tasks using the commercial cloud solutions saved enough time/salaries so that they were worthwhile even purely from cost perspective. Also, for academic purposes it was possible to get quite large amounts of cloud computing for free, so most users never bothered to even try to approach the academic HPC environment as it had no advantages whatsoever.

The_Amp_Walrus• 5 years ago

Thanks! AWS was costly compared to the free HPC cluster but it was sub $1000 per month and our bottlenecks were person-hours rather than money. We had a lot of work to do. As some have mentioned there wasn't a strong technical argument for switching to AWS, it was more about productivity with those tools. I'd built up a bit of credibility with the team at that point and was like "I can make this better" and my boss was said "ok cool". As I mentioned in the article the team was very open to try new things.

I can see an alternate timeline where I set up Buildkite in AWS to run Slurm jobs on the HPC cluster, rather than using EC2 spot instances. Using Buildkite + S3 were probably the more important infra changes.

DevOps in Academic Research

Related Stories