Future of OSU Open Source Lab in Jeopardy

by aendruk • 1 year ago

289 102 1 year ago

EMH333• 1 year ago

The Open Source Lab was a fundamental part of my college experience. I would not be the person I am now if not for the experience gained while employed there. It was such a great feeling to help hundreds of open source projects maintain infrastructure and services, especially some of the larger projects which have colocated hosts

jawilson2• 1 year ago

Am I reading correctly that of the $250K they need, $150K of that goes to a single staff member for 60% of their time? Does that seem...excessive?

skyyler2• 1 year ago

No, you are not reading correctly.

The 60% number is the percentage of the budget, not the staff member's allocated time.

However, what do we know about the duties of this staff member? $150k isn't a very high salary for an experienced systems administrator

indrora• 1 year ago

It's higher than an SDE in Seattle, but less than a senior position at those same companies, for people who want some perspective.

Firmly "Middle ground of the area"

avs7332• 1 year ago

From a university grants perspective that likely includes benefits.

Grant hiring math is

Salary + benefits = cost

Where benefits = salary *~.4

cycomanic3• 1 year ago

Presumably the $150k also includes all oncosts, so the actual salary is quite a bit lower still? As a side note I don't understand the arguments about salaries for nonprofits. Sure they should not be outrageously higher than the average, but shouldn't we want to get the best people for these jobs (instead of them working on aware?), or is the argument that if you work for a nonprofit you should be doing it out of altruism and be glad you receive a salary at all?

unquietwiki• 1 year ago

I've been in the IT field for 20+ years, but I've never made that much in a year. What's stopping someone from saying, "Hire two cheaper guys, so there's redundancy?"

jmull• 1 year ago

The $150K probably needs to cover other costs, like payroll taxes. Perhaps other benefits as well? (Health insurance would be the big one -- in the rough ballpark of $20K).

tikhonj• 1 year ago

Assuming that's the "fully loaded" cost (ie including taxes, benefits, etc), seems like it would translate to a take-home salary of $100k or less.

ecnahc5154• 1 year ago

That's 60% of the _budget_ not 60% of their time.

Also: Lance is almost certainly working more than 40 hours a week. Also, he isn't just a systems administrator. He's a mentor, fundraiser, any literally everything else that is needed to keep the lab running. There used to be more staff, but it's hard to retain qualified individuals. He's been there for 17 years, he's not doing it for the money, he does it because the OSL is important!

ecnahc5154• 1 year ago

Oh, and since he's a public employee, you can look up the current salary and history.

https://hr.oregonstate.edu/sites/hr.oregonstate.edu/files/er...

https://www.openthebooks.com/oregon-state-employees/?F_Name_...

I'll summarize it:

$107k in 2017 and $124k in 2023. I don't know about you, but someone with 17 years experience could easily be making 2-5x that depending on the company and role.

fuzzy_biscuit• 1 year ago

Not if we're talking about an experienced engineer that is the only full-time staff for a year. That's the entire budget, so it feels pretty spot on. Maybe I'm out of touch or misunderstanding your point, though.

firesteelrain5• 1 year ago

GP is pointing out that OSU Open Source Lab is only in jeopardy because a majority of the shortfall is in labor costs. 86% to be exact.

johnvanommen• 1 year ago

Am I reading correctly that of the $250K they need, $150K of that goes to a single staff member for 60% of their time? Does that seem...excessive?

When I was doing AIX and Solaris system administration in Salem Oregon, they paid me $75 an hour.

A lot of people here are comparing Corvallis to Seattle, but they’re hundreds of miles apart.

Salem is the nearest big city.

TBH, making $75 an hour in Salem was like making $150 an hour in Seattle. You can live REALLY WELL on $75 an hour in Corvallis.

svnt• 1 year ago

Maybe ten years ago, but not since the pandemic.

Seattle is still more expensive, but $150k is just buy-a-house and have-a-kid money in Corvallis anymore.

erikerikson• 1 year ago

And that $75 an hour (which annualizes to $156,000) doesn't include employer overhead (taxes, benefits, etc.) which can double their cost.

Ar-Curunir4• 1 year ago

No? Less qualified software engineers make more with less experience, and while doing orders of magnitude less important work

mulmen• 1 year ago

No, it doesn’t seem excessive at all to me. If the lab delivered the same services for $2.5mil would you still be questioning that $150k salary?

jimbob45• 1 year ago

That’s going to be a tough sell to any company. At some point, this is less of a charity and more of a failing company.

bananapub• 1 year ago

I really am constantly surprised at how foolish some posters on this site are, but your comment is pretty exceptional even by recent low standards.

do you actually know what OSUOSL is and what it does? do you care? if not, why are you even commenting on this post?

saagarjha2• 1 year ago

This is not a company. It is not expected to turn a profit.

_mlbt2• 1 year ago

Oregon State University has a $1.651 billion endowment according to Wikipedia…

https://en.m.wikipedia.org/wiki/List_of_colleges_and_univers...

Would that be an option to save it if corporate sponsorship doesn’t work out?

jldugger2• 1 year ago

Most gifts that fund university endowments are earmarked by the donor for specific purposes. And for the money not earmarked, you're competing against all the other priorities, including making up for various unplanned shortfalls of Federal funding.

timewizard• 1 year ago

The Foundation itself has nearly $1b in assets. The problem with these foundations is they're often streched across the entire university. Which means the foundation behind OSL is also the foundation behind intercollegiate athletics and tons of other completely unrelated programs.

https://projects.propublica.org/nonprofits/organizations/936...

Vortigaunt• 1 year ago

I ctrl-f'd Oregon State and didn't find it in the link you provided. I think you found UofO's endowment: University of Oregon - $1.651

Wikipedia states that OSU's endowment is $829.9 million (2023).[0]

[0] https://en.wikipedia.org/wiki/Oregon_State_University

_mlbt2• 1 year ago

You’re right, I mixed them up! Thanks for correcting me on that.

dvrj101• 1 year ago

people have started using the word "endowment" against colleges like it's a term for slush fund.

natebc• 1 year ago

indeed they have. In reality they're like complicated heterogeneous investment funds with untold numbers of varying restrictions governing them.

They're not just a pile of gold coins in the university presidents basement.

aseipp• 1 year ago

When I was working on GHC many years ago OSUOSL helped us by providing us access to some nice POWER7 machines (courtesy of an IBM kernel hacker who recommended and endorsed us) and we used them for years to solve weird issues. I've always thought very highly of the Open Source Lab. I hope someone can help them make it through this.

xbar• 1 year ago

I was always happily surprised to find that they were hosting what I needed when I needed it.

A great lab with a long history.

floren• 1 year ago

I hear Microsoft loves open source, so they should be able to step up and cover this, right?

firesteelrain5• 1 year ago

Why stop there? Facebook, Google, OpenAI, Netflix, Amazon…

kev009• 1 year ago

A lot of the fun parts of the computing industry have, predictably, been hollowed out by the rent seeking model of cloud and *aaS. There is some grace as it's easier than ever to build some scalable web business.. but the most fun of my career was rabbit holing on computers for the sake of computers.. working on operating systems and device drivers and network stacks. And it did and still does matter to a lot of bottom lines, but corporates have a hard time connecting the dots or doing something other than what the flock is doing.

It's a little awkward because the AI datacenter boon is a little bit of a revival for physical and systems work but it is limited to that and I am skeptical of the longevity.

Those days of having fun working on network stacks, operating systems, setting up FOSS development labs and being a good steward of things.. harder and harder to do and even harder to get started.

mitchellh• 1 year ago

OSUOSL and Lance specifically (the writer of this post) was extremely supportive of me during the early days of Vagrant and Packer. Lance tried many times to try to find a way for OSUOSL to help my projects but I don't think we ever formalized anything.

Regardless, they were always big users and big proponents of the OSS work I was doing. And I remember that. I think more than the OSS project support they do, the support and education they help provide for students is laudable.

I personally think corporate sponsors shouldn't blink twice at supporting OSU OSL, but I'm not surprised given the state of... things. And the individuals choosing to judge and criticize based only on a 4 bullet point budget are infuriating.

Well, I'll help. I've emailed to setup a donation.

Thanks for everything you've done Lance, OSUOSL. And thanks to anyone else who helps support them!

ecnahc5154• 1 year ago

As someone who was a student at the OSL when Vagrant was hip, also thanks to Mitchell for creating Vagrant! We used it a ton for testing all our our configuration management.

jratkevic• 1 year ago

Lance is a great guy and that lab for a decade plus has done great work and supported so much of community. Happily supported their start and will continue to.

rdtsc• 1 year ago

OSU OSL provides CI machines for some of the more exotic architectures like Linux on Z and POWER to some ASF projects. It would a loss to close it down.

Maybe some unicorn billionaires could spare a few millions? Especially the ones who built their wealth on top of open source libraries or databases.

damnitbuilds• 1 year ago

OpenAI should fund them - they're obviously very heavily behind Open Source, as it's what they were created to do.

https://help.openai.com/en/collections/12496919-open-source

Ah.

mburns• 1 year ago

The OSL was transformative for my career as a budding CS student in Corvallis many years ago. I can’t say enough good things about the positive impact it has on the Open Source community and the students it employs.

In my experience, there isn’t a great on-ramp for learning to be a SysAdmin (or devop, etc) in a practical sense. Learning what it takes to support systems in “Production” with actual users, and all that entails, at some point requires a hands-on approach. Finding entry-level opportunities to do that isn’t easy until you have /some/ experience. The OSL provides that, and supports countless FOSS projects in the process. It’s really a great arrangement.

Obviously I’m biased, but the Open Source Lab should be viewed as one of the Crown Jewels of OSU.

ecnahc5154• 1 year ago

I worked at the OSL as a student years ago, and it was one of the most impactful places I've ever worked at. I learned a lot, and I wouldn't be the engineer I am today without having worked there.

Since graduating, I've also hired, and worked with multiple alumni from the OSL and they're always top notch. Anyone looking for interns or new graduates with devops/SRE or SWE experience should be looking at the OSL for talent. It's not too often you can hire a new graduate with potentially multiple years of production experience, especially in devops.

In context of HN/Y Combinator, https://www.ycombinator.com/companies/coreos was a successful container/Kubernetes focused startup founded by two OSUOSL alumni, Alex Polvi and Brandon Philips, which was eventually acquired by Red Hat.

The OSL is something special.

For a list of projects the OSL helps host, check out https://osuosl.org/communities/. You might see a project you care about in that list! As an example: they provide aarch64 and powerpc VMs for a ton of projects to do their CI/builds on.

chneu2• 1 year ago

Same. I helped out/worked there out of high school(back around 2010).

One of the best experiences of my life.

I still prefer to use the OSL for my linux repos.

mulderc2• 1 year ago

I feel like that shouldn’t be impossible to raise. I would be more than happy to donate $250, now we just need 999 more to do the same.

mulderc2• 1 year ago

Link to donation page: https://osuosl.org/donate/

sregister• 1 year ago

Jensen is an OSU alum--it would be nice if this reached him.

don-bright• 1 year ago

They are part of gnu compile farm which donates compute to open source projects.

I used them a lot when I worked on OpenSCAD build system, there weren't a lot of places 12+ years ago you could go 'make -j 30' on a PowerPC or 'ctest' and have it run dozens of builds/tests in parallel. Really helped alot, that C++ template stuff would barely build at all on my personal machine.

Sorry to hear this

Grae• 1 year ago

A gap like cannot be closed without corporate and foundation gifts. That said, individual gifts can also contribute to the progress.

As is common with schools, parks districts, etc., the Open Source Lab partners with a 501(c)(3) organization, the Oregon State University Foundation, to accept tax-deductible donations.

For anyone who would like to directly support the Open Source Lab in staying open, please be sure to indicate "Open Source Lab Fund" on the Oregon State Foundation donation page [0]. Note that their form is *not* set up with any tracking to attribute your gift from your clickthrough, and that any general donations to the Foundation will likely *not* support the Lab in this effort to stay open.

[0] https://give.fororegonstate.org/PL1Uv3Fkug, or click through from the general donation page.

myself2482• 1 year ago

Oooo dang, the lack of an "opt out of future emails" checkbox really gives me pause there. Charitable donations are second only to vehicle purchases in the avalanche of shit they trigger.

Welp. Did it anyway. There's a "make my donation anonymous" checkbox on the second page, but that doesn't actually specify whether it opts out of emails, or merely hides my name from some list. Well, it's worth it either way.

mensetmanusman2• 1 year ago

Use the hyphenated email filter trick.

gkanai• 1 year ago

I was with Mozilla when OSU's OSL was supporting the mirroring/serving of Firefox and Thunderbird globally. They were a key mirror/supporter during the early days of Mozilla and definitely contributed to the growth of Firefox.

pimlottc• 1 year ago

While the Oregon State College of Engineering (CoE) has generously filled this gap, recent changes in university funding makes our current funding model no longer sustainable.

I suspect this is related to the recent Trump administration actions to withhold funding from colleges and universities. OSU already voted to increase tuition recently amid concerns about future federal funding [0]:

“New federal priorities and proposed funding cuts, especially for research, may have direct, negative consequences for OSU,” [OSU President Jayathi Murthy] said.

0: https://archive.is/iGX5X

nl2• 1 year ago

"I have reached out to our largest corporate sponsor and they are working to increase their support as we update our contract, but that still may not be enough."

Why wouldn't they name the sponsor? Seems like a great thing some company (presumably Google?[1]) is doing, but they don't get the credit they deserve.

[1] "Google sponsors the OSL through financial grants." (Noting that none of the other sponsors seem to provide financial grants) https://osuosl.org/sponsors/

pabs32• 1 year ago

They should reach out to IBM too, since POWER and Z hardware are mostly an IBM thing. ARM should be willing to help too, since they host AARCH64 stuff too.

junar• 1 year ago

"Below are groups who supported the Open Source Lab through annual contributions of $25,000 or more during fiscal years 2019 and 2020."

It seems that all of them have contributed financially (though of course, it's unclear which ones do so at present).

nl2• 1 year ago

"annual contributions of $25,000"

I think includes hardware/service contributions (eg the IBM hardware and the bandwidth suppliers). It's a bit unclear though.

ajdlinux• 1 year ago

Give them the credit after the sponsor actually finalises the updated contract.

ThinkBeat• 1 year ago

I find it odd that they can provide all the infrastructure compute / storage / bandwidth for all of these projects for $35K

Debian and Fedora on their own must be highly demanding??

Form the article: """ Currently provides infrastructure hosting for projects such as Drupal, Gentoo Linux, Debian, Fedora, phpBB, OpenID, Buildroot/Busybox, Inkscape, Cinc and many more! """

pabs32• 1 year ago

Most of Debian's infra is elsewhere, but OSU does host a backup server, logs server, a bugs.d.o frontend, POWER/MIPS/HPPA/RISC-V build/porting servers.

https://db.debian.org/machines.cgi

Similarly, I assume RedHat hosts most of Fedora's infra and OSU does POWER/IBM Z stuff for them.

bigfatkitten2• 1 year ago

I would've assumed that Red Hat/Fedora are well-catered for now in terms of POWER and Z, considering who now owns Red Hat.

richardwhiuk• 1 year ago

Bandwidth is donated by TDS Telecom. I suspect that physical hosting is provided by the University and servers seem to be provided by IBM.

altairprime• 1 year ago

They need university budget for items that aren’t donated in-kind rather than as cash by sponsors. Build servers can be donated by granting OSU access and paying the bill. But someone has to manage those donated services, and none of the existing donation agreements apparently include cash on table to pay for that headcount (since they did not until this year).

jldugger2• 1 year ago

My understanding is that a lot of ISPs use linux mirrors as an offset for asymmetric bandwidth, which can lead to one-sided network charges. As such, they're happy to donate bandwidth to the degree it benefits their negotiations elsewhere.

phillipcarter• 1 year ago

I didn't work at OSL directly, but I did work for the sibling organization (then called BSG, now called CAAS) and it's sad to see this. I donated and would encourage other reads to do so too.

OSL and the sibling organizations at OSU are such an incredible program for students, and OSL in particular has had an outsized impact on the OSS ecosystem relative to its size and cost.

Moreover, it's institutions like this which represent some of the best of the software world, and we can't let them fall by the wayside as the capitalistic focus of larger companies just gobbles everything up with little regard for what they destroy.

system7rocks• 1 year ago

I am not working in the computer science field any more, but my experience working with computer systems in an environment like this helped me gain a lot of confidence as an individual. This is amazing work and deserves to be funded!

musicale• 1 year ago

I want to go to Open Source University!

kinow• 1 year ago

Learned about the OSL way back in Brazil via their support to Open Source and some of their alumni who worked in OSS projects too. I hope they figure out a way forward. And I was surprised to see they ask funding for a single position. Always imagined they would be a team of 3-7 engineers.

Firefishy• 1 year ago

I help to run the OpenStreetMap infrastructure, and for many years the OSUOSL has generously hosted some of our servers.

Without OSUOSL OpenStreetMap would be more difficult to host and significantly slower to access from North America.

I hope OSUOSL can get the financial support they rightfully deserve.

eunos• 1 year ago

OOT, How'd you distinguish between Oklahoma, Oregon and Ohio State University when you see the word OSU?

myself2482• 1 year ago

OSU.edu is Ohio, all others have their own domains. This is also why Michigan State is the one true MSU, etc. (Being a Michigander, it pains me to give Ohio any credit for anything, but that goes to show how much I think this is a sensible way to distinguish them.)

genunix• 1 year ago

The donation page is not easy to use and one must search the drop down menu for the tag "Open Source Lab Fund". However it is worth the effort. I did my part.

Future of OSU Open Source Lab in Jeopardy

Related Stories