Crypto miners are killing free CI

https://layerci.com/blog/crypto-miners-are-killing-free-ci/

by Lyn_layerci • 5 years ago

120 34 5 years ago

Sharing some thoughts from our own experience fighting cryptominers and the negative externalities for CI companies and their users. I'd be curious to hear if any other services have been affected.

abayer• 5 years ago

Yup, we’re dealing with it on CodeShip. I’m pretty sure all CI services are swamped by mining on free accounts or fraudulent paid accounts.

Ramiro2• 5 years ago

We have the same problem in Okteto. We've been investing a lot on building tech to prevent this (I gave a talk on this during the last eBPF community days -> https://www.youtube.com/watch?v=tplv3Hjjv2Q), but it's tough. We spend a LOT of resources fighting it.

tulsua• 5 years ago

Do you see yourself able to fully automate that process? The Falco -> slack notification -> manual ban doesn't sound like it will scale very well (but a nicer workaround than outright removing the free tier!).

miltoncd• 5 years ago

Interesting! Didn't realize it was affecting so many services...

salawat• 5 years ago

Once you have any way of allowing other people to use cycles... They will do it. And you can't really be surprised when you have these cryptocurrencies that folks in need of cash with few if any other options use it. It's why I object to the activity on principle. It becomes the new default+ activity.

Any computation not explicitly provisioned in a way that guarantees pre-empting a cryptocurrency generating process never has a chance to happen.

Lyn_layerci2• 5 years ago

Yes, this needs to be solved at the crypto level - attackers always have the advantage.

davidgerard• 5 years ago

This is usually Monero mining, which is still feasible on general CPUs - particularly if you're stealing the compute power.

Monero is quite hard to trade directly to actual money - you usually have to go via BTC or ETH.

So a systemic fix basically involves crashing the price of crypto in general, to take away the financial motivation.

(This is also one obvious solution to bitcoin's CO2 production - it's all about the US dollars, so hit those gateways.)

ggghgrrrrr• 5 years ago

How about online compilers: you encode the mining as a C++ template and do the mining at compilation time, or you use their "run" functionality.

sydd• 5 years ago

Sadly they are able to attack anything that allows arbitrary code execution

ddevault• 5 years ago

We're moving builds.sr.ht to only support paid users from May forward because of crypto mining abuse. Background here:

https://man.sr.ht/ops/builds.sr.ht-migration.md

I've been in touch with many other people working in the CI industry and this has become a massive problem for all of us over the past few months. Entire industry working groups have been set up for knowledge sharing to combat the crypto mining epidemic.

In hindsight, cryptocurrency is an abject disaster and one of the worst inventions of the tech industry in the last few decades. I am absolutely ashamed to share an ecosystem with such an obscene, exploitative grift. In addition to entirely failing to meet its basic objectives as a useful currency, it has introduced perverse incentives into the entire technology sphere, reduced the integrity of the entire industry, been the subject of hundreds, if not thousands, of scams and ponzi schemes, has created shortages for consumer and server hardware, and is hugely wasteful and harmful to the environment. Fuck cryptocurrency.

codekansas• 5 years ago

Crypto actively rewards anti productive behavior. I wonder how many smart, ambitious people will decide to exert their talents doing basically useless stuff simply because of the financial incentives

nix232• 5 years ago

You could say the same about the whole Finance-industry..has nothing todo with Crypto.

nix232• 5 years ago

In addition to entirely failing to meet its basic objectives as a useful currency.

That's the biggest problem, and it should be like IOTA where you "proof" when you do an transfer and just then, and not by "printing" coins, it's a massive waste of energy.

judge2020• 5 years ago

This is still happening right now on circleci (requires login but is otherwise public https://app.circleci.com/pipelines/github/testronan/MyFirstR...)

Ayesh• 5 years ago

GitHub recently changed its policy to not allow CI runs on first time contributor PRs until approval, and to flag PR maker instead of the repo owner on potential abuse.

quickthrower22• 5 years ago

Contrarian point of view: why do we “need” free CI? Open source can run CI locally on docker etc. The free CI is marketing for the CI companies. Don’t offer free compute. We need to train the industry in general to pay for trials. Some companies eg those in the SEO vertical manage to do this. Ahrefs for example.

lifeofguenter• 5 years ago

Running something locally is not necessarily free either. Even if maybe the "host" is for free (which I can not imagine, unless someone throw something at you), it will still require your time to setup and ensure its running.

For private/small open source projects I would highly welcome such a pricing tier from travis-ci but 69USD is just not in my pocket (not saying its unjustified though!).

For larger open-source projects that rely on many builds throughout the day/month, this might be a bigger problem.

So while its "free" marketing, it might be much more crucial to our open-source structure than we think it is (which might be a problem as well).

And I think we can all agree: open source software is an important part not only for the industry but our lives :)

mellyjelly• 5 years ago

This is part of the reason open source is so important, but should be enough on its own to argue that free services should exist: these CI services have been an important piece of education, which should be free.

quickthrower22• 5 years ago

I use a bunch of different CI at work and they are great but if they are offline you are stuffed as they are not portable and running local while possible is a nightmare.

Containers are a solution for this right? If I was open source I’d run CI in docker or similar so you can easily run it locally or on a server.

The assumption is people doing FOSS can’t afford maybe $20 a year to spend on some kind of elastic compute to run this or don’t have a computer powerful enough to run it. I doubt this is the case and if it is just do without CI - just run your tests from IDE before pushing to master.

lacker• 5 years ago

I mean we don't need it, society can survive without free CI, but it is nice. I'm happy to get free CI for an open source project in return for the CI company marketing to me a bit about how great their product is.

anurag• 5 years ago

It's not just CI providers: we're seeing the same thing on Render (https://render.com) and I bet Heroku and AWS are all equally impacted.

gjvnq2• 5 years ago

Noob question: why not make a big delay to start processing build jobs (something like 1 hour). By then, whatever input they wanted to hash will be useless as a new block will be already minted.

As for build scripts that require a network connection, just make the connection painfully slow.

xyzzy123• 5 years ago

Because then your CI provides slow feedback on changes :/

hnarn2• 5 years ago

Isn't this discussion about free CI? Seems like the answer is quite simple then: if you want it faster, pay for it.

BrendanLong2• 5 years ago

They're not starting a CI job per hash (that would be too slow). I'm not sure exactly how each of these cryptocurrencies works, but presumably what they're doing is starting a miner which attempts hashes for a while and then stops. And the only reason the jobs stop at all is that it would be too obvious if they ran continuously.

Ygg2• 5 years ago

Silly question. But couldn't CI say in TOS, that any crypto currency mined using their CI/CD resources belongs to the host?

ronsor• 5 years ago

They could, but that wouldn't mean much for a few reasons:

* Cryptocurrency can't really be confiscated

* These malicious miners likely live in a country with a dubiously competent legal system.

* Lastly, it's not worth the money to try to collect.

BrendanLong2• 5 years ago

Expanding on that last point, one of the examples in the article is someone making ~$70 per month in cryptocurrency. The CI people -could- send a lawyer to Vietnam to try to collect that $70 but even if they succeed it's very not worth it.

hnarn2• 5 years ago

It seems like a law of the Internet that "nothing nice will last". If there's a potential for abuse, it will be abused and the rule-abiding majority will suffer for it. Firefox Send is another example of this, it was pretty obvious from the start that the threat vector of abuse would make it untenable in the long term even if the service itself was awesome.

Havoc• 5 years ago

In a way I think this was somewhat inevitable. Arbitrary code execution is somewhat commoditised.

I guess I’ll have to have another look at activating the CI on my home gitlab install

urza• 5 years ago

If only there was a way how to anonymously charge something like $0.1 for each action/api call.... I don't know... I heard maybe something like cryptocurrencies can do it?

Crypto miners are killing free CI

Related Stories