Frontier AI has broken the open CTF format

https://kabir.au/blog/the-ctf-scene-is-dead

by frays • 2 months ago

418 466 2 months ago

kabir.au

walletdrainer3• 2 months ago

I started playing CTFs in 2021
and the old game is not coming back

For many people the CTF scene was already dead in 2021 because it had turned into something unrecognisable.

In reality it’s just different.

lukan2• 2 months ago

Well, I had to google what CTF means (capture the flag, a hacking competition), so surely cannot judge here, but the text indicates that with AI some things are very different today:

"That makes open CTFs pay-to-win. The more tokens you can throw at a competition, the faster you can burn down the board. Specialised cybersecurity models like alias1 by Alias Robotics are becoming less relevant compared to general frontier LLMs. The competition is turning into "who can afford to run enough agents, with enough context, for long enough.""

mock-possum• 2 months ago

Isn’t that the bitter lesson in a nutshell? “Specialised cybersecurity models … are becoming less relevant compared to general frontier LLMs.”

walletdrainer3• 2 months ago

There are two different schools of thought:

1) It’s OK to do just about anything to win a CTF, including installing malware on the organisers computers months before the actual event so you’ll have an easy time stealing the flags.

2) It’s not ok to try and win the CTF with a solution the authors did not intend.

Recently the #2 crowd has been winning because the hacking scene has turned corporate and boring. People started to partake in CTFs in the hopes of landing a job(!)

CTFs are indeed ruined for those people, I personally don’t mind.

For the people in group #1 LLMs change little. Attacking the challenges directly was always a last resort.

Grimburger2• 2 months ago

Learning about eternal September in May 2026

Hits different doesn't it

Retr0id6• 2 months ago

I started playing in 2015 or so and had mostly stopped by 2020. Not because I felt it was "dead" exactly but it just wasn't hitting the same for me. By then it wasn't "the winner has the most LLMs", but "the winner has the most members on their team". I merged into one of the mega-teams and it just wasn't fun any more.

deafpolygon• 2 months ago

Unrelated, but does anyone find this site incredibly hard to read?

walletdrainer3• 2 months ago

Bizarre font and poor contrast, yep.

The text itself being exceedingly long for no obvious reason doesn’t help.

lukan2• 2 months ago

Poor contrast? White on black?

And if you think it was too long, what part would you have shortened? I never knew about the scene and found it interesting to read this personal take on it.

tromp• 2 months ago

https://en.wikipedia.org/wiki/Capture_the_flag_(cybersecurit...

still has no mention of AI, but that will likely change as they increasingly dominate competition.

sumeno3• 2 months ago

Using AI on CTF is like using a car to get better at the 100 yard dash

Choco31415• 2 months ago

Except some people want to win and don't care about their own personal performance.

low_tech_love3• 2 months ago

Yeah but for the brief time window (which is probably coming to an end right now) where getting your name on the leaderboards was still worth it anyway, because people had not yet realized that the game is over, players will use the car anyway. Now that the game is over, leaderboards are meaningless, so we will figure out ways to move past that (like playing with people who care). But that will change the game in unavoidable ways: the tiny, fragmented scale will give less incentive for creators to come up with massive intricate complex challenges (it would be like hoping Bethesda would make Skyrim for a handful of die hard players). And soon, maybe on a few years, people will invariably start questioning whether it even makes sense to waste their time with this hobby if learning CTF skills is basically useless in an AI dominated world. There are still people learning Assembly for fun, but almost nobody does Assembly programming challenges as a hobby.

vasco• 2 months ago

My first ever was Stripe CTF in 2012 I think, I still wear the shirt I got (now super fainted) from passing some challenges. I was a student in portugal and remember receiving the shirt for it and thinking, maybe those Americans aren't any better than me and I can compete at the same level.

I never got super into security but it gave me the confidence to play in the same field and lose the stupid aura I had that somehow "rich americans" would be better than me at everything because they had better universities or because of Hollywood or something.

Sad that another cool thing is lost to AI but I guess kids will learn in other ways.

chvid• 2 months ago

What is CTF? And why is the cyber security world filled with silly gaming references?

throwa356262• 2 months ago

https://en.wikipedia.org/wiki/Capture_the_flag_(cybersecurit...

Its a war game reference I guess?

mort963• 2 months ago

Capture The Flag is a cybersecurity game where the organizers set up a bunch of intentionally vulnerable computer systems with a "flag" on them, a string that's "supposed to be" secret but is accessible through exploiting the vulnerabilities. This may be a line in /etc/password, a string in memory, a field in a database, whatever. The goal of the game is to hack into the computer systems, find ("capture") the flag, then copy/paste it into the organiser's scoreboard website to prove that you solved that particular challenge.

It's pretty fun. Or at least it was, back when you had some sense that your competitors were competing on an even playing field and just beat you because they were better than you.

I wouldn't say the name is a "gaming reference", it's just a descriptive name for a game.

monarx• 2 months ago

used to see some really good CTF videos show up on youtube and now nothing like that shows up on the feed

susam• 2 months ago

I have normally found any sort of timed technical competition intimidating. Even so, about 6 or 7 years ago, after being persuaded by a colleague, I participated in a few CTFs. I am glad I did, back when this type of thing still meant something. I have kept a screenshot from one of the CTFs that I am quite fond of: https://susam.net/files/blog/ctf-2019.png

eecc• 2 months ago

“solve”, why not solution? Like “spend” and not expenditure, why use the verb as a noun and not care about grammar?

iainmerrick• 2 months ago

They’re shorter.

Why so pedantic?

sheept3• 2 months ago

These examples that you're calling "verbs as a noun" are standard grammar. You can't just invent simplified rules about a language and declare it wrong when the rules fall apart.

msm_2• 2 months ago

In addition to what others have said, this usage is very common in the CTF world. "The challenge has no solves", "We just got the first solve" etc are very idiomatic. It would actually look weird to me if this was "solution".

tripzilch• 2 months ago

I don't understand the complainiture, it's an improve

kevinsimper• 2 months ago

You could make it offline and with provided laptops only, just like with the competitive CS2 scene.

hsbauauvhabzb3• 2 months ago

Ctfs need preparation and unconstrained internet, even if you block domains it’s possible to tunnel out

belabartok39• 2 months ago

Use jumpbox to access CTF. Disable all wireless for the playing hall.

sheept3• 2 months ago

Presumably if you block domains, you wouldn't be able to use AI to find a way around the block. So doing so demonstrates at least some human skill

Retr0id6• 2 months ago

Unconstrained internet is nice, but I don't think it's a hard requirement. Just tricky to enforce, even in-person.

eastbound2• 2 months ago

Since real-life situations involve AI, banning AI would make CTFs just a simple game, not a demonstration of capabilities and talent.

loeg• 2 months ago

They always were just a game?

mort963• 2 months ago

What do you mean? Solving a CTF challenge demonstrates way more capabilities and talent than just asking a chat bot to solve a CTF challenge.

sheept3• 2 months ago

Offline CTFs could also incorporate physical security challenges, like lockpicking

tylerchilds• 2 months ago

I do like the idea of escape the room games becoming the cybersecurity employable competition meta

Retr0id6• 2 months ago

They often do

hofiflo• 2 months ago

The recent LakeCTF onsite finals had exactly that. LLM usage was forbidden (but players still used their own devices) and there were real-life challenges such as lockpicking as well. I’m part of the organizer team and what we’ve heard so far from participants was that it was really enjoyable not to have any LLM help because suddenly the actual skill and thrill when solving a challenge mattered again. I think what helped in this case as well was that the prizes weren’t high-value enough to incentivize cheating but that participating in the event itself and the social aspect around it are the main point.

rurban• 2 months ago

I don't do CTF's but took part at the security workshop for fun ~2 years with my Android phone only. I was first with the first simple challenge, but then couldnt continue because my phone was just too limited. But I watched what the others did. And a young Indian guy did everything with ChatGPT then. I found it silly, but amusing, because he actually got second. There was no Codex nor Claude then. Nowadays it must be dead for real, because I would solve everything with my agents, as I do in the real world.

Grimburger2• 2 months ago

Very impressed that OP has gone from starting university in 2021 to becoming a Senior Security Engineer.

It's an incredibly exciting time in security research in my humble old man opinion.

Think the cadence of new exploits is perhaps a good measure of that rather than subjective thoughts by anyone regardless of experience.

toraway2• 2 months ago

Okay, but none of that is actually responsive to what the article is discussing, which is competitive CTFs. There's not a single criticism of using AI for actual security research in anything they wrote and they mention being a heavy user of GPT-5.5 and GPT-5.5 Pro so belittling the author's experience to defend LLMs wasn't actually necessary.

himata41132• 2 months ago

I was writing an obfuscator recently, I just had the model deobfuscate and optimize the code back to original and I kept improving the obfuscator until it couldn't. The funny thing is that after all this I also ended up with a really strong deobfuscator and optimizer which is probably more capable than most commercial tools.

The solution is just to make CTFs harder, but when do CTFs become too hard? Maybe the problem is that 'hard' CTFs are fundementally too 'simple' where it's just a logic chain and an exhaustive bruteforce towards a solution since there really are limited ways to express a solution in plain sight.

Or maybe human creativity has been exhausted and we're not so limitless as we thought. Only time will tell.

I had another idea spring to mind: we could hide two flags, one that could only be found by ai agents and not humans or tools written by humans.

koolala2• 2 months ago

A portion could require astral projection and computers can't do that. Or maybe just a VR mini-game like the 90s always imagined.

himata41132• 2 months ago

bringing CTF solutions into the real world is a really good idea! I didn't even think of this until you mentioned it.

we have very powerful simulation tools so something like "project a pattern at these angles" wouldn't really work as you could simulate that.

I guess something cool is that we can make simulating the solution very expensive, but in real world it would be free since it's analog... As long as simulations take longer than it takes for a human to find a solution it would be a pretty good way to deal with it. I am sure people smarter than me can come up with something.

Maybe I was too early to dismiss human creativity.

a_vanderbilt• 2 months ago

I used to help build the CTFs for BSides Orlando. I ended up moving to another con, and at our last event we collected extensive logging for post mortem analysis.

We found that AI usage is basically guaranteed now, but certain challenge designs did thwart it. Challenges built with temporal visual elements made AI fall flat on its face, as it could not ingest/process the data fast enough to act on them in time. We also found that counterfactual challenges (ie. the result you get did not match what we suggested you'd get) made AI-assisted solve time slower compared to pure humans, indirectly penalizing over-reliance on AI. Multimodal challenges combining audio and visual elements were also very effective, but were not as accessible to players.

This paper gave us some ideas about designing those challenges: https://arxiv.org/pdf/2308.02950.

For our next event we figured out a way to thwart AI in our CTF: embed the CTF in a game engine. The loop essentially becomes something like this: Connect to a simulated access point in the game, the K8s cluster connects their attack container to a private network with the challenge box(es). Hacking the boxes doesn't render a flag, but rather changes in game state. AI did very poorly coping with this in our testing, as it can't derive the spatial state of the game world very well and it soft decouples the inductive reasoning loop it relies on to know if it is on the right track.

The downside to this approach is it is far more labor intensive for CTF organizers, and requires players to have a computer capable of running the game. We are also betting on AI to not advance enough by the time we ship to be able to just ingest the entire game state in realtime and close the loop that way.

Trung0246• 2 months ago

Interesting, what I just did recently is basically the same of this as I tried to push the limit of js obfuscator as much as possible by keep forcing gpt/claude deobfuscate final output then having gpt improve the tool to break the deobfuscator.

Do you publish it somewhere? Here's a sample my my js obfuscator output: https://gist.github.com/Trung0246/c8f30f1b3bb6a9f57b0d9be94d...

amingilani• 2 months ago

I don’t think CTFs are dead, they’ll just evolve. The difficulty level will need to be increased or the rules locked down. Just like sports and racing persist despite the existence of performance enhancing drugs and rocket technology.

I just did a CTF where I was in the top 10. It was the first CTF I completed and I used AI because the rules permitted it. That said, I couldn’t solve all challenges.

But yes, it was significantly easier now than I last attempted one. Even manually solving with AI assisted assembly interpretation was much easier.

mort963• 2 months ago

Increasing the difficulty level is a terrible solution. The problem with CTFs isn't that they're too easy. Making them harder just makes them even less accessible to people who don't cheat. It'd be like seeing people who put hidden electric motors in their bikes during Tour de France and conclude, "oh we just need longer distances and steeper hills".

StrauXX2• 2 months ago

LLMs don't tend to help much when solving challenges beyond their skill level. Either they one-shot a challenge, or thei are almost useless as a companion for them.

viccis3• 2 months ago

Exactly. The whole point of CTFs is that you could start on a simple one (CSAW was usually my go to one to recommend) as a complete novice who'd never done a second of computer security work and, after a few days of 8+ hours of running into concepts you hadn't encountered, googling, reading tutorial, practicing, overcoming the challenges to get a flag, etc., you'd come out the other end knowing a solid bit of security practitioner basics and likely whether you'd like to continue. Then you could keep going upwards and onwards. I went from 0 knowledge to a nice job in the field in a year.

Raising the difficulty only matters for the (imo) less important part: the dick measuring competition between the very top teams.

The actual point of CTFs was usually to keep your skills sharp and stay learning. Eventually you build your own challenges, thereby completing the "have it taught to me, then do it myself, then teach another person" three step process towards mastering concepts.

You can just say "let the people who want to learn from it do so" but honestly the entire culture of learning in the US at least is DEAD. We turned "education" into a rote system of maximizing incentives to the extent that that's all the youth know it as, and (increasingly) all educators can do. It's just gone without some kind of major reckoning, and we all know things will just collapse before that happens. The ball is in the court of whatever country can learn how to force its youth to learn the real way and use AI productively only AFTER learning the concepts it's being used to accelerate.

acters2• 2 months ago

When ctf organizers attempt to make a challenge "harder", I find they push the challenge into a more "guessy" state. Instead of proving skill, you basically need to guess some obscure or random step in the puzzle that the challenge is meant to give you. It is one of the most common problems with any puzzle based challenge system.

Retr0id6• 2 months ago

That doesn't work. The thing that made CTFs fun is the fact that the challenges are solvable in a short-ish timeframe, usually a day at most, if you have the requisite skills and talent.

rfoo• 2 months ago

Glad to hear that as I have some fun challenge ideas that would be otherwise too tedious to solve.

ec109685• 2 months ago

The issue is they become pay to win, which just isn’t as much fun.

raphman• 2 months ago

Interesting and well written article that mirrors/foreshadows how LLMs do and will change other scenes.

As I don't know much about the CTF scene, I looked for other takes on this topic.

Here's an article from 2015 about how tool-assistance already changed CTFs:

Individual skill will undoubtedly be a factor next year. But, I'm left wondering whether next year's DEFCON CTF will tell us anything more than how well-developed each team's tools are (and how well they can interpret the results).

https://fuzyll.com/2015/ctf-is-dead-long-live-ctf/

But there are quite a few recent (2026) articles with the same core message as in the original article, e.g., https://blog.includesecurity.com/2026/04/ctfs-in-the-ai-era/ or https://k3ng.xyz/blog/ctf-is-dead

And here's someone explaining how Claude Max allowed them to win CTFs:

I had always been interested in CTF as one of the only ways people could compete and show off their skill in coding/problem solving on a global scale. It was just too difficult and didn't make sense for me to learn the fundamentals as an electrical engineer. As time went on, I got better and better, and it was hard to tell whether it was because of experience or if it was because of improvements in AI.
I accomplished my goals, and for that reason I'm quitting CTF, at least for now. [...] I'd like to think I highlighted the problem before it became a bigger issue. So, how do we fix this? Teams and challenge authors losing motivation is not good. CTF dying is not good. AI bad. Or is it?

https://blog.krauq.com/post/ctf-is-dying-because-of-ai

The only article that saw LLMs as a non-negative force for CTFs was this one. Fittingly, it sounds like LLM output ("Let's be honest", "This is where things get interesting.") and only contains hallucinated references.

https://caverav.cl/posts/ctfs-not-dead/ctfs-not-dead/

utopiah2• 2 months ago

Right, the same way that car racing has "broken" jogging. This is so dumb. /s

The whole point of competitions is to provide a safe environment thanks to a set of rules all participants AGREE on in order to progress together.

If new tools "break" the competition, we change the rules and that's A-OK.

CTF isn't a natural phenomenon, if tools change, rules change, simple.

swiftcoder4• 2 months ago

The only way this actually works is if you move CTF to in-person only. There's no other way to reasonably prevent the whole leaderboard being taken up by whoever spent the most on tokens.

utopiah2• 2 months ago

Sure, I don't know how to make it work. I just know that DeepBlue didn't kill competitive chess. We simply have at least 3 different rule sets, namely

- no computer assistance, which does also mean no mobile on competition, human only

- advanced chess with assistance

- computer only, no human assistance

and arguably chess itself is not doing worst since.

rqd3• 2 months ago

tldr; adapters took my elo

r4indeer• 2 months ago

I'm conflicted on the use of AI in CTFs. On the one hand, they are supposed to mirror real-life scenarios, so of course you should be able to use any tool that would be available to you in real life.

On the other hand, CTFs are fundamentally a game and a competition which are supposed to be fun and compare and improve ones skill. So when I let an LLM generate the entire solution for me, what's the point anymore? I did not learn anything. I did not work for that place on the leaderboard, I just copied the solution. And worst of all, I did not have any fun. It's boring.

So how does using AI as a solver not feel like cheating?

baq• 2 months ago

Replace ‘CTF’ with ‘high school’ or ‘university’ and you’ve described the total slow motion collapse of education; the only saving grace is that most of it requires in person presence.

We’ve figured out the human replacement pipeline it seems, but we haven’t figured out the eduction part. LLMs can be wonderful teachers, but the temptation to just tell it ‘do it for me’ is almost impossible to resist.

daymanstep2• 2 months ago

Wonderful teachers that give unreliable information with total confidence?

k__• 2 months ago

Anti-intellectualism is at it again, hu?

Bawoosette• 2 months ago

To be fair, that was much of my actual experience with human professors in university.

victorbjorklund2• 2 months ago

Like humans.

entropyneur• 2 months ago

I had human teachers who did that in middle/high school. Took me many years to pick out all the hallucinated bits of "knowledge". I don't think the current models are any less reliable that what we currently have on average.

p-e-w2• 2 months ago

The amount of bullshit and blatant lies I’ve heard from my human teachers dwarfs the hallucinations produced by today’s LLMs.

Levitz• 2 months ago

Off the top of my head: DOMS being little crystals in muscles, tongue having separate areas for each type of taste, food pyramid, blue blood in the veins, the appendix being useless, body temperature doesn't change disregarding whether it's exposed to cold or to heat, and a whole lot of stuff related to politics and history I'd rather just omit (I don't live in the US).

All things I learned in school which were wrong information.

Not to mention, the current state of education is far worse. I don't think most realize how low the bar is.

autoexec• 2 months ago

They'll also encourage and praise you even when you're heading down the wrong path until you think you've uncovered the secret of the universe or proven that established science was wrong this whole time when really you've just been bullshitting with an engagement bot.

pjc50• 2 months ago

"Education is just a CTF for the valuable flag of a credential. In this essay I will --"

mold_aid4• 2 months ago

LLMs can be wonderful teachers

Are they or aren't they

p-e-w2• 2 months ago

A million times better than any human teacher I’ve ever had, for sure.

Now I’m certain that there exist those mythical human instructors who can do better, but that’s not worth much if 99.99% of people don’t have access to them. Just like a good human physician who takes their time with the patient is better than an LLM, but that’s not worth much either given that this doesn’t match most people’s experience with their own physicians.

IanCal5• 2 months ago

They can be incredible. One on one teaching with an infinitely patient teacher who can generate interactive problems on the fly, for dollars a month? Wild. A year of paid ChatGPT would pay for about 9 hours of cheap tutoring here.

tardedmeme6• 2 months ago

As usual it depends. When it does well it's because it can do well. When it does poorly it's because you're prompting it wrong.

thin_carapace2• 2 months ago

hammers are both a great tool and a deadly weapon at once

viccis3• 2 months ago

Mostly, no. They will explain things to you and you'll feel like you understand them. When you have to do it, though, you'll find you're not any better off than when you started.

I used to see this with students in calculus who abused the tutoring resources. They'd have tutors just work problems (often their homework...) in front of them. "Ah! Obviously that trig substitution integral worked that way. Oh, of course, that proof is very obvious in retrospect." And then they'd walk away from the exam with a 30% and no idea how their 20 hours of "study" for it didn't result in the same performance as their peers who worked problems, read the materials and asked questions, etc., got.

Most AI use is that same in my experience. "Show me how the fundamental theory of calculus works." The LLM puts together a very elaborate and flashy presentation that they skim. Great. That's no different than reading a text book. Even if you ask the LLM questions and have it elaborate on things, you've never once done one of the most important things a student can do: spend time confused trying to work hard at understanding something that's not obvious. The LLM will make it obvious at every point. Total lack of friction. Works about as well as a spotter who does the lifting for you.

magic_hamster• 2 months ago

Education is also figured out. You just need to learn, do and practice for yourself. Telling the agent "to just do it for you" is tempting, but it's not learning. You need to be deliberate when you're trying to actually learn and internalize.

Also, you could spin up your own educational agent with very strict instructions on guiding the user instead of just doing the work. Of course you can always go around it but if you're making an effort to learn, this is a good middle ground.

repelsteeltje• 2 months ago

I found this interview [0] on the subject of AI in CS education on the Oxide & Friends podcast very illuminating. Of course, Brown University CS != All education, but interesting angle nevertheless.

[0] Episode webpage: https://share.transistor.fm/s/31855e83

Gigachad7• 2 months ago

We are interviewing for a software dev role and we made the first round in person to prevent cheating. The gap between people who learned pre ai vs post is immense. I had a dev with supposedly 3 years experience and a degree in software who wouldn't have been able to write fizzbuzz without AI.

Retr0id6• 2 months ago

I had a dev with supposedly 3 years experience and a degree in software who wouldn't have been able to write fizzbuzz without AI.

If you remove the "without AI" and the end, I've been hearing similar anecdotes about fizzbuzz for years (isn't the whole point of fizzbuzz to filter out those candidates?)

IanCal5• 2 months ago

Can’t say you’re wrong but the last anecdote describes many I’ve had to review for jobs long before LLMs. Fizzbuzz is a classic thing that shockingly many devs genuinely cannot do, even at home.

baxtr• 2 months ago

I wonder if you’re filtering for the right things.

We usually hire for problem solving capabilities and not so much for technical know-how.

That’s at least how I read your comment.

andai2• 2 months ago

That's actually the origin of FizzBuzz! A puzzle invented to weed out the perplexing multitude of CS graduates who apparently cannot program.

https://blog.codinghorror.com/why-cant-programmers-program/

brookst5• 2 months ago

Isn’t this like interviewing accountants but prohibiting use of calculators or spreadsheets?

I don’t care what someone can do without the tools of their trade, I care deeply about their quality of work when using tools.

skeptic_ai• 2 months ago

I developed for 15 years. I don’t think I can do with AI anymore. Why would I even want to do that? It’s like telling a car driver to build an engine.

mannanj4• 2 months ago

I first did fizz buzz about 10 years ago fresh out of college. Now, after 10 years in full stack and fully vibe coding, I forgot basic python syntax. An interview like yours would have false positives if you are checking for syntax because well, its like looking up spelling, I just ask the AI for the syntax inline.

Glohrischi• 2 months ago

Don't worry, i never thought I would see someone unable to write fizzbuzz, but it happened 9 years ago.

Also how many people work with linux and can't tell you what 'ls -alh' is doing is staggering (lets ignore the h, even al people struggle hard).

People working with docker for YEARS and don't even understand how docker actually works (cgroups)...

Interviewing was always a bag of emotions in sense of "holy shit my job is save your years to come" and "srsly? how? How do you still have a job?"

amarant• 2 months ago

Meh. Before AI I've had "senior" colleagues with 10 and 8 years experience each, doing pair programming for 2 days straight, and in that time they hadn't managed to checkout a new branch in git.

It's not even that they got distracted, they sat there trying, for 2 whole days, with concerned colleagues giving them hints like "have you tried checkout -b"... They didn't manage!

How the hell do you work for a decade in this business without learning even the most basic git commands? Or at least how to look them up? Or how to use a gui?

Incompetent devs is not a new thing.

otabdeveloper4• 2 months ago

The best frontier LLMs can't solve 4th grade math homework yet. Don't hold your breath on that collapse of education.

(Real mathematics problems, not American-style ""math"".)

npilk• 2 months ago

Do you have an example of a 4th grade problem in mind that isn't "American-style"?

andai2• 2 months ago

They were a forcing function for skillz and they no longer are. We need new forcing functions for skillz or we will become WALL-E blobs.

Well, they were ostensibly forcing functions... ten years ago everyone was paying the exchange student to do their homework and assignments for them, and that guy was paying his cousin back in his home country, but the whole thing is a bit more efficient now.

amazingamazing• 2 months ago

We’ve figured out the human replacement pipeline it seems, but we haven’t figured out the eduction part.

No we have not.

aschla• 2 months ago

We've already had consolidation of education for a while now. Even before all the edutech courses, there were Youtubers educating better than many university professors. 10-15 years ago students were already skipping lectures and just showing up for tests.

jaybrendansmith3• 2 months ago

Everything we've learned in the last 10 years is telling us that computers do not help human education in the slightest. We remember better when we write with pen and paper. We learn better with whiteboards and paper books. The simple answer: Remove most computing from education entirely. Blue composition books, pencils, whiteboards is what trains humans. Calculators are helpful perhaps but it is quite possible that slide rules are better. We need humans that can critically think from first principles to counter the recycled information generated by AI.

__MatrixMan__2• 2 months ago

I think this overlooks the potency and scarcity of 1:1 time with the teacher. If you've only got maybe a few minutes of that in an average schoolday there's a huge difference between whether or not you've talked it through with an AI before trying the question out on the teacher.

They're wrong sometimes, but usually in verifiable ways. And they don't seem to know the difference between medicine and bioterrorism, so often they refuse. But these limitations are worth tolerating when the alternative is that our specialists in topic X are bogged down by questions about topic Y to the point where X isn't getting taught.

skulk• 2 months ago

computers do not help human education in the slightest

I had no access to anyone who could teach me calculus as a kid except Khan Academy, so I think this is a gross exaggeration. But I agree in the end, that all my "real" learning did come from pen-and-paper practice, not watching videos.

peter-m80• 2 months ago

I learned calculus thanks to wolfram alpha step by step solving feature

jaybrendansmith3• 2 months ago

I'm not going to disagree with step by step videos ... those are a HUGE help. I'm really talking about solving problems using pen and paper, whether math or writing, is how my problem-solving patterns actually changed.

ralph84• 2 months ago

humans that can critically think from first principles

This has never been achieved by, nor is it the point of, education for the masses.

sometimelurker• 2 months ago

I don't think computers automatically make us more educated, but if you want to make a point don't use reductive exaggerations. > We need humans that can critically think from first principles to counter the recycled information generated by AI.

I agree with this.

PunchyHamster4• 2 months ago

I disagree with that statement. There is nothing inherently wrong with using computer to learn and if your personal goal is to learn it in lot of cases makes it much easier, whether to search for or visualise a piece of knowledge you're' learning.

The problem is frankly computer and now computer with LLM makes it easy to cheat.

The kid doesn't want to learn, the kid wants good grades so parent is happy with them, and the young adult wants to get the paper coz they were told that is required for good life. It's misalignment of incentives.

tempaccount5050• 2 months ago

Nah, I wrote physics programs on my computer at home in high school and it absolutely helped with my schooling. Yeah, maybe iPad apps aren't the best things in schools but you're throwing the baby out with the bathwater. Computers bad is simply not true.

wslh• 2 months ago

I would start saying that many people need presence in a real environment with people to learn. We don't use all our senses in a remote environment.

UltraSane• 2 months ago

Smart people will use LLMs to learn things faster. Education will adapt by doing all assessments in person.

djoldman• 2 months ago

Replace ‘CTF’ with ‘high school’ or ‘university’ and you’ve described the total slow motion collapse of education; the only saving grace is that most of it requires in person presence.

So something like, "Frontier AI has broken the 'high school' or 'university' format"?

The hype surrounding AI is just pervasively exhausting: you've got the folks talking about an entire new age for humanity where we're shortly going to take over the entire universe. And you've got the folks talking about how our entire society is crumbling.

Education is one place folks seem to throw up their hands and say nothing can be done.

The fix is simple: students are to be evaluated on their performance in person. That's it.

Any other "collapse of education" isn't due to AI, it's something else.

HPsquared• 2 months ago

In my university education (2007-2011), 80% of the grade was based on exams at the end of each year, with no resits.

AndrewKemendo• 2 months ago

I started teaching “how to build quality products using LLMs” full time recently, and most of what I teach is literally just the 101s of systems engineering, reliabily engineering, product development and project management:

Exceptional clarity on the problem you have

Know how to measure the problem you’re solving

Numerically define what “done” is

Make a deterministic and fully observable prototype

Iterate in production with the user

Expand user base as desired with user iteration in parallel forever

Etc…

Obviously a lot more in the details and these are all case by case, but these chatbots are basically perfect productivity machines for this process.

The massive caveat to all of this is this only works for people that can reliably and truthfully define those items above, are willing to structure organization to make those your priorities.

And actually most financial incentives demand the opposite of this process

If most organizations were honest about it, they would simply say “we’re here to make the most money possible and we’re gonna do whatever it takes to do that”

A lot of people don’t like that, so they don’t say it to come up with other bullshit.

Ultimately that’s why I felt like my only option right now is to teach people how to do this because I assumed it was obvious and it is not.

ori_b• 2 months ago

You haven't explained why anyone should value education in the world we're building, other than as a hobby.

SoylentOrange• 2 months ago

Great article, well written, and good analogy to chess. I’ve been playing competitive chess most of my adult life and I think that the solution lies in how chess dealt with this problem:

Explicit ELO measurements with some cheating detection. AI assistance wholly banned. As you climb the ELO ladder, detection gets more onerous. At top level during online events, anti cheating teams require the use of both monitoring software and multiple cameras.

Idea is that you can cheat pretty easily at the lowest levels but it gets less easy the higher you go. This allows for better feeding into the truly elite competitions.

I think chess’s very firm stance that AI is never allowed in competition (neither online nor in person), rather than CTF’s acceptance, was the right call.

salt4034• 2 months ago

Yes, chess has been dealing with AI for decades at this point, and it's amusing/frustrating that so many other communities are deciding to re-discover everything from scratch, rather than just learn from the chess experience.

If CTF is a player-vs-player event, then AI should just be banned outright, otherwise it will devolve into AI-vs-AI, which is just not an interesting competition format, as we learned in chess. Compared to FIDE top events (which bans AI), only a tiny niche audience actually watches the Top Chess Engine Championship (AI-centered). It turns out what we care about is not whether chess can be solved by any means available, but what are the limits of the human mind in learning chess.

Pretty much all chess coaches/educators also warn against relying heavily on AI during learning; engines only give you an illusion of understanding.

3qw128• 2 months ago

The article is the thickest of AI slop. Don't believe anything.

sevindob• 2 months ago

ikr, if bro can't be bothered to write an article himself then anything he says is automatically suspect

hoyd• 2 months ago

«That feedback loop is breaking. If the visible scoreboard is dominated by teams using AI, a beginner is pushed toward using AI before they have built the instincts the AI is replacing. That is an anti-pattern. It prevents active learning, and active struggle is the bit that actually teaches you. It is also completely demotivating to put in real effort and see no visible progress because the ladder above you has been automated.»

This stands out to me, and speaks perhaps broader than the article itself? I’m sure this has been in the spotlight before, but well put for many areas I think.

black_knight• 2 months ago

I see this with beginner programming students at university. They get AI to help them with assignments, with the intention of learning, but ultimately they do not get the understanding they would have if they had done the assignment themselves. Then they are at a deficit for learning more advanced topics.

My fear is that they never get to the level they need to be at to create good software even with the help of AI. So, although an expert with AI can create great software, that is not where we end up. In stead we will have vibe coded messes by people who barely have any grasp of what is going on.

chrismorgan• 2 months ago

Meta: this was submitted with the article’s title “The CTF scene is dead” which I found very easy to understand. It has just been updated to use the subtitle’s first sentence, “Frontier AI has broken the open CTF format”. I find that much harder to grasp, rather like a garden-path sentence. My immediate thoughts were that “Frontier” was a company name, and that there was some file format named CTF. If you don’t know about Capture The Flag contests, the change doesn’t help. If you do, I think the change makes it worse.

IanCal5• 2 months ago

If it helps I understand the second much better and feels less clickbaity and includes more info. I do agree with the points you made about the confusion although I find frontier a term used in this area a lot, “frontier AI models have” would probably resolve that.

jofzar2• 2 months ago

Imo frontier is too niche and specific, if you know what a frontier model means then it's fine, but if you don't then it's negative/detrimental to the title.

"new" does the same thing and is probably just a better descriptor then frontier

Jenk• 2 months ago

If the title simply said "AI is out-performing humans at CTF" then none of this confusion exists. Nothing is "broken," we don't need to be superfluous with "frontier," and the point is still there.

jsoaoxhd• 2 months ago

Why do people always hijack threads to discuss titles? Most articles have terrible titles. Just downvote it and move on.

dandellion• 2 months ago

Why do you contribute to making this thread longer? Just downvote an move on.

KomoD• 2 months ago

You can't downvote a submission.

keeganj• 2 months ago

I agree, it took me a second to parse. It may be because this is the first time I've seen "frontier models" described as "Frontier AI". That sounds more like a company name, especially when the F is capitalized.

SomeHacker44• 2 months ago

The article never defined CTF. Nor have the top comments here. Skip.

Basic rule: define every abbreviation when it is first used.

gbnwl• 2 months ago

Frontier as in "Frontier Model" is a legitimate vocabulary term you should probably be aware of in 2026. It's not something the author made up or chose randomly, it's common parlance in the space.

motbus3• 2 months ago

I think soon there will be ways to trick this models and I think when it happens it will be yet another layer like aslr

These models seems completely unbeatable only in the ads. There are 100+ times way someone puts Hindi Yoda talk In Morse Code and it goes nuts. The reason they are going to hard for PR Marketing on this is because they know it is a matter of time.

Avamander3• 2 months ago

The more you obfuscate a topic against LLMs the lower the educational value of a challenge.

The only things that works is novelty and obscurity. LLMs still suck with things mentioned in the footnotes of datasheets and manuals, things that deviate in subtle ways, unique constructions that alter something very very common. It's hard for LLMs to avoid common pitfalls in terms of making assumptions, while staying on track.

SirHumphrey• 2 months ago

Competitive programming scene always included offline competition and with AI they are becoming more important (and in general they were more fair even before). If CTFs are to survive, they should probably try to adopt this strategy.

You could even go so far that anything loaded on your computer is fair game, but not more than that (certain competitive programming competition for example allow unlimited amount of paper material - for CTFs you probably need much more than that, therefore electronic).

virtualritz2• 2 months ago

Chess and Go are not dead just because Ai got better than humans at these games.

What am I missing here?

hnlmorg• 2 months ago

You aren’t allowed to use tools to play competitive Chess / Go but that are required for solving CTF.

jofzar2• 2 months ago

These have very strong anti cheats and in person is very stringent on no electronics.

Its not really a good comparison

lugu• 2 months ago

Read the article.

artninja1988• 2 months ago

https://news.ycombinator.com/newsguidelines.html

"Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that"."

virtualritz2• 2 months ago

I read the article. Their chess section makes no sense as in "why this wouldn't work for CTF".

But I don't know enough that's why I asked.

I imagine one could do CTF in public, machines you work on vetted/prepared to some spec, yada yada.

If chess and Go can do it why can't CTF?

That was my question when I wrote "what am I missing here".

aymenfurter• 2 months ago

Chess banned engines from competition. CTFs can't really do that because you need internet access and tooling to play.

vagab0nd• 2 months ago

This left a strange feeling. The article reads as extremely bleak. But from a different perspective this is extremely bullish for AI.

Avamander3• 2 months ago

LLMs managing the "coloring book" equivalent of something is not bullish for the "art" version of something.

The intent for most CTFs is to provide a meaningful challenge that concerns a single topic without introducing noise that wastes time. Of course a training exercise is easier to complete for an LLM.

kangalioo• 2 months ago

I agree. The article mourns the death of pentesting as an art form due to automation. But you could also celebrate the death of pentesting as an arduous necessary evil due to automation

lokrian• 2 months ago

Is AI also superior to humans at black box challenges and attacking actual targets on the internet? That seems like a really important question.

Avamander3• 2 months ago

No, the search space is much more vast and the feedback loop almost nonexistent.

The reason LLMs can do CTFs so well is partially because the challenges are usually designed to avoid wasting time and to introduce a single concept without noise.

slurpyb• 2 months ago

How to motivate cybersec best outcome reddit 2026 no mythos

parasti• 2 months ago

I can't help but draw parallels with video games. Aimbots in competitive multiplayer games is a well defined issue: it's considered cheating and frowned upon, players caught cheating are banned from the game. Tool-assisted speedruns (TAS) where a player attempts a world record at completion in a single-player game is another face of the same concept (computers help you win), but one that is socially accepted as long as runs are clearly labelled as TAS.

ViscountPenguin• 2 months ago

The biggest difference would be the fact that you can discover video game cheating through some kind of trace. Speed running communities go pretty hardcore on that kind of thing nowadays.

It's a lot harder to detect cheating when your only trace is how fast someone submitted the string CTF{DUck1e_Pwned}

justanotherjoe• 2 months ago

Sure if the goal is entertainment and sports, you're right. However, unlike chess or counter strike it's downstream from a real needed utility. Like, is there a point to do it anymore? (ofc there is, but still, it's been devalued from the perspective of the 'real utility')

nrabulinski• 2 months ago

It’s literally not. The most interesting and satisfying CTFs have never been grounded in reality, it’s just been an expression of mastery, both from players and authors, with a few notable exceptions. But they’re that, exceptions, not the rule.

tptacek3• 2 months ago

Aimbots in competitive multiplayer games are (almost always) game-breaking abuses. CTFs have always rewarded tooling and automation. They're different cultures.

copx• 2 months ago

If adaptation means accepting that the scoreboard is now an AI orchestration benchmark, then we should say that honestly instead of pretending the old competition still exists.

This is like someone complaining that making machine parts has been ruined: Skillful craftsmen used to make them by hand using manual tools!

Nowadays the CAD/CAM/CNC cheaters have almost completely automated the whole thing. How is the next generation of craftsmen going to learn how to craft a gear by hand when the process of gear making has been reduced to pressing start on a CNC machine?!

See what I mean? Sorry, I think this article is just Luddite. I can empathize with the pain of your beloved craft basically being rendered obsolete by new technology, but the process can neither be stopped nor is it bad in general.

The manual skills you trained with CTF puzzles are now simply no longer relevant . (Field-specific) "AI orchestration" is the new cyber securtiy skill if LLMs really have become so good at this, and what the author used to do manually then has the same value as being able to craft a gear by hand.

raddan• 2 months ago

The way I read the post is that the author is disappointed that the community is gone. The CTF was just a reason for a number of like-minded people to organize around an activity.

Indeed, in the real world, plenty of people organize to do formerly-skillful tasks together. I have not personally crafted a gear by hand, but I have built a house in a long-abandoned style with a group of people only using hand tools.

There _is_ a danger that society forgets how to do these things. During that house-building exercise, there were many tricks of the trade that, while likely documented somewhere in a book, would have been difficult to reproduce without seeing a demonstration. From the standpoint of “does it matter?” it depends on what you care about. We absolutely do not need cruck-framed houses with scribed joints. Modern construction is faster and cheaper and lasts long enough. But it would sadden me greatly if practices like this faded from memory, because it’s one of those things that makes you gasp “wow!” when you see it. And your appreciation only deepens when you try it yourself.

toraway2• 2 months ago

Just parachuting in to reflexively throw the "Luddite" label at someone lamenting the decline of a niche community they've enjoyed participating in and contributing to is certainly ... a choice.

Within the framework of your analogy, it's like responding to someone active in DIY maker groups suddenly dealing with an influx of influencers in meetups showing off Chinese junk from Etsy to post on Tiktok, and accusing them of being a Luddite blinded by their zealous hatred of mass production -- both strangely abrasive and also fairly nonsensical except as a "mass production supporter" social signifier.

Not to mention, in the article they specifically describe themselves as a heavy user of frontier models for security research ever since the release of Opus 4.5, calling them "useful within the field". In fact I don't see any actual criticism of AI/LLMs anywhere whether for security research, programming or anything else, except for making competitive CTFs no longer viable.

What does it take to avoid the "Luddite" brand? Using AI themselves and praising AI as useful (to the point of having a lopsided advantage over humans) isn't enough? Do they also need to say "I haven't written a line of code in 6 months/it's easily a 100x multiplier for my job" every time they mention it too?

spacedcowboy2• 2 months ago

The first paragraph on anything with an acronym in it should explain the bloody acronym. I assumed CTF was an encryption standard, given the headline. It was only coming here and reading the comments that made me realise it's a game-format ("Capture The Flag").

jaffa22• 2 months ago

Capture the flag the only expansion of CTF that i know but even if it is capture the flag this still doesnt make any sense. Like Quake CTF?

arm• 2 months ago

https://en.wikipedia.org/wiki/Capture_the_flag_(cybersecurit...

msm_2• 2 months ago

I don't know what to tell you. If you don't know what "CTF" is you're not the target of this blog post. It's like stumbling upon article "What's new in HTTP/2" and complaining that "HTTP" acronym is not explained.

I don't mean that everyone must know what CTF is, but sometimes it's OK to write things just for your community (CTF community in this case), not for general population.

spacedcowboy2• 2 months ago

Ok, so picture the situation:

1) You see a headline on HN about some open format being broken by frontier-level AI. You don't recognise the acronym.

2) You visit the site, you read the first few paragraphs, you still have no effing clue what the site is talking about

3) You come back to HN and read the comments to figure out WTF is going on. Oh, it's just some game style, so not a Cryptographic Trust File, or something you have to care about after all.

The point is you can't know what some opaque acronym is about until you visit the source of something that will hopefully explain this opaque acronym. Leaving the site being still none-the-wiser is a failure of the site, in my view. If you don't agree, that's fine, we're adults, we can differ, but it seems like a valid complaint to me.

FWIW (this means "for what it's worth" :) I'm not railing against acronyms in general, and HTTP is probably one of the most-used ones on the internet so I'm not sure it really applies as a good counterpoint. Using CTF without an explanation is more like using SSTP (Secure socket tunneling protocol) without one, IMHO (this means In My Humble Opinion :) ...

jimnotgym2• 2 months ago

You can still do competitions. But you'll all need to fly to the same place and work on laptops with a fresh install of Linux. 1 hour to install tooling then Internet off, challenge revealed.

Not as easy logistically...

xiphias2• 2 months ago

,,a beginner is pushed toward using AI before they have built the instincts the AI is replacing. That is an anti-pattern.''

The same article talks about CTF skills as a way to learn about security best practices and separately a sport.

In reality it was all about learning an extremely important skillset (securing/attacking software and systems) that is getting automated.

The real thing the author seems to be frustrated about is AGI is coming in computationally verifiable domains first, and lot of his skillset was taken over in a big part.

TrackerFF• 2 months ago

Question: Was this website made with Claude?

I've seen that exact font and color scheme a dozen of times the past weeks.

saidnooneever• 2 months ago

Do CTFs like Lan parties or factor in new tooling avalable to people. change is not death. or death is not an end. either way, people will enjoy applying and showing off their skill. competing with eachother on a human level,.with or without ai tools.

JackSlateur• 2 months ago

No relationship with the CTF (Common Trace Format) format ..

petterroea• 2 months ago

I helped arrange my country's longest living CTF this year. Our CTF is *made for amateurs*, but we always have challenges for intermediate to skilled players and the top of the scoreboard is usually topped by them. It is the compromise we have - amateurs get so many tasks they struggle to solve them all, and the pro's get to win. Our goal is to nerdsnipe people who are curious into trying our CTF by offering easy beginner tasks, and then get them hooked enough to stick around for the intermediate ones, even if it takes them a day to solve one.

This year, multiple groups on the top of the leaderboard were clearly abusing LLMs. You can tell because they know nothing of what a CTF is nor the terminology, nor really the fields the challenges were about when they were talked to. They were obviously amateurs.

It was pretty depressing to hear how unaware they were of how obviously they did not fit in to the type that usually is on the top of the leaderboard. It seems they seriously think they were under the radar. If it was one group it could be a freak incident - some times someone just shows up and curbstomps competition. But there were many groups like this this year. They also had a certain smugness to it - one staff reported that a group was hinting to other teams about their "super weapon". Another group credited their "secret third team member they didn't want to talk about".

I use LLM frequently and experiment with it a lot, both at work and on my free time. Nowadays they are good enough to have value and I am interested in learning more about that. They let me spend more time on hard problems and avoid spending the day on simple CRUD. I say this to say that LLM doesnt have to equal bad, it is a tool, that's all. However, I generally avoid LLM communities because many LLM fans are lazy and unskilled people who are just happy they can feel they are worth something even if they have no skill. They don't really have much to provide of conversation. If anything, from reading the CTF crowd this year, the rise of LLMs has just meant more of these people can stomp on and harvest the CTF scene for self validation.

This is not me trying to gatekeep who can play CTF. Anyone is welcome, but there is one condition: You are here to learn and have fun.

The conclusion many I talk to has come to is that nowadays, it is harder to learn to put in hard work and become good at something because there are just too many ways to cheat and take shortcuts. I suspect in the future there will be a shortage of useful people - the kind that have critical thought and know the value of doing something properly. This doesn't mean "Not using LLM", but as said by many on HN before you need a certain seniority before LLMs are useful augmentations to your skills and not just stopping you from learning yourself.

I agree with the article. Anything but physical competitions with strong security - think professional e-sports with organizer-provided PCs, is over. But I think one of the most interesting things to take away from my CTF experience is that the bottom of the leaderboard was still full of amateurs slowly working their way up - it is a few rotten apples that ruin the fun for most, and there are still plenty of people who want to learn and deep-dive.

Gathering6678• 2 months ago

I thought a company called Frontier broke a file format CTF.

dostick• 2 months ago

Unable to find what “CTF” means, since it doesnt look like referring to Capture The Flag gaming

yc-kraln• 2 months ago

It does--but a particular form of Capture The Flag where there is a computer system and the "capturing" is breaking in or exploiting a security issue in that system.

bornfreddy• 2 months ago

I guess this is very similar to what happened to demo scene, in some way. The limits are what makes these problems interesting, and once we have better machines / tools, the incredible skill is no longer prerequisite, making everything less interesting for participants. Sad, but - such is life...

yk• 2 months ago

There's something funny about complaining about cheating in a hacking competition.

Well actually I get it. In cycling motor doping, putting a hidden engine into the bike, seems more offensive than regular doping. I think this is because there is a continuum from eating well to taking supplements to injecting stuff, but having a engine breaks a fundamental idea about cycling. Similar hacking is about cleverly abusing the rules.

tkel• 2 months ago

Pretty ironic that this article was also written using LLMs. It has all the LLM-isms.

qassiov• 2 months ago

Surprised to not see more discussing this. It's so grating, and nobody noticing (or believing others that say it's AI generated) makes me feel like I'm going crazy

lachiflippi• 2 months ago

The "CTF for fun" aspect has been dead ever since the winning teams had thousands of dollars of rewards waiting for them. Of course people are going to use anything that's not explicitly forbidden by the rules to win. Introducing what amounts to an "I win" button that both can't be prevented by rules and is accessible to anyone didn't "break the format" anymore than the epidemic of giant merger teams did a couple years ago, it just broke the community because you now don't have to actually talk to other people to cheat anymore.

Many CTFs have switched to a dual-leaderboard format recently, one for "agentic teams," one for the rest. If all you care about is "learning" and imaginary internet points, you can just participate as a human team and adblock the AI scoreboard, and maybe lobby CTFTime into splitting their rankings as well.

tardedmeme6• 2 months ago

When I did my first CTF, it was close to the deadline and I thought I had the extracted the flag from the program and the rest of the program was just filler, so I entered the flag, and it told me it was not the flag. It turns out the program multiplies the input by a pseudorandom matrix before comparing it against the flag, so I had to implement a matrix inversion and then get the flag. That's not the story though.

The matrix was always the same and the challenge was clearly designed so that the point was being able to read anything at all, not knowing how to invert a matrix, so I asked the creator what was up.

He told me that there were tools that would trace input values until they reached a comparison instruction, then print what they were compared against. Therefore it was necessary for every deobfuscation challenge to scramble the input in some way too complex for these tools to undo, before comparing it. Hence the multiplication by a pseudorandom matrix.

The point is, cheating tools aren't new.

mpeg• 2 months ago

Yes but you can't compare some ollydbg script that would maybe be useful in a super specific challenge to LLMs which trivialise absolutely every challenge in a ctf and are de facto necessary to compete now

simonTrace• 2 months ago

AI-generated phishing is the scariest development in cybersecurity right now. Click rates on AI-written phishing emails are 54% compared to 12% for traditional attacks. Automated real-time detection is the only scalable answer at this point

brandonwindson• 2 months ago

Click rate is higher because AI removes the broken English and generic scams. But detection is not the only answer.

Most phishing comes from a small set of hosting providers that ignore reports. The real fix is making abuse reports actionable. Fix that, and you cripple the economics of phishing. AI detection is reactive; killing the source is proactive.

not_a9• 2 months ago

I’m interested in finding out how attack-defense style CTFs are affected by slopping. ENOWARS skorbor will probably significantly differ from the last time around.

JoshGG• 2 months ago

CTF = Capture The Flag

https://en.wikipedia.org/wiki/Capture_the_flag_(cybersecurit...

lmeyerov2• 2 months ago

It's tough. We run botsbench.com , which tracks AI progress on a top CTF, and I gave a talk at CCC a few months ago on our own results doing AI speed runs, so I think about this a lot.

In our own trainings we give (AI agents for security, and a graph masterclass), we ended up leaning into it. For example, we ship with a skills bundle. There are plus sides, like less code-forward participants can go further and are appreciating that, and less of a gap between high-level concepts and successful hands-on. But at the same time, manual work does build a lot of intuition & knowledge that gets missed in auto modes.

nine_k2• 2 months ago

Will this bring back the age of LAN parties, where the LAN is disconnected from the internet, and mobile connectivity is blocked?

lmeyerov2• 2 months ago

I think that ship has sailed as well --

botsbench.com shows Sonnet 4.5+ with Claude Code harness does pretty well, and Sonnet roughly tracks the edge of what self-hosted models do on the upper tier of affordable GPUs, like running 1-2 DGX Sparks and waiting 6mo for oss to catch up a bit

lg5689• 2 months ago

This is happening to other forms of competitive programming too. The most recent AIs have problem solving skills rivaling top humans, and so if AI can't be easily banned, the competition is dominated by AI agents.

I thought code golf would take longer for AIs because there's so little training data (it's more niche), but we're seeing AIs starting to match expert humans there too. Sucks because golf has been my favorite type of programming puzzle.

It's crazy how far AIs have come in problem solving ability.

Legend2440• 2 months ago

Code golf is well-suited for AI because you have a easily verified objective (minimize code size while passing tests) and can run an LLM in a loop to churn away at it.

notepad0x90• 2 months ago

You can introduce canaries, and ban auto-pwning in general. that's usually banned anyways. Some challenges just can't be solved by a human in under a certain period of time.

Another idea is deep red herrings. solves that lead to more solves, on and on, except only if the previous solves were solved quickly. The effect will be that participants who solve things quickly will keep finding things to solve. they can't know that the path they're on will lead to victory, even if they artificially slow down, unless they consistently slow down just as a human would. It will eliminate the speed advantage. For the skill advantage, other than having another LLM procedurally generate challenges, I don't know of a good solution.

There are always things like captchas. or the good 'ol honor system. A person can spend only so much for things that have no financial reward in the end, only clout.

---

Alright, all that said, i think i really do have a good solution for this, as well as academic exams. Or I think I do, because it's so simple, I've been scratching my head as to why everyone isn't doing it already.

Require screen sharing/recording. LLMs can't fake that well enough. Have another LLM audit the video for mouse, key stroke, window movement and other details to see if it looks human-generated or not.

If a student has an essay assignment, have them record their screen as they research, and actually type out the whole thing. In the extreme, require anti-cheat proctoring software installed, as is done in remote examination. In an even more high-stakes and extreme scenario, have them share their face. Their eye and face movement, correlated with the screen-share, and correlated with the activity observed on the server end, should be pretty hard to beat, even in the next ~5 years of LLM advances.

ChiperSoft• 2 months ago

Neither the article nor the comments in this thread explain which of the many meanings the acronym CTF is being applied to...

mr_mph• 2 months ago

In this context, it stands for capture-the-flag: A type of computer security competition, usually in a 'jeopardy' style, where challenges that fewer teams have solved are worth more points.

hemlock45932• 2 months ago

I feel the post. For me AI has ruined both, playing CTFs and also building CTFs challenges. The most annoying thing to me is the "yeah idk but here is the flag" mentality.

Before when playing CTFs with my mates was usually sitting there for hours tackling a challenge until some other mate joined, had some look together and solved it with you together in 30 minutes which is the most rewarding learning experience. Nowadays mate joins in throws the clanker on it and solved it in 5 minntes. Asking on how it worked you always get the "yeah idk what it did, but who cares, here is the flag" response.

Same for creating challenges. Whenever I ask for writeups or if some people solved it differently I usually get the "yeah idk, clanker solved that one" response taking the fun out of it.

So yep, this CTF format is definitely dead. Mainly because the strong competitiveness and prices. This encourages people to cheese challenges and sometimes solving them differently was fine as you still had a creative out-of-the-box thinking moment, but nowadays with AI there is no brainpower needed, no cheesing needed, no human needed. As you mentioned, it's pay to win.

My two cents is that the 24/7 CTFs will get more attraction as the scoreboard doesn't matter there and simply doesn't give you any price.

gmm1990• 2 months ago

I don’t know like chess engines didn’t kill chess. You could just play with people that don’t use the “engine”

hemlock45932• 2 months ago

Yea, but chess adapted to it and is restricting use of engines. When you play a tournament you are banned from using a phone and will be disqualified if you do so. Online tournaments don't have a prize money for that reason, so there is no real benefit for cheating. Lichess and chess.com additionally add rankings for bots and have a strict anticheat if you use bots for regular games.

For me it feels like this is not really possible for live CTFs. In contrast to chess you can't ban AI, as live CTFs are about breaking things by design, so they'll always try to circumvent an AI ban.

low_tech_love3• 2 months ago

It’s different, unfortunately. I wish you were right. The problem is that creating interesting and fun CTF challenges is a very active, time consuming, creativity-heavy task. A chess board is always the same, and always will be, but every CTF competition is unique. There is little to no incentive anymore to spend time creating the challenges. You might say “well create the challenges and share them with people who care and who want to play honestly” which is probably the right answer here, and might happen at a smaller scale. I picture CTF in the future almost like a tabletop RPG experience, one where a small amount of people will share with close friends who they trust. But the usual “open” CTF scene (as mentioned by op) is probably over for good, if we’re being honest.

nicce2• 2 months ago

I don’t know like chess engines didn’t kill chess. You could just play with people that don’t use the “engine”

Impossible to do online, at least. People go beyond things for virtual fame and "owning" others. Even if it is based on cheating.

charcircuit• 2 months ago

Imagine giving every competitive chess player the best chess engine and letting them use it freely during matches. Would that be considered fair?

Imagine every competitive chess player being allowed to video call with a hundred other people to help them make a move. CTF have never been fair, nor has it ever been effectively structured for learning.

Nifty3929• 2 months ago

Must I beg to have an acronym spelled out a least once, the first time it's used? Even if you assume 90% of readers already know, the other 10% (including me, in this case) will thank you, it doesn't take much effort, and it expands the reach of your communication or idea.

Exceptions for cases where the acronym is just so well known that a lot of people don't even know what it stands for even though they know the concept well. I recall one corporate training I was sitting through and they used the term "Border Gateway Protocol" and it took me a half beat to think through "oh, you mean BGP?"

Thanks!

bawolff4• 2 months ago

Which acronym do you mean? CTF? I think that acronym, just like BGP, is more well known by itself than what it stands for.

More generally, not every piece of writing is meant for every audience. Like if someone writes a blog post about CTFs aimed at people who like CTFs, nobody in the target audience needs to have CTF explained to them. Ultimately HN is a link aggregator, but sometimes its a bit like eavesdropping on a conversation. When you are just listening in you don't get the full context sometimes.

doublescoop• 2 months ago

Best practice in writing about technical concepts is to spell out acronyms like this on their first use. There is a ton of stuff I learn about here on HN that I didn't know anything about before.

It doesn't help that the linked article never bothers to explain this either.

Gigachad7• 2 months ago

The annoying thing is even if you know what it means, multiple groups will use the same initialisms for different terms. So without more context you can’t know what it means.

It isn’t common but I feel it would be best when posting to HN to just expand the initialisms even if the source title didn’t.

0x20cowboy• 2 months ago

I dont know what CTF stands for so I dont know if I am interested in this article or learning anything about it. Maybe I am.

Are you really arguing for not just typing out whatever 3 words this stands for once in the name of clarity?

ergonaught• 2 months ago

You could have just said “No”, if you had to say anything at all, rather than continuing the behavior.

Actively rude.

razster• 2 months ago

What I see CTF I think Capture The Flag, Tribe player in me.

watt• 2 months ago

If you think you know what every acronym stands for, I recommend CBT.

MrSkelter• 2 months ago

It’s just bad writing. There is no such thing as a universally understood acronym. Even within fields. The cost of clarity is zero. Not elucidating means that over time the writing rots as meanings change and new “universal understandings” supersede old ones.

pastel8739• 2 months ago

I think so many acronyms have meaning that isn’t explained by the words that the stand for. The other day I was explaining what CI is and they asked what it stood for; I realized that Continuous Integration is almost completely useless for someone trying to understand what CI actually is

cco• 2 months ago

Semantic names are great, but that's a separate issue. With the full term you can now go search for yourself and find explanations more easily.

graceful6800• 2 months ago

Since this is the top comment at the moment: CTF stands for Capture The Flag.

Personally I have never, ever heard that concept referred to by the initialism. Granted, it's almost never come up in my circles, so... shrug

worble• 2 months ago

CTF is a game mode for popular online games like halo (or at least, that's how I know it), so paragraphs like

My first CTF was HCKSYD, a 48-hour solo CTF. I full solved it and won in 2 hours. I was completely hooked. That led me to win DownUnderCTF, Australia's largest CTF, with Blitzkrieg multiple times. Blitzkrieg was one of Australia's strongest teams at the time. I later joined TheHackersCrew, an international top-tier team that was consistently ranked highly on CTFTime, the main global ranking and event calendar the scene uses as its scoreboard. With them, I competed in some of the most prestigious CTFs in the world, consistently placing well within the top 10 until the end of 2025.

Are still completely nonsensical to even those that understand the acronym

bawolff4• 2 months ago

Just to give the actual answer, CTF in this context means a computer security competition. Generally the way they work, is you get some programs, and you have to hack them to get some string called the flag (e.g. maybe the server has a root owned file called flag, so you have to get root somehow to read the file). Team with the most flags at the end wins.

In this context, CTF is almost exclusively referred to by the initialism, i think to help distinguish from other uses of the term.

tptacek3• 2 months ago

Apart from everything else people have said in response to this, it's rude to presume that an article has HN as an audience simply by dint of it being available for us to link to. It's totally reasonable for people to write for an audience they know understands these terms.

So, in fact, you must not beg to have authors include courtesy definitions for you. That's not reasonable. Instead, you should simply ask here, on the thread, without complaining about the article.

xnickb3• 2 months ago

Exactly. Especially now in the era of LLMs where you can feed the article to a chatbot and ask it to spoon feed you at the pace you can digest at.

amirhirsch2• 2 months ago

Let’s reduce this to absurdity:

I think you only wanted clarification of CTF (Capture the Flag) and not AI (Artificial Intelligence) and not GPT-4 (Generative Pre-Trained Transformer version 4) and not CLI (Command Line Interface) and not MCP (Model Context Protocol) and not LLM (Large Language Model)

Quoting TFA (The Fucking Article): “just adapt bro”

lol at the BGP example

w23j2• 2 months ago

For people who are confused by this comment: lol = laughing out loud!

toofy• 2 months ago

i try not to over feed tangents but this is precisely how i feel every time i speak to someone who is recently enlisted in the military. i have to constantly stop them and be like “i have no idea what you just said” over and over and over again. it’s like trying to make sense of a random bowl of alphabet soup.

fragmede5• 2 months ago

We live in the goddammed future. Huamnity's knowledge is at your fingertips. Right clicking the Nth word of the article and putting in any semblance of effort to learn on your own is too much to ask?

I don't know everything, there's tons of stuff I don't know about, but when I'm at my web browser, the least I can do about something is ask Google about a word or phrase or subject that isn't familiar instead of being spoonfed information like I'm a baby.

fullstackchris• 2 months ago

There are also widely accepted standards to written word.

The best example is when an abbreviation can be expanded to more than one phrase, and both are widely used.

circus1540• 2 months ago

“hacker” news, ladies and gentlemen

ajnin• 2 months ago

Your two paragraphs are completely contradictory. I agree with the first one.

alsetmusic• 2 months ago

At the same time, I did a search for "what is a ctf to play" and got the answer. We know how to find answers to these problems. I agree the blog post was poor form.

mafro• 2 months ago

I didn't know what BGP is, but I did know CTF. YMMV

snayan• 2 months ago

Hah, same. Always entertaining how people map their tendencies to the world at large.

w23j2• 2 months ago

CTF stands for Capture the Flag. So with that definition you have exactly zero more information about the article then without it. So I assume next you want a short description of what it actually means, like "CTF (Capture the Flag) are security competitions where the objective is to break...", which is completely ridiculous to include in an article aimed specifically at the CTF crowd.

fartfeatures3• 2 months ago

It is easier to search for keywords (Capture the Flag) vs acronyms (CTF) which likely resolve to other terms as well. Child trust fund is the first result when I put CTF into Google. Admittedly searching CTF security solves that issue. A quick link to an article on CTF to make the post digestible by outsiders seems reasonable enough.

thomascountz• 2 months ago

   ...which is completely ridiculous to include in an article aimed specifically at the CTF crowd.

What compels you to hold this belief?

plaguuuuuu• 2 months ago

you could try

* googling "CTF security"

* asking literally any AI to explain the article

Yes, you must beg. If you don't know what a CTF is, and don't want to find out, why read the article anyway

watwut• 2 months ago

Using AI to get factual information?

Also well written articles allow you to figure out what they mean after clicking onto them. People often read to learn.

fg1373• 2 months ago

I had no idea "security" is the keyword to add, even after reading the leading paragraph and the first paragraph under "What makes me qualified to say this?".

In fact, I know what "capture the flag" is but am not familiar with the acronym. Still, the article confused the hell out of me, so I came back to HN comments for more context.

Asking AI to explain the article

That's how we are expected to consume content these days?

jere3• 2 months ago

It's kind of amusing that you're asking for acronyms you don't know not to be used and acronyms you do know to not be spelled out.

fg1373• 2 months ago

As someone who spent years reading academic journals where spelling out acronyms on first occurrence is mandatory, the comments here are jarring. Think about it -- those are academic journals where most people do know what those acronyms mean.

It's such a small but immensely helpful thing to do.

kortex• 2 months ago

This isn't an academic journal or general purpose news article though. It's a small blog about a subculture. It'd be like spelling out GPU on a blog for building custom PCs, or BDSM on a kink blog.

Spelling it out when 99.9% of your audience doesn't need it actually is the opposite of in-group signaling, it makes it feel like it's aimed at a wider audience, when it's not.

https://en.wikipedia.org/wiki/Cooperative_principle

RealCodingOtaku3• 2 months ago

The comments are annoying. No matter the niche, it is always good to write the abbreviation the first time it is used, in fact, W3C recommends it[0]. Anyone who does not follow this either are not informed well enough, or has ableism. Most replies in this thread shows the latter.

https://www.w3.org/WAI/WCAG21/Understanding/abbreviations

ajam15072• 2 months ago

This page has 5 abbreviations that are not spelled out before you even get to the body of the article.

wglb• 2 months ago

If you wish to explore a new concept, perhaps ask duckduckgo "what is ctf" an you may well get a response such as:

    CTF stands for Capture The Flag, which is a cybersecurity competition where participants find hidden "flags" in vulnerable programs or systems to test and develop their security skills. There are two main types of CTFs: jeopardy-style, where teams solve various challenges for points, and attack-defense, where teams defend their systems while trying to exploit their opponents'.

codemog• 2 months ago

We’re in an age where, to be possibly a bit rude but blunt, pseudo-intellectuals are obsolete. A pseudo-intellectual prided themselves on being able to efficiently solve closed, man made problems such as leetcode, CTF problems, or even math Olympiad problems. They could do good in school by memorizing a rote technique and applying it to some test. They typically don’t have any real creativity and if you put them to work on a problem you can’t Google or isn’t a fake man made one, they fall apart incredibly fast.

They may as well be the human equivalent to what LLMs currently are.

I do not mourn these people, as they’re usually the most arrogant types. I hope for their sake they adapt.

nektro• 2 months ago

easy, CTFs should ban it. then it'd be more like the chess community

archi422• 2 months ago

The article addresses this:

Rules that ask people not to use LLMs are ignored and almost impossible to enforce in open online events.

It's quite sad to see CTFs dying. I never had the time do seriously participate in CTFs, but I always respected those who did, as well as the people organizing these events.

legacynl3• 2 months ago

Rules that ask people not to use LLMs are ignored and almost impossible to enforce in open online events.

That's such a non-reason. If your competition cannot enforce the rules of the competition, then what's the point? Does the CTFs specifically need to be 'open'?

Dzugaru2• 2 months ago

It's not only CTFs. I strongly believe being a programmer at a gamejam like Ludum Dare, or hackathons is pretty much over.

wasmperson• 2 months ago

Ludum Dare 59 just wrapped up last week, and both first and second place were won by developers using "Agentic" coding tools, something the community there is still discussing:

https://ldjam.com/events/ludum-dare/59/setidream/about-ai-ar...

For what it's worth, the non-AI-coded entries were still quite good relative to the winners, so it's not so obvious that AI use confers an unbeatable advantage.

Dzugaru2• 2 months ago

I did vibe code jam 59 entry with friends, the spirit of the rules there's a lot more lax. We didn't even get to top 100, but that's mostly due to gamedesign errors, not tech. This is the first entry in years which was vibecoded 100%, and I have very mixed feelings about it. It's no doubt anymore - 1.5x-2x speedup, which makes not using it (if allowed) a complete no-go. But psychologically it's tough losing control, and changing workflow to managerial one substantially, it diminishes the craft.

s3p• 2 months ago

Don't hate me, I do agree with the premise of the article (I really do!) but I can’t help but notice:

The issue was never that AI could help. proceeds to write the next 3 sentences about how the problem IS in fact ai help
Teams that refused to use AI were not just missing a convenience; they were playing a slower version of the competition.
CTFs were not just a set of puzzles. They were a ladder.
The claim is not that every challenge is solved. The claim is that...
The loss is not just a scoreboard. It is the ladder from

Guys I'm so sorry I just can't stop noticing stuff like this. Anyone else?

Yenrabbit• 2 months ago

I got some AI writing vibes too, but looking closer, I think it might be human-written (or at least partly so) - perhaps just picking up some AI conversation styles? FWIW, Pangram gives it a mixed but mostly-human score too. Maybe AI is not just changing the way we speak; it's changing the way we perceive all writing ;)

tptacek3• 2 months ago

A big fraction of the comments on this thread are about the impact of cheating on competitive games. It's important to understand that automating CTF challenges isn't usually cheating. It's normally part of CTF culture. The better teams have toolboxes ready to shred the early challenges; it's not a level playing field and was never intended to be.

(The author of the piece understands this; I think they're broadly right, though I think these games will find other ways to incentivize participation without the now-meaningless leaderboards.)

viccis3• 2 months ago

This is already addressed in the blog post about the fast that frontier LLMs have moved to being able to solve the kind of problem you'd expect a talented amateur or mid-level pro to do (aka top level CTF problems)

atleastoptimal• 2 months ago

The competition is turning into "who can afford to run enough agents, with enough context, for long enough."

This will basically become true for everything.

low_tech_love3• 2 months ago

It’s unavoidable, but really sad, isn’t it? Thinking about the incredible creativity and hard work that went into creating such challenges and now it’s probably history. I feel something similar with free-to-play gacha video games. The gacha mechanics are slowly creeping into every type of game, and where you once had very clear and obvious slot machines, now you get them transparently mixed in with beautiful and fun games (e.g. ZZZ, Where Winds Meet, Genshin, etc) in a way that sounds like in the near future no company will have any incentive to not have gacha mechanics in any game.

SebFender• 2 months ago

Yes you're right - But just like many other stuff things change - CTF Veteran for more than 3 decades I find lots of fun figuring out how to use some of my agents and new tools to find vulnerabilities - The goal is the same / tools change and that's good.

somesortofthing• 2 months ago

I have no experience in the CTF scene so I'm curious - why not lean in and design the puzzles with an AI harness like the one top teams use in the loop and use the(presumably) expert skills of the designers to patch up the holes until the AI can't find them? Do you just end up designing ~perfectly secure systems that no human can break without finding monetizable 0days?

a_t48• 2 months ago

I think that misses the point - it's a little bit like asking why FPS game developers don't lean into aimbot usage. You could, but by default it's a bit boring, and a different type of game.

SadWebDeveloper• 2 months ago

Kinda FUD article... the reality is that common problems are going to be easy because the solution is probably inside the training dataset, the challenge should be adapted to make LLM's useless for example once at Defcon CTF the problems were for an unknown CPU architecture based on octal that required to write even your own disassembler... this are the kind of things that will probably be hard for frontier LLM's

netsec_burn• 2 months ago

Speaking from experience, the LLM agents adapt fairly well to these contexts too. It's not at all FUD, you're at a significant disadvantage if you don't compete with AI now. I went to a CTF recently against teams I have won against every year, and within 10 minutes of the event starting they had solved every challenge. They have an agent loop and it solves everything immediately, so they won. Anyone attempting to solve the challenges on their own has no chance, even if you think "maybe this is too out of the box for LLMs". Furthermore, the DEFCON CTF you're referring to has quals, and if you don't qualify you don't get those challenges in the finals. Quals has mainly binary exploitation challenges which Opus (and others) solve as long as you hold the gas pedal down on your API bill. I don't believe it's hyperbole to say CTF is dead, as a competitor.

rmac• 2 months ago

i asked hackathon judges about this - the net is More than ever teams of one are winning

I guess this goes in parallel with the whole building for one narrative

And while I have ideas to excite and promote LLM use in these style games I've still not been able to crack the human collaboration component that is at the forefront of all of this change

legacynl3• 2 months ago

The issue was never that AI could help. CTF players have always used tools. [...] Teams that refused to use AI were not just missing a convenience; they were playing a slower version of the competition.

So the obvious solution is to fully ban AI and AI generated tools? To destroy your own hobby just because AI can semantically be considered a tool, seems very stupid to me. If the point of these CTFs is to practice and measure your skill, what becomes the point of the competition once everyone uses AI?

nicce2• 2 months ago

Many existing challenges are purely based on knowing the existing precise tool. It has been the difference of "very easy" and "insane" challenge, whether you knew the tool or not.

So, I would not start banning the tools. They always been there. We just need to fine tune the challenges where bar goes beyond things and you really need to use AI as tool. Maybe the definition of insane starts to be custom kernel fork with planted bugs and you need to use AI to find the bugs, and use some exploit chains against that kernel with specific web server. That is the real world right now, I guess.

electr1cBugaloo• 2 months ago

This article hits really close to home for me. I have just recently started doing CTFs and feel like I am already a somewhat decent intermediate player. Much of the motivation for me is learning but also comparing/succeeding vs. my peers.

Asking AI for learning / explaining purposes is absolutely fine in my book - but I have absolutely no motivation to set up AI to solve challenges. Without AI you can't really compete successfully. So AI is really taking motivation away for me which in turn again prevents me from learning more. I am not sure that this is solvable.

richardfey• 2 months ago

I have a different take on this: how many people using a rifle do you see at composite bow tournaments? We might just move this kind of activities to sandboxes and implement more strict requirements for participation. It might make them niche or indeed disappear.

Frontier AI has broken the open CTF format

Related Stories