Serving 200M requests per day with a CGI-bin

https://simonwillison.net/2025/Jul/5/cgi-bin-performance/

by mustache_kimono • 1 year ago

336 327 1 year ago

simonwillison.net

eyberg2• 1 year ago

Outside of nostalgia there's no engineering reason to do this - definitely not for performance.

That same go program can easily go over 10k reqs/sec without having to spawn a process for each incoming request.

CGI is insanely slow and insanely insecure.

simonw8• 1 year ago

What makes it insecure? It's a pretty simple protocol - anything in there that makes is insecure beyond naive mistakes that could be avoided with a well designed library?

EDIT: Looks like the way CGI works made it vulnerable to Shellshock in 2014: https://en.m.wikipedia.org/wiki/Shellshock_(software_bug)

I agree that there's probably not much of an argument to switch to it from the well established alternative mechanisms we are using already.

The one thing in its favor is that it makes it easier to have a polyglot web app, with different languages used for different paths. You can get the same thing using a proxy server though.

eyberg2• 1 year ago

CGI has a very long history of security issues stemming primarily from input validation or the lack thereof.

tonyedgecombe• 1 year ago

EDIT: Looks like the way CGI works made it vulnerable to Shellshock in 2014:

From your linked article: If the handler is a Bash script, or if it executes Bash...

But we are talking about Python not Bash.

slashdave2• 1 year ago

Because... it can execute an arbitrary executable? In the old days, it also ran as root.

mike_hearn2• 1 year ago

IIRC the main issue was finding ways to convince a CGI script to write something to disk, at which point you could sometimes make it be treated as another CGI script. More of an issue on Windows than UNIX.

cperciva2• 1 year ago

Hey, it could be worse. Some people launch entire VMs to service individual requests.

klysm• 1 year ago

That’s cloud-native!

giantrobot• 1 year ago

You mean "Web Scale".

jasonvorhe• 1 year ago

How is that done?

kragen62• 1 year ago

I mean, a process kind of is an entire VM. But yeah, serverless is a marketing term for CGI scripts.

__float• 1 year ago

Not everything needs 10k RPS, and in some sense there are benefits to a new process – how many security incidents have been caused by accidental cross-request state sharing?

And in a similar vein, Postgres (which is generally well liked!) uses a new backend process per connection. (Of course this has limitations, and sometimes necessitates pgbouncer, but not always.)

Tractor86264• 1 year ago

Even without pgbouncer postgres uses long lived connections (and long lived processes) . So bad example.

Uber famously switched from pg to mysql because their SWEs couldn't properly manage connections

reddec• 1 year ago

Few years ago I felt the same and created trusted-cgi.

However, through the years I learned:

- yes, forks and in general processes are fast - yes, it saves memory and CPU on low load sites - yes, it’s simple protocol and can be used even in shell

However,

- splitting functions (mimic serverless) as different binaries/scripts creates mess of cross scripts communication - deployment is not that simple - security wise, you need to run manager as root and use unique users for each script or use cgroups (or at least chroot). At that moment the main question is why not use containers asis

Also, compute wise, even huge Go app with hundreds endpoints can fit just few megabytes of RAM - there is no much sense to save so few memory.

At worst - just create single binary and run on demand for different endpoints

simonw8• 1 year ago

Also discussed yesterday: https://news.ycombinator.com/item?id=44464272

petesergeant4• 1 year ago

The nascent web community quickly learned that this was a bad idea, and invented technologies like PHP

Well ackshually ... the technology here that was important was mod_php; PHP itself was no different to Perl in how it was run, but the design choice of mod_php as compared to mod_perl was why PHP scripts could just be dumped on the server and run fast, where you needed a small amount of thinking and magic to mod_perl working.

simonw8• 1 year ago

Right, but mod_php was an early addition to the PHP ecosystem and quickly became the default way of deploying it - I believe the first version of the Apache module was for PHP/FI Version 2.0 in 1996: https://www.php.net/manual/phpfi2.php#module

petesergeant4• 1 year ago

It was indeed, and I spent much time wailing and gnashing my teeth as a Perl programmer that nothing similar existed in Perl.

fcatalan• 1 year ago

At that time I was developing with a friend what later was called a Learning Management System: It had content management, assignment uploads, event calendar, grade management, real time chat, forums... It was all plain C via CGI and it was hell to work with.

What almost brought us to tears the day we learned about PHP was how everything we had been painstakingly programming ourselves from scratch reading RFCs or reverse engineering HTTP was just a simple function call in PHP. No more debugging our scuffed urlencode implementation or losing a day to a stray carriage return in an HTTP header...

faizshah• 1 year ago

I’ve also thought about this moreso as part of a workflow for quickly prototyping stuff. At least for a lot of the modern JIT languages I believe their startup times will be dominated by your imports unless you go with a fastcgi model. This came up as I started adopting h2o web server for local scripts since it has clean and quick to write config files with mruby and fast-cgi handlers and is also crazy fast: https://h2o.examp1e.net/configure/fastcgi_directives.html

Another place this can be useful is for allowing customers to extend a local software with their own custom code. So instead of having to use say MCP to extend your AI tool they can just implement a certain request structure via CGI.

dolmen• 1 year ago

An MCP frontend to CGI programs would not be a bad idea for a end user environment.

This makes me wonder if an MCP service couldn't be also implemented as CGI: an MCP framework might expose its feature as a program that supports both execution modes. I have to dig into the specs.

zokier• 1 year ago

Fastcgi kinda loses all the benefits of cgi though.

aaronblohowiak• 1 year ago

Ehhhhhhh. I believe but cannot cite that fork() got a lot cheaper over the last 30 years as well (independent of machine stats, I believe the Linux impl is inherently cheaper now but I can’t remember the details.) cgi bin works really well if you don’t have to pay for ssl or tcp connections to databases or other services, but you can maybe run something like istio if you need that. I have long thought that (fast)cgi is better model than proprietary “lambda” /“faas”. But the languages de jure and vendor lock in didn’t favor a standards based approach here.

seabrookmx• 1 year ago

You should check out OpenFaaS. It uses a very CGI-inspired architecture for self hosting "functions" (in the lambda sense) on more conventional infra.

AdieuToLogic4• 1 year ago

I believe but cannot cite that fork() got a lot cheaper over the last 30 years as well ...

The fork[0] system call has been a relatively quick operation for the entirety of its existence. Where latency is introduced is in the canonical use of the execve[1] equivalent in newly created child process.

... cgi bin works really well if you don’t have to pay for ssl or tcp connections to databases or other services, but you can maybe run something like istio if you need that.

Istio[2] is specific to Kubernetes and thus unrelated to CGI.

0 - https://man.freebsd.org/cgi/man.cgi?query=fork&apropos=0&sek...

1 - https://man.freebsd.org/cgi/man.cgi?query=execve&sektion=2&a...

2 - https://istio.io/

colesantiago• 1 year ago

dupe: https://news.ycombinator.com/item?id=44464272

kragen62• 1 year ago

Even with things like Python, CGI is pretty fast these days. If your CGI script takes a generous 400 milliseconds of CPU to start up and your server has 64 cores, you can serve 160 requests per second, which is 14 million hits per day per server. That's a high-traffic site.

That is, if your web service struggles to handle single-digit millions of requests per day, not counting static "assets", CGI process startup is not the bottleneck.

A few years ago I would have said, "and of course it's boring technology that's been supported in the Python standard library forever," but apparently the remaining Python maintainers are the ones who think that code stability and backwards compatibility with boring technology are actively harmful things, so they've been removing modules from the standard library if they are too boring and stable. I swear I am not making this up. The cgi module is removed in 3.13.

I'm still in the habit of using Python for prototyping, since I've been using it daily for most of the past 25 years, but now I regret that. I'm kind of torn between JS and Lua.

simonw8• 1 year ago

Here's the justification for removing cgi - https://peps.python.org/pep-0594/#cgi

Amusingly that links to https://peps.python.org/pep-0206/ from 14th July 2000 (25 years ago!) which, even back then, described the cgi package as "designed poorly and are now near-impossible to fix".

Looks like the https://github.com/jackrosenthal/legacy-cgi package provides a drop-in replacement for the standard library module.

kragen62• 1 year ago

That fails pretty hard at providing a rationale. Basically it says that CGI is an inefficient interface because it involves creating a new process! Even if that were true, "You shouldn't want to do such an inefficient thing" is very, very rarely a reasonable answer to a technical question like "How do I write a CGI script in Python?" or "How do I parse a CSV file in Python?"

There are certainly some suboptimal design choices in the cgi module's calling interface, things you did a much better job of in Django, but what made them "near-impossible to fix" was that at the time everyone reading and writing PEPs considered backwards compatibility to be not a bad thing, or even a mildly good thing, but an essential thing that was worth putting up with pain for. Fixing a badly designed interface is easy if you know what it should look like and aren't constrained by backwards compatibility.

WD-424• 1 year ago

I don't get it. Having a complaint about Python removing CGI from the stdlib is well and fine. But then you say you'd rather consider JS, which doesn't even have a std lib? Lua doesn't have a CGI module in stdlib either.

bravesoul23• 1 year ago

Node.js provides the defacto standard lib for JS backend and its got a good feature set.

That said these days I'd rather use Go.

kragen62• 1 year ago

I think it's fine to not have functionality in the standard library if it can be implemented by including some code in my project. It's not fine to have a standard library that stuff disappears from over time.

rollcat5• 1 year ago

Lua doesn't have a CGI module in stdlib either.

Lua barely has any stdlib to speak of, most notably in terms of OS interfaces. I'm not even talking about chmod or sockets; there's no setenv or readdir.

You have to install C modules for any of that, which kinda kills it for having a simple language for CGI or scripting.

Don't get me wrong, I love Lua, but you won't get far without scaffolding.

sitkack3• 1 year ago

The argument isn't about who has the standard lib, what I think Kragen is saying is that that the Python leadership has no qualms about removing functionality that people rely on and making up lame reasons to do so.

bravesoul23• 1 year ago

Yeah high performance web used to be an art. Now it's find what you are doing that's stupidly wasteful that you did to ship fast, and stop doing that thing.

Your app could add almost no latency beyond storage if you try.

kqr8• 1 year ago

Consider Perl. It's not quite as batteries-included as Python, but it is preinstalled almost everywhere and certainly more stable than JS and Lua. (And Python.)

kragen62• 1 year ago

You may be interested in an article I wrote in ;login: 23 years ago: https://www.usenix.org/publications/login/june-2002-volume-2...

At the time Perl was the thing I used in the way I use Python now. I spent a couple of years after that working on a mod_perl codebase using an in-house ORM. I still occasionally reach for Perl for shell one-liners. So, it's not that I haven't considered it.

Lua is in a sense absolutely stable unless your C compiler changes under it, because projects just bundle whatever version of Lua they use. That's because new versions of Lua don't attempt backwards compatibility at all. But there isn't the kind of public shaming problem that the Python community has where people criticize you for using an old version.

JS is mostly very good at backwards compatibility, retaining compatibility with even very bad ideas like dynamically-typed `with` statements. I don't know if that will continue; browser vendors also seem to think that backwards compatibility with boring technology like FTP is harmful.

rollcat5• 1 year ago

What's the landscape like, for when you need to scale your project up? As in: your project needs more structure, third-party integrations, atomic deployments, etc - not necessarily more performance.

Python has Werkzeug, Flask, or at the heavier end Django. With Werkzeug, you can translate your CGI business logic one small step at a time - it's pretty close to speaking raw HTTP, but has optional components like a router or debugger.

tonyhart7• 1 year ago

Yeah but 400ms is unacceptable this days

kragen62• 1 year ago

That's intended as an unreasonably high upper bound. On my cellphone, in Termux, python3 -m cgi takes 430–480ms. On my laptop it takes 90–150ms. On your server it probably takes less.

I agree that tens of milliseconds of latency is significant to the user experience, but it's not always the single most important consideration. My ping time to news.ycombinator.com is 162–164ms because I'm in Argentina, and I do unfortunately regularly have the experience of web page loads taking 10 seconds or more because of client-side JS.

cenamus• 1 year ago

If the whole site takes 5 seconds to fully hydrate and load its 20megs of JS I'll gladly take a server side rendered page that has finished loading in a second.

nurettin2• 1 year ago

<PHP> I would like to have a word.

lucb1e6• 1 year ago

The PHP interpreter (+mods) don't take anywhere near 400ms to start up, even on very old CPUs, though? Not sure what you mean

mjw10072• 1 year ago

The Python maintainers are removing the module _named_ cgi, but they're not removing the support for implementing CGI scripts, which is CGIHTTPRequestHandler in the http.server module.

All that was in the cgi module was a few functions for parsing HTML form data.

kragen62• 1 year ago

It would be very difficult indeed to make it impossible to implement CGI scripts in Python; you'd have to remove its ability to either read environment variables or perform stdio, crippling it for many other purposes, so I didn't think they had done that. Even if they removed the whole http package, you could just copy its contents into your codebase. It's not about making Python less powerful.

As a side note, though, CGIHTTPRequestHandler is for launching CGI programs (perhaps written in Rust) from a Python web server, not for writing CGI programs in Python, which is what the cgi module is for. And CGIHTTPRequestHandler is slated for removal in Python 3.15.

The problem is gratuitous changes that break existing code, so you have to debug your code base and fix the new problems introduced by each new Python release. It's usually fairly straightforward and quick, but it means you can't ship the code to someone who has Python installed but doesn't know it (they're dependent on you for continued fixes), and you can't count on being able to run code you wrote yourself on an earlier Python version without a half-hour interruption to fix it. Which may break it on the older Python version.

pjmlp6• 1 year ago

I rather stick with PHP or JS, due to having a JIT in the box for such cases.

Since I learnt Python starting in version 1.6, it has mostly been for OS scripting stuff.

Too many hard learnt lessons with using Tcl in Apache and IIS modules, continuously rewriting modules in C, back in 1999 - 2003.

int0x292• 1 year ago

I don't think the JIT will help that much as each request will need to be JITed again. Unless Node and PHP are caching JIT output

fulafel• 1 year ago

Python got a JIT recently (an experimental / off by default feature in 3.13).

mathiaspoint2• 1 year ago

The scary thing about CGI for me is more the shell insanity than the forking. Although I think after the shell shock RCE most servers probably switched to directly execing the CGI process.

gsliepen• 1 year ago

Indeed. There is no reason why CGI would need shells or scripting languages though, you can just write them in any programming language. It's not that hard; I wrote this pastebin clone in C: https://github.com/gsliepen/cbin/

kragen62• 1 year ago

I think they were already doing that. I'd be very surprised if Apache's mod_cgi in 01997 had been spinning up a whole bash process to spawn off the CGI script as a subprocess! Certainly I haven't seen the passels of useless bash processes in ps that you would get with that approach.

ksec3• 1 year ago

Zen 6c is about to get 256 Core Per Socket, 512 vCPU / Thread, or 1024 vCPU in a Dual Socket System. That is 2560 Request Per Second ( or PageView ), and this doesn't even include caching.

If I remember correctly that is about half of what StackExchange served on daily average over 8 servers. I am sure using Go or Crystal would have scale this at least 10x if not 20x.

The problem I see is that memory cost isn't dropping which means somewhere along the graph the memory cost per process together will outweight whatever advantage this has.

Still, sounds like a fun thing to do. At least for those of us who lived through CGI-Bin and Perl era.

diordiderot• 1 year ago

People use crystal?

chasil3• 1 year ago

The problem child for classic CGI is Windows, where process creation is 100x slower than any other POSIX implementation.

You can measure this easily, get a copy of Windows busybox and write a shell script that forks off a process a few thousand times. The performance difference is stark.

kragen62• 1 year ago

That's interesting. I hadn't thought about that. Still, fork() (plus _exit() and wait()) takes 0.7ms in Termux on my phone for small processes, as measured by http://canonical.org/~kragen/sw/dev3/forkovh.c.

Are you really saying that it takes 70ms on Microsoft Windows? I don't have an installed copy here to test.

Even if it does, that would still be about 15% of the time required for `python3 -m cgi`, so it seems unlikely to be an overriding concern for CGI programs written in Python, at least on manycore servers serving less than tens of millions of hits per day. Or does it also fail to scale across cores?

layer8• 1 year ago

That’s more an argument against using Windows for a web server than against using CGI.

lucb1e6• 1 year ago

If your CGI script takes a generous 400 milliseconds of CPU to start up and your server has 64 cores, you can serve 160 requests per second

...and then you're wasting a 64-core server at 100% CPU load on just starting up and tearing down script instances, not yet doing any useful work. This is doing 160 startups per second, not requests per second

Would be a pretty big program for it to require 400ms on a fast CPU though, but the python interpreter is big and if you have one slow import it's probably already not far off

kragen62• 1 year ago

Responded at https://news.ycombinator.com/item?id=44483385.

Levitating• 1 year ago

If your CGI script takes a generous 400 milliseconds of CPU to start up

then that endpoint will have at least 400ms response times, not great

chgs3• 1 year ago

I have cgi scripts which take dozens of seconds to run. Load time isn’t a problem for many uses.

refulgentis• 1 year ago

Surprised this is the top comment 12 hours in, should be intuitive to HN that 160 req/s on 64 cores is...not great!...and that's before the errors it takes to get that # (ex. all we're doing is starting the per-request executable)

kragen62• 1 year ago

Obviously you can do orders of magnitude better than 160 req/s! But if you only have tens of thousands of users or less, like all but the top thousand web sites in the world, there's at best very marginal benefit in doing so. It's spending effort to solve a problem that no longer exists.

I agree that the number I gave is erroneous, but not in the direction you imply. 400ms is extremely conservative, and it's not just starting an executable. I took my 100-millisecond estimate for the time to start up Python and `import cgi`, which loads 126 modules in the version I have here on my cellphone, and multiplied it by a safety factor of 4. Even on my cellphone it doesn't take that long, although `python3 -m cgi` does, because it loads all those modules and then runs a debug dump script. The time I measured for `python3 -m cgi` on my laptop is 90–150ms.

If all you are doing is starting a per-request executable, that will typically take more like 1 millisecond than 100 milliseconds.

Perhaps you meant to suggest that the actual logic triggered by the request would increase this 400ms number significantly.

Consider first the case where it would, for example because you do 200ms of computation in response to each request. In this case the best you could do by switching from CGI to a more efficient mechanism like FastCGI is a 3× increase in req/s. If that allows you to increase from 10 million requests per day on one server to 30 million, that could be worthwhile if you actually have tens of millions of requests per day. But it's kind of a marginal improvement; it saves you from needing two or three servers, probably.

Now consider the case—far more common, I think—where your computation per request is small compared to our hypothetical 400 ms Python CGI overhead. Maybe you do 20ms of computation: 80 million machine instructions or so, enough to tie an IBM PC up for a few minutes. In such a case, the total one-core request time has gone from 400ms to 420ms, so 400ms is a very good estimate, and your criticism is invalid.

jwatte2• 1 year ago

If you're necessarily going to go with Lua (not recommended, IMHO) you should at least try luau.

kragen62• 1 year ago

Thanks! What do you like and dislike about Luau and Lua in general?

swyx2• 1 year ago

I'm kind of torn between JS and Lua.

why lua?

kragen62• 1 year ago

Six months ago here, in response to a Lua-advocacy post that I didn't like very much, I wrote a comment where I gave an overview of what I like about Lua: https://news.ycombinator.com/item?id=42519070

Shortly after that, we had a lengthy thread about "how Lua compares to Python": https://news.ycombinator.com/item?id=42655158

Two years ago I commented https://news.ycombinator.com/item?id=38862372 listing Lua's worst flaws from my point of view.

slashdave2• 1 year ago

Maybe my memory is bad, but I don't remember people jumping ship from CGI-bin because of performance. I do remember a lot of security problems.

kragen62• 1 year ago

I remember performance being the main reason people jumped ship from CGI in the period 01995–02002. The switch didn't solve security problems itself (except Shellshock, if you wrote CGI scripts in bash, but Shellshock wasn't publicly known until much later) but it sometimes came with a less slapdash approach to building web services which did solve security problems. On the other hand, it often instead came with a move to PHP, which had just unbelievable levels of security problems.

It's possible that your experience with people switching was later, when performance was no longer such a pressing concern.

sitharus2• 1 year ago

The main security issue I recall from CGI was caused by the web server having to execute the binary. This meant either executing as www-data, running the web server as root so it can call setuid, or using setuid binaries which have their own issues.

These were real issues on multi-user hosts, but as most of the time we don’t use shared hosting like that anymore it’s not an issue.

There were also some problems with libraries parsing the environment variables with the request data wrong, but that’s no different from a badly implemented http stack these days. I vaguely recall some issues with excessively log requests overflowing environment variables, but I can’t remember if that was a security problem or DoS.

sramsay• 1 year ago

I remember megatons of Java Servlet(tm) hype convincing us that real programmers do OO.

Java remains the only programming language I've ever heard covered in a feature story for NPR.

cb3217• 1 year ago

Yeah. I mentioned this the other day: https://news.ycombinator.com/item?id=44466245

Combined with the dot-com boom "general hype", I'm sure a lot of managers pushed heavyweight solutions where lightweight would have sufficed. Well, that may be an eternal problem, but maybe more succeeded in pushing them with a lot of hype. :-)

Not enough people I guess saw this as Sun trying to be the new Microsoft (which was the new IBM, which still has MVS & Cobol!), namely the company in control of The Platform, where here "The" just means the hip new thing kids learn in school and want to continue doing before they become expensive old timers.

petesergeant4• 1 year ago

Tangentially related, I wrote a brief history of CGI and the evolution away from it here (as conference slides):

"A brief, incomplete and largely inaccurate history of dynamic webpages"

https://www.slideshare.net/slideshow/psgi-and-plack-from-fir...

Tractor86264• 1 year ago

2400 rps on this hardware on hello world application - isn't it kinda bad?

And we trading performance for what exactly? Code certainly didn't become any simpler.

slyall• 1 year ago

It's only bad if you need to get more than 2000 rps

Which is only a small proportion of sites out there.

kqr8• 1 year ago

I'd argue it's bad even if you get more than 1000 Bq of requests. You never want to approach 100 % utilisation, and I'd aim to stay clear of 50 %.

Tractor86264• 1 year ago

If there is no some other advantages - it is just bad.

gred3• 1 year ago

I'd rather not pay for 8 cores / 16 threads, though...

masklinn• 1 year ago

It's only bad if you need to get more than 2000 rps

Or if you don't want to pay for an 8/16 for the sort of throughput you can get on a VPS with half a core.

YmiYugy• 1 year ago

Yes, but it's running on pretty powerful hardware. Try this with 1 vCPU, 512MB RAM and a website, that makes a lot of requests for a single page visit. Until recently I used to maintain some legacy B2B software, where each customer got their own container with very strict resource limits. A single page visit could cause 20-50 requests. Removing CGI was a significant performance win, even with a single user loading just one page.

kqr8• 1 year ago

It's not great, but it is enough for many use cases. Should even handle a HN hug of death.

Tractor86264• 1 year ago

But why? What advantages we getting?

procaryote3• 1 year ago

The marging for error becomes tiny though... Performance regression making some requests slow? Suddenly you don't handle even those 2k requests. And a Denial of service attack doesn't even have to try hard.

jarofgreen2• 1 year ago

Had a similar chat with someone recently after I used Apache for a side project in part because of it's .htaccess feature.

This let's you drop .htaccess files anywhere and Apache will load them on each request for additional server config. https://httpd.apache.org/docs/2.4/howto/htaccess.html

One big reason to avoid them was performance; it required extra disk access on every request and it was always better to put the configuration in the main config file if possible.

But now? When most servers have an SSD and probably spare RAM that Linux will use to cache the file system?

Ok, performance is still slightly worse as Apache has to parse the config on every request as opposed to once, but again, now that most servers have more powerfull CPU's? In many use cases you can live with that.

[ Side project is very early version but I'm already using it: https://github.com/StaticPatch/StaticPatch/tree/main ]

rollcat5• 1 year ago

Quoting Rasmus Lerdorf:

I'm not a real programmer. I throw together things until it works then I move on. The real programmers will say "Yeah it works but you're leaking memory everywhere. Perhaps we should fix that." I’ll just restart Apache every 10 requests.

PHP got a very long way since then, but a huge part of that was correcting the early mistakes.

PHP 8 is significantly better because it contains a lot less of my code.

jarofgreen2• 1 year ago

I'm not sure in which spirit you mean that, so I'm going to choose "approvingly" :-)

I do have thoughts for later about modes which could take all the config from .htaccess files and build them into the main config so then you avoid any performance issues - however you have to do that carefully to make sure people don't include any bad config that crashes the whole server. One of the nice things about using .htaccess files as intended is Apache has the Nonfatal flag on AllowOverride so you can avoid that. https://httpd.apache.org/docs/2.4/mod/core.html#allowoverrid...

malwrar• 1 year ago

I don’t understand why apache wouldn’t just watch the filesystem, this choice means 99.99% of http requests are going to be slowed down by an unnecessary disk reads.

kragen62• 1 year ago

When you get a notification from watching the filesystem, you have to read and parse the files. It's simpler to just always read and parse the files, and the <20 milliseconds that takes are not a big deal in many cases. I just tested curl http://canonical.org/~kragen/tmp2.html on localhost and it generally comes in under 15ms, despite the 2-kilobyte .htaccess in that directory.

But 25 years ago it was a significant performance hit. So you might wonder why Apache didn't just watch the filesystem to avoid it. The answer, as best I can reconstruct it, is that only IRIX had a usable filesystem change notification API; Linux had a shitty broken one, and Solaris and FreeBSD didn't have one at all. Linux's inotify is still a pain in the ass, but at least now it's good enough that it can actually work. https://groups.google.com/g/mailing.freebsd.fs/c/T64SiVOfyUE has Mark Felder 10 years ago describing it as "a world of hurt", but not having a viable alternative for FreeBSD even then.

So 25 years ago there was just no way to do this, so people just got in the habit of turning off .htaccess to make Apache fast. That meant that there was no incentive to make .htaccess fast.

jekwoooooe• 1 year ago

But why? A lot of modern stacks aren’t just about performance but development speed and stuff

WhereIsTheTruth• 1 year ago

"Just pump more HW at the problem bro"

RedShift1• 1 year ago

I would gladly spend a little more on hardware if it makes the code much simpler and easier to debug.

BirAdam2• 1 year ago

Simpler and easier to debug would also mean less expensive to maintain… thus, more hardware is likely cheaper long term.

ivovk• 1 year ago

It stops working when you need to connect to any external resource. Database, http clients etc maintain connection pools to skip initial connection phase, which can be costly. That’s why you usually need a running web application process

ben0x539• 1 year ago

At my last job, a lot of our web services also benefited immensely from in-process caches and batching (to be fair, some of them were the cache for downstream services), and their scaling requirements pretty much dominated our budget.

I can totally see how the cgi-bin process-per-request model is viable in a lot of places, but when it isn't, the difference can be vast. I don't think we'd have benefited from the easier concurrency either, but that's probably just because it was all golang to begin with.

8organicbits• 1 year ago

You can solve that with a sidecar, a dedicated process (or container) that pools connections for you. Pgbouncer as one example.

rvnx9• 1 year ago

Great, additional things to maintain that can break, all of that to work around the original sandcastle instead of fixing the root issue

phplovesong2• 1 year ago

The way "old" stacks like PHP work also makes it impossible to do stateful stuff, like websockets.

There is work-arounds but usually it a better idea to ditch PHP for a better technology more suited for modern web.

johnisgood6• 1 year ago

Better technology? Please, do tell.

And for your information, you can have stateful whatnots in PHP. Hell, you can have it in CSS as I have demonstrated in my earlier comments.

phplovesong2• 1 year ago

I mean something that has a "running program" is a good start. Makes it trivial to do long lived stuff, like websockets.

gvalkov• 1 year ago

We're still serving a cgi-bin directory at work for the occasional quick and dirty internal web app. The ergonomics are great as long as you keep it simple. The fact that it's cgi doesn't mean you have to print http/1.0 to stdout manually. For example, in python the builtin wsgiref.handlers.CGIHandler lets you run any wsgi app as a cgi script:

  import wsgiref.handlers, flask
  app = flask.Flask(__name__)
  wsgiref.handlers.CGIHandler().run(app)

The way we run the scripts is with uwsgi and its cgi plugin[1]. I find it simpler and more flexible than running apache or lighttpd just for mod_cgi. Since uwsgi runs as a systemd unit, we also have all of systemd's hardening and sandboxing capabilities at our disposal. Something very convenient in uwsgi's cgi handling that's missing from mod_cgi, is the ability to set the interpreter for a given file type:

  cgi = /cgi-bin=/webapps/cgi-bin/src
  cgi-allowed-ext = .py
  cgi-helper = .py=/webapps/cgi-bin/venv/bin/python3  # all dependencies go here

Time to first byte is 250-350ms, which is acceptable for our use case.

[1] https://uwsgi-docs.readthedocs.io/en/latest/CGI.html

kragen62• 1 year ago

This is great information, thanks! And it looks like wsgiref.handlers.CGIHandler isn't deprecated yet.

johnisgood6• 1 year ago

I remember when I used a C program with CGI. It was quite fast, and this was decades ago. There were no >100 cores or threads, and RAM was not abundant either, it was at best 1 GB. It was doable then, pretty sure it is even more doable today.

PaulDavisThe1st2• 1 year ago

Amazon in 1995 was a C++ executable invoked via CGI () That couldn't scale once load-balancing was required, but it worked pretty well up until that point.

() technically, two of them: one handled the front (customer visible) end, one handled the back-office side.

kragen62• 1 year ago

Oh right! And then you moved to ISAPI, right?

indigodaddy4• 1 year ago

Speaking of old stuff like newspro, anyone remember php scripts like CuteNews and sNews (single file CMS) ?

doublerabbit• 1 year ago

Yes, CuteNews was clever. PHPNuke, e107, all those happy days downloading source files on 56K and uploading them to my webhost to play with.

indigodaddy4• 1 year ago

Some super fun times for sure and the scene is not easily replicated by anything today

reidrac• 1 year ago

If the cgi bin needs DB access, every time the process starts it needs to open a connection. Having the code in memory, for example using fastcgi, is not only to avoid the startup time penalty; you can also have a DB connection pool or at least a persistent DB connection per thread.

procaryote3• 1 year ago

Do it at scale and your database will be sad about the number of connections

At least that was the case when I did the "python is single threaded, let's run many of them" + "python is slow, let's run many of them" dance

At scale you end up using shared connection pools outside of python (like pgbouncer) and a lot of tuning to make it serve the load while not killing the database

Of course, then we reimplemented in a multithreaded somewhat performant language and it became dead simple again

Twirrim• 1 year ago

There were standard ways to handle that, such as hosting a separate daemon that acts effectively as your proxy. Using Unix sockets instead of TCP/IP makes connecting to it relatively cheap.

Iwan-Zotow• 1 year ago

Use udp

pjmlp6• 1 year ago

That is why CGI eventually evolved into a model that would keep some of the stuff around between requests.

genghisjahn2• 1 year ago

I recently ran a test where I had a $350 mini server using a golang binary, rabbitmq, redis and MySQL (all hosted on the same mini server)handle 5000 reqs/s sustained. That translates to 400 million reqs in a 24 hour day.

I’m amazed at how great the tools are these days that are free and yet we pay so much to cloud providers. I know it’s not an apples to apples comparison but it was so great to develop all that and fine tune it on a box in my basement.

jgalt2122• 1 year ago

Indeed. When I look at our VM spend, it's high because we need servers with a lot of local disk--not a lot of compute. For some reason, you cannot get one without the other.

Can you imagine the size of the business / service you could run with 4 attached 20TB drives, and a modest CPU? Good luck getting such from a cloud provider.

zepolen• 1 year ago

You can get 4x20TB + decent cpu for under 200$/month.

stavros• 1 year ago

Yep, and I hear everyone deploying some Kubernetes microservices monstrosity that slows development down tenfold because they don't know that servers can do more than 1 req/sec.

It's crazy to me that we keep paying all these overheads for no reason other than "it's what Google does". I really need to write my article about my modular monolith architecture up, it's worked really well for us.

papichulo20232• 1 year ago

I kinda blame the mentality that the only way to use containers is with K8s. Docker compose and basic scrips can take wayyy too far.

Aeolun2• 1 year ago

they don't know that servers can do more than 1 req/sec

Not if you are using Next.JS. It’s bizarre that even static files can only do like 10r/s on Next.JS

varispeed• 1 year ago

You don't have to pay cloud providers. You can rent a dedicated service from a hosting provider and go to town with it (very much limited by your bandwidth or transfer limit).

Cloud providers are used because there is a lot of vested interest - e.g. VC and investors also having shares in cloud companies, then there is fear their investment might not survive imaginary surge of traffic that will never happen in reality. Cloud sales people are masters of playing at investors insecurities.

x_may• 1 year ago

I think it’s also largely driven by the apparently cheapness of turning the CapEX of server buying to the OpEX of cloud renting. Less up front investment and auditing/access controls for SoC2 compliant are so much easier m.

beala2• 1 year ago

I've thought about hosting my side projects in my basement, but then I'd be susceptible to power outages and ISP downtime. If a drive fails while I'm traveling I'm SOL. If I accidentally lock myself out, there's no serial terminal to fall back on. I could go the route of those folks on r/homelab, but then it's no longer clear I'm saving money, especially if you factor in my time. My conclusion is that cloud providers are actually a great deal and benefit from huge economies of scale.

genghisjahn2• 1 year ago

I’ve thought through this as well. I’m not saying your conclusion is wrong, but if I get to where I really need backups to all this, there are solutions available. And I’ve had to learn lot more about logging and alerting to make sure my services stay up. But that time spent is repaid many times over with solutions that I understand. There is a time trade off but it’s usually a few hours of me learning “oh this is how you solve this” and then I don’t worry about it anymore.

In my experience, cloud at scale has ALWAYS required someone with a pager willing/paid to get up at 3am on Christmas Eve. So someone’s time is being used no matter.

lucb1e6• 1 year ago

If a drive fails while I'm traveling I'm SOL [...] My conclusion is that cloud providers are actually a great deal

You realise you can always still restore that backup onto someone else's server? When you need to restore from backup either way. I don't really see why one would pre-emptively pay for it

If I accidentally lock myself out, there's no serial terminal to fall back on.

Why not? That sounds like a choice you can make. The hardware I hosted on either had a KVM built in or can just attach a USB keyboard and VGA (or nowadays HDMI) display

Power outages aren't common in my area, and otherwise a UPS is not that expensive (compared to if you pay a third party to set up redundant power for your hobby system)

You can choose to pre-emptively pay the cloud premium and give them access to your server so you can also social engineer yourself back in via customer support (after all, if you aren't expecting to lose the password and thus don't need to convince a human to let you into your hosting account, then you could also hold onto your own server's password). It just all seems very opposed from the self-hosting spirit where you're self-reliant, which apparently you value since you were considering whether to self host?

kalleboo• 1 year ago

I've been hosting my side projects on a Synology NAS I bought back in 2018. I would have the NAS anyway to backup my laptop etc, so the hardware is effectively free for hosting.

Power draw is low so it's on a basic consumer UPS which has worked fine for the short power outages we've had (minutes).

RAID means that a drive dying doesn't impact service, although I've yet to have a drive outright die, I've just had them start developing bad sectors and preemptively replaced them (btrfs checksumming keeps the data safe)

I have backups going to a cloud storage service, so if the crap hits the bucket when I'm traveling, I can just spin up a VPS and restore to there (which is what I would have to do if a cloud server died anyway)

lucb1e6• 1 year ago

and yet we pay so much to cloud providers

Not all of us we don't

0xbadcafebee• 1 year ago

Even in the early 2000s CGI was really old tech. People were moving to more modern systems that solved the problems with CGI. What's funny is, modern systems actually have exactly the same problems as some of those newer solutions, yet they're just ignored/accepted now?

In the mid-2000s I worked on a very-large-scale website using Apache2 w/mod_perl. Our high-traffic peaks were something like 25k RPS (for dynamic content; total RPS was >250k). Even at that time it was a bit old hat, but the design scaled very well. You'd have a fleet of mod_perl servers that would handle dynamic content requests, and a fleet of Apache2 servers that served static content and reverse-proxied back to the mod_perl fleet for dynamic requests. In front of the static servers were load balancers. They'd all keep connection pools open and the load balancers avoided the "maximum connection limit" of typical TCP/IP software, so there was no real connection limit, it was just network, memory, and cpu limits.

The big benefit of Apache2 w/mod_perl or mod_php was that you combined the pluggability and features of a scalable and feature-filled web server with the resident memory and cache of an interpreter that didn't need to keep exiting and starting. Yes you had to do more work to integrate with it, but you have to do that today with any framework.

The big downside was bugs. If you had a bug, you might have to debug both Apache and your application at the same time. There was not as much memory to be had, so memory leaks were a MUCH bigger problem than they are today. We worked around it with stupid fixes like stopping interpreters after taking 1000 requests or something. The high-level programmers (Perl, PHP) didn't really know C or systems programming so they didn't really know how to debug Apache or the larger OS problems, which it turns out has not changed in 20 years...

FastCGI and later systems had the benefit that you could run the same architecture without being tied to a webserver and dealing with its bugs on top of your own. But it also had the downside of (in some cases) not multiplexing connections, and you didn't get tight integration with the web server so that made some things more difficult.

Ultimately every backend web technology is just a rehashing of CGI, in a format incompatible with everything else. There were technical reasons why things like FastCGI, WSGI, etc exist, but today they are unnecessary now that we have HTTP/2 and HTTP/3. If you can multiplex HTTP connections and serve HTTP responses, you don't need anything else. I really hope future devs will stop reinventing the wheel and go back to actual standards that work outside your own single application/language/framework.

pvtmert2• 1 year ago

To me, the `inetd` _is_ the CGI. Which made internet waay more fun. I, myself hosted various shell scripts through inetd, even an HTTP one which was written completely in Bash.

Although the VPS is long gone, and I did not use any version control at the time. The laptop I wrote the stuff are gone too :'(. But it was certainly quite fun. Deployment was as easy as Makefile + scp and testing was the another Bash script with bunch of `netcat`s and greps.

What a time to be alive :)

vivzkestrel2• 1 year ago

can anyone with expertise in CGI actually shed some light on what problems do CGIs actually have? like why did we hop from this to React?

zh32• 1 year ago

The oft-cited reason is each call fires up a new process so is expensive in terms of resources (though resources are cheap these days). Additionally, the CGI program has to faff about to retrieve data and parameters from the request (easy to get wrong) plus you have to watch out for security issues (hard to get right). Plus, of course, since the process exits at the end of the request any kind of session tracking needs to implemented separately.

Having said, for inhouse use, embedded and/or trusted environments they allow super quick hacks, e.g. complete CGI 'program' to print the date (put it in cgi-bin and make it executable):-

  #!/bin/sh
  echo "Context-Type: text/plain\r\n"
  echo "Your IP is $REMOTE_ADDR"
  date

vivzkestrel2• 1 year ago

is there a benchmark that you are aware of that takes a cgi program and pits it against modern frameworks like node.js express , hono etc? just to get a feel of how many requests will be handled at scale

Sami_Lehtinen• 1 year ago

For CGI endpoints that are called infrequently—daily, weekly, or even monthly—the script startup delay is negligible. Even a persistent process waiting to handle a request can have its memory swapped to disk by the OS under varying server load. When a request finally arrives, the delay from swapping that memory back in can negate any theoretical advantage over a cold-starting CGI script.

Serving 200M requests per day with a CGI-bin

Related Stories