You can serve static data over HTTP

https://ignore.pl/you_can_serve_static_data_over_http.html

by cxr • 3 years ago

156 120 3 years ago

alecst• 3 years ago

Thanks, I think I can actually apply this idea actually. I'm writing a chat app for fun and one of the (admittedly small) issues I'm dealing with is cleanly storing the local state of the chat so that it appears correctly on refresh. There's some complexity involved with how messages are served, received, etc. with notifications, dings, animations, whatever -- but maybe saving that information to a file (as an alternative to localStorage or database read) could be a robust alternative to whatever I was trying to do. I'll give it a try.

nesarkvechnep2• 3 years ago

Database reads all the time for your use-case.

est• 3 years ago

that's why I really, really hope ActivityPub or Nostr protocol could be designed as static-website compatible. But nope.

schemescape• 3 years ago

RSS seems closest to that.

nl• 3 years ago

I think you should be able to to a read-only, unauthenticated implementation of ActivityPub just using ActivityStreams files.

Best as I can tell the part that might be non-compliant here is that new followers should be added to the follower collection.

jeroenhd• 3 years ago

Parts of ActivityPub will work with static files: https://justingarrison.com/blog/2022-12-06-mastodon-files-in...

You need something to deal with follows (cryptographically verify that the follow attempts actually originate from the servers they claim to) and submitting posts to your followers (push based, not pull based in practice) but all the metadata and user lookup stuff can be done with JSON files in the right place.

faizshah• 3 years ago

Not really in the spirit of the post but I learned recently that you can also back your API Gateway apis in AWS with an s3 bucket: https://docs.aws.amazon.com/apigateway/latest/developerguide...

Makes it super simple to implement mocks of your api design before building the api.

graypegg• 3 years ago

Rails, by default, will put a “.json” suffix on json routes. This is so supremely useful if you have a Rails api app, speaking to some over complicated front end that might need a quick and dirty mock for something. Just serve your /api/thing.json from somewhere. Only issue is with HTTP verbs other than GET, but usually at that point I need a lot more than this 15second mock. Good for testing little things though.

Alifatisk3• 3 years ago

I was about to say this!

Lio• 3 years ago

This also lends itself to easy caching with a versioning key and proxy servers, just like with static files.

…I guess that was kind of the point of REST in the first place but it’s easy to forget.

REST has a wonderful simplicity to it.

I’m yet to hear about an equivalent caching solution for GrqphQL APIs for example.

nesarkvechnep2• 3 years ago

Let me tell you about hypermedia controls and discuss simplicity then? CRUD data API’s are simple, until some level of complexity.

userbinator• 3 years ago

Not from a modern developer. You need to jump down the rabbit hole of the history of solid and proven solutions.

Indeed. I've noticed an increasing number of people who call themselves "developers" and appear to create software, but all they can do is follow step-by-step tutorials to glue together some massively bloated thing that they have absolutely no understanding of, much less the skills to debug it when something doesn't work perfectly as expected.

keyle2• 3 years ago

At some point, the plane flew so high above the clouds, and it hasn't seen the ground for so long, the passengers might think the clouds is the ground.

turtleyacht• 3 years ago

When "Hello World" is a cloud instance spun up and billed on your tuition.

That won't really happen, right? No Computer Science (CS) professor would designate special cloud Interactive Development Environments (IDEs) to students.

On the other hand, it would really resolve a lot of "doesn't work on my machine" issues.

geek_at• 3 years ago

glue together some massively bloated thing that they have absolutely no understanding of

Welcome to the world of npm and supply-chain hell

busterarm• 3 years ago

Java and Golang aren't any different.

quectophoton• 3 years ago

To be fair, I would also say the mainstream IDEs have contributed to this situation to a certain extent, since they lower barrier of entry. That's a very good thing, but also sometimes bad.

Here's what I mean. There's a difference between:

* Using an IDE knowing what you want to do but you just like the convenience. You know what the IDE is doing for you, but you want your tool to do it for you quickly and get out of the way. If the IDE does something you don't know, you try to learn what this was.

* Using an IDE because you literally can't work without one. Not knowing your programming language and depending 100% on autocomplete to do most stuff. Not even knowing how to write the files that the IDE automatically updates when doing certain things.

It's a subtle difference, but the former is more likely to leave better comments in a web-based code review ("pull request") than the latter.

And if a project consists only of the latter type, it's unlikely that there will be comments like:

* "Use this other class with the same name but different package because this one is buffered" (because depending on autocomplete without knowing the language prevented them from even caring about which import to use, or the difference between those, as long as it works).

* "Would this change affect this other file that's not in the PR but calls this function a thousand times per request?" (because not knowing their own projects general structure due to depending on their IDE for navigation, so they're more likely to NOT notice a file that's not mentioned in the PR).

So if a project has only people that depend on an IDE (instead of using an IDE for convenience), it's easy to end up with stuff that technically works, but copies 40MB of strings around multiple times, and thrashes your performance due to reading one character at a time without buffering.

Granted, this technically is not related to IDEs, but in my experience (emphasis on "my"), most of the people I've worked with that use an IDE, are of the "can't program without it" type, and not the "I use it because it's convenient" type.

Not all of them. But most. Even if there's a learning curve at play here, I'd say there's usually overlap between "use IDE because convenient" and "actually bothering to put an effort into learning".

I understand this is an inevitable consequence of lowering the barrier of entry and making programming more accessible, and it doesn't matter if we're talking about npm, IDEs, programming languages, or anything.

intelVISA5• 3 years ago

good time for APTs though!

sfvisser• 3 years ago

You might not want them in your team maybe, but isn’t it really cool we reached a state where you can actually do this? Democratization of software engineering is a great good!

marginalia_nu6• 3 years ago

Dunno if it actually is democratizing software engineering. Looks more like a cargo cult to me. You get a second class of developers that don't quite get it. Not because they aren't capable, but because nobody shared the secrets with them and they had to piece together everything from flotsam.

BigJono• 3 years ago

Democratization of software engineering means giving everyone the opportunity to learn. Not handing them a black box with a few instructions then sending them off to plow head first into a brick wall of insurmountable knowledge gaps so they give up.

re-thc• 3 years ago

but isn’t it really cool we reached a state where you can actually do this

As in be a terrorist and produce bombs? These are all ticking time bombs.

JohnFen4• 3 years ago

Is it a great good if it reduces the quality of what our industry produces?

nousermane• 3 years ago

glue together some massively bloated thing that they have absolutely no understanding of

Relevant: https://xkcd.com/1988/

pikseladam• 3 years ago

exactly that. I couldn't describe better.

VoodooJuJu• 3 years ago

Then how are these people getting jobs but not my more competent friend? Asking for friend.

xigoi• 3 years ago

Because many hiring managers don't understand programming at all, so they select people by guessing.

msla• 3 years ago

Welcome to the world of programming on an OS.

intelVISA5• 3 years ago

Turn back, nothing but pain and disappointment if you continue this dangerous line of thought.

gofreddygo• 3 years ago

but all they can do is follow step-by-step tutorials

Lego's dream has come true.

chrisshroba• 3 years ago

What if the data is dynamic but small and low qps? What are the current easiest projects and services for authed data CRUD? Is firebase still good?

firecall2• 3 years ago

I’ve used it a bit recently and it doesn’t seem terrible!

But only for prototyping and testing the auth and database tools.

Haven’t really made a decision on it vs rolling my own in Rails.

dkh• 3 years ago

Supabase[1] is a popular, open-source[2], and very modern alternative with a great free tier!

[1] https://supabase.com/

[2] https://github.com/supabase/supabase

villgax• 3 years ago

Was this is a joke or something? You could have always done this with whatever framework you were using including Express, FastAPI, Flask & everything else....

LVB• 3 years ago

It was a sarcastic reminder to KISS.

oneeyedpigeon3• 3 years ago

You could even do it without a framework altogether!

Zaheer• 3 years ago

This is essentially how Levels.fyi operated for several years: https://www.levels.fyi/blog/scaling-to-millions-with-google-...

firecall2• 3 years ago

I’ve got some JSON being served to an iOS App from the cheapest of hosting which is ~5 lines of PHP to get the data from MySQL and return JSON.

It’s been running unattended for a decade….

Alifatisk3• 3 years ago

The good ol’ days

chii2• 3 years ago

And serving it over plain http is how you can get Man-In-the-Middled!

marginalia_nu6• 3 years ago

To be fair, you can get MITMd with HTTPS too. How do you think Cloudflare inspects web traffic?

grokkedit• 3 years ago

not sure that MITM definition applies here: cloudflare is serving the data, they're not in the middle, they're either alice or bob

intelVISA5• 3 years ago

Please, don't gender Cloudflare they prefer 'CFitM'!

etoxin• 3 years ago

It's pretty common for cicd pipelines to generate static json like this to track environment info.

fiddlerwoaroof• 3 years ago

This technique of producing an “API” from directories of json generated by batch jobs and creative use of symlinks can be surprisingly effective. I wrote a simple program to transform a list of RSS feeds into a directory tree that I’ve been using for years as my feed reader API to great effect.

captn3m0• 3 years ago

We do this at https://endoflife.date API, and it works quite well.

jarofgreen• 3 years ago

I really like endoflife! It's great for looking up dates with a consistent UI and also codenames - I could never learn that Debian Bullseye was 11 or all the other silly codenames so I look them up here. Thank you!

We also do something similar on Datatig, which is a tool I'm writing for the general situation of crowdsourcing data in Git repos. It makes it easier for people to submit data and then creates usable data and handy websites. The static website output has an API which is just static JSON files. https://pypi.org/project/DataTig/ or https://datatig.readthedocs.io/en/latest/

oneeyedpigeon3• 3 years ago

Yeah, I've always tried to generate RSS feeds when needed rather than on-demand — it strikes me that the ratio of requests to updates must be so enormous that it doesn't make sense doing anything else.

mkleczek• 3 years ago

KISS is good... until it isn't.

At my work I struggle with the opposite: all problems are being squeezed into "let's put it into static JSON on the CDN" - which ends up with a complex custom JSON based language (schema) to support sharing information between apps, subsetting information (ie. search) etc - ie. implementing an ad-hoc one-file database. Ahh... and don't forget about complex CI/CD pipelines and Git setup that allow business users to manipulate data in these files.

I found "just use Postgres for everything" a much saner default in the end.

xlii• 3 years ago

don’t forget about complex CI/CD ...

So it’s simple or complex because I got confused ;)

KISS is ALWAYS good. But simplicity doesn’t mean easy or naïve solutions. Naïve solutions get very complex since the initial input is low and they don’t cover edge cases by design making it tangled mess.

Simple solutions are often very difficult to create, require a lot of input to be simple, but they cover a lot of cases behind the curtains and are easy to describe.

Your “use Postgres” strategy is more KISS than doing complex mocking. Sure, initial effort is higher, but using well known, powerful yet ergonomic singular entity is as simple as one can get on some levels. It’s equivalent of “just use Excel for it”.

I agree with you, but I think tagline is misplaced.

Freedom2• 3 years ago

What is KISS? Apologies, I'm unfamiliar with some acronyms.

justinclift2• 3 years ago

https://en.wikipedia.org/wiki/KISS_principle

keyle2• 3 years ago

That sounds like a PITA solution more than KISS. That said, I'd rather walk into that business and solve that problem, than walk into a business that is fully cloud serverless hooks all over the place.

So yes, Kiss is good until it isn't... because it wasn't kiss anymore. Moving to the postgres solution sounds like it's the new Kiss.

raxxorraxor• 3 years ago

fully cloud serverless hooks all over the place

I don't know how this will ever not be a complete maintenance hell. It is even far worse than the monolithic "I push every bit that needs to be pushed for the whole enterprise - Sync Tool".

ptman• 3 years ago

"Make things as simple as possible, but not simpler." - Einstein (paraphrased)

foul• 3 years ago

The crazy habits proponent in your workplace is simply ahead of his time: there is the "SQLite for edge" vulgata now and he/she may skim some of the wheel reinvention and funny engineering problems with throwing around SQLite files, like in the good old Access days.

People should not cache something that changes very often.

JohnFen4• 3 years ago

The situation at your workplace isn't KISS. It's the opposite.

The fuller version of KISS is "keep it as simple as possible, but no simpler".

kokanee• 3 years ago

I do this when possible. Caching can be a pain to get right though. If you’re set up to do hashed filenames for JS and CSS already, you can leverage that for cache busting your json, and then you need to inject the hashed filename into the app somehow.

2143• 3 years ago

I like your website. Nice and clean. Even the underlying HTML is clean. Well done.

mholt• 3 years ago

   caddy file-server

does this, but more production-ready, honors cache headers, sets Etag, etc.

Also:

    caddy respond

can be used to hard-code specific responses if you're testing an HTTP client. It can even spin up on a whole port range and supports templates in the response text right there on the command line:

    caddy respond --listen :2000-2004 "I'm server {{.N}} on port {{.Port}}"

Here's an example maintenance page:

    cat maintenance.html | caddy respond \
         --listen :80 \
         --status 503 \
         --header "Content-Type: text/html"

https://caddyserver.com/docs/command-line

Alifatisk3• 3 years ago

Is caddy that easy to setup and run?

yjftsjthsd-h2• 3 years ago

I'd say "easy to setup and run" is arguably caddy's entire point:) Single binary file, yes running is that easy for simple stuff, and honestly it's not exactly complicated for more complex stuff. It got a lot of its popularity because it made HTTPS, including automatic cert provisioning, utterly trivial.

iainmerrick• 3 years ago

It’s pretty easy, but I find the config file format rather confusing. In the classic Unix style, you’re learning yet another little language, so there’s a learning curve. Once you learn it, it’s fairly powerful and expressive, but the docs aren’t very clear.

e12e• 3 years ago

Yes.

Caddy excels at making simple [1] use-cases trivial - either with no config, or 3-5 lines of config.

It might seem jarring comming from Apache or nginx - but i think a lot of it comes from the developers having destilled a lot of common use-cases and carefully chosen reasonable defaults.

[1] Simple to articulate; not necessarily simple to execute - eg letsencrypt works out of the box. Caddy will give you https if you ask (or rather unless you go out of your way to avoid it).

It can be a great stop-gap in running an ssl proxy in front of a legacy application server stuck on some horrible old distro, for example - since the binary just needs a Linux kernel (and can ignore old ssl libraries).

yjftsjthsd-h2• 3 years ago

It would be polite to note that you wrote caddy. (It's a good tool and appropriate here, but disclaimers for self-promotion are always in style)

praveen9920• 3 years ago

I was working on content heavy web app few years ago and realised that lot of database reads are happening to very few "popular" posts then it hit me that we can just host that content as json files using Nginx without API getting involved. That worked like a charm. Decreased usage of database, decrease in latency. etc. In hindsight we could have just used CDN but for the shoestring budgets we had, we achieved lot of scalability with single nginx server.

justinclift2• 3 years ago

A common approach in between these two is to use something like memcached (or redis) to cache the results of common requests.

That also tends to provide results faster than the database layer, so better latency and less database load.

It's not as fast as serving static content via nginx or a CDN though. :)

Attummm2• 3 years ago

The python server is not meant for production, serving files from it is incredibly slow and resource intensive. It also is not full webserver. There are better options for serving files

psnehanshu• 3 years ago

Ofcourse, this was just an example. One of the prod solutions can be Nginx.

Attummm2• 3 years ago

Adding that to the blog post would strengthen the argument. We shouldn't assume the knowledge level of the reader. Examples become production all the time. Using files with unix philosophy is a tried and tested way to be really productive.

jgaa• 3 years ago

You can serve static data over HTTP

Yes. Just like you can put letters together and form words. It's kind of what it was designed to do.

fredrikholm• 3 years ago

You'd be surprised talking to the younger crowd.

FabHK• 3 years ago

Some time ago, a website was posted here (forgot the topic) which consisted of .md files, and a JavaScript library which would then, on the client, render the markdown on the fly.

Why on earth the markdown was not pushed through a renderer so that then the server could serve static HTML is probably a question that only occurs to old timers like us.

oneeyedpigeon3• 3 years ago

I also initially misread the tone of the title. It's heavily dripping with sarcasm.

vermon• 3 years ago

I built and API for an internet radio station archive. Initially I tried to build an app but instead I realised I can run a scheduled Github Action that runs every hour, scrapes the archive, builds a Json file and deploys it to Github Pages for free: https://bd.maido.io/api.json

willwinger• 3 years ago

Had a smile reading through this. So many web developers out there who have no idea about history of HTTP. That it was designed for static document serving and since been twisted to serve applications.

Roark66• 3 years ago

Here I was expecting this will be about not needing HTTPS for everything static...

shinycode• 3 years ago

There is https://github.com/typicode/json-server that is really useful for that matter

SenHeng• 3 years ago

For a 'fake' API, I can't overstate just how powerful this is.

I once spent an entire month building a proper API server out of this. Then one day while tweaking some DTOs for some POST endpoints, I realised what I was doing and switched to Postgres.

pSYoniK• 3 years ago

This is great advice and its most of the time everything you need for a personal website.

A generation that grew up with Wordpress et al. will have dificulties getting that really all there is to a website is some html. I switched from a Spring Boot web app that used Thymeleaf for the templating engine, with a MySQL db on a Linux vps and Heroku (I did it to learn/practice) to a Hugo blog that I publish locally then scp to a VPS with Caddy setup to serve the static content. Nothing to hack/maintain/worry about.

As an extra point I would even complete the suggestion with adding direct HTTP not S, support. Your page will load even faster and you're transmitting some static content anyway.

bladelessninja2• 3 years ago

I hope the author means HTTPS, not HTTP - otherwise, I don't think this is a good idea.

Anyway, as everything in software development: it all depends - serving a simple JSON over an existing "full blown web application back-end" is usually more convenient and less confusing for developers than configuring hosting for static content. Although, if you JUST want to serve a JSON then, yeah, you can add it as a static resource.

edwinjm2• 3 years ago

It's not illegal to use static files and it does work.

No, it doesn't. Maybe for your personal homepage with three hits per month, but even on a low traffic server, static files gave me headaches.

1) When two visitors write to the file at the same time, the data of one of the visitors will be gone. This can be solved by using lock files. This is a topic on its own (what happens when two processes both create a lockfile at the same time?). Also, when there are a lot of writes happening, multiple processes will wait for the lockfiles, creating load on the server and wait times for the visitor.

2) When a file is read during while another process is doing a write, the file is only read partially. There are workarounds for this as well, you can first write to another file and when done, move this other file to the correct file.

As the number of visitor increases or the file size increases, problems will get harder.

Nowadays it's not hard to add a database. Whether it's on your server or in the cloud. It's performant, it works and it will grow with increasing demand.

It's probably easier to set up than a file based solution.

Sometimes what seems to be simple isn't really.

iofiiiiiiiii• 3 years ago

Sounds like your files are not static, therefore your problem is not the same as the problem the article is addressing.

pjc502• 3 years ago

If there's writing going on, you're not using static files.

edwinjm2• 3 years ago

Then I'm mistaken.

KomoD• 3 years ago

Doesn't sound static to me.

ygouzerh• 3 years ago

For mocks, it's a good idea.

For the rest, even for a prototype, it seems a lot of trouble to insert, update and delete data.

nulbyte• 3 years ago

Actually, text editors are pretty good these days.

But on a(n only slightly) more serious note, maybe the data is generated outside of the web server. Different projects have different needs, but there are plenty of them that just need to serve some mostly static data.

ThatPlayer• 3 years ago

I've taken this to an extreme myself with 10MB+ JSON files. I would recommend enabling gzip compression, but also not letting it get to that point. I'm considering switching to sqlite as that has been ported to WASM and allow pages

politelemon• 3 years ago

Along similar lines. If you're starting out with feature toggles for a web application, in many cases you don't need a separate third party application that manages the flags for you. You can just serve a .json file.

mpweiher• 3 years ago

Yes, so much in programming is so much simpler than we think it is.

I first came across this when creating a demo for a smartpen that was supposed to have live interactivity features, which I was going to implement by putting a little HTTP server on the pen.

When it came to simulating my solution, I took away features until I came up with the "serving static files over HTTP" trick the post describes.

Then I realised that I could do the whole thing by just putting the demo files in a directory and changing the base URL in the client from https://... to file:///...

Another time we built a backend system for feed processing and delayed putting in a database until we discovered we didn't actually need it.[1]. The standard operating procedure at the time was to create an Oracle DB for everything, including a system that needed a total of 6 data items. Wow.

My boss at Wunderlist used to brag that the database we used for the iOS/macOS clients was just a bunch of JSON files in the filesystem. Worked wonders and was really, really fast.

These and many other experiences led me to come up with Storage Combinators[2]. Once you try them, you'll wonder why things always have to be so complicated.

And yes, I agree with the idea that "gluing complex libraries together that we don't fully understand" is a problem. And also with the notion that it's an achievement that we can and should be proud of, at least partially, because that used to be something we couldn't do at all[3]. Nowadays we can at least do it badly[4].

My take on this is that we don't have the right kinds of glue, particularly procedural/functional abstraction is often no the right glue, and OOP notwithstanding that's really all we got. [5][6]

[1] https://link.springer.com/chapter/10.1007/978-1-4614-9299-3_...

[2] https://2019.splashcon.org/details/splash-2019-Onward-papers...

[3] https://blog.metaobject.com/2021/07/glue-code-is-success-con...

[4] https://blog.metaobject.com/2021/06/glue-dark-matter-of-soft...

[5] https://blog.metaobject.com/2019/02/why-architecture-oriente...

[6] https://objective.st

bhanu423• 3 years ago

I usually just use `npx http-server` which gives a server that supports concurrent streams for faster downloads over the python server(which I use on devices where python is present and node isn't). I often use it transfer file over wifi from a device to other.

billyhoffman• 3 years ago

Reminds me of a funny take from @marginalia: The Static File Startup:

We’re going to disrupt Internet hosting and serve static files in a novel way through a cloud based SaaS.

https://www.marginalia.nu/log/51-the-file-startup/

marginalia_nu6• 3 years ago

Looks like first few lines the post got a bit mangled when I switched blog systems :-/

andrewmcwatters• 3 years ago

It’s funny but it hurts. I read the whole thing amused but with disdain on my face.

dolmen• 3 years ago

Content-Type?

intelVISA5• 3 years ago

the heck's an octet-stream?

slim• 3 years ago

This is called PCM (path controller mapping), it's very new, I'm going to give a talk about it at next aws summit /s

You can serve static data over HTTP

Related Stories