Lobste.rs is now running on SQLite

https://lobste.rs/s/ko1ji1

by abetusk • 9 days ago

228 169 9 days ago

lobste.rs

tetris11• 9 days ago

Oh wow. Off-topic but quite refreshing to see a site not absolutely plastered in AI posts

grebc2• 5 days ago

HN is increasingly bot central, I kind of assume 80% of the respondents are artificial.

Lobsters still got a pulse, largely.

rickcarlino• 5 days ago

Invitation-only is the future of online communities. Lobsters still has a pulse because you need to know someone to get in and it’s a bad look if you invite annoying people.

OutOfHere3• 5 days ago

That's an unsubstantiated and false accusation wrt the 80%. If you believe it, why are you even here? Are you one of them?

gspr• 5 days ago

It's a wonderful place. It feels like HN, but with the Silicon Valley attitude dialed way way way down, and with a much narrower scope (politics, economics and entrepreneurship are all off-topic, for example).

christophilus2• 5 days ago

Huh. I’ve largely avoided it, but that does sound pretty nice.

NetOpWibby• 5 days ago

The SV attitude isn't there but there's active assholes who wield their moderation weirdly. I deactivated my account before deactivations were added to the logs and I don't remember what made me quit. I've largely avoided Lobsters since.

I joined their IRC channel last week and it's been rather chill.

I do like the invite-only nature of the site though, keeps quality high...while making some folks tyrannical.

ai_critic• 5 days ago

Slight clarification: politics that disagree with the zeitgeist of the site and the mods is off-topic.

maxloh6• 5 days ago

Hi. Could you send me an invite? Email in my profile :)

Aurornis4• 5 days ago

Lobste.rs is very anti-AI. They force any LLM topic to be tagged with “vibecoding” even when the majority of LLM posts are not about vibecoding.

AI posts are usually the most commented on. Linus Torvalds’ comments saying that LLMs are actually useful is still on the front page, tagged as vibecoding, and has a lot of comments from people mostly disagreeing with him.

Lobste.rs is more of a monoculture than Hacker News. If you get downvoted enough they stick a banner on the top of the page inviting you to delete your account. It has a nasty side effect of driving away good contributors who don’t align perfectly with the hive mind. It’s a site where you learn to keep your mouth shut if your input doesn’t agree with what the core users want to hear.

dgellow• 5 days ago

I honestly don’t find any of this concerning

xtracto• 5 days ago

I dont remember why I was kind of barred from lobsters , but I applaud all this behavior you mentioned. In the end, that's how normal clickes of friends/interest groups form (or formed 30 years ago in real life.)

If you got into my group of death metal people, and suddenly started talking about Oasis... we would invite you to get out haha.

Same with patriots.win . They may be a group of right wing biggots, but hey, they did their platform and are happy talking there.

joemi• 5 days ago

It's a lot smaller of a community than HN, so it will naturally have a smaller range of opinions merely due to the fact that it has a lot fewer users.

That said, I don't believe it's as much of a hivemind as you've portrayed. I've seen dissenting opinions voiced there which weren't downvoted (or at least not noticeably), while the same opinion in a similar thread here on HN would be downvoted and flagged. So I don't think that getting downvoted enough so that you're invited to delete your account is a common occurrence with productive members.

mort96• 5 days ago

Driving away silicon valley tech bros who spend most of their time boosting AI is actually a prerequisite for building a good community.

abc42• 5 days ago

Yeah, it's starting to be a bit idiotic. Some of that stuff is here too.

arjie3• 5 days ago

I originally went to lobste.rs because I wanted to avoid repetitive discussion and get more technical stuff. In time, though, lobste.rs got fairly boring since it's always the same stuff over and over again. I found an alternative means[0] of filtering out boring things here on HN and it's enjoyable to use again.

0: deleting comments of people who aren't interesting from my view and highlighting ones who often say things that are interesting, the user I'm replying to is one of the latter

BoingBoomTschak• 5 days ago

Being anti-AI is fine by me, but the hardcore Tumblr dogma can be very tiring sometimes; even if only as a reader interested in the valuable technical comments.

cromka3• 9 days ago

Wait, no edge cache, just one node with SQLite db?

thomasdziedzic• 7 days ago

Caddy does cache a portion of the traffic, and what it can't cache makes its way to rails.

packetlost2• 5 days ago

For a relatively low-traffic site like Lobster's, yeah I'm not surprised they don't need anything crazy. SQLite is really efficient and modern hardware is really fast.

booi• 5 days ago

I don't think SQLite is particularly optimized for modern hardware. It's just crazy efficient with a regression test suite many projects wish they could have.

homebrewer• 5 days ago

It's been fairly unstable recently, pages sometimes render for several seconds which I've never seen under MariaDB. Used to be instantaneous, always.

Sometimes (maybe 5% or less) the request won't render at all, and you get a browser error page.

Today they ran into this bug, lost a bunch of voting data, and went into read-only mode for several hours:

https://github.com/rails/rails/pull/57128

I wonder how much of this is usual bugs which crop up during major database migrations, and how much is caused by choice of SQLite.

srcreigh• 5 days ago

Pages (especially threads while logged in) taking seconds to load definitely happened to me pre SQLite migration.

eterm4• 5 days ago

SQLite definitely seems like a poor choice for dealing with many concurrent requests.

Maybe it's improved since I last used it, but to my knowledge SQLite essentially forces all writes to be serialised, at risk of data corruption otherwise.

There are tricks for improving the performance such as WALs, but that is merely a performance boost rather than genuine concurrency with things like row-level locks that you might find in other databases.

I guess if the whole thing is architected with a write-through cache that handles concurrent writes and deals with serialising all the writes, then it can be a single writer streaming changes through to the database, but then you still have a point of serialisation, it just will manifest itself slightly differently.

And SQLite is something that will give you constraints you will always have to consider.

Whereas running mariadb or postgres on the same machine would deliver similar benefits without a risky migration.

If your DB is small enough to run as a SQLite database, then it probably ought to have never been on a different machine in the first place.

There is a very happy medium between SQLite and a database on a different machine, one I am continually surprised to see people ignore.

veqq3• 5 days ago

edit: misread requests as writes

wild_egg• 5 days ago

SQLite definitely seems like a poor choice for dealing with many concurrent requests

Can you qualify "many"? SQLite easily handles 100k+ writes per second and it's not hard to have app layer code serialize and batch writes to take advantage of that speed. Concurrent writes require a ton of overhead and your system and code can be quite a lot faster and simpler if you just skip the idea of them altogether.

alfiedotwtf4• 5 days ago

Depends on the architecture..

They could have one SQLite instance per user and then have a single sweeper that goes through all last writes and then replicates it to the main instance - eventual consistency fanned out across files

torginus• 5 days ago

Can't you do infinity read replicas? I don't think the write load on lobster.rs is severe - maybe 1 comment/sec, with an acceptable replication delay of 60 sec?

If you can do that, you can have infinity instances.

anonzzzies• 5 days ago

Not sure if this is SQLite or driver/Rails issues? We use it for sites and systems with a lot more (but similar) traffic and have no issues; we did write the drivers we use ourselves (for Node) as we got into much issues with the popular/existing ones.

abrookewood• 5 days ago

I jsut had this happen - it was my first visit to the site and probably my second page load. Tried opening the About page and it was completely blank. Thought it was odd & reloaded, which rendered correctly.

ekjhgkejhgk• 5 days ago

It's super slow.

Kuyawa• 5 days ago

No intention to antagonize but... why not postgre? Even my side projects run on postgresql with no overhead at all

adenta• 5 days ago

Where are you hosting postgres

dewey2• 5 days ago

On the same machine where they are currently storing the sqlite file maybe? It's not that hard to run. A sqlite backup cron is also not that much more work than a pg_dump cron.

inigyou13• 5 days ago

if they have only one server, then on that server. If they have a load-balanced front end and central database, then on the database.

gnulinux• 5 days ago

run on postgresql with no overhead at all

Citation needed. I love postgres as much as the next person, but it does have more overhead than Sqlite which is in an in-process db linked through compiled C code, it doesn't run as a separate server. Very significant difference that when you use Sqlite db, there is typically no db process other than your application logic, unless you implement the server yourself. If you don't want your application to have multiple processes (say, as a toy example) then it totally makes sense to prefer Sqlite just for this reason. Sqlite and postgres are different tools, they serve different purposes.

christophilus2• 5 days ago

If you run it on your one app server, it’s not much more work than SQLite. SQLite requires some special care when you’re dealing with multiple processes, and backups are about the same amount of effort.

dewey2• 5 days ago

It's also explained in the post shared:

---

I've heard "why not PostgreSQL?" a few times this week. It was even our original plan in #539! Well, it was a pragmatic choice in two different ways:

The person who volunteered to do the work used SQLite.

I don't want to use solutions that are bigger and more complex than our likely needs. Postgresql is my default for projects, but it does have the added complexity of being a separate service to run, tune, and maintain.

ozim• 4 days ago

The person who volunteered to do the work used SQLite.

That is a totally valid reason.

Though if you can run it from SQLite there is not much tuning on Postgres that would be needed. I would expect migration to Postgres from MariaDB would be easier.

cromka3• 5 days ago

Ironically, this post resurfaced on HN with my comment here shown as if made 15 minutes ago, which I in reality did last week?

stymaar2• 5 days ago

Second chance pool, most likely.

mh-2• 5 days ago

I didn't realize that re-timestamped comments. That's a pretty unexpected behavior, if true. I wouldn't like if that happened to a comment I left - a lot can happen in a week.

dredmorbius• 2 days ago

<https://news.ycombinator.com/item?id=48740177> (dang, on 2CP & timestamps).

More: <https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...>

evanelias5• 5 days ago

Some of the cited reasons for moving off MariaDB [1] seem misguided, in my opinion. Especially the part about "K1 are very enterprise-focused, so the database is likely to focus its work on features that are not relevant to us. There's increased risk they drop the free/open source version we use"

K1 acquired the commercial entity behind MariaDB Enterprise, but that's separate from the non-profit MariaDB Foundation. And there's literally zero risk of the MariaDB server suddenly going closed-source; as a fork of MySQL (which is GPL), this is not even legally possible!

[1] https://github.com/lobsters/lobsters/issues/539#issuecomment...

edoceo• 5 days ago

And even if it did go closed-souce, your existing install can run for ages because the vendor cannot force upgrade. One of my favorite features of FOSS.

NuclearPM• 5 days ago

How could it go closed source?

vasco• 5 days ago

This makes little sense but I guess they get little traffic so pretty much any database will work for them.

If traffic grows they will find problems, then limitations, then move to postgres. And if not it's fine. Story old as time.

saila• 5 days ago

The GPLv2 doesn't prevent a company from distributing binaries and making the source available on request while development is done behind closed doors. This wouldn't really be "open source" as we typically understand it and might be cause to exclude it from some Linux distros, for example. I'm not saying that K1 would do this (I have no idea who they are or what they do), but I also think it's a reasonable risk to consider.

evanelias5• 5 days ago

I don't see how that would be a realistic risk? Any MariaDB user (including Lobste.rs) could request the source, and per the GPL it must be supplied, so this isn't an effective strategy for any entity to go closed-source.

And in any case, https://github.com/MariaDB/ links to mariadb.org which is run by the MariaDB Foundation, not the commercial enterprise owned by K1.

inigyou13• 5 days ago

That was precisely how the original free software movement started, actually. Work was done behind closed doors and if you didn't like it, you released your own version with your changes.

jcgl• 4 days ago

What Linux distros do you think would exclude on the basis of closed door/cathedral-style development? I’ve never heard of this.

zinodaur3• 5 days ago

As a big user of MySQL - moving off of it does seem like a good idea, if you can. The longer you wait, the harder it will be to flee

alfiedotwtf4• 5 days ago

Any reason besides trust me bro?

evanelias5• 5 days ago

It's a large ecosystem with multiple server vendors (Oracle MySQL, MariaDB, Percona, VillageSQL) that are relatively easy to migrate between, compared to moving to a completely different DBMS ecosystem.

Sometimes there are valid technical motivations to move away completely, but a nonsensical fear of a massively-used GPL database somehow becoming closed-source (with no surviving forks or continuity) is not one of them in my book!

kenforthewin• 5 days ago

The site is performing poorly for me. For example, the login page took 9 seconds to load, same with the home page.

cenazoic• 5 days ago

May I have an invite, if anyone has one to spare?

abetusk8• 5 days ago

Provide an email or email me (abetusk@mechaelephant.com) so I can get yours for the invite.

abofh• 5 days ago

I hope it's profitable, because this feels like engineering for engineering sake and not actually improving the product in a way that draws customers.

rfmoz• 5 days ago

Another SQLite user here. My site gets constantly hammered by crawlers, and SQLite just keeps going on hit miss. Anyway, the real magic is on the ram and the ssd perf.

cdrini5• 5 days ago

keeps going on hit miss

(Is this a typo?)

varun_ch• 5 days ago

Does HN still use simple files for storing its data?

luciana1u• 5 days ago

sqlite for a site that debates database migrations is like a chef who eats microwave dinners at home. makes perfect sense and somehow still feels like a betrayal.

oandrew2• 5 days ago

I'd appreciate a lobste.rs invite if anyone has one. Email is in my profile. Thanks!

Imustaskforhelp2• 5 days ago

I'd appreciate a lobste.rs invite if anyone has one. Email is in my profile. Thanks!

I'd like one too if someone wishes to invite me as well, Thanks and have a nice day!

uproarchat3• 5 days ago

Successfully e-mailed invitation to anonplusplus@proton.me

uproarchat3• 5 days ago

Successfully e-mailed invitation to hn@oandrew.dev.

maxloh6• 5 days ago

Hi. Could you send me an invite too? Email in my profile :)

oandrew2• 5 days ago

Thank you!

chasil• 5 days ago

Some observations:

1. Stats are used by "Cost-Based Optimizers."

I think this is the most appropriate wiki:

https://en.wikipedia.org/wiki/Query_optimization

On my Oracle databases, I run the dbms_stats.gather_database_stats package procedure once a week. SQL Server also has stats for sure, and I would be surprised by a relational database that did not.

SQLite will quietly gather stats with each database connection. Running an explicit analyze devotes full attention to an otherwise piecemeal group effort.

2. I don't think rows are physically removed when deleted, instead they are marked. Removal will definitely happen at vacuum.

3. There is a backup command in the CLI that will make a transaction consistent copy of the database file, and there are lower-level C functions behind it that can be used with object code.

4. It is important to remember the many restrictions of WAL mode, including the impact on ATTACHing multiple database files: ACID consistency is lost.

https://sqlite.org/lang_attach.html

'Transactions involving multiple attached databases are atomic, assuming that the main database is not ":memory:" and the journal_mode is not WAL.'

amazingamazing2• 5 days ago

Software “engineering”. Thank goodness bridges aren’t built with this level of rigor.

In any case I will take an invite if one has one because the community is solid.

abetusk8• 5 days ago

Give your email or send me a message (abetusk@mechaelephant.com) so I can send the invite.

amazingamazing2• 5 days ago

Done. Just added to profile, thank u!

Peanuts99• 5 days ago

I've got a few planks over a ditch in my garden. The bridge only needs to be designed for the load and then some. In this case, this seems to work just fine.

blobbers• 5 days ago

Do you feel like lobste.rs is a better version of HackerNews?

veqq3• 5 days ago

They're very distinct. Both are good in their own ways like elephants and dolphins. Though HN suffers more under AI spam, it permits wider topics and has a wider userbase.

xboxnolifes• 4 days ago

I only lurk occasionally, but lobsters feels like a much smaller, less jaded community then HN. The content is very similar. Hopefully not too many people who get invites from this thread bring over the jadedness.

arjie3• 5 days ago

Pretty cool. I do default to SQLite these days simply because I can just put it on the host, set up a PVC to it, and run the app in Kube pointing at a persistent SQLite. Absent some other need (it's a fork of psql- or mysql-based software) I prefer using it for simplicity.

I do have to say that I've found lobste.rs generally a pretty slow site to render and navigate and I wonder whether that's the Rails backend, or underpowered hardware, or lack of caching. It's more feature-ful than a HN or Reddit though with much lower usage so I wonder what it is.

dmix• 5 days ago

Nice to see Lobsters is made with ruby

DHH has made a big deal about using SQLite in rails

kgeist2• 5 days ago

They use WAL in SQLite. If I continuously perform reads/writes so that they overlap with no gaps, I can make their VM go down because SQLite will not have time to initiate a checkpoint to trim the WAL file. SQLite waits for a time window without any active reads/writes before starting a WAL checkpoint. If there isn't one, the WAL will grow indefinitely, eating up all the disk space on the VM.

It's in SQLite's documentation, and almost no one switching to Sqlite seems to be aware of it because no one discusses it in blog posts like these. I guess most projects switching to SQLite have very low traffic and no malicious users (yet)

abetusk8• 5 days ago

Could you go into more detail about this issue and provide some links to the documentation?

I'm absolutely no expert, I'm just reading about it now, but from the SQLite WAL documentation [0]:

7. There is the extra operation of checkpointing which, though automatic by default, is still something that application developers need to be mindful of.

and from their "checkpointing" documentation [1]:

By default, SQLite does a checkpoint automatically when the WAL file reaches a threshold size of 1000 pages.

I'm only skimming but I see no mention of waiting for an idle time window.

Have you been successful in DoS'ing lobste.rs?

[0] https://sqlite.org/wal.html

[1] https://sqlite.org/wal.html#ckpt

kgeist2• 5 days ago

On the page you linked:

However, if a database has many concurrent overlapping readers and there is always at least one active reader, then no checkpoints will be able to complete and hence the WAL file will grow without bound.
This scenario can be avoided by ensuring that there are "reader gaps": times when no processes are reading from the database and that checkpoints are attempted during those times.

Dunno, maybe Rails has a built-in workaround for this.

My workaround was to run a separate thread that monitored the WAL size on disk every second. If it went above the target size of 8 MB, my framework would enter "slow down" mode, where all reads and writes were artificially delayed by calling "sleep()", starting at 16 ms and gradually increasing the sleep time based on a few heuristics.

This allowed the application to have short gaps with no reads or writes, so the checkpointer could actually proceed.

shirol• 4 days ago

That's only true in theory. In practice, you'll be rate-limited first. Even if you aren't, the database won't be the bottleneck because SQLite is so fast. Their Ruby on Rails stack will suffer thread starvation way before the WAL grows. Rapid requests still leave microsecond gaps between context switches for SQLite to run an automatic checkpoint. To actually make SQLite stall, you'd need an endpoint that lets you hold a massive write transaction open indefinitely, which lobste.rs doesn't expose anyway (I hope).

mikae1• 4 days ago

I guess most projects switching to SQLite have very low traffic and no malicious users (yet)

Lobsters is invite only. That helps. :)

Onavo• 4 days ago

In your previous thread the solution was either WAL2

https://news.ycombinator.com/item?id=40688336

chaz6• 4 days ago

Is that solved with wal2 mode?

HackerThemAll2• 3 days ago

You can force explicit CHECKPOINTs to combat that.

rsync• 5 days ago

sqlite3_rsync exists:

https://sqlite.org/rsync.html

... and is installed and available:

  # sqlite3_rsync -vv ./test.db user@rsync.net:test.db

"Ordinary rsync does not understand SQLite transactions. Rsync will make a copy of ORIGIN into REPLICA, however the copy might not be consistent. Parts of the copy might be from one transaction, while other parts might be from a different transaction. The database copy might be corrupt."

dcmatt• 5 days ago

Wow, lobsters seems refreshing! Invite only to contribute must be why they can run on sqlite so effectively. I've run into so many issues with WAL (real time market data) unless I'm painfully deliberate with reads and writes. Don't feel like jumping on the "ohhh invite me" bandwagon, but definitely subscribing to their feed.

hnarn• 5 days ago

Moving from MariaDB to SQLite and not Postgres for a web application of this scale is insane. I wonder what went into the discussion behind this.

HackerThemAll2• 3 days ago

Have you ever contemplated the true performance of SQLite? Expensify reached 4M point database queries per second, and an artificial (more like parser-testing, not database-touching) 160M qps. If your backend is on a single machine of predictable performance, and you properly index your queries, and you don't require remote DB connections, you don't need anything else but SQLite for 95% of use cases.

https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...

MitziMoto• 5 days ago

Although the site is very fast for me as of this writing, it looks like they're running into an issue:

https://lobste.rs/s/ko1ji1/lobste_rs_is_now_running_on_sqlit...

I hope they get it sorted soon, I'm rooting for sqlite.

p2hari• 5 days ago

Everytime Lobste.rs get posted, I get to think should I dive in. This time maybe. Could somebody be kind enough to give me an invite. Email in my profile. Thanks.

abetusk8• 4 days ago

Invite sent. Let me know if you didn't get it.

wewewedxfgdf• 5 days ago

I would visit lobste.rs more if it filtered out stories that are on Hacker News.

I'm not interested in a rehash of Hacker News but I am interested in more tech news.

Anyhow on topic: I find it puzzling the whole "use sqlite in production!" thing.

- you're stuck on a single machine now.

- you don't have remote database access.

- you have to live with the inflexibility of sqlite's schema change capability.

- all writes go through a single writer.

- fewer data types

- fewer backup/security/high availability options

- foreign keys and strict disabled by default.

- you're single core in a multi core world.

It's not a terrible decision but when you can choose Postgres, why would you go with sqlite?

simonsan• 5 days ago

A single page answers that question: https://www.sqlite.org/whentouse.html

Levitating• 5 days ago

Lobste.rs seems cool. Would someone invite me?

abetusk8• 4 days ago

Invitation sent. Let me know if you didn't receive it.

artirdx• 10 hours ago

Could I also get an invitation? Email in profile.

dzonga• 4 days ago

my take to benefit from SQLite:

make sure you're not doing a bunch of migrations. that's the SQLite weak point that I know of & have experienced.

use SQLite - as if you're using a KV or better yet maybe a CQRS approach.

Lobste.rs is now running on SQLite

Related Stories