some_developer

Docker in docker, with opencode.

Opencode plus some scripts on host and in its container works well to run yolo and only see what it needs (via mounting). Has git tools but can't push etc. is thought how to run tests with the special container-in-container setup.

Including pre-configured MCPs, skills, etc.

The best part is that it just works for everyone on the team, big plus.

It's hyperbole, but like so often, there's a grain of it truth somewhere.

Also, like most things, Laravel is built on the shoulders of others. It sometimes makes hard things easy to access (ppl like this) and hides complexity away, clouding how things actually work (ppl don't like this).

I primarily use Laravel but like to think I code in a generalist if way as to not get stuck in its system.

It's the creator of Laravel, Taylor Otwell

Years ago I wanted to use native FTS (because of tall the things mentioned, having to sync to external simply adds complexity) and it failed at another point.

Not completely surprising, but on a table with _potentially_ couple of thousand of inserts / seconds, it slowed down the overall updates to the point that transactions timed out.

We already added an index for one of the columns we wanted to index and were running the statement for the second one. The moment this the second index finished, we started to see timeouts from our system when writing to that table, transaction failing etc.

We had to drop the indices again. So, sadly, we did never get to the point to test the actual FTS performance :/ I would have like to test this, because didn't necessarily had to search hundreds of millions of documents, due to customer tenants this would always be constrained to a few million _at most_.

ps: I already wrote about this -> https://news.ycombinator.com/item?id=27977526 . Never got a chance to try it nowadays (newer versions of everything, never hardware, etc.)

I learned the hard way that Gin updates are too slow, and in my case it was not even 100 updates per seconds on average, but could peak to 1000.

How does Pg_bm25 compare here with maintaining the index & performance?

Every time I read blogs like this I'm tempting to get back to the product, still in use, where we scraped the search integration due to the issues we had with pgsql (performance). Although ElasticSearch (nowadays: OpenSearch) is available, we never deemed it important enough for this part of the product.

In case you care, my comment from a few years: https://news.ycombinator.com/item?id=27977526

for ~$188/month.

Note that this is still the on-demand price.

Although it's still not Apple to Apple, no one (sane) would useRDS 24x7 without RI. That probably brings you down another 50 bucks or so.

Last year I also learned that ransomware attackers are not computer gods, necessarily.

My QNAP NAS was open to the internet (a forgotten but-not-in-use port forward on my router) and the famous QLocker ransomware got hold of it.

I got out, I got really lucky. I've millions of files (useless to anyone in the world but me) on my NAS and the ransomware encryption took literally days. I probably discovered it on the 3rd day of its activity.

Someone brighter than me already figured things out and posted this on the qnap forums, that there's a 7zip process running which encrypts the files and they process all files sequentially (aka: takes a long time in my case). This 7zip process gets executed for every file and gets passed the password for it on the command line, though in the process view it was masked for me. But I could just replace the invoked 7zip binary with a shell script and redirect all the arguments -> presto, I had the password.

I then wrote some more shell scripts to decrypt all the files infected so far, wrote it more efficient than the attacker, and within 24 hours everything was back to normal.

Before anyone cries "BACKUPS": yes, of course I've off-site backup and they also were not (yet) affected. But actually I only backed up the real important data for me and my family, due to the costs at that time, I didn't backup _everything_. But I since switched cloud backup storage to Backblaze which I figured was the cheapest vs. acceptable ergonomics for recovery, so I could increase the amount of data I back up.

Lesson learned, I guess. I know not everyone can help themselves like I could, I feel really lucky I got away with just some blood sweating.

Try Octane, it's the app server we were longing for.

I need to use them almost daily for work. AWS SSO and often need access to multiple accounts at once. Without multi containers this would be a serious PITA

Our roll-outs first apply migrations, then deploy the code. Migrations are applied by iterating all databases, and it sometimes it can take for up to several hours, before the code is finally deployed (so that it can use the new schema in every DB). It creates a very large window where old code can see new DB schemas, so we have to be careful for our migrations to be forward- and backward-compatible.

I read this answer last week but had to let this sink in.

I can't fathom how this would feel being "the norm". As a postgres user, I'm accustomed that usually most DDL statements can be done inline in a few seconds, ignoring things like creating indices concurrently which may takes up to 1-2 hours depending on the table.

OTOH I'm sitting more or less on a monolithic database, if there's even a term for this. Yes, it's multi-tenant for tables where necessary (though we don't include that particular customer ID key on all tables as some FK relations naturally provide this separation) and there are times when I'm not sure a tables with 700+ million rows and rapidly growing is a good idea.

From where I sit, having multiple database per server and/or application instances sounds an impossible thing to manage. Let alone that the "application" is a collection of multiple services (micro to medium) and "one just does not run multiple instances" of this cohort. Then there's is the additional challenge that we're receiving "lots-o-webhooks" from various services, a few hundreds to thousands per second for any customer and would need a central service to know to dispatch to which database etc.

If I may ask kindly, is it possible for you to share a bit of insights on how you got on this road to where your company is now? Did you start out _that_ way from the start?

Thanks

For tech ppl, linear has so many pain points (ofc heavily depends on your workflow style) I cry in agony every night.

- email notification are batched unless it's an urgent issue. This is such bollocks, I don't know how they get away with this. Gh issues and jira just work. Workaround: use their Webhook and do it yourself. Makes me so angry.

- can't search comments WTF

- I'm a "tabber", but linears super-fancy-SPA bootstrapping is way behind loading a single issue page especially compared to github but also jira

- github has no wysiwyg and linear ONLY has it, but it has so many subtle bugs, especially with lists. But I'm spoiled with notion. Has its own set of problems, but pure editing is great

Oh boi I could go on.

But I agree, for cross teams with non-devs it works great.

I used to know lot of terminal commands but I'm seriously falling behind due to jetbrains integration which covers 99% of my daily use case. Together with local history, I've never "lost" work in years.

Never heard of switch /restore and will probably forget about it the next time I'm on the terminal.

Rebase and interactive Rebase is so well integrated for my use cases and I feel much more productive without having to switch context.

Reflog and bisect are when I nowadays switch to the terminal, have not discovered an equivalent last time I checked.

Anectodal note:

A few years ago we added yet-another part to our product and, whilst ES worked "okay", we got a bit weary of ES due to "some issues" (some bug in the architecture keeping things not perfect in sync, certain queries with "joins" of types taking long, demand on HW due to the size of database, no proper multi-node setup due to $$$ and time constraint, etc.; small things piling up over time).

Bright idea: let's see how far Postgres, which is our primary datastore, can take us!

Unfortunately, the feature never made it fully into production.

We thought that on paper, the basic requirements were ideal:

- although the table has multiple hundreds of millions of entries, natural segmentation by customer IDs made possible individual results much smaller

- no weighted search result needed: datetime based is perfect enough for this use-case, we thought it would be easy to come up with the "perfect index [tm]"

Alas, we didn't even get that far:

- we identified ("only") 2 columns necessary for the search => "yay, easy"

- one of those columns was multi-language; though we didn't have specific requirements and did not have to deal with language specific behaviour in ES, we had to decide on one for the TS vectorization (details elude me why "simple" wasn't appropriate for this one column; it was certainly for the other one)

- unsure which one, or both, we would need, for one of the columns we created both indices (difference being the "language")

- we started out with a GIN index (see https://www.postgresql.org/docs/9.6/textsearch-indexes.html )

- creating a single index took > 15 hours

But once the second index was done, and had not even rolled out the feature in the app itself (which at this point was still an ever changing MVP), unrelated we suddenly got hit by lot of customer complains that totally different operations on this table (INSERTs and UPDATEs) started to be getting slow (like 5-15 seconds slow, something which usually takes tiny ms).

Backend developer eyes were wide open O_O

But since we knew that second index just finished, after checking the Posgres logs we decided to drop the FTS indices and, lo' and behold, "performance problem solved".

Communication lines were very short back then (still are today, actually) and it was promptly decided we just cut the search functionality from this new part of the product and be done with it. This also solved the problem, basically (guess there's some "business lesson" to be learned here too, not just technical ones).

Since no one within the company counter argued this decision, we did not spend more time analyzing the details of the performance issue though I would have loved to dig into this and get an expert on board to dissect this.

--

A year later or so I had a bit free time and analyzed one annoying recurring slow UPDATE query problem on a completely different table, but also involving FTS on a single column there also using a GIN index. That's when I stumble over https://www.postgresql.org/docs/9.6/gin-implementation.html

Updating a GIN index tends to be slow because of the intrinsic nature of inverted indexes: inserting or updating one heap row can cause many inserts into the index (one for each key extracted from the indexed item). As of PostgreSQL 8.4, GIN is capable of postponing much of this work by inserting new tuples into a temporary, unsorted list of pending entries. When the table is vacuumed or autoanalyzed, or when gin_clean_pending_list function is called, or if the pending list becomes larger than gin_pending_list_limit, the entries are moved to the main GIN data structure using the same bulk insert techniques used during initial index creation. This greatly improves GIN index update speed, even counting the additional vacuum overhead. Moreover the overhead work can be done by a background process instead of in foreground query processing.

In this particular case I was able to solve the occasional slow UPDATE queries with "FASTUPDATE=OFF" on that table and, thinking back about the other issue, it might have solved or minimized the impact.

Back to the original story: yep, this one table can have "peaks" of inserts but it's far from "facebook scale" or whatever, basically 1.5k inserts / second were the absolute rare peak I measured and usually it's in the <500 area. But I guess it was enough for this scenario to add latency within the database.

--

Turning back my memory further, I was always "pro" trying to minimize / get rid of ES after learning about http://rachbelaid.com/postgres-full-text-search-is-good-enou... even before we used any FTS feature. At also mentions the GIN/GiST issue but alas, in our case: ElasticSearch is good enough and, besides the thwarts we've with it, actually easier to reason about (so far).

We've multiple PHP repositories but the longest one currently takes around 7 minutes wall time (that includes: tests, static analysis, code style and a few other misc smaller things). Scaling the phpunit tests is actually easy in terms of throwing money at it, as the suite of +15k tests can be diced and sliced to run segments in parallel (a bit of scripting + github action matrix). Billing time is around 40 minutes I'd say.

The frontend/TS stuff takes longer, usually 10-11 minute, where it's "truly building" and we can hardly parallelize this one. Or we lack the expertise to fix it probably.

At the moment though this is non-container environment; once we add building/deploying into the mix, I'd assume the time will go up a bit.

How many times have you worked on a project and, after 1-2 years of development, you have changed to a different database? I can tell you how many times it happened to me, I counted them: exactly never.

I guess this is the crux.

You hardly know in advance this is going to happen.

I experienced this transition (mysql>pgsql), though not with Django.

The ORM was perfect. If I recall correctly, only the customized SQL queries (only a few present) had to br adapted, everything else just worked.

11/10 would chose such a framework again.

Everything which is a list becomes harder to read / get an overview due to the new width (files, issues, PRs). Color changes don't exactly help either.

But for the diff view, it's great to see more of the line width.

Miss the last commit message and especially the status of it. Seems like a major loss of information to me.

I applaud this, I really do.

I'm also worried about HyperSwitch. When I converted to OSX half a decade ago, coming from Windows, this was the one app which I needed to be able to say "I can use this machine".

I can't comprehend how people otherwise can organize their things; it's just not my thing.

Also Tilde-Tab is the killer; Windows really missed out here.

I looking for a replacement for the same reason: the uncertainty of the future of this app.

The author seems to keep it alive, but it has (or had?) timebombs and once a year the author had to release a new patch version due to this.

I tried contact the author to throw money at it, to make sure I can have this as long as I use OSX, but he didn't really respond to that.

However, what can I say, I still prefer HyperSwitch (HS).

- HS is much faster "coming up", this is essential for me. I want to quickly use this. Basically I configured HS to be instant, no delay. I can live with the flashes, actually doesn't really bother me - HS supports CMD-Tab and Tilde-Tab; I need both. And in practice they work amazingly well for me. Tile-Tab is when I strictly need to jump between in-app windows. CMD-Tab can do both, in fact. Jump between apps or within the same apps window.

Also CTRL-Tab hurts to hit combined with TAB.

Anyway, this was on my mind for some time and there was nothing where I could mention it to similar thinking people so I already apologize if this seems inappropriate as a feedback for the hard work you put into it!

thanks

Same here.

Using GraphQL in the PHP backend und frontend TS. The schema is exported to TS and thus everything is fully types. A blessing.

Just the other day I learned about https://github.com/unite-cms/unite-cms/ .

Though obviously GraphCMS is different (technology-wise and business-wise it seems).

Hard to say. Now that we switched to PostgreSQL, operations run much more smoother. You literally can't compare it. But this is also attributed to the major refactoring we did under the hood too; as I pointed out in the article, I simply "driver switch" didn't cut it :-)

We hit the max connections limit in postgres too at peak times (but no lock-up or similar thing happened) and per advice of our hosting provider we added pgbouncer to the stack last week and will hopefully don't have much problems here anymore.

Disclaimer: I wrote that article.

We just recently switched from MySQL to PostgreSQL. God we're so glad:

- add a null column to a big table => have fun and plan your maintenance accordingly. Good luck estimating it. This is basically a no-op in Postgres

- partial indices; absolutely important for big tables. Think about ticketing systems where in general the number of "open tickets" is much lower than the closed ones

- being able to use advanced SQL features like window functions can easily be underestimated; you can't use potential you don't have

- renaming indexes, just like that. If you're into refactoring and lessen developer surprises, you want to have proper names and fix accidently mistakes or just legacy stuff

- in my experience (but I'm no DBA), the logging of PostgresSQL is much more useful then what I get with MySQL (e.g. deadlocks or slow logs)

- the EXPLAIN (ANALYZE, BUFFERS) output is ... what you get is beyond comparison to MySQL

The only technical downside we experienced with Postgres (but in comparison, these things are benign):

- limit of length of identifiers; even with aliases. This can trip ORMs which try to do some clever naming under the hood

- you cannot change the order of columns, you can only append at the end

To me, everything I do with Postgres feels much more predictable, whereas with MySQL it always felt a game of gamble.