HN user

__s

5,656 karma

https://serprex.github.io https://etg.dek.im

Posts18
Comments2,034
View on HN

Every year I just say I want to improve my work life balance & work out more

Doesn't happen, but hey at least I'm being honest about what I'd like to improve

I find it irritating with humans. "last but not the least" always distracts me as I then consider maybe the last item _is_ the least. & what is with everyone saying they want to "double click" into meeting items

I've experienced the opposite: wishing I were dead so life wouldn't go on with me falling into just another glum existence. That at the emotionally present moment I snapshot it, preventing inevitable dullness that time moves on

Unfortunately here I am, life moves on, now I'm just wasting away on HN

Half-Baked Product 19 days ago

Sai is https://news.ycombinator.com/user?id=saisrirampur

Maybe we were saved by being acquired before hiring sales. Sai knew the problem & understood customers. He'd sometimes oversell a bit, but managed it: kept pulse of capacity for new development, would ask about how hard requested features were, would feel out customer intent & guide customer adapt to what was already there

When we had our pepepizza moment, there was an understanding that it wasn't going to work, took learnings of what would be involved there, but kept focus on improving what we already had

For kafka connector we had a design partner, I got to work with them directly. They wanted 30 microsecond message processing, so didn't want json. Original ask was flatbuffers. I decided to put message formatting into a scripting layer using gopher-lua. Spent a weekend getting flatbuffers working with lua (it was buggy, opened half a dozen PRs to flatbuffers repo which got ignored). It was clearly awful having to manage flatbuffer schema files & update scripts every time schema changed. But I had alternative already made: msgpack. Throughput needed work but addressed that by creating pool of lua interpreters

Overall I overworked myself (put my hands out of commission & spent months relearning how to type on split ergo colemak-dh), but I enjoyed the work. Team was very open with each other & when performance is your selling point there's an understanding that engineering quality needs to be maintained. Sure there were parts of the system I hated, & sometimes I'd try chip away at those

Hopefully that helps, hard to say the difference, but I really feel in my work that when customer has problem I'm part of conversation. Most recently there was talk of customer wanting cold data offloaded from postgres which is what inspired https://github.com/ClickHouse/pg_clickhouse/pull/298 where we get Postgres to do most the work

Raised problems trying to mix C++ into postgres extension, decided fix was to write clickhouse-c library to replace clickhouse-cpp, there was some doubt on team about value, but demonstrated value (https://github.com/ClickHouse/pg_clickhouse/pull/254) & I appreciate my colleagues not being afraid to change their mind

There's a level of trust where instead of being assigning tasks on a board I instead work on what I think is important based on information available. Nobody was asking for wal-rus, but I know my fleet

ClickHouse Cloud similarly took route of taking its time hiring sales. Better to have a small sales team that can work directly with engineering on quality leads than overwhelming everyone so that sales becomes the enemy. Guess the difference is agency. When engineering is involved in making commitments they're invested in delivering & there's push back so sales doesn't start hallucinating features

If you're running pg yourself I recommend pgbackrest. It doesn't run as a daemon, & it forks multiple processes for concurrency. But it's simple to run as archive_command & is light on resources outside concurrency

wal-g/wal-rus have higher throughput

Correct. We tune overcommit so postgres reliably returns out of memory. It becomes complicated to accurately tune overcommit for every AWS instance type. We configure GOMEMLIMIT/cgroups but those are about RSS. Outliers come together: instances running queries out of memory on our service tend to also be pushing other resource limits, causing wal-g & prometheus exporters to start having more erratic memory usage at the worst time

This helps on both ends of the cost spectrum. Large 64 core instances are where our heuristics fall off the most as variance increases, & tiny instances with 8GB of memory can use every 100MB of RSS we can get

no. but wal-g & wal-rus both have parallelism over wal-e. however are you more asking about handling build up of wal / vacuum prevention caused by long running transactions? those are up to postgres, archive command only keeps pushing wal so that when postgres is ready to get rid of wal it can. seems like your scenario wouldn't care much what the archiver is since wal should be shipped long before postgres is ready to get rid of wal