pjd7

Engineering hasn't gone away, you're now just directing things at a higher level. You are now a architect & manager (but you're managing agents not people).

Who sometimes has to deep dive & mentor a agent on solving the right problem.

Tell us why you think so at least.

I'm not sure I would want this even if I could have it TBH. Engingeering org size is about ~200 with infra/sre/ops around ~25.

Different teams want to move at difference cadences. At a certain scale splitting up things feels a little more natural (maybe I am stockholmed by prior limitations with TF though or just used to this way of operating now).

But even then, we're moving to k8s operators to orchestrate a bunch of things and moving off terraform apart from the stuff that doesn't change much (which will eventually get retired as well). Something like https://www.youtube.com/watch?v=q_-wnp9wRX0

Terraform variable management is our larger problem (now/nearterm) when we have to deploy numerous cells of infra that use the same project/TF files with different variables. Given the number of projects/layers of TF getting cell specific variables injected is meh.

Those variables are instance size, volume size, addresses, IAM policy, keys etc.

This is in the b2b saas world with over a million MAU. We've got islands of infra for data soverignty, some global cells where each cell can communicate back / host some shared services (internal data analytics, orchestration tooling, internal management tooling and the like).

This is awesome, I will be trying this out in the coming months. Its just made it to the top of my R&D shortlist for things that could massively simplify our data stack for a b2b saas.

https://www.youtube.com/watch?v=oYiFTBO67uU

https://innovation.ebayinc.com/stories/graphload-a-framework...

It looks like people are using it to build graph related models on it.

I am looking at it & considering doing something similar for graph data sets. As well as a transactionally safe key value store to store roaring bitmaps.

Caught a steam train that operates on the mainline in NSW, Australia last year.

https://www.picnictrain.com.au/

We had to stop & wait at various points on our return day trip for other passenger services to pass us. Had to wait for some freight services to clear a block at a few places too.

There was an auxiliary diesel unit not actively used apart from for power AFAIK (and additional emergency mobility should things go kaboom/excess wheel slip).

Maybe that has some the extra safety systems on it though.

I started looking into podman quadlets on the weekend. All because docker & nft (nftables firewall) don't play nice together.

I have my finely crafted nftables ruleset that I want to keep. Docker is going to abuse the crap out of my ruleset/make me jump through extra hoops to have it work.

Having the containers operate under systemd looks fine (ignoring the fact that it is systemd, I guess I have finally accepted systemd is here to stay).

I was interested in using traefik & docker labelling to have ingress sorted out and I believe this will all play nicely enough to replace docker-compose on a single server.

Unifying things slows engineers down, so probably not (for some time).

https://youtu.be/qcNNB2qZ5kE?t=2776 around this time code. If the USA doesn't want a nation to fight then there are more issues than a F35 software kill switch etc.

This looks super interesting.

Just starting to review it but my front of mind questions: 1) How do I handle persistence? Looks like some code is missing. 2) Do you support multi-tenancy (b2b saas graph backend for handling relations scoped to a tenant)

Thanks

Interesting in hearing some thoughts about using roaring bitmaps stored in a bytea postgres column to represent adjacency matrixes.

I was thinking that given RDS has support for plrust and PostgreSQL's SPI I could use the fact they support croaring-rs there as a crate and build upon that.

I figure I can use that to represent many graph's with say 100s to ~100m nodes and many relations between these things. But each graph would be tenanted to a tenant (company/b2b saas use case).

I was thinking that by using plrust storing the roaring bitmap on the DB server in a bytea and using SPI, I can benefit from the minimal network overhead to mutate and query against the bitmap with croaring. Using SPI locally in the DB server I eliminate network overhead shipping that back to my application code.

PostgreSQL also gives me transaction safety to updates etc. And a bunch of support for other column base data such as my tenant ID column, some JSONB for relationship metadata to query on etc.

Basically something like https://jazco.dev/2024/04/20/roaring-bitmaps/ but on postgres. Given I need to support many tenanted graphs & we're already using citus this seems like something that is feasible at a larger scale too.

I was wondering though if I am going to need to create some operator classes to allow me to index relations a bit better (probably seems likely I think).

I am aware of https://github.com/ChenHuajun/pg_roaringbitmap but would prefer to use int64s and maybe start out on RDS instead of having to add another workload to our self hosted citus cluster/s.

Happy to be told I am fool and any insights would be nice. I am potentially going to try this out on some of our data sets we have because our product team is basically laying out a vision where they want us to have a graph powering a bunch of things.

I don't like the idea of neo4j when we're already deep into PostgreSQL for a bunch of workloads (~20+ TB table workloads etc so we have some reasonable inhouse PG experience).

Also huge thanks to the author of the blog post. I had been looking at pgRouting and wondering with a tilted head.. hmm seems like we can just use this as a graph DB. So that is also on my list to test out.

Where does one find out about all of these conventions for rails?

250 Pitt St Sydney around 2002-2003.

They work with sulphuric acid, oxy/acetylene/propane torches. Some processes give off cyanide fumes. Ultrasonic baths with ammonia based solutions to clean polishing compounds off etc.

You need outside air ventilation.

There is lots of mechanical suction for things like polishing to capture the waste material for post processing.

Most will wear a leather apron for heat / burn protection and capturing fine dust/dirt from polishing compounds. I suppose you could destroy that eventually in a giant smelter.

I did my jewellery trade in Australia (hence the correct spelling for me). We used to keep all our emery paper, old polishing wheels etc and send them off ever few years to be burnt & refined.

When the building we were in got renovated some enterprising guys in another workshop ripped up their floor boards and their neighbouring empty suites and got all the precious metals out of the gaps between the floorboards.

The building was 11 stories and was predominantly filled with small jewellery workshops with 2-5 people per business. And a lot of adjacent businesses (trade supplies, stone merchants etc).

I'm not a frontend engineer or really a fully experienced backend engineer. I have done systems stuff, lots of OPS, scaling and other things adjacent.

Flutter is interesting to me because I can use a tool like flutterflow to build an app and throw together the backend reasonably comfortably (with some possible future footguns included).

Not knowing Kotlin means this is less attractive to me, but I can see the benefit if I knew one language that could span frontend and backend very nicely.

Laughs at a PostgreSQL schema change backfill jobs that need a few days to complete on i4i.16xlarge.

Or that time I spent $120k USD on testing some upgrade paths on a dataset in elasticsearch from version 2.4 -> 5 -> 6 -> 7 that took about 5-6 days runtime on imports per upgrade step.

Really depends on some factors:

Do you have billions of rows with only a very small % in use/locked or millions of rows that are jsonb blobs with a relatively high % in use/locked?

Is your workload mostly write once read lots or read/write evenly split etc.

How fast is your IO. Are you using network storage (EBS for example) vs local NVMe?

How much other load are you contending with on the system.

I have a JSONB heavy workload with lots of updates. JSONB blobs avg around 100KB. But can go up to 20MB. We can see < 10 updates on a blob all the way to thousands of updates on the larger ones.

We use Citus for this workload and can move shards around and that operation will use logical replication to another host effectively cleaning up the bloat that way.

We also have some wide multi-column indexes over text fields and date fields that see a fair bit of rewrite activity as well. Those indexes get bloated a fair bit too and we run re-indexes every few months (used to be every 6-12 months, but as the workload is getting busier we're re-indexing a bit more frequently now).

The index bloat is far more of a problem than the raw table bloat.

In the past I would use pg_repack when we were on a single RDS solution.

I have a machine, took a few weeks to dial in the settings right for me. Tried a few different masks & machines from a sleep study clinic. In a few years I have gone less than 7 nights without using it.

Using the machine has been life changing for me.

I used to struggle to get up before 10am. Now I wake up naturally around 6:30am-7am most mornings.

Learning new things isn't a mountain to climb to retain just a bit of information. I feel like if I had done a sleep study 10 years earlier I would be a much better technologist than I am now.

I would recommend you suggest to your wife that she tries a few different mask styles. I initially found I wasn't fitting the mask well or had a slightly wrong size and I would get some leaking that would keep me awake at night.

Taxi's in NSW/Sydney are much better these days. Uber X are typically shit by comparison.

A $95 UberX ride home is about $109 in a taxi for me.

Talking to a few friends who also live a similar distant from the CBD/downtown find that Taxi's are the better / more reliable option these days too.

The taxi is pretty much always newer, with a professional driver who usually isn't almost falling asleep behind the wheel. The quality of the driving is typically significantly higher. It feels like a safer ride.

Of the recent UberX's I have caught, the car was typically in need of a suspension replacement or significant servicing and around 6-7 years old or older.

Most recent taxi rides I have had a car around 12 months old or less.

Taxi's are better regulated and provide a better service. I'll happily pay the little premium there is in this market to use one over an Uber.

On a recent trip to NZ the uber to taxi price comparison was massively different.. Taxi was like 2.5x more expensive..

Yup, been following this build as well. There are plenty of other decent layouts that can be built in a shed, half a room etc.

A different aspect is the shunting layout, this guy's channel is amazing. https://www.youtube.com/@boomerdioramas

I'm 41, just recently got back into the hobby. Have rejoined the local club I was a member of as a teen.

Membership average age is like 50-60ish. We do have about 10-15% of members like me under 50. Some even in their 20s. Doesn't seem quite like its in too much trouble around here. The pool is shirking with apartment living becoming more normal though in the area. But maybe because this is a club with its own layouts that members can use is one way for the hobby to survive & thrive.

There are 5 layouts, 2 in N scale. 3 in HO. 3 of the layouts are able to be broken down and put on a trailer and taken to an exhibition.

I do think that home built & run layouts are probably on the decline with cost of living & housing so high compared to years past. But I think club layouts that members can use is a way to help keep the hobby relevant.

There are cool projects like https://dcc-ex.com/index.html and arduino based control for layout automation bits. That historically cost a lot.

My observations of the US is that mostly ready to run models are sold now, haven't seen too many kit makers around for US stuff. Still kit makers for Australian & NZ stuff. With some decent ready to run Australian prototype available now.

The next house will have a train room or area of the garage available. This is definetly a middle class hobby. Its not cheap.

I have wanted to work with my hands again. Sitting at a computer all day or parts of the evening playing video games is only so fun. Would rather build something physical.

Can't wait for Elon to declare himself sovereign of Mars.

Overall the changes seem sensible. For those wanting to self host there are plenty of guides out there on how to configure various MTAs with all of the required bits.

BUT, Why does IP reputation matter so much these days when you have DKIM, MTA-STS, DANE and other mechanisms that provide verification of the sender?

Say I want to startup a Email Service Provider, I need to go and source a bunch of IPv4 typically to have a premium upsell for end users to really ensure cross sender reputation does not impact other tenants. Crazy.

IPv6 historically at least was anecdotally punished by the likes of GMAIL, Yahoo, Hotmail, Office365 etc. Does anyone know if IPv6 hosted email severs still suffer additional spam scoring?

https://www.freightwaves.com/news/flexport-to-launch-eastern...

The leasing or contracts on the air freight must be a bit of a problem if shopify has trimmed their business to flexport. Probably needs to free up cashflow to cover the expenses.

Wonder how the % of business shopify was for flexport.

I guess any "easy" way to solve this right now is to put some authentication on the celery communication bus? I wonder if flower is being used.

Once auth is required for the celery processes, the old zombie wont have access. And if this occurs again OP could revoke access etc.

I would agree that in some organization sizes it is fading out & becoming hidden by ever increasing raw compute/disk performance which inevitably hides many bad queries.

However for some orgs this becomes a brick wall they eventually smash into.

The cloud vendors now provide moderately ok database offerings on a variety of virtual hardware that allows you hide a lot sub optimal data modelling, index usage and inefficient data modelling.

Any database workload is performant while all the data fits in ram.

I agree with some other comments that there is a bit of a shift left approach to the DBA role now. I find that dedicated DBA skill sets are somewhat useful up until a point then they struggle to have context of the application stack beyond.

If you want a solid DBA career you may want to look at a SRE track/pathway that includes some deep focus on databases.

You will want to know the how various index types work, when and why to use them. Data types. Intricacies of various database mechanics that impact performance and scaling workloads. Solid foundations in highly available system design, maintenance and operation. A willingness to be on-call. A willingness to write code to a software engineer level or send in changes to the dev team/s to help optimise queries, drivers, connection pool settings/timeouts or ORM usage (shudder).

I think the days of a pure DBA role are coming to a close in technology lead organisations. Maybe not in banking/insurance or other orgs where DBAs found a home to sit in the back office doing minute query tuning on a reporting query etc.

If more developers have read and understood the content in https://use-the-index-luke.com/ then there would be less need for DBAs.

I have been in a similar situation. First step in my opinion is to go on a holiday, at least 1 week long (ideally 2 weeks or slightly longer). And get away from a computer. Head somewhere where you can relax. Working on a side project is not a holiday, its still working. Get outdoors, exercise, use tips you can find online for beating depression like characteristics.

This is to give you some perspective and time to decompress from the day to day pressures at work & regular home life.

Your burnout is going to be clouding your view and judgement of the situation at work. The time off will help you look at the situation with some fresh eyes hopefully.

If the idea is worth millions, what sort of stock options do you have? How vested you are is a consideration. Also can you afford to exercise those options? Would you be financially better offer continuing to work and vesting options to exercise later (assuming you're in the USA under their style of share plans). If you don't have stock options (or something equivalent) well then jumping ship to another job is probably the best course of action (maybe).

What are your motivations for working in this job? Is there the potential of a large pay day if the company has successful liquidity event under favorable terms? Get to build X and have your name on it? Understanding those will help you work out what you want to do and why you want to do it.

You need to talk to your boss as well. Why is the feedback bad? What are actionable things you can do to improve the feedback? Are you not working well with others now that the team is growing?

Talking to the founders about some struggles you're having may help, but how good is your relationship with them? How understanding of the work you have done are they? If they're not across the detail of what you have delivered this may backfire depending on their personality types etc.

To the OP, You need to check out this http://www.workthesystem.com/

Most people at this stage are probably happy to push along ipv6 more so.