HN user

diek

443 karma
Posts3
Comments94
View on HN

FWIW, any accredited CS degree program in the US will have rigorous math and science requirements: https://www.abet.org/accreditation/accreditation-criteria/cr...

I don't think it was worded very well, but I think the parent comment was saying, "the bulk of CS can be covered in a masters program, so take an undergrad degree that has the same overlap in math/science, but a different focus". I'm not sure I agree, spreading the absorption of that knowledge over 4 years can be beneficial.

You're correct. If you have:

  public int someField;
 
  public void inc() {
    someField += 1;
  }
that still compiles down to:
  GETFIELD [someField]
  ICONST_1
  IADD
  PUTFIELD [somefield]
whether 'someField' is volatile or not. The volatile just affects the load/store semantics of the GETFIELD/PUTFIELD ops. For atomic increment you have to go through something like AtomicInteger that will internally use an Unsafe instance to ensure it emits a platform-specific atomic increment instruction.
Postgres as queue 2 years ago

For my use cases the aim is really to not deal with events, but deal with the rows in the tables themselves.

Say you have a `thing` table, and backend workers that know how to process a `thing` in status 'new', put it in status 'pending' while it's being worked on, and when it's done put it in status 'active'.

The only thing the backend needs to know is "thing id:7 is now in status:'new'", and it knows what to do from there.

The way I generally build the backends, the first thing they do is LISTEN to the relevant channels they care about, then they can query/build whatever understanding they need for the current state. If the connection drops for whatever reason, you have to start from scratch with the new connection (LISTEN, rebuild state, etc).

Postgres as queue 2 years ago

Admittedly I used SQL Server pretty heavily in the mid-to-late-2000s but haven't kept up with it in recent years so my dig may have been a little unfair.

Agree on READPAST being similar to SKIP_LOCKED, and filtered indexes are equivalent to partial indexes (I remember filtered indexes being in SQL Server 2008 when I used it).

Reading through the docs on Event Notifications they seem to be a little heavier and have different deliver semantics. Correct me if I'm wrong, but Event Notifications seem to be more similar to a consumable queue (where a consumer calling RECEIVE removes events in the queue), whereas LISTEN/NOTIFY is more pubsub, where every client LISTENing to a channel gets every NOTIFY message.

Postgres as queue 2 years ago

In a large application you may have dozens of tables that different backends may be operating on. Each worker pool polling on tables it may be interested on every couple seconds can add up, and it's really not necessary.

Another factor is polling frequency and processing latency. All things equal, the delay from when a new task lands in a table to the time a backend is working on it should be as small as possible. Single digit milliseconds, ideally.

A NOTIFY event is sent from the server-side as the transaction commits, and you can have a thread blocking waiting on that message to process it as soon as it arrives on the worker side.

So with NOTIFY you reduce polling load and also reduce latency. The only time you need to actually query for tasks is to take over any expired leases, and since there is a 'lease_expire' column you know when that's going to happen so you don't have to continually check in.

As far as documentation, I got a simple java LISTEN/NOTIFY implementation working initially (2013?-ish) just from the excellent postgres docs: https://www.postgresql.org/docs/current/sql-notify.html

Postgres as queue 2 years ago

Postgres is great as a queue, but this post doesn't really get into the features that differentiate it from just polling, say SQL Server for tasks.

For me, the best features are:

  * use LISTEN to be notified of rows that have changed that the backend needs to take action on (so you're not actively polling for new work)
  * use NOTIFY from a trigger so all you need to do is INSERT/UPDATE a table to send an event to listeners
  * you can select using SKIP LOCKED (as the article points out)
  * you can use partial indexes to efficiently select rows in a particular state
So when a backend worker wakes up, it can:
  * LISTEN for changes to the active working set it cares about
  * "select all things in status 'X'" (using a partial index predicate, so it's not churning through low cardinality 'active' statuses)
  * atomically update the status to 'processing' (using SKIP LOCKED to avoid contention/lock escalation)
  * do the work
  * update to a new status (which another worker may trigger on)
So you end up with a pretty decent state machine where each worker is responsible for transitioning units of work from status X to status Y, and it's getting that from the source of truth. You also usually want to have some sort of a per-task 'lease_expire' column so if a worker fails/goes away, other workers will pick up their task when they periodically scan for work.

This works for millions of units of work an hour with a moderately spec'd database server, and if the alternative is setting up SQS/SNS/ActiveMQ/etc and then _still_ having to track status in the database/manage a dead-letter-queue, etc -- it's not a hard choice at all.

EDIT: On top of that, there's usually a bit of 'related' work you need for a task, by example when you find an edge case related to your feature, and now you also needed to fix a bug, or you did a bit of refactoring on a related service, or needed to change the data on a badly formatted JSON file.

I agree that's related work, but I'd argue that work doesn't belong in that branch. If you find a bug in the process of implementing a feature, create a bugfix branch that is merged separately. If you need to refactor a service, that's also a separate branch/PR.

That's actually the most common pushback I get from people when I talk about squashing. They say "but then a bunch of unrelated changes will be lumped together in the same commit", to which I respond, "why are a bunch of unrelated changes in the same branch/PR?"

In general I agree with you, there are absolutely times where you want to retain commit history on a particular branch (although I try to keep the source tree from knowing about things like commit IDs).

I would argue that those are by far the minority of PRs that I see. As I mentioned in another comment, _most_ PRs that I see have a ton of intermediary commits that are only useful for that branch/PR/review process (fixing tests, whitespace, etc). Generally the advice I give teams is, "squash by default" and then figure out where the exceptions to that rule are. That's mainly because, in my opinion, the downsides of a noisy commit graph filled with "addressing review comments" (or whatever) commits are a much bigger/frequent issue than the benefits you talk about. It really depends on the team.

If the useful commits are the "baby" in your bathwater analogy, all the useful information in those commits is in the squashed commit.

This assumes a branch being merged in represents one logical change (a feature/bugfix/etc) that is "right sized" to be represented by one commit.

The golden rule is "do not rewrite history of a public branch". Rebase/squash your PR branches to your heart's content, but once it's merged that's it.

You get clean history by not merging branches with 50 intermediary "fiddling with X" commits in them.

destroying Commit information just to keep the graph tidy is a bad idea in my opinion

The commit information I see when telling teams to squash their branches on merge is not valuable.

* "fixing whitespace" * "incorporate review comments" * "fix broken test" * "fix other broken test"

(note, the broken tests were broken by the changes in the PR)

As soon as that PR is merged those commits are worthless. And there are branches with dozens of those "fixing X" commits that would otherwise pollute the commit graph.

I die a little bit each time I try to understand what changes were related to a line when tracking down a bug

A change/feature/bug is a branch, which is squashed into a commit on your main branch, right? So your main branch should be a linear history of changes, one change per commit.

How does that impact the ability to git blame?

Rather than waiting for the client to download a JavaScript bundle and render the page based on its contents, we render the page’s HTML on the server side and attach dynamic hooks on the client side once it’s been downloaded.

The fact that they don't make a reference like, "hey, ya know, how _everything_ worked just a few years ago" tells me they think this is somehow a novel idea they're just discovering.

They then go on to describe a convoluted rendering system with worker processes and IPC... I just don't know what to say. They could have built this in Java, .Net, Go, really any shared memory concurrency runtime and threading, and would not run into any of these issues.

The Iridium Museum 5 years ago

This is a fun blast from the past.

My dad was one of the "original six" engineers that worked on Iridium within Motorola, so my 90s childhood was filled with Iridium posters and satellite footprint tracking software.

I've mentioned it previously here when talking about SpaceX and Starlink. Iridium launched with satellite-to-satellite links. None of this "bent pipe" stuff where the satellite can only route between itself and a ground station in its same coverage area. As long as you could talk to an Iridium satellite, your data could be routed _in orbit_ to its destination along those inter-satellite links.

async/await in the general sense doesn't have anything to do with the preemption policy of a particular language implementation. For instance, C# implement async/await as keywords on top of its existing concurrency model.

That's really what it comes down to: cooperative multi-tasking vs preemptive multi-tasking.

Windows 3.1 had cooperative multi-tasking. Everything went great until one program didn't yield control, and then the whole system ground to a halt. That should sound familiar to any Node.js developers with their event loop.

Cooperative multi-tasking also extremely complicates code that actually uses the CPU. If you look inside libuv, it even has to jump through hoops with its encryption functions by effectively calling yield() in between rounds of calculation.

PHP 8.0.0 beta 2 6 years ago

Most web workloads will use some sort of process cache, like php-fpm, simply because spawning a process for each request is very inefficient. So the process and any extensions used can leak memory over time across requests.

CompletableFuture is the closest concept in Java, to my knowledge.

java.util.concurrent has a ton of great classes. If you need to do shared memory concurrency, it probably has something there to help you.

SDLC is often used to emphasize that you need to be familiar with the full "Software Development Lifecycle". When I read SDLC I just assume they mean, "you need to be able to take a project from concept to deployment (including documentation for supporting it long term)". Depending on org structures, a lot of engineers only work on one part of that pipeline throughout their career.

I recently bought a 2020 Bolt LT and I really enjoy it - $26k "out the door" price (after taxes, registration, etc) - 260 miles of range (66kWh battery by LG Chem) - DC Fast Charging - upright seating - lots of headroom/legroom in the front and backseat - dedicated LCD for the driver, separate from infotainment system

I compared it to a Tesla (test drove a Model 3), and a Tesla would be about $15k more. I have no use for self-driving. As a software engineer, I feel that full self-driving requires a general AI, which I think is on the order of decades away.

I really just wanted a simple electric car with minimal maintenance and no fossil fuel dependencies. Minimal complexity, no gimmicks (fart generator, Tesla? come on...)

With those requirements, the Bolt is awesome.

You may deploy your application package (however that is packaged up), but what happens when a hard drive starts to die? It may not _die_, it just may have elevated write latency. What about a RAID controller firmware having issues above a certain IOPS threshold? What about a critical kernel security patch that has to go out, and your application runs on 1,000 servers?

None of those things are related to your code directly, but may interact with it at some level. At some point, you get so far removed from the work on your actual application that it makes sense to move that to another 'Infrastructure' group.

Fun detail: node internally will use thread pools to do CPU-intensive tasks that would normally block the main thread.

For example: https://github.com/nodejs/node/blob/master/src/node_crypto.c...

I generally use that as an example when explaining to people why Node isn't a great fit for a lot of workloads. They have to use these features internally, but you as the user with a CPU-intensive job don't have access to those features.

It depends on what you mean when you say you want it to "support queries", but then also ask for a "nosqldb". It really depends on whether you want a full query grammar (in that case, why not SQL?), or if a key/value store is enough.

I use lmdb pretty extensively. It is single-file (well, it has a lock file) key/value store, can be read-from/written-to by multiple processes, and embeds nicely. It has Python bindings, but I've only used it from C.

https://symas.com/lmdb/

Why would they not include the inter-satellite linkage capability? That doesn't make any sense, given that was a solved problem in the Iridium constellation 20 years ago. That was a huge benefit of Iridium, in that it didn't have to rely on the "bent pipe" of just relaying data between a client and a ground station within the same coverage area. It was literally, "if you could talk to one satellite, anywhere in the world, you would have service".