HN user

evanrich

113 karma

c2d1c8

Posts0
Comments23
View on HN
No posts found.

For 1, no example really comes to mind, but i guess there could be cases where a service went from publishing an event with all of its related data, then split into a service where that becomes more expensive to do (like that data is no longer in memory its behind the api of the old service). In some cases you can have very simple services that consume a message, make a few calls to services or databases to hydrate it with more information, then produce that message to another topic that the original consumers could switch to. More commonly though if the data model is making a drastic change where the database is being split and owned by two new services, you will have to get consumers in on the change to make sure everyone knows the semantics of the new changes.

For 2, it completely depends on the source of the trigger. The first event in a chain probably only has enough information to know that it should produce an event, usually as quickly possible, so no additional db or api fetches. So you might get something in the driver status topic that contains {driver_uuid, new_status, old_status}, then based on what downstream consumers may want to do in response to that event, you may need more info, so you may get more entity information in derived topics. Even pure-entity-based messages would have needed a trigger, so in our topics that tail databases, you may have the full row as a message along with the action that occurred like {op: insert, msg: {entity data… }}.

Like others have said, it is just one tool in the tool box.

We used Kafka for event-driven micro services quite a bit at Uber. I lead the team that owned schemas on Kafka there for a while. We just did not accept breaking schema changes within the topic. Same as you would expect from any other public-facing API. We also didnt allow multiplexing the topic with multiple schemas. This wasn’t just because it made my life easier. A large portion of the topics we had went on to become analytical tables in Hive. Breaking changes would break those tables. If you absolutely have to break the schema, make a new topic with a new consumer and phase out the old. This puts a lot of onus on the producer, so we tried to make tools to help. We had a central schema registry with the topics the schemas paired to that showed producers who their consumers were, so if breaking changes absolutely had to happen, they knew who to talk to. In practice though, we never got much pushback on the no-breaking changes rule.

DLQ practices were decided by teams based on need, too many things there to consider to make blanket rules. When in your code did it fail? Is this consumer idempotent? Have we consumed a more recent event that would have over-written this event? Are you paying for some API that your auto-retry churning away in your DLQ is going to cost you a ton of money? Sometimes you may not even want a DLQ, you want a poison pill. That lets you assess what is happening immediately and not have to worry about replays at all.

I hope one of the books you are talking about is Designing Data Intensive Applications, because it is really fantastic. I joke that it is frustrating that so much of what I learned over years on the data team could be written so succinctly in a book.

I fully agree here. I am working through Deep Learning for Coders now. I have created several errors as I am implementing chapters, but the coding style of fastai makes in impenetrable to debug. It’s an incredible book and a great library when its working as expected, but the number of times i have run into some variant of `method takes N parameters but M were given` is pretty frustrating. Looking through the stack, these are not patterns that would have been accepted in a code review from me for the same reason you mentioned. Making small changes and debugging are both hampered by the style.

Why GPT-3 Matters 6 years ago

For me, it has kind of broken HN’s comment sections. I find myself jumping to the bottom of longer comments to look for “btw, this comment was written by gpt3”. To me it seems like we are going to be entering a perpetual April fools day where we never really know what’s real.

I am in the same boat as you. People who love it are crazy about it. I started it on a plane flight and the guy next to me chatted my ear off about how it changed his life for 20 mins. My current theory is that if you read it in your childhood, it resonates with you. I was 25 when I read it and it did not.

It seems to me that these articles are talking about different sorts of jobs. They are both quoting the same numbers, but choose to use different application for those numbers. I would think there is some truth in both sides. For Rowe's part, he is talking about jobs that can not be outsourced and are not optional to those who need the work finished. Pipe fitting, maintenance through welding, and vague post-Katrina engineering jobs are jobs and skills that have to be done on site, and therefore can not be moved overseas. They also pay well. These are jobs that make up part of that 3.5M jobs number, but I would guess a small portion of it. The NYT article talks about machining and factory work. In this field, work can be done here or abroad, so the cost of workers is a commodity. Unless tax breaks help these factories bring higher wages or the owners are altruistic, in my mind we can count these jobs as already gone. Someone could take that job for a bit, making $10-15 an hour but as soon as the commodity cost of wages exceeds the cost of added transportation abroad or the cost of transportation falls or any number of little pieces that fall into the outsourcing equation change, those jobs are gone just like the rest. The issue with these jobs are that they are still considered skilled when technology is quickly making them unskilled.

Eric Schmidt and Jared Cohen paint a pretty scary picture for drones in their book The New Digital Age. They predict a similarity between how terrorists and guerrilla warriors use cheap consumer technology now, such as cell phones in IEDs, and how they might use drones in the future as their prices continue to drop. "Imagine [a drone] that is fitted with a homemade bomb on its undercarriage, producing a whole new level of domestic terror that is just around the corner."