runT1ME

Let's say your immutable DAG has four different 'transformations' (think either a map or a fold on the data). Beam or Spark will take a chunk of your data, partition it somehow (so you get parallelism), and do a transform on the data.

Now if you're doing a bunch of computations off of tens or hundreds machines you don't want to fail the whole thing because one hypervisor crashes or there's a JVM segfault or whatnot. So each step is usually saved to disk via checkpoint and then moved along to the next transformation in batches, so if the subsequent transfomation is where it dies, you have a safe spot to restart that particular computation from.

In addition, your (chunk of) data might need to be shipped to an additional machine to be transformed so you're not so much 'updating' data as you are creating new data in a space efficient way of the changes and shuffling those along the various steps.

It's awkward to work with fully nested structures. Think about having a map of customer objects which have a list of addresses and you need to update everyone's first address to be capitalized for some raeson. You'd really want a fully fleshed out lens library and maybe even macros for auto generating them on your concrete data structures to make it easy to 'update' (via a persistent structure) without having to deal with all the cruft of doing it by hand.

I really think this is where pure FP shines.

If you look at the architecture of something like Apache Beam, while you describe your computations in a language like Java or Python, you're really using a DSL for creating an immutable DAG that processes chunks of immutable data, persisiting each chunk (being loose with terms here) for durability, 'modifying' it in the immutable sense and then passing it on to the next edge and so on.

In a single standalone system, many argue that a purely immutable conceptual approach has pros and cons. In the Big Data world, I've never heard anyone argue for an imperative approach. Immutability is necessary, and as soon as you need immutability you want all the other goodies that facilitate it in a clean way like monads, lenses, and the like.

I don't have time (nor the endurance) for a 90 minute run, so could you combine the two? I often do a 14 hour fast and then go for a 20 minute run combined with some calisthenics on my lunchbreak before eating. How can you tell your body is in a ketogenic state?

I can definitely say the first few times I exerted a ton of energy after not eating for 12+ hours I felt like I had the flu for a few hours, but now I don't really get that.

Would love to get your take on if python is popular in large organizations because of the existing libraries or if it is the language asethetic itself?

To an outsider who knows a bit of python because of Airflow and Spark, it seems Python has become a popular metaprogramming language that various disparate ecosystems have all adapated.

Pyspark is python but kind of its own language and much of the processing is happening outside the python runtime. I think you could say the same for people doing Numpy or Pandas.

Tensorflow probably even more so where I believe Python is the most popular interface, but you're really just programming a program to run somewhere else. Again this is the same with Airflow conceptually, though I believe the runtime is python.

If my hypothesis is overall correct, and that a majority (or large share) of python programmers aren't sharing the same ecosystem, libraries and packages, then shifting to a different runtime or language that just encapsulates a subset of the language is much less challenging.

This is the opposite problem of the JVM, where no one really likes Java the langauge, so everyone tries to create enjoyable (and productive) languages for the jvm to keep using the libraries and the runtime optimizations.

This isn’t legal. A corporation that leases someone a car and a house is paying them a taxable income, regardless of it not being a cash income.

When I would commute from Orange County so Santa Clara the flight was often ~40 minutes. Combine with takeoff and landing it didn't make a lot of sense to pull out an addition device but it was easy enough to watch a short TV show on my phone.

Depends on the language (I don't use it for Java), but for Scala and python I definitely prefer it. I'd definitely recommend using TMux with Vim, I've mapped my tmux navigation to mirror Vim's window navigation, and I can keep a terminal and REPL open and easily copy-paste between Vim and my tmux windows.

We can write a general procedure to see it any given program halts within n-steps however.

Sure, that's possible, but the opposite is also possible: it might be wrong for most programs we actually write.

We could confirm this though! It's not like we can't find out if a given program halts or is inconsisent. Godel talks about it in his letter to Von Nuemann.

It might only be wrong for programs that have never existed and will ever exist.

Only one thread actually mutates anything, and its the same one over and over

Sounds great in theory and usually results in fantastic average case, but you get a hot partition and suddenly you can't work share and things go south. It's not a perfect solution.

I think it's important to point out that you can do calculus and arithmetic and only construct proofs that are provable. The possibility of deriving unprovable but true statements in an axiomatic system doesn't mean they're common or you can't prove all the satatements one has encountered thus far.

The day rates look high, but they have to pay self-employment taxes and health insurance

Most SWEs doing 'contract work' aren't actually 1099 contractors, what it means is they work for an intermediary and then 'contracted out' to a big company with a recognizable name (Google/BofA/Verizon/IBM). These jobs are W2 so you don't pay self-employment tax, they usually have mediocre insurance options, and you lose your job as soon as the work dries up.

It's simply a way for a company to staff up quickly without going through as strict of hiring committee and having any obligation to keep said staff around if plans change.

It's often easier for a department to get a req for a senior/staff contractor (and make salary exceptions) than a FTE, so what happens is departments will offer 200, 250 an hour for temp work but only able to hire engineers for 150k a year because their 'employee' req is slotted lower.

Both at my current startup job and at my previous Verizon job, a lot of my time is scaling out existing services and also boostrapping core infrastructure projects that prioritize reliability over adding features or addressing rapidly changing requirements.

Part of that is because I'v become very comfortable diving into very complex, established code bases because of my OSS experience, and enabled by the fact that I've had coworkers who didn't enjoy the deep dive/scaling part of coding compared to adding new features.

A lot of developers really enjoy the spotlight of having added the new killer feature that everyone recognizes, and sometimes that's even different skill set.

So my long winded answer is that there are two main ways to accomplish your goal:

1. Be able to refactor, address technical debt, scale out solutions, debug any problems, and find a team that complements your skillset, and get good at making sure your non visible contributions are known and appreciated. This often means having clear and easily available metrics that can show over time things like: less errors, more tests, faster build times, and so on. Really anything that you can point to that makes it easy to understand your value to those outside your team/org.

2. Become a team lead. A lot of times the tech debt/refactoring work isn't celebrated or recognized, but necessary. The job of a team lead is to both give guidance and enable your engineers to accomplish their tasks. Making code more maintanable, easy to read, and robust ensures your team will quickly and happily progress on their task, and you'll get to share in their successes even if you aren't the one contributing features. Unlike scenario 1, you don't necessarily even have to show as many metrics, as your value proposition is much simpler: Make your team productive and happy.

You don't need async/await to do monadic comprehension in Scala, it's built into the language from the very beginning with `for`.

This was inspired by do notation, which came about ~1998.

I think he means that the Fed can directly and indirectly increase the money supply, which disproportionately benefits people and corporations leveraged into assets. Hence, increasing the money supply makes the rich even richer.

I'm postulating that any modern CRUD app framework has Monads, Monoids, and other categorically inspired structure, even if they don't call it that or have a way to abstract over it.

I was on a team building good old crud apps using monads, monoids, categories

Here's the thing, if you look at a basic Spring crud app, you are also using Monads, Monoids, Categories, Traverse, etc. but you aren't expressing it in the type system. Seriously go look at modern Spring's flux stuff, it's all there minus the type classes.

I've seen teams that tried to over engineer Spring, teams that tried to over engineer Node applications, and yes there are teams that over engineer Functional style Scala.

All the teams I worked with (and managed) at Verizon leaned pretty heavily into FP style Scala without much over engineering and the experience was extremely pleasant. The only production issues in my three years there I remember were performance related, finding out how to get more throughput or lower latency out of FP Scala. I literally can't remember any 'bugs' that made it to production.

SBT in the last few years is nothing like SBT of five years ago. I agree it was a convoluted mess at one point, and now I much prefer it to Gradle.

I have the same issue with my PX, every two months or so I have to use the paper clip to hard reboot them. Kind of annoying to need to do for a $400 pair of headphones, but they sound great, look good, and are very comfortable.

Interesting comment, definitely want to hear more. I have concerns about Beam/Dataflow, but they seem different to yours.

The dataflow product seems to run older versions of Apache Beam just fine, so minor deprecations don’t seem like an issue in practice, but maybe I’m mistaken.

“No visible contributions to Apache BEAM itself”. I don’t think this is true, I’m a contributor and somewhat active on the developer mailing list, it seems the majority of the contributions these days come from google employees.

If the questionnaire you’re referring to was the paid Apache Beam survey, I participated and definitely didn’t get the impression that they were considering deprecating the service. It was much more focused on how they can improve docs, examples, and help developers use it.

Now, I think the project is too ambitious even for google. They don’t need to support Spark/Dataflow/Flink on three different languages (java/python/go) imo. I’m also frustrated with some of the bugs that slip through.

The fact that there is no back pressure support for a streaming framework is such a google thing to do: why worry about back pressure if you can just tell another team to increase their throughput for downstream sinks? /s

Dataflow does seem to be one of GCP’s most popular services (spotify and twitter are both users now) so I would guess it is here to stay in some form.

I've been approached about various data engineering jobs over the last couple years and the job descriptions have varied wildly. It has been everything from:

1. SQL/analytics wizard, capable of building out dashboards and quickly finding insights in structured data. Oracle/MSSQL/PostGres etc. Maybe even capable of FE development.

2. Pipeline expert, capable of building out data pipelines for transforming data, Flink, Spark, Beam on top of Kafka/Kinesis/Pubsub run from an orchestration engine like Airflow. Even this could span from using mostly pre-built tools wiring together things with a bit of python to move data from A to B, to the other exteme of full fledge Scala engineer writing complex applications that run on these pipelines.

3. Writing infrastructure software for big data pipelines, customizing Spark/Beam/Flink/Kafka and/or writing custom big data tools when out of the box solutions don't work or scale. Some overlap with 2, but really distinguished by it being a full fledged software engineer specializing in the big data ecosystem.

So, are all three of these appropriate to call Data Engineer? Is it mainly #1 and people are getting confused? I would certainly fall into the #3, so I'm always surprised when people approach me about 'SQL transform' type jobs.

I'm not even sure the core Beam engineers understand it all! Look at how Kafka offset acks are handled now:

https://github.com/apache/beam/blob/master/sdks/java/io/kafk...

It would be helpful if this thorough of an example could live on the apache beam website, I think it would avoid a lot of confusion. I certainly found the way it handles windowing with triggers to be quite diffrent than say, Spark.

I'm surprised that they haven't taken a page out of Netflix's book: sign small artists that sound close enough to what's popular to their own internal record label and push them in playlists?

I really recommend Functional Programming in Scala. It takes a first principles approach to Functional Programming and starts with the idea of "how do we write programs if we don't mutate state"? Lots of exercises, from small to longer, and ends up being very practical.

The problem is this incentivizes all the wrong things. This encourages the ultra rich to invest in land or large properties, driving up that particular asset class. Instead of venture capital, we’ll just get a bunch or people hoarding land

The beauty of inflation is that it hurts you in direct proportion to how well off you are,

This is the opposite of true. In practice, inflation is wealth transfer to the upper class. Inflation benefits debt holders, the people who are most levereged (with debt) into inflation proof assets are those who are rich.

Thank you so much for your response, that example makes a lot of sense. Algorithmically speaking in computer science, we can formalize efficiency with complexity theory.

Can we do the same with neural networks? Is there a formalization of why 'skip connections' (which I know nothing about) are better, why transformers are more efficient than recurrance, etc?

Is it useful to talk about their complexity or universal properties or size (and I realize this is muddled up a bit by the fact that hardware architecture can sometimes trump efficiency).