HN user
I recently shared a similar rationale for why a custom query language can help users even in the age of LLMs.
https://nchammas.com/writing/custom-query-language-design#wh...
Users can't "own" queries they don't understand. An LLM helping a user avoid annoying syntax errors is very useful; but helping a user write a query they ultimately cannot understand is fraught with the problems you describe.
I don't understand this concern. How exactly are you copy/pasting code such that significant indentation causes "real problems"?
I remember the creators of Go explained [1] that they chose explicit block delimiters because of problems they saw when embedding snippets of Python in other languages. But this seems like a very niche kind of problem.
[1] https://go.dev/talks/2012/splash.article#:~:text=we%20have%2...
There is an old project out of Berkeley called BOOM [1] that developed a language for distributed programming called Bloom [2].
I don't know enough about it to map it to the author's distributed programming paradigms, but the Bloom features page [3] is interesting:
disorderly programming: Traditional languages like Java and C are based on the von Neumann model, where a program counter steps through individual instructions in order. Distributed systems don’t work like that. Much of the pain in traditional distributed programming comes from this mismatch: programmers are expected to bridge from an ordered programming model into a disordered reality that executes their code. Bloom was designed to match–and exploit–the disorderly reality of distributed systems. Bloom programmers write programs made up of unordered collections of statements, and are given constructs to impose order when needed.
[1] https://boom.cs.berkeley.edu
The examples I'm referring to are in that page I linked to in my comment above.
Here's one of them:
# Polars
df.select(
pl.col("foo").sort().head(2),
pl.col("bar").sort(descending=True).head(2),
)
But in Polars (and maybe in Pandas too) you can do this easily, and I'm not sure why. There is something qualitatively different about the Polars DataFrame that makes this possible.
There is something I don't get about the Polars DataFrame API.
https://docs.pola.rs/user-guide/migration/spark/
Look at the examples on this page of the Spark vs. Polars DataFrame APIs. (Disclaimer: I contributed this documentation. [1])
Having used SQL and Spark DataFrames heavily, but not Polars (or Pandas, for that matter), my impression is that Spark's DataFrame is analogous to SQL tables, whereas Polars's DataFrame is something a bit different, perhaps something closer to a matrix.
I'm not sure how else to explain these kinds of operations you can perform in Polars that just seem really weird coming from relational databases. I assume they are useful for something, but I'm not sure what. Perhaps machine learning?
They _did_ serve the data. From TFA:
To demonstrate its scale, we operated the instance with 100M bots posting 3,500 times per second at 403 average fanout.
The linked post about their demo [1] has more details about the challenges involved, including rendering home timelines.
[1] https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the...
This is pretty neat, and it reminds me of my experiment solving the water jug problem from Die Hard 3 using Hypothesis [1] (a Python library for property-based testing).
Though I doubt Hypothesis's stateful testing capabilities can replicate all the capabilities of Datalog (or its elegance for this kind of logic problem), I think you could port the author's expression of the game rules to Hypothesis pretty straightforwardly.
[1] https://nchammas.com/writing/how-not-to-die-hard-with-hypoth...
Hypothesis is really a neat library. You can use it to implement stateful testing, which allows you to cover some of the more complex problems that might normally require something like TLA+.
I saw a solution to the water jug problem from the movie Die Hard 3 written in TLA+ and ported it to Hypothesis [0]. It was surprisingly straightforward and easy to read.
[0] https://nchammas.com/writing/how-not-to-die-hard-with-hypoth...
This idea was briefly addressed in PEP 635 [0]:
_Statement vs. Expression._ Some suggestions centered around the idea of making `match` an expression rather than a statement. However, this would fit poorly with Python's statement-oriented nature and lead to unusually long and complex expressions and the need to invent new syntactic constructs or break well established syntactic rules. An obvious consequence of match as an expression would be that case clauses could no longer have arbitrary blocks of code attached, but only a single expression. Overall, the strong limitations could in no way offset the slight simplification in some special use cases.
That talk by Martin Kleppmann is fantastic.
Another, closely related article is "The Log: What every software engineer should know about real-time data's unifying abstraction" by Jay Kreps.
https://engineering.linkedin.com/distributed-systems/log-wha...
Interesting project! (The interactive slides are cool btw.)
Could you share a bit about how engineers express data transformations in Flow?
From a quick look at the docs, it doesn't look like you are using SQL to do this, which is interesting since it bucks the general trend in data tooling.
Author here.
Are you thinking of a specific implementation of materialized views? Most implementations from traditional RDBMSs would indeed be too limiting to use as a general data pipeline building block.
The post doesn't argue that, though. It's more about using materialized views as a conceptual model for understanding data pipelines, and hinting at some recent developments that may finally make them more suitable for more widespread use.
From the conclusion:
The ideas presented in this post are not new. But materialized views never saw widespread adoption as a primary tool for building data pipelines, likely due to their limitations and ties to relational database technologies. Perhaps with this new wave of tools like dbt and Materialize we’ll see materialized views used more heavily as a primary building block in the typical data pipeline.
Do you know how sophisticated SQL Server is about updating indexed views? How granular are the locks when, for example, an indexed aggregate is updated? That will have a big impact on write performance when there are many concurrent updates.
A long time ago, I tried to use SQL Server indexed views to maintain a bank account balances table based on transaction history [0].
I forget what I ended up doing, but I remember that one of the downsides of using indexed views was that they didn't support any constraints. There are many restrictions on what you can and can't put in a SQL Server indexed view [1].
In this regard, I think Oracle has a more mature materialized view offering, though I personally haven't used Oracle much and don't know how well their materialized views work in practice.
[0] https://dba.stackexchange.com/q/5608/2660
[1] https://docs.microsoft.com/en-us/sql/relational-databases/vi...
Of the traditional RDBMSs, I believe Oracle has the most comprehensive support for materialized views, including for incremental refreshes [0].
As early as 2004, developers using Oracle were figuring out how to express complex constraints declaratively (i.e. without using application code or database triggers) by enforcing them on materialized views [1].
It's quite impressive, but this level of sophistication in what materialized views can do and how they are used does not seem to have spread far beyond Oracle.
[0] https://docs.oracle.com/database/121/DWHSG/refresh.htm#DWHSG...
[1] https://tonyandrews.blogspot.com/2004/10/enforcing-complex-c...
If you process data one row at a time, that is clearly a streaming pipeline, but most systems that call themselves streaming actually process data in small batches. From a user perspective, it’s an implementational detail, the only thing you care about is the latency target.
Author here. 100% agreed.
As an aside, I just came across your post about how Databricks is an RDBMS [0]. I recently wrote a similar article from a slightly more abstract perspective [1].
Having worked heavily with RDBMSs in the first part of my career, I feel like so many of the concepts and patterns I learned about there are being re-expressed today with modern, distributed data tooling. And that was part of my inspiration for this post about data pipelines.
providing analytics and a database type experience over a data lake.
It's interesting how the modern data lake is developing in this way, recreating many patterns from the traditional database for distributed systems and massive scale: SQL and query optimization, transactions and time travel, schema evolution and data constraints...
Having started out as a database developer / DBA many years ago, working with data lakes today reminds me in many ways of that early part of my career.
I wrote a post tracing a common interface from the typical relational database to the modern data lake.
* AWS has a managed Spark offering called EMR
There is also my rinky-dink open source project, Flintrock [0], that will launch open source Spark clusters on AWS for you.
It's probably not the right tool for production use (and you would be right to wonder why Flintrock exists when we have EMR [1]), but I know of several companies that have used Flintrock at one point or other in production at large scale (like, 400+ node clusters).
[0] https://github.com/nchammas/flintrock
[1] https://github.com/nchammas/flintrock#why-build-flintrock-wh...
From the what I understand, the FAGGA basically expands your maxilla forward. Here's a good example [1] that shows the appliance in action. The photos showing the upper jaw after FAGGA insertion match my experience.
The ALF can be applied to both the maxilla and mandible (whereas the FAGGA, I believe, is just for the maxilla), but the growth it promotes is more gradual and subtle.
I don't think either appliance shares much in common with lingual braces, functionally speaking. I'm not a doctor, but from what I understand lingual braces act primarily on the individual teeth whereas the ALF and FAGGA act primarily on the jaw bones.
[1] https://www.robinatowndental.com/jaw-expansion-without-surge...
I'm in my early 30s, and I recently had my maxilla and mandible non-surgically expanded with a combination of appliances, one called Advanced Lightwire Functional (ALF) and the other called Fixed Anterior Growth Guidance Appliance (FAGGA).
Treatments that make use of the Orthotropics philosophy or of appliances like the ALF or FAGGA are relatively new and are not mainstream. However, there are many case studies available online showing them in action and showing before/after pictures.
I don't know what kind of resources you're looking for, but this video [1] provides a quick introduction to Orthotropics.
You don't need surgery to move the jaw forward in order to treat obstructive sleep apnea.
Some dentists (like mine) offer a removable orthotic [0] that you wear during sleep which accomplishes the required jaw movement and improves sleep quality. It's an alternative to treatment with CPAP that doesn't require anything invasive like surgery.
I haven't undergone this particular treatment myself, but I know from wearing other styles of precisely designed orthotics (both removable and semi-permanent) that they can improve the quality of your breathing, and thus improve the quality of your sleep.
My town recently replaced the dull, orange-ish street lights in my neighborhood with very bright, white LEDs.
The street is now very well-lit, but I share the author's concerns and experience with this kind of light creating an environment that's not conducive to sleep, or to unwinding in general. When I look at the new street lights I feel like they were taken out of a hospital operating room.
Another thing I noticed when the new lights first went in was that the birds started chirping at all hours of the night. I don't know if they were always doing that and I only noticed after the new street lights were installed, but it makes sense that the light would have a similar impact on animals as it does on humans.
I wonder if anyone else noticed this when the lights in their neighborhood were upgraded to these harsh, white LEDs.
This post reminds me of a similar one by the CEO of Basecamp giving his take on the problems with group chat and their solution to those problems.
https://m.signalvnoise.com/is-group-chat-making-you-sweat-74...
Group chat is like being in an all-day meeting with random participants and no agenda.
Related: A YouTube channel [0] dedicated to videos of "everyday life" in North Korea, from an Indonesian expat living there.
Couple of random examples:
* Walking around Pyongyang in the winter. [1]
* Going to a computer store in Pyongyang. [2]
[0] https://www.youtube.com/channel/UCzvCf_q10UZkUJE0lOav0ag
These ratings are not great because they incentivize doctors not to take more complex cases.
This reminds me of a scene in the recent Dr. Strange movie where the main character -- a neurosurgeon -- turns down a case because it had a high risk of failure and he didn't want to tarnish his perfect record.
They claim to be risk-adjusted, but in reality you cannot adjust for all possible risk factors. Lots of things in medicine are very unique. There's very little that's more complex than human disease.
Isn't it better to approach the problem by improving how we do these risk assessments so that the risk-adjusted ratings are fair, as opposed to leaving patients in the dark about where to get the best care?
I was amazed to stumble across a few case studies on YouTube of a doctor treating Tourette's by adjusting the resting position of the patient's jaw.
https://www.youtube.com/watch?v=nLkPBUBmxDo
I assume there are different types of Tourette's with different causes and different treatments, but it was really amazing to see this link to the jaw.
The section titled "So Why Does Abstract and Concrete Matter?" on Donald's post explains pretty clearly why you can't have these two types of dependencies in the same file.
There is another discussion on the Pipfile repo about this that may also clarify things for you [0]. The example I posted there [1], which I'll post again here, is:
---
A project can only have 1 set of abstract dependencies (setup.py/pyproject.toml), but different users working with that project can have different sets of concrete dependencies (requirements.txt/Pipfile) which allow them to fulfill those abstract dependencies from PyPI mirrors, private package indexes, personal forks on GitHub, or somewhere else.
So if project Car depends on Engine (an abstract dependency), I can choose to install Car but grab Engine specifically from a fork I made on GitHub (a concrete dependency) that has some performance improvements. Meanwhile, someone else working at a big company that doesn't want to depend on external services to build and deploy their internal Python projects can choose to install both Car and Engine from their private package index as opposed to PyPI (another concrete dependency).
You can't merge these two types of dependencies together into one file without hampering people's ability to choose where to get their dependencies from.
---
According to another commenter on that issue [2], both Rust and Ruby have a similar split in how they specify dependencies.
[0] https://github.com/pypa/pipfile/issues/27
[1] https://github.com/pypa/pipfile/issues/27#issuecomment-26228...
[2] https://github.com/pypa/pipfile/issues/27#issuecomment-26226...
And since this Pipfile is just a bunch of function calls
That hasn't been settled yet. It's being actively discussed [0].
Or is the idea to eventually get rid of setup.py?
Yes, a replacement for setup.py has already been agreed on in PEP 518 [1]. It's called pyproject.toml.
As of Spark 2.0 spark-ec2 doesn't ship with Spark anymore [0]. It's been moved here [1].
[0] http://spark.apache.org/releases/spark-release-2-0-0.html#re...
This is exactly why large stores of data should be considered a liability and NOT an asset.
Your comment reminded me of a post by Bruce Schneier with a similar theme, called "Data Is a Toxic Asset" [0].
[0] https://www.schneier.com/blog/archives/2016/03/data_is_a_tox...