drej

There's one for the Czech network (one of the densest in the world, if not the densest) https://grapp.spravazeleznic.cz

Please do run this on your own workloads! It's fairly easy to set up and run. I tried it a few weeks ago against a large test suite and saw huge perf benefits, but also found a memory allocation regression. In order for this v2 to be a polished release in 1.26, it needs a bit more testing.

Read my posts again, I'm not complaining it's not for the masses, I know it isn't. I'm complaining that it's being forced upon people when there are simpler alternatives that help people focus on business problems rather than setting up virtual environments.

So I'm very much advocating for people to "[u]se whatever tools work best".

(That is - now I'm doing this. In the past I taught a course on pandas data analytics and spoke at a few PyData conferences and meetups, partly about dataframes and how useful they are. So I'm very much guilty of what all of the above.)

I find it much more beneficial to lower the barrier for entry (oftentimes without any sacrifices) instead of spending time and money on upskilling everyone, just because I like engineering.

Fun aside - I actually used polars for a bit - first time I tried it, I actually thought it was broken, because it finished processing so quickly I thought it silently exited or something.

So I'm definitely a fan, IF you need the DataFrame API. My point was that most people don't need it and it's oftentimes standing in the way. That's all.

I was talking about data engineering, because that was my job and all analysts were downstream of me. And I could see them struggle with handling infrastructure and way too many toggles that our platform provided them (Databricks at the time).

Yes, I did write tests and no, I did not write 1000-line SQL (or any SQL for that matter). But I could see analysts struggle and I could see other people in other orgs just firing off simple SQL queries that did the same as non-portable Python mess that we had to keep alive. (Not to mention the far superior performance of database queries.)

But I knew how this all came to be - a manager wanted to pad their resume with some big data acronyms and as a result, we spent way too much time and money migrating to an architecture, that made everyone worse off.

As a user? No, I don't have to choose. What I'm saying is that analysts (who this Polars Cloud targets, just like Coiled or Databricks) shouldn't worry about instance types, shuffling performance, join strategies, JVM versions, cross-AZ pricing etc. In most cases, they should just get a connection string and/or a web UI to run their queries, everything abstracted from them.

Sure, Python code is more testable and composable (and I do love that). Have I seen _any_ analysts write tests or compose their queries? I'm not saying these people don't exist, but I have yet to bump into any.

Having done a bit of data engineering in my day, I'm growing more and more allergic to the DataFrame API (which I used 24/7 for years). From what I've seen over the past ~10 years, 90+% of use cases would be better served by SQL, both from the development perspective as well as debugging, onboarding, sharing, migrating etc.

Give an analyst AWS Athena, DuckDB, Snowflake, whatever, and they won't have to worry about looking up what m6.xlarge is and how it's different from c6g.large.

For those not familiar with this kind of computing challenges, I must link this wonderful video about TypeScript types running... DOOM. https://www.youtube.com/watch?v=0mCsluv5FXA

Whoa, I've been battling with an issue where my new M4 Mac Mini, disconnected from all peripherals, just sips power and gets rather warm overnight. Cools itself when I wake it up. I wonder if the changes suggested in the post and the comments here will help me resolve this issue. Thanks for upvoting this.

I remember just adding one letter to the instance type in EMR to switch from Intel to ARM... and saw a 20% speedup as well as additional 20% hourly cost saving, all in all 1/3 cost decrease from a single character.

Some migrations are easier than others.

Note that this already exists on top of SQLite proper - authored by Ben Johnson (Litestream, Fly.io etc.) - https://github.com/benbjohnson/postlite

I encourage anyone to study these protocols, they are sometimes a lot more prevalent than one might think. Take the Postgres wire protocol, it's supported by a number of databases out there.

Some years I built a data grepping tool, but didn't want to build a UI around it, so I started looking into the Postgres protocol, so that I could plug in existing tools. Getting the basics right was not that difficult, the docs are pretty clear. https://www.postgresql.org/docs/current/protocol.html

For me the main deterrent from Gitea has been their approach to UI design. The shameless inspiration (if you can call it that) they took from Github is pretty startling, the two interfaces look pretty much identical (well, Github has recently redesigned its repo page, but many can remember the old design).

Sure, get inspired, make something similar here or there, but this is an outright copy of the whole product design.

Anyone can share their experience with the somewhat new STRICT mode? Does it help? I tend to use Postgres when available, primarily for the added strictness, but I'd surely prefer SQLite in more scenarios as it's easier to operate.

It's wonderful tool and has simplified workflows for many, just be mindful of its one giant security implication. Should your database credentials ever leak in any way (lost/stolen property, incorrect git commit, screencasting mishap, ...), having an Adminer/phpMyAdmin instance running on an otherwise unrestricted address, you're opening up (quite literally) your server to a world of trouble.

(I speak from experience. I've seen a lot of credentials leak together with source code. This meant I saw "secret" paths where Adminer was hosted on a given site. Having database credentials meant I didn't have to somehow circumvent network security to get to the database itself. [Yes, I disclosed all of this to the server operator and even got a measly $50 gift card out of it :-)])

We used to run it self hosted, but the architecture is rather complex, so we ended up switching to their SaaS offering. I tried advocating [0] for a leaner architecture for simpler setups + to run integration tests against... but it wasn't met with much enthusiasm.

I imagine 99% of installations (and certainly CI pipelines) would be fine with their wsgi webserver and sqlite instead of Clickhouse, Relay, memcached, nginx, Postgres, Kafka and a host of other services. I wanted to take a stab at this myself, but given the complexity of the system, uncertainty of being able to merge it back, and, last but not least, their license, I decided against it.

[0] https://github.com/getsentry/sentry/issues/32794

There are several principles that have helped me:

  - Do NOT migrate to new tools, even if they are really shiny and popular - this may end up being a big hurdle you'll regret down the line (looking at you, Notion, Todoist, Google Keep, the list goes on). Just stick to your tools for at least a couple of years before exploring alternatives.
  - Try search instead of organisation. I used to have elaborate folder structures for everything... but have since given up, still keep some basic structure, but I find most things via search anyway.
  - Help your non-technical friends and family. You may have discovered the best clouds, NAS solutions, software, methodology... but they still use post its for passwords and keep family recipes in that one notebook in the cupboard. Guide them, share your subscriptions with them, give them pointers.
  - Use fewer services, even if they are not the best tools for the job. I find it much easier to reason about a handful of storage media (and their backups, pricing etc.) than having an app for every single activity in my life (e.g. I stopped using apps for recipes, I use my synchronised note taking app instead. Same for shopping lists, use my todo app. Etc.)

Clicking on the link I though "I wonder it it starts with time.sleep"

It was a pretty safe guess, though, the number of bugs related to this is pretty epic.

Perhaps I'm mistaken, but from an interview with the game author, I gathered that you can use 10k+ words, only ~2300 of those being candidates for the winning word.

But the analysis only uses the latter subset. So in theory there could be a valid 5-letter that could be a good opener, even if it cannot be a winning word.

What I find hilarious is that companies argue who can query 100 TB faster and try to sell this to people. I've been on the receiving end of offers by both of the companies in question and used both platforms (and sadly migrated some data jobs to them).

While they can crunch large datasets, they are laughably slow for the datasets most people have. So while I did propose we use these solutions for our big-ish data projects, management kept pushing for us to migrate our tiny datasets (tens of gigabytes or smaller) and the perf expectedly tanked compared to our other solutions (Postgres, Redshift, pandas etc.), never mind the immense costs to migrate everything and train everyone up.

Yes, these are very good products. But PLEASE, for the love of god, don't migrate to them unless you know you need them (and by 'need' I don't mean pimping your resume).

I remember benchmarking fast compression algorithms a while back and had lz4 as one of the top contenders, but I remember going with Snappy over lz4, because I ran into some compatibility issues in lz4 land... I can't recall any specifics, but I think it was some Hadoop libraries not being able to read files lz4-compressed by some other versions of the library.

Has anyone run into the same issue? I'm considering reopening this investigation (even though I'm very happy with Snappy).

That being said, I now remember I stopped contributing for a slightly different reason. While I tried to fill WD with complete information about a given subject, this was never leverage by a Wikimedia project - there is certain resistance to generating Wikipedia articles/infoboxes from Wikidata, so you're fighting on two fronts, you always have to edit things in two places and it's just a waste of everyone's time.

Unless the attitude becomes "all facts in infoboxes and most tables come from WD", the two "datasets" will continue diverging. That is obviously more easily said than done, because relying on WD makes Wikipedia contribution a lot more difficult... and that pretty much defeats its purpose.

It’s fairly difficult from the other side as well - contributing. I’ve been trying to complete wikidata from a few open source datasets I am intensly familiar with… and it’s been rather painful. WD is the sole place I have ever interacted with that uses RDF, so I always forget the little syntax I learned last time around. I have some pre-existing queries versioned, because I’ll never be able to write them again. I even went to a local Wikimedia training to get acquainted with some necessary tooling, but I’m still super unproductive compared to e.g. SQL.

It’s sad, really, I’d love to contribute more, but the whole data model is so clunky to work with.

Nice. Reminds me of an optimisation trick from a while ago: I remember being bottlenecked by one of these trigonometric functions years ago when working with a probabilistic data structure... then I figured the input domain was pretty small (a couple dozen values), so I precomputed those and used an array lookup instead. A huge win in terms of perf, obviously only applicable in these extreme cases.

Hey Mike, thanks for all the work you've been doing - I first used D3 back in 2013/2014 and been using them for side projects since.

I haven't dug into the whole v3/v4 modularisation, so that might answer it, but is there a way to minimise the dependencies that Plot brings? You say it needs D3, but what of it does it need specifically, the whole thing? It's just that it's 250K or so, so I was wondering what the minimal setup here is.

Cheers and congrats on the release!

I remember using this editor in the very early 2000s, it was quite something back then. You could easily live edit files off of FTP. With PHP being all the rage, we were changing things pretty live :-)

Good times.

I have seen several deployments of Databricks (including first hand experience) and... most use cases could be better served by Postgres (or Redshift, Athena, Snowflake for larger scale). It has honestly been such an overkill for so many workloads, it was quite astonishing. I've seen people move from Excel to Spark... to handle the same volume of data. That's obviously not Databricks' fault, but their PR is pretty much "please do all your data work in our product, it's well suited for it".

Yes, it's very good if you don't like setting up clusters (few do/can) and the UI is rather useful for getting up and running (not so much for writing code though). But you need to really understand the platform before adopting it. Please, please don't just adopt it because it's popular.

Note: We cannot accept non-trivial PRs for legal reasons. We need to retain copyright over the entire codebase.

Does a CLA solve this? (Genuine question, I don't know.)

I used to have a boss who took this logic to an extreme and basically preached “always buy, never build”

Problems mounted - cost, security, integration issues, performance, ... - we couldn’t do anything about any of that, because we just wrote glue code to combine all the SaaS stuff.

I think about that job from time to time and I have two theories as to why this guy was like this - a) lack of trust/knowledge his engineers can build stuff, b) bragging about using a shiny-new-tech is better than talking about writing a Python script that saves data to Postgres.

All in all - good points in the article, but there is more to the decision (as hinted in the summary), building stuff is not just about scratching one’s itch.