alexott

:facepalm: "orchestration/management layer" that just runs dozens of millions of VMs per day, across the three clouds, with things like model serving, AI gateway, OLTP (Lakebase/Neon), Unity Catalog, etc.

Forget Databricks == Apache Spark...

Parquet alone is not for modern data engineering. Delta, Iceberg should be in the list

Want to be at airport at time? Go with DB a day before and sleep at hotel near airport

I’m also was wondering that it wasn’t mentioned. At some point I did all programming in it, compiling via command line, switching to Borland IDEs only for debugging….

It really depends. I was at talk of architect of car company when he was talking about need to develop and support a car software for 20-30 years - few years before release, 10-20 years of production, and the critical fixes after end of support. And it includes not only soft itself, but all compilers, etc.

Plain parquet has a lot of problems. That’s why iceberg and delta arise

It’s already supported for quite a while: https://duckdb.org/2024/06/10/delta.html

For Germany it’s far from reality… it shows from Paderborn to Dortmund in less than hour, but usually it’s good if you get there in two hours by train…

I still use muse + emacs lisp to generate my CV into html and pdf (via Latex with custom template)

SM-1800 was Intel based, not PDP-11 based. 1800 was based on Russian variant of 8080, and 1810 had both 8080 and 8086 as I remember. https://ru-m-wikipedia-org.translate.goog/wiki/%D0%A1%D0%9C_... will give overview of the SM series. Regarding OS - PDP-based were initially on RSX, later on Soviet variants of unix. Intel based had either custom OS for 8080, or ms-dos like for x86 - Wikipedia article covers it well.

Another nice feature was data exchange between different kernels

Unfortunately, it didn’t get enough community around, and development has stalled. For some time it was sponsored by Alibaba, but at some point of time, the main maintainer left it. Similar story with other people

P.S. I was committer there until changed job.

I was in Thailand in 2017th - mobile internet with 4g was almost everywhere, including small islands. It was a huge contrast to Germany where you need to get 15 minutes drive from most of cities to get only Edge at best, or no mobile coverage at all

You can already have it in Delta with Delta Rust and Python bindings: https://github.com/delta-io/delta-rs

Article (in russian language) about the product and why Scheme was used: https://web.archive.org/web/20210506123442/http://fprog.ru/2...

~20 years ago I worked on commercial software for email security that was using MzScheme (before it became PLT Scheme) as the base language. Code was cross-platform (Solaris, Linux, HP-UX) - OS-specific code was in C, with about 1k lines. Filter rules were compiled into Scheme itself. The whole code was about 30k LoC, including web based UI, and only had 5-6 developers… Later I immigrated and joined company that had similar product with less features with code in C++, with hundred thousand LoC and more developers.

A lot of things has changed quite long ago - not everything is notebook, local dev is fully supported, version pinning wasn’t a problem, cluster startup time heavily dependent on underlying cloud provider, and serverless notebooks/jobs are coming

It’s less about notebooks, but more about SDLC practices. Notebooks may encourage writing throwaway code, but if you split code correctly, then you can do unit testing, write modular code, etc. And ability to use “arbitrary files” as Python packages exists for quite a while, so you can get best of both worlds - quick iteration, plus ability to package your code as a wheel and distribute

P.S. here is a simple example of unit testing: https://github.com/alexott/databricks-nutter-repos-demo - I wrote it more than three years ago.

There is VSCode extension, plus databricks-connect… plus DABs. There are a lot customers doing local only development

Mosaic AI Training (https://www.databricks.com/product/machine-learning/mosaic-a...) as it's mentioned in the announcement blog (https://www.databricks.com/blog/announcing-dbrx-new-standard... - it's a bit less technical)

There is another discussion about it - in context of the technical blog post that has more details. https://news.ycombinator.com/item?id=39838104

Ah, I remember, the self-extracted .msi file was one of the quite challenging files - it's executable, it's a .cab file, and OLE2-container

Mime type detection is very interesting thing. I wrote media type detection for McAfee Web Gateway 7.x and because it was a high performance proxy, the detection speed was a major focus, but also the precision, especially for "container types, like, MS Office, OLE-based files, etc. The base of it was a simple Lisp-like language that allowed to write signatures very fast, and everything was combined with very aggressive caching of the data, so we avoided to read data again and again, and used internal caches a lot. In tests, the detection was ~10x faster than file, and with more flexible language we got more file types recognized precisely. Although there were challenges with some formats, like, OLE-based files had FAT directory structure at the end of the file, and you were need to walk the tree to find the top-level structure to distinguish Excel file from Excel file embedded into Word.

Streams detection was also quite funny task...

There are also pneumatic computers. Here is the recent article: https://cacm.acm.org/news/275366-pneumatic-computing-gains-a..., but I remember (around 30 years ago) mentions of them in the field of nuclear reactors…

As I remember, delta-rs was started by Scribd, not by Databricks: https://youtu.be/2jgfpJD5D6U, https://youtu.be/scYz12UK-OY

Unity Catalog isn’t comparable with Iceberg Catalogs. It’s not required for Delta to function…

There was a paper at VLDB about Delta Lake: https://www.vldb.org/pvldb/vol13/p3411-armbrust.pdf - it describes why it was created, plus details of implementation.

Around 20 years ago I worked on commercial email filtering/archiving system implemented in MzScheme (now PLT) - it was relatively small code base, but very powerful.

I wrote an article about it in Russian (almost 15 years ago), for some reason google translate doesn’t work for archive url: https://web.archive.org/web/20210506123442/http://fprog.ru/2...

Almost 20 years ago I worked at company that developed a commercial product on top of MzScheme (prior to DrScheme and Racket). There was an article about it in Russian, unfortunately now only available via web archive https://web.archive.org/web/20210506123442/http://fprog.ru/2...

Yes, but may depends on the DBR version. >= 12.2 (LTS) should work

New Python SDK covers all public Databricks REST API operations, so you can use it for all things around automation on Databricks...