HN user
It’s surprising to find a join waitlist button for an open source project.
I see that it looks like it will be self-hostable and that the repo is forthcoming but I had to scroll a lot to see that.
Looks interesting though!
I spent a while mixing this up with PiKVM and was having trouble understanding how any of it would fit in with that project. Made a lot more sense once I got over that.
What type of battery life do you see when running these off of a power bank?
This is the book I like for an overview of the problem. https://www.amazon.com/Temporal-Relational-Kaufmann-Manageme...
[grizzled kdb+ user considers starting an argument but then thinks better of it]
Looks interesting! It's not obvious to me on the website what the pricing or licensing situation is with DBSnapper. How does that work? Thanks!
Very excited to hear this! Congrats!
Obviously there is no actual model. ChatGPT is just tens of thousands of people writing responses to user's queries in realtime.
Would the licensing of Grist allow for building widgets/components to use it in dashboard tools like Streamlit, Jupyter, or Holoviz's Panel? Looks cool!
Pandas, Django, and Flask.
I’ve found the Python Speed [1] site an excellent resource for the quirks of working with Python in containers for data science use cases.
I've been wishing that JetBrains and Posit (nee RStudio) would adopt the devcontainer standard, but then I just read this article [1] about how MSFT is using VS Code's weird mix of open-source and proprietary components to fracture the market and ensure that any competitors who try to build off of VS Code are at a permanent disadvantage. Now I'm having second thoughts.
Devcontainers are a good example of their strategy. VS Code's source code is open source, but the devcontainers extension is not [2] and alternative vscode-in-the-browser providers are not able to use the devcontainers extension as they're not allowed to use the official VS Code extension marketplace.
[1] https://ghuntley.com/fracture/ [2] https://twitter.com/castrojo/status/1671544329402302464?s=20
I'm kind of obsessed with this space and have spent way too long setting up VS Code and RStudio IDEs on my homelab.
One area I struggle with when thinking about building container-based development environments is the best way to avoid mixing in your IDE specific dependencies with your project dependencies. I think some of the commercial tools do this, but I haven't gotten in set up well in my homelab.
I just came across two articles by a former GitPod employer who moved to Coder (these are the two main providers of open-source VS Code in the browser solutions). They're both really interesting.
The first is on how effectively Microsoft has used VS Code to fracture the market place to their advantage by strategically open-sourcing parts of VS Code while keeping many of its best features proprietary (Pylance, the python language server is a good example of this). https://ghuntley.com/fracture/
The second article is about why he thinks Coder's strategy is more promising than GitPod's prompting him to go work for them. It's not as detailed, but it touches on some of the parts of container-based development environments that I've found overly limiting. https://ghuntley.com/integrate/
Why does “pip install gcc” keep failing then? ;)
Conda isn’t perfect but takes on a lot of problems that pip doesn’t deal with at all. Regular conda is really slow these days but you can use mamba instead or just configure conda to use the libmamba solver and it’s much nicer.
The folks at prefix.dev seem to be building some pretty cool drop in replacements for conda too.
If you use conda there are extensions that can help with this by automatically registering any available conda environments that include ipykernel in your Jupyter Lab environment.
nb_conda_kernels is pretty reliable but not actively maintained. Gator from the mamba folks is new and still a bit rough around the edges but looks like it will be pretty slick eventually.
Nice article, thanks!
It would be great if DuckDB handled this itself, but it seems like to be competitive with Athena on really massive datasets, you need to have a metadata layer that is used to figure out which parquet files in S3 DuckDB actually needs to query and then potentially run those in parallel. This seems to be the architecture of Puffin (which I haven't personally tried using yet).
[1] https://www.boilingdata.com/ [2] https://boilingdata.medium.com/lightning-fast-aggregations-b... [3] https://github.com/sutoiku/puffin
One possible thing to look into would be whether this dataset is partitioned too much. My understanding is that the recommended file size for individual parquet files is 512MB to 1GB, whereas here they are 50MB. It would be interesting to see the impact of the partitioning strategy on these benchmarks.
[4] https://parquet.apache.org/docs/file-format/configurations/ [5] https://www.dremio.com/blog/tuning-parquet/
I think the issues of privilege are worth considering, but I think they’re not the only dimension worth thinking about.
Consider this hypothetical. You’ve got two equally wealthy kids who go to the same private school, take all the same classes, and take the same SAT prep class.
Kid 1 has a perfect GPA and a 90th percentile SAT score. Kid 2 has perfect SAT score and a 90th percentile GPA. If there’s only one spot, which kid should a college admissions office pick?
How so?
Really wish they'd released this as open source. Sounds interesting.
Portable hard drive is probably the most practical. Another option not on this list would be rclone [1].
Designing Data Intensive Applications is pretty popular.
I’m really excited about the potential for this in my work. I’ve worked in data science / data engineering roles and had to learn a lot of random web stuff that I honestly didn’t really want to learn (e.g. CORS, HTTPS, etc). I’d much rather build a TUI than a web app with these features.
Quarto is also pretty interesting for this.
FWIW, I think the comparison with Bloomberg is a bit of a distraction. It seems unlikely that you can live up to that 90% claim, but that doesn’t mean it’s not a good product or good value. It looks like you’re targeting a different market than Bloomberg users. I took a quick look at your site and found the framing there much more compelling than the direct dig at Bloomberg here.
Not a marketer though, so what do I know. Good luck!
Honestly, I’m skeptical that Bloomberg chat is really going to add much value to a random individual who springs for a Bloomberg terminal.
If you’re a bond trader at some bank and all the other bond traders are chatting on Bloomberg, you’re missing out. But they’re not going to chat with me even if I got myself a Bloomberg terminal.
I made a similar transition at 35. Have you looked at quant developer roles? They may value your experience and background more and be more willing to compromise on your software engineering experience.
This is really interesting!
Have you given much thought to either type annotations or decorators. One or both of them could potentially be useful for attaching schema information to the DataFrame inputs/outputs of functions and for reducing boilerplate when you might want to validate those inputs/outputs before or after calling the functions.
Agreed about the similarities between Excel and Pandas. I started out as more of an data analyst and am now a SWE and I think one of the things that SWEs who dismiss Excel and Jupyter don't understand is how little you can assume about the data you might be working with.
If you're an analyst who knows some VBA, that can be super useful but it would probably be a mistake to try to make your VBA-driven applications bullet proof. Nobody wants you to spend that much time on it, and the odds that something completely out of your control will change and break it anyway are quite high.
Postmodern Pandas sounds like a great title for a blog post.