radus

It's simple, just tell it not to hallucinate.

Okay, but why?

I'm working on adding features to the snakemake aws batch executor plugin. The existing plugin supports execution on AWS Batch by dynamically creating job definitions based on rule resource configuration, but was missing support for features like using different containers for different rules, consumable resources, secrets, etc. Two approaches:

1) https://github.com/radusuciu/snakemake-executor-plugin-aws-b... (my fork). Just add the features to the batch job building code 2) https://github.com/radusuciu/snakemake-executor-plugin-aws-b.... This is more experimental and not yet fully working. I wanted to try a few things. a) can we rely on existing job definitions (managed through IaC instead). b) can we implement a fire-and-forget model where the main snakemake process runs on Batch as well? c) Can we slim down the snakemake container by stripping off unnecessary features.

Polars has a much more consistent API, give it a shot.

Regarding your plotting question: use seaborn when you can, but you’ll still need to know matplotlib.

But are you really going to repair it?

Yes

Polars and duckdb interoperate nicely and can enable this flexibility

I like using EFK (ElasticSearch-Fluent-Kibana) for this

I guess then you've got to abstract one level further and formulate your advice as an allegory.

Example: you want to set your local docker context to the production environment, so that when you type `docker system prune --volumes` you delete your production data.

The answer is “it depends”

Quick critique: module contains functions with many parameters, many branches, deep nesting, and multiple return points.

docker swarm is also a decent solution if you do need to distribute some workloads, while still using a docker compose file with a few extra tweaks. I use this to distribute compute intensive jobs across a few servers and it pretty much just works at this scale. The sharp edges I've come across are related to differences between the compose file versions supported by compose and swarm. Swarm continues to use Compose file version 3 which was used by Compose V1 [1].

1: https://docs.docker.com/engine/swarm/stack-deploy/

They were defending rocks. They had not uncovered evidence of native martian life.

See more discussion here: https://scifi.stackexchange.com/questions/160959/is-or-was-t...

You've explained your value proposition vs. celery, but I'm curious if you also see Hatchet as an alternative to Nextflow/Snakemake which are commonly used in bioinformatics.

Example: you use a large plotting library, but it's only needed on some pages. Chunking allows you to selectively load that code only where it's needed.

My approach for this is to use multi-stage dockerfiles. Build tools live and operate in a build stage, and you copy the /venv into your runtime stage. I think this same approach should work with uv so long as it doesn't bundle itself into the venv.

This is great! I’ve been waiting a long time for this and it seems like more debug features are in the pipeline too: https://github.com/docker/buildx/issues/1104

Docs: https://docs.docker.com/engine/reference/commandline/buildx_...

This is my experience as well :/

The term translational is frequently used to describe basic research (eg. with cells or mice) that has applications to human health - findings in lab that make it to the clinic.

I’m not sure that dropping flaming batteries from the sky will be very popular with those on the ground.

Great tool, I use it with this alias:

  alias dive='docker run -ti --rm -v /var/run/docker.sock:/var/run/docker.sock wagoodman/dive'

Totally agree with the critique, though it bears mentioning that one way that polars differentiates itself from pandas is the expressions API. See [1] for an example.

1: https://kevinheavey.github.io/modern-polars/method_chaining....

That's awesome, thanks for sharing! Though tbh I'm not likely to use it.. it's a bit too magical - though still a delicious hack.

I'm not including parsing time, both pandas and polars versions started from an in-memory data structure parsed from two XML files (low GB range). This is on my workstation with a single Xeon 4210 (10 cores, 20 threads @ 2.20-3.20Ghz).

Perhaps I can focus on a subset of this processing and write this up since it seems like there's at least some interest in real examples. As pointed out in a reply to a sibling comment, I don't guarantee that my starting code is the best that pandas can do -- to be honest, the runtime of the original code did not line up with my intuition of how long these operations should take. Maybe someone will school me but either way switching to polars was a relatively easy win that came with other benefits and feels right to me in a way that pandas never did.

To add some balance:

- I can't rule out that a pandas wizard couldn't have achieved the same speed-up in pandas

- polars code was slightly more verbose. For example, when calculating columns based on other columns in the same chain, in pandas, each new column can be defined as a kwarg in a single call to `assign`, whereas in polars, columns that depend on other must be defined in their own calls to `with_columns`

- handling of categoricals in polars seemed a little underbaked, though my main complaint, that categories cannot be pre-defined, seems to have been recently addressed: https://github.com/pola-rs/polars/issues/10705

- polars is not yet 1.0, breaking changes will happen

It allowed me to take some code that reads in a bunch of data and performs a few rounds of some pretty standard operations (groupby, filtering, calculating mean/stdevs) and going from pandas to polars allowed me to go from ~1 minute per dataset to 1 second (yes, I tried the Arrow backend for pandas too). This was after spending some time profiling the pandas code and fixing up the slowest parts as best as I could. The translation was pretty straightforward. The output of this pipeline code was a few different dataframes (each to be inserted into a separate table) and each dataframe was output from a function. I was able to migrate one function at a time after asserting that the outputs of the two functions were identical and that all relevant tests passed (I used `to_pandas()` where needed).

I'm not sure how much faster I could go, since ~1 second/dataset allowed me to answer some questions that I had that required scanning values for a few parameters. The biggest wins for me were in grouping and merging operations.

I'm a complete convert now. The API is simpler and more obvious IMO, and the ability to compose expressions (`polars.Expr`) is awesome. The performance benefits are nice and what motivated me in the first place, but I'm more swayed by the aforementioned benefits.

Many plots contain thousands of data points. Eg: 10 x 100 heatmap which supports sorting by various metadata. This is a common visualization for biological data where your data matrix is samples x proteins, so potentially much larger than 1000 data points.

Nitrile is a fine default at home and in the lab if you make sure to change your gloves when they are contaminated. You can also double glove.

For most items you don’t need to do anything beyond a scrub and rinse. Others have pointed out that glassware is expensive and sometimes unique. Additionally, if you don’t need much product (eg. you’re screening reaction conditions), you can work with small disposable vials or even plates.

It's easier to re-use if you need the same generated field in multiple places.