boyd

Throwing cores at the problem with `pytest-xdist` is typically the lowest hanging fruit, but you still hit all the paper cuts the authors mention -- collection, DB fixtures, import time, etc.

And, further optimization is really hard when the CI plumbing starts to dominate. For example, the last Warehouse `test` job I checked has 43s of Github Actions overhead for 51s of pytest execution time (half the test action time and approaching 100% overhead).

Disclosure: Have been tinkering on a side project trying to provide 90% of these pytest optimizations automatically, but also get "time-to-first-test-failure" down to ~10 seconds (via warm runners, container snapshotting, etc.). Email in profile if anyone would like to swap notes.

I think we're somewhat talking past one another -- in any case, we'll add more in the README on minimal perfect hash functions and the differences. In short, you'd need to also have a data structure (e.g., a Bloom filter) for checking if the key is in your MPHF and then also a mapping of 1..n MPHF values to your actual values.

PTHash and other minimum perfect hash functions return an arbitrary value if the query key did not exist when building the MPHF, so they can be a lot smaller. B-field can identify query keys that don't exist in the set (with high probability?).

Yes, exactly.

What I'm wondering is why the Kraken2 probabilistic hash table doesn't work.

I just skimmed the paper again (has been a while since a close reading), but my refreshed understanding is:

* Like the B-field, there are also false positives.

* When multiple hashed keys (k-mers) collide in the Kraken2 hash table, it has to store a "reduced" value for those key-value pairs. While there's an elegant solution for this issue for the problem of taxonomic classification (lowest common ancestor), it still results in a loss of specificity. There's a similar issue with "indeterminate" results in the B-field, but this rate can be reduced to ~0 with secondary arrays.

* The original Kraken2 paper describes using 17 bits for taxonomic IDs (~131K unique values). I don't know how many tax IDs current Kraken2 DB builds use offhand, but the error rate climbs significantly as you use additional bits for the value vs. key (e.g., to represent >=2^20 values, see Fig S4). I don't have a good sense for the performance and other engineering tradeoffs of just extending the hash code >32 bits. I also don't know what the data structure overhead is beyond those >32 bits/pair.

So, for a metagenomics classifier specifically, some subtle tradeoffs but honestly database quality and the classification algorithm likely matters a lot more than the marginal FP rates with either data structure -- we just happen to have come to this solution.

For other applications, my sense is a B-field is generally going to be much more flexible (e.g., supporting arbitrary keys vs. a specific fixed-length encoding) but of course it depends on the specifics.

... meaning it is an "injective" function that maps unique key-value pairs, correct? Genuinely asking, I have glancing familiarity via their use in assembly algorithms but (a) don't have a formal math/CS background; and (b) haven't read any of the papers recently.

My understanding is that a perfect hash function maps elements elements to a unique integer (i.e., it's a one-to-one mapping). I think PHF data structures will also always return a value. So if you look up an element not in the constructed PHF, you'll always get a "false positive" value.

In contrast, a B-field lets you map a key to an arbitrary number of (typically non-unique) values. So I could map a million elements to "1", another million to "2", etc.

I'm not especially current (or fluent!) in that literature though, so would love pointers to anything that doesn't have the above constraints.

So a bitmap index requires a bit per unique value IIRC (plus the key and some amount of overhead). So for ~32 unique values you're already at 4 bytes, 40 bytes per key-value pair for 320 values, etc.

In comparison, a B-field will let you store 32 distinct values at ~3.4 bytes (27 bits) per key-value pair at a 0.1% FP rate.

Thank you! The "Space Requirements" section in the README has a few examples, and your comment has made me realize our (micro-)benchmark link in the README is broken.

We'll get that fixed and maybe find the time to do a larger post with some benchmarks on both space/time tradeoffs and overall performance vs. other data structures.

So it’s for cases where you have any key but associated with one of only (preferably few) discrete values

We use it for a case with ~million unique values, but it's certainly more space efficient for cases where you have tens, hundreds, or thousands of values. The "Space Requirements" section has a few examples: https://github.com/onecodex/rust-bfield?tab=readme-ov-file#s... (e.g., you can store a key-value pair with 32 distinct values in ~27 bits of space at a 0.1% false positive rate).

all the docs say “designed for in-memory lookups”

We use mmap for persistence as our use case is largely a build-once, read many times one. As a practical matter, the data structure involves lots of random access, so is better suited to in-memory use from a speed POV.

fyi, you use temp::temp_file() but never actually use the result, instead using the hard-coded /tmp path

Thank you, have opened an issue and we'll fix it!

Ah, we need to clarify the language! The B-field will always return the correct value for an inserted key.

False positives are only returned for keys that have not been inserted. This is akin to a Bloom filter falsely returning that a key is in the set).

Yes you can manage the error rate by controlling the overall size of the allocated bit array and several other parameters. There's a (slightly obtuse) section on parameter selection here: https://github.com/onecodex/rust-bfield?tab=readme-ov-file#p...

And a Jupyter notebook example here: https://github.com/onecodex/rust-bfield/blob/main/docs/noteb...

We do need a better "smart parameter selection" method for instantiating a B-field on-the-fly.

I think those are both good examples of where you can manage the cost of a false positive.

In genomics, we're using this to map a DNA substring (or "k-mer") to a value. We can tolerate a very low error rate for those individual substrings, especially since any erroneous values will be random (vs. having the same or correlated values). So, with some simple threshold-based filtering, our false positive problem goes away.

Again, you'll never get the incorrect value for a key in the B-field, only for a key not in the B-field (which can return a false positive with a low, tunable error rate).

We’re more focused on data visualization vs. general “chat with your data”, but building in this space: https://minard.ai

Upload or import a CSV, Excel, Google sheet, etc. and also just launched Postgres and Snowflake connectors today!

It looks like on mobile in portrait that link isn’t showing up in the header. Thanks for flagging!

We’ll get that fixed, but in the meantime the direct login link is https://minard.ai/accounts/login/

Note that the site is not mobile optimized at the moment, so you’ll want to be on desktop or at least a tablet to actually play around with it.

It’s fairly common to use two hash functions and then compute the remaining n hashes as hn(x) = h1(x) + n * h2(x).

Paper: https://link.springer.com/chapter/10.1007/11841036_42

Feel obliged to point out this is misleading / not especially relevant: GDP is the value of the annual output of a country, while market capitalization is the (discounted) value of all future cash flows of a company. LVMH is ~0.5T market cap French company, but it’s nowhere near 1/6th of French economic output.

This is a lot like the recent “Novo Nordisk’s market cap is greater than the GDP of Denmark!” comparisons — true, but they’re not like numbers (and indeed, much of Novo Nordisk’s revenue is captured in Danish GDP).

I was looking at this a while back and it was difficult/tedious at the time (and I think also not supported on Amazon EKS).

Has it gotten better? Any resources you recommend? Cursory Google searches on this have so much outdated info it can be hard to quickly wade back in.

Futons are neither good couches nor good beds.

One Codex (YC S14) | San Francisco (Mission) | Software + Scientist Roles | Onsite + Remote | https://www.onecodex.com

One Codex is a platform for microbial genomics. We are a technical, experienced team working on meaningful problems that range from infectious disease diagnostics to outbreak epidemiology to improving our understanding of the microbiome. We work with top researchers, medical institutions, and biotechs, and have processed samples from all seven continents (and space!). Here's what we're doing to help out with COVID-19: https://www.onecodex.com/blog/2020/03/16/covid-19-sequencing...

We're currently looking for engineers across multiple positions, including both full stack and DevOps roles. Our stack includes Python, Rust, and Javascript/Typescript (React), and we write everything from D3 visualizations to low-level bioinformatics algorithms. We are also hiring for a computational biologist, an account manager, and a marketing manager.

Challenges include: (1) developing novel algorithms for analyzing complex microbial communities; (2) working with terabytes of genomic data; (3) building scientifically reproducible analyses suitable for both research and the clinic; and (4) supporting scientists and developers building on our platform with extensible APIs.

We are based in San Francisco and offer a competitive salary and meaningful above-market equity. Benefits include full medical, dental, and vision coverage, and a flexible vacation policy.

Please apply here: https://www.onecodex.com/careers/

One Codex (YC S14) | San Francisco (Mission) | Software + Scientist Roles | Onsite + Remote | https://www.onecodex.com

One Codex is a platform for microbial genomics. We are a technical, experienced team working on meaningful problems that range from infectious disease diagnostics to outbreak epidemiology to improving our understanding of the microbiome. We work with top researchers, medical institutions, and biotechs, and have processed samples from all seven continents (and space!). Here's what we're doing to help out with COVID-19: https://www.onecodex.com/blog/2020/03/16/covid-19-sequencing...

We're currently looking for engineers across multiple positions, including both full stack and DevOps roles. Our stack includes Python, Rust, and Javascript/Typescript (React), and we write everything from D3 visualizations to low-level bioinformatics algorithms. We are also hiring microbiologists/computational biologists.

Challenges include: (1) developing novel algorithms for analyzing complex microbial communities; (2) working with terabytes of genomic data; (3) building scientifically reproducible analyses suitable for both research and the clinic; and (4) supporting scientists and developers building on our platform with extensible APIs.

We are based in San Francisco and offer a competitive salary and meaningful above-market equity. Benefits include full medical, dental, and vision coverage, and a flexible vacation policy.

Please apply here: https://www.onecodex.com/careers/

One Codex (YC S14) | San Francisco (Mission) | Software + Scientist Roles | Onsite + Remote | https://www.onecodex.com One Codex is a platform for microbial genomics.

We are a technical, experienced team working on meaningful problems that range from infectious disease diagnostics to outbreak epidemiology to improving our understanding of the microbiome. We work with top researchers, medical institutions, and biotechs, and have processed samples from all seven continents (and space!). Here's what we're doing to help out with COVID-19: https://www.onecodex.com/blog/2020/03/16/covid-19-sequencing...

We're currently looking for engineers across multiple positions, including both those who are backend- and frontend-leaning. Our stack includes Python, Rust, and Javascript/Typescript (React), and we write everything from D3 visualizations to low-level bioinformatics algorithms. We are also hiring microbiologists/computational biologists.

Challenges include: (1) developing novel algorithms for analyzing complex microbial communities; (2) working with terabytes of genomic data; (3) building scientifically reproducible analyses suitable for both research and the clinic; and (4) supporting scientists and developers building on our platform with extensible APIs.

We are based in San Francisco and offer a competitive salary and meaningful above-market equity. Benefits include full medical, dental, and vision coverage, and a flexible vacation policy.

Please apply here: https://www.onecodex.com/careers/

One Codex (YC S14) | San Francisco (Mission) | Software + Scientist Roles | Onsite + Remote | https://www.onecodex.com

One Codex is a platform for microbial genomics. We are a technical, experienced team working on meaningful problems that range from infectious disease diagnostics to outbreak epidemiology to improving our understanding of the microbiome. We work with top researchers, medical institutions, and biotechs, and have processed samples from all seven continents (and space!). Here's what we're doing to help out with COVID-19: https://www.onecodex.com/blog/2020/03/16/covid-19-sequencing...

We're currently looking for engineers across multiple positions, including both those who are backend- and frontend-leaning. Our stack includes Python, Rust, and Javascript/Typescript (React), and we write everything from D3 visualizations to low-level bioinformatics algorithms. We are also hiring microbiologists/computational biologists.

Challenges include: (1) developing novel algorithms for analyzing complex microbial communities; (2) working with terabytes of genomic data; (3) building scientifically reproducible analyses suitable for both research and the clinic; and (4) supporting scientists and developers building on our platform with extensible APIs.

We are based in San Francisco and offer a competitive salary and meaningful above-market equity. Benefits include full medical, dental, and vision coverage, and a flexible vacation policy.

Please apply here: https://www.onecodex.com/careers/

Those systems process up to 384 or 1056 samples in ~8 hours. The proposed method should support a similar volume (even a bit higher) than the Roche 8800, but with a different large installed instrument base. So this should add a lot of capacity to that qPCR.

They accepted more tests than they had the ability to quickly process. They also appear to have accepted many tests that required use of a lower throughput assay before switching to higher throughput testing on Roche 8800s: https://www.theatlantic.com/health/archive/2020/03/next-covi...

One Codex (YC S14) | San Francisco (Mission) | Software + Scientist Roles | Onsite + Remote | https://www.onecodex.com

One Codex is a platform for microbial genomics. We are a technical, experienced team working on meaningful problems that range from infectious disease diagnostics to outbreak epidemiology to improving our understanding of the microbiome. We work with top researchers, medical institutions, and biotechs, and have processed samples from all seven continents (and space!). Here's what we're doing to help out with COVID-19: https://www.onecodex.com/blog/2020/03/16/covid-19-sequencing...

We're currently looking for engineers across multiple positions, including both those who are backend- and frontend-leaning. Our stack includes Python, Rust, and Javascript/Typescript (React), and we write everything from D3 visualizations to low-level bioinformatics algorithms. We are also hiring microbiologists/computational biologists.

Challenges include: (1) developing novel algorithms for analyzing complex microbial communities; (2) working with terabytes of genomic data; (3) building scientifically reproducible analyses suitable for both research and the clinic; and (4) supporting scientists and developers building on our platform with extensible APIs.

We are based in San Francisco and offer a competitive salary and meaningful above-market equity. Benefits include full medical, dental, and vision coverage, and a flexible vacation policy.

Please apply here: https://www.onecodex.com/careers/

No, sensitivity is TPs / all positives (detected TP and FN). It’s the green half of the diagram in your link. The language is very easy to get tripped up by though. :)

One Codex (YC S14) | San Francisco (Mission) | Software + Scientist Roles | Onsite + Remote | https://www.onecodex.com

One Codex is a platform for microbial genomics. We are a technical, experienced team working on meaningful problems that range from infectious disease diagnostics to outbreak epidemiology to improving our understanding of the microbiome. We work with top researchers, medical institutions, and biotechs, and have processed samples from all seven continents (and space!).

We're currently looking for engineers across multiple positions, including both those who are backend- and frontend-leaning. Our stack includes Python, Rust, and Javascript/Typescript (React), and we write everything from D3 visualizations to low-level bioinformatics algorithms. We are also hiring microbiologists/computational biologists.

Challenges include: (1) developing novel algorithms for analyzing complex microbial communities; (2) working with terabytes of genomic data; (3) building scientifically reproducible analyses suitable for both research and the clinic; and (4) supporting scientists and developers building on our platform with extensible APIs.

We are based in San Francisco and offer a competitive salary and meaningful above-market equity. Benefits include full medical, dental, and vision coverage, and a flexible vacation policy.

Please apply here: https://www.onecodex.com/careers/

One Codex (YC S14) | San Francisco (Mission) | Software + Scientist Roles | Onsite + Remote | https://www.onecodex.com

One Codex is a platform for microbial genomics. We are a technical, experienced team working on meaningful problems that range from infectious disease diagnostics to outbreak epidemiology to improving our understanding of the microbiome. We work with top researchers, medical institutions, and biotechs, and have processed samples from all seven continents (and space!).

We're currently looking for engineers across multiple positions, including both those who are backend- and frontend-leaning. Our stack includes Python, Rust, and Javascript/Typescript (React), and we write everything from D3 visualizations to low-level bioinformatics algorithms. We are also hiring microbiologists/computational biologists.

Challenges include: (1) developing novel algorithms for analyzing complex microbial communities; (2) working with terabytes of genomic data; (3) building scientifically reproducible analyses suitable for both research and the clinic; and (4) supporting scientists and developers building on our platform with extensible APIs.

We are based in San Francisco and offer a competitive salary and meaningful above-market equity. Benefits include full medical, dental, and vision coverage, and a flexible vacation policy.

Please apply here: https://careers.onecodex.com/

One Codex (YC S14) | San Francisco (Mission) | Software + Scientist Roles | Onsite + Remote | https://www.onecodex.com

One Codex is a platform for microbial genomics. We are a technical, experienced team working on meaningful problems that range from infectious disease diagnostics to outbreak epidemiology to improving our understanding of the microbiome. We work with top researchers, medical institutions, and biotechs, and have processed samples from all seven continents (and space!).

We're currently looking for engineers across multiple positions, including both those who are backend- and frontend-leaning. Our stack includes Python, Rust, and Javascript/Typescript (React), and we write everything from D3 visualizations to low-level bioinformatics algorithms. We are also hiring microbiologists/computational biologists.

Challenges include: (1) developing novel algorithms for analyzing complex microbial communities; (2) working with terabytes of genomic data; (3) building scientifically reproducible analyses suitable for both research and the clinic; and (4) supporting scientists and developers building on our platform with extensible APIs.

We are based in San Francisco and offer a competitive salary and meaningful above-market equity. Benefits include full medical, dental, and vision coverage, and a flexible vacation policy.

Please apply here: https://careers.onecodex.com/

One Codex (YC S14) | San Francisco (Mission) | Software + Scientist Roles | Onsite + Remote | https://www.onecodex.com

One Codex is a platform for microbial genomics. We are a technical, experienced team working on meaningful problems that range from infectious disease diagnostics to outbreak epidemiology to improving our understanding of the microbiome. We work with top researchers, medical institutions, and biotechs, and have processed samples from all seven continents (and space!).

We're currently looking for engineers across multiple positions, including both those who are backend- and frontend-leaning. Our stack includes Python, Rust, and Javascript/Typescript (React), and we write everything from D3 visualizations to low-level bioinformatics algorithms. We are also hiring microbiologists/computational biologists.

Challenges include: (1) developing novel algorithms for analyzing complex microbial communities; (2) working with terabytes of genomic data; (3) building scientifically reproducible analyses suitable for both research and the clinic; and (4) supporting scientists and developers building on our platform with extensible APIs.

We are based in San Francisco and offer a competitive salary and meaningful above-market equity. Benefits include full medical, dental, and vision coverage, and a flexible vacation policy.

Please apply here: https://jobs.onecodex.com/

Does anybody know if (and how) this applies to spot instances? None of the language seems to exclude application to spot prices, though I realize the use-it-or-lose-it nature of these discounts has some tensions with bursty spot workflows.