HN user

akarve

471 karma

http://www.aneeshkarve.com

Posts14
Comments106
View on HN

This is one of those “so cool yet so silly” brainstorms that I’m grateful someone was audacious enough to entertain. I’m both relieved and saddened that it never came to fruition.

There’s a word, chindogu, to describe things that are less than useless. In some sense this project engendered more problems than it solved. Like so many other attractive brainstorms.

Bad NEWS, Emacs 3 years ago

I was sure to read “the forces of vim have invaded our headquarters and now hold our CEO hostage.”

Hard agree. Civil & mechanical engineering have a culture and history of blameless analysis of failure. Software engineering could learn from them.

See the excellent To Engineer is Human in just this topic of analyzed failures in civil engineering.

If you just make up a number claim it’s prime and nobody disputes it

You can test if a number is prime in polynomial time, much faster than a sieve. There’s no need to test every divisor to know whether a number is prime or not.

Algos like RSA generate large primes millions of times every day—-there’s nothing to take on faith.

Algorithms do not shun creativity. They are born creatively. But they do set a high bar for new contributions. You either beat the existing complexity or you don't.

If you can improve upon either the space/time complexity or even the comprehensibility/maintainability of algorithms, the industry is your oyster.

[dead] 6 years ago

The question one must ask: where were the chances of zoonosis greatest (where did people have the most contact with relevant animals)? and everything else falls into place

Good questions. First, services like open.quiltdata.com and Amazon's Registry of Open Data cover the S3 costs for public data. So that's one incentive. Second, the cost of cloud resources are highly competitive (if not superior) to on-premise data centers (see https://twitter.com/mohapatrahemant/status/11024016152632238... I don't think it's correct to think of S3 as expensive.

There are many ways to shave S3 costs (e.g. intelligent tiering, glacier), but at some point the data become so slow to access that you can't offer a pleasant user experience around browsing, searching, and feeding pipelines.

Most importantly, the "my data, my bucket" strategy gives users control over their data. A university with their own bucket has more control over their data than they do if Google, Facebook, etc. host and monetize it.

We have a data curators program on Quilt and I encourage you to apply (page bottom on open.quiltdata.com). For high-value public data sets, AWS's registry of open data will, if accepted, cover the costs of storage and egress. We went through this process with Allen Cell and I'm happy to help.

Oh I would love to get the UH Cancer Center data into Quilt! Do you happen to have an S3 bucket with that data live? If the bucket is publicly permissioned it should "just work." We can talk about indexing the data for search. We are comfortable in the TB-PB range :)

I will look more closely at Ohayo.

Not quite ;) S3 is the primary data and metadata store, so that the rest of the stack is a pure function of S3 data (including Elastic). We don't use git at all yet. We use S3 object versioning and then capture the version, SHA-256, etag, etc. in a JSONL-based manifest https://open.quiltdata.com/b/quilt-example/tree/.quilt/packa.... Said JSONL manifest is simply a "locked list" of all the S3 objects in that package. The same manifests can be checked into git for fork/merge of data sets, but we're still exploring the right way to do that.

I'll let Kevin answer the database fragments question.

Not yet. But it's closer than one might think. Spaces has an S3-compatible API, and we have plans to use something like min.io to make Quilt work "all the blobs": GCP, Azure, Digital Ocean, etc. OSS contributions welcome :)

What would you use Quilt for in Spaces?