MichaelSalib

So, my spouse was a CPU designer at AMD for many years and now does secure computing work for, well, the US government. I showed her your comment. She laughed. A lot.

This is all completely wrong.

To be clear, our experience using gcsfuse and friends to do basically the same things was extremely painful and a performance nightmare. The HDF format was designed for a world where seeks are free which makes cloud access very high latency and very low throughput.

Ah, thanks for these! But I see nothing has changed. * pyfive is interesting but immature and doesn't seem to have any cloud bucket support * h5s3 is an abandoned experiment that hasn't been touched in two years * h5py is fine but again, no cloud support * kita is a commercial offering from the HDF Group and -- I cannot stress this enough -- these people are shockingly incompetent; plus when I last looked at their system architecture diagram I thought it was a joke (well, I thought it was an intentional joke)

Efficient access to scientific datasets hosted on S3/GCP is a full blown crisis in the scientific computing community. People aren't switching to zarr for the fun of it, but because zarr is here, today, and isn't a joke, and is actually open.

No, you can certainly write in parallel, despite the GIL. The GIL makes this inefficient if your work is CPU-bound, but for IO-bound workloads it can be fine.

Otherwise, HDF5 offers every single advantage that zarray has and is much more mature, stable, better documented, and has better support.

Absolutely not. HDF5 is an awful format with terrible implementations. For example, try writing a python program with multiple threads where each thread writes to a different HDF5 file. This should just work -- there's no concurrent access. And yet it doesn't because HDF5 implementations are piles of ancient C code that use lots of global state. There's no technical reason for this; one could easily store all the state needed in a per-file object. But back in the day, software eng standards were lower (especially for scientists) and HDF5 changes at a glacial place.

I've been bitten by this particular bug, but you really have to wonder: given how poorly it speaks to the software engineering behind HDF5 implementations, what else is broken in the code or specifications?

If you're working in a situation where it makes sense to have things on disk or some sort of NFS share, use HDF5. If you're working with objects in a cloud bucket, you'll incur additional overhead with HDF5, as you'll have to read its table of indices, then make range requests to each chunk. Zarr is optimized for the cloud use case.

When last I looked, there were no open source HDF5 implementations that were smart enough to do range requests to cloud hosted hdf files. Has this changed?

What's the plan for when grandma develops dementia? The paranoia will make her unwilling to part with her beloved firearm...do we just wait until she murders a family member or home health aid?

That seems like a much more important question than bickering over precisely which firearm grandma should have as she slides into dementia.

https://khn.org/news/dementia-and-gun-safety-when-should-agi...

Which do you think looks better on your promotion packet?

1. Write a new libc and get it used all over Google, generating lots of users automatically; or

2. Shepard Google's adoption of code some other folks wrote.

The idea that children should spend their days staring at screens and never interacting with other human beings seems really sick. But oh so on brand for the Valley.

I worked at FB briefly so maybe I can explain. FB has a corporate culture that really discourages critique. When things are broken, especially internal things, people look at you funny if you speak up about it. A big part of that is that quarterly bonuses are given for "making an impact" and your group's status (and part of your bonus) is based on delivering a consistent set of "impacts" over time. So it is better for your comp to do things badly really fast since you get (1) did something super fast! and (2) get to record a big impact a few months later when you fix the obvious brokenness.

Pretty quickly, people learn to keep their mouth shut.

Also, many, many FB engineers are early-career folk who are fresh out of school. More senior folk are few and far between and are even more strongly incentivized to keep their mouth shut, because their bonuses are bigger.

The way that white supremacy works is that black people are penalized even in the small enclaves where they constitute a local majority. Note that the laws that punish crack (used by lower income black people) 100 times more harshly than powder cocaine [0] are federal in nature.

[0] https://www.aclu.org/other/cracks-system-20-years-unjust-fed...

When it comes to exceptions, it's generally true on x64 that you don't pay for what you don't use (there's no performance penalty to exception handling if you don't throw) although that hasn't always been true for all platforms and implementations. It's also generally true that you couldn't implement that kind of non local flow control more efficiently yourself, although the value of that guarantee is a little questionable with exception handling.

I'm inclined to agree with you but just about everyone at Google says the opposite and most C++ shops I've seen agree with them. I've made this argument and lost repeatedly. So, it seems like the community can't even agree on which abstractions are zero cost (or maybe whether some zero cost abstractions are actually zero cost?). To the extent that the community itself has no consensus about these things, maybe they're not a marketing slogan that's helpful to use.

(1) I love your writing.

(2) Even reducing it to runtime costs, it seems a bit nonsensical. Are C++ exceptions a zero cost abstraction? All the googlers I argued with about them would insist that they have unacceptably high runtime costs.

OK, but templates are surely zero (runtime) cost abstractions, right? Unless you start to worry about duplicate code blowing out your instruction cache but if that's a problem, no profiler in the world will ever be able to tell you, so I guess you'll never know just how costly the abstraction is, so you might as well continue believing it is zero...?

Zero-cost abstractions only exist in a world where you don't highly value language simplicity and comprehensibility.

Simplicity and comprehensibility were things the committee had to give up in order to pretend they had "zero-cost" abstractions. Nothing in life comes free: everything, including all abstractions, comes at some cost.

To clarify: of course they like simplicity when it costs nothing. But they consistently value other goods over simplicity.

For example: maintaining backwards compatibility. The community believes that it is more important that 20 year old C++ code run unmodified than that the language should be simplified. There's lots of stuff you could do to simplify the language but options dry up in a world where 20 year old code must be able to run unmodified.

So sure, the committee talks a lot about simplicity, but it isn't willing to sacrifice much.

Don't get me wrong: I'm glad that finally, in 2020, C++ will be almost but not quite as good as Common Lisp was at metaprogramming back in 1982. But it remains the case that eval-when and defmacro are both more powerful and dramatically simpler than anything the C++ committee has ever considered.

I think that's dependent on language culture. Because the C++ community doesn't take language simplicity or comprehensibility seriously, there are lots of C++ developers who can't use or reason about surprisingly large parts of the language. So the community has rallied around the notion that "library" developers need to understand everything and that most developers will just glue together bits that the library devs made.

I mean, how many C++ developers actually write serious template code? How many of them could reliably explain what the keywords in post do?

The idea that every developer is a library author (or the lisp extension that every developer is a language author) is common in many other language communities but it relies on the community working hard to make mastery of the language feasible for lots of people. The C++ community never bought into that notion; they inherited a very stratified class structure from Bell Labs.

That's (maybe) great for folks in China, but what about folks outside? Is the rest of the world really better off if Google can easily be bullied and manipulated by the Chinese government?

Once they start taking orders from Beijing about what results to show in China, why shouldn't they take orders about what to show in the US? After all, the Chinese government controls their access to an enormous market worth lots of money. Will we be able to find articles about the concentration camps for Muslims in China after they're operating in China?

Japan is intensely racist. My spouse lived there for a year. When she went about her business outside, small children who saw her would literally burst into tears screaming to their parents about gaijin.

I saw this up close in Boston (really Cambridge MA). In most American municipalities, neighborhood groups have enormous power to shape development. All development has to get approved by the city, and the city listens to neighborhood groups. Those groups can also petition a landmark commission to landmark a building making redevelopment much more difficult. And they can sue. Even when these tactics don't work, they're extremely time consuming. I've seen one court case drag on for years even though the people filing literally submitted briefs written in crayon. So it is often better to just settle with neighborhood groups, but like patent trolls, that only encourages them.

One issue is that in the US, we treat housing as the primary source of wealth for middle class people. So everyone is expected to be a homeowner. A world where the price of your primary wealth asset is rising is good! And a world in which its value is flat or falling is disastrous. And so local laws get shaped so as to keep housing prices high; in practice that means choking the supply of new more dense housing. After all, any change might harm the value of your home!

Another good resource from Google is [1] which focuses more on operational impacts after you deploy a system that relies on ML (I'm a coauthor). [2], which was written by my boss, is also great.

[1] https://sites.google.com/site/wildml2016nips/SculleyPaper1.p...

[2] https://research.google.com/pubs/pub43146.html

It looks like they're using a 65nm process, so yeah, fab space appears to be limiting them. Modern designs are well under 20nm.

Around 2003 at MIT I spoke with a guy who told me about a home lighting system that the French were using. Basically, you take an entire wall in a house and sandwich thin conductive layers between insulator layers. Then you have these special light fixtures with a sharp pronged tip that exposes three different wire heads at different lengths.

You just stab one of these fixtures into the wall anywhere you want; each of the three prongs travels to a different depth and makes contact with a layer representing the hot, neutral, and ground lines so that your light fixture is now plugged into 220 Volt wall socket current.

At the time, I remember saying "that sounds totally insane". I still think it is. But if you made a special map backing board rather than entire walls and ran a lot less power through it....

A good friend of mine works for a major electronics manufacturer (trust me, you'd recognize the name) where some employees exfiltrated 200,000 internal design documents/artifacts to a competitor that they were planning on joining. As a result, no one can use gmail/gchat from work machines or VPNs anymore. I think a MITM proxy that detected giant file exfils but allowed people to send gchat messages to their family would be a huge plus in that environment.

It is a bit astonishing what lengths the go community is willing to, er, go, in order to avoid compile-time code execution (the basis of lisp macros) or generics. I understand that neither of these ideas were invented at Bell Labs during the 1970s, but at some point, can't we just put aside the not invented here syndrome?

Could we all just agree to pretend that Bell Labs folks invented those ideas? Would that work?

Literally every single C programmer in the world is writing x86 or arm machine code. C programmers have this bizarre blind spot where they don't realize that though. Isn't that weird?

I'm not sure about that; the paper talks about parallel machines that can search out many keys at once. Its focus is on known plaintext attacks against AES, but it seems like the techniques described could apply to IFP for RSA.

I'm not a cryptographer so I could be totally off base here.

Is it really $10M per key? DJB points out that if you're willing to spend a lot of money building parallel hardware, you can get some surprising performance benefits allowing you to crack large numbers of keys at once.

http://cr.yp.to/snuffle/bruteforce-20050425.pdf‎

What can Snowden promise them, anyway, that they would make this deal?

They don't know exactly how much he has and the government has some interest in securing the data that he hasn't released.

Is this true? I thought Clapper knew about the questions in advance, which means he could have easily asked committee staff to only ask those questions in closed session. Congress is usually very deferential about those sorts of issues.

And what happens when someone whom you voted as shows up to vote? I mean, if this is going to happen on a large enough scale to affect the outcome of an election, that event has to happen many times. Which means poll workers and police are going to record many cases of voter fraud.

So, can you show me an election in the US where that has happened? Even just one?

Or is this a fantasy?

The point is that if each fraudulent voter has to go to multiple polling stations, that dramatically increases the cost and raises the likelihood of detection. In order to scale that attack up to be practical, you'd need buses full of fraudulent voters going from one station to the next.

I think you should do some research on voting in the US. You can start by looking at web pages like this: http://www.sec.state.ma.us/ele/eleifv/howreg.htm