celoyd

Done. Thank you!

It’s based on the example for 2-pass encoding, with tweaks for things like QuickTime support: https://gist.github.com/celoyd/90cc919105248a2ddc95

I know it’s odd, but I wanted it not to be visible all at once.

Oh, hello! I enjoyed your motion interpolation experiments, and I look forward to the blog post.

Is there a good public reasonably live feed of this data?

Right here: http://himawari8.nict.go.jp

Also, it'd be nice to be able to pause it and really take in the colors.

You can zoom in on the NICT browser: http://himawari8.nict.go.jp

Their color mix is more cloud-oriented, so it’s darker, but at least the image quality is higher than the video’s.

Sorry. If there’s one thing I’ve learned from this project, it’s that video support is hard.

I’m happy to take advice (preferably in the form of ffmpeg flags).

Well-worn gray: https://en.wikipedia.org/wiki/Albedo

It’s a loose comparison, since both asphalt and the moon’s surface vary. Plus, there are tricky things like specular reflection and the opposition effect.

How can its color be compared to something on earth? Doesn't the (lack of) atmosphere have some effect?

You can hold a moon rock up against something on Earth. Or you can look at Earth and the moon from a distance with the same sensor, which is what Himawari-8 is doing.

The atmosphere does affect things: A huge plain of moon rocks on Earth’s surface would look bluer (hazier) than the moon would in the same picture from Himawari-8 (or an Apollo Hasselblad, etc.). And a moon rock in sunlight on Earth will look slightly yellower than one in the same light on the moon. But your eyes are constantly adapting to the surrounding illumination. Color perception is really complicated.

Casual readers may not get your reference – the idea for DSCOVR literally came to Al Gore in a dream. I still think this is hilarious.

One can indeed argue that DSCOVR has merit, and people including climate scientists do. (Though arguably it’s really too early to judge either way until there’s time for publications to come out of its data.) Earth’s radiation budget is surprisingly underdetermined, considering its importance.

But it’s certainly a fair criticism of DSCOVR as launched that it’s not state-of-the-art. That’s because it was delayed for reasons of politics, only loosely disguised as reasons of science.

The only places I’d push back on your remarks are:

1. The “Uh no”, which is unsubstantiated (perhaps just facetious, but I’m sensitive because I supported my point relatively laboriously, with the links); and

2. the idea that the NYT should refrain from pointing out political interference in science funding.

I think they’re just responding to my joke with a joke ;)

I made this, and I’m happy to answer questions. (And to politely ignore contentious remarks about how I characterized GOES-R.)

Discovery at Udvar-Hazy is why I started this project. I saw the serial numbers on the tiles – almost close enough to touch on tiptoe – and remembered reading about some kind of database. I imagined a visualization with a line for every tile over time, and decided to find the records to make the viz.

That was two years ago. I’ve asked a lot of people since then, but after making a web site for the project (on @doingitwrong’s advice), this week is by far the most attention it’s gotten. I’m hoping this is what finally gets word to someone who knows something.

How odd. I just switched fonts entirely (though still from Google Fonts). If anyone has a sense of what’s going on here, I’d love to fix it properly.

The environmentalist threw a fit about this "glue" and it was replaced with a glue that all the engineers said was not as safe, in regards to the tiles staying attached...

This is a common myth. It’s false: http://mediamatters.org/research/2005/08/09/limbaugh-promote...

Fixed (I hope). Thanks for reporting.

In the images in the blog post and the live map? Those aren’t pansharpened at all. If we do add pansharpening in a later version, it’ll likely be naïve, without spatially aware modeling of the multispectral data. (Specifically, it’ll probably be a cleaned-up, rasterio-based, null-aware, parallelized descendant of this sketch of the Brovey transform in numpy: https://gist.github.com/celoyd/2e7beed82951d22b9b90 .)

From what I’ve seen – and I haven’t tested it carefully yet, so I could be wrong – the more elaborate methods are severe overkill on Landsat 8. It has only 4 pan px per multi px (where some commercial data has 9 or 16), and the pan band is almost exactly R+G+B (without NIR). So my gut and some simple experiments suggest that doing PCA-or-whatever is overthinking it.

I wrote that Mapbox post, and helped with some of the processing that happens to get this data on AWS. Yep, you get multispectral resolution of 30 m, and with pansharpening (http://en.wikipedia.org/wiki/Pansharpened_image, etc.) you can get visually acceptable quality at 15 m in RGB.

Landsat is basically intended for science about seasonal/annual/decade-scale changes in Earth’s land surface. When you see an estimate of how a city’s built-up area has grown since 1980, or how the Everglades are changing, it probably has Landsat as one source. This explains a lot of design decisions that might seem weird to a layperson who wants to use it for everyday RGB imagery. Most use of Landsat imagery is basically off-label. It’s just very good data in terms of accuracy, precision, and general ease of use. And if I say so myself, it looks real pretty: https://www.mapbox.com/blog/landsat-live-live/

it could be in the order of magnitude of 1.8cm per year at some places of Antartica and Canada.

Here’s a visualization of elevation over time recorded by a GPS observatory at Baker Lake, Nunavut, relatively near the fastest modeled rebound in Canada: http://www.sonel.org/spip.php?page=gps&idStation=2407

By eyeball, that’s 0.08 m over 2003–2011, or just about 1 cm/year. (Notice the tabs on the bottom: the x and y velocities also show clear trends. That’s continental drift. And when the news says something like “the earthquake permanently moved the ground almost two feet south”, the data comes from observatories like this one, among other methods.)

And the third microsat company with operations coming online soon is Planet Labs: http://www.planet-labs.com

Robinson Meyer at “The Atlantic” did a nice overview two months ago: http://www.theatlantic.com/technology/archive/2014/01/silico...

It’s an interesting time in remote sensing.

The video field of view is 1.1 by 2 km: http://skybox.com/uploads/10/08/imageryandvideospecsheet.pdf

Purely speculating, not going on anything they’ve said: theoretically when they have more satellites you might be able to patch together a larger area with simultaneous video from several birds. You’d have a slightly ugly angle difference at the seam, of course. There may be technical constraints here that I’m not thinking of, though.

In any case, at 1 m GSD and 30 fps, 1.1 by 2 km is 66 megapixels a second. Even efficiently compressed, that’s a lot of information already.

Exactly. This is why distant city lights often twinkle even more than stars do: there’s more atmosphere between you and them than between you and the stars.

Another effect here is that Skybox’s video uses the pan band, which includes infrared to 900 nm[0], and smog is usually pretty transparent in NIR, depending.

0. http://skybox.com/uploads/10/08/imageryandvideospecsheet.pdf

And a third important factor is that smog is very diffuse (by definition: otherwise you call it a smoke plume), so if you have the bit depth you can just increase contrast until you get a good image.

(I work at Mapbox on satellite imagery, but wasn’t involved with this particular blog post; what I’m saying here is stuff that people in remote sensing Just Know.)

I took a break from dealing with an in-band null problem in image data to check HN, and here’s this. Oof.

Don’t represent missing data with a value in the same domain as real data.

Maybe I’m just grumpy, but that seems like a clear, persuasive, and generally realistic goal. In cases where you’re constrained to break that rule, at least approximate it. For example, if you’re stuck using strictly floats for a percentage like this one – without recourse to an undefined/null/none type – you can still use -1.0 or Inf or NaN and not draw the lines.

I’ve worked with NASA and USGS a fair deal both for fun and professionally, and this is typical.

They are extremely competent and sincerely want you to have good data. They are also hampered by the bureaucratic limits of any large organization. So it’s like working with a large, well-run business that’s hired a lot of the best people in its field and is working on good problems, yet is large enough that it can’t move to meet the exact needs of any one customer.

But on top of that there are political concerns. They have an institutional fear that a congressperson in a budget debate is going to stand up and say something like “And apparently we’re paying the Geological Survey $N million a year to run a web server for something called geotiffs that tell you how tall hills are!” That’s my impression from reading between the lines, anyway; no government employee I know has been indiscreet enough to deliberately hint at such a fear.

For example, the best interface to SRTM isn’t from the agencies that made it, it’s a single-page project from Derek Watkins at the NYT: http://dwtkns.com/srtm/

Working with NASA in particular feels like working with an industry leader that has a mysterious policy against advertising, or even going out of its way to help you find resources. (Individuals do, but not the organization, at least not anywhere near in proportion to the number and value of its resources.)

NOAA too: they have some amazing satellite imagery that’s public domain, but they simply do not have the budget to do anything but the most halfassed job of hosting, publicizing, and documenting it, because from a funding perspective that’s frivolous. They barely archive their images, because no one with budget control gets why a weather agency should save its input data. Look up “VIIRS granule” – that’s technically open data, but yikes.

The resources are there, and if you make the effort to figure it out, the people who manage them are pretty much all a delight to work with. But you have to deal with the damage created by a political culture that too often treats our civilian space and geospatial agencies as afterthoughts rather than as highly multiplied public goods.

I’d be interested to know the ratio of the L0 data size to the L2/3/4 data[0] size – in other words, how many downlinked bits of sensor readout are processed per bit of useful science data? Of course that won’t be a hard number; I’m just curious to get a sense of the general scale on which the pipeline reduces pixels to physical parameters.

0. http://en.wikipedia.org/wiki/Remote_sensing#Data_processing_...

Yet until a credible leadership figure actually _offered_ unconditional surrender, it was going to be war as usual.

But it wasn’t. The A-bombs were quite unusual. They were extraordinary weapons, and would have required extraordinary justifications to use on the military, let alone on civilians.

(To the argument that it wasn’t possible to target the military apart from civilians: that’s exactly why nuclear weapons are so problematic.)

it deliberately ignores both the wide diversity of opinion which existed within Japanese leadership

An alternate interpretation is to not take it alone, and to see it as deliberately summarizing the wide diversity of opinion which was already going through the complex process that you rightly point out. News of Nagasaki’s destruction interrupted a meeting by the Japanese Supreme War Council, not so much about whether to surrender as about how. I don’t think it’s facile at all to call this, in context, “ready to surrender”.

The US did not know the details of the meeting, of course, but they chose to spend 70,000 human lives on an assumption with no particular justification, while under no special immediate military threat – at that point, Japan was relatively contained, and the fear was that if left alone it would reach out again in a matter of months or decades, not weeks. The threat was diplomatic: that Japan might surrender to the USSR. That seems worth presenting as something other than the rueful conduct of justified, proportional warfare as an absolute last resort.

As an American, sometimes I look at other countries that haven’t come to terms with major war crimes – the Armenian Genocide, for example, or indeed Imperial Japan’s crimes in China – and I shake my head.

Then I remember that the US has the A-bombs on our record, and we’ve made it almost 70 years now without admitting as a nation (meaning in standard textbooks, in mainstream entertainment, etc.) that these were, by any sensible definition, crimes against humanity of the highest gravity.

There’s this objection people sometimes raise: that it’s easy to say these things if you weren’t there, if you weren’t under the pressures of command, knowing your nation’s people were dying by the day. That’s true. I don’t know what that was like. It’s easy for me to moralize from a distance. But that argument isn’t actually justifying the act of the A-bombings, merely moderating judgment on the people who did it. Saying they didn’t really know what they were doing is very different from it was a good idea.

a clever, possibly elegant, algorithm

It’s just barely clever enough that no one thought of it before. It has to be really simple, because the innermost loop (estimating how cloudy a given pixel is) runs several trillion times.

using tools anyone could use (aside from a massive server farm)

EC2’s m2.4xlarge (> 64 gigs of memory, 8 decent cores) is going for $0.14/hour right now. You could run ten of those all day and spend less than on dinner at a restaurant with cloth napkins.

Heh, yes, we seriously considered the Honda Civic full of external drives option.[0]

Some of our images were from NASA’s hot new GIBS[1] service, which was extremely fast. However, it’d only backfilled about 2/5 of the data we needed, so for the rest of it we were using a legacy endpoint that was launched around 2004ish and has some, let’s say, idiosyncratic caching and throttling systems.

Instead of setting up a special channel, we talked to them and figured out how to shotgun the downloads in a way that wouldn’t kill their cache or require them to give us special treatment. We thought of this on a Friday and wanted to have it ready when we came in on Monday, and it worked. Thus the somewhat blunt methods.

0. For any young persons in the audience, the reference is to “Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.” from Tanenbaum’s “Computer Networks”.

1. https://earthdata.nasa.gov/about-eosdis/system-description/g...

Blue Marble was a big inspiration here, and we’ve had fun comparing notes with some of the people who made it. Blue Marble is, however, ten years old, and has a number of (justifiable) limitations. You can get a sense of these if you zoom in on its depictions of, say, the Niger Delta, or the northern tip of Greenland.

This release is down to zoom level 8, which is roughly 500 m to 1 km/px – what I think of as regional rather than local. You’re seeing mountain ranges and river valleys, not what you have planted in your garden.

Right now with this project we’re thinking about how to serve people making small- to medium-scale maps on topics like global environmental issues, shipping, travel, etc. At higher zoom levels (higher resolutions), our more conventional satellite and aerial imagery seems to be meeting people’s needs for now.

Oh, oh, I like doing these analogies, and we badly need AWS stickers for the office.

A typical grain of beach sand – which must be about the lightest solid thing that’s easy for most people to envision – is 3 mg. That many grains of sand would mass 6e6 kg (6000 metric tons), which in turn is near the upper limit of masses that make sense to most people in everyday terms.

So you could say that if every S3 object were a grain of sand, S3 would be 3× the launch mass of the Space Shuttle, or somewhat more than the gold held in Fort Knox, or as much as the heaviest living thing: http://en.wikipedia.org/wiki/Pando_(tree)

And the the 1.1 million requests per second would be about 3.3 kg, or as much as a gallon of milk/water.