eref

I do not get what makes this post so popular. Google Maps is often better and uses automatically extracted features from photogrammetry. Eh. 350 points at most.

For me on Firefox on macOS that is just a couple of key strokes away:

  cmd-l cmd-c ctrl-a r <space> u r l : <enter> cmd-t cmd-v ctrl-a h n <enter>

[1] https://hn.algolia.com

Why not both?

.

.

The term "circuit" is used in neuroscience all of the place.

Capsules basically do a kind of self-attention. But there the parent features compete for a coupling, not the child features.

Same with 57.0 on macOS 10.12.

Edit: I have FF Studies disabled under about:preferences#privacy. I guess that is the reason why it is not installed on my machine.

The new XPS 13 has a better camera position and a good touch pad but still not a great battery. It’s close. Maybe next year.

simulate a physical environment very fast

That's probably only a problem if it is must faster than everbody else.

let alone faster than what happens in our environment

That is often not very hard. When a bottle rolls off the table, you can catch it by approximately predicting it's trajectory without computing the precise evolution of the ~10^26 atoms that make up the water bottle. Compression is a corner stone of intelligence. The second corner stone is using compression to choose actions that maximize expected cumulative future reward.

The question is whether the overall gains outnumber the local losses.

The main problem is that we still lack good generative models and good ways of interrogating them. GANs are unstable and difficult to apply to time series, VAEs suffer from posterior collapse, WaveNet/PixelRNN grow with the input size and overemphasize the details, RNNs are hard to train because we lack good training algorithms. Generally, small errors tend to compound in step-wise predictions because NNs do not generalize very well and gradients tend to vanish and shatter. If you just regard computation time to roll out the future, modeling domains in which the rules are simple enough to be hand-coded and evaluated quickly (such as Go and Chess) probably makes MCTS a million times more suitable compared to domains in which you need a complex model.

But what does have to do with smoothness and translation invariance which this paper is a demonstration of? You even learn Gabor filters with local connectivity without spatial weight sharing.

What do Gabor filters have to do with this?

I think that can be mainly attributed to the fact that the last few deconvolutional features are overfitted to features in the image and are somewhat robust to noise. The network does not even learn features to produce e.g. white noise as output. This is probably much less magical than the paper makes it seem to be.

I think it is right to „soft“-censor in this way (i.e. the information is accessible but not promoted). Reality without sorting and filters would be inhuman because market incentives to grab people’s attention would maximally exploit our tribal instincts and thereby completely destabilize our societies. This is already happening and it needs to stop immediately.

Maybe I misunderstand stoic philosophy, but the rule to override emotional responses to failure with serenity strikes me as overly broad. Sometimes it is simply easier to get angry and let it out; especially if that sends a valid corrective signal to someone who is at fault. And even if the failure was just bad luck, it seem totally fine to be mad for a couple of minutes or hours. It is simply the way our brains have evolved to work and that's fine as long it is not harmful as long as nobody is harmed. Just be careful to not turn it into a constant state of anger because that will likely worsen your life rather than improve it. All of that can be subsumed under rationality.

High representation efficiency

Capsules do require much fewer parameters, they generalize 10-20% better to new viewpoints, they are much more robust to adversarial examples and can better recognize overlapping objects; but, on the other hand, capsules currently require much more training data to achieve the same performance, even though in theory (if they would actually learn inverse graphics) they should require less data, and they add a lot of expensive additional structure (roughly 10X). I am rather pessimistic about whether the approach will lead us anywhere; it seems sub-optimal to model all possible child-parent configurations explicitly. That has a quadratic nature to it and my hunch is that can be done sub-linearily.

I might be wrong, and I don't want to question the possibility that someone from a non-technical domain can contribute to technical domains (in particular because neural networks and style transfer are basically high school mathematics; and since this happened before [1]), but it seems to me that the paper in question does not contain much technical innovation, but it is rather a demonstration of how existing techniques can be used in practice [2].

[1] https://en.wikipedia.org/wiki/Hedy_Lamarr

[2] https://arxiv.org/abs/1701.04928

I think that is correct because otherwise they would have mentioned it as an outstanding feature of the model. It does require fewer parameters than a CNN to reach the same accuracy.

The upgrade to 10.13.1 has effectively locked my SSD in read-only mode because reading from it would always cause an APFS kernel panic (both under 10.13 and 10.13.1) also making it impossible to boot from it. It took me several hours to figure that out and to restore my data. I now downgraded to 10.12.6 and I am not missing anything. I think I am going to switch to Ubuntu as soon there is a comparable laptop to MacBooks from ~2015 (perhaps in 1-2 years?). This kind of nonsense is just not worth my time and money when I just need Firefox, ZSH and Emacs anyway.

Skimming the two papers I could not find any figure about data efficiency. Did you?

I was thinking that the next layer in the network would respond to multiple samples (i.e. convolutions of the Gaussian at different positions) and, as long as you didn't have too many active neurons on the previous layer, it could extract a measure of position.

I think, it would still be a very "blunt tool" for feature detection. If you are going to compute weighted sums in a convolution anyway (as opposed to just summation in avg pooling or maximum search in max pooling), then the question is really why not simply learn arbitrary feature detectors instead of fixed Gaussian kernels? You can separate Gaussian kernels in x and y direction, which allows you to compute it in 2 * N^2 * K + N^2 instead of N^2 * K^2 operations (with image size N and kernel size K), but in practice, that probably won't give you enough improvement to make up for how few bits of information a Gaussian filter can extract. You would also need to use a very strong sparsity regularizer to get few enough active neurons in the previous layer such that that multiple Gaussians can infer a location. I am not entirely sure it would not work, maybe it is worth a try.

If you have too many active neurons then, as you say, you encounter aliasing effects, but I think the same is true with capsule networks - they're not expected to handle particularly high-frequency features, are they?

That is a very good point. In neuro lingo, this aliasing is called "crowding". A multi-channel filter kernel (as in standard CNNs) can in principle deal with that by learning filters for representing multiple entities in different spatial configurations within the receptive field, but that requires large amounts of filters and spatial codes which are also not trainable very well in CNNs. Capsules can indeed only represent one entity within their respective receptive fields. I think, capsules fail more gracefully in case of crowding than standard CNNs because the agreement detection can decide on one out of multiple objects being predicted by the capsules below.

You'd also lose most of the information. If there is only a single active neuron among the inputs to a Gaussian kernel neuron, you would at least have info about the distance of that to the center of the receptive field, but no directionality. If there are multiple active neurons among the inputs, you'd lose most distance-to-center info. Basically imagine avg pooling as spatial downsampling by box filter or surface area integration, and Gaussian pooling as downsampling by Gaussian filtering.

The same problem occurs with avg pooling. Strided conv also allows to "pool" neurons in the layer below to reduce the number of neurons in subsequent layers, but, in practice, deeper neurons then also have trouble learning precise representations of the locations of the things below (but much more info is retained compared to avg/max pooling). Capsules can presumably learn such things much more accurately because they can, in principle, learn precise geometric mappings to infer positions independently of the viewpoint. However, the results so far are not much better than scalar output neurons. Capsules do perform a bit better in terms of robustness against adversarial examples and overlapping objects.

You can maybe find a alternatives here: https://docs.google.com/spreadsheets/u/1/d/1TFcEXMcKrwoIAECI...

It is a great opportunity to shake up the sphere of extension development a bit. New alternatives pop up every day and eventually the best will stick. Here is a list: https://docs.google.com/spreadsheets/u/1/d/1TFcEXMcKrwoIAECI...

I think it is a pretty unnecessary sentence. 81 comes from the 9x9 kernel size. It is obvious that those will overlap despite of the stride of 2. Maybe they mean the projective field.

Neither is there for reproducible code, but now there are artificial incentives such as the Reproduction Challenge.

I wished more academic papers would have a "Limitations" section. Often it is so hard to tell what these are, e.g.: Does it scale well? How much faster/slower is it than other methods? Does it converge reliably?