Edge detection doesn’t explain line drawing

https://aaronhertzmann.com/2020/04/19/lines-as-edges.html

by KqAmJQ7 • 3 years ago

246 167 3 years ago

cheschire2• 3 years ago

I have a different hypothesis. I think line drawings are the representation of vectors. The details we focus on are the vectors with the highest magnitude of change, but that change is not always visual. For example, momentum often has an outsized representation in line drawings. Edges can be a high magnitude change as well, but it's not the only thing.

cropher• 3 years ago

Interesting; can you elaborate? Also could you clarify what you mean by changes that are "not always visual"? How would non-visual information exist in a purely visual medium?

cheschire2• 3 years ago

The example I mentioned was momentum. That's not visual information, that's extrapolated information of position over time. It can be represented in line drawings as motion lines in comics, for example. Interestingly, by simply implying motion, I hypothesize that the brain deprioritizes processing detail on the object that is implied to be moving, and focuses instead on the interactions that will follow.

If I were a researcher, my contrived test of this would be to simply have people recreate drawings of "static" objects, and have others recreate drawings of objects implied to be in motion.

Other non-visual information would be emotions. The shape of eyes and mouth lines are highly critical to passing emotion. I suspect that people's interpretation of emotion directly impacts how strong the emotional representation of those parts of the face would be drawn. For example if a test subject is told to draw the face of a model in front of them, but they are told the person is experiencing an emotion, I hypothesize that the group of people who are told the person is happy would more frequently bias their interpretation of the eyes, eyebrows, and corners of the mouth towards a "happy" representation than those who are told the person is experiencing great inner turmoil.

To be clear though, I'm not saying we only draw based on non-visual information. I'm saying the sum total of all vectors has an influence on the drawing. Colors, in my opinion, have as much of an impact as edges. And it would be interesting to compare the drawings of a person with less common color sensitivities to more common color sensitivities.

plaidfuji• 3 years ago

You’re on to something here. In the past I did a bunch of work on extracting line drawings from images, and the fundamental goal was always to vectorize the lines - then you get an abstract representation of the figure that you can scale up or down.

cnity• 3 years ago

As someone (probably like many here) who graduated from a university which taught computer vision and peripheral neuroscience courses, with such titles as "Computational Neuroscience of Vision", I always felt that trying to understand the human brain as a kind of algorithm was a bit of an artefact of computer scientists as they approach biology.

The truth is the visual cortex is vast, and not sufficient to explain the human classification and perception of objects visually. Never mind individual neurons or edge perception. Edge detection is an interesting isolated example for study and learning, but you will never come close to explaining human recognition and cognition in such simple terms.

hprotagonist2• 3 years ago

ah, someone who paid attention in lecture!

(incidentally; there’s a fairly deep literature of historians of science that have carefully documented that we describe ourselves as analogous to the most sophisticated technology of the day: see “to lose one’s temper”, “to blow a gasket”, “i got my wires crossed”, “sorry, cache miss”, … as metaphors and idioms of mental state through the centuries that reflect the cool tech of the time in which they were coined )

pavlov3• 3 years ago

And also the universe itself is often seen through the lens of contemporary technology.

Are we living on an island that floats on a giant turtle’s back? Or are the heavens like giant clockworks? Or maybe it’s all a computer simulation?

These cosmological speculations are separated by thousands of years, but they are all simply a reflection of what the person finds most awe-inspiring in their everyday life.

adolph2• 3 years ago

I think in the case of “to lose one’s temper”, there isn’t an obvious match between the technology of the day and the early medical theory of four humours, not of metal-working.

Origin of Temper, New Oxford American Dictionary:

Old English temprian 'bring something into the required condition by mixing it with something else', from Latin temperare 'mingle, restrain'. Sense development was probably influenced by Old French temper 'to temper, moderate'.

The noun originally denoted a proportionate mixture of elements or qualities, also the combination of the four bodily humours, believed in medieval times to be the basis of temperament, hence temper (sense 1 of the noun) (late Middle English). Compare with temperament.

https://en.m.wikipedia.org/wiki/Four_temperaments

ben_w2• 3 years ago

to lose one’s temper

Huh, never thought about that one before.

Linguistic stuff like this is fun to find; these days I mostly spot it via learning German as a second language, so the artifice in artificial intelligence becomes “Künstliches Intelligenz” where “Kunst” is artist and “Kunststoff” is plastic, and in Middle Low German “kunst” is knowledge and ability.

coined

Deliberate choice to exemplify your point, or accidental because it’s almost impossible to avoid examples like this in modern English?

HPsquared3• 3 years ago

Now the technology is starting to reflect the biology (neural networks). Inception!

dylan6043• 3 years ago

as metaphors and idioms of mental state through the centuries

after listing a bunch of things from the previous century that barely stretches a bit further back. i would have been impressed if you had come up with a phrase from medieval tech, or roman/greek/egyptian. hell, i'd settle for pioneer days tech to allow for "centuries". otherwise, it just feels like modern day analogies.

detourdog10• 3 years ago

I think it is more that as technology grows it spreads its terms and contexts to the point of entering pop culture. I'm not sure if the populous uses terms simply becuase they have heard them before in the same context or due to an understanding.

I have to point out that P.G. Wodehouse is often used as an example of this style in recent "literature". I can't even figure out the words to describe the sources of his terms. Wodehouse use terms from anywhere in English language culture (including French I think). The odd part about it is that Wodehouse's writings are so old I find it easy to miss the references.

I don't doubt we do this but do expect that is no different than my love being as deep as the ocean.

jeepers6• 3 years ago

When my legs give out from below me, I don’t shout “loss of hydraulic pressure!” like some kind of arthropod.

Yet we’re obsessed with framing ourselves as chains of matmul.

wpietri3• 3 years ago

Ooh, that's very interesting. How would I find the metaphors used before, say, the industrial revolution?

kurtoid• 3 years ago

cache miss

Ooh, I like that one. Stealing it

syntaxing• 3 years ago

Especially not purely RGB cameras. There’s a reason why you automatically focus to something that’s moving or fluttering. I think DVS camera would have bridged a huge gap in perception sensing but unfortunately there’s not enough demand for it to scale so most manufacturers dropped it.

jofla_net• 3 years ago

Agreed, There certainly is, in CS, no shortage of engineers cramming square pegs into round holes.

AndrewKemendo2• 3 years ago

Great way to put it

The fields of Cognitive Science would view humans as proto-robots

I'd love to revitalize the field of "cybernetics" because it really answers all of this long ago

jdietrich• 3 years ago

trying to understand the human brain as a kind of algorithm was a bit of an artefact of computer scientists as they approach biology

I think the advances in neural networks over the past few years have shown that the failure of such an approach was mostly a matter of scale. Trying to reduce the visual system into a few kilobytes of code is of course a fool's errand, but trying to emulate it with ~10^11 parameters is looking much less foolish.

"The brain is a computer" is a stupid analogy if you think of a computer as a scalar or vector machine, but it's much less stupid if you're thinking in tensors.

colechristensen• 3 years ago

The truth is the visual cortex is vast, and not sufficient to explain the human classification and perception of objects visually.

There have been experiments which have exactly located individual neurons and sets of neurons responsible for the first layers of image recognition. i.e. a neuron that fires when a specific spot on a retina is stimulated (a single pixel) and neurons that fire which detect lines. This is not theoretical but actual probing of living brains. I'll find the paper(s) later.

bloppe• 3 years ago

As someone who minored in neuroscience, I took away that edge detection is actually quite important to the way your vision works. Google search "center-surround receptive field of retinal ganglion cells". This happens in your eye, before the signal even enters the optic nerve to go to the brain. The brain itself is not detecting the edges; its input already has that information.

I was also struck by the similarity between the way your cochlea (the organ in your ear that picks up sound waves) functions, and the way a Fourier transform works. They both transform the signal into the frequency domain, but your cochlea does it via its mechanical properties rather than by convolving the signal with a bunch of sine waves.

amelius• 3 years ago

The human brain also triggers on soft features.

It's a combination.

zellyn• 3 years ago

Wouldn't a simple explanation of line drawings be that segmentation of images into shapes is an important part of vision, using whatever information is available? The reason it would be somewhat invariant across color and lighting is that those change so much. (eg. we can see by moonlight or faint illumination, when color signal is absent)

In some cases, simple segmentation fails (like with the shadowed face in the article), and you have to rely on other features.

smokel2• 3 years ago

Line drawings are just easy to produce, historically. This makes them embedded in our culture, and kids learn to read them from an early age.

The same skepticism could be raised towards letters and words.

If we had invented the photo camera before the paint brush or pen, things might have looked different.

smokel2• 3 years ago

Here's a link to the paper that contains the "my hypothesis". The link in the article runs into a paywall.

https://arxiv.org/pdf/2002.06260.pdf

simias3• 3 years ago

Ten years or so ago I was working on a video chip that had an upscaler feature. While prototyping and simulating it, we first started by applying a mathematically-correct (i.e. information preserving) FIR filter to do the upscale. Then we compared the result with other solutions and found that ours looked worse. We asked our colleagues to blind-test it and they all picked third-party-scaled images over ours.

At first we assumed that we must have had a bug somewhere because the Fourrier transform told us that our approach was optimal, but after more testing everything matched the expected output. Yet it looked worse.

So we started reverse-engineering the other solutions and, long story short, what they did better is that they added some form of edge-enhancement to the upscaling. Information-theory-wise it actually degraded the image, but subjectively the sharper outlines were just so much nicer to look at and looked correct-er. You felt like you could more easily tell the details even though, again, in a mathematical sense you actually lost information that way.

I don't think it makes a lot of sense to reduce human vision to edge detection (we can still make sense of a blurry image like this one after all: https://static0.makeuseofimages.com/wordpress/wp-content/upl... ) but it's clear to me from empirical evidence that edge-detection is a core aspect of how we parse visual stimuli.

As such I'm a bit confused as to why the author seems to see this as a binary proposition. That being said, I could just be misunderstanding completely the point the author is trying to make.

JoeAltmaier2• 3 years ago

The difference between the data in the image, and the information? If for instance you upscaled text so large that it became blurry and unrecognizable, you lost information.

Our cortext is all about interpreting what we see. Almost before our brain proper has the data, nerves have begun extracting information (edges etc). Probably because it was the difference between hitting and missing the animal with the spear. Or seeing or missing the tiger in the grass.

simias3• 3 years ago

Precisely! I also find it interesting how, from an information theory standpoint, audio processing and image processing are effectively the same thing (audio resampling is fundamentally 1D image scaling for instance) but because humans process sounds very differently from images we end up doing things pretty differently.

For instance when we want to subjectively make images more attractive we tend increase contrast and sharpness, whereas for sound we tend to compress it, effectively reducing "audio contrast".

avereveard• 3 years ago

Works in games too. Adaptive contrast will increase noise, but after a while games without it will look blurry and undefined.

heavenlyblue• 3 years ago

Surely if you're upscaling pixel art you're loosing information when you create gradients between pixels. It doesn't seem to me that your metric of information loss was ideal.

photonerd• 3 years ago

Quite, no new information so no “loss” but not the information that needs to be there.

It’s putting an 8oz coffee brew in a 20oz cup & giving it to the customer as a large saying they had no coffee loss. While true, it’s not the same as delivering a 20oz coffee.

simias3• 3 years ago

Conservation is not just about preserving info, it's also about not adding information that's not here. If you upscale without those gradients (effectively sharpening to the max with nearest neighbor extrapolation) you introduce high frequencies that could not exist in the original data. You've created new information out of nowhere.

But of course you're correct that in this case it may be the desirable outcome. I still think that this idea of creating information using algorithms in order to get a subjectively more pleasant result is really one of the biggest issues of our time. Not a day passes where I don't see AI-colorized pictures, AI-extrapolated video footage, AI-cleaned family portraits, AI-improved smartphone footage etc...

It's both amazing and a bit scary, because in a certain way we rewrite history when we do this, and since the information is not present in the original it's very difficult to ascertain how close we truly are to reality. We're creating a parallel reality, one Instagram filter at a time. Maybe that's the true metaverse.

Kinrany• 3 years ago

The upscaled image stores more information than the original image, so it must be possible to keep all the information while still doing edge enhancement!

pbhjpbhj• 3 years ago

Stores more data, but the same information, and if there is any interpolation then some of the data is modified, meaning that you lose a little data. In fact even without interpolation I think you change the data.

If you imagine a hard edge that aligns with a pixel [] boundary then you imagine upscaling in various ways, I think it's QED. You change data about the sharpness of the edge.

[] I use an Android phone, with Google's keyboard it genuinely rendered "pixel" with a capital letter. I've never written about Google's device of that name. Silly Google.

CamperBob2• 3 years ago

It's like playing a familiar song on an old cassette tape on a modern system. The user will likely be tempted to crank up the treble, trying to recover content that isn't there. In the image-scaling example, the HF content wasn't there to begin with -- since you upsampled it, it couldn't have been! -- but there's a strong psychological expectation for it to be there.

With images, it's a bit easier to put the high frequencies back in via judicious use of edge enhancement, perhaps because you have two dimensions to work with rather than one dimension in the audio case.

olaulaja• 3 years ago

Sounds like using the wrong metric, the upscaled image should be compared against the original full resolution one, not the downscaled one. Obviously you can't know what the full resolution one looks like when actually upscaling (vs testing), but you can make an educated guess.

willis9362• 3 years ago

You can remove the guesswork. You can start with a high resolution (or even raster export of a vector image for an extreme example), downsample it (various methods and downsampling factors for completeness), then attempt to upscale it.

nestorD• 3 years ago

A lot of smart TVs do that. Being able to spot it (played a lot with edge detection when discovering computer vision) is a curse.

bmicraft• 3 years ago

I would put it like this:

Sharpening increases the high frequencies to cover the loss of even higher ones lost in downscaling.

willis9362• 3 years ago

Yes. Similar to pre emphasis in baseband telecommunication standards in band limited media.

debugnik• 3 years ago

I'm a bit confused as to why the author seems to see this as a binary proposition

The author mentions this twice:

This hypothesis is compatible with Lines-As-Edges, while answering many of these questions.

cameldrv• 3 years ago

I don't think it's just subjective in this case. The theoretical signal processing approach assumes that the signal is band-limited to frequencies less than two pixels wide, and it's not. There are lots of sharp edges that have higher frequency components than that.

Another way of looking it, more along the lines that you're talking about, is that it depends on your error model. The traditional way of measuring error is RMS pointwise in pixels. Doing some sort of interpolation on pixels gives a pretty good result for that. However, another way to look at it is that it may be better to have a positional error, i.e. a particular color or intensity level is in the wrong spot, than to have an intensity/color error, i.e. you have a pixel that has an intensity/color that's not present in the source signal.

This same basic issue was the basis of a big divide in font rendering for many years, where the Mac would render fonts with the exact geometry of the letters, but then anti-aliased, while Windows would use font hinting to make the shape snap to the pixel grid. Personally I thought that the Windows approach was a lot easier to read on a screen, but the Mac approach had the advantage that the geometry of the text would be exactly the same in print as it was on the screen, back when print was something that was important, especially for Mac users.

a2128• 3 years ago

My completely uninformed hypothesis is that it is edge detection, but on a depth map rather than a color map. A figure in the article even shows a situation where a depth sketch succeeds while edge detection on a color map fails

regularfry2• 3 years ago

Kinda both, would be my guess. The brain reconstructs depth from visual information, so where the visual edge can be interpreted as a depth cue, you're faking out that mechanism. It also explains why the pure edge detection image just looks noisy, especially in the hair: most of the edges are effectively incidental colour shifts that don't provide depth information.

jaked89• 3 years ago

Line drawings work because they represent the centers of symmetry for surfaces and volumes. They trigger the same center-neurons that the original shape would.

A hand could appear on you retina in different sizes and orientations. According to distance, the size will grow and shrink. But the center of symmetry will stay the same.

This goes further. There's also a center of symmetry between edges, and higher-level features as well. Our brain has no issue detecting these.

JoeAltmaier2• 3 years ago

There was a science fiction story of a race of aliens that had to draw what they found instead of photographing it. A photograph was flat and uninformative to them, almost completely lacking in meaning. Their science was based on interpretation at the root.

An interesting concept to put in a story anyway.

bschne• 3 years ago

Problem #1: What about all the other features?

Is this necessarily a problem in the argument? If we consider e.g. color, and construct a blurry image that has a color distribution very similar to a real face, but doesn't fire the edge detection in the same way, we can still recognize it as a face. This could just mean that a sufficiently close match to other examples of the same class on any one of the strong "dimensions" of the image our brain processes is sufficient to make us recognize it, no? The author makes it sound like if edge detection is what makes us recognize line drawings, this implies that our brain discards every other feature in attempts to recognize visual input, but I don't think that's a sound conclusion.

CapsAdmin• 3 years ago

This stood out to me as well. If I close my eyes and walk into a wall I will still perceive the wall.

Our various senses help build an understanding of what's going on. It's not like we fail to understand anything once you remove one part of the system.

Maybe this person is trying to say something deeper or more nuanced and I fail to understand the meaning behind it.

bduffany• 3 years ago

The "Problem #2" that they mention also has an easy solution.

If the "intermediate" variable / internal representation is just "the input image but with edges only," then of course you can see internal representations.

When you compute that intermediate variable for the line drawing, it will just happen to behave like the identity function for that particular case. So if you have already filtered out non-edges then the transformation is basically a no-op.

The "types" mismatching as they mention is not a concern because the type is just "image" i.e. a big vector of HSL values or something. Edge detection is just a convolution filter so it's going to have approximately the same type as the input.

armoredkitten• 3 years ago

Yes that stuck out to me as well. The author seems to be setting up a straw man, as if people are arguing that the brain can't distinguish between a line drawing and the real thing (or a line drawing and a photograph).

The other information isn't necessarily discarded. It's just used to identify that this is a line drawing and not the real thing. It's still remarkable that just the lines themselves (I make no claim as to whether it really is edge detection or something else) are still enough information to be able to identify the representation, but it doesn't mean the brain is discarding the other information.

coldtea3• 3 years ago

A classic answer to this question is what I will call the Lines-As-Edges hypothesis. It says that drawings simulate natural images because line features activate edge receptors in the human visual system.

Why does the explanation need to go to that direction?

Line drawings have a liking to the thing being depicted.

They're a crude representation of it (compared to a photograph or a photo-realistic oil painting) but are nonetheless a represenation.

That's why they work, in the sense of people understanding what they show: they share similar patterns with the things being show. And we are pattern matches.

In this case the patterns are edge patterns, but it could just as well be non-edge patterns. Imagine a color drawing of a human face, where the ink drawn edges have been removed, and it's just blocks of color for the head, the eyes, the pupils, the mouth, the nostrils, ears, etc. We could still tell it's a face, even if we applied some gaussian blur to those blocks.

The most basic statement of the problem with Lines-As-Edges is that the human visual system isn’t just an edge detector. You can see colors, you can see absolute intensities. You can tell the difference between a thin black line and the silhouette of a dark object against a light background; we have both kinds of receptors in the primary visual cortex, as well as others. Yet Lines-As-Edge supposes that the vision system discards all of this other information present in an image, for just this one special case. Why?

Isn't this taking things backwards?

It's not the vision system which "discards all of this other information present in an image" in our regular operation.

Rather, it's the line drawing with does away with (discards) all of this other information and only focuses on a thing's edges.

In other words, our visual system has capabilities A, B, C (say edge detection, color, 3d placement, etc). And, a line drawing gives it only A - which is still enough.

When our visual system also gets B and C, it can perceive objects even better. But for merely identifying something, A is apparently enough.

What I described here is totally compatible with Lines-As-Edges hypothesis, and makes sense too, so I don't see where the author's issue is, and why he thinks the fact that "the human visual system isn’t just an edge detector" invalidates the lines-as-edges hypothesis.

Now you get a sense of the color of the object, and not just its outlines. How would one generalize Lines-As-Edges to account for these different types of depiction? The visual system is no longer ignoring everything aside some gradients; it’s now paying attention to some colors (and not others).

Yeah, so? It just means that the visual system can work with less or with more (and multi-type) information.

Lines-as-edges is a hypothesis for why line drawings "work" (are recognizable as the thing). The hypothesis doesn't say however that edges are the only thing the visual system can understand.

So, there's no need to "generalize Lines-As-Edges to account for these different types of depiction".

Lines-as-edges explains how line drawings are understood, period.

Line drawings with shading and color, add additional information, aside from the edges.

That's fine: no proponent of Lines-As-Edge ever said that the visual system only works with edges. Just that it works with edges when interpreting a line drawing which only offers edges.

empath-nirvana2• 3 years ago

People are so used to seeing images that it's impossible to recognize that they're an _illusion_. If you see a drawing of an apple, it is colored pigment on a page, it's not an apple, but it's _impossible_ to look at it and not see an apple.

_Why_ do we recognize it as an apple? It's certainly not an exact duplicate of the light rays that would enter your eye from a real apple. How different can it be from an apple and still be recognizable? What exactly is the mechanism by which it triggers the recognition?

Calling an image a "representation" or saying that it "has a liking" to the real thing is sort of begging the question. The question to be answered is: "In what way is it a representation? And how does it have a liking to the thing represented"?

coldtea3• 3 years ago

Calling an image a "representation" or saying that it "has a liking" to the real thing is sort of begging the question. The question to be answered is: "In what way is it a representation?

Represenation or having-a-liking is not some hazy notion though. It means there are things in our drawing that map to how those things are in the actual thing. And there are: edges, proportions, shapes.

But it's not some close mapping of edges in the literal sense. That is, a drawing doesn't have to follow the actual edges of the thing depicted with any accuracy for it to be recognized as such.

E.g. we could draw a stick figure instead of a detailed line drawing of a person, or a highly stylized "child drawing" style house, and their edges would look nothing like the edges of the real thing's. But those line drawings would still be easily recognizable.

henry_pulver• 3 years ago

This is fascinating!

I remember reading that most optical illusions don't work on people raised in remote tribes in the Amazon, as their visual perception has been 'fine-tuned' for jungle contours, instead of the straight lines in the west.

Is it possible that we _learn_ how to perceive line drawings in our early years?

lacrimacida• 3 years ago

I think so and this is true about music as well as other arts.

wzdd• 3 years ago

I don't understand the essay at all, perhaps because I'm not a domain expert. Edges are clearly a very strong signal in the visual system -- as the article points out, edge detection is one of the first things that happens in the visual pathway from the eyes. However, edges are clearly not the only signal, as the article demonstrates with cross-hatching, colour, and so on (and is also obvious to presumably all sighted humans).

If you remove edges as a signal -- say, by taking the coloured apple in the article, removing the lines, and blurring the colours around the silhouette -- you'd probably still recognise an apple, but not as quickly. For the same reason, if you defocus your eyes (or take off your glasses -- the popularity of these is a strong indicator that contour matters), you'll have more difficulty navigating, even though many signals (colour, shade, depth) are still present. Clearly edges are important, but are not the only thing we're working with.

Optical illusions also don't invalidate the hypothesis, because they almost by definition rarely occur in nature. Similarly, objects in extreme shadow don't invalidate it either, because, e.g., we are quite capable of recognising half a lion as a lion.

I think possibly there's a difference in interpretation here. The claim the author has an issue with is (his phrasing): "the lines in a line drawing are drawn at natural image edges, where an edge receptor would fire. These lines activate the same edge receptor cells that the natural image would. Hence, the line drawing produces a cortical response that is very similar to that of some natural image, and thus you perceive the drawing and the photograph in roughly the same way."

But the paper the author links to doesn't say that. It says "The likely explanation is that lines trigger a neural response that has evolved to deal with natural scenes." It's not claiming that line drawings and photographs are perceived in "roughly the same way", only that we evolved to recognise outlines and line drawings are outlines or at least contain outlines.

So problem 1 (what about all the other features), problem 3 (what is the benefit), problem 4 (visual art isn't just line drawings), and problem 5 (edge detection is not a line drawing algorithm) don't seem to really address the hypothesis.

That just leaves problem 2, "we can't see internal representations". I'm not sure what the statement is here. The author writes "The idea is that we have neurons that activate for object contours and similar, and that line drawings directly activate these neurons. Lines-As-Edges is a special case of this hypothesis. I don’t understand this claim at all." But that claim, as a hypothesis, seems very reasonable? The author seems to be saying that you can't bypass all the neuronal machinery to get directly to the contour-recognising neurons. But that's not true if all you're bypassing is the stuff to isolate the feature of interest (edges, in this case).

Nevermark• 3 years ago

The article sheds doubt on edge detection as being the major reason why we can interpret drawings as what they attempt to portray, because drawings are often missing shading and color.

Drawings also often drop a great amount of detail, and proper proportion information, and remain recognizable.

Think of cave paintings or Picasso’s bull.

But if you consider image interpretation as a competitive classification, then missing information doesn’t present a problem.

If color, shading, detail & proportion are missing, then they are missing from all possible pattern interpretations equally. That leaves the final classification problem relatively unchanged despite all the missing info.

EDIT: in fact, if my hypothesis is true we should be able to see patterns with even less information!

For instance, dropping all internal detail, and even most shape. As in seeing a face profile on the side of a cloud that otherwise looks nothing like a human head.

Or dropping edges completely. Perhaps seeing an object where there are only stars creating points of light.

Please post your anecdotal experiments at 404experimentreports.com!

cvoss• 3 years ago

The brain's ability to do "shadow removal" is really impressive. You can see it happening in the final example depicted in the article. I think this ability is what was used to explain "the dress". It also comes up in a classic optical illusion [0] where a checkerboard of light and dark gray squares has an object sitting on it and casting a shadow. The shadowed light gray squares are exactly the same shade as the unshadowed dark gray squares, but if you point this out to someone, they'll have a really hard time accepting it because they are so good at unconsciously accounting for shadows.

[0] https://en.wikipedia.org/wiki/Checker_shadow_illusion

aftbit• 3 years ago

Hey thanks, this one is new to me. 100% it is impossible to convince myself that those two squares are the same color.

I have a camera looking over my driveway with motion detection that triggers all the time because of the shadow of a tree waving in the wind. My current weekend project is to use FastSAM to detect cars / animals / people in the driveway instead of just looking at a threshold of changed pixels.

1: https://github.com/CASIA-IVA-Lab/FastSAM

thehappypm• 3 years ago

It's because the squares are not the same color.

The alternating squares are different colors... in absence of a shadow.

Your brain is correctly adjusting for the shadow to correctly identify the "true" color.

mistercow• 3 years ago

Are there any studies on what, if any, animals respond to line drawings (and to what extent)? In particular, do chimpanzees and other apes closely related to humans?

It seems somewhat plausible that “lines-as-edges” was a foot in the door for something that specifically evolved in humans as our ancestors began using paintings and drawings for communication. Maybe it was initially just hijacking edge detection so that some images could be conveyed through drawings, and over time, and that developed into a kind of “grammar” for artificially depicting more nuanced images in media where realism was impossible.

This would be similar to how recognizing and interpreting different kinds of basic vocalizations (as many animals can) developed into a much more sophisticated mechanism for developing complex language.

drcode• 3 years ago

when a plane is receding from us, it is more likely to contain color variation, which or eyes sees as a line. it is more likely to contain color variation simply because a plane receding from us has more surface area relative to a unit of space in our visual cortex, versus a plane we are viewing at the position of a normal vector. the edges of a 3d object tend to be receding planes, hence we evolved to detect this

timacles2• 3 years ago

As a person that has been practicing and studying line drawing for 15+ years, this article seems like its way off the mark. Its just not asking the right questions.

The brain uses all kinds of context sensitive cues to try to link what its looking at to whats its already seen before. If you happen to look at something thats completely new, it doesnt matter if its a line drawing, 3d 4k image or whatever, your brain will be confused. OTOH, when you're looking at something that you've seen before the brain will do all kinds of tricks and cheats to make that thing as real as possible.

Anyone that draws will know you can use plain black and white lines to depict everything from textures, depth, color etc..

Line drawing works because literally anything will work. You could see a distorted, flat black silhouette of one of your family members, and you will instantly know who it is. Its your brain that "makes" things work because its job is to take incomplete information and make it fit an existing mental model.

Line drawings carry so much visual information about objects that its really not that impressive your brain can "figure" out what its looking at. It can do so much more with so much less.

regularfry2• 3 years ago

The last picture in the article is an interesting counter-example: a line drawing that's constructed in a fairly natural way that is nevertheless almost impossible to interpret. I'm sure there are "line art rules" that picture breaks which are almost second nature to someone who's been in the game for as long as you have; the interesting thing is how that translates to the visual cortex and why those rules work.

timacles2• 3 years ago

That one isn't a line drawing, its a shadow relief drawing with depth removed. Then with with outlines applied to the shadows.

Even then, the only reason why its so "confusing" to us is because they applied some kind of random black shape to the background which extends above his head to the left.

The black background blends with the man's body making an amorphous blob. If it was removed you'd be able to instantly tell thats a bald mans head.

You are correct about the "line art rules" they're all just basic visual rules that the brain uses that artists have been playing with for 100s of years. Its no different than those illusions that look like a Grandma or young woman at the same time. All it takes is one conspicuously placed shape and your whole perspective can be thrown off.

No different from how camo works: https://en.wikipedia.org/wiki/Disruptive_coloration

Anyway, my point being, this isn't some new phenomenon, its a well understood at this point.

AndrewKemendo2• 3 years ago

Yet Lines-As-Edge supposes that the vision system discards all of this other information present in an image, for just this one special case. Why?

The "other information" is not discarded, it's just not processed yet

I think this is a misrepresentation of how the visual system works and views the vision system as more "batch" than the continuous process it is.

So if you think about it as a time series problem, when light hits the retina then "inference and processing" starts with a kind of "fast and rough" inference and then proceeds to fill in details and contextual coherency follows

I'd have to go pull out the textbooks but if you look at visual interpretation sequence it's something like:

Movement > edges > color > details

So your visual system acting as an object detector - from sensor to inference - makes inferences about movement first, then infers the edges, then infers color and finally additional details in the last few ms

Nothing is "discarded" it's just less relevant in the first pass and additional refinements happen mostly sequentially - this all happens in nearly imperceptible time.

SanjayMehta• 3 years ago

Whilst reading Dr Seuss to kids I observed that even at a very early age they can mimic the expressions on the cartoon characters’ faces, even when they’ve never seen that drawing before.

An easy experiment to reproduce.

uglycoyote• 3 years ago

Very frustrating article to read. The article is setting up a straw man and attacking it. He is acting like everyone else thinks:

1) edges are the only important features in images and 2) line drawings can only represent edges.

Who are these brainless absolutists that he is attacking?

Then he's acting like he is the only one with other bright ideas that nobody will listen to.

I think it is obvious to anyone who thinks about this that:

1) edges are a useful feature for recognizing objects in images but not the only useful feature 2) lines in line drawings can and often do represent edges, but there are a lot of other things they can represent. Light and shading and texture of various kinds.

It would be fine to write an article that goes in to depth on the different nuances, but it is annoying that this author pretends that most other experts have naive and simplistic views, with "uncritical certainty", and "no one seems to question it", and the author "has a hard time convincing them otherwise". It is a very condescending tone that comes off sounding like the author is presenting themselves as some brilliant but misunderstood outcast, and the only one who can see the light of truth.

we could do without the drama!

arketyp• 3 years ago

The author comes across as ignorant at best, but then to present his own work as the Realism Hypothesis of Hertzmann, it leans more towards arrogance.

6D794163636F756• 3 years ago

There is a subset of the tech bro that believes everything can be reduced to a problem with clearly defined taxonomy and as such every problem can be solved by an engineer with no subject knowledge. This article very much reads like one of those people wrote it.

moron4hire• 3 years ago

One of the problems right off the bat is not understanding that the classical "edge detection" algorithm doesn't actually detect edges. It detects rapid change in contrast. To then claim this computer algorithm's flaws are somehow proof that a psychological theory is therefore wrong is itself the categorically wrong thing.

bambax• 3 years ago

Figure 4 in the article [1] is absolutely fascinating and seems to prove conclusively that the edge detection hypothesis is completely misguided: when one retains only the edges (B) of the original image (A), all meaning is lost and the resulting image is unrecognizable.

What A has that B lacks is shapes. There is obviously a rotund shape of the face that is reconstructed by the brain, that one can almost see, although technically it's not actually present in A. Not present, yet visible. Same for the eyes, smile, cheekbones.

It must trigger some kind of pattern-matching in the brain.

This problem seems to be similar to the famous optical illusion of the old woman / young woman [2] that works well also in strict black and white.

In this optical illusion there is surprisingly little information on the image, yet it can trigger two very different representations (that one can see alternatively, but never at the same time).

I think the brain tries to fit the shapes it sees in one box or another, and when it finds a box it builds a whole concept around it.

I also think that boxes have to pre-exist or be learned: they can't be inferred from an image, if one has never encountered the original representation in the wild.

One evidence for this is children are completely blind to optical illusions that have one "innocent" representation and one involving some kind of nudity or sexual activity, while adults tend to see the NSFW version first.

And so, to come back at the original question of why line drawing works, I think it's because it triggers concepts. The word "square" or "circle" are unambiguous and designate precise geometric shapes (provided one has learned the relevant concepts of square and circle).

Same with shapes, except that there are an infinite amount of different shapes that we can "discuss" using the language of line drawings.

[1] https://aaronhertzmann.com/images/howtodraw/sayim.jpg

[2] https://cdn.mos.cms.futurecdn.net/rQkQZ6pDZbEHz23rxckWPm-320...

oaktowner• 3 years ago

I am not a computer vision specialist, but I wanted to add an anecdote from years of parenting.

One thing that amazed me about babies was how early they understand line drawings. Long before a baby can talk, it knows what a dog is (and can imitate a dog sound), and can also identify a dog in a photo, in a realistic drawing, and in a very simple line drawing.

It seems so easy, but in reality those things all look so very, very different. And after 6ish months, babies have mastered that recognition.

golol2• 3 years ago

Eh. To me it looks like the authors extends "lines-as-edges" explanation of why we can understand line drawings to the claim that the visual cortex works almost only on edge detection, which is obviously bunk and so easily refuted by him. Then later he shows a shillouette image we can understand even though the edges themselves are not understandable, and somehow this is supposed to be evidence against lines as edges. No, it just means that we have other methods of understanding images: shillouette, color, texture etc.

In my opinion these kinds of arguments trying to decompose the brains functioning into a couple of distinct techniques become quite obviously pointless once you look at the activation patterns of neural networks. Just looking at the features that neurons detect throughout the different layers of an inage classifying neural network tells you more than these kinds of papers ever will.

You see that edges, shillouettes, circles, circles with holes, textures, shinyness patterns, grid patterns etc. up to complex patterns and then real things like heads or arms are detected.

There is some manifold of all the images a being is somehow likely to see on this world and it has a complicated structure. You can extract the major features of the geomatry of this manifold and you come across the usual patterns. At simple complexity you find things like edges and textures, at higher complexity things like eyes or appendages. You try to find these features in an image and hope one of them works well. Maybe edges work, maybe shillouettes work or maybe both or neither.

Look I know ANNs and NNs are quite different, but the experimental evidence with NNs shows that what I described above, a mix of feature detectors that just approximate the structures of the data to deeper and deeper detail and are just all somehow applied to see what works, is much more plausible than some constructed algorithm a neuroscientist or philosopher would write down.

kortex• 3 years ago

Look I know ANNs and NNs are quite different, but the experimental evidence with NNs shows that what I described above, a mix of feature detectors that just approximate the structures of the data to deeper and deeper detail and are just all somehow applied to see what works, is much more plausible than some constructed algorithm a neuroscientist or philosopher would write down.

Precisely. Deep CNNs were directly inspired by studying the structure and behavior of visual cortical tissue.

The algorithm is just recursive feature detection, where the visual elements are transformed into abstractions, based on both sensory input and projected expectations. If anything, that's the real exciting part of the visual system, why do illusions occur, how does expectation affect perception, etc. Not "how do line drawings evoke similar concepts to images". That's just bypassing the first few layers of filters. Basic ass tiny MNIST nets can do this.

bofkcykf• 3 years ago

The word is spelled "silhouette".

tabtab• 3 years ago

"shillouettes" are what politicians use ;-)

golol2• 3 years ago

thank you :)

sdwr• 3 years ago

Thanks, was looking for this.

porphyra• 3 years ago

Drawing lines around semantic segmentation, such as using Segment Anything [1] seems to make a lot more sense than just doing edge detection on image brightness.

[1] https://segment-anything.com/

radicaldreamer• 3 years ago

Figuring this out is one of the central questions of phenomenology... how is it that after seeing a couple of trees, we can recognize almost any kind of tree from any angle? What is intrinsic about a tree that makes this possible? What (or is) there a pure form of a tree?

mrguyorama3• 3 years ago

Since definitions aren't concrete, and language is explicitly a popularity contest, you are not learning "what makes a tree a tree", but rather "what people call a tree"

You are re-deriving an existing definition, not an actual physical reality. This is a great example actually, because there is no genetic or biologic definition of a tree, and what we call a tree is matched better by some idealized representation of info like "has a trunk maybe". Also that the definition you develop is rarely rigorously tested in practice, so you only need about 80%, and basic pattern matching can get you that really easily.

lookdangerous• 3 years ago

A drawing of a cup has 'cup-ness', as far as our mind is concerned. It's not about the micro mechanisms of edge detection or color or whatever, it's about how we recognize the quality of a cup in real cups and things that push the margins for what a cup is and in drawings of a cup.

bazzargh• 3 years ago

Huh, right up my alley... a while back I was playing with trying to turn photos into sketches ( https://hachyderm.io/@bazzargh/109928618521729073 ); one of the effects I found really noticeable with my super-naive approach was that it tended to overemphasise very dark areas in a way we don't see.

Then last week I saw this, which also does some fill in shading (but, using ML)-worth the click, it's a project that makes it appear as if paper held over an object causes it to be sketched...but in fact it's all post-processing... https://www.youtube.com/watch?v=vArIkzYtW6I

This got me back to wondering about the shading problem, and I ended up down a rabbithole reading papers like https://www.yorku.ca/rfm/pub/2021annrev.pdf (review article on state of the art), https://www.frontiersin.org/articles/10.3389/fpsyg.2022.9156... (recent markov model, with links to code) about lightness and brightness perception, and how our mental model of how the scene is lit can explain a bunch of optical illusions.

While I was going for a charcoal effect, this approach https://openprocessing.org/sketch/486307 - scribble using brightness as 'gravity' on the pen - is pretty nice.

Anyhoo, I'm not going to critique the article, because I'm a total amateur just doing this for fun, but it _is_ fun, and very satisfying when you get the computer to draw something that looks hand drawn instead of just a sobel filter.

brap• 3 years ago

This was a very interesting read.

From childhood we instinctively draw lines, but I never questioned how we produce them from real objects, and why they work so well at representing real objects. It’s one of those things we just take for granted.

eternityforest• 3 years ago

Re. problem 2, Running edge detection twice will still give a recognizable result. Maybe some functions DO let you take the result of an internal layer, feed it to the input, and not get complete nonsense

rajnathani• 3 years ago

Why do line drawings work? Why is it that we can immediately recognize objects in line drawings, even though they are not a phenomenon from our natural world.

This is wrong, as humans have been drawing and viewing art for tens of thousands of years. Also, for other animals, a form of edge detection is present in for example the tiger paw mark that a tiger leaves on the bark of a tree to mark territory, or simply paw/foot marks on the ground.

Edge detection doesn’t explain line drawing

Related Stories