Extending Van Gogh’s Starry Night with Inpainting

http://blog.wolfram.com/2014/12/01/extending-van-goghs-starry-night-with-inpainting/

by lelf • 12 years ago

281 43 12 years ago

blog.wolfram.com

jwise0• 12 years ago

I wish that the Wolfram articles would describe a little more about what their algorithms do, rather than the mechanics of how to use their primitives. The articles remind me some of the era of web engines optimized for a specific task -- there were plenty of MVC frameworks that work really really well for writing a todo list in 25 lines, but doing anything more complicated than a todo list meant dissecting the guts of the framework.

It'd be great to hear how they did an efficient implementation of texture synthesis. But if you want to read at least about the basics, the dissertation they linked [1] gets into the details on page 18, anyway, in section 2.2 (and beyond).

[1] http://www.logarithmic.net/pfh-files/thesis/dissertation.pdf

devilsdounut• 12 years ago

Wolfram has a habit of creating blog posts about the test cases for their algorithms.

jordigh4• 12 years ago

He also has a habit of not talking about how things work. All the time. Source code, mathematical proofs, citations, all of these things are relatively unimportant in Wolfram's world. He even thinks you should not try to find out how things work:

http://reference.wolfram.com/language/tutorial/WhyYouDoNotUs...

dahart• 12 years ago

Re: basics, here is one of the least efficient texture synthesis algorithms, but is detailed, an excellent starting place for beginners, and produces very high quality texture synthesis: http://graphics.cs.cmu.edu/people/efros/research/NPS/alg.htm...

That is from my favorite texture synthesis paper of all (Efros & Leung '99), for these reasons:

- I know Efros, and he's a genius, and also an absolutely adorable human being.

- This paper was inspired by Markov text generators, its the two-dimensional equivalent, e.g., http://en.wikipedia.org/wiki/Mark_V_Shaney. Its very simple, produces great results, - AND - this texture synthesis algorithm will synthesize written text from a picture of written text. Almost none of the faster texture synthesis algorithms do this, and it seems meta; I like to think there is some deep fundamental truth hiding in there somewhere.

(edited for typos and formatting)

spot2• 12 years ago

afaik this technique was first implemented in the early 90s: http://draves.org/fuse/

jordigh4• 12 years ago

Wolfram has a habit of not giving a lot of citations or hiding them under the small print in order to make it seem like he or his company came up with everything in a void.

JacobEdelman3• 12 years ago

Although this article isn't by Wolfram I agree that the philosophy of his company seems to be to hype products like any normal consumer company does. The problem is this doesn't really work when your selling and academic product. No programming language management organization will claim they are "revolutionizing programming" without strong qualifiers.

sp3322• 12 years ago

FTA: The “TextureSynthesis” method is based on the algorithm described in “Image Texture Tools,”[1] a PhD thesis by P. Harrison. This algorithm is an enhanced best-fit approach introduced in 1981 by D. Garber in “Computational Models for Texture Analysis and Texture Synthesis.”[2]

[1] http://www.logarithmic.net/pfh-files/thesis/dissertation.pdf

[2] http://dl.acm.org/citation.cfm?id=910609

spot2• 12 years ago

is Garber's thesis (or better yet, a paper summarizing it) available online?

thu• 12 years ago

That's really neat. This makes me think to image analogies[0], something that impressed me enough that I keep remembering it after 13 years!

[0] http://www.mrl.nyu.edu/projects/image-analogies/

adamb_• 12 years ago

Thanks for sharing this. I was really impressed by this example: http://www.mrl.nyu.edu/projects/image-analogies/potomac.html

JoeAltmaier• 12 years ago

Its tiling the painting seamlessly, which is cool. Not really Van Gogh; but Van-Gogh compatible?

maho2• 12 years ago

Mathematica demos sure look pretty [0]. But I recently tried to get actual work done with Mathematica and found the experience utterly ridiculous.

The high level functions, while nice for demos, never quite do what you need. Want to display data vs. time with errorbars around the data? Sorry, no function for this completely exotic use case, please hack a DateListPlot on top of an ErrorListPlot [1].

The WolframAlpha integration "transparently" interprets code for you in the background. This interpretation can change any day [2] and leaves you guessing why your notebook doesn't run anymore.

The bugs I encountered were severe: One of the output function swallows the "-" sign if the result is between -1 and 0 [3]. Currently I am battling file corruption of notebooks that are saved on network drives and open while the drive is disattached/reattached.

Had it not been for the awesome support by the mathematica. stackexchange.com community, I would have quit Mathematica completely. Now I vowed to use it only for small hacks. And, if I want to impress someone, for demos.

[0] https://www.youtube.com/watch?v=EjCWdsrVcBM

[1] http://mathematica.stackexchange.com/questions/18962

[2] The input

    Quantity[1, "Kelvins"]

is interpreted by Mathematica itself and yields an object equivalent to "1 Kelvins". The input

    Quantity[1, "Kelvin"]

however is interpreted "transparently" by Wolfram Alpha. A few weeks ago it evaluated to "1 KelvinsDifference", but now has changed to "1 Kelvins". Don't get me started on the KelvinsDifference vs. Kelvins thing, which are both represented by the same symbol to maximize your confusion when your unit conversions are failing...

[3] http://mathematica.stackexchange.com/questions/65600

leephillips• 12 years ago

Really interesting, detailed comment. The problem is not that Mathematica has bugs - all complex software has bugs (except TeX). The problem is that you can't fix it, or even see how it works, because the source is secret. Nothing will get fixed until the company decides to fix it. Access to source is not an ideological imperative, it's a practical necessity.

cpach2• 12 years ago

I’m not convinced open source would really help. I get the feeling that Mathematica’s underlying philosophy just isn’t a good fit for practical programming.

taliesinb3• 12 years ago

Actually, the source code for most things that aren't implemented in C are available. They're just distributed in the compiled MX form, so there isn't a source file to read. But you can still easily 'disassemble' parts of the language at runtime.

Here's a very simple way to do it for DateListPlot:

   Get["https://raw.githubusercontent.com/szhorvat/Spelunking/master/Spelunking.m"]
   DateListPlot;
   Spelunk[DateListPlot]

This produces a hyperlinked 'code browser' in your notebook that lets you chase down function definitions out of the live kernel. Here's an example of what it looks like:

http://imgur.com/ZjWJvEO

So the OC could actually rewrite DateListPlot to add error bar support, if he wanted to. He could even overwrite the original MX file in his installation once he had done this, if he so desired.

coldtea• 12 years ago

Access to source is not an ideological imperative, it's a practical necessity.

I have access to the source of tons of stuff, and yet I can't fix it.

Not without spending months on understanding their codebase and going out of my actual projects and work.

(And even then, my fix might not be accepted upstream, so it could be invalidated with any major or even minor version update).

That other's have access too, so someone might fix it, also doesn't work that much. Tons of projects, even extremely popular ones like GTK have but a few commiters, and tons of open bugs.

taliesinb3• 12 years ago

The high level functions, while nice for demos, never quite do what you need. Want to display data vs. time with errorbars around the data? Sorry, no function for this completely exotic use case, please hack a DateListPlot on top of an ErrorListPlot [1].

'Never' seems like quite a generalization. What about e.g. Classify and Predict, or FormFunction, or Interpreter, or Nearest? Do those all not do quite what you need? Conversely, does other software do better, on average, in 'doing what one wants'? Which software?

There are gaps, definitely, one small example you've pointed out. My pet peeve, for example, is that it is unnecessarily difficult to map colors and sizes onto graph vertices based on vertex properties, which I needed to do for [1].

Still, the scope of visualization functionality is so vast [4] that it's not surprising there are gaps (which we continue to plug, the whole language is a never ending project). And having users report to us what they want lets us prioritize those things. Anyway, clearly DateListPlot should learn about error bars, and I'm sure in a year or two it will. Although it is frustrating you can't just go in there and tweak the source code yourself, I agree.

Don't get me started on the KelvinsDifference vs. Kelvins thing, which are both represented by the same symbol to maximize your confusion when your unit conversions are failing...

I think Quantities should probably gain a little box, like DateObjects have [2], that lets you mouse-over for more information, which should make this problem easier to diagnose. But the distinction is apparently important [3], although I don't want to "get you started" :-). Either way, Alpha shouldn't change its interpretation from under you, that's definitely a bug.

http://mathematica.stackexchange.com/questions/65600

Unfortunate, but fixed within a day of being reported! 10.0.2 has passed many weeks of SQA and will ship shortly. So at least this bug will only have been around for 0.0.2 versions :-)

[1] http://blog.stephenwolfram.com/2013/04/data-science-of-the-f...

[2] http://reference.wolfram.com/language/ref/DateObject.html, http://reference.wolfram.com/language/ref/Entity.html

[3] http://reference.wolfram.com/language/tutorial/TemperatureUn...

maho2• 12 years ago

'Never' seems like quite a generalization. What about e.g. Classify and Predict, or FormFunction, or Interpreter, or Nearest? Do those all not do quite what you need?

Certainly, there are many functions that do what I need - but in my experience, the "higher" the level of the function is, the smaller the chances that it fullfills my needs.

Conversely, does other software do better, on average, in 'doing what one wants'? Which software?

Over the years I have come to appreciate Matlab. Like Mathematica, it also has some maddening idiosyncrasies [0] and is mostly closed source, but it let's me inspect its high level functions. For instance, if I am not happy about the errorbar plots, I can execute the command "edit errorbar" to get an idea what I have to change. If the change is complex, Matlab is a decent enough IDE to implement it anyways. In comparison, Mathematica seems like a front-end to the kernel to me. It is not a proper IDE. I heard there is a Mathematica IDE called Wolfram Workbench [1], but it costs extra.

For me, my biggest problem is not about the sofware "doing what one wants", but that it is doing things I don't want. File corruption and sign error bugs are absolutely not ok for software that I base my publications on, and I have not encountered them in the alternatives [2].

Mathematica is a powerful piece of software and has many use cases. Of course it can't be a "eierlegende Wollmilchsau" that is good at everything [3]. But personally (disgruntled customer rambling here) I wonder why Wolfram Research is devoting so much energy into developing all these new, shiny features while the foundation is crumbling.

[0] Try running a Matlab script that has - gasp - a space in its name. It will fail and produce an "undefined function" error message. (Not very helpful to new users.) The newer versions at least warn you if you try to save a script with an illegal name - but they still allow you to run it and produce the non-descript error message.

[1] http://www.wolfram.com/products/workbench/

[2] The alternatives I actually use are Igor Pro, Matlab, and Excel. Excel can easily lead to sign errors because of the unexpected behaviour of the unary minus. It takes precedence over the power operator, so if you evaluate "-1^2", the result is +1. While you can get used to that behaviour, you can easily forget it when you work with references. Say A1 contains 1, then "-A1^2" evaluates to +1. If, like me, you prefer working with named cells and name A1 as x, then "-x^2" also evaluates to +1. I once botched an analysis due to this behaviour. Ironically, this was one of the reasons why I wanted to base my current analysis on Mathematica.

[3] http://commons.wikimedia.org/wiki/File:Wollmilchsau.png?usel...

tyilo• 12 years ago

For the units thing, you might want to take a look at http://mathematica.stackexchange.com/questions/15338/simpler...

Also I have this in my init.m: Unprotect[WolframAlpha]; ClearAll[WolframAlpha]; WolframAlpha[x__] := Null; Protect[WolframAlpha];

andrey-p• 12 years ago

Worth checking out the rest of Yarin Gal (the PhD student whose work prompted this post)'s extrapolated works of art:

http://extrapolated-art.com/

Some of these look so seamless it's staggering.

trop• 12 years ago

Not to be totally grumpy here, but this is classic mathematicians-think-art-is-images-looking-like-famous-art. The expanded "starry night" isn't an interesting painting, of course. And it adds nothing to Mr. V. V. Gogh's work. It just mimics aspects of the extant work, in a spatially naive manner (e.g., what more do we learn except that the new rendition exists on a black mesa surrounded by more village-with-church?).

SIGGRAPH-style research shouldn't validate itself merely because it teaches the computer to adopt a cliched mode of image-making.

infinitone• 12 years ago

Have you looked at the other examples? http://extrapolated-art.com/

Some of these are pretty impressive and do more than just pattern repetition.

TheOtherHobbes• 12 years ago

I'm going to agree with the critics - I think you may be confusing technical problem-solving with artistic awareness.

They're all interesting as technology demonstrations, but none of them improve on the original compositions, and some of them are almost completely unsuccessful as visual art.

Rapzid• 12 years ago

I think that makes the whole just as interesting. Photographers often frame a tiny portion of the entire view available to them. That frame is where their art is, not the expanded view. So in a sense the wall just highlights that what's inside the frame is special and it contrasts nicely.

djloche• 12 years ago

Not to make this any less impressive, but doesn't the latest versions of Photoshop have these types of features built in?

special_ghost• 12 years ago

Such an approach might be useful to the print industry for generating seamless "bleed" around images, which is required to allow some margin for error when trimming the physical prints.

mistagiggles• 12 years ago

From the comments on the article: http://imgur.com/a/BfcK4

Some more tests of the algorithm with less impressive results (though still good looking)

soperj• 12 years ago

Only problem is, I've been to the village where Starry night was painted, it's not that big.

thenomad• 12 years ago

Extension algorithms are absolutely amazing - and incredibly useful if you're a practical working visual artist.

I've personally found Photoshop's Content-Aware Fill to be so useful on films and comics that it now gets its own credit, as:

"Matte extension: Content-Aware Phil".

Extending Van Gogh’s Starry Night with Inpainting

Related Stories