FastPhotoStyle from Nvidia

https://github.com/NVIDIA/FastPhotoStyle/blob/master/README.md

by scraft • 8 years ago

231 79 8 years ago

github.com

simonhamp• 8 years ago

What witchcraft is this?

sannee• 8 years ago

Of course there is a relevant XKCD: https://xkcd.com/1838/

supermdguy• 8 years ago

Paper this is based on: https://arxiv.org/pdf/1802.06474

It's really great that NVIDIA is releasing code for their deep learning research.

TD-Linux• 8 years ago

I'm going to be that person - why a non-OSI approved license? Given that it's CUDA-specific, I'd expect NVIDIA to want people to use it.

dingo_bat2• 8 years ago

Licensed under the CC BY-NC-SA 4.0 license

Seems fine to me. If you want to develop something commercial you'd roll your own anyway. Nothing else is restricted by this license.

daeken3• 8 years ago

Consider artists. There's a tremendous potential in using technology like this in art, and preventing someone from selling their works will often put them off of using it at all.

ttoinou3• 8 years ago

Is it faster than previous implementation ?

koverda• 8 years ago

Looks like it's a lot faster. They compare their approach to the Luan et al. approach, and for a 1024x512 image, they are about 30-60x faster. They also seem to be more accurate with better results.

ttoinou3• 8 years ago

Oooooh no I'm going to get back on nerding this 100 % of my time :'(

JeffreyKaine• 8 years ago

Is it really all that hard to have a demo site for these things? It would be a lot of fun to play with crossing pictures. I'm guessing it's because using a graphics card in the browser isn't good enough yet?

volker48• 8 years ago

I'm not sure how fast their FastPhotoStyle approach is, but a TensorFlow implementation of the original neural style transfer can take upwards of 20 minutes to create the final stylized image. If someone had the pre-trained model and neural net code in JS to read it and you could do it all client side then it would be possible, but still very slow.

ehsankia• 8 years ago

The tech has come a long long way since the original, even before this FastPhotoStyle project.

A few months ago, there was TensorFire [0] that was able to do it in the browser. Quick google also gives other results [1]. There's also many apps that can do it in seconds. Speed definitely isn't an issue anymore, but getting it to work in browser can be tricky.

[0] https://tenso.rs/demos/fast-neural-style/

[1] https://reiinakano.github.io/fast-style-transfer-deeplearnjs...

krn1p4n1c• 8 years ago

That top left style will be perfect for the family xmas photo.

ttoinou3• 8 years ago

Thoses interested in that technology : I made two videos 18 months ago, No optical flow and youtube compression kills everything but still decent if watched in 4K on a big screen :)

https://www.youtube.com/watch?v=2YRVt80g2Ek

https://www.youtube.com/watch?v=i69cBYI6f-w

scribu• 8 years ago

Notice that all of the examples ilustrated in the paper contain similar scenes. The content image is a building, while the style image is also a building. Or an image of trees is styled using another image of trees.

But how well does it fare when you give it an image of a house and an image of something completely different, like a dog or a slipper?

JackFr• 8 years ago

What would you expect the outcome to be?

What is the correct answer to a question that's not well-formed?

semi-extrinsic2• 8 years ago

The interesting question, then, is how far off can this be and still work? Is the limit "reasonable", or is there room for improvement of the algorithm?

E.g. I think most humans would say taking this content picture:

https://wallpapershome.com/images/pages/pic_hs/10150.jpg

and styling it with this picture:

https://c2.staticflickr.com/4/3499/3876547311_c2e32759d9_z.j...

is a pretty well-posed operation. How does that look using this algorithm?

amelius• 8 years ago

Download the code, run it, and let us know!

dingo_bat2• 8 years ago

The examples seem to be too good to be true. I don't have a GPU lying around so I cannot try it unfortunately.

medhir• 8 years ago

paperspace provides pretty easy setup cloud GPUs for ~$0.40/hr if that's of interest :)

SeanBoocock• 8 years ago

I wonder whether this could be applied to a real-time scenario. Modern Real-time renderers for games will often have a tone mapping step that let artists color grade the final output. The paper cites a 11+ second runtime for 1K inputs, which is orders of magnitude off what it would need to be, but perhaps a simpler version run on the GPU is feasible.

josephpmay• 8 years ago

Notice that the research was done by Nvidia

piracykills• 8 years ago

Nvidia is pretty big in the machine learning space in general, not game specific these days - GPUs are pretty general purpose highly multithreaded number crunchers and Nvidia's been making moves further in this direction with their own CUDA-based training tools, the DGX-1, the Jetson and other products.

dbranes3• 8 years ago

I'm probably missing something obvious here, maybe someone can explain the following to me.

- Their approach is a composition of 2 steps, what they call "stylization" and "smoothing".

- Top left of 2nd page they claim: "Both of the steps have closed-form solutions"

- Equations 5 is the closed form solution for the "smoothing" step.

My question: Where's the closed-form solution for the stylization step that they're claiming?

Are they calling equation 3 a closed-form expression? In this case the title and the claim in the introduction are rather misleadinng, because computing 3 requires you to train autoencoders.

saurik• 8 years ago

You don't train it for every image; in this way, a neural network often is a "closed-form solution": it provides you an equation, admittedly a very convoluted one, which can be used to obtain its solution, admittedly usually an approximation, in a finite amount of time. The normal solution to this problem (according to the paper) is an iterative technique "to solve an optimization problem of matching the Gram matrices of deep features extracted from the content and style photos", whereas this one is simply two passes: stylization and smoothing.

dbranes3• 8 years ago

Not sure if I understand, don't every neural network ever produce some approximation in finite time? In what sense is this approach "closed-form"?

grenoire• 8 years ago

I really don't think that it's fair to call neural networks closed-form solutions. The term immediately makes me assume that it enabled you to bypass the training stage altogether.

p1necone• 8 years ago

These very low res example images aren't particularly useful for judging how good this actually is.

limaoscarjuliet• 8 years ago

Is there some research doing the same in voice area? - Fix/change accent, - Improve person's voice, - Perhaps even make one sound like another.

STRiDEX• 8 years ago

I saw a clip from adobe a while ago

https://www.youtube.com/watch?v=I3l4XLZ59iw

arbie• 8 years ago

Yes, Adobe's VoCo, is one example.

Images and speech require different architectures (CNNs vs RNNs).

etaioinshrdlu• 8 years ago

https://lyrebird.ai/ -- they do the last thing. But they all seem related.

arto• 8 years ago

https://news.ycombinator.com/item?id=16426585

abledon• 8 years ago

Has anyone had luck using this for their tinder profile?

skocznymroczny• 8 years ago

Unfortunately, it only transfers style, not attractiveness

grondilu• 8 years ago

Only tangentially related, but has anyone ever tried to apply style-transfer on human faces for artificial aging or rejuvenation? Like for the movie industry or something?

robbomacrae• 8 years ago

FaceApp does this (inc gender swapping) and its quite fun for an hour or so of messing around.

andybak4• 8 years ago

The only machines with decent GPUs in them I have access to run Windows and Windows Subsystem for Linux doesn't allow GPU access. Other than dual-booting or running Linux in VirtualBox - is there any way I can try this?

flipp3r• 8 years ago

The user manual literally has a setup for Ubuntu, using CUDA & cupy.

andybak4• 8 years ago

I'm not sure I understand how that helps me.

executesorder66• 8 years ago

OT, but why do those machines need to run Windows? Why can't you install Linux?

andybak4• 8 years ago

They are dev boxes for Windows VR apps. I'd like to play with this out of curiosity. It's not worth the hassle of a dual boot for that.

exDM693• 8 years ago

None of the dependencies seemed to be Linux-specific at a quick glance. You might be able to install all that on Windows (not sure how pleasant experince it'll be).

Virtualbox won't help you, because you can't give proper access to the GPU for the VM guest unless you set up PCI-e passthrough and dedicate your whole GPU to the VM guest (and use your integrated graphics for the host). Not sure if this is even possible if Windows is the host.

If you don't feel like setting up a Linux install on your box, you could try some of the GPU cloud services.

ATsch• 8 years ago

Also I am told the proprietary nvidia drivers have a software lock that prevents you from using GPU passthrough unless you buy certain more expensive models.

jamespo• 8 years ago

poppingtonic• 8 years ago

Anaconda can probably help

dharma1• 8 years ago

what's the max resolution with this?

ivanceras• 8 years ago

Preparation 1: Setup environment and install required libraries Python Library Dependency
conda install pytorch torchvision cuda90 -y -c pytorch

What is conda? How do i install it on ubuntu 16.014?

sungam• 8 years ago

Conda is the anaconda python distribution widely used for machine learning and numerical computing.

dagw• 8 years ago

Conda is basically an alternative to pip and virtualenv, used by the Anaconda python distribution that's really popular in the data science and machine learning community. The easiest way to get it is to install miniconda: https://conda.io/docs/user-guide/install/linux.html

jczhang• 8 years ago

I assume you need a Nvidia card for this? Also has anyone tested it and seen how long it takes to render?

milanfar2• 8 years ago

(a) This problem is long known as color/contrast transfer, and it was solved > 10yrs ago, (b) the results shown in this paper aren't objectively or subjectively better/more photo-realistic than Kokaram etc.'s work; and (c) I question whether this task even requires deep learning at all.

https://francois.pitie.net/colour/

milanfar2• 8 years ago

This problem is long known as color/contrast transfer, and it was solved more than 10yrs ago. The results shown aren't objectively or subjectively better/more photo-realistic than Kokaram etc.'s work which is far simpler. I question whether this task even requires deep learning at all.

https://francois.pitie.net/colour/

FastPhotoStyle from Nvidia

Related Stories