Mitsuba 3 Physically Based Renderer

by oumua_don17 • 4 years ago

219 56 4 years ago

solarmist2• 4 years ago

Anyone know how this relates to GPU ray tracing or if it does or why it is exciting?

keyle• 4 years ago

I know nothing of Mitsuba, but one of the optional requirements is nvidia drivers for gpu compute. So it looks like it does.

mattkrause2• 4 years ago

There are “variants” that use simple scalar floats, CUDA, and an intermediate mode that does CPU-based vectorization. The same folks have made some vectorization libraries: enoki (used in v2) and a newer one called Dr. Jit.

More about it here and why it’s different from (e.g.,) PyTorch https://drjit.readthedocs.io/en/latest/intro.html

erichocean3• 4 years ago

It does use the GPU, but this isn't really for production use (at all).

It's a research renderer and its key functionality is reversible rendering—going from an image to the scene parameters that would create that image.

The purpose of this kind of renderer is to develop research ideas that will eventually make it into a production renderer someday…

solarmist2• 4 years ago

Thank you. I didn’t have a clue why I should care about this.

IIAOPSW2• 4 years ago

Unless it simply encodes scene information as imperceptible color gradients, this is impossible as a render necessarily projects from 3d to 2d.

dahart• 4 years ago

There’s a page with a paper and video about to their Just-In-Time compilation engine: http://rgl.epfl.ch/publications/Jakob2022DrJit

It relates to GPU ray tracing because the engine can compile either CPU or GPU specific code, specialized to the available hardware and tailored to the specific scene data; makes it so you can render faster than otherwise.

dang• 4 years ago

Mitsuba – A physically based renderer - https://news.ycombinator.com/item?id=10096737 - Aug 2015 (8 comments)

kristianov• 4 years ago

The tagged line is really weird. Physically Based? Not really based indeed.

mattkrause2• 4 years ago

How so?

Raytracing tries to create images by simulating how light bounces around an environment. It’s physically-based in that it uses laws of physics/optics to do that (e.g., Snell’s Law when light passes through different media), rather than some method that just happens to look cool.

jiggawatts2• 4 years ago

We're at a cusp in technology where "reverse rendering" is rapidly progressing from a research curiosity to a mainstream product. As soon as high-quality model generation becomes possible on a typical workstation, there will be a dramatic shift in how computer game art assets are produced. Many artists will be replaced by this mechanical, automated method of 3D model production.

No longer will you need a team of 100 artists for every 2-3 developers to manually model and texture objects, instead you'll have 10-20 photographers running around taking pictures of real-life objects and scenes. These will then be reverse-rendered into models, which can then be directly placed into the game world by just a handful of map designer artists.

You can see this in Unreal Engine 5, where they combined their "infinite detail" polygon engine Nanite with their acquisition of Quixel's Megascans library of models captured via photogrammetry.

However, photogrammetry is not the same as reverse rendering: it's only really suitable for diffuse objects captured in good conditions (foggy or overcast days). It can't handle transparency, reflections, or complex/dynamic self-shadowing. Currently it's mainly used for backdrops like rocks, logs, and road surfaces.

Reverse rendering can in principle capture anything a renderer can do, including specular materials. It can "undo" self-shadowing and produce objects that can be re-lit in different scenes.

For gamers, these are going to be exciting times!

For graphic artists... perhaps also exciting, but not necessarily in a good way.

paisawalla• 4 years ago

It seems like this means artists would mostly be needed for creative art, e.g. spaceships, monsters, etc. That is, things that don't exist, so can't be photographed.

But there are already commercial and free libraries of commonplace items like furniture, cars, and buildings. So isn't what I described pretty close to the same situation today already?

taylorius5• 4 years ago

But spaceships and monsters do exist. There are thousands of them, just look on google images. An AI inverse rendering system, could just as easily ingest a ton of space sci-fi images, as real world photos.

esperent• 4 years ago

For graphic artists... perhaps also exciting, but not necessarily in a good way.

I think you're missing an important aspect of art - style.

The CG industry has been chasing photorealism for decades and we've been slowly inching closer. Photogrammetry is an important step that will provide a big boost in that goal.

But what happens when we get there? Once it's (relatively) cheap to make a AAA game that's virtually indistinguishable from reality, what happens next?

Everyone will love it for a few years but then they'll start to demand something new, and that might be non-photorealistic styles. Or it might be off world or impossible environments. Photogrammetry won't help much with a game set in space in the distant future, populated with non Earth creatures, plants, and architecture. Although it will help a bit, of course. There will still be a place for ultra realistic environments (war games, spy games, romance games etc.) but it will become just one style amongst many.

If anything this might lead to a revival in making real world models and then scanning them. Imagine a Dark Crystal style world except you are the player. All the characters and models were crafted by hand and then scanned into a 3d world where they are indistinguishable from their real world counterparts when viewed in VR/AR or on screen. That would be cool.

As for artists, perhaps they should be worried about AI stealing their jobs. But not photogrammetry, that'll just remove some of the tedious stuff like modeling furniture.

_carbyau_• 4 years ago

I second the "style" aspect. For a games example, Call Of Duty vs Borderlands - both FPS but the visual styles are worlds apart.

This could provide a new tool, and maybe remove some jobs. But it will make new jobs related to this new tool.

jiggawatts2• 4 years ago

Neural nets can do style transfer. It might be as simple as making photorealistic models and then transferring a style from a handful of non-realistic samples.

psyc• 4 years ago

I've been both a game programmer, and a computer graphicist coveting photorealism, for ~30 years. In my opinion, this is exactly right. Photorealism is and was never going to put artists out of a job. But other algorithms... might. I can't rule it out, but the older I get the less certain I am of where so-called 'AI' is going.

themerone• 4 years ago

I think you misjudge how much artistry goes into placing those photographed assets.

You are probably way off on your estimates in how this will change the workforce.

Small teams of artists can now produce television VFX that rival what took hundreds of film artists to do decades ago, but if you pay attention to movie credits, the number of artists working on big films seems to be increasing.

Also, it seems to me that AI and procedural generated assets are more likely to be the future than real world scans.

naillo2• 4 years ago

It'll be interesting which of these win; scanning real life objects as you describe or just generating 3d assets directly through a DALL-E style model.

alleycat5000• 4 years ago

This is a nice tutorial on differentiable and inverse rendering (one of the Mitsuba authors also gives the tutorial):

https://shuangz.com/courses/pbdr-course-sg20/

Inverse rendering is such a cool idea... solving for the inputs (3D models and materials) to a renderer such that they match real world inputs (photographs) by computing gradients automatically.

airstrike2• 4 years ago

Couple this with VR glasses and you can relive your favorite memories in first person rather than through a photograph. I've been thinking about this for probably a decade now, and can't wait for it to become reality.

taylorius5• 4 years ago

I'd be careful with that. My suspicion is that our memories would transform themselves into memories of the simulation, rather than the original event.

airstrike2• 4 years ago

I’d watch that sci-fi film

ykl• 4 years ago

As with almost everything that comes out of Wenzel Jakob's lab at EPFL, the documentation for Mitsuba 3 is _incredible_. Check out this writeup of how polarization is handled in Mitsuba 3 (complete with animated diagrams): https://mitsuba.readthedocs.io/en/latest/src/key_topics/pola...

That writeup is PhD Thesis material in terms of scope and detail, but the student that wrote it, Tizian Zeltner, actually wrote his PhD on another topic entirely. This is just the level of stuff that routinely comes out of that lab.

throwaway17_17• 4 years ago

Agree on the quality. I have several documents from the Mitsuba 2 era that are of the absolute highest quality. And for anyone wondering, this counts both the quality of content and the actual success of communicating complex and varied ideas in commendable fashion.

bayindirh• 4 years ago

This is a result of a group in harmony with people who are in love with what they are doing. When you like the subject, and pour what you have into it, this is what comes out. I'm sure this wasn't tiring for them.

Make no mistake, this is a very high achievement, and is very rare in both private companies and research labs. It's a very hard point to reach if you don't like or care about what you are doing.

Kudos to them.

AdamH12113• 4 years ago

What is a physically based renderer? It sounds like ray tracing that can be done in reverse on a final image?

cornstalks• 4 years ago

I’ve always seen “physically based” to mean the renderer does a physical simulation (photons bouncing, refracting, having wavelengths stretched, subsurface scattering, BSDF calculations, etc.).

So much in computer graphics are just (clever) hacks upon hacks to get something that looks “good enough” but isn’t really simulating physics in any meaningful way (like SSAO, texture baking, bump mapping, etc.). These hacks are much, much faster than simulating the physical process of photons interacting with the world.

I don’t know where “physically based” originated but my first introduction to it was pbrt, which I suspect popularized “physically based” naming.

Differentiable rendering is the name of going from the final image and “reverse rendering” it to reconstruct a 3D model of it.

fluxem• 4 years ago

Some clarification regarding the differentiable rendering vs inverse rendering.

Differentiable rendering is what it says: differentiating the rendering process. Imagine that x is the scene, then f(x) is the function that renders the image. Then, Differentiable rendering is simply taking the derivatives: f'(x).

Inverse rendering is a process of finding scene parameters x, such that f(x) produces the given image y. This is often achieved by using differentiable rendering together with an optimization algorithm (like SGD or Adam). However, due to the nature of rendering, it's easy to get stuck in a local minimum. Therefore, even a perfect differentiable rendering engine is not sufficient for the inverse rendering.

ohgodplsno2• 4 years ago

Mind you, physically based renderers also use a bunch of clever hacks upon hacks to be able to run in realtime. Nobody is simulating photons bouncing in the nanostructure of a material. We just have formulas thay say "for this roughness and metallicness, your material will look like that and it'll be good enough". We still abuse the hell out of UV maps to fake things, still include ambient occlusion (as a stylistic choice most of these days).

It's a different workflow that gives you consistency once you know what you're doing, as opposed to the days of Phong shading.

hypertele-Xii3• 4 years ago

Traditionally rendering has been concerned with the RGB colors, because that's what the screen has to output. So your materials are defined in terms of what color objects are. Then they started adding "effects" on top like reflectiveness, bump and normal maps, noise, etc. to make objects look more... real.

Physically based rendering approaches from the opposite end. It asks, what are the characteristics of the material, in terms of simulated light interacting with a nontrivial surface, so that the thing appears on screen as it should if it were a real object with real light bouncing off its nanostructure surface.

Ray tracing is one method of rendering, but you can physically based render the old way too with fragment shaders. You just won't get global illumination so it won't look as "real".

ohgodplsno2• 4 years ago

There are ways of faking GI, even with current techniques and without using ray tracing. Screenspace GI can look convincing (and also miserably fail). Radiosity techniques exist, etc. We've been able to fake global illumination for a while now. The results aren't perfect, but then again, that's what realtime computer graphics have been forever, a bunch of tricks that are close enough to reality, and 10 artists crying in the corner because they had to spend 5 hours moving an asset around so it looks just right.

zokier• 4 years ago

Traditionally rendering has been concerned with the RGB colors

I do need to point out that many (most?) so called physically based renderers are still RGB based, with all the downsides that come with that. Mitsuba is notable for also having spectral rendering, which is a distinct feature that models light as a spectrum instead of rgb triplet.

taylorius5• 4 years ago

It just means that the renderer is making some attempt to mimic the way light behaves in the real world. This, hopefully, implies that the resulting image will look realistic.

magicalhippo2• 4 years ago

A physically-based renderer tries, as much as possible, to ground the equations and techniques in physical principles. For example conservation of energy[1], surfaces shouldn't reflect more light that what shines on them, and Helmholtz reciprocity[2].

This affects the choice of approaches and algorithms, such as using unbiased rendering[3], and their implementation, like using energy-conserving bidirectional scattering distribution functions[4] to describe how light interacts with a surface.

[1] https://en.wikipedia.org/wiki/Conservation_of_energy

[2] https://en.wikipedia.org/wiki/Helmholtz_reciprocity

[3] https://en.wikipedia.org/wiki/Unbiased_rendering

[4] https://en.wikipedia.org/wiki/Bidirectional_scattering_distr...

magicalhippo2• 4 years ago

It sounds like ray tracing that can be done in reverse on a final image?

I forgot to mention, if the renderer respects Helmholtz reciprocity, then you can chose to either do forward rendering (rays originate at light and bounce until they hit the camera or dissipate) or backwards rendering (rays originate at the camera and bounce until they hit a light or dissipate), or even do both, so-called bidirectional path tracing[1].

[1] https://pbr-book.org/3ed-2018/Light_Transport_III_Bidirectio...

daniel-thompson• 4 years ago

Sidenote: the author of Mitsuba is also a contributor to https://www.pbr-book.org/, which is a great (graduate-level) introduction to this topic.

nl• 4 years ago

The inverse rendering (Radiance Field Reconstruction (NeRF-like)) is very impressive. It can reconstruct physical models from multiple photos.

See the final results at https://mitsuba.readthedocs.io/en/latest/src/inverse_renderi... (scroll down a bit to see the reconstructed image vs the original)

junon• 4 years ago

This is missing examples of the outputs. I don't see any actual renders of note.

Am I missing something?

moonchrome• 4 years ago

From what I see the point of this renderer is that it can invert the rendering to come up with a 3d scene that matches the input render image, so I guess outputs aren't as important ?

andybak• 4 years ago

IMHO it's "differentiable" that's the most interesting part of this. There's tons of physically based renderers.

It's odd that that word doesn't appear on http://www.mitsuba-renderer.org/ - unless "retargetable" carries a similar meaning?

doctorhandshake• 4 years ago

This was my reaction - I clicked for PBR and was surprised to find the much more interesting lede buried

naillo2• 4 years ago

Just waiting for someone to apply CLIP guidence onto this (so I don't have to).

majoe• 4 years ago

As someone coming from the physics side and not so much from the Computer graphics side of physically based rendering, I have high hopes for differentiable rendering as a general tool to extract knowledge from light-based measurements.

Most differentiable renderers stop at the object surface, but differentiable volumetric light transport is what would allow us to use the renderer to optimise a measurement device or for medical diagnostics or just to estimate the optical properties of a material (that is absorption and scattering coefficent and scattering phase function), which is a pretty hard problem in itself.

From a quick look, Mitsuba 3 seems to have everything in place to be used in such a setting and I'm really excited to see some use cases outside of Computer graphics.

Mitsuba 3 Physically Based Renderer

Related Stories