Every now and then, and definitely not when we could find any other speakers to come in, WGD holds Lightning Talks. Any member is free to come along and opine for a few minutes about whatever game-related topic they please. In the past, we’ve had talks on using OpenGL in fancy ways, using fancy DAWs (Digital Audio Workstations) to create funky tunes and “games for people”, for those fancy-pants people who love games involving no equipment except the bodies of you and your friends.
This time round, there were three talks – one was on palette-swapping and different techniques used in the past, and another was about making your own art (it was sorta like the Shia LaBeouf Just Do It talk, but with less shouting and more creepy drawings). Finally (or rather, in the middle of those two), was a talk from someone not at all built for public speech – me. I did an analysis of some of the cool shaders used in Super Mario Odyssey‘s Snapshot Mode, and I’m gonna do it again, right here. It was somewhat refreshing having a whole session of talks about arty subjects, even if they often swayed into technical artistry, given that the society is somewhat programmer-heavy. Never mind that the Lightning Talks took place last term – let’s pretend I’m prompt at blogging.
Anyway, it’s time to jump up in the air, jump up don’t be scared, jump up and make shaders every day!
I’ve set up this demo scene with a few coloured cuboids, one directional light and a boring standard Unity skybox. It’ll do the job for demonstration purposes. In the executable version, you can freely move the camera around.
I took the most saturated environment I could find, just to desaturate it.
First up is a simple colour transformation. To convert an image to greyscale, all we do is take the luminance of each pixel in the scene and use that value for the red, green and blue values of that pixel’s colour.
float lum = tex.r * 0.3 + tex.g * 0.59 + tex.b * 0.11;
The luminance value takes into account the sensitivity of the human eye to each component of colour; your eyes are more sensitive to green, so it is heavily weighted in this calculation. That’s basically all there is to the greyscale shader.
This is pretty much what you’d expect; the effect is very similar to Odyssey’s.
The city environment felt the most appropriate for a sepia-tone shot.
The sepia-tone shader is very similar to the greyscale one, although the transformation is a little more involved. Before, we used the same value for each of the resulting red, blue and green channels to get greyscale values, but this time, each of the resulting red, blue and green are separate functions of the original RGB values.
half3x3 magicNumbers = half3x3
0.393 * 1.25, 0.349 * 1.25, 0.272, // Red.
0.769 * 1.25, 0.686 * 1.25, 0.534, // Green.
0.189 * 1.25, 0.168 * 1.25, 0.131 // Blue.
half3 sepia = mul(tex.rgb, magicNumbers);
We have to resort to some matrix multiplication to achieve this. The 3×3 matrix represents a bunch of numbers magically pulled out of Microsoft’s rear end – the mapping from the original colours to the new ones. The last line here essentially represents a system of three equations, where, for example:
sepia.r = tex.r * 0.393 * 1.25 + tex.g * 0.349 * 1.25 + tex.b * 0.272
If you’ve never done linear algebra or matrix multiplication before, this might look a bit exotic, but trust me – it works!
We end up with another unsurprising result. You might find this a bit too yellow, in which case just remove all the multiplications by 1.25 I added.
This one is where things get a bit more interesting. Upon seeing this effect, I immediately thought to myself, “Aha! They’re just using the depth buffer values! I’m a goddamn genius”! So, what is the depth buffer I hear you ask?
In computer graphics, you need to render things in the correct order. If you have a crate fully obscured by another crate, you wouldn’t want to render the obscured crate over the other one – it’d look really strange! Whenever a pixel is drawn, its depth – the distance from the camera plane to the object being drawn into this pixel in the z-direction – is recorded in the depth buffer (z-buffer); this is just a big ‘ol 2D array the same size as the program’s on-screen resolution. When you try and draw something, you first check the new object’s z-buffer value against whatever is already recorded at that position, and if it’s a higher value (the distance from the camera is further), that means it’s occluded and we discard that pixel.
This is the same scene from above, but from a different angle. Each pixel is just showing the greyscale z-buffer value of the final rendered image, which means it’s representing how far away the closest thing at that pixel position is. But why does this not look as good as the Odyssey example? Well, first of all, I’m not Nintendo EPD and they have a few more resources than I do. Second, I’ve not coloured the image myself, although as we saw earlier, this would be a fairly trivial change. The main reason is actually because the scene isn’t very large and we need to understand how the z-buffer does things to understand why.
There needs to be a bound on how small or large the z-buffer values are, else the values would be pretty meaningless. A camera has a near clip distance and a far clip distance, which are the minimum and maximum distances from the camera plane an object can be to be rendered. If you’ve ever clipped the camera through part of a wall in a game, that’s because the near clip plane is further away than the wall is, so that section of wall is culled (not rendered). Values in the z-buffer are typically floating point values between 0 and 1, so a small scene (i.e. one where the clip planes are fairly close together) would result in two objects having more different z-buffer values than the same objects in a large scene.
In the scene above, the near plane is very close to the camera and the far plane is just far enough to contain the whole scene. By default in Unity, the far clip plane is 1000 units away, which would result in the entire scene here being black – all z-buffer values would be close to 0. But having a small scene means you don’t really get much variation in colours, because things are really close and there’s nothing in the background – in the Odyssey screenshots above, there are things clearly in the foreground (Mario and the rocks he’s stood on) and the background (the section of island behind the waterfall). One detail that’s kind of hard to make out is that the furthest island actually shows up in very faint green in the snapshot, so it’s real geometry rather than part of a skybox. Also, the water particles don’t show up in this mode, so they’re likely reading from the depth buffer but not writing to it, and are rendered after all the opaque level geometry.
Seaside Kingdom Doggo raised this game’s review average by 19 points.
Next up is a blur effect. I won’t be talking about the radial blur starting from the edge that Super Mario Odyssey uses though; I’ll talk about a simpler blur that I had time to write – a Gaussian blur that operates on the whole image.
This type of blue works by throwing a ‘kernel’ over each pixel in the original image and looking at the neighbouring pixels. The larger the kernel, the more pixels are considered. Then, a weighted average of the pixel values, with the central pixel having most weight, is calculated for the new image. It’s a pretty simple concept, and probably how you’d expect, but is one step more complicated than a box blur which doesn’t bother with weighting the pixels and just does an unweighted average. Commonly, you would do two or three passes with different sized kernels, so the algorithm is run more than once and the resulting blur has more fidelity.
This image is doing three blurring passes, each with a larger and larger kernel. Each blurring pass actually requires two shader passes, one in the x-direction and another in the y-direction. You could reduce the amount of blurring by making the kernels smaller or reducing the number of passes.
TFW you turn up to a formal occasion in casual clothing.
The final algorithm I implemented is an image-space Sobel filter to attempt to detect edges in the image. This also acts in separate x-and y-direction passes; in each direction, a gradient is calculated, where a higher gradient means a more ‘different’ colour. By definition, you can’t do a gradient in both directions simultaneously, hence the need for two passes. This effect wouldn’t be used in a game if you wanted objects to stand out against objects of similar colour, but it’s pretty simple to understand conceptually.
Running the algorithm on our sample scene results in pillars that are pretty cleanly defined, but this is mostly due to their block-colouring. You can also see a couple of spots where pillars seem to ‘blend’ into one another, and all shadows are also edge-detected because they are dark areas right next to light flooring.
For a bit of fun, I ‘converted’ this effect into a cel-shading effect. Again, it’s not exactly Borderlands quality, but it was a fun exercise for me. This effect is also not present in Odyssey, so there is no reference image comparison.
The Sobel edge detection filter from above is inverted and multiplied by the source image, such that ‘edge’ areas become black and all other areas retain their original colouring. The areas where colours are similar clearly lack cel-shading here, such as the red pillars right-of-centre. You can also see on the yellow pillars where the skybox turns lighter that the black lines ‘break’ a little. Of course, because it’s based on the Sobel edge detection filter, you also get cel shading on the shadows, but the effect is less pronounced here because the shadows are dark.
All of the source code and a built executable for Windows can be found over on my Github. It’s really fun to pick apart effects you’ve seen in video games and I wholeheartedly recommend you try the same! If I have time over the holidays, I may try to pick apart some of the other effects, such as the NES/SNES filters which are probably just pixelation filters with a palette swap applied on top.