Flying back on the plane with many ideas spinning in my head, I was wondering how I could engage more with the industry and help game developers achieve better performance and improved visual quality.
A few days later, I still couldn’t sleep as ideas were spinning in my head. I imagined being in a windowed room with the sun outside travelling across the sky. Then I thought, if the room is static geometry, why do I need to render it to the shadow map every frame? The things which are moving around are everything but the room: the sun, the camera, the dynamic objects etc. I realized that a texture could represent the whole static environment (the room in my case). Going a bit further, the alpha channel could represent how much light can enter the room. My initial assumption was that a cubemap texture would only fit a uniform room but, as we later found out, cubemaps seem to be very good approximation of many kinds of local environment - not only square rooms, but also irregular shapes such as the cave we used in the Ice Cave demo which you can see below.
With the whole room represented by a cube texture I could access arbitrary texels of the environment from within fragment shader. With that in mind, the sun could be in any arbitrary position and the amount of light reaching a fragment calculated based on the value fetched from the cubemap.
I shared the theory with my colleague, Roberto Lopez Mendez, and within a couple of hours we had a working prototype running in Unity. We were very pleased to see the prototype achieve very efficient soft shadows while maintaining a high level of visual quality and we decided to use this technique in the Ice Cave demo which was premiered at GDC 2015.
I only recently got back from staffing the ARM booth at GDC and I had a fantastic time meeting so many interesting and knowledgeable people in the game industry. I was really delighted to be able to show the Ice Cave demo and explain all the visual effects we used. The shadow technique in particular garnered a lot of interest with many developers keen to use it in their own engine.
In this blog I would like to give a technical overview of what you need to do in order to make the shadow technique work in your own projects. If you are not familiar with the reflections based on local cubemaps technique, I recommend reading my colleague's blog on the topic. There are also many other internet resources on the subject.
Why did I mention reflections based on local cubemaps? The main reason is that once you have implemented these reflections you are almost there with the shadow technique. The only additional thing you need to do alongside generating a reflection cube is to store alpha along with the RGB colours.
The alpha channel (transparency) represents the amount of light entering the local environment. In your scene, attach the cubemap texture to the fragment shaders that are rendering the static and dynamic objects on which you want to apply shadows. In the fragment shaders, build a vector from the current fragment to the light position in world space. As long as we are using local cubemaps we cannot use that vector directly to fetch a vector and we need just one more calculation step which is a local correction. In order to do that we need to calculate the intersection of that vector with the bounding volume (the bounding box of the room/environment) and then we need to build another vector from the position where the cubemap has been generated to the intersection point.
This vector (Q and P) can now be used to fetch a texel from the cubemap. Once you have fetched the texel you will have information about the amount of shadow/light to be applied on the fragment which is being processed. It is as simple as that.
That was a short summary of the technique, below you will find more details and a step by step guide of what you need to do in order to make shadows look really stunning in your application.
If you are already familiar with reflections based on local cubemaps, this step is exactly the same as creating the reflection cubemap (probe). Moreover, you can reuse your reflection cubemap for this shadow technique. You just need to add an alpha channel.
Let us assume you have a local environment within which you want to apply shadows from the light sources outside of the local environment e.g. room, cave, cage, etc. As an aside, the environment does not have to be enclosed, it can be an open space, but this is a subject for another blog. Let’s focus on the local environment, a simple room, in order to understand better how the technique works.
You need to work out the position from which you will render six faces of the cubemap – in most cases it will be the centre of the environment’s bounding volume (a box). You will require this position not only for the generation of the cubemap, but later the position needs to be passed to shaders in order to calculate a local correction vector to fetch the right texel from the cubemap.
Once you have decided where to position the centre of the cubemap you can render all faces to the cubemap texture and record the transparency (alpha) of the local environment. The more transparent an area is, the more light will come into the environment. If required, you can use the RGB channels to store the colour of the environment for coloured shadows like stained glass, reflections, refractions etc.
Applying shadows to the geometry couldn’t be simpler. All you need to do is build a vector “L” in world space from a vertex/fragment to the light(s) and fetch the cubemap shadow by using this vector.
However there is tiny step you need to do with the “L” vector before fetching each texel. You need to apply local correction to the vector. It is recommended to make the local correction in the fragment shader to obtain more precise shadows. In order to do this you need to calculate the intersection point with the bounding volume (bounding box) of the environment and use this intersection point to build another vector from the cubemap origin position to the intersection point. This gives you the final “Lp” vector which should be used to fetch the texel.
An example code snippet which you can use to correct the vector:
// Working in World Coordinate System. vec3 intersectMaxPointPlanes = (_BBoxMax - V) / L; vec3 intersectMinPointPlanes = (_BBoxMin - V) / L; // Looking only for intersections in the forward direction of the ray. vec3 largestRayParams = max(intersectMaxPointPlanes, intersectMinPointPlanes); // Smallest value of the ray parameters gives us the intersection. float dist = min(min(largestRayParams.x, largestRayParams.y), largestRayParams.z); // Find the position of the intersection point. vec3 intersectPositionWS = V + L * dist; // Get the local corrected vector. Lp = intersectPositionWS - _EnviCubeMapPos;
Then use the “Lp” vector to fetch a texel from the cubemap. The texel’s alpha channel [0..1] provides information about how much light (or shadow) you need to apply for a given fragment.
float shadow = texCUBE(cubemap, Lp).a;
At this point you should have working shadows in your scene. There are then two more minor steps to improve the quality of this shadowing.
Chessroom in shadows
As you may have noticed, we are not using depth information to apply shadows and that may cause some faces to be incorrectly lit when in fact they should be in shadow. The problem only occurs when a surface is facing in the opposite direction to the light. In order to fix this problem you simply need to check the angle between the normal vector and the vertex-to-light vector, L. If the angle, in degrees, is out of the range -90 to 90, the surface is in shadow. Below is a code snippet which will do this check.
if (dot(L,N) < 0) shadow = 0.0;
As always there is room for improvement. The above code will cause each triangle into a hard switch from light to shade, which we would like to avoid. We need a smooth transition which can be done with the following simple formula:
shadow *= max(dot(L, N), 0.0);
shadow – alpha value fetched from the shadow cubemap
L – the vertex-to-light vector in world space
N – the normal vector of the surface, also in world space
The last, and I would say coolest, feature of this shadow technique is its softness. If you do it right you will get realistic shadow softness in your scene.
First of all make sure you generate mipmaps for the cubemap texture and use tri-linear filtering.
Then the only thing you need to do is measure the length of a vertex-to-intersection-point vector and multiply the length by the coefficient which you will have to customize to your scene. For example, in the Ice Cave project we set the coefficient to 0.08. The coefficient is nothing but a normalizer of a maximum distance in your environment to the number of mipmap levels. If you want you can calculate it automatically against bounding volume and mipmap levels. We found it very useful to have manual control to allow us to tweak the settings to suit the environment, which helped us to improve the visual quality even further.
Let us get to the nitty-gritty then. Here we can reuse computations which have been done in calculating local correction. We can reuse the intersection point to build the vertex-to-intersection-point vector and then we need to calculate the distance.
float texLod = length(IntersectPositionWS - V);
Then we multiply the lodDistance by the distance coefficient:
texLod *= distanceCoefficient;
To implement softness, we must fetch the correct mipmap level of the texture by using texCUBElod (Unity) or textureLod (GLSL) function and construct a vec4 where XYZ represents a direction vector and the W component represents LOD (Level of Detail).
Lp.w = texLod; shadow = texCUBElod(cubemap, Lp).a;
High quality smooth shadows in chessroom
Now you should see high quality smooth shadows in your scene.
In order to get the full experience you will need to combine cubemap shadows with the traditional shadow map technique. Even if it sounds like more work it is still worth it as you will only need to render dynamic objects to the shadow map. In the Ice Cave demo, we simply added the two shadow results in the fragment shader to get our final shadow value.
Chessroom with (left) and without (right) shadowmaps
In traditional techniques, rendering shadows can be quite expensive as it involves rendering the whole scene from the perspective of each shadow-casting light source. The technique described in this blog is mostly prebaked (for improved performance) and independent of output-resolution, producing the same visual quality for 1080p as it produces for 720p and other resolutions.
The softness filtration is calculated in hardware (via the texture pipeline) so the smoothness comes almost for free. In fact, the smoother the shadow the more efficient the technique is. This is due to the smaller mipmap levels which result in less data transferred from main memory to the GPU compared to traditional shadow map techniques which require a large kernel to make shadows smooth enough to be more visually appealing. This obviously causes a lot more data traffic and reduces performance.
The quality you get with the shadow technique is even higher than you may expect. Apart from realistic softness, you have very stable shadows with no more shimmering on the edges. Shimmering edges can be observed when using traditional shadow map techniques due to rasterization and aliasing. None of the anti-aliasing algorithms can fix this problem entirely. Obviously enabling multi-sampling helps to improve quality, but the shimmering effect is still visible. The cubemap shadow technique is free from this problem. The edges are stable even if you use a much lower resolution than is used in the render target. You can easily use four times lower resolution than the output and see neither artefacts nor unwanted shimmering. Needless to say that having four times lower resolution saves massively on bandwidth and improves performance!
Let us get to the point with performance data. I already covered the texture fetch above which is very efficient. Apart from fetching a texel you only need to make few calculations related to the local correction. The local correction on ARM Mali GPU-based devices does not take more than three cycles per fragment. This perfectly aligns with the texture fetch pipeline and while the GPU is calculating the next vector for the cubemap, the texture pipeline is preparing the texel for the current fragment.
This technique can be used on any device on the market which supports shaders i.e. OpenGL ES 2.0 and higher. But as everything else in our world it has its downsides. The technique cannot be used for everything in your scene. Dynamic objects, for instance, receive shadows from the cubemap but they cannot be prebaked to the cubemap texture. The dynamic objects should use shadow maps for generating shadows, blended with the cubemap shadow technique.
As a final word we are seeing a lot of implementations of reflections based on local cubemaps and this shadow technique is based on more or less the same paradigm. If you already know where and when to use the reflections based on local cubemaps technique, then you will be able to easily apply the shadow technique to your implementation.
More than that, you will find the shadow technique is less error prone when it comes to local correction. Our brains do not spot so many errors with shadows as we tend to spot in reflections. Please have a look at our latest demo, Ice Cave below, and try to implement the technique in your own projects. I hope you enjoy using it!
[CTAToken URL = "https://developer.arm.com/graphics/game-engines/unity/unity-demos/ice-cave" target="_blank" text="View Ice Cave demo" class ="green"]
Really nice,and how to use the shadow parameter，I implementing it as this article，the surface is black。