Jan Allmenröder jallmenroeder.de

← All posts

Linearly Transformed Spherical Harmonics

The arcade scene, a cabinet and two plastic chairs in a room lit by a rectangular area light on the wall, whose glossy reflection spreads across the cracked red and white floor tiles. Specular highlights from area lights like this are what LTC and LTSH approximate.

LTSH got accepted as a Poster at I3D 2021.

Here I present my Bachelor’s Thesis and its implementation about Linearly Transformed Spherical Harmonics (LTSH). To keep things short I expect readers to be familiar with bidirectional reflectance distribution functions (BRDFs). If this is new to you, check out section 5.2 of my thesis.

Direct lighting with area lights is complicated, as it involves integration of the BRDF over spherical domains. Usually, modern BRDFs are too complex to “simply integrate” them. This calls for creative alternatives to solve the integral in real-time applications. Sampling the light source is too expensive for most use cases (although there is a shift happening through the advances in real-time ray tracing).

An existing technique uses linearly transformed cosines to approximate BRDFs and performs shading with them. They achieve good approximations, especially for highly specular BRDFs but the limited flexibility of the cosine function becomes apparent for rough BRDFs and at grazing angle. We swap out the rigid cosine function with more versatile spherical harmonic (SH) expansions to improve the quality of the BRDF approximation. Our technique produces realistic renderings and offers improvement in image quality compared to linearly transformed cosines, but it performs significantly slower.

Linearly Transformed Cosines

In 2016, Heitz et al. derived Linearly Transformed Cosines (LTC). There is a short intuitive introduction to the technique at the linked page, so I’ll only give it a quick rundown here.

The idea is to linearly transform a cosine function so it matches the BRDF as close as possible. Now we can approximate the integral of the BRDF with the integral of the linearly transformed cosine function. And the integral over the light polygon of the linearly transformed cosine is equivalent to the integral over the inversely transformed polygon of the untransformed cosine function. We need to account for a Jacobian term of our linear transformation but I’ll leave this out of here for brevity. Since the cosine function possesses an analytic integral, we can now solve the rendering integral.

Diagram of the LTC idea: on the left, the light polygon P over a narrow, BRDF-like lobe made by linearly transforming a cosine; the inverse transform M inverse leads to the right, where the transformed polygon P org lies over the broad original cosine, whose integral over the polygon has an analytic solution.
Integral over light polygon of linearly transformed spherical function (left) is equivalent to integral over inversely transformed light polygon of original spherical function (right)

At a specific shading point we have the view direction and material properties. For the GGX BRDF this would mean we have four input values: the view direction consisting of two angles θ and ϕ, a surface roughness a and an index of refraction. To reduce the dimensionality of the problem, we only look at isotropic BRDFs to drop the ϕ-component of our view direction. We also use a fixed value for the index of refraction. With this we precompute a 2D look-up table. Heitz et al. achieved good results with a 64×64 table of value pairs for the θ component of the view direction and a roughness value a and we will follow their example.

Our approach

As the name of our technique suggests, LTSH is pretty close to LTC. Instead of using the cosine function to approximate the BRDF, we use spherical harmonic (SH) expansions. Spherical harmonics are “Fourier expansions on the sphere” figuratively spoken. By taking linear combinations of the SH basis functions, we can approximate any spherical function. Much like Fourier expansions, the higher the order of your SH expansion the closer your approximation gets as higher frequencies are added in. But directly using an SH expansion to approximate the BRDF would need a lot of SH bands, especially for highly specular BRDFS and is thus not feasible. However, by linearly transforming the BRDF, good approximations are found even for low order SH expansions.

Spherical harmonics become an eligible candidate because both Belcour et al. and Wang and Ramamoorthi simultaneously derived a closed form expression for the integration of SH expansions over spherical domains. In their paper, Belcour et al. state “One avenue of future work could combine these ideas to explore the expressive power of linearly-warped SH.” which is ultimately where the idea for my thesis came from. So we approximate the BRDF with LTSH and then calculate the integral over the inversely transformed light polygon of the original SH expansions. This adds some degrees of freedom to the BRDF approximation process as we now modify both the linear transformation and the SH expansion itself. A useful side fact: the cosine function is a subset of SH expansions with a degree of 1 or higher, so LTC is a subset of LTSH which means the quality of our technique is never worse than LTC in theory.

I won’t go into detail of the BRDF approximation or how the SH integral is actually solved, as it is mathematically involved and would be repetitive to what I wrote in my thesis. Refer to section 6 for BRDF fitting and section 7 for shading with LTSH.

The results

Now I showcase some rendering of the implementation. If you want to try it out yourself, there is a precompiled Windows release. Go ahead and play around, move the light source and render using different techniques. Since the technique only accounts for specular shading, using the specular debug view helps in spotting differences between techniques. Here is an exemplary screenshot:

Screenshot of the renderer: an arcade cabinet with a steering wheel next to an orange chair, its glossy, textured surfaces lit by a rectangular area light on the wall. The image looks plausible, but on its own says little about the accuracy of the technique, which the comparisons below examine.
Rendering of the arcade scene. Our technique (LTSH of order 4) produces plausible results with normal and roughness mapped surfaces and complex geometry.

Although such images look nice, they tell little about the quality of our technique. So let’s look at a comparison of the specular highlight on a simplistic test scene (use the buttons to switch between different techniques):

LTC rendering of the test scene, a rectangular area light above a glossy plane, with the blue inset magnifying the area behind the light and the green inset the edge of the highlight. Its highlight shape is the furthest from ground truth, and the area behind the light is clearly lit where ground truth is dark. LTSH order 2 rendering of the area light above the glossy plane: the highlight shape in the green inset is closer to ground truth than with LTC, and the area behind the light in the blue inset is darker. LTSH order 4 rendering of the area light above the glossy plane: the highlight shape comes closest to ground truth, but the area behind the light source, in the blue inset, is over lit, an artifact ground truth does not show. Ground truth rendering of the area light above the glossy plane, the reference for the other three: the highlight shape the approximations aim for, and the area behind the light, in the blue inset, almost black.

The shape of the specular highlight grows closer to ground truth from LTC to LTSH of order 4, but an artifact on the area behind the light source is over lit.

In the next scenario, we show significant improvement for rough surfaces. LTC has an unnatural highlight due to the rigidity of the cosine function. Even the LTSH of order 2 removes this artifact.

LTC rendering of a rough surface lit by an area light at the right edge, with blue insets magnifying the falloff: the highlight ends in an unnatural hard boundary with black beyond it, because the rigid cosine cannot spread as broadly as the rough BRDF. LTSH order 2 rendering of the rough surface: the hard LTC boundary is gone, and the highlight fades out in a broad, soft glow like ground truth. LTSH order 4 rendering of the rough surface: like order 2, a broad, soft highlight without the LTC artifact, very close to ground truth. Ground truth rendering of the rough surface, the reference: the highlight spreads in a wide, soft glow that fades out gradually, with no hard boundary.

The problem is seen when looking at the BRDF approximations. The BRDF has important contributions over the whole hemisphere. The linear transformation cannot transform the cosine in a way that matches such a broad distribution. With our more versatile SH expansion however, we achieve very close approximations in this scenario.

Heat-map spheres of the three lobes seen from behind. The LTC lobe on the left is saturated over a wide band that ends in a hard edge, a poor match for the broad GGX lobe on the right, while the LTSH order 4 lobe in the middle is nearly identical to GGX.
“Backside” (opposite of the highlight) of LTC (left), LTSH of order 4 (middle), and GGX (right); roughness = 0.63

However, our technique also introduces numerical instability. It becomes apparent when the ratio intensity/area of the light source grows large. We inherit this instability from the SH integration implementation of Wang and Ramamoorthi. For most of the cases this can be avoided by using point lights instead of tiny area lights, but using this technique to emulate line lights or other shapes with small surface areas is not advised.

To give an impression of the quality improvements of LTSH over LTC we look at the square root of the mean squared error over the whole hemisphere. Please note, that we used simple equidistant sampling in the angular domain in our BRDF fitting process. This makes our results prone to error especially for low roughness values. To be fair, the LTC errors presented here are likely worse than those of the original LTC implementation, as they used more sophisticated sampling for their BRDF fitting. But since our technique would benefit from this sampling in the same way as our LTC implementation would, I think it’s still fair to use those values for the sake of comparison here. The colormap is capped at 0.1.

Heat map of LTC error over roughness a (left to right) and view angle θ (top to bottom): the error hits the cap across the whole lower-left triangle of low roughness and grazing angles, rises again at high roughness near normal view, the rough-surface problem seen above, and is low only along a diagonal band. Heat map of LTSH order 2 error on the same axes: low almost everywhere, including at high roughness where LTC struggles; the cap is reached only in a narrow strip of very low roughness and in a corner of low roughness at grazing angles. Heat map of LTSH order 4 error on the same axes: the lowest error of the three, with a thinner raised band near grazing angles than order 2; the remaining capped error sits at low roughness, where the post notes its simple equidistant sampling makes results prone to error.
Square root of mean squared error over hemisphere of LTC (left), LTSH order 2 (middle) LTSH order 4 (right). Visualization for the whole 64×64 look-up table. θ represents the view direction (90 is grazing angle) and a is the roughness.

As you can see, a clear tendency of improvement is visible from LTC over LTSH order 2 to LTSH order 4. Especially the problems that LTC has with highly rough surfaces, as discussed earlier, are visible here.

Another important factor are run times. Our technique introduces significantly higher run times than LTC. Here it is important to note, that our implementation is only a proof of concept. We did not have the time to go into a lot of optimizing. Also, the SH integration code provided Wang and Ramamoorthi is optimized for SH expansions of order 8 and does not provide optimal run times for lower order integration. Since the number of SH basis functions grows quadratically in the SH order, we expect the run time of our technique to also grow quadratically when increasing the number of SH bands used. The baseline is the application running without doing any lighting calculations. For more quality examples and a broader discussion visit the thesis section 7.

Horizontal bar chart of run time in milliseconds, each bar split into the baseline and the lighting pass: the lighting pass more than doubles from LTC to LTSH order 2 and roughly doubles again for order 4, the cost that comes with the quality gain of LTSH.
Baseline: 0.14ms, LTC: 0.44ms, LTSH 2: 0.95ms, LTSH 4: 1.78ms

Conclusion

Our technique offers quality improvement over LTC that comes at a cost. The frame cost for LTSH is significantly higher and grows with the number of SH bands used in a quadratic manner. But using LTSH expansions of different orders makes this approach adaptive to different needs in quality/run times. Unlike LTC, we do not offer textured area lights and alike LTC there is no support for shadows. Our technique needs further investigation and optimization before it is suitable for production use but it is an interesting alternative to LTCs.

I had a lot of fun while researching this and hope my results prove helpful to someone in the future, so I’m happy to share it here. I’d like to thank my advisor Christoph Peters for the large amount of time he put into this thesis and providing me with useful tools for example to create visualizations of spherical functions or an SH evaluation tool. For other graphics related posts check out his blog. Also thanks to NVIDIA for providing the Falcor framework and the ORCA library to help aspiring researchers.