My question is about "compute_ideal_colors_and_weight_1plane" in astc encoder , it computes a color line to fitting the texels in blocks, but it doesn't use the least squal method to fit the texels , instead it use the average color and dominant directions. I guess the least square method will get less fitting errors , am I right? and I also found the fitting errors is not included in the mode selection, instead it use the color line weight as the ideal weight which error is 0, but actually it has included different fitting error.
Thanks for reading
The method we use for finding the dominant direction is a bit unusual, but has a very similar outcome to least squares. We have tried using a more traditional PCA implementation instead, and it does improve quality slightly but it's quite a bit slower so it doesn't really pay for itself. In either case - both approaches ultimately end up with a direction line - we interpolate along a line segment so we always need to end up with that.
I'm not sure what you mean by fitting error not being used for mode selection. We test a lot of modes, so what we care about is the error after the mode selection has been applied. This is what compute_error_of_weight_set_1plane does, and it is indeed doing least sum of squares. The effect of quantization is non-linear, and there are multiple levels of quantization used that play off against each other (weight value, weight grid density) so this nearly always has to turn into a "try it and see" approach.
Note that there is no "right answer" here. It's possible to write a higher quality compressor that does a better job - use PCA, test more modes, search more of the search space. That's definitely possible. The trick is making it fast enough that developers actually want to use it.
The compressor is open source, so PRs are welcome if anyone has any bright ideas to improve things ;P
Thanks for answering my questions. For fitting error I mean the distance colors in the block to the color line, but I finally realized that it would get the same result when it is included in the block mode selection. Thanks .