Light Field Denoising by Sparse 5D Transform Domain Collaborative Filtering

Abstract

In this paper, we propose to extend the state-of-the-art BM3D image denoising filter to light fields, and we denote our method LFBM5D. We take full advantage of the 4D nature of light fields by creating disparity compensated 4D patches which are then stacked together with similar 4D patches along a 5th dimension. We then filter these 5D patches in the 5D transform domain, obtained by cascading a 2D spatial transform, a 2D angular transform, and a 1D transform applied along the similarities. Furthermore, we propose to use the shape-adaptive DCT as the 2D angular transform to be robust to occlusions. Results show a significant improvement in synthetic noise removal compared to state-of-the-art methods, for both light fields captured with a lenslet camera or a gantry. Experiments on Lytro Illum camera noise removal also demonstrate a clear improvement of the light field quality.

Publication
IEEE International Workshop on Multimedia Signal Processing
Date

Additional results

Visual results complementing the paper are shown below. We then show in Tables 1 and 2 the average results presented in the paper. The corresponding detailed results are shown in Table 3 and 4.

Visual results

We show here side by side comparisons of noisy and de-noised light fields. On the top left corner of each video is highlighted the sub-aperture image being displayed. Note that some videos may exhibit encoding artifacts.

First, we show results for different synthetic noise levels where σ corresponds to the white gaussian noise standard deviation (click on a σ to start a video).

Below, we show results for lenslet camera (Lytro) noise removal.

Average PSNR results

The ΔPSNR lines in Tables 1 and 2 correspond to the PSNR gap between the proposed approach and the best state-of-the-art method. Values highlighted in bold correspond to the best performing method for a given noise level.

Table 1 - Average denoising performances in PSNR for the EPFL dataset (Lytro Illum).
σ=10σ=20σ=30σ=40σ=50
HF4D 31.07025.79822.60720.33818.586
BM3D 35.42132.85231.35730.24729.321
BM3D EPI 36.08833.47631.90530.71229.671
VBM4D 36.07533.52231.92330.67429.630
VBM4D EPI 36.12933.51031.92530.71929.721
LFMB5D 1st step 34.38832.81031.68430.74329.911
LFMB5D 2nd step 36.50334.21432.86831.84330.987
ΔPSNR 0.3740.6920.9431.1241.266
Table 2 - Average denoising performances in PSNR for the Stanford dataset (Gantry).
σ=10σ=20σ=30σ=40σ=50
HF4D 30.57725.43222.21319.92118.164
BM3D 38.80535.26833.12631.56030.267
BM3D EPI 38.89535.64533.48931.89630.594
VBM4D 39.26935.58833.21231.43630.019
VBM4D EPI 38.69735.60433.55832.02630.809
LFMB5D 1st step 39.34035.81733.37731.49629.971
LFMB5D 2nd step 40.38937.77236.03134.671333.511
ΔPSNR 1.1202.1272.4732.6452.701

Detailed PSNR results

Table 3 - Denoising performances in PSNR for each light field in the EPFL dataset (Lytro Illum).
Ankylosorus & Dipplodocus 1BikesColor Chart 1Danger de MortDesktopFlowersFountain & Vincent 2Friends 1ISO Chart 12Magnets 1Stone Pillar OutsideVespa
HF4D
σ=10 30.88130.94031.03831.46031.16031.30230.79130.98730.90130.94330.97331.463
σ=20 25.38325.62225.95726.08126.19626.07125.65425.93325.47825.39325.68826.116
σ=30 22.04722.45922.77522.95323.07222.96522.47822.83222.13922.00622.55622.999
σ=40 19.67220.24720.48920.75820.81620.76620.20020.61419.75619.58320.34520.810
σ=50 17.84618.55318.71519.06919.06319.06418.42618.89017.92517.72118.63419.121
BM3D
σ=10 36.09434.76035.71135.03935.70634.25034.30835.51834.94836.04433.92838.750
σ=20 34.60431.83433.64032.08233.03430.99631.37332.80432.53734.48530.87036.310
σ=30 33.66130.17532.42630.36831.52829.18229.73431.23131.02733.40529.31934.226
σ=40 32.86528.96831.45629.13630.43227.91028.55930.08329.87632.45228.31032.919
σ=50 32.14027.93930.60628.14029.53126.89327.57429.13528.91231.56427.56531.849
BM3D EPI
σ=10 36.13135.58934.97736.09737.06135.83134.63736.76335.06536.28635.28439.332
σ=20 34.75432.63532.72633.07733.89432.53431.81834.14132.44034.88132.28036.532
σ=30 33.85330.97931.25631.27531.90330.60730.19932.49430.96633.92130.76534.638
σ=40 32.95129.78330.12629.83930.78129.18829.03231.19229.96332.93729.65833.094
σ=50 32.03228.77929.10228.54229.86627.99228.12630.07929.21931.93628.63731.746
VBM4D
σ=10 35.96635.62835.40136.00836.46435.96734.74236.63135.32636.10135.46739.195
σ=20 34.21932.77233.30233.04933.86932.77632.10433.91433.04034.42132.48336.310
σ=30 33.03631.06532.01431.26632.24230.84030.55732.24131.54933.19030.71434.358
σ=40 32.01329.79230.94729.94130.96529.40629.39130.96530.35032.10329.38832.831
σ=50 31.07428.76229.99928.86829.89128.28228.43029.90029.31631.11728.32731.592
VBM4D EPI
σ=10 36.37435.69235.51536.05936.46735.64834.63836.68535.33436.32635.53739.271
σ=20 34.30332.86533.08133.17533.86032.70831.89234.05732.88534.33032.58936.376
σ=30 33.02531.18531.61831.44632.31730.96630.30032.44531.39133.05830.91834.433
σ=40 31.97829.95130.46730.17231.13129.68929.11931.22530.24232.00629.72132.927
σ=50 31.04528.96029.50229.14530.12928.67828.16130.20329.28031.08128.76731.697
LFMB5D 1st step
σ=10 35.26933.12333.28734.71734.30034.01632.88534.82033.79235.16734.17337.103
σ=20 34.33331.59432.09832.66032.77332.09731.11733.28031.99734.43531.74535.597
σ=30 33.69030.41531.34331.20731.65230.56729.95832.07130.79133.77230.36734.370
σ=40 33.02429.45930.68830.01130.72529.26629.08031.08929.85133.07329.39333.259
σ=50 32.33428.61630.03728.99329.88128.22128.36630.16629.05532.29028.57732.396
LFMB5D 2nd step
σ=10 35.854 36.275 35.263 36.86836.76736.81334.94137.40635.661 36.062 36.21339.910
σ=20 34.79133.577 33.612 33.95434.53833.69032.33434.95833.42335.20032.94237.549
σ=30 34.21632.00232.73632.29533.19231.77930.87333.48231.97534.56531.27236.030
σ=40 33.66730.85132.02831.03732.15630.35229.87232.38230.91033.93830.14334.773
σ=50 33.08629.90831.39030.02731.29229.24529.10631.46330.04033.26729.24633.769
Table 4 - Denoising performances in PSNR for each light field in the Stanford dataset (Gantry).
AmethystBraceletChessEucalyptus FlowersJelly beansLego BulldozerLego KnightsLego TruckTarot Cards and Crystal Ball (Large Angle)Tarot Cards and Crystal Ball (Small Angle)The Stanford BunnyTreasure Chest
HF4D
σ=10 31.37430.46431.36229.46631.33029.85030.35231.31529.08031.05031.36129.913
σ=20 25.86225.20125.81625.17525.46825.24325.21026.00024.73825.43025.57825.463
σ=30 22.54321.86422.57622.15422.03522.18121.96622.81321.66022.08822.18922.492
σ=40 20.20019.50820.28419.90819.67119.96719.64620.57019.39619.75619.81820.325
σ=50 18.40617.73518.52518.14317.90118.25917.86718.84517.65017.98918.01718.633
BM3D
σ=10 37.54938.43840.82935.17744.86537.42940.13538.39438.15437.91439.81436.962
σ=20 33.96434.56537.54231.24641.86733.81436.66634.91734.57834.26736.74433.047
σ=30 31.92532.29035.39829.09339.69631.68034.48932.79632.44532.09934.78830.816
σ=40 30.51830.64533.77727.60037.98530.15632.87231.25930.88430.51133.32029.195
σ=50 29.42229.27132.41226.35436.54828.93431.56630.03729.58629.17532.13727.764
BM3D EPI
σ=10 38.91538.87941.02935.68843.87137.07638.43039.17136.68437.95541.33837.703
σ=20 36.23435.47238.87131.76041.68533.12234.47336.18032.28334.97039.32933.358
σ=30 34.48132.17437.20729.39840.22930.71432.31733.97029.49333.04037.75431.090
σ=40 31.20031.43035.89427.74839.08029.03131.01532.30727.59631.55736.48429.412
σ=50 29.87528.99234.80026.52938.12727.81430.11731.00326.23230.33135.30527.997
VBM4D
σ=10 38.56138.99741.00436.53143.94138.33440.25339.16437.90638.43340.22537.882
σ=20 34.97435.15637.39132.91740.09834.55836.52535.58034.19034.83036.73634.094
σ=30 32.65732.74834.92130.67937.41132.14234.06633.21131.89332.55134.47631.792
σ=40 30.94430.95933.04129.02035.34130.37032.21231.44030.19230.85332.76330.099
σ=50 29.59229.52631.53527.70433.66728.98030.72930.03528.83429.49431.38428.755
VBM4D EPI
σ=10 38.41738.13941.13334.46844.27836.95939.32038.80936.96037.85340.92737.097
σ=20 35.77234.89738.52131.82941.13133.52136.22735.46033.35035.07738.12733.340
σ=30 34.01632.81036.60829.99938.95031.35534.07733.23631.10533.23336.24531.061
σ=40 32.70831.26335.09528.63437.26729.78532.44331.55629.49631.85134.83229.384
σ=50 31.66830.03333.86427.56135.90128.55631.17530.20228.24130.74933.70628.054
LFMB5D 1st step
σ=10 38.84738.90740.79736.71043.92638.63040.35139.38537.04338.43240.82638.229
σ=20 35.52935.03737.41533.39940.39935.02636.97235.87333.35535.22836.78734.786
σ=30 33.19032.51535.16231.30637.29432.62434.78133.47930.94032.85633.91132.469
σ=40 31.36830.85733.46629.65834.40030.82833.03831.66429.12230.77332.07830.702
σ=50 29.86229.58732.04628.17432.25729.32131.58830.17627.74929.02130.74329.129
LFMB5D 2nd step
σ=10 39.59339.88442.15837.26345.63639.60341.59840.35738.35439.34441.88139.003
σ=20 37.02036.98840.04534.33243.82636.76238.93637.85235.37136.67339.38336.071
σ=30 35.25735.30538.43732.57542.28134.87837.16736.06233.61835.007 37.560 34.227
σ=40 33.87434.07737.05631.28641.01633.43835.76134.59232.30133.708 36.090 32.858
σ=50 32.68533.02035.85130.14239.92032.15534.59833.34331.26932.615 34.837 31.693