- Author : Pablo Arias pariasm@gmail.com, see
AUTHORS - Copyright : (C) 2019, Pablo Arias pariasm@gmail.com
This code provides an implementation of the video denoising methods described in:
Please cite the paper if you use results obtained with this code in your research.
The code is in C with some BASH helper scripts. Known dependencies are:
- OpenMP: parallelization [optional, but recommended]
- libpng, libtiff and libjpeg: image i/o
- libfftw3-dev: computing the DCT of patches
- GNU parallel: parallelization in some helper scripts
Compilation was tested on Ubuntu Linux 16.04 and 18.04. Configure and compile the source code using cmake and make. It is recommended that you create a folder for building:
$ mkdir build; cd build
$ cmake ..
$ make
NOTE: By default, the code is compiled with OpenMP multithreaded
parallelization enabled (if your system supports it). Use the
OMP_NUM_THREADS enviroment variable to control the number of threads
used.
The compilation populates build/bin with the following binaries:
nlkalman-fltnon-local Kalman filtering of a framenlkalman-smoRTS smoother of a frametvl1flowcompute TV-L1 optical flow between two imagesawgnadd noise to an imageiionconvert image to a different formatimprintfdisplay statistics of an image in printf formatplambdaevaluate lambda expression at all pixels of an image.decomposeDCT pyramid decompositionrecomposerecomposition from a DCT pyramid
In addition, the following helper scripts will be installed in bin/
nlkalman-seq.shcomputes NL-Kalman filtering (and optionally) the smoothing over a noisy image sequence.nlkalman-seq-gt.shgiven a clean sequence, adds noise, runsnlkalman-seq.shand computes PSNR.msnlkalman-seq.shmultiscale version of nlkalman-seq.sh (experimental)msnlkalman-seq-gt.shgiven a clean sequence, adds noise, runsmsnlkalman-seq.shand computes PSNR.psnr.shcomputes MSE/RMSE/PSNR between two images
Denoising a noisy sequence
The simplest use is via the helper scripts:
nlkalman-seq.sh /my/video/frames-%03d.png first-frame last-frame sigma out-folder [filt-params] [smoo-params] [flow-params]
The method reads the video as a sequence of images. The sequence of images is passed
as a pattern in printf format, thus frame-%03d.png means that frames have the following
filenames: frame-001.png, frame-002.png, etc. The first and last frame
numbers have to given, as well as the standard deviation of the noise.
The denoising results are stored in the out-folder. The script produces the following
output sequences:
bflo_%03d.flo: backward optical flow (ie flow from frame t to t-1)bocc_%03d.png: masks of backwards occluded pixelsflt1_%03d.tif: output of 1st NL-Kalman filtering iterationflt2_%03d.tif: output of 2nd NL-Kalman filtering iteration (if 2nd iteration is enabled)
If smoothing is performed, the following additional sequences will also be left in out-folder
fflo_%03d.flo: forward optical flow (from from frame t to t+1)focc_%03d.png: masks of forward occluded pixelssmo1_%03d.tif: output of the smoothing pass
You can pass options to the filtering and the smoothing thought the optional
arguments [filt-params] and [smoo-params]. For a list of
all parameters run nlkalman-flt -h and nlkalman-smo -h. If no parameters are
given, the parameters are set automatically based on the noise level sigma. The
filtering and smoothing parameters have to be passed between quotes.
Some examples:
# Run the denoising with automatic parameters from frame 3 to 56 with noise 10.
nlkalman-seq.sh /my/video/frames-%03d.png 3 56 10 out/path
# Set patch size during both filtering iterations at 12x12, toggle verbose output:
nlkalman-seq.sh /my/video/frames-%03d.png 3 56 10 out/path "--f1_p 12 --f2_p 12 -v 1"
# Filter with automatic parametes, smoothing with a patch size of 6x6
nlkalman-seq.sh /my/video/frames-%03d.png 3 56 10 out/path "" "--s1_p 6"
# Filter with automatic parametes, do not enable smoothing
nlkalman-seq.sh /my/video/frames-%03d.png 3 56 10 out/path "" "no"
Finally, you can also provide a string with parameters for the optical flow and occlusions detection. The string has to have 6 numbers, three parameters for the backward optical flow computed during filtering and three for the forward flow computed for the smoothing pass:
"fscale-filt data-weight-filt occl-th-filt fscale-smoo data-weight-smoo occl-th-smoo"
fscalefinest scale of the multiscale TV-L1:0means the finest scale, and1means that the optical flow is computed at half resolution and then upscaled (default is1).data-weightdata-attachment weight to control the smoothness of the flow (default is0.25)occl-ththreshold on the divergence of the flow use to compute occlusions (default is0.75)
For example, to run the denoising with automatic filtering and smoothing parameters but with custom parameters for the optical flows
nlkalman-seq.sh /my/video/frames-%03d.png 3 56 10 out/path "" "" "1 0.2 .75 0 0.2 0.75"
Add noise, denoising and compute PSNR
Finally, if you want to compute the flow on a sequence with synthetic noise and then compute the PSNR on the result, you can use:
nlkalman-seq.sh /my/clean/video/frames-%03d.png first-frame last-frame sigma out-folder [filt-params] [smoo-params] [flow-params]
In addition to the previous outputs, you will find in out-folder:
out-folder/%03d.tif: frames with noise added (as tif floating point images)out-folder/measures: text file with RMSE and PSNR computed globally and per-frame
The following libraries are also included as part of the code:
- For computing the optical flow: the IPOL implementation of the TV-L1 optical flow method of Zack et al..
- For image I/O: Enric Meinhardt's iio.
- For basic image manipulation: a reduced version of Enric Meinhardt's imscript.
- For command line parsing: Yecheng Fu's argparse.
- For multiscale denoising: Pierazzo and Facciolo's DCT multiscaler
The project is organized as follows
root/
├── lib/ 3rd party libraries
├── scripts/ helper scripts
└── src/ kalman filtering and smoothing code
The code of BNLK is licensed under the GNU Affero General Public License v3.0,
see LICENSE. The 3rd party libraries are distributed under their own licences
specified inside each folder.