Sampling, rendering, and denoising light fields for challenging environments
Abstract
Light field imaging is an important area of image-based rendering (IBR) that
is on the cusp of becoming an important area in the field of robotics. While born
into the field of graphics, light fields have evolved into and taken the form of high
resolution depth sensors that can simultaneously generate high quality views of a
subject.
Modern focused light field cameras are capable of capturing video at over 160
frames per second (FPS) and producing depth at half that rate. These modern
light field cameras have demonstrated impressively high resolution depth results,
with much of the appeal being in that it is a passive, single sensor system with the
ability to generate depth from a single shot. However, with such high framerates,
the amount of light collected decreases, affecting light field rendering and depth
estimation processes.
The objective of this thesis is to build upon existing work in light fields and
robotics. We compare methods for collecting light fields and develop a system
around focused light fields that makes them more robust to challenging conditions,
particularly in very low-light scenarios.
Our first contribution explores the usage of robotic platforms for collecting light
fields using a traditional, 2D camera and compares localization accuracy with popular visual techniques such as structure from motion (SfM) and simultaneous localization and mapping (SLAM)
We shift our focus for our second contribution to the sensor that said robot might
be equipped with, in this case a focused light field camera. We develop a method
based on deep learning that allows for enhancement of low-light light fields captured
under a time constraint. We build upon this by developing a simple algorithm that
generates smooth light field video in the absence of a reference video. To accomplish
this we also create two datasets: one containing low-light light field stills at various
shutter speeds, and another containing low-light light field videos captured at a
single shutter speed. In our third contribution we further build on the work of the previous contribution by exploiting unique aspects of the light field. In particular, we develop
a network that can learn depth from the low resolution microlens images (MLIs)
that make up the focused camera light field. As well, we develop a deep neural
network (DNN) that takes into account the light field context and angular feature
redundancies, taking inspiration from burst denoising methods for single shot light
fields.
In conclusion, we show that robot platforms are the best platforms for sensor
placement in light field acquisition and set a standard where none existed in low-light
focused light field enhancement.