Sampling, rendering, and denoising light fields for challenging environments

Abstract

Light field imaging is an important area of image-based rendering (IBR) that is on the cusp of becoming an important area in the field of robotics. While born into the field of graphics, light fields have evolved into and taken the form of high resolution depth sensors that can simultaneously generate high quality views of a subject. Modern focused light field cameras are capable of capturing video at over 160 frames per second (FPS) and producing depth at half that rate. These modern light field cameras have demonstrated impressively high resolution depth results, with much of the appeal being in that it is a passive, single sensor system with the ability to generate depth from a single shot. However, with such high framerates, the amount of light collected decreases, affecting light field rendering and depth estimation processes. The objective of this thesis is to build upon existing work in light fields and robotics. We compare methods for collecting light fields and develop a system around focused light fields that makes them more robust to challenging conditions, particularly in very low-light scenarios. Our first contribution explores the usage of robotic platforms for collecting light fields using a traditional, 2D camera and compares localization accuracy with popular visual techniques such as structure from motion (SfM) and simultaneous localization and mapping (SLAM) We shift our focus for our second contribution to the sensor that said robot might be equipped with, in this case a focused light field camera. We develop a method based on deep learning that allows for enhancement of low-light light fields captured under a time constraint. We build upon this by developing a simple algorithm that generates smooth light field video in the absence of a reference video. To accomplish this we also create two datasets: one containing low-light light field stills at various shutter speeds, and another containing low-light light field videos captured at a single shutter speed. In our third contribution we further build on the work of the previous contribution by exploiting unique aspects of the light field. In particular, we develop a network that can learn depth from the low resolution microlens images (MLIs) that make up the focused camera light field. As well, we develop a deep neural network (DNN) that takes into account the light field context and angular feature redundancies, taking inspiration from burst denoising methods for single shot light fields. In conclusion, we show that robot platforms are the best platforms for sensor placement in light field acquisition and set a standard where none existed in low-light focused light field enhancement.