How to Train Neural Networks for Flare Removal

1Rice University 2Google Research 3Adobe Inc.

Abstract

Lens flare is a common artifact in photographs occurring when the camera is pointed at a strong light source. It is caused by either multiple reflections within the lens or scattering due to scratches or dust on the lens, and may appear in a wide variety of patterns: halos, streaks, color bleeding, haze, etc. The diversity in its appearance makes flare removal extremely challenging. Existing software methods make strong assumptions about the artifacts' geometry or brightness, and thus only handle a small subset of flares. We take a principled approach to explicitly model the optical causes of flare, which leads to a novel semi-synthetic pipeline for generating flare-corrupted images from both empirical and wave-optics-simulated lens flares. Using the semi-synthetic data generated by this pipeline, we build a neural network to remove lens flare. Experiments show that our model generalizes well to real lens flares captured by different devices, and outperforms start-of-the-art methods by 3dB in PSNR.

Overall pipeline

Our approach consists of three steps: 1) We generate training input by randomly compositing a flare-free natural image and a flare image. 2) A convolutional neural network is trained to recover the flare-free scene (in which the light source may also have been removed, which is undesirable). 3) After prediction, we blend the input light source back into the output image.

BibTeX

@article{wu2020how,
  title={How to Train Neural Networks for Flare Removall},
  author={Wu, Yicheng and He, Qiurui and Xue, Tianfan and Garg, Rahul and Chen, Jiawen and Veeraraghavan, Ashok and Barron, Jonathan},
  journal={arXiv preprint arXiv:2011.12485},
  year={2020}
}