View article

Compositing a person into a scene to look like they are really there is a fundamental technique in visual effects, with many other applications such as smartphone photography [Tsai and Pandey 2020] and video conferencing [Hou and Mullen 2020]. The most common practice in film-making has been to record an actor in front of green or blue screen and use chroma-keying [Wright 2013] to derive an alpha matte and then change the background to a new one. However, this does nothing to ensure that the lighting on the subject appears consistent with the lighting in the new background environment, which must be solved with laborious lighting placement or elaborate LED lighting reproduction systems [Bluff et al. 2020; Debevec et al. 2002; Hamon et al. 2014]. Our goal is to design a system that allows for automated portrait relighting and background replacement. There is a significant body of work both in relighting, eg [Barron and Malik 2015; Debevec et al. 2000; Nestmeyer et al. 2020; Sun et al. 2019; Wang et al. 2020; Zhou et al. 2019], and in determining alpha mattes and foreground colors, eg [Cai et al. 2019; Forte and Pitié 2020; Hou and Liu 2019; Lutz et al. 2018; Xu et al. 2017]. A few techniques simultaneously consider foreground estimation and compositing in a unified framework [Wang and Cohen 2006; Zhang et al. 2020b] and produce convincing composites when the input and target lighting conditions are similar. However, the absence of an explicit relighting step limits realism when the input and target illumination conditions are different. To generate convincing relit composites, Einarsson et al.[2006] and Wenger et al.[2005] captured …