Computer Sciences and Information Technology
Title : | A deep learning framework for restoration of damaged heritage antiquities using low-cost, hand-held, time-of-flight (ToF) light field depth sensors |
Area of research : | Computer Sciences and Information Technology, Engineering Sciences |
Focus area : | Digitization of cultural heritage |
Principal Investigator : | Dr Rajiv Ranjan Sahay, Assistant Professor, Indian Institute of Technology (IIT) Kharagpur |
Timeline Start Year : | 2018 |
Contact info : | rajiv@ee.iitkgp.ac.in ; anirban@ee.iitkgp.ac.in |
Details
Executive Summary : | Depth scanning for digitization of our cultural heritage would facilitate libraries and museums to reconstruct 3D models of monuments/statues. In the last decade, several projects in developed nations have utilized expensive and cumbersome sensors such as laser scanners to reconstruct 3D models of famous heritage sites. Today sophisticated computer vision algorithms and low-cost sensors are available for digital archiving of heritage antiquities for posterity. Investigators, therefore, propose a framework for the restoration and enhancement of damaged heritage structures using inexpensive, hand-held ToF and light-field cameras. Depth maps acquired from multiple views of the scene will be utilized for reconstructing a 3D mesh representation of the damaged structures. However, depth maps obtained from such low-cost hand-held cameras are of poor quality and necessitate the application of appropriate enhancement techniques. Generally, depth maps captured by such low-cost sensors suffer from missing data at occlusions, secularities, low resolution and noise. A more challenging problem arises when the heritage object is itself degraded or defaced by various weathering agents such as wind, sea, human factors, etc. In that case, even using expensive laser scanners does not solve the problem of faithful 3D reconstruction of the cultural artifact. Depth completion and depth super-resolution have been investigated by several researchers in the literature. Existing methods for depth restoration suffer from artifacts when dealing with large regions of missing data. Hence, novel techniques for reconstructing heritage statues containing large degraded regions are required. Today, deep learning has emerged as a new paradigm in computer vision which has set new benchmarks in several traditional research problems such as image inpainting and super-resolution. Specifically, generative adversarial networks (GAN) which have been proposed very recently have shown good promise for the inpainting of large missing regions in images. However, it is yet to be fully exploited for depth map enhancement. Hence, Investigators propose to exploit GANs to build more accurate and efficient solutions for depth map inpainting and super-resolution. In order to perform depth scanning of heritage antiquities, investigators plan to employ a heterogeneous acquisition system that will combine noisy and low spatial resolution depth data with high-resolution image data. Investigators will also collect ground truth depth data by laser scanner to validate the proposed algorithms for depth enhancement. They will develop novel deep learning algorithms to fuse the enhanced depth maps and perform 3D reconstruction of the underlying broken/defaced heritage structure. Apart from evaluating the proposed method on the available benchmark depth datasets in the literature, they propose to develop and release a new specialized depth dataset of Indian historical monuments for testing and benchmarking 3D depth map completion algorithms. |
Co-PI: | Dr Anirban Mukherjee, Associate Professor, Indian Institute of Technology (IIT) Kharagpur |
Total Budget (INR): | 49,23,908 |
Achievements : | Investigators are still in the phase of collection and experimentation with different algorithms. |
Organizations involved