Detecting and Tracking Skin Lesions on 3D Whole Body Skin Images using Deep Learning

Abstract

Given the increasing need for automatic diagnosis of skin-related diseases, it’s important to leverage full-body image acquisition as well as AI tools. Many computer-aided approaches to analyze skin conditions rely on static 2D color images; however, limited works have focused on 3D whole-body images or tracking lesions across time. Imaging the entire body gives additional context that may not be visible in a single localized photograph. Further, imaging and tracking skin conditions across time may allow for lesion changes or the progression of treatments to be monitored.
Given the importance of early detection, diagnosis, and treatment of skin conditions, we propose a new dataset and a novel computational pipeline to detect and track lesions using 3D human skin images. Our pipeline is based on recent advances in computer vision for multi-view 3D colored skin surface reconstruction (such as structured light 3D scanning using Artec EVA scanner and 3D photogrammetry scanning using smartphones), 3D geometry processing, point correspondence, and deep learning-based object localization and classification (such as Faster R-CNN). To perform analysis on 3D whole-body scans, we map the 3D scans of human subjects
to 2D texture images, where a trained region-based convolutional neural network localizes the lesions within the 2D domain. For subjects with multiple scans, we apply a matching algorithm to track and monitor the lesions across time. We test our methods on a scanned mannequin, with varying postures and artificial skin lesions, as well as a newly created synthetic 3D dataset comprising 900 whole-body, skin-colored meshes with lesions of varying appearances and diagnoses under different postures, and a public digital 3D dataset with manually annotated lesions. 3D full-body imaging and deep learning-based lesion detection methods show great potential in the future of automatic skin-related disease diagnosis.