Virtual audit of microscale environmental components and materials using streetscape images with panoptic segmentation and image classification

Microscale environmental components, such as street furniture, sidewalks, and green spaces, significantly enhance street quality when properly identified and managed. Traditional in-person audits are time-consuming, so virtual audits using streetscape images and computer vision have been explored as alternatives. However, these often lack a comprehensive range of microscale components and do not consider attributes like materials. This paper proposes an automatic virtual audit method that recognizes microscale component types and materials in streetscape images using panoptic segmentation and material classification of segmented images of detected components. By surveying components affecting pedestrian-perceived street quality to include as many essential components as possible, 33 types of microscale components, as well as materials of sidewalk pavement, architectural elements, and street furniture, were identified with an overall F1 score of 0.946, demonstrating significantly improved performance compared with previous studies. This approach helps enhance street quality by evaluating built environments through an automatic virtual audit.