Urban 3D segmentation and modelling from street view images and LiDAR point clouds

Pouria Babahajiani, Lixin Fan, Joni-Kristian Kämäräinen, Moncef Gabbouj

    Research output: Contribution to journalArticleScientificpeer-review

    21 Citations (Scopus)
    1143 Downloads (Pure)


    3D urban maps with semantic labels and metric information are not only essential for the next generation robots such autonomous vehicles and city drones, but also help to visualize and augment local environment in mobile user applications. The machine vision challenge is to generate accurate urban maps from existing data with minimal manual annotation. In this work, we propose a novel methodology that takes GPS registered LiDAR (Light Detection And Ranging) point clouds and street view images as inputs and creates semantic labels for the 3D points clouds using a hybrid of rule-based parsing and learning-based labelling that combine point cloud and photometric features. The rule-based parsing boosts segmentation of simple and large structures such as street surfaces and building facades that span almost 75% of the point cloud data. For more complex structures, such as cars, trees and pedestrians, we adopt boosted decision trees that exploit both structure (LiDAR) and photometric (street view) features. We provide qualitative examples of our methodology in 3D visualization where we construct parametric graphical models from labelled data and in 2D image segmentation where 3D labels are back projected to the street view images. In quantitative evaluation we report classification accuracy and computing times and compare results to competing methods with three popular databases: NAVTEQ True, Paris-Rue-Madame and TLS (terrestrial laser scanned) Velodyne.

    Original languageEnglish
    Pages (from-to)679–694
    Number of pages16
    JournalMachine Vision and Applications
    Issue number7
    Publication statusPublished - 2017
    Publication typeA1 Journal article-refereed


    • LiDAR
    • Point cloud
    • Robotics
    • Semantic segmentation
    • Street view
    • Urban 3D

    Publication forum classification

    • Publication forum level 2

    ASJC Scopus subject areas

    • Software
    • Hardware and Architecture
    • Computer Vision and Pattern Recognition
    • Computer Science Applications


    Dive into the research topics of 'Urban 3D segmentation and modelling from street view images and LiDAR point clouds'. Together they form a unique fingerprint.

    Cite this