Efficient 3D visual perception for robotic rock breaking

Longchuan Niu, Ke Chen, Kui Jia, Jouni Mattila

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

1 Citation (Scopus)


In recent years, underground mining automation (e.g., the heavy-duty robots carrying rock breaker tools for secondary breaking) has drawn substantial interest. This breaking process is needed only when over-sized rocks threaten to jam the mine material flow. In the worst case, a pile of overlapped rocks can get stuck on top of a crusher's grate plate. For a human operator, it is relatively easy to make the decisions about the rock locations in the pile and the order of rocks to be crushed. In an autonomous operation, a robust and fast visual perception system is needed for executing robot motion commands. In this paper, we propose a pipeline for fast detection and pose estimation of individual rocks in cluttered scenes. We employ the state-of-art YOLOv3 as a 2D detector to perform 3D reconstruction from point cloud for detected rocks in 2D regions using our proposed novel method, and finally estimating the rock centroid positions and the normal-to-surface vectors based on the predicted point cloud. The detected centroids in the scene are ordered according to the depth of rock surface to the camera, which provides the breaking sequence of the rocks. During the system evaluation in the real rock breaking experiments, we have collected a new dataset with 4780 images having from 1 to 12 rocks on a grate plate. The proposed pipeline achieves 97.47% precision on overall detection with a real-time speed around 15Hz.

Original languageEnglish
Title of host publication2019 IEEE 15th International Conference on Automation Science and Engineering, CASE 2019
Number of pages7
ISBN (Electronic)9781728103556
Publication statusPublished - 1 Aug 2019
Publication typeA4 Article in a conference publication
EventIEEE International Conference on Automation Science and Engineering - Vancouver, Canada
Duration: 22 Aug 201926 Aug 2019

Publication series

NameIEEE International Conference on Automation Science and Engineering
ISSN (Print)2161-8070
ISSN (Electronic)2161-8089


ConferenceIEEE International Conference on Automation Science and Engineering

Publication forum classification

  • Publication forum level 1

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'Efficient 3D visual perception for robotic rock breaking'. Together they form a unique fingerprint.

Cite this