A method for understanding and digitizing manipulation activities using programming by demonstration in robotic applications

Robots are flexible machines, where the flexibility is achieved, mainly, by the re-programming of the robotic system. To fully exploit the potential of robotic systems, an easy, fast, and intuitive programming methodology is desired. By applying such methodology, robots will be open to a wider audience of potential users (i.e. SMEs, etc.) since the need for a robotic expert in charge of programming the robot will not be needed anymore. This paper presents a Programming by Demonstration approach dealing with high-level tasks taking advantage of the ROS standard. The system identifies the different processes associated to a single-arm human manipulation activity and generates an action plan for future interpretation by the robot. The system is composed of five modules, all of them containerized and interconnected by ROS. Three of these modules are in charge of processing the manipulation data gathered by the sensors system, and converting it from the lowest level to the highest manipulation processes. In order to do this transformation, a module is used to train the system. This module generates, for each operation, an Optimized Multiorder Multivariate Markov Model, that later will be used for the operations recognition and process segmentation. Finally, the fifth module is used to interface and calibrate the system. The system was implemented and tested using a dataglove and a hand position tracker to capture the operator’s data during the manipulation. Four users and five different object types were used to train and test the system both for operations recognition and process segmentation and classification, including also the detection of the locations where the operations are performed.