Description

In this project, computer vision and motion planning is used to control a robotic arm to grab a pen and leave it iside a box.

Demonstration

The following video is a demonstration of the grab the pen:

Fig. 2 Grab The Pen

Layout

This project contains 2 parts:

1. A computer vision algorithm for recognizing the 3D location of a purple pen, using the realSense RGBD camera.
   The software recognize the pen by its color, and locate its coordinate using the 3D camera.
2. Controlling a PincherX 100 robotic arm to accurately grab the pen and leave it inside the box.

Overview

Part 1 - Pen Recognition

In order to locate the pen in the 3D space we used OpenCV2.
First step is to transform our RGB image to HSV where it is easier to filter the image and isolate only the purple color (see figure 2 left image).
Since we have isolated the pen from the image we are detecting contours and keeping only the ones that exceed a threshold based on the contour area (see figure 2 right image).

Now that we are able to detect the pen at the RGB image we know the location of the pen at the Depth image, and then retrieving the 3D coordinates.

Fig. 2 Pen recognition

Part 2 - Pen Grab

In this part, The interbotix motion and control packages are used to move the arm to a specified coordinates and grab the pen.

The arm motion can be split to 4 parts:

1. Move the arm to the specified coordinates with an open gripper. The coordinates of the pen found using the pen recognition on part 1
2. wait for 4 sec, if the goal coordinates have changed repeat step one otherwise continue
3. Close the gripper on the pen
4. Move the arm to a location above the box
5. Open the gripper and release the pen inside the box