Teach Pendant at Fingertips: Intuitive Vision-Based Gesture-Driven Control of Dexter ER2 Robotic Arm

Teaching a robotic arm can be unintuitive and challenging due to the disparity between natural human limb movements and robotic manipulator control. Substantial training is often required to effectively operate such systems. This work presents the development of an intuitive, easy-to-use interface that maps human palm gestures to control the movement of a 5-axis robot, the Dexter ER2. The proposed vision-based system enables the robotic arms to autonomously follow hand gestures through a three-stage methodology: gesture detection, gesture tracking, and robotic manipulator control. A deep learning-based model is employed for accurate gesture recognition, allowing the robot to mimic the controller’s limb movements with minimal cognitive load. For trajectory computation and joint movement generation, an adaptive gradient descent-based inverse kinematic solver is implemented, enabling efficient and smooth convergence of joint angles corresponding to the desired end-effector positions. The interface was successfully implemented, enabling the robot to follow gesture commands and reach target positions. The proposed vision-based gesture-driven control scheme of the Dexter ER2 robot arm is validated through both simulation and experimental studies. The system employed serial communication, ensuring a continuous and lossless data stream between the controller and the computer. A detailed comparative analysis is also presented to highlight the advantages of the proposed approach. Future work includes enhancing visual appeal with a graphical interface and addressing the latency issue in real-time control.