Advancing End-User Development in Robotics with Large Language Models and Digital Twins

This paper introduces an end-user development environment designed to support non-expert users in creating pick-and-place robot tasks. The environment includes a chat-based interface powered by OpenAI’s GPT-4o, guiding users through the task definition process in an intuitive and domain-oriented manner. Once a task is defined, it is visualized in a graphic interface built with Google Blockly, enabling users to review, edit, or expand the task through direct manipulation of block-like components. The system also allows users to generate a corresponding robot program and simulate its execution using a digital twin of the robot. This final simulation step is essential for validating the task and identifying potential issues before deploying the robot program in a real-world setting.