LLM-Based DAG Creation for Data Enrichment Pipelines in SemT Framework

The design of data enrichment pipelines is a complex task that can be simplified by leveraging Large Language Models (LLMs) for natural language-based creation of Directed Acyclic Graphs (DAGs). This approach aims to democratize data pipeline development by enabling DAG generation through natural language text inputs. Our study explores using LLMs to generate Apache Airflow DAGs. This approach streamlines workflow design and makes complex data enrichment more accessible, while preserving the flexibility and power of Airflow's ecosystem. We conducted preliminary experiments using the SemT framework to address data enrichment challenges at scale. SemT service model, based on the Explore-Design-Operate methodology, enables flexible data transformation pipelines that work across different environments.