Airflow Developer

Job Summary

We are seeking a skilled Airflow Developer to lead the migration of ETL workflows from Talend to Apache Airflow. The ideal candidate will have hands-on experience in designing, developing, and deploying data pipelines using Airflow, and a solid understanding of Talend ETL processes. This role involves analyzing existing Talend jobs, re-engineering them in Airflow, and ensuring seamless integration with data sources and destinations.

Responsibilities

• Analyze existing Talend ETL jobs and understand business logic and data flow.
• Design and implement equivalent workflows in Apache Airflow using Python.
• Develop Directed Acyclic Graphs (DAGs) for scheduling and monitoring data pipelines.
• Ensure data quality, error handling, and logging in Airflow pipelines.
• Collaborate with data engineers, analysts, and business stakeholders to validate migrated jobs.
• Optimize performance and scalability of Airflow workflows.
• Document migration processes, pipeline architecture, and operational procedures.
• Provide support during User Acceptance Testing (UAT) and production deployment.

Qualifications

• Strong experience with Apache Airflow (2+ years).
• Hands-on experience with Talend ETL tools (Open Studio or Talend DI).
• Proficiency in Python for scripting and pipeline development.
• Experience with SQL and working with relational databases (e.g., PostgreSQL, MySQL, Oracle).
• Familiarity with cloud platforms (AWS, GCP, or Azure) and cloud-native data services.
• Understanding of CI/CD pipelines, version control (Git), and DevOps practices.
• Knowledge of data warehousing concepts, ETL best practices, and data governance.

Preferred Skills

• Experience with containerization (Docker) and orchestration (Kubernetes).
• Exposure to Airflow plugins, sensors, and custom operators.
• Experience with monitoring tools (e.g., Prometheus, Grafana).
• Talend certification or Airflow certification is a plus.

Experience

• 3-5 years of relevant experience in ETL processes and data pipeline development.

Environment

• Location: Noida; this position is expected to be on-site or a hybrid work model based on company policies.