Senior Data Engineer

Job Summary

We are looking for a highly experienced Senior Data Engineer to design and implement robust data pipelines, manage large-scale data processing, and ensure smooth orchestration using modern cloud and data technologies. The ideal candidate will have extensive hands-on experience with Python, Spark, SQL, and modern cloud data services.

Responsibilities

• Develop and maintain scalable ETL/ELT pipelines for batch and streaming data.
• Design, implement, and optimize big data processing using Spark and Pandas.
• Work with cloud-based tools like Azure Data Factory or similar for data orchestration.
• Collaborate with data scientists, analysts, and other engineers to ensure clean, reliable, and timely data delivery.
• Monitor and troubleshoot pipeline performance, leveraging tools like Airflow and cloud monitoring solutions.
• Manage, transform, and query data across structured and unstructured databases.

Qualifications

• Extensive experience in data engineering and pipeline development.
• Proficiency in Python, Spark, and SQL.
• Experience with cloud data technologies (e.g., Azure Data Services).
• Strong understanding of ETL/ELT processes.
• Familiarity with big data frameworks and tools.
• Ability to troubleshoot and optimize data processing workflows.
• Excellent collaboration and communication skills.
• Bachelor's degree in Computer Science, Engineering, or a related field.

Preferred Skills

• Experience with additional cloud platforms (e.g., AWS, Google Cloud).
• Knowledge of data warehousing solutions.
• Familiarity with data governance and security practices.

Experience

• Extensive experience in data engineering roles, typically 5+ years preferred.