Senior Data Engineer

Company Overview

Not specified.

Job Summary

The Senior Data Engineer will play a pivotal role in developing high-performance, large-scale data applications within the Aviation Industry. The primary focus will be on the architecture, design, and implementation of data products, particularly involving the migration from Azure Synapse to Azure Data Lake. This position requires a strong commitment to data quality, performance, and cost optimization, while fostering collaboration in an Agile environment.

Responsibilities

• Architect, design, develop, test, and maintain complex data architectures that support data integration, storage, processing, and orchestration.
• Migrate existing data solutions from Azure Synapse to Azure Data Lake, ensuring best practices are followed.
• Construct and optimize Apache Spark jobs and clusters within the Databricks platform.
• Collaborate with cross-functional teams, including Product, Engineering, Data Scientists, and Analysts, to translate data requirements into actionable solutions.
• Provide mentoring and guidance to junior team members while promoting best practices in data engineering and governance.
• Continuously monitor and optimize workloads and clusters for enhanced performance and efficiency.

Qualifications

• Full-time Bachelor's degree in Computer Science or a related field is required.
• At least 5 years of experience as a Data Engineer with expertise in Databricks, Azure, and DevOps.
• Strong programming skills in Python (mandatory) and familiarity with Scala.
• Proven track record of delivering large-scale data projects, including migrations.
• Certifications:
  • Databricks Certified Associate Developer for Apache Spark
  • Databricks Certified Data Engineer Associate
  • Microsoft Certified: Azure Fundamentals
  • Microsoft Certified: Azure Data Engineer Associate
  • Microsoft Exam: Designing and Implementing Microsoft DevOps Solutions (nice to have)

Preferred Skills

• Familiarity with BI solutions, such as Power BI.
• Experience with Apache Airflow or other orchestration technologies.
• Knowledge of data governance frameworks and best practices.