Lead Data Engineer - Azure/AWS, Synapse/Fabric, PySpark, Databricks, ADF, Medallion Architecture, Lakehouse, Data warehouse

Job Summary

The Lead Data Engineer is responsible for designing and implementing data solutions in cloud environments such as Azure and AWS. This role involves leveraging technologies like Synapse, Fabric, PySpark, Databricks, and Azure Data Factory (ADF) to build robust data pipelines and facilitate the adoption of modern data architectures including Medallion Architecture and Lakehouse frameworks. The Lead Data Engineer will play a critical role in transforming raw data into meaningful insights while ensuring data integrity and efficiency.

Responsibilities

• Design and implement scalable data pipelines using Azure and AWS services.
• Develop and manage data solutions leveraging technologies such as Databricks, Azure Synapse, and ADF.
• Collaborate with cross-functional teams to define data requirements and usage policies.
• Optimize existing data systems and workflows for improved performance and reliability.
• Implement Data Lakehouse and Medallion Architecture best practices for data storage and processing.
• Ensure data quality and compliance with organizational standards and governance.

Qualifications

• Proficiency in Azure and AWS cloud platforms.
• Strong experience with data engineering tools such as PySpark, Databricks, Synapse, and ADF.
• Expertise in designing and implementing data warehouses and lakehouses.
• Familiarity with Medallion Architecture and its practical applications.
• Excellent problem-solving skills and attention to detail.
• Strong communication and teamwork abilities.
• Bachelor’s degree in Computer Science, Engineering, Data Science, or a related field.

Preferred Skills

• Experience with machine learning frameworks and data analysis tools.
• Knowledge of SQL and other programming languages related to data processing.
• Familiarity with ETL processes and data integration methodologies.

Experience

• A minimum of 8 years of experience in data engineering or a related field.
• Demonstrated ability to lead data-centric projects and initiatives.

Environment

The typical work setting may include remote, hybrid, or in-office environments, depending on the organization's policies.