Data Engineer

Job Summary

The Data Engineer will be responsible for constructing and maintaining robust data infrastructures, optimizing data pipelines, and ensuring the efficient delivery of data solutions to meet business intelligence needs. This role plays a critical part in empowering data-driven decision-making within the organization.

Responsibilities

• Design and implement scalable data architectures using Medallion Architecture principles.
• Develop and maintain ETL pipelines utilizing Azure Data Factory.
• Implement data processing solutions using PySpark in Data Bricks.
• Collaborate with data scientists and analysts to define and implement data models.
• Ensure data quality and integrity through consistent monitoring and optimization of data workflows.
• Document data processes and maintain data pipeline codebases for future reference and compliance.

Qualifications

• Proficiency in Python for developing data processing applications.
• Extensive experience with Azure Data Bricks for big data processing.
• Strong knowledge of Data Factory for data integration and ETL processes.
• Solid understanding of Medallion Architecture for structuring data platforms.
• Familiarity with data modeling and database design.
• Excellent problem-solving and analytical skills.
• Ability to work collaboratively in a team-oriented environment.
• Bachelor's degree in Computer Science, Data Engineering, or a related field.

Preferred Skills

• Experience with cloud-based data solutions and services.
• Knowledge of data governance and data security best practices.
• Familiarity with machine learning frameworks and tools.

Experience

Typically seeks candidates with 5 to 15 years of relevant experience in data engineering or a related field.