Data Engineer - Medallion Architecture, Python, Databricks, Azure

Job Summary

The Data Engineer will be responsible for designing, building, and maintaining scalable data pipelines (Medallion Architecture) using various technologies including Python, Databricks, and Azure. This role is crucial in ensuring that data is accurately collected, transformed, and made accessible for analytics and business intelligence.

Responsibilities

• Design and implement scalable data processing pipelines using Medallion Architecture.
• Develop ETL processes to extract, transform, and load data from various sources.
• Utilize Python, Databricks, and Azure services to analyze and manage large datasets.
• Collaborate with cross-functional teams to define data requirements and specifications.
• Monitor and optimize data pipelines for performance and reliability.
• Ensure data quality and integrity through automated testing and validation.

Qualifications

• Proficient in Data Engineering concepts and best practices.
• Strong programming skills in Python.
• Experience with Databricks platform for data processing and analytics.
• Familiarity with Azure cloud services and data management solutions.
• Knowledge of data modeling concepts and Medallion Architecture.
• Ability to work with large datasets and perform data transformation.
• Excellent problem-solving skills and attention to detail.
• Bachelor's degree in Computer Science, Information Technology, or related field (preferred).

Preferred Skills

• Experience with data warehousing solutions.
• Familiarity with data visualization tools (e.g., Power BI, Tableau).
• Understanding of machine learning concepts and frameworks.
• Knowledge of SQL for database querying.

Experience

• Minimum of 5 to 15 years of relevant experience in data engineering or a similar role.