Data Engineer

Company Overview

Company details and industry information are not specified.

Job Summary

The Data Engineer role is essential in managing and optimizing the data pipeline. This position is focused on designing, implementing, and maintaining ETL (Extract, Transform, Load) processes that support data analytics and reporting. The Data Engineer will work with various technologies and frameworks to ensure seamless data flow and accessibility, contributing to the organization’s data-driven decision-making.

Responsibilities

• Design and develop ETL processes to facilitate data integration from multiple sources.
• Utilize Databricks on AWS to perform data transformations and processing.
• Schedule and manage workflows using Apache Airflow for reliable task orchestration.
• Collaborate with data analysts to understand data requirements and optimize data outputs in Snowflake.
• Write and optimize SQL queries for data extraction and analysis.
• Implement version control using GitLab to track code changes and collaborate with team members.

Qualifications

• Strong knowledge of ETL processes and design principles.
• Proficient with Databricks on AWS and Snowflake for data processing and storage.
• Experience with Apache Airflow for workflow management.
• Proficiency in PySpark and SQL for data manipulation and analysis.
• Familiarity with version control systems, particularly GitLab.
• Experience using AWS S3 for data storage.
• Knowledge of data visualization tools like Looker.
• Experience with Continuous Integration/Continuous Deployment (CI/CD) using Jenkins.

Preferred Skills

• Familiarity with cloud computing concepts and AWS services.
• Knowledge of data governance and data quality standards.
• Experience with machine learning workflows.

Experience

Relevant years of experience in data engineering or a similar role, with a focus on data pipeline management and ETL processes.