Data Engineer

Company Overview

(No specific company overview provided)

Job Summary

We are seeking a highly skilled and experienced Data Engineer to join our data platform team. The ideal candidate should have a strong background in Python, Spark/PySpark, Databricks, and Azure Data Factory (ADF), with a passion for building scalable and efficient data pipelines in a cloud environment.

Responsibilities

• Design, develop, and maintain robust ETL/ELT pipelines using PySpark and Databricks.
• Build and orchestrate data workflows using Azure Data Factory (ADF).
• Develop and optimize data processing scripts and jobs using Python and Spark.
• Work with structured and unstructured data from various sources, ensuring quality and performance.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements.
• Monitor and troubleshoot data pipelines, ensuring high availability and fault tolerance.
• Implement data governance and security policies for handling sensitive data.

Qualifications

• 5+ years of hands-on experience with Python and PySpark.
• Strong experience working with Databricks for big data processing.
• Proficiency in Azure Data Factory – pipeline creation, dataflows, linked services, and triggers.
• Good understanding of Azure services related to data engineering.
• Experience in writing complex SQL queries for data extraction and transformation.
• Strong debugging and performance tuning skills for Spark and distributed systems.

Preferred Skills

(No additional preferred skills provided)

Experience

• 5+ years of relevant experience in data engineering.

Environment

• Employment Type: Freelance
• Work Mode: Remote

Salary

(No salary range provided)

Growth Opportunities

(No specific growth opportunities mentioned)

Benefits

(No benefits listed)