Senior Data Engineer (Azure Databricks / Apache Spark)

Company Overview

Information about the company is not specified.

Job Summary

We are seeking an experienced Senior Data Engineer with strong expertise in Azure Databricks, Apache Spark, and Azure Data Engineering tools. The primary purpose of this role is to design, develop, and maintain large-scale data pipelines that ensure high performance, scalability, and reliability of enterprise data solutions.

Responsibilities

• Design and develop ETL pipelines using Azure Databricks, Apache Spark, and Azure Data Factory for both batch and real-time data integration.
• Work with Scala/Python for advanced data transformations and pipeline development.
• Write and optimize stored procedures and queries for efficient data processing.
• Perform data ingestion, preparation, integration, and operationalization to meet business requirements.
• Design and implement solutions using Azure Synapse, Snowflake, and other cloud-based analytical warehouses.
• Build and maintain end-to-end data architectures including Data Lakes, Data Warehouses, and Data Marts.
• Work with Azure DevOps, GitHub, and orchestration tools for CI/CD, automation, and pipeline management.
• Ensure data quality, consistency, and reliability by applying relational data processing (MS SQL, Delta Lake, Spark SQL).

Qualifications

• Minimum 8 years of Data Engineering experience.
• Hands-on expertise with Azure Databricks, Apache Spark, and Azure Data Factory.
• Proficiency in Python or Scala for data engineering tasks.
• Strong experience in ETL and Data Warehousing concepts, methodologies, and strategies.
• Experience with Azure Synapse Analytics, Snowflake, and cloud-based data warehouses.
• Knowledge of data lake architecture and big data processing frameworks.
• Strong problem-solving, analytical, and communication skills.
• Experience working in the Healthcare domain (preferred).

Preferred Skills

• Familiarity with Healthcare business data models.

Experience

• At least 8 years of relevant Data Engineering experience is required.

Environment

• The work setting is hybrid, requiring 2 days onsite per month.

Salary

Salary details are not specified.

Growth Opportunities

Information regarding potential career advancement opportunities is not specified.

Benefits

Details on offered benefits are not specified.