Azure Databricks Engineer

Company Overview

(Company overview is not specified.)

Job Summary

The Azure Databricks Engineer will play a crucial role in developing, maintaining, and optimizing data processing and analytics pipelines using Azure Databricks. The position requires a strong background in data modeling, PySpark, and SQL to ensure efficient data governance and management within the enterprise.

Responsibilities

• Develop, maintain, and optimize PySpark notebooks on Azure Databricks for data processing and analytics pipelines.
• Implement and manage Spark Structured Streaming solutions for real-time data processing.
• Work extensively with Spark SQL and Unity Catalog to manage and govern enterprise datasets.
• Design and implement robust data models supporting efficient querying and data integrity.
• Write complex SQL queries and optimize them for performance and scalability.
• Collaborate with cross-functional teams, including Data Scientists, Analysts, and Architects, to understand data requirements.

Qualifications

• Hands-on experience with Azure Databricks, PySpark, and Spark SQL.
• Proficiency in Spark Structured Streaming.
• Experience with Unity Catalog for data governance.
• Strong SQL skills for data extraction, transformation, and analysis.
• Solid understanding of Data Modeling principles.
• Bachelor’s degree in Computer Science, Engineering, or a related field preferred.
• 6-9 years of relevant experience in data engineering or related roles.

Preferred Skills

• Experience in CI/CD pipelines for Databricks.
• Knowledge of Delta Lake and Data Lakehouse concepts.
• Familiarity with Azure Data Factory or other orchestration tools.

Experience

• A minimum of 6-9 years of relevant experience in Azure Databricks and associated technologies.

Environment

• The position is hybrid, located PAN India, allowing flexibility for remote and in-office work.

Salary

(Salary information is not specified.)

Growth Opportunities

(Growth opportunities are not specified.)

Benefits

(Benefits information is not specified.)