Databricks SQL Engineer (Pharma Background)

Company Overview

[Company details are not provided.]

Job Summary

We are seeking a highly skilled Databricks SQL Engineer with a background in the pharmaceutical industry to join our Data Engineering team. The ideal candidate will utilize strong expertise in various technologies including Databricks, Pyspark, SQL, Spark SQL, Delta Lake, and data lakehouse architectures. This role is essential for building scalable and efficient data models, queries, and pipelines that power analytical insights and business decisions.

Responsibilities

• Design, develop, and optimize SQL-based data models and queries in Databricks for large-scale datasets.
• Work with Delta Lake, Unity Catalog, and Databricks Notebooks to manage and secure data assets.
• Implement and maintain ETL/ELT pipelines using Databricks workflows and SQL.
• Develop documentation and data dictionaries to promote self-service analytics.
• Ensure data quality and consistency through testing, validation, and monitoring.
• Optimize data lakehouse architecture for cost and performance, leveraging Spark SQL best practices.
• Support CI/CD practices for deploying notebooks and SQL jobs to production.

Qualifications

• 6+ years of experience working with Databricks SQL, PySpark, Spark SQL, or similar distributed SQL engines.
• Experience in the Pharma Domain.
• Strong experience with Delta Lake, structured streaming, and data lakehouse paradigms.
• Proficiency in SQL development, query tuning, and handling large datasets.
• Experience with version control tools (Git) and collaborative development practices.
• Experience working in cloud environments (AWS, Azure, or GCP), preferably with Azure Databricks or Databricks on AWS.
• Understanding of data governance, access control, and tools like Unity Catalog or Azure Purview.
• Familiarity with CI/CD practices for data pipelines and automation tools like Databricks Repos, Terraform, or dbx.
• Exposure to Python or PySpark for scripting and data manipulation.

Preferred Skills

• Knowledge of data science methodologies and analytics frameworks.
• Familiarity with data visualization tools (e.g., Tableau, Power BI).

Experience

• Minimum of 6 years of relevant experience as a Databricks SQL Engineer or similar role.

Environment

• This position is remote. Candidates may need to collaborate with team members across various time zones and environments.

Salary

[Salary details are not provided.]

Growth Opportunities

[Information on career advancement opportunities is not provided.]

Benefits

[Benefits information is not provided.]