Databricks Platform Administrator

Company Overview

[Not provided]

Job Summary

We are seeking a skilled Databricks Platform Administrator / Engineer with proven experience supporting modern data platforms in a Property & Casualty (P&C) insurance setting. The ideal candidate will be responsible for managing, configuring, optimizing, and maintaining the Databricks environment to support data engineering, analytics, and machine learning workloads in a secure and compliant manner.

Responsibilities

• Platform Administration & Management:

  • Manage and administer Databricks workspaces, clusters, jobs, libraries, and notebooks.
  • Implement access controls using Unity Catalog, SCIM, and workspace-level Role-Based Access Control (RBAC).
  • Configure and monitor autoscaling, cluster policies, and compute usage to optimize cost and performance.
  • Support platform upgrades, patching, and operational automation.

• Security, Compliance & Governance:

  • Enforce security policies aligned with P&C insurance industry regulations.
  • Integrate with enterprise identity providers (Azure Active Directory, Okta) and manage Single Sign-On (SSO).
  • Support data governance using tools like Unity Catalog, Immuta, or Collibra.

• DevOps & Automation:

  • Build Continuous Integration/Continuous Deployment (CI/CD) pipelines for notebooks, jobs, and ML workflows using tools like Azure DevOps, GitHub Actions, or Jenkins.
  • Automate platform provisioning and configurations using Infrastructure-as-Code (Terraform, ARM, etc.).

• Monitoring & Troubleshooting:

  • Set up logging, monitoring, and alerting using tools like Azure Monitor, Datadog, or Splunk.
  • Troubleshoot performance issues, job failures, and cluster errors.

• Collaboration & Enablement:

  • Work closely with Data Engineers, Data Scientists, and Business Analysts to ensure optimal platform usage.
  • Develop platform usage guidelines, onboarding documentation, and best practices.
  • Provide technical support and training to users.

Qualifications

• Education:

  • Bachelor’s or Master’s degree in Computer Science, Engineering, Information Systems, or related field.

• Required Qualifications:

  • 10+ years of experience in data platform engineering or system administration.
  • 2+ years of hands-on experience with Databricks administration on Azure.
  • Solid understanding of Apache Spark, Delta Lake, and Lakehouse architecture.
  • Proficiency in scripting and automation using Python, Bash, PowerShell, or Terraform.
  • Familiarity with data security, audit, and compliance frameworks relevant to P&C insurance (e.g., GDPR, NYDFS, SOC 2).
  • Experience with Azure Data Services, Azure Data Lake Storage (ADLS), Key Vault, and Azure DevOps.
  • Strong understanding of the P&C insurance data ecosystem (policy, claims, underwriting, actuarial, etc.).

Preferred Skills

• Databricks Certified Associate / Professional (Administrator / Data Engineer) certification.
• Experience integrating Databricks with core systems like Guidewire, Duck Creek, or custom P&C platforms.
• Knowledge of industry standards like ACORD data models and ISO data feeds.
• Exposure to tools like Unity Catalog, Purview, Collibra, Informatica, or similar.

Experience

• Minimum 10 years of relevant experience in data platform engineering or administration, with at least 2 years of specialized experience in Databricks.

Environment

• This role is located in Pune/Ahmadabad and may involve a mix of in-office and remote work settings, depending on company policies and project requirements.

Salary

[Not provided]

Growth Opportunities

[Not provided]

Benefits

[Not provided]