Data Engineer

Job Summary

We are seeking a highly skilled Data Engineer with expertise in Kafka, Python, and Azure Databricks to drive our healthcare data engineering projects. The ideal candidate will have extensive experience in real-time data streaming, cloud-based data platforms, and large-scale data processing. This position demands strong technical leadership, excellent problem-solving abilities, and the capability to collaborate effectively with cross-functional teams.

Responsibilities

• Lead the design, development, and implementation of real-time data pipelines using Kafka, Python, and Azure Databricks.
• Architect scalable data streaming and processing solutions to support healthcare data workflows.
• Develop, optimize, and maintain ETL/ELT pipelines for structured and unstructured healthcare data.
• Ensure data integrity, security, and compliance with healthcare regulations (e.g., HIPAA, HITRUST).
• Collaborate with data engineers, analysts, and business stakeholders to understand requirements and translate them into effective technical solutions.
• Troubleshoot and optimize Kafka streaming applications, Python scripts, and Databricks workflows.
• Mentor junior engineers, conduct code reviews, and enforce best practices in data engineering.
• Stay updated with the latest cloud technologies, big data frameworks, and industry trends.

Qualifications

• 4+ years of experience in data engineering with strong proficiency in Kafka and Python.
• Expertise in Kafka Streams, Kafka Connect, and Schema Registry for real-time data processing.
• Experience with Azure Databricks (or willingness to learn quickly).
• Hands-on experience with cloud platforms (Azure preferred; AWS or GCP is a plus).
• Proficiency in SQL, NoSQL databases, and data modeling for big data processing.
• Knowledge of containerization (Docker, Kubernetes) and CI/CD pipelines for data applications.
• Experience working with healthcare data (EHR, claims, HL7, FHIR, etc.) is a plus.
• Strong analytical skills, problem-solving mindset, and ability to lead complex data projects.
• Excellent communication and stakeholder management skills.

Preferred Skills

• Additional experience with data visualization tools would be advantageous.
• Familiarity with machine learning concepts and frameworks.

Experience

• 5 to 8 years of relevant experience in data engineering, specifically with real-time data streaming technologies.

Environment

• Locations: Bangalore, Noida, and Hyderabad
• Work mode: Hybrid (with a requirement of 2 days in the office per week).