Python Databricks Developer

Company Overview

Company information is not specified.

Job Summary

The Python Databricks Developer will play a vital role in developing and maintaining data pipelines and processing solutions. Utilizing a strong background in data engineering and Python development, the candidate will contribute to the organization's ability to manage and process healthcare data efficiently. This position requires collaboration within a team and the ability to work independently with minimal technical guidance.

Responsibilities

• Develop and maintain robust ETL pipelines using Databricks and DataStage or similar systems.
• Implement data processing workflows leveraging Spark, Python, and SQL for efficient data handling.
• Collaborate with team members to design scalable data architectures compatible with AWS technologies including S3 and Lambda.
• Monitor and optimize existing data pipelines for performance improvements and cost efficiency.
• Ensure proper documentation of data engineering processes and methodologies.
• Work with Airflow for orchestration of data workflows and tasks.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related field.
• 5+ years of experience in Data Engineering.
• 2+ years of hands-on experience developing in Python.
• Proficient in using Databricks and Spark in data processing tasks.
• Experience with AWS technologies such as S3 and Lambda.
• Familiarity with DataStage or similar ETL systems.
• Knowledge of Airflow for data pipeline orchestration.
• Ability to work independently with minimal technical guidance.

Preferred Skills

• Experience with healthcare data processing systems.
• Familiarity with batch processing frameworks.
• Understanding of best practices in data engineering and data management.

Experience

• 7-10 years of overall experience with at least 5 years specifically in Data Engineering.

Environment

• Hybrid work environment requiring candidates to work 2-3 days from the office each week.
• Locations may include Bangalore, Pune, Mumbai, Hyderabad, and Noida.
• Working hours are from 11 AM to 8 PM, with a need for flexibility to overlap with the Eastern Standard Time (EST) zone as per project requirements.

Salary

• Estimated salary is 2.4 LPM + Tax. Please note that this is subject to negotiation based on feedback and the candidate's experience.

Growth Opportunities

Career advancement opportunities within the company are not specified.

Benefits

Details about offered benefits are not provided.