Senior Data Engineer (Python, PySpark, AWS)

Job Summary

The Senior Data Engineer will play a vital role in the organization by designing and enhancing data-processing pipelines and orchestration workflows. This position is focused on utilizing open-source tools, AWS services, and automation to drive data capabilities that align with the company’s objectives.

Responsibilities

• Design, develop, and enhance data-processing pipelines, orchestration workflows, and monitoring solutions using open-source tools and AWS services.
• Collaborate with product managers and technical teams to define and implement data platform capabilities.
• Identify and implement process improvements to automate repetitive tasks, optimize usability, and enhance scalability.
• Provide ongoing technical support and usage guidance to internal users of data platform services.
• Develop and maintain metrics, monitoring, and alerting tools to ensure visibility and performance of production services.

Qualifications

• Proven experience in building and optimizing data pipelines in distributed computing environments.
• Solid experience working with cross-functional teams in agile environments.
• Strong working knowledge of Linux/Unix environments.
• 4+ years of hands-on experience in:
  • Advanced SQL
  • Python
  • PySpark (MUST)
• Familiarity with tools like Git/Bitbucket, Jenkins/CodeBuild, and CodePipeline.
• Exposure to platform monitoring and alerting tools.
• Prior experience or understanding of Palantir Foundry is an added advantage.

Preferred Skills

• Knowledge/Experience with Palantir.

Experience

• 6+ years of relevant experience required.

Environment

• Onsite work location in Hyderabad, 5 days a week.