Data Engineer – Core Python, PySpark, Hive, Hadoop HDFS, Oozie, Yarn

Company Overview

Not specified.

Job Summary

The Data Engineer will play a critical role in building and optimizing data pipelines, contributing to overall data management and analytics initiatives. The position requires strong proficiency in Core Python, PySpark, and the Hadoop ecosystem. This role will ensure high availability and performance of data systems while working collaboratively with data scientists and analysts to support data-driven decision-making within the organization.

Responsibilities

• Develop and maintain robust data pipelines using Core Python and PySpark.
• Implement data ingestion and transformation processes in the Hadoop ecosystem, including Hive, HDFS, Oozie, and Yarn.
• Ensure the integrity and quality of the data throughout the architecture and pipelines.
• Apply Object-Oriented Programming (OOPs) concepts to design reliable and maintainable code.
• Collaborate with cross-functional teams to understand data needs and deliver solutions that meet requirements.
• Optimize SQL queries and enhance database performance and efficiency.

Qualifications

• 6–10 years of experience in data engineering or a related field.
• Strong knowledge of Core Python and PySpark for data processing.
• Proficiency in SQL for database interaction.
• Familiarity with cloud platforms such as Azure or AWS is a plus.
• Experience with the Hadoop ecosystem: specifically, Hive, HDFS, Oozie, and Yarn.
• Solid understanding of OOPs design principles, including Class/Object, __init__, Constructor, Instance/Class/Static methods, Encapsulation, Abstraction, Property, and Decorators.
• Bachelor's degree in Computer Science, Information Technology, or a related field.

Preferred Skills

• Experience with data visualization tools.
• Understanding of data warehousing concepts.
• Knowledge of data model design and ETL processes.

Experience

6–10 years of relevant experience in data engineering or similar roles.

Environment

Hybrid work environment based in Bangalore (Whitefield), with a requirement to work 3 days from the office.

Salary

Estimated CTC of 1.5L.

Growth Opportunities

Not specified.

Benefits

Not specified.