Azure Databricks Pyspark SQL

Job Summary

The Data Engineer will play a crucial role in the Business Intelligence and Data Warehousing domain. This position involves the development and management of data solutions using Azure's modern data platform to facilitate data analysis, transformation, and storage. The successful candidate will contribute to the organization's goals by optimizing and ensuring data integrity across various platforms.

Responsibilities

• Develop and maintain data pipelines using Apache Pyspark, Azure Databricks, and Azure Data Factory.
• Perform data analysis and transformation leveraging Python, R, or Scala in Azure environments.
• Design and implement data structures optimized for storage and varied query patterns, such as Parquet and Delta Lake.
• Utilize distributed processing capabilities within Databricks or Apache Spark and debug using tools like Ganglia UI.
• Collaborate with teams to manage and handle data securely, adhering to Information Security principles.
• Work effectively with structured, semi-structured, and unstructured data to identify linkages and extract valuable insights from diverse datasets.

Qualifications

• Educational Qualification: Bachelor's degree in Computer Science, Information Technology, or a related field.
• Minimum 5+ years of relevant IT experience specifically in Business Intelligence (BI) and Data Warehousing (DW).
• At least 4 years of hands-on experience with Azure modern data platforms, including Data Factory, Databricks, Synapse (Azure SQL DW), and Azure Data Lake.
• Experience in NoSQL data store concepts and a strong understanding of traditional RDBMS technologies (e.g., MS SQL Server, Oracle).
• Proficiency with MPP technologies (e.g., Teradata, Netezza) and NoSQL databases (e.g., MongoDB, Cassandra, Neo4J, CosmosDB, Gremlin).
• Familiarity with traditional Data Warehousing and ETL tools such as Informatica, IBM Datastage, and Microsoft SSIS.
• Effective communication skills with the ability to articulate technical concepts to various stakeholders.
• Experience working with large codebases collaboratively using Git methodologies.

Preferred Skills

• Familiarity with Agile methodologies such as SCRUM, Extreme Programming (XP), and Kanban.
• Knowledge of debugging and optimizing Spark Jobs.

Experience

• 5+ years in the IT field with a focus on Data Engineering, specifically in BI and DW domains.

Environment

• Location: Pune/Hyderabad.
• Work setting and any specific physical or environmental conditions are not specified.

Salary

Salary information is not provided.

Growth Opportunities

Potential career advancement opportunities within the company are not specified.

Benefits

Details on offered benefits are not provided.