Principal Data Engineer

Company Overview

(Details about the company are not specified.)

Job Summary

The Principal Data Engineer will be responsible for developing and maintaining data ingestion, transformation, and integration pipelines within the Microsoft Azure ecosystem. This role plays a crucial part in ensuring secure and efficient data storage and processing, contributing to the organization’s overall data strategy and goals.

Responsibilities

• Develop and maintain data ingestion, transformation, and integration pipelines using PySpark, Scala, and SQL within the Microsoft Azure ecosystem.
• Manage and maintain Databricks Delta Lake environments for secure and efficient data storage and processing.
• Design and implement data models (ER and dimensional) to optimize data access and reporting.
• Build pipelines to integrate data from various sources including flat files, SQL Server, Oracle, Informix, JSON, XML, Blob storage, and APIs.
• Leverage Azure services such as Azure Data Factory, Azure Storage, Blob Accounts, and Key Vaults to ensure seamless data operations and security compliance.
• Ensure best practices for code creation, migration, and deployment across multi-environment setups.

Qualifications

• Strong experience with PySpark, Scala, and SQL.
• Proficiency in designing data models and building ETL pipelines.
• Familiarity with Databricks Delta Lake and Azure services.
• Ability to troubleshoot and resolve data-related issues effectively.
• Excellent communication skills for stakeholder engagement.
• Experience in providing technical mentorship to junior team members.
• Knowledge of data engineering best practices.

Educational Qualifications

• A degree in Computer Science, Data Science, Information Technology, or a related field is preferred.

Preferred Skills

• Experience in handling both traditional and non-traditional data sources.
• Knowledge of data governance and security compliance measures.
• Familiarity with metrics and data science team collaboration.

Experience

• Minimum of 5-7 years of experience in data engineering or a related role, with a focus on Azure technologies.

Environment

(Details about the work setting, location, or physical conditions are not specified.)

Salary

(Salary details are not specified.)

Growth Opportunities

(Details about potential career advancement opportunities are not specified.)

Benefits

(Details about offered benefits are not specified.)