Senior Data Engineer

Company Overview

(Not specified)

Job Summary

The Senior Data Engineer will play a crucial role in ensuring the accuracy, integrity, and trustworthiness of data assets across our cloud-native infrastructure. This role focuses on the development of data quality and validation frameworks, contributing significantly to the organization’s goals through reliable data management.

Responsibilities

• Design, develop, and deploy automated data validation and quality frameworks using Python.
• Build scalable and fault-tolerant data pipelines supporting quality checks across data ingestion, transformation, and delivery.
• Integrate with REST APIs to validate and enrich datasets across distributed systems.
• Deploy and manage validation workflows using AWS services (EKS, EMR, EC2) and Kubernetes clusters.
• Collaborate with data engineers, analysts, and DevOps to embed quality checks into CI/CD and ETL pipelines.
• Develop monitoring and alerting systems for real-time detection of data anomalies and inconsistencies.
• Write clean, modular, and reusable Python code for automated testing, validation, and reporting.
• Lead root cause analysis for data quality incidents and design long-term solutions.
• Maintain detailed technical documentation of data validation strategies, test cases, and architecture.
• Promote data quality best practices and evangelize a culture of data reliability within the engineering teams.

Qualifications

• Experience with data quality platforms such as Great Expectations, Collibra Data Quality, or similar tools.
• Proficiency in Docker and container lifecycle management.
• Familiarity with serverless compute environments (e.g., AWS Lambda, Azure Functions), Python, PySpark.
• Relevant certifications in AWS, Kubernetes, or data quality technologies.
• Prior experience working in big data ecosystems and real-time data environments.
• Strong background in programming with expert-level knowledge of Python.
• Practical experience in data pipeline engineering and API integration.
• Understanding of cloud-native workloads specifically on AWS and Kubernetes.

Preferred Skills

• Additional experience with data governance best practices.
• Knowledge of data modeling and database design.
• Understanding of machine learning data requirements and data processing frameworks.

Experience

• A minimum of 5+ years of relevant experience in data engineering, with a focus on data quality and validation processes.

Environment

• Typical work setting will be a multi-cloud environment with a focus on AWS and Kubernetes.

Salary

(Not specified)

Growth Opportunities

(Not specified)

Benefits

(Not specified)