Data Engineer

Job Summary

We are seeking a skilled Data Engineer to build and optimize data pipelines for a real-time Digital Twin platform that powers mobility simulation, complex event processing, and multi-agent learning. In this role, you will design the backbone for scalable, low-latency ingestion and processing of high-volume sensor, vehicle, and infrastructure data to support prediction models and simulations.

Responsibilities

• Design and implement streaming data pipelines from IoT sensors, cameras, vehicle telemetry, and infrastructure systems.
• Build scalable infrastructure using AWS Kinesis and Apache Flink/Spark to support both real-time and batch workloads.
• Enable time-series feature stores and implement sliding window processing to capture mobility patterns.
• Integrate simulation outputs and model predictions into AWS data lakes.
• Maintain data validation, manage schema versioning, and ensure high-throughput ingestion of data.
• Collaborate with Data Scientists and Simulation Engineers to optimize data formats such as Parquet, Protobuf, and Delta Lake.
• Deploy and monitor pipelines on AWS cloud and/or edge infrastructure.

Qualifications

• Mandatory Skills: Data Engineering, AWS, Data Lakes, Message Serialization, Airflow, Python, SQL, and distributed data systems (e.g., Kinesis, Spark, Flink).
• Proven experience (3+ years) in data engineering with a strong track record in designing and building real-time systems.
• Solid understanding of event-driven architectures and data ingestion platforms.
• Proficiency in Python and SQL for developing complex data transformations.
• Familiarity with building data pipelines and workflow orchestration tools such as Apache Airflow.
• Bachelor's degree in Computer Science, Engineering, or a related field is preferred.

Preferred Skills

• Experience with processing sensor data, telemetry ingestion, or mobility data.
• Exposure to smart city platforms, V2X ecosystems, or other time-series paradigms.
• Experience integrating data from cameras or other sensor technologies.
• Familiarity with containerization (Docker), CI/CD practices, Kubernetes, and cloud-native architectures.

Experience

• 4-7 years of overall experience in data engineering, with a focus on real-time systems and distributed data environments.

Environment

• Work Location: Bangalore/Chennai with the requirement to work on-site at the client location for one week each month.
• Typical work setting includes a collaborative environment with exposure to both cloud and edge infrastructure.

GrowthOpportunities

• Opportunity to contribute to cutting-edge projects in real-time data processing and digital twin simulations.
• Potential for career advancement into senior technical or leadership roles within an innovative technology-driven team.