Data Engineer

Company Overview

Company information is not specified.

Job Summary

The Data Engineer will design and implement core data pipelines and semantic data models for the AI Platform, enabling high-quality retrieval, grounding, and reduced hallucination for multi-domain Large Language Model (LLM) applications. This role is crucial in facilitating effective data architecture and ensuring the performance and efficiency of the AI-driven systems.

Responsibilities

• Design and implement scalable data pipelines that support AI applications.
• Develop and manage the semantic data models to ensure reliable data retrieval and processing.
• Collaborate with AI teams to enhance data quality and minimize hallucination in LLM applications.
• Utilize various data engineering tools and technologies to optimize performance and data processing capabilities.
• Manage and maintain ETL/ELT systems to ensure seamless data integration.
• Monitor data workflow and troubleshoot any issues that arise within the data infrastructure.

Qualifications

• 5+ years of experience in data engineering or data science.
• Proficient in Python and SQL with strong experience in ETL/ELT systems.
• Hands-on experience with GraphQL schema design.
• Familiarity with at least one vector database: Pinecone, Milvus, Weaviate, or FAISS.
• Solid understanding of embeddings, chunking, metadata enrichment, and retrieval pipelines.
• Experience with workflow management tools such as Airflow, Spark, Flink, or similar systems.

Preferred Skills

• Knowledge of machine learning concepts and data models.
• Familiarity with cloud platforms and their data management tools.
• Good problem-solving skills and a collaborative approach to work.

Experience

• Minimum of 5 years of relevant experience in data engineering or related fields.

Environment

Work setting, location, and conditions are not specified.

Salary

Salary information is not specified.

Growth Opportunities

Potential career advancement opportunities are not specified.

Benefits

Benefits information is not specified.