Data Engineer (Python, FastAPI, Kubernetes, Azure)

Company Overview

Not specified.

Job Summary

We are seeking a Data Engineer with expertise in Python, FastAPI, Kubernetes, and Azure. The primary responsibility of this role is to design, build, and optimize scalable data pipelines and microservices that enhance our data handling capabilities. This position plays a critical role in ensuring that our systems deliver high performance and reliability, crucial for supporting high concurrency workloads.

Responsibilities

• Design, build, and optimize scalable data pipelines and microservices.
• Deploy and manage APIs in Kubernetes (K8s) with high availability.
• Orchestrate I/O-heavy API calls for high concurrency workloads.
• Work with queues, databases, and event-driven systems for real-time data processing.
• Implement and manage Azure cloud data solutions including Data Factory, Event Hub, and Key Vault.
• Collaborate with cross-functional teams to ensure high performance and reliability of systems.

Qualifications

• Educational Requirement: Bachelor's degree in Computer Science, Engineering, or a related field.
• Technical Skills:
  • Strong expertise in FastAPI, Socket.IO, Python, and Kubernetes.
  • Experience in production deployments with high concurrency (10k+ users).
  • Hands-on experience with queues such as Celery, RabbitMQ, or AWS SQS.
  • Strong knowledge of databases including PostgreSQL, MongoDB, Databricks, and Iceberg.
  • Experience with web sockets, async programming with async.io.
  • Strong hands-on experience with Azure Cloud services, including:
    • Azure Data Factory
    • SQL/NoSQL Databases
    • Event Hub
    • Storage
    • Key Vault
  • Experience in Microservices and Containers.
  • Experience in SQL.
  • No gaps in experience and no overlapping work across multiple companies.

Preferred Skills

• Exposure to GoLang (optional).
• Knowledge of API and automation tools like Logic Apps.
• Familiarity with CI/CD pipelines, DevOps practices, and Azure monitoring tools.

Experience

• Years of Experience: 4–6 years, preferably 3–5 years focused in Data Engineering.