Databricks Data Engineer

Company Overview

Company details are not specified.

Job Summary

We are seeking an experienced Databricks Data Engineer with a proven track record in building and optimizing big data pipelines. The ideal candidate will leverage strong expertise in Apache Spark, Databricks, and AWS to contribute to our organization's data engineering goals and best practices.

Responsibilities

• Design, build, and maintain scalable data pipelines using Databricks and Apache Spark.
• Develop and optimize ETL/ELT processes for both structured and unstructured data.
• Implement and manage Lakehouse architecture for data storage, processing, and analytics.
• Orchestrate data workflows using Databricks Workflows, Jobs API, and Airflow.
• Work with AWS data services (S3, Lambda, CloudWatch) for integration and automation.
• Optimize query performance using pushdown techniques, indexing, and partitioning.
• Ensure data security and compliance through Unity Catalog, governance policies, and access controls.
• Monitor, troubleshoot, and improve Databricks jobs, clusters, and pipelines.
• Participate in end-to-end data migration projects to AWS Cloud.
• Collaborate using Git/GitHub for code versioning and CI/CD.

Qualifications

• 5–9 years of experience in data engineering roles.
• Strong hands-on experience in Databricks, Apache Spark, AWS, and Python.
• Expertise in data orchestration and workflow management.
• Proficient in SQL, performance optimization, and troubleshooting.
• Excellent analytical, problem-solving, and communication skills.
• Bachelor's degree in Computer Science, Information Technology, or a related field (preferred).

Preferred Skills

• Familiarity with machine learning frameworks.
• Experience with data governance tools and practices.
• Knowledge of additional programming languages or frameworks (e.g., Java, Scala).

Experience

• Minimum of 5 years of relevant experience in data engineering, with a focus on big data technologies.

Environment

This position is remote, allowing for flexible work arrangements while requiring effective collaboration with team members in various locations.

Salary

Estimated budget for the position is ₹1.2 LPM.

Growth Opportunities

Growth opportunities within the company are not specified.

Benefits

Benefits associated with this role are not specified.