Associate Big Data Engineer

Kimshuka is actively seeking an experienced Data Engineer - Databricks Migration Specialist to join our dynamic team in Bangalore on a contract basis. This is an exciting opportunity for a seasoned professional passionate about modern data platforms and cloud migration. You will play a pivotal role in driving the crucial migration of enterprise data workflows from legacy systems like DataIKU to the advanced Databricks environment, contributing significantly to our data modernization initiatives.

As a key member of our engineering team, you will leverage your expertise in Databricks, PySpark, and Azure data services to build scalable and efficient data solutions. If you possess a strong background in data engineering, coupled with specific experience in Databricks migrations, and thrive in a hybrid work setting, we encourage you to explore this challenging yet rewarding role with us.

Key Responsibilities

• Lead and execute the end-to-end migration of complex DataIKU workflows and data pipelines to the Databricks platform.
• Design, develop, and implement highly scalable and robust ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks.
• Build, optimize, and tune PySpark and Spark SQL workloads for performance, cost efficiency, and reliability.
• Architect and implement scalable data models leveraging modern technologies such as Delta Lake and Azure Data Lake Storage (ADLS Gen2).
• Automate data processing workflows and establish robust CI/CD pipelines, with a strong preference for Azure DevOps, to ensure seamless deployment and operations.
• Perform comprehensive data validation, reconciliation, and end-to-end testing to guarantee data accuracy and integrity post-migration.
• Create detailed technical documentation, operational runbooks, and facilitate knowledge transfer sessions for seamless team integration and support.

Requirements

• Minimum of 5-10 years of progressive professional experience in Data Engineering roles.
• At least 2 years of hands-on, demonstrable experience with Databricks and Apache Spark.
• Proven expertise in enterprise-level data migration and modernization projects, particularly from legacy data platforms like DataIKU.
• Strong proficiency in PySpark, Spark SQL, Python, and Advanced SQL for data manipulation, analysis, and pipeline development.
• Solid understanding and practical experience with Azure Data Factory (ADF), Azure Data Lake Storage (ADLS Gen2), and Delta Lake architecture.
• Comprehensive experience in designing and developing robust ETL/ELT pipelines and effective data modeling strategies.
• Excellent analytical, problem-solving, troubleshooting, and communication skills to collaborate effectively with cross-functional teams.
• Familiarity with CI/CD tools such as Azure DevOps or GitHub Actions, Git, and Shell Scripting is highly desirable.

What We Offer

• Opportunity to work on cutting-edge cloud migration and modern data platform projects at the forefront of technology.
• A collaborative and innovative hybrid work environment based in Bangalore, fostering growth and learning.
• Exposure to industry-leading data technologies and architectures including Databricks, Azure, and Delta Lake.
• A chance to make a significant impact by shaping the future of enterprise data solutions for our clients.
• A challenging role that promotes continuous learning and professional development within a supportive team.