Data Architect - Databricks & DBT Migration

Sparshcorp is actively seeking an experienced and highly skilled Data Architect / Lead with a strong background in Databricks and DBT migration. This pivotal role involves leading large-scale data transformation projects, specifically migrating existing data infrastructure from Snowflake and AWS Glue to a modern Databricks Lakehouse platform. If you are a visionary Data Architect passionate about building robust, scalable, and high-performance data solutions, we invite you to join our dynamic team and drive significant impact in our data engineering initiatives.

Key Responsibilities

• Lead the comprehensive end-to-end migration strategy and architecture design for transitioning data platforms from Snowflake/AWS Glue to Databricks.
• Design, develop, and implement scalable data transformation models using DBT (Data Build Tool) to ensure data quality, consistency, and reusability.
• Architect and optimize Python-based ETL pipelines for efficient data ingestion, processing, and storage within the Databricks environment.
• Establish and enforce Lakehouse architecture principles (Bronze, Silver, Gold layers) to ensure data governance, accessibility, and performance.
• Collaborate closely with global stakeholders, including product owners, data scientists, and engineering teams, to understand requirements and deliver impactful data solutions.
• Conduct performance tuning and optimization of data pipelines and Databricks clusters to enhance processing speed and reduce operational costs.
• Mentor and guide junior and mid-level Data Engineering teams, fostering a culture of technical excellence and continuous learning.
• Stay abreast of emerging technologies and best practices in the data engineering and cloud analytics space, recommending innovative solutions to improve our data infrastructure.

Requirements

• Minimum of 7+ years of progressive experience in data architecture, data engineering, or a related field.
• Proven expertise in designing and implementing large-scale data solutions using Databricks and Apache Spark.
• Extensive hands-on experience with DBT for data modeling, transformation, and pipeline orchestration.
• Demonstrable experience in migrating data platforms from Snowflake or AWS Glue to Databricks is essential.
• Proficiency in Python for developing complex ETL processes and data manipulation scripts.
• Deep understanding of Lakehouse architecture patterns and best practices, including data layering and metadata management.
• Strong knowledge of SQL for data querying, analysis, and validation.
• Excellent communication, leadership, and collaboration skills, with the ability to influence technical and non-technical stakeholders in a fast-paced environment.

What We Offer

• An opportunity to lead and shape the data strategy of a forward-thinking organization.
• A challenging and innovative work environment where your contributions have a direct impact on business outcomes.
• Exposure to cutting-edge data technologies and large-scale, complex migration projects.
• Collaborative culture with opportunities for continuous learning and professional growth.
• Competitive compensation and a chance to work with a talented team of data professionals.