Databricks Senior Data Engineer - Pyspark

Ascendra-IT is urgently seeking an exceptionally skilled and experienced Databricks Senior Data Engineer with profound expertise in PySpark to join our innovative team on an onsite basis in New York. This critical role offers an exceptional opportunity for a seasoned professional with 11+ years of experience to lead the design, development, and optimization of robust data pipelines and advanced analytics solutions within a fast-paced environment. As a pivotal member of our data engineering group, you will leverage your advanced skills in Databricks and PySpark to build and maintain scalable, efficient, and high-performance data platforms that drive strategic business insights for our valued clients. We are looking for a proactive individual who thrives on solving complex data challenges and contributing to impactful projects, focusing on delivering high-quality data solutions using the latest technologies.

Key Responsibilities

• Architect, design, develop, and maintain highly scalable and efficient data pipelines and ETL processes using Databricks and PySpark.
• Implement robust data ingestion, transformation, and loading strategies from diverse data sources into data lakes and data warehouses, ensuring data quality and integrity.
• Optimize Databricks notebooks, PySpark jobs, and underlying infrastructure for superior performance, cost-efficiency, and operational reliability, handling large volumes of data.
• Collaborate closely with data scientists, data analysts, and cross-functional engineering teams to translate complex business requirements into tangible data solutions and ensure data availability for analytics and machine learning initiatives.
• Establish and enforce best practices for data engineering, including code review, version control with Git, automated testing, continuous integration/continuous deployment (CI/CD) pipelines, and documentation.
• Monitor, troubleshoot, and resolve issues within data pipelines and data systems, ensuring high availability, accuracy, and timely delivery of data.
• Contribute to data governance frameworks, security protocols, and compliance standards across all data platforms and processes.
• Provide technical leadership and mentorship to junior data engineers, fostering a culture of technical excellence, innovation, and continuous learning within the team.
• Stay abreast of emerging technologies and industry trends in Big Data, cloud computing, and data engineering, proposing and implementing improvements to our existing architecture.

Requirements

• A minimum of 11 years of progressive professional experience in data engineering, with a strong emphasis on designing and implementing Big Data solutions.
• Demonstrated expert-level, hands-on experience with the Databricks platform and extensive proficiency in developing, deploying, and optimizing data solutions using PySpark.
• Exceptional programming skills in Python and advanced proficiency in SQL for complex data manipulation, analysis, and database management.
• Solid understanding and practical experience with data warehousing concepts, dimensional modeling, and schema design for analytical databases.
• Experience working with major cloud platforms such as AWS, Azure, or GCP, specifically utilizing their data-related services (e.g., S3, ADLS, GCS, EMR, Data Factory, BigQuery).
• Familiarity with containerization technologies like Docker and orchestration tools like Kubernetes in a data context is a plus.
• Strong understanding of distributed computing principles and experience with other Big Data technologies (e.g., Apache Spark, Hadoop ecosystem) is highly desirable.
• Excellent problem-solving, analytical, and critical thinking skills, with the ability to debug and optimize complex data flows.
• Superb communication and interpersonal skills, capable of effectively collaborating with technical and non-technical stakeholders and presenting complex technical information clearly.
• Bachelor's or Master's degree in Computer Science, Engineering, or a related quantitative field.

What We Offer

• A stimulating and rewarding contract opportunity within a dynamic and forward-thinking IT solutions provider, working on impactful projects.
• Exposure to cutting-edge technologies and complex, large-scale data challenges that will enhance your professional profile as a Databricks Senior Data Engineer.
• A collaborative, supportive, and inclusive work environment that encourages innovation, continuous learning, and significant professional growth.
• The opportunity to play a crucial role in delivering strategic client initiatives and shaping the future of data architecture and analytics.
• Highly competitive compensation package reflecting your extensive experience and expertise in the field.