AWS Data Engineer - Databricks

Are you an experienced AWS Data Engineer passionate about building robust and scalable data solutions? Join our dynamic team at IBM, where we are at the forefront of leveraging cutting-edge cloud technologies to transform data into actionable insights. We are seeking a highly skilled and motivated AWS Data Engineer with a strong background in PySpark and SQL to design, develop, and optimize our next-generation data platforms.

This pivotal role offers an exciting opportunity to work on complex data challenges, contributing to critical projects that drive business intelligence and innovation. If you thrive in a collaborative environment and possess a deep understanding of cloud-based data engineering, we invite you to be a part of our journey as an AWS Data Engineer.

Key Responsibilities

• Design, develop, and maintain highly scalable and reliable data pipelines using core AWS services, ensuring data integrity and performance.
• Implement and optimize ETL processes for extracting, transforming, and loading data from various sources into data lakes and warehouses.
• Utilize PySpark for advanced data processing, transformation, and analytics on large datasets within the AWS ecosystem.
• Develop and manage data solutions using AWS Lambda, AWS Step Functions, AWS Glue, and AWS S3 to support diverse business requirements.
• Collaborate closely with data scientists, analysts, and other engineering teams to understand data needs and deliver effective data solutions.
• Ensure the quality, security, and governance of data throughout its lifecycle, adhering to best practices and industry standards.
• Monitor and troubleshoot data pipeline performance, proactively identifying and resolving issues to ensure continuous data flow.
• Contribute to the continuous improvement of data engineering processes, tools, and methodologies within the IBM framework.

Requirements

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related technical field.
• A minimum of 6 years of professional experience in data engineering, with a strong focus on AWS cloud environments.
• Demonstrable expertise in various AWS services including AWS Lambda, AWS Step Functions, AWS Glue, and AWS S3.
• Proven hands-on experience with PySpark for big data processing and strong proficiency in Python.
• Exceptional SQL skills for data querying, manipulation, and optimization of relational databases.
• Solid understanding of data warehousing concepts, ETL methodologies, and data modeling techniques.
• Familiarity with data governance, security, and compliance best practices in a cloud environment.
• Ability to join with immediate or a short notice period is highly preferred.

What We Offer

• An engaging and challenging work environment at a global technology leader, IBM.
• Opportunities for professional growth and continuous learning in cutting-edge cloud and data technologies.
• Work on impactful projects that leverage your skills to solve real-world business problems.
• A collaborative culture that encourages innovation, teamwork, and knowledge sharing.
• Competitive compensation package and comprehensive benefits.