Senior Data Engineer

We are actively seeking a highly skilled and experienced Senior Data Engineer for a long-term remote contract opportunity. In this pivotal role, you will be instrumental in designing, developing, and maintaining robust data platforms and pipelines, leveraging cutting-edge cloud and data technologies. This position is perfect for a seasoned professional passionate about data engineering, cloud architecture, and contributing to innovative AI/GenAI initiatives within a dynamic remote environment.

As a Senior Data Engineer, you will play a crucial role in shaping our data infrastructure, ensuring scalability, reliability, and security. You will work with a diverse tech stack, including Databricks, Spark, Snowflake, and a comprehensive suite of Azure Cloud Services, to deliver high-impact data solutions. If you thrive in a challenging yet rewarding remote work setting and are eager to contribute to the future of data and AI, we encourage you to apply.

Key Responsibilities

• Architect, develop, and maintain scalable and efficient data pipelines using Databricks, Spark (PySpark), and Python/Scala.
• Design, implement, and optimize data warehousing solutions, primarily utilizing Snowflake.
• Develop and manage ETL/ELT processes and data orchestration workflows with Azure Data Factory (ADF) and Airflow.
• Leverage Azure Cloud Services such as ADLS, Azure Functions, and Key Vault to build secure and resilient data platforms.
• Collaborate with data scientists and machine learning engineers on AI/GenAI projects, including LLM Fine-Tuning, RAG, and Prompt Engineering.
• Implement and maintain DevOps practices for data infrastructure, including CI/CD pipelines using GitHub and GitHub Actions, and infrastructure automation with Terraform.
• Monitor and troubleshoot data systems, ensuring data quality, integrity, and availability.
• Contribute to the strategic planning and architectural decisions for future data and analytics initiatives.
• Work independently and proactively manage project deliverables in a fast-paced remote setting.

Requirements

• Proven extensive experience as a Senior Data Engineer, preferably in long-term contract roles.
• Deep expertise with Databricks and Spark (PySpark or Scala) for large-scale data processing.
• Strong proficiency in Python, Scala, and SQL for data manipulation and analysis.
• Hands-on experience with Azure Cloud Services, including Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), Azure Functions, and Azure Key Vault.
• Solid understanding and practical experience with Snowflake data warehousing.
• Demonstrable experience with DevOps methodologies and tools like Airflow, GitHub, CI/CD, and Terraform.
• Familiarity with Kafka for real-time data streaming is highly desirable.
• Exposure to AI/GenAI concepts such as LLM Fine-Tuning, RAG, and Prompt Engineering.
• Exceptional problem-solving abilities and a strong commitment to delivering high-quality, robust data solutions.
• Excellent communication skills and the ability to work effectively in a fully remote team environment.

What We Offer

• An engaging long-term contract opportunity to work as a Senior Data Engineer on cutting-edge projects.
• The flexibility and convenience of a fully remote work environment.
• Opportunity to significantly impact the data strategy and AI initiatives of the company.
• Exposure to a modern and diverse technology stack, fostering continuous learning and skill development.
• A collaborative and supportive team culture focused on innovation and excellence.