Data Engineer

AiLogic Neural Network Pvt. Ltd. is seeking a talented and passionate Data Engineer to join our innovative team in Hyderabad. This is an excellent opportunity for professionals who are enthusiastic about leveraging Python and NLP to construct robust data pipelines for cutting-edge AI applications. The successful Data Engineer will play a crucial role in transforming raw, unstructured data into valuable assets for our advanced AI and Large Language Model (LLM) initiatives.

Key Responsibilities

• Design, build, and maintain scalable data pipelines for AI and LLM applications.
• Implement efficient PDF parsing, OCR, and advanced text extraction techniques.
• Perform comprehensive text preprocessing, normalization, and chunking on diverse datasets.
• Apply Natural Language Processing (NLP) and tokenization methods to unstructured text.
• Develop and optimize data deduplication and masking processes to ensure data quality and privacy.
• Work with vector databases and develop document processing pipelines to support AI models.
• Collaborate with AI researchers and machine learning engineers to understand data requirements and deliver tailored solutions.
• Ensure data integrity, reliability, and accessibility for various AI development stages.

Requirements

• Strong proficiency in Python programming is essential.
• Proven hands-on experience with unstructured text and document processing.
• Familiarity with NLP techniques, including tokenization and text normalization.
• Experience with Hugging Face transformers library is highly desirable.
• Demonstrated experience with Apache Spark or PySpark for big data processing.
• Understanding of vector databases and their application in AI data pipelines.
• Ability to prepare and structure data specifically for AI and LLM applications.
• Strong problem-solving skills and an analytical mindset.

What We Offer

• An exciting opportunity to work on cutting-edge AI and LLM technologies.
• A dynamic and collaborative work environment at the forefront of neural network innovation.
• The chance to make a significant impact on the development of intelligent systems.
• Professional growth and learning opportunities in a rapidly evolving field.
• Competitive compensation and a supportive team culture.