Senior Data Quality Engineer, ML (India) - Code and Theory

Job Overview

As a Senior Data Quality Engineer at Code and Theory, you will be part of the AI/ML engineering team, tasked with ensuring high-quality outputs from large language models (LLMs). Your role will involve collaborating with cross-disciplinary teams to build software solutions that solve real-world problems while maintaining high standards of performance, security, and accessibility.

Technical Requirements

Responsibilities

• • Write Python and SQL scripts to evaluate outputs from large language models (LLMs)
• • Design and implement LLM-as-Judge evaluations with clear scoring rubrics
• • Define and calculate quality metrics such as exact match, token-level F1, ROUGE, and subjective rubric scores
• • Build and maintain ground-truth datasets for benchmarking and regression testing
• • Automate evaluation pipelines and integrate them into CI/CD workflows
• • Conduct in-depth analysis of large unstructured datasets to identify inconsistencies and biases
• • Diagnose and report failure modes such as hallucinations and formatting errors
• • Collaborate with AI engineers, QA, data scientists, and product managers
• • Document processes and maintain reproducibility of evaluation runs
• • Create comprehensive technical documentation

Technical Environment