The Senior AI QA Engineer is responsible for developing and executing AI model evaluation strategies, implementing automated and manual testing for LLM-based applications, detecting biases and hallucinations, and collaborating with engineers to optimize model performance and ensure high-quality AI outputs.

• Develop and execute AI model evaluation strategies, ensuring accuracy, consistency, and fairness.
• Implement automated and manual testing for LLM-based applications.
• Collaborate with the AI Engineer to integrate testing into early-stage development.
• Build and manage test datasets, ensuring high-quality, diverse, and balanced samples.
• Develop synthetic data pipelines to enhance model evaluation.
• Design and maintain hallucination, bias, and robustness detection frameworks.
• Define and track AI performance metrics (e.g., factual accuracy, coherence, latency, response quality).
• Work closely with AI engineers to debug failures, identify root causes, and optimize model performance.
• Provide feedback on prompt effectiveness, suggest improvements, and collaborate with the Prompt Engineer to refine prompts.
• Implement continuous monitoring tools to track AI model drift, performance degradation, and unexpected failures.
• Develop and maintain comprehensive test reports, summarizing findings and recommendations.

• Experience with AI/ML testing frameworks and LLM evaluation methodologies.
• Strong understanding of LLM behaviors, biases, failure modes, and edge cases.
• Proficiency in Python and familiarity with ML testing frameworks (e.g., PyTest, Unittest, custom ML evaluation tools).
• Experience with test dataset management and annotation tools.
• Familiarity with synthetic data generation and adversarial testing techniques.
• Strong problem-solving and debugging skills to analyze AI failures and inconsistencies.
• Strong English language proficiency with the ability to evaluate AI-generated text and improve prompts.