Principal AI QA Engineer

Job Description

We are seeking an experienced Principal AI QA Engineer to lead the quality strategy for enterprise Artificial Intelligence products, including Large Language Models (LLMs), Generative AI applications, AI agents, Retrieval-Augmented Generation (RAG) systems, and machine learning platforms. As a technical leader, you will define AI testing standards, develop scalable quality frameworks, and ensure AI solutions meet the highest levels of accuracy, reliability, safety, performance, and regulatory compliance.

In this role, you will work closely with AI researchers, machine learning engineers, software developers, product managers, MLOps engineers, and security teams to establish end-to-end quality assurance processes throughout the AI development lifecycle. You will drive automation, benchmark model performance, evaluate AI outputs, and lead initiatives to improve the robustness and trustworthiness of production AI systems.

The ideal candidate has deep expertise in software quality engineering, AI testing methodologies, test automation, cloud-native technologies, and modern Generative AI platforms.

Key Responsibilities

• Define and lead the enterprise AI Quality Assurance strategy across multiple AI products and platforms.
• Design and implement comprehensive testing frameworks for Large Language Models (LLMs), Generative AI applications, AI agents, and machine learning systems.
• Develop automated validation pipelines to evaluate model accuracy, consistency, robustness, fairness, and reliability.
• Lead functional, regression, integration, API, performance, security, and end-to-end testing for AI-powered applications.
• Design evaluation methodologies for prompt engineering, prompt optimization, and LLM response quality.
• Create benchmark datasets and automated scoring frameworks to measure AI model performance.
• Validate Retrieval-Augmented Generation (RAG) pipelines, vector search quality, and knowledge retrieval accuracy.
• Conduct adversarial testing to identify hallucinations, prompt injection vulnerabilities, jailbreak attempts, unsafe outputs, and model weaknesses.
• Monitor production AI systems for model drift, latency, reliability, inference quality, and operational health.
• Establish AI quality metrics, testing standards, governance processes, and release criteria.
• Collaborate with cross-functional engineering teams to resolve quality issues and improve AI system performance.
• Mentor QA engineers and promote best practices in AI quality engineering and test automation.
• Support Responsible AI initiatives by evaluating bias, fairness, explainability, privacy, and compliance requirements.
• Analyze production incidents, perform root cause analysis, and implement preventive quality improvements.
• Drive continuous improvement of AI testing frameworks, automation coverage, and engineering processes.

Required Qualifications

• Bachelor’s or Master’s degree in Computer Science, Artificial Intelligence, Software Engineering, Data Science, or a related technical field.
• 8+ years of experience in Software Quality Assurance, Test Automation, or Quality Engineering.
• 4+ years of hands-on experience testing AI, Machine Learning, or Generative AI systems.
• Strong experience with Large Language Models (LLMs), AI agents, and enterprise AI platforms.
• Advanced programming skills in Python and experience developing automated testing frameworks.
• Experience with RESTful API testing, automation tools, and CI/CD pipelines.
• Strong understanding of machine learning concepts, model evaluation, and AI validation techniques.
• Experience working with cloud platforms such as AWS, Azure, or Google Cloud Platform.
• Excellent analytical, problem-solving, leadership, and communication skills.

Preferred Qualifications

• Experience evaluating GPT-based models, Claude, Gemini, Llama, or other foundation models.
• Knowledge of Prompt Engineering and prompt evaluation techniques.
• Experience testing Retrieval-Augmented Generation (RAG) applications.
• Familiarity with vector databases and semantic search technologies.
• Experience with Docker, Kubernetes, and MLOps workflows.
• Knowledge of Responsible AI, AI governance, and model risk management.
• Experience performing AI security testing and adversarial evaluations.

Technical Skills

Programming & Scripting

• Python
• SQL
• Java
• JavaScript
• Bash

AI & Machine Learning

• Large Language Models (LLMs)
• Generative AI
• Retrieval-Augmented Generation (RAG)
• AI Agents
• Prompt Engineering
• Machine Learning
• Model Evaluation
• AI Benchmarking
• Responsible AI
• AI Safety Testing

Testing & Automation

• AI Model Validation
• Functional Testing
• Regression Testing
• Integration Testing
• Performance Testing
• Security Testing
• API Testing
• Test Automation
• End-to-End Testing

Tools & Platforms

• PyTest
• Playwright
• Selenium
• Postman
• JMeter
• Jenkins
• GitHub Actions
• Git
• Docker
• Kubernetes

Cloud Technologies

• AWS
• Microsoft Azure
• Google Cloud Platform (GCP)

Preferred Certifications

• ISTQB Advanced Test Automation Engineer
• AWS Certified Machine Learning – Specialty
• Microsoft Azure AI Engineer Associate
• Google Professional Machine Learning Engineer
• Certified Kubernetes Application Developer (CKAD)

Benefits

• Annual performance bonus and potential equity/stock awards.
• Comprehensive medical, dental, and vision insurance.
• Flexible paid time off and company holidays.
• Remote and hybrid work opportunities.
• Professional development and certification reimbursement.
• Wellness and employee assistance programs.
• Paid parental leave.