AI Automation QA Engineer

Job Summary

We are looking for a skilled AI Automation QA Engineer to design, develop, and maintain automated testing solutions for AI-powered applications, machine learning models, and Generative AI systems. In this role, you will collaborate with AI Engineers, Software Developers, Data Scientists, and DevOps teams to ensure AI applications are reliable, scalable, secure, and production-ready. The ideal candidate has strong experience in automation testing, API validation, CI/CD pipelines, and testing AI-driven products such as LLMs, chatbots, recommendation engines, and intelligent automation platforms.

Key Responsibilities

• Design and implement automated test frameworks for AI and machine learning applications.
• Develop and maintain UI, API, regression, integration, and end-to-end automated test suites.
• Validate AI model outputs for accuracy, consistency, reliability, and performance.
• Test Large Language Models (LLMs), AI copilots, chatbots, and Retrieval-Augmented Generation (RAG) applications.
• Create automated test scripts using Python, Selenium, Playwright, or similar frameworks.
• Automate API testing using REST Assured, Postman, or similar tools.
• Integrate automated testing into CI/CD pipelines using Jenkins, GitHub Actions, or Azure DevOps.
• Monitor AI application performance, identify defects, and collaborate with development teams to resolve issues.
• Validate training datasets, model deployments, and inference pipelines.
• Conduct performance, load, security, and compatibility testing for AI applications.
• Generate detailed test reports, defect documentation, and quality metrics.
• Support continuous improvement of QA processes and automation frameworks.

Required Qualifications

• Bachelor’s degree in Computer Science, Software Engineering, Information Technology, or a related field.
• 3–6 years of experience in QA Automation or Software Testing.
• Experience testing AI/ML or Generative AI applications is highly preferred.
• Strong programming skills in Python, Java, or JavaScript.
• Hands-on experience with Selenium, Playwright, Cypress, or similar automation frameworks.
• Experience with REST API testing and API automation.
• Knowledge of software development lifecycle (SDLC) and Agile methodologies.
• Familiarity with machine learning concepts and AI workflows.
• Experience with Git, Docker, Kubernetes, and CI/CD tools.
• Excellent analytical, problem-solving, and communication skills.

Preferred Skills

• Large Language Models (LLMs)
• Prompt Engineering
• LangChain or LangGraph
• Retrieval-Augmented Generation (RAG)
• OpenAI APIs or similar AI platforms
• Vector Databases (Pinecone, FAISS, ChromaDB)
• SQL and NoSQL databases
• Jenkins, GitHub Actions, Azure DevOps
• AWS, Azure, or Google Cloud Platform
• Performance testing tools (JMeter, k6)
• AI model evaluation and Responsible AI principles

Technical Skills

• Python
• Selenium
• Playwright
• PyTest
• REST API Testing
• Postman
• SQL
• Git & GitHub
• Jenkins
• Docker
• Kubernetes
• Azure DevOps
• AWS / Azure / GCP
• LangChain
• OpenAI API
• JMeter
• Machine Learning Fundamentals

Benefits

• Competitive salary with annual performance bonus.
• Medical, dental, and vision insurance.
• Paid vacation, holidays, and parental leave.
• Flexible remote or hybrid work options.
• Professional development and AI certification reimbursement.
• Stock options or equity (company-dependent).
• Career growth opportunities in AI Engineering and Automation.