AI Evaluation Specialist (Emerging Role)

Job Description

We are seeking a detail-oriented AI Evaluation Specialist to evaluate, benchmark, and improve the performance of cutting-edge Artificial Intelligence systems, including Large Language Models (LLMs), Generative AI applications, AI agents, and multimodal models. This emerging role sits at the intersection of AI Quality Assurance, Data Analytics, Prompt Engineering, and Responsible AI. The ideal candidate will help ensure AI systems deliver accurate, safe, unbiased, and high-quality outputs while supporting continuous model improvement.

Key Responsibilities

• Evaluate AI-generated content for accuracy, relevance, coherence, safety, and factual correctness.
• Develop evaluation frameworks, scoring rubrics, and benchmark datasets for AI model assessment.
• Conduct prompt testing and adversarial testing to identify weaknesses and edge cases.
• Analyze model outputs to detect hallucinations, bias, toxicity, and performance gaps.
• Collaborate with AI Engineers and Data Scientists to improve model quality and user experience.
• Perform Human-in-the-Loop (HITL) reviews and provide structured feedback for model refinement.
• Design and execute A/B testing experiments for AI applications.
• Monitor production AI systems and track key performance metrics.
• Review training datasets and identify data quality issues.
• Support Responsible AI, AI Governance, and compliance initiatives.
• Create reports and recommendations for leadership and product teams.
• Stay current with emerging trends in Generative AI, Agentic AI, and AI evaluation methodologies.

Required Qualifications

• Bachelor’s degree in Computer Science, Artificial Intelligence, Data Science, Linguistics, Mathematics, Psychology, or a related field.
• 1–3 years of experience in AI evaluation, data annotation, quality assurance, machine learning operations, or content analysis.
• Strong analytical and critical-thinking skills.
• Excellent written communication and documentation abilities.
• Understanding of Generative AI, Machine Learning, and Large Language Models.
• Experience working with structured evaluation processes and quality standards.

Preferred Qualifications

• Experience with Prompt Engineering and LLM optimization.
• Familiarity with AI Safety, Responsible AI, and model governance practices.
• Knowledge of Python, SQL, and data visualization tools.
• Experience with AI benchmarking platforms and evaluation metrics.
• Exposure to Retrieval-Augmented Generation (RAG) systems and AI agents.
• Experience evaluating multilingual AI systems.

Technical Skills

• AI Model Evaluation
• Large Language Models (LLMs)
• Prompt Engineering
• Generative AI Testing
• AI Benchmarking
• Human-in-the-Loop (HITL) Evaluation
• Python
• SQL
• Data Analytics
• AI Safety & Governance
• Machine Learning Fundamentals
• Quality Assurance (QA)

Key Competencies

• Critical Thinking
• Attention to Detail
• Problem Solving
• Research & Analysis
• Communication Skills
• Decision Making
• Collaboration
• Ethical AI Awareness