Data Scientist, Evaluations – AI

Job Description

Meta is seeking a highly motivated Data Scientist to join the Evaluations team within Meta Superintelligence Labs (MSL). This team plays a critical role in measuring and accelerating AI progress by developing evaluation frameworks, benchmarks, and methodologies for advanced AI systems and Large Language Models (LLMs).

As a Data Scientist, Evaluations, you will partner with AI researchers, machine learning engineers, and product teams to design metrics that assess model capabilities, safety, reliability, and performance. Your work will directly influence the development and deployment of next-generation AI technologies.

Key Responsibilities

• Design and develop evaluation frameworks for AI and Generative AI systems.
• Build quantitative metrics to assess model performance, accuracy, and safety.
• Analyze large-scale datasets and AI model outputs.
• Create benchmark datasets for testing and validation.
• Conduct statistical analyses to identify model strengths and weaknesses.
• Collaborate with AI researchers to improve evaluation methodologies.
• Develop dashboards and reporting tools to track model performance.
• Monitor trends in AI quality, fairness, and reliability.
• Support experimentation and A/B testing for AI products.
• Present insights and recommendations to technical and executive stakeholders.

Required Qualifications

• Bachelor’s, Master’s, or Ph.D. in Computer Science, Data Science, Statistics, Mathematics, Machine Learning, or a related field.
• 4+ years of experience in Data Science, Machine Learning, Analytics, or AI Evaluation.
• Strong knowledge of statistical modeling and experimental design.
• Experience working with large datasets.
• Proficiency in Python and SQL.
• Experience with machine learning and AI systems.
• Strong communication and presentation skills.

Preferred Qualifications

• Experience evaluating Large Language Models (LLMs).
• Knowledge of Generative AI systems.
• Familiarity with Reinforcement Learning from Human Feedback (RLHF).
• Experience with model benchmarking and AI safety evaluation.
• Background in machine learning research.
• Experience working with cloud platforms and distributed data systems.

Technical Skills

• Python
• SQL
• Machine Learning
• Data Science
• Statistics
• Experimental Design
• A/B Testing
• Generative AI
• Large Language Models (LLMs)
• Data Visualization
• Tableau / Power BI
• AI Benchmarking

Day-to-Day Activities

• Evaluate AI model responses across multiple tasks.
• Analyze performance metrics and identify improvement opportunities.
• Develop new benchmarks for measuring AI capabilities.
• Collaborate with research teams on evaluation experiments.
• Create reports and visualizations for stakeholders.
• Support the development of safer and more reliable AI systems.
• Investigate model failures and recommend improvements.

Benefits

• Competitive salary and performance bonuses.
• Health, dental, and vision insurance.
• Retirement savings plans.
• Paid time off and wellness programs.
• Professional development opportunities.
• Access to cutting-edge AI research and technology.