Applied Scientist • Mid-Level

Who you are

You have a robust background in machine learning and statistics, with experience in quality assurance and auditing methodologies. Your expertise in automated evaluation systems ensures the highest standards of data quality, which is critical for building industry-leading technology with Large Language Models (LLMs) and multimodal systems.

You are skilled in designing comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. Your ability to create detailed SOPs, quality metrics, and sampling methodologies will help improve performance benchmarks for Amazon Nova models.

You possess strong analytical skills and can perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. Your experience in developing and maintaining LLM-as-a-Judge systems, including designing judge architectures and creating evaluation rubrics, is essential for this role.

You are a collaborative team player who communicates effectively with stakeholders, providing quality feedback and insights to enhance data collection workflows. Your innovative mindset drives you to continuously seek improvements in quality assessment processes.

What you'll do

As an Applied Scientist on the AGI team, you will lead the development of quality strategies that ensure the integrity of data collection workflows. You will design auditing strategies and create detailed SOPs that help the Nova models improve their performance on benchmarks.

Your role will involve conducting expert-level manual audits and meta-audits to evaluate auditor performance, providing targeted coaching to enhance the overall quality capabilities of the team. You will also develop and maintain LLM-as-a-Judge systems, designing judge architectures and creating evaluation rubrics for automated quality assessment.

You will set up the configuration of data collection workflows and communicate quality feedback to stakeholders, ensuring that the highest standards of data quality are met. Your contributions will directly impact the success of the AGI team and the technology being developed at Amazon.

What we offer

This is a 12-month temporary contract opportunity with the possibility of extension based on business needs. Starting pay will be based on factors including experience, qualifications, and location, with a salary range of $136,000.00 - $184,000.00 USD annually. Amazon offers a comprehensive benefits package starting Day 1 of employment, including EAP, Mental Health Support, Medical Advice Line, and 401(k) matching.