Applied Scientist • Mid-Level

Who you are

You have a robust background in machine learning and statistics, with experience in quality assurance and auditing methodologies. Your expertise in automated evaluation systems ensures the highest standards of data quality, and you are skilled in building industry-leading technology with Large Language Models (LLMs) and multimodal systems.

You thrive in collaborative environments, working closely with core scientist teams to develop innovative models. Your ability to lead the development of comprehensive quality strategies and auditing frameworks is essential for safeguarding the integrity of data collection workflows.

You are adept at designing auditing strategies, including detailed SOPs, quality metrics, and sampling methodologies that help improve performance on benchmarks. Your experience includes performing expert-level manual audits and conducting meta-audits to evaluate auditor performance, providing targeted coaching to uplift overall quality capabilities.

You have a critical mindset, focusing on developing and maintaining LLM-as-a-Judge systems. This involves designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. You are also responsible for setting up the configuration of data collection workflows and effectively communicating quality feedback to stakeholders.

What you'll do

As an Applied Scientist, you will collaborate with the AGI team to lead the development of quality strategies that enhance the performance of Amazon Nova models. You will design and implement comprehensive auditing frameworks that ensure data integrity and quality throughout the data collection process.

Your role will involve conducting expert-level audits and meta-audits, evaluating auditor performance, and providing coaching to improve quality capabilities across the team. You will also be responsible for developing LLM-as-a-Judge systems, which includes designing architectures and creating evaluation rubrics that support automated quality assessments.

You will work closely with stakeholders to communicate quality feedback and insights, helping to drive improvements in data collection workflows. Your contributions will directly impact the effectiveness of the AGI team's initiatives, ensuring that the technology developed meets the highest standards of quality and performance.

What we offer

This is a 12-month temporary contract opportunity with the possibility of extension based on business needs. Amazon offers competitive compensation, with a starting salary range of $136,000.00 - $184,000.00 USD annually, depending on experience and qualifications. Starting Day 1 of employment, you will have access to benefits including EAP, Mental Health Support, and a 401(k) matching program. Join a team that is at the forefront of developing innovative AI technologies and make a significant impact in the field of Artificial General Intelligence.