Applied Scientist • Lead

Who you are

You have a robust background in machine learning and statistics, with at least 5 years of experience in applied science or a related field. Your expertise includes quality assurance and auditing methodologies, which you have applied in previous roles to ensure high standards of data quality. You are skilled in developing automated evaluation systems and have a strong understanding of Large Language Models (LLMs) and multimodal systems. Your leadership experience includes mentoring teams and guiding them in designing effective auditing strategies and quality metrics.

You possess excellent communication skills, enabling you to effectively convey complex concepts to both technical and non-technical stakeholders. You are detail-oriented and have a proven track record of developing comprehensive quality strategies that align with organizational goals. Your ability to coach and uplift team capabilities is a key strength, and you thrive in collaborative environments where you can drive innovation and quality improvements.

What you'll do

As the Applied Science Manager, you will lead a team of Applied Scientists in the Artificial General Intelligence (AGI) team at Amazon. Your primary responsibility will be to develop and implement quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. You will oversee the design of auditing strategies, including detailed standard operating procedures (SOPs) and quality metrics, ensuring alignment with the core scientist team developing Amazon Nova models.

You will manage expert-level manual audits and meta-audits to evaluate auditor performance, providing coaching to enhance overall quality capabilities across the team. A critical aspect of your role will involve overseeing the development and maintenance of LLM-as-a-Judge systems, which includes designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment.

You will also be responsible for configuring data collection workflows and ensuring effective communication of quality feedback to stakeholders. Your leadership will directly impact customer experiences by ensuring high-quality training and evaluation data that supports the development of industry-leading technology.

What we offer

Amazon offers a competitive salary range of $183,800 to $248,700 annually, along with a comprehensive benefits package that includes health insurance, retirement plans, and paid time off. You will have the opportunity to work in a dynamic environment that fosters innovation and collaboration. We encourage you to apply even if your experience doesn't match every requirement, as we value diverse perspectives and backgrounds.