Applied Scientist • Mid-Level

Who you are

You have a robust background in machine learning and statistics, with experience in quality assurance and auditing methodologies. Your expertise in automated evaluation systems ensures the highest standards of data quality, which is critical for building industry-leading technology. You are innovative and dedicated, ready to collaborate with a core scientist team to develop Amazon Nova models.

You possess strong analytical skills and are comfortable designing comprehensive quality strategies and auditing frameworks. Your ability to create detailed SOPs, quality metrics, and sampling methodologies will help improve performance benchmarks for the AGI team. You are also skilled in conducting expert-level manual audits and meta-audits to evaluate auditor performance, providing targeted coaching to uplift overall quality capabilities.

Your experience includes developing and maintaining LLM-as-a-Judge systems, where you design judge architectures and create evaluation rubrics. You are adept at building machine learning models for automated quality assessment and setting up configurations for data collection workflows. Your communication skills enable you to effectively relay quality feedback to stakeholders, ensuring alignment and understanding across teams.

What you'll do

As an Applied Scientist, you will lead the development of quality strategies that safeguard the integrity of data collection workflows. You will collaborate closely with the core scientist team, focusing on enhancing the performance of Amazon Nova models. Your role will involve designing auditing strategies that include detailed SOPs and quality metrics, which are essential for maintaining high standards in data quality.

You will perform expert-level manual audits and conduct meta-audits to evaluate the performance of auditors, providing coaching to enhance their capabilities. A critical aspect of your work will be developing LLM-as-a-Judge systems, which involves designing judge architectures and creating evaluation rubrics that facilitate automated quality assessments.

You will also be responsible for setting up the configuration of data collection workflows, ensuring that they align with the quality standards established by the AGI team. Your ability to communicate quality feedback to stakeholders will play a vital role in the continuous improvement of data integrity and performance metrics.

What we offer

This position is a 12-month temporary contract opportunity with the possibility of extension based on business needs. Amazon offers a competitive salary range of $136,000.00 - $184,000.00 USD annually, depending on experience and qualifications. Starting Day 1 of employment, you will have access to benefits such as EAP, Mental Health Support, and a 401(k) matching program. Join Amazon and contribute to groundbreaking advancements in Artificial General Intelligence while enjoying a supportive work environment.