Applied Scientist • Contract

Who you are

You have a robust background in machine learning and statistics, with experience in quality assurance and auditing methodologies. Your expertise in automated evaluation systems ensures the highest standards of data quality, which is critical for developing industry-leading technology. You are innovative and dedicated, ready to contribute to the Artificial General Intelligence (AGI) team at Amazon.

You possess strong analytical skills and are adept at designing comprehensive quality strategies. Your experience includes developing auditing frameworks that safeguard the integrity of data collection workflows. You are comfortable performing expert-level manual audits and conducting meta-audits to evaluate auditor performance, providing targeted coaching to uplift overall quality capabilities.

Your role will involve developing and maintaining LLM-as-a-Judge systems, where you will design judge architectures and create evaluation rubrics. You have a solid understanding of building machine learning models for automated quality assessment, which is essential for the success of the AGI team.

You are a collaborative team player, working closely with core scientists to develop Amazon Nova models. Your ability to communicate quality feedback to stakeholders will be vital in ensuring that the team meets its benchmarks and performance goals.

Desirable

Experience with large-scale data collection and analysis is a plus. Familiarity with the latest advancements in artificial intelligence and machine learning will enhance your contributions to the team.

What you'll do

As an Applied Scientist, you will lead the development of quality strategies and auditing frameworks for the AGI team. This includes designing detailed standard operating procedures (SOPs), quality metrics, and sampling methodologies that help improve the performance of Amazon Nova models on benchmarks. You will be responsible for setting up the configuration of data collection workflows and ensuring that they meet the highest standards of quality.

You will conduct expert-level manual audits and perform meta-audits to evaluate the performance of auditors. Your role will also involve providing targeted coaching to enhance the overall quality capabilities of the team. You will develop and maintain LLM-as-a-Judge systems, which will require you to design judge architectures and create evaluation rubrics that facilitate automated quality assessment.

Your contributions will directly impact the integrity of data collection workflows and the overall success of the AGI team. You will collaborate closely with core scientists, ensuring that the models developed meet Amazon's high standards for performance and reliability.

What we offer

This is a 12-month temporary contract opportunity with the possibility of extension based on business needs. Amazon offers competitive compensation, starting from $136,000 to $184,000 annually, depending on experience and qualifications. Starting Day 1 of employment, you will have access to various benefits, including EAP, Mental Health Support, and 401(k) matching. Join Amazon and be part of a team that is at the forefront of artificial intelligence and machine learning technology.