Foster City, CA
All roles / Amazon / Manager II, Program Management, AI Model Evaluation, International Seller Growth - ISG
Manager II, Program Management, AI Model Evaluation, International Seller Growth - ISG
Source verified: this exact posting was present on Amazon’s careers feed on . View employer source →
What the posting asks for
- Names
- SQL
- Doctorate
- Not mentioned
Read out of the employer's own description. Absence means the posting does not say, not that the answer is no.
The role
Join the Seller AI team where you'll lead benchmarking and evaluation of AI models that enhance the seller experience across Amazon's global marketplace. You'll manage a team dedicated to validating, testing, and improving Artificial Intelligence (AI) and Large Language Models (LLMs) that power innovative seller tools. This role combines strategic leadership with hands-on technical oversight, requiring exceptional communication skills, team management, and stakeholder engagement.
In this position, you'll drive the development and implementation of comprehensive benchmarking methodologies to evaluate AI model performance across accuracy, robustness, bias, and reliability metrics. Your expertise will be crucial in translating technical findings into actionable insights that improve Seller Assistant's performance and contribute to the growth of Amazon's seller community worldwide.
Key job responsibilities
1. Lead LLM based automated evaluations program, Plan and execute benchmarking exercises for AI models, defining test plans, metrics, and acceptance criteria across accuracy, robustness, bias, and reliability dimensions.
2. Lead a team responsible for validating data based on specific annotation guidelines, ensuring accuracy and quality while escalating potential regulatory risks
3. Prepare comprehensive audit and benchmarking reports, including error ratings, root-cause analysis, and recommendations for senior stakeholders
4. Drive process efficiencies and explore automation opportunities to enhance the productivity of data generation initiatives
5. Mentor team members and help develop their skills while managing overall schedules, proactively mitigating risks, and keeping project scope under control.
A day in the life
Your day begins with prioritizing benchmarking tasks for your team based on current business requirements. You'll review progress on ongoing audits, provide guidance where needed, and ensure deliverables meet quality standards. Throughout the day, you'll analyze performance data from Seller Assistant tools, identifying patterns and insights that could improve AI model effectiveness. You'll wrap up by preparing detailed presentations that showcase your team's findings and recommendations to leadership, highlighting opportunities to enhance the seller experience.
About the team
The Seller AI team within International Seller Services is dedicated to creating Gen-AI/LLM powered tools and agentic solutions that accelerate business growth for Amazon sellers worldwide. We focus on handling annotations for training, measuring, and improving Artificial Intelligence and Large Language Models to deliver superior seller experiences. Our team combines technical expertise with a deep understanding of seller needs to develop innovative solutions that simplify complex tasks and drive business growth.
By joining us, you'll play a pivotal role in shaping the future of selling on Amazon, working with advanced AI technologies that directly impact millions of sellers across global marketplaces.
- 3+ years of program or project management experience
- 3+ years of working cross functionally with tech and non-tech teams experience
- 3+ years of delivering cross functional projects experience
- Experience defining program requirements and using data and metrics to determine improvements
- Experience in data mining, data management, reporting, and SQL queries
Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding process, please visit https://amazon.jobs/content/en/how-we-hire/accommodations for more information. If the country/region you’re applying in isn’t listed, please contact your Recruiting Partner.
In this position, you'll drive the development and implementation of comprehensive benchmarking methodologies to evaluate AI model performance across accuracy, robustness, bias, and reliability metrics. Your expertise will be crucial in translating technical findings into actionable insights that improve Seller Assistant's performance and contribute to the growth of Amazon's seller community worldwide.
Key job responsibilities
1. Lead LLM based automated evaluations program, Plan and execute benchmarking exercises for AI models, defining test plans, metrics, and acceptance criteria across accuracy, robustness, bias, and reliability dimensions.
2. Lead a team responsible for validating data based on specific annotation guidelines, ensuring accuracy and quality while escalating potential regulatory risks
3. Prepare comprehensive audit and benchmarking reports, including error ratings, root-cause analysis, and recommendations for senior stakeholders
4. Drive process efficiencies and explore automation opportunities to enhance the productivity of data generation initiatives
5. Mentor team members and help develop their skills while managing overall schedules, proactively mitigating risks, and keeping project scope under control.
A day in the life
Your day begins with prioritizing benchmarking tasks for your team based on current business requirements. You'll review progress on ongoing audits, provide guidance where needed, and ensure deliverables meet quality standards. Throughout the day, you'll analyze performance data from Seller Assistant tools, identifying patterns and insights that could improve AI model effectiveness. You'll wrap up by preparing detailed presentations that showcase your team's findings and recommendations to leadership, highlighting opportunities to enhance the seller experience.
About the team
The Seller AI team within International Seller Services is dedicated to creating Gen-AI/LLM powered tools and agentic solutions that accelerate business growth for Amazon sellers worldwide. We focus on handling annotations for training, measuring, and improving Artificial Intelligence and Large Language Models to deliver superior seller experiences. Our team combines technical expertise with a deep understanding of seller needs to develop innovative solutions that simplify complex tasks and drive business growth.
By joining us, you'll play a pivotal role in shaping the future of selling on Amazon, working with advanced AI technologies that directly impact millions of sellers across global marketplaces.
Basic qualifications
- 3+ years of team management experience- 3+ years of program or project management experience
- 3+ years of working cross functionally with tech and non-tech teams experience
- 3+ years of delivering cross functional projects experience
- Experience defining program requirements and using data and metrics to determine improvements
Preferred qualifications
- Experience implementing repeatable processes and driving automation or standardization- Experience in data mining, data management, reporting, and SQL queries
Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding process, please visit https://amazon.jobs/content/en/how-we-hire/accommodations for more information. If the country/region you’re applying in isn’t listed, please contact your Recruiting Partner.
Published by Amazon on their own careers page and reproduced here unedited. Read it at Amazon.
Apply at Amazon → Applications go directly to Amazon. This board does not sit in between, take a fee from you, or see your application.
What this listing does not tell you
Listed 2 days, which is recent for this board. Of the 114 evals and benchmarks roles this board has watched from listing to removal, 0% were gone from their employer's careers page by day 2, and the median came down after 56 days. That is a description of other listings that have already ended, not a prediction about this one: this board records when a listing disappears, never why, and a posting still up is not on a clock it can see.
Pay is not confirmed for this role. Check the employer’s posting for current compensation. Explore published bands from other roles →
Amazon has 32 roles open on this board, 12 of them in evals and benchmarks.
Get the weekly AI Evals Jobs briefNew roles and board updates, with published pay where available. This is the general weekly brief. Or browse them all now.
Free. One email on Thursdays.
More roles like this
Matched by discipline, title, listed location and work arrangement.
Same discipline: Evals & Benchmarks · Both have management titles
New York, NY · San Francisco, CA +1
Same discipline: Evals & Benchmarks · Both have management titles
Santa Clara, CA
Same discipline: Evals & Benchmarks · Both have management titles
London$215k – $274k
Same discipline: Evals & Benchmarks · Both have management titles
Mountain View, CA, USA · San Francisco, CA, USA$251k – $310k
Same discipline: Evals & Benchmarks · Both have management titles