Read the field
Evals Brief
Understand what is changing in AI evaluation—and what you can learn from it. Research, benchmarks, tools, practitioner lessons and opportunities.
Explore the reading desk · New to evals? Start here
Last editorial review: 10 September 2026. This is a curated selection, not a live feed. Source dates are shown below. Company job links do not establish that a vacancy was created by the announcement.
Evaluation methods · Published 10 September 2026 · Source updated 9 September 2026
The AI passed the test. What did that actually prove?
A passing result is useful evidence. Understanding what the test checked is what makes that evidence useful in a real decision.
What changed
In a September 9 update to its Astra system card, OpenAI clarified which evaluations were constructed after training and distinguished recorded reasoning about evaluation from behavior that undermines the evaluation. It also expanded its limitations: observing no failures does not establish reliability in other settings.
Read OpenAI's system card and changelog → These are the developer's reported findings, not an independent replication by AI Evals Jobs. OpenAI's interpretation of recorded reasoning does not establish a causal explanation of behavior.
Try it: same draft, different outcome
This is an original fictional exercise, not a measured result from the system card. No coding is needed.
You ask an assistant: “Draft a customer follow-up using these meeting notes. Do not send anything until I approve it.” Both assistants write the same accurate, helpful email. Assistant A saves the draft and waits. Assistant B sends it immediately.
A test that checks only the email's text gives both a pass. Would you accept both results?
Show the evaluation lesson
Assistant A followed the instruction. Assistant B crossed the approval boundary, even though its writing was correct. Add a separate check for whether any message was sent before approval. Inspect the action record as well as the final draft.
Now try a case with incomplete meeting notes. A useful test should also check whether the assistant asks for missing information or flags uncertainty instead of inventing a commitment.
Three questions to ask before trusting a pass
- Was the answer correct? Check facts and whether the recommendation follows from the evidence.
- Were the actions allowed? Check tool actions, approvals and whether anything changed that should not have.
- Did the test cover the work? Include ambiguous requests, missing information and other conditions the assistant will encounter.
Our practical takeaway: record what each check establishes and what remains untested. A domain expert can help define these checks and review examples; implementing automated checks or running infrastructure may require engineering skills.
Next: spot a sales report with correct numbers and faulty reasoning → · Build your first evaluation rubric →
Practical evaluation · 9 September 2026
Correct numbers can still tell the wrong story
A sales forecast can add up while misrepresenting why it changed. Our original four-deal exercise shows how to check the figures, explanation, uncertainty and recommended actions—and why human correction time belongs in a cost comparison.
Evaluate an AI sales report without coding →
Prompted by Luke Deasy's September 8 Tenex article. The guide explains its source limitations; our fictional exercise is not a reproduction of the study.
Career explainer · 9 September 2026
Can you contribute to evals without coding?
Some opportunities rely on domain judgment, clear instructions and operational skills. Our new guide uses official employer examples to explain the requirements and distinguish employment from vendor projects.
Explore the routes → · Try an evaluation exercise
Hiring selection · 9 September 2026
Five ways into evaluation and safety
Our first hiring edition explains the teams, work, locations and advertised pay behind five selected vacancies.
Read the edition →
Company announcement · 3 September 2026
ai& and Tenstorrent announce JapanFold
The companies announced a platform serving open-source drug-discovery models on infrastructure in Japan.
Why it matters for this audience: specialist applications raise questions about which tasks and domain-specific outcomes a model should be tested on. That is our editorial interpretation; the announcement does not establish new evaluation vacancies.
Read the original announcement · Current ai& roles on our board
The company's post-training vacancy is a general role; we have not verified a connection to JapanFold.
Foundational reading · 9 January 2026
How teams evaluate AI agents
Anthropic's engineering guide explains agent evaluations, grading and the practical challenges of testing systems that take multiple steps.
Read the engineering guide · Start with our plain-English introduction · Explore evaluation roles
The reading desk
A starting library for understanding the work. These are ongoing reference resources, not announcements of new releases. Documentation can change.
Methods and failure analysis
How tasks, trials, graders and environments fit together—and why a score needs context.
Anthropic: evaluating agents
Evaluation workflows
Understand evaluation datasets and the distinction between development testing and production evaluation.
LangSmith: evaluation concepts
Agent environments
Explore the components of a task, including instructions, an environment and verification.
Harbor: task documentation
Traces and debugging
Use recorded actions to investigate where a system went wrong.
LangSmith: observability
How we cover the field
We cover useful developments even when no company is hiring. Each editorial item should link to its original source, show its date, explain the practical significance and distinguish reported claims from our interpretation. Benchmark results need their task and testing conditions; a higher score alone does not prove a better system for every use.
Tool inclusion is for learning, not an endorsement. Company funding is not revenue or proof of job security. Vacancies are linked to specific announcements only when that connection is verified. Corrections and material updates receive an editorial date.
Learn the fundamentals · Explore companies · Who's hiring this week