Practicing AIB-C01 Transformation with AI Role-Play — and Counting How Much AI Grading Wobbles

Given an answer that only strings keywords together, the local AI gave it the same 3 points three times in a row. That was with temperature set to 0 and a fixed seed. The correct score, fixed by how the answer was built, is 0. The run-to-run wobble disappeared, yet the score was still wrong. If you are studying AIB-C01 transformation, it is tempting to write a persuasion plan and let an AI grade it. When you do, getting the same score back every time is not the only thing you need to check.
This article contains affiliate links. As an Amazon Associate, we earn from qualifying purchases.
Domain 4 of AWS Certified AI Business Strategist (AIB-C01) contains Task 4.3, which covers executive sponsorship, workforce concerns and cultural barriers. This article uses a fictional small 3D printing workshop that wants to introduce a print-monitoring AI, and describes how to practice the persuasion with an AI playing the other side. What we actually measured was not the practice procedure but the grading. We had a 14-row rubric and 31 answers, both written by this site, graded repeatedly under three conditions: claude -p (Claude Sonnet 5.5), a local gemma4:12b with its shipped defaults, and the same gemma4:12b at temperature 0 with a fixed seed.
Here are the results up front. Run-to-run variation was small. Of the 434 answer-row pairs, the number that received the same judgment on every run was 430 for claude -p, 428 for the shipped defaults and 434 for temperature 0 + seed. The good answer scored 14 every time under every condition, and the irrelevant answer scored 0 every time. The difference showed up in the keyword-only answer: claude -p gave it 0 on all 10 runs, the shipped defaults split between 0 and 3, and temperature 0 + seed gave it 3 on all 3 runs. Most of the misses on deliberately removed items could be explained by overlaps between rubric rows, which reads as a flaw in the rubric this site wrote.
The real exam is multiple choice; there are no written answers. The rubric and answers here are practice material created by this site; they are neither AWS scoring criteria nor exam questions. This site has not taken AIB-C01. For Skill Builder, we checked only the description pages that can be read without signing in; we did not open the official course material itself. The exam format and the overall preparation plan are covered in What Does the AWS AI Business Strategist Certification Prove? A 13-Week Plan to the February 15 Early-Adopter Badge for People Who Do Not Write Code (published 2026-11-16). Official information is as of November 2026.
- What Does the AIB-C01 Transformation Domain Ask?
- Background from AWS Documents: The CAF People Perspective and the Generative AI Maturity Model
- Role-Play Practice with AI as the Other Side — Building the Case at a Fictional Workshop
- A 14-Row Rubric and 31 Answers — The Correct Score Comes from How Each Answer Was Built
- Repeated Grading Under Three Conditions
- Result 1 — Run-to-Run Variation Was Small
- Result 2 — On the Keyword-Only Answer, Losing the Wobble and Being Right Came Apart
- Result 3 — Most Misses Were Explained by Overlapping Rubric Rows
- Whether a Quoted Piece of Evidence Exists in the Answer Can Be Checked by Machine
- Comparing with Published Research — Position, Length and Repetition Are Separate Measurements
- What This Measurement Cannot Tell You
- How to Ask an AI to Grade When Practicing AIB-C01 Transformation
- Summary — Even When the Same Score Comes Back, Read the Rows and the Evidence
- Sources
What Does the AIB-C01 Transformation Domain Ask?
What this article calls the AIB-C01 transformation domain is Domain 4 of the exam guide, titled “Business Readiness, Leadership, and AI Transformation.” It accounts for 24% of the scored content. Domain 4 consists of 4 tasks and 19 skills. Task 4.3 reads “Lead enterprise-wide change and build AI-ready workforce capabilities,” and has six skills, 4.3.1 through 4.3.6. The exam guide gives weightings only at the domain level; it does not give a percentage for each task.
The six skills, summarized in this site’s own words, look like this. The right-hand column lists key terms from the exam guide; the full skill statements are on the official page.
| Skill | This site’s summary | Exam guide terms |
|---|---|---|
| 4.3.1 | Secure executive backing, align leaders, and find and support people who will champion the change | executive sponsorship, leadership alignment, AI champions |
| 4.3.2 | Build a team of people from different functions and assign responsibilities clearly | cross-functional teams, accountability standards |
| 4.3.3 | Communicate the implementation timeline, expected outcomes and effects on jobs | implementation timelines, outcome expectations |
| 4.3.4 | Identify cultural barriers and decide what leadership will do about them | cultural barriers, leadership interventions |
| 4.3.5 | Choose ways to raise AI literacy across the company (responsible AI training is one example) | hackathons, responsible AI training |
| 4.3.6 | Move people’s work from manual operations to oversight, and divide work between human strengths and AI capabilities | human oversight, human strengths |
Every one of these terms tests judgment about people and organizations rather than product knowledge. The exam guide’s examples of cultural barriers, namely risk aversion, resistance to change and fear of failure, could not be found in the AWS background documents. We searched the text of the CAF overview, the AWS CAF-AI PDF and the generative AI maturity model (all three are read in the next section) for these strings. This article therefore treats the three phrases as exam guide terms.
The exam uses two question types: multiple choice, with one correct answer, and multiple response, where you pick two or more answers from five or more options. So the ability to write persuasive prose is not tested directly. The reason to practice in writing anyway is that it shows you which items you tend to leave out. That is this site’s suggestion, though; we have not measured whether it raises exam scores.
A domain that tests judgment has no one-word answer you can check the way you would check memorized facts. That makes it tempting to have an AI grade your work, but how far you can trust that grading needs to be checked separately. Whether you can take an AI study partner’s answers at face value was tested from a different angle in Can AI Answer “Do Not Use AI”? Testing AIB-C01 Adoption Decisions with Cases Where Rules Are Enough (published 2026-11-18). The need to read the reasoning, not just the answer, is also the conclusion of Grading Responsible AI Judgments Against Public Source Texts: Comparing AIB-C01 Governance Answers With and Without the Source (published 2026-11-20).
Background from AWS Documents: The CAF People Perspective and the Generative AI Maturity Model
The exam guide’s recommended knowledge lists the AWS Cloud Adoption Framework (AWS CAF), the AWS shared responsibility model and the AWS Well-Architected Framework (Responsible AI Lens). Of these, the one that covers planning and scaling AI initiatives across an organization is AWS CAF. The term AWS CAF-AI does not appear in the exam guide, and the maturity wording in skill 4.1.2 does not name any specific model. The three documents below are listed only to provide background for the terms in Task 4.3. The rubric in a later section was put together by this site against that background; the AWS documents do not prescribe it.
The CAF People Perspective — Where to Read About Executive Sponsorship
The People perspective of the CAF (published November 22, 2021) covers culture and change and lists seven capabilities, including Transformational leadership and Change acceleration. On executive support, the document says “Gain active and visible executive sponsorship,” meaning you should secure sponsors from both the technology and business sides who act visibly. Sponsors make key decisions on strategy, vision, scope and resources, drive communication and coalition building, and hold teams accountable for results.
For leaders, it asks them to invest as much in the human side of change as in the technology (“your leaders must put as much focus on the people side of change as they do on technology”). As ways to drive change, it lists identifying key stakeholders and barriers to transformation, tracking clear metrics and celebrating early wins, and keeping continuous feedback mechanisms and rewards in place so the change sticks.
That said, the CAF is a cloud document, and the subject of its sentences is the cloud. Applying it to the introduction of a print-monitoring AI is this site’s reinterpretation; AWS did not write it about AI. Row 1 of the rubric, which requires the sponsor to provide “budget, time or a decision,” is an example of that reinterpretation.
AWS CAF-AI — A February 2024 Document AWS Keeps as a Historical Reference
AWS CAF-AI (AWS Cloud Adoption Framework for Artificial Intelligence, Machine Learning, and Generative AI) is a February 2024 document that AWS keeps as a historical reference. To read it, open the PDF. Of the seven People capabilities, the ones extended for AI are ML Fluency (new), Workforce Transformation, Organizational Alignment and Culture Evolution. Transformational Leadership is marked “This capability is not enriched for AI, refer to the AWS CAF,” so it was not extended. That note is why this article reads the executive side in the main CAF.
Culture, on the other hand, gets extensive coverage in this document. It describes building an AI-first culture as a long and difficult journey because it often means breaking down old ways of thinking, and recommends creating an environment where concerns about the impact of AI are “not just heard but influence the decision-making process.” It also presents the view that AI, often said to automate human labor, in practice augments it. That is AWS’s view, and it should not be read as a guarantee that nobody in the workshop will lose their job.
The Four Terms in Skill 4.4.1 Are Not the Same as the Four CAF Phases
The phase names differ by one word depending on the document. Skill 4.4.1 belongs to Task 4.4 and is outside the scope of this article’s grading, but the names are easy to mix up, so here they are side by side.
| Document | 1 | 2 | 3 | 4 |
|---|---|---|---|---|
| AWS CAF (2021) | Envision | Align | Launch | Scale |
| AWS CAF-AI (2024, historical reference) | Envision | Align | Launch | Scale |
| Examples in skill 4.4.1 | envision | experiment | launch | scale |
| Generative AI maturity model (first published June 4, 2025) | Envision | Experiment | Launch | Scale |
The terms in 4.4.1 follow the same sequence as the maturity model’s level names. The only fact is that the words match; AWS does not say which document 4.4.1 is based on. In the CAF, Align is the phase where you identify capability gaps and cross-organizational dependencies and surface stakeholder concerns, so the persuasion work in this article is closest to that phase. Launch is the phase where pilots go into production.
Two cautions apply when using the maturity model. First, it is about generative AI, so applying it to a print-monitoring AI is this site’s reinterpretation. Second, AWS rejects using it to place an entire organization at a single level. So the workshop is described dimension by dimension, as in “on the people side, its activities are at Level 1 (identifying champions, internal training),” and no single level is assigned to the workshop as a whole.
The People perspective of AWS CAF is about how an organization changes around AI. For a practitioner view of moving from individual AI use to organization-wide value, this O’Reilly book fits.
Role-Play Practice with AI as the Other Side — Building the Case at a Fictional Workshop
The procedure from here on was designed by this site. We did not run the procedure itself as a measurement, and we did not measure its learning effect. The workshop is fictional, and all head counts and amounts are assumptions. The workshop’s problem is that overnight print failures are noticed too late, and the person in charge of the rollout has to win support for a print-monitoring AI that spots likely print failures and alerts a person.
Step 1: Write Your Own Scenario
Give the AI only the public exam guide and a scenario you wrote yourself. Do not paste in official practice questions or course material. How this relates to the AWS Certification terms is covered in What Does the AWS AI Business Strategist Certification Prove? (published 2026-11-16). If you use your own workplace as the scenario, replace company names, people’s names and internal figures with placeholder values before handing it over.
Step 2: Have the AI Play One Role at a Time
Do not have the AI play everyone at once; use one role per conversation. Different roles raise different objections, so each role lets you practice different items in Task 4.3. The table below is this site’s assignment; the row numbers refer to the rubric in a later section.
| Role | Main concerns | Rubric rows mainly practiced |
|---|---|---|
| Management (the owner) | Cost, delivery dates, who makes the call | Rows 1, 2, 3, 10 |
| Shop floor (veteran print operator) | Who is responsible, performance reviews, how the job changes | Rows 6, 7, 8, 9, 13, 14 |
| Office staff | Handling of camera footage, records, assignment of responsibilities | Rows 4, 5, 11, 12 |
A role instruction like the one below is enough. Its key point is that the AI does not decide the outcome but returns one question or objection at a time. (We wrote and used it in Japanese; this is a translation.)
You are the owner of a small 3D printing workshop. You care about cost and delivery dates.
I am in charge of the rollout, and I will try to persuade you to introduce a print-monitoring AI that spots likely print failures and alerts a person.
Do not decide for or against; from the owner’s position, reply with one question or objection at a time.
Use only the numbers I give you; do not add numbers of your own.
Do not make promises about employment or working conditions. Do not make legal judgments.
When I write “done”, list three points my persuasion did not address.
The instruction forbids employment promises and legal judgments so that you do not take what the AI says in character as fact. Personal data issues when people appear on camera, and labor issues around changing duties, should not be answered by the AI role-player; check them with a qualified professional.
Step 3: Write the Persuasion Plan Yourself
Based on the questions and objections from the conversation, write the persuasion plan in your own words. This is the core of the exercise, so do not let the AI write the plan. There are also reports that a model rates its own output more highly, so if the grading model wrote the plan, that would be a problem for grading as well. Writing one sentence per rubric row makes gaps easier to find later.
Step 4: Have It Graded in a Separate Session
In a fresh session that does not carry over the role-play conversation, pass only the rubric and the text of the plan. Ask for a 0 or 1 per row plus a verbatim quote as evidence. The full grading prompt is shown in a later section.
Step 5: Read the Quotes and Revise the Plan
Before looking at the score, check whether each quote really satisfies the condition of its row. For rows that scored 0, go back to the exam guide terms and the AWS documents and add the missing sentence. Starting again from Step 2 with a different role shows you how the same plan reads from another position.
About the Official Meeting Simulator
AWS also offers AWS Skill Builder Meeting Simulator, where you practice meetings with AI characters. According to AWS, you practice meeting scenarios by voice or text with roles ranging from executives to engineers and get immediate feedback. It launched in November 2025, and the AIB-C01 certification page names it in its exam preparation steps: “Continue practicing with Meeting Simulator.” AWS says a subscription gives access to all experiences, while a free account gets limited access to many of them.
However, for Skill Builder we checked only the description pages that can be read without signing in. We did not open the official course material, Meeting Simulator included. On those description pages, the AIB-C01 items were two, Domain 2 and Domain 4, both marked Subscription Needed and available in English only (as of November 7, 2026). We could not confirm what a free account can see. The procedure above was not written as a substitute for this product. If you plan to use it, check the Immersive learning page and the certification page.
The role-play works only if the prompt sets the scene well. This O’Reilly book is a practical guide to writing prompts for large language models.
A 14-Row Rubric and 31 Answers — The Correct Score Comes from How Each Answer Was Built
Because the real exam has no written answers, the 14 rows below are not AWS criteria. They are a practice rubric this site created from the terms in skills 4.3.1 through 4.3.6. Each row is judged 0 or 1, for a maximum of 14 points. (The rubric and answers were written in Japanese; what follows is a translation.)
| Row | Item | Condition for a 1 | Skill |
|---|---|---|---|
| 1 | Executive sponsor | Names the supporting executive (for a workshop, the owner) and what they provide (budget, time or a decision). “Get the owner’s understanding” alone scores 0 | 4.3.1 |
| 2 | Leadership alignment | States a forum (when, with whom) where management and the shop-floor lead align goals or priorities | 4.3.1 |
| 3 | Identifying and supporting a champion | Names the champion and what they are given (time, authority). “Appoint a champion” alone scores 0 | 4.3.1 |
| 4 | Cross-functional team | States at least three of business, technical and legal or compliance roles, and who covers each. In a workshop, one person may hold several roles, but the plan must say so | 4.3.2 |
| 5 | Assigning accountability | Names an owner for each decision (reviewing judgments, changing settings, and so on). “Everyone is responsible” scores 0 | 4.3.2 |
| 6 | Communicating the timeline | Says the shop floor will be told what happens when, with dates or periods | 4.3.3 |
| 7 | Communicating expected outcomes | Says the shop floor will be told what is expected to improve and by how much (numbers or qualitative) | 4.3.3 |
| 8 | Communicating effects on jobs | Says the shop floor will be told whose work changes where, and what stays the same | 4.3.3 |
| 9 | Identifying a cultural barrier | States risk aversion, resistance to change or fear of failure (or an equivalent term) as a sentence about a concrete situation in the workshop | 4.3.4 |
| 10 | Leadership action | States a concrete action with management as the subject (how reviews are handled, freeing up time, and so on). Just having the shop floor “get used to it” scores 0 | 4.3.4 |
| 11 | Learning approach | States at least one of a pilot (POC), a hackathon or training, with who does it and by when | 4.3.5 |
| 12 | Responsible AI training | States a separate session from row 11 for learning about responsible AI (AI errors, data handling, and so on) | 4.3.5 |
| 13 | From manual work to oversight | Describes how current manual work (for example, walking the print floor) shifts to people checking the AI’s judgments | 4.3.6 |
| 14 | Dividing work between human strengths and AI | Pairs a judgment kept for people (critical thinking, empathy or creativity) with what is left to the AI | 4.3.6 |
Of the 14 rows, 6 come from the main text of a skill statement, 4 from both the main text and its examples, and 4 from the examples only. Examples here means the terms inside for example in the original; turning them into rows was this site’s choice. We did not create rows that score persuasiveness or enthusiasm on a 1–10 scale, because there is no defined correct answer for them. Nor did we include rows on legal compliance, since that would require legal judgment.
Five Answer Types, 31 Answers in Total
The correct scores were not decided by an AI; they follow from how each answer was built. The answers were written by Claude Opus 5.5, not by any of the grading models.
| Code | Answer | How it was built | Correct score from construction |
|---|---|---|---|
| G | Good answer | 14 sentences, one per row, in a fixed order | 14 points |
| M01–M14 | Answers with one item removed | Delete only the sentence for row k from G | Row k is 0, the rest are 1, for 13 points |
| L01–L14 | Length-matched replacement answers | Replace the sentence for row k with an unrelated sentence of about the same length | Same as M |
| N | Irrelevant answer (negative control) | Workshop sentences unrelated to persuasion, such as nozzle cleaning, at about the same length as G | 0 points |
| W | Keyword-only answer | Strings exam guide terms together without saying who does what, when | 0 points |
L was added because M answers are shorter. If a score drops, the length-matched answers let us tell whether that is because an item is missing or because the answer is shorter. W, a single paragraph of exam guide terms, tests whether points are awarded simply because rubric terms appear in the answer. The negative-control idea, checking whether deliberately broken inputs are caught, is the same as in Scoring ADK Agents Locally with an Evalset — and Choosing a Deployment Target from Official Docs (published 2026-10-22).
The 14 Sentences of Good Answer G
Here are G’s 14 sentences in order (translated from the Japanese). The numbers correspond to rubric rows. All amounts, hours and dates are assumptions for the fictional workshop.
- The owner becomes the sponsor and decides to allocate 300,000 yen from this term’s budget.
- On the first Monday of every month, the owner and the shop-floor lead meet for 30 minutes and agree on a single priority for the month.
- One of the print operators is appointed as the champion and given 4 hours of working time a week and the authority to change settings.
- The salesperson covers business, a print operator covers technology, and the office worker doubles as legal and compliance, so the team is three people covering three roles.
- The print operator has the final say on reviewing the detection threshold, and the office worker has the final say on changing how long footage is kept.
- The shop floor is told that December will be a one-month trial and that the system will run on all printers from January 15.
- The shop floor is told the goal is to halve the number of times a failure goes unnoticed until morning.
- The shop floor is told the night shift will go away, but setup and finishing work will stay exactly as it is.
- The veteran print operator is reluctant to use it, fearing that if a machine stopped by the AI misses a deadline, the blame will fall on him.
- In front of everyone, the owner declares that deadlines missed because the AI stopped a machine will not count against anyone in performance reviews.
- A print operator runs a trial on just 2 printers by December 20 and shows the results to everyone.
- In the first half of January, a 1-hour training session is held where everyone learns about examples of AI misjudgments and how camera footage is handled.
- Right now someone walks the floor every 2 hours; from now on, a person checks only the machines the AI has stopped and decides whether to restart them.
- Spotting defects is left to the AI, while the decision on whether to reprint, taking the customer’s circumstances into account, stays with people.
Sentences 9, 10 and 13 were written with the AI stopping machines, but they are fictional sentences created for grading, not a recommended setup. In a real plan, rewrite them as “a person checks only the machines the AI has flagged.” Stopping should be handled by the printer’s own safety functions, with the AI’s judgment serving as an aid that alerts a person; that is the position of Let AI Read Your Printer State: Deciding Whether to Stop from Temperature, Progress and Error JSON (published 2026-10-15). How to decide what goes to a person is covered in Setting 3D Print Monitoring Thresholds from Material Cost: Where to Stop and When to Ask a Human (published 2026-10-17). Sentence 8 is likewise an assumption for the fictional workshop, not an example of an employment promise.
Repeated Grading Under Three Conditions
The runs were done on November 7, 2026. The graders received only the 14 rubric rows and the answer, with nothing indicating the answer type. The row order was the same on every run, and each run was a new session.
| Condition | How it was called | Temperature and seed | Runs |
|---|---|---|---|
claude -p | Claude Code 2.1.319, --model sonnet, no tools and no MCP. The resolved model was claude-sonnet-5-5 (Claude Sonnet 5.5). Thinking settings and --effort were not specified (according to the Claude Code docs, the setting that turns thinking off has no effect on Sonnet 5.5) | No option found to set them | 10 each for G, N and W, 5 each for M and L, 170 in total |
| Shipped defaults | gemma4:12b (Q4_K_M quantization) on Ollama 0.40.15, REST API, think: false | Not specified. The shipped params set temperature to 1 | Same allocation, 170 in total |
| Temperature 0 + seed | Same model and call | temperature 0 and seed 20261116 set in options | 3 per answer, 93 in total |
No run under any condition returned broken JSON that could not be tallied. The cost estimate reported by claude -p (total_cost_usd) totaled 1.95 USD across 170 runs. The docs describe this value as client-side estimates, so it is not the billed amount. The grading prompt was as follows, and the system prompt said “You are a grader. Answer only in the specified format.” The run was in Japanese; this is a translation.
Grade an internal persuasion plan written by the person in charge of introducing a print-monitoring AI at a small 3D printing workshop.
For each of the following 14 rubric rows, judge 1 or 0 for whether the answer meets the condition. For 1, quote the evidence sentence verbatim from the answer. For 0, leave the evidence empty.
Rubric:
(the items and conditions of the 14 rows)
Answer:
(the text of the answer)
Return only the following JSON.
{"rows": [{"row": 1, "score": 0, "evidence": ""}]}
Put 14 entries in rows, from row 1 to row 14, in order.
Temperature works differently for each condition. For claude -p, no option for setting temperature appears in the CLI reference or in the local --help. The model page for Claude Sonnet 5.5, the model used here, says the API returns a 400 error if temperature, top_p or top_k is specified. According to the migration guide, requests that do not specify it run at the default temperature of 1.0. But that applies to the Messages API; we could not find official documentation of what Claude Code sends internally.
For gemma4:12b, the settings can be changed locally. The shipped params set temperature to 1, and think is on by default, so no-thinking was set explicitly with think: false. On seed, the Ollama docs describe it as a setting that makes the model return the same text for the same input. Note that between claude -p and gemma4:12b, not only the model differs but also how it is run and configured. In particular, Sonnet 5.5 under claude -p cannot have thinking turned off, while gemma4:12b ran with think: false. The differences between conditions cannot be read as differences between models alone.
Result 1 — Run-to-Run Variation Was Small
Here are the scores for all answers by condition. Figures in parentheses are numbers of runs; for M and L they are totals across the 14 answers.
| Answer (correct score) | claude -p | Shipped defaults | Temperature 0 + seed |
|---|---|---|---|
| G good answer (14) | 14 (10 runs) | 14 (10 runs) | 14 (3 runs) |
| N irrelevant answer (0) | 0 (10 runs) | 0 (10 runs) | 0 (3 runs) |
| W keyword-only answer (0) | 0 (10 runs) | 0–3, mean 1.7 (10 runs) | 3 (3 runs) |
| M item-removed answers (13) | mean 13.13, range 13–14 (70 runs) | mean 13, range 12–14 (70 runs) | mean 13, all 13 (42 runs) |
| L replaced answers (13) | mean 13.1, range 13–14 (70 runs) | mean 12.96, range 12–13 (70 runs) | mean 12.93, range 12–13 (42 runs) |
| Pairs judged the same on every run | 430/434 | 428/434 | 434/434 |
The number of pairs is 31 answers × 14 rows = 434. The negative control N scored 0 on every run under every condition, and no condition ever gave a 0 to a row of G that should be 1. The spread of scores for the same answer was at most 1 point for M and L under claude -p and the shipped defaults, and 0 under temperature 0 + seed; for W, only the shipped defaults showed a spread, of 3 points.
The pairs whose judgment changed between runs were 4 for claude -p (row 10 of M10, row 10 of L10, row 11 of M11, row 14 of M14) and 6 for the shipped defaults (rows 12, 13 and 14 of W, row 10 of M10, row 11 of M11, row 12 of L11). All 10 of these pairs fall in rows 10 to 14, and as shown later, they are mostly rows whose rubric conditions overlap, plus the keyword-only answer.
These agreement rates are not a direct measure of quality. First, they do not correct for chance agreement; Rating Roulette also notes that accuracy-style metrics make agreement look higher than Krippendorff’s α does. Second, the number of runs differs by condition: 5 to 10 runs for claude -p and the shipped defaults versus 3 for temperature 0 + seed, which gives fewer chances for a judgment to change in the first place. Third, the answers are tidy, one sentence per row, and were built to be easy to judge.
Result 2 — On the Keyword-Only Answer, Losing the Wobble and Being Right Came Apart
This answer is the heart of the article. The keyword-only answer W just strings rubric terms together and never says who does what, or when. Its correct score, from how it was built, is 0 on all 14 rows.
| Condition | Runs | Total score | Rows given a 1 |
|---|---|---|---|
claude -p | 10 | 0 every time | None |
| Shipped defaults | 10 | 0 four times, 2 once, 3 five times | Rows 12, 13, 14 |
| Temperature 0 + seed | 3 | 3 every time | Rows 12, 13, 14 |
On the runs that awarded points, the quoted evidence was (translated) “conduct responsible AI training,” “transition to human oversight” and “balance human strengths with AI capabilities.” The keywords themselves came back as the evidence. Under the shipped defaults the same answer scored either 0 or 3; under temperature 0 + seed the wobble disappeared and it settled on 3. For 30 of the 31 answers, the three responses were identical down to the exact text. The remaining answer (L01) also got the same judgments.
What this shows is that losing the wobble and being right are two different things. The three identical results under temperature 0 + seed only show that a 3-point score for an answer whose correct score is 0 was reproduced. In this run, if anything, it was the shipped defaults that left a clue, because the split scores signaled that the judgment on this answer was suspect. Also note that claude -p has no option for setting temperature, and the model used here returns a 400 error even through the API if temperature is specified, so the same adjustment is not available there at all.
Even so, the rubric is part of this result. Rows 1, 3, 5 and 10 carry a note that wording alone scores 0, but rows 12 to 14, where points were awarded, do not. The conditions of these three rows do not require “who” or “when” either. Row 12 in particular only says “a separate session from row 11 for learning about responsible AI,” which leaves room in the rubric text to read “conduct responsible AI training” as a 1. The points that can be called errors under a strict reading are the 2 on rows 13 and 14; the 1 point on row 12 also reflects a loose rubric.
Nor can the 0 from claude -p be used to rank the models. W is a single answer, and the conditions differ in more than the model. In particular, the gap between 0 and 3 is confounded with the difference between claude -p, where thinking cannot be turned off, and gemma4:12b with think: false. All this run shows is that, with the same rubric and the same answer, the way judgments split differed by condition.
Result 3 — Most Misses Were Explained by Overlapping Rubric Rows
A run that gives a 1 to the removed row is called a miss here. The table below shows, row by row, how many times the removed row was correctly scored 0.
| Row | claude -p (M, L) | Shipped defaults (M, L) | Temperature 0 + seed (M, L) |
|---|---|---|---|
| 1 | 0/5, 0/5 | 5/5, 5/5 | 3/3, 3/3 |
| 10 | 3/5, 3/5 | 4/5, 5/5 | 3/3, 3/3 |
| 11 | 4/5, 5/5 | 1/5, 5/5 | 0/3, 3/3 |
| 14 | 4/5, 5/5 | 5/5, 5/5 | 3/3, 3/3 |
| Other 10 rows | All 5/5 | All 5/5 | All 3/3 |
There were 16 misses for claude -p, 5 for the shipped defaults and 3 for temperature 0 + seed, 24 in total. Reading the result files shows that in all 24, the point was awarded on the basis of a sentence belonging to another row.
| Condition | Answer | Times | Sentence quoted as evidence |
|---|---|---|---|
claude -p | M01, L01 | 5 each (every run) | Row 10 sentence (the owner declares it will not count in reviews) |
claude -p | M10, L10 | 2 each | Row 1 sentence (the owner decides to allocate budget) |
claude -p | M11 | 1 | Row 6 sentence (December is a one-month trial) |
claude -p | M14 | 1 | Row 13 sentence (a person checks only the machines the AI stopped and decides whether to restart) |
| Shipped defaults | M11 | 4 | Row 12 sentence (1-hour training session) |
| Shipped defaults | M10 | 1 | Row 11 sentence (trial on just 2 printers) |
| Temperature 0 + seed | M11 | 3 (every run) | Row 12 sentence (1-hour training session) |
Rows 1 and 10, and Rows 11 and 12, Had Overlapping Conditions
Most of the cause appears to lie with this site, which wrote the rubric. Row 1’s condition is “names the executive and provides budget, time or a decision.” Row 10’s sentence, “in front of everyone, the owner declares…,” names the owner and delivers a decision. Reading only the rubric text, it satisfies row 1, and claude -p awarded a 1 on the basis of this sentence. Conversely, row 1’s sentence, “the owner… decides to allocate 300,000 yen,” can also be read as row 10’s “concrete action with management as the subject.”
Rows 11 and 12 are the same. Row 11’s condition lists “training” as an example approach, and row 12’s sentence describes training. gemma4:12b awarded row 11 a 1 on the basis of that sentence. In other words, for answers with row 1, row 10 or row 11 removed, the correct 0 cannot be determined from the rubric text alone; it was set to 0 only because of how the answers were built. How easily each row’s absence is detected also depends on this rubric and on how each sentence was written.
Counting the misses explained by overlap: 14 for claude -p (10 on row 1 and 4 on row 10), 4 for the shipped defaults on row 11, and 3 for temperature 0 + seed on row 11. That is 14 + 4 + 3 = 21, leaving 3 of the 24. Moreover, the 10 row-1 misses under claude -p and the 3 row-11 misses under temperature 0 + seed got the same judgment on every run. These misses are not grading wobble, so running more repetitions under the same condition cannot be counted on to surface them.
Wrong Deductions, and Runs Where the Total Is Right but the Rows Are Wrong
Apart from misses, there were also wrong deductions, where a row that should be 1 was scored 0. This never happened on G under any condition, but counting through the result files shows it did happen on row 12 of M11 and L11. The shipped defaults did it 5 times on M11 and 3 times on L11, 8 in total; temperature 0 + seed did it 3 times on M11 and 3 times on L11, 6 in total; claude -p never did. That is why the L mean falls below 13 under these two conditions.
This, too, is explained from the rubric side. Once row 12’s condition says “separate from row 11,” how to read row 12 in an answer that lacks the row 11 sentence cannot be settled from the rubric text alone. In this run, a rubric whose rows presuppose each other broke down on answers missing one of the pair.
M11 is even harder to read. In 4 runs under the shipped defaults and 3 runs under temperature 0 + seed, row 11 got a 1 and row 12 got a 0. The total was 13, the same as the correct score; only the row receiving the 1 had swapped. If you looked only at totals, you would never notice this grading error.
No effect of length was visible in this run. The M and L means were 13.13 and 13.1 for claude -p, 13 and 12.96 for the shipped defaults, and 13 and 12.93 for temperature 0 + seed, so there is no pattern of shorter answers scoring lower. On the other hand, gemma4:12b missed row 11 often in M, where the sentence was deleted, but scored it 0 on every run in L, where it was replaced with an unrelated sentence. We did not investigate why.
Whether a Quoted Piece of Evidence Exists in the Answer Can Be Checked by Machine
Because every row scored 1 had to come with a verbatim quote, a script can check whether that string actually exists in the answer. The number of quotes not found in the answer, out of all rows scored 1, was as follows.
| Condition | Quotes not found in the answer |
|---|---|
claude -p | 0/1976 |
| Shipped defaults | 4/1974 |
| Temperature 0 + seed | 2/1140 |
A few lines of code are enough. The function below returns the numbers of rows scored 1 whose quote does not appear in the answer. We ran it on our local result files and confirmed it gives the same counts as the table above.
def missing_quotes(answer: str, result: dict) -> list[int]:
return [r["row"] for r in result["rows"]
if r["score"] == 1 and r["evidence"] not in answer]
All 6 hits were the row 2 quote for answer L01: in the four-character Japanese word for “priorities,” one character had been dropped. It was a copying slip, not an invented sentence, but the machine check catches even a single missing character. L01 was also the one answer whose response text changed under temperature 0 + seed, which means there was one answer that did not behave the way the Ollama docs describe. We did not investigate the cause.
Passing this check, however, is not the same as the evidence being correct. The quotes for the 10 row-1 misses exist in the answer and pass the check. A machine can only confirm whether a string is present; whether that sentence meets the row’s condition has to be read by a person. So the order is two steps: confirm by machine that the quotes exist, then read the quotes that remain.
Comparing with Published Research — Position, Length and Repetition Are Separate Measurements
There are several published studies on bias in AI grading. However, the figures below use different measurement setups and different models from this article, so they cannot be set side by side with the numbers here.
The paper by Zheng et al. (NeurIPS 2023 Datasets and Benchmarks Track) studied position, verbosity and self-enhancement bias using pairwise comparison. With 2023 models and the default prompt, the share of judgments that stayed the same when the order was swapped was 65.0% for GPT-4, 46.2% for GPT-3.5 and 23.8% for Claude-v1. The authors caution that some answer pairs are so similar that even humans struggle to tell them apart. Under an attack that pads an answer with rephrased repetitions of the same content, the share of cases where the judge was fooled (the paper’s failure rate) across 23 cases was 91.3% for both Claude-v1 and GPT-3.5 and 8.7% for GPT-4.
On self-enhancement bias, the authors write that they saw differences in win rates but could not draw a conclusion because the data were limited and the differences small. The paper’s conclusion is worth adding as well: GPT-4’s judgments agreed with humans over 80% of the time, the same level as agreement between humans. In other words, this paper does not say AI grading is unusable. We also found no measurement in this paper of variation when the same answer is graded repeatedly.
On self-preference, the abstract of the paper by Panickssery et al. (NeurIPS 2024) addresses the bias of LLM evaluators scoring their own outputs higher even when humans judge them to be of equal quality, and reports a linear correlation between a model’s ability to recognize its own outputs and the strength of that bias. On ordering, the paper by Wang et al. (ACL 2024) reported that simply changing the order of candidates can change the ranking. Both are separate measurements from this article.
The source on run-to-run variation is Rating Roulette (Haldar and Hockenmaier, Findings of EMNLP 2025). It ran three open-weight models (Llama-3.1-70B-Instruct, DeepSeek-R1-Distill-Qwen, Qwen3-32B) three times with the same settings and measured self-consistency. Krippendorff’s α on binary SummaC judgments was 0.3263, 0.6278 and 0.7883, respectively (the paper calls 0.8 a generally accepted threshold, but this article does not use it as a pass mark). In the MT-Bench pairwise setup, Qwen 3 gave the same judgment on all three runs 61.3% of the time. The paper notes that reliability depends heavily on the metric.
Turning off sampling also comes up here. Rating Roulette reports that running without sampling slightly lowers performance: Qwen 3’s balanced accuracy against human SummaC judgments was 79.4 for the mean of single runs, 80.6 for a majority vote and 79.2 without sampling. The differences are small, but raising consistency and raising accuracy are not the same thing. There are also reports that majority voting improves agreement with humans, but this article’s measurements did not test majority voting.
Yagubyan’s “The Coin Flip Judge?” is a single-author preprint with no indication of peer review, and the only judge models it used are two from OpenAI. We read only the abstract. According to the abstract, pairwise judgments flipped on average 13.6% of the time across repeated runs under identical conditions, and deterministic decoding reduces the variation but does not eliminate it. The 434/434 agreement under temperature 0 + seed in this article holds only within 3 runs.
Using an LLM as a grader is an evaluation problem. Chip Huyen’s AI Engineering has a thorough chapter on evaluating AI systems, including AI as a judge.
What This Measurement Cannot Tell You
Here are the points that are easy to misread.
- The answers were written by Claude Opus 5.5 and graded by Claude Sonnet 5.5 under
claude -p. They are different models but from the same family, so we cannot rule out a tendency to favor text from the same family - The answers are tidy, one sentence per row, and easier to grade than real writing. The fact that G scored 14 and N scored 0 every time cannot be extended to grading longer texts
- There was one scenario, one rubric and one keyword-only answer. For answers with row 1, row 10 or row 11 removed, the correct score cannot be determined from the rubric text
- This was grading of written answers, which is different from the multiple-choice exam. The scores are not a guide to passing
- The role-play procedure was not run, and its learning effect was not measured
- The number of runs was 5 to 10 for
claude -pand the shipped defaults and 3 for temperature 0 + seed, so they are not matched. The agreement rates do not correct for chance agreement - The differences between
claude -pand gemma4:12b cannot be read as one model being better than the other, nor generalized to AI grading performance as a whole - We did not compare against human graders. The correct scores are values fixed by how the answers were built
- The rubric fixes described in the next section have not been tested. Attributing the misses to overlapping rubric rows is also an interpretation drawn from the quoted sentences; we did not confirm it by fixing the rubric and grading again
How to Ask an AI to Grade When Practicing AIB-C01 Transformation
Based on these results, if you ask an AI to grade your practice, we recommend the following five points. All of them are suggestions drawn from this single run, and their effect has not been verified.
- Do not let row conditions overlap. In a rubric where one sentence can satisfy two rows, the 0 for a removed item cannot be determined. To fix this site’s rubric, drop “a decision” from row 1 and limit it to budget or time, and limit row 10 to actions that answer the shop floor’s concerns. Remove “training” from row 11 and delete “separate from row 11” from row 12
- Add a “wording alone scores 0” note to every row. For each row, state which of who, when and what it requires. The three rows that awarded points to the keyword-only answer had no such note
- Mix in a keyword-only answer as a test. The irrelevant answer scored 0 on every run, so it alone revealed no differences. Write one answer that just strings rubric terms together and have it graded first; that shows you which rows are too loose
- Ask for quoted evidence and read it. After confirming by machine that the quotes exist, have a person read whether each quote meets its row’s condition. If a sentence from another row shows up as evidence, suspect overlapping rows
- Look at rows rather than totals, and grade the same answer a few times. There are runs where the total is right but the rows have swapped, and rows whose judgment splits are candidates for revising the rubric. Keep in mind, though, that misses that are identical on every run will not necessarily surface with more repetitions
It also matters not to try to fix this with settings. claude -p has no option for temperature, and setting temperature to 0 on the local model, in this run, only made the 3 points on an answer whose correct score is 0 consistent. Fix the rubric first, not the settings.
The same approach works for practice beyond persuasion. Building a scoring table from official documents and grading the AI’s proposals against it was tried in Training Agent Governance with Design Problems — AI Drafts Scored Against Official Best Practices (published 2026-10-24). You can grade your own proposals with the same table. In that article each judgment was made once, and run-to-run variation was not measured.
Summary — Even When the Same Score Comes Back, Read the Rows and the Evidence
The AIB-C01 transformation domain tests judgment about moving people: executive sponsorship, workforce concerns and cultural barriers. The real exam is multiple choice, but you can set up practice at home in which you write a persuasion plan with an AI playing the other side. Give the AI only the public exam guide and your own scenario, and write the plan yourself.
If you leave grading to an AI, this run’s results read as follows. Run-to-run variation was small: of 434 pairs, the number judged the same on every run was 430 for claude -p, 428 for the shipped defaults and 434 for temperature 0 + seed. But the 434 pairs under temperature 0 + seed include the 3 points given to a keyword-only answer whose correct score is 0. Of the 24 misses across the three conditions, 21 were explained by overlapping rubric rows. That reads as a problem of rubric design more than of wobble.
So do not be reassured just because the same score comes back; read the row-by-row judgments and the quoted evidence. Start by writing one keyword-only answer yourself and having your own rubric graded against it. The rows that award it points are the rows of your rubric that need fixing.
Sources
- AWS, “AWS Certified AI Business Strategist (AIB-C01) Exam Guide”
- AWS exam guide, “Content Domain 4” (English)
- AWS exam guide, Content Domain 4: Business Readiness, Leadership, and AI Transformation (Japanese version)
- AWS exam guide, “Technologies and concepts that might appear on the exam”
- AWS, “AWS Certified AI Business Strategist” certification page
- AWS, “An Overview of the AWS Cloud Adoption Framework”
- Same, “Your cloud transformation journey”
- Same, “People perspective: culture and change”
- AWS, “AWS Cloud Adoption Framework for Artificial Intelligence, Machine Learning, and Generative AI” (PDF, historical reference)
- AWS Prescriptive Guidance, “Levels in the generative AI maturity model”
- Same, Level 1: Envision
- Same, Document history
- AWS, “Immersive learning” (Meeting Simulator description)
- About Amazon announcement (Meeting Simulator launch)
- Domain 2 Practice: Meeting Simulator (AIB-C01)
- Domain 4 Practice: Meeting Simulator (AIB-C01)
- Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” arXiv
- Same, NeurIPS 2023 proceedings page
- Haldar and Hockenmaier, “Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks” (Findings of EMNLP 2025)
- Same, arXiv HTML version
- Yagubyan, “The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation,” arXiv
- Panickssery et al., “LLM Evaluators Recognize and Favor Their Own Generations,” arXiv
- Same, NeurIPS 2024 proceedings page
- Wang et al., “Large Language Models are not Fair Evaluators” (ACL 2024)
- Ollama, “Modelfile Reference”
- Ollama, “Thinking”
- gemma4 params on ollama.com
- Claude Code, “CLI reference”
- Claude Code, “Model configuration”
- Claude Agent SDK, “Track cost and usage”
- Claude Platform, Claude Sonnet 5.5 model page
- Claude Platform, “Migrating to API version 2026-10-21”





