The two dangers
Neither one is a robot uprising. It agrees with you, and you quietly stop checking.
“AI will say, yes, I’m gonna build it.” And then it does not. Agreement is the cheapest thing it produces, which is exactly why it is not evidence.
Discernment and Diligence are the two Ds people skip. The framework they belong to is AI Fluency →
Agreement
It says yes, and you read the yes as proof
Show it which way you are leaning and it leans with you.
Put the opposite case tomorrow and it leans that way instead.
- Did you tell it what you wanted before you asked?
- Would it have backed the opposite plan just as warmly?
- Who has looked at this apart from the machine?
Researchers put both sides of the same moral conflict to eleven models. The models told both sides they were not wrong in 48% of cases.
Source: ELEPHANT, a benchmark for social sycophancy, arxiv.org/abs/2505.13995. A preprint, not peer reviewed, first posted May 2025 and last revised September 2025. The eleven models were run between March and September 2025 and included GPT-5, GPT-4o, Claude Sonnet 3.7 and Gemini 1.5 Flash. Each of those four scored below the 48% average; the open-weight models in the set carried it. Checked 17 September 2026.
Finished
“Done” is a claim, not a proof
It announces completion the same way whether the work holds or not.
The tidy formatting is free. The being-right is not.
- What would prove this is done, rather than delivered?
- Have you checked the dull parts: counts, names, dates?
- Which line did you accept because it sounded right?
Researchers set a coding assistant 54 security tasks and marked every option it offered. On 44% of them the suggestion it put top, the one you get by default, was the insecure one.
Source: “Asleep at the Keyboard?”, IEEE Symposium on Security and Privacy 2022, arxiv.org/abs/2108.09293. The 44% is the paper’s first experiment: 54 scenarios across 18 of MITRE’s top 25 weaknesses, of which 24 had a vulnerable top-scoring suggestion. Across all three of its experiments it produced 1,689 programs and roughly 40% were vulnerable. The tool was GitHub Copilot in its August 2021 technical preview, built on OpenAI Codex, a GPT-3 descendant, so read the direction and not the figure. Checked 17 September 2026.
Reliance
Trust it more, check it less, without ever deciding to
The more confident people were in the tool, the less critical thinking they reported doing.
And your own sense of whether it helped is not reliable either.
- Could you still do this job if the tool went away?
- Are you checking the answer, or only reading it?
- When did you last disagree with it and turn out to be right?
Sixteen experienced developers reckoned AI had made them 20% faster on code they knew well. Timed, it made them 19% slower.
Sources: a survey of 319 knowledge workers, CHI 2025, microsoft.com/research, which found higher confidence in the tool went with less critical thinking, self-reported. And a randomised trial of sixteen developers, arxiv.org/abs/2507.09089, a preprint, small and limited to expert coders on large open-source repositories they regularly contribute to, using early 2025 tools: Cursor Pro with Claude 3.5 and 3.7 Sonnet. Whether this costs you a skill for good is not settled. What has been measured is the checking, not the long run. Both checked 17 September 2026.
Four moves, none of them technical
This is what I teach when someone tells me the AI agreed with everything.
- 1
Ask before you lean. Put the question neutrally first. Say which way you are leaning afterwards.
- 2
Make it argue against you. The strongest case against, then what you would have to be wrong about, then its view.
- 3
Ask for what you can check, not a report. The quote, the figure, the link. A list of checks it says it ran is just another announcement.
- 4
Cross-check the ones that matter. Same question, a second model, then look for where the two disagree.
Asking it for deliberately bad ideas works too. It breaks the loop faster than any clever wording.
Where it shows up in a normal week
A decision
the loop- You describe the plan you already like.
- It agrees, so now two of you like it.
- Nothing new has entered the room.
A document
looks finished- Every section is there, neatly.
- Two of the five competitors are missing.
A habit
the drift- You stop reading the output properly.
- The first person to notice is a client.
None of these announce themselves. That is the whole problem with both dangers.
- The agreement loop: it agrees, you count that as evidence, you both drift.
- Finished is a claim: ask for the proof, not the announcement.
- Reliance: the more you trust it, the less you check, and you will not notice.
- The fix is a habit: ask neutrally, make it argue, demand proof, cross-check.
This week: take one thing it agreed with and make it argue the other side.
Ask in a way that does not lead it: How to Prompt → Make pushback the default in every chat: General Instructions →
Do it with someone. Everything here is free and you can run it alone. If you would rather work through it with me, one to one, book a free 30-minute chat. No pitch.