Article
My AI Interview: How I Would Rebuild the AI Interviewer
I build AI tools and learning systems. This time, I was the person being assessed.
An AI interviewer assessed my learning-design skills. I ended up giving it repeated feedback on how to conduct the conversation.
THE SETUP
I applied for a learning designer job. Took some time to submit all the material. Then I was told I would get an AI interview. Sounded great. Wasn’t.
Joined with a cup of coffee. The AI interviewer started with legal and technical disclaimer. That’s the kind of chemistry you get from an FBI interview.
Anyway.
It asked how my day was going. Alright. I asked about its day. It was doing well too.
So far, an ordinary conversation. Except for the FBI chemistry bit.
Then… it jumped straight into the interview, and handed me an exercise: design training for employees using AI-assisted writing to answer customer complaints. Sure thing, boss.
THE EXPERIENCE
The first technical question came immediately. It had the decency to ask me about my background. I don’t think it factored in the following conversation. Without further ado, it asked me:
“How would you write three measurable Bloom-style learning objectives for a 22-hour adult training on using AI-assisted writing at work?
…
💭 What? 💭
Don’t get me wrong. I know what Bloom’s Taxonomy is. A learning model shaped as a pyramid, stacking learning objectives by what they achieve (remember, understand, apply, etc.).
But knowing what it is, using it to create a course, and straight up conjuring it out of the void to answer that first question are quite different things.
At that point, I started feeling like a young Spock learning math and morals in Vulcan school.
I asked the AI interviewer to repeat, slower. It did. I needed time to ingest the ask. I fumbled my way through the answer, and managed to deliver a decent answer.
Then followed a bunch of questions of the same caliber. Spoken at Youtube speed x2. Again.
I asked it to speak slower. Again. It said it could. It didn’t, though.
I tried to get back to the substance. At some point, I explained an assessment strategy.
The AI interviewer acknowledged the approach. Then hit me with a Mortal Kombat-worthy combo question:
“How would you deliver that exercise remotely, keep it accessible, distinguish the employee’s decisions from the writing assistant’s suggestions, capture evidence for another assessor, and decide what counts as a pass?”
5 questions in one sentence. Delivered at warp speed. This time I begged it to speak slower.
“Of course. What specific pace works best for you? Should I use shorter sentences or pause between ideas?”
Both, I said.
To which it literally answered:
Understood. When delivering that total phase assessment remotely, how would you ensure test integrity? Pause.”
“Pause”. It did not pause. It said pause.
At least the pauses were clearly labelled.
“Pause” instead of actually pausing appeared again in later replies. The questions still came quickly and bundled several demands together. When I’d ask for clarification, the IA would sometimes narrow the task. With a catch. I would then answer the narrowed task… but be evaluated on the initial ask. Think of it as the AI asking me 5 questions, narrowing it down to 2 upon request for clarification… then silently judging me for not answering the 5 (even though it did not follow up on the remaining 3.
That’s a lot of math overhead for a job interview.
The AI was assessing my answers. I was spending attention on getting a usable question.
Near the end, I asked for a concrete situation, because its questions became increasingly abstract. Which, thankfully, it delivered: a disagreement between stakeholders.
Now I had a situation to work through. I proposed separate conversations to understand each team’s concern, followed by a joint discussion and a documented decision.
With that concrete situation, I could spend more attention on the problem itself.
But here is where it failed: it complied with my requests, albeit in a strange way (“clarify” became 2 questions instead of 1), but forgot the context of those requests, and ended up judging me on the 5 questions that it deliberately narrowed down to 2.
KEY TAKEAWAYS
1. Design your AI interviewer to keep always retrieve or keep context across turns in a single-thread conversation
The AI interviewer acknowledged my feedback repeatedly. And failed to apply it. In other words: what mattered was whether its later behavior changed. It didn’t. It got worse.
AI Design Lesson #1: Check what follows the acknowledgment.
2. A question helps produce its answer
In cognitive science, it’s called “context/order framing”: prior questions prime the frame of reference for later ones, degrading independent judgment. In AI parlance, it’s simpler: garbage-in/garbage-out. The concrete scenario brought out clearer reasoning than the broad prompt before it.
AI Design Lesson #2: Good assessment design makes competence easier to observe.
3. Confusion needs investigating
A long answer or a request for clarification can reflect difficulty with the task, the wording, or both. The AI interviewer needs to investigate that distinction before judging. I suspect that the AI was expecting specific answers and applying some sort of boolean logics:
- AI: Asks Question A.
- Me: Asks Clarification.
- AI (under the hood): Expects answer that answers the initial parameters of the question. Flags “true” (“pass”) if answered in those terms. Anything else is flagged “false” (“fail” or partially wrong).
AI Design Lesson #3: Build your AI to parse clarification questions as non-relevant to “did he answer correctly” and only grade the actual answer. But that brings us back to Lesson #1: the AI needs to carry context across turns, but also not to be binary.
WHAT AI BUILDERS CAN DO
- Keep adjustments in effect. A pacing request should survive more than one reply.
- Explain changes to the task. Make narrowing explicit.
- Probe the gap. Ask a useful follow-up when an answer is incomplete or unclear.
MY PERSPECTIVE
I did not pass the screening. Some of my answers could have been more direct, sure; but the failure alone does not establish why I failed, beyond my speculations above.
The AI interviewer helped create the performance it assessed. I would want a human reviewer to examine the whole exchange, including the AI interviewer’s contribution.
A DESIGNER’S & DEVELOPER’S NOTE
My work is about making humans and machines intelligible to each other.
This experience clearly shows a failed case (and I do acknowledge that being rejected might paint my opinion on the matter. Yet, my points still stand on their own).
Two golden rules I abide with:
- “When people have trouble with things… it’s not your fault. Don’t blame yourself: blame the designer.” — Don Norman, The Design of Everyday Things
- The manager blames the employee; the employee blames the manager. The teacher blames the student; the student blames the teacher… We rarely think to blame the structure of the relationship itself.” — Greg Mckeown, Effortless
When someone struggles with an exercise I designed, I examine my instructions as well as their answer. And I redesign. An AI-led assessment deserves that scrutiny too.
Lorenzo Colombani designs and builds AI products, learning systems and interactive web experiences — lorenzocolombani.com.