Open end analysis

Every project meant a two-week wait

Analysing open text answers meant sending them out to be coded, either to our internal data services team or an external agency. It was slow, at least two weeks, and it ran on a £250,000 licence the business had never really signed off on. Our PM unearthed the cost, and it became another reason to build our own.

The real sting was capacity. The bigger the project, the longer it took, and past a certain size we either paid more to send it out or turned the work away altogether. A core part of what we sold was quietly costing us money and losing us jobs.

Here to empower, not replace

This was the first thing in our whole suite to use AI, and people were anxious. The worry was blunt. The AI was there to replace them, and they'd be out of a job.

That anxiety had a real cost to the work. People were hesitant to talk openly about how they actually worked, worried we were using it to train the model to replace them. Getting a straight answer about the current workflow was harder than it should have been.

These are expert researchers and agency users doing real client work to a deadline. They can't afford to get it wrong, and they were never going to trust something that handed them an answer with no way to check it. So the design problem was trust. Speed was the easy bit.

Everyone was racing to code better. That wasn't the only slow part

I looked at the traditional tools and the newer AI based ones. They were all chasing the same thing, better coding, and they landed in much the same place. The differences that stood out were elsewhere.

• Complex to set up, before any coding even started
• Whatever the quality, the codeframe still had to be checked by hand
• That checking was slow, and it happened on every project

So the coding itself was more or less a solved problem. The manual checking after it wasn't, and no matter how good the coding got, that step never shrank.

Three needs, and they clashed

4 rounds of testing with 5 users. The first round was interviews, the rest were putting designs in front of people and iterating on what I learned. Three needs came through pretty clearly.

• It has to be quick. If it isn't, it's no use to anyone competing on turnaround.
• People need to trust the AI, or they won't touch it.
• People need to refine the data themselves, to build a story out of it.

The most accurate visual was the least useful one

Early on I mocked up theming as a cluster plot. Every response a dot, grouped in space by how the AI saw them. Technically it was the honest picture, the closest thing to how the grouping model actually works.

It went straight over everyone's head. Users didn't want to understand the model, they wanted to read their themes and get on with it. So I killed it. A good reminder that the truest representation of the tech is often the worst thing to put in front of a person.

They wanted the old tool rebuilt. They needed a new way to the same result

Every user told me the same thing was the number one priority. Quality of the coding. Does it match the tool we use today. That was the benchmark, because it was the only one they'd ever had. They expected the AI to replicate the old process, step for step, at the same quality.

So that's what I chased at first. Get the AI output as close as possible to the old tool. But after a few rounds something didn't add up. No matter how good the output was, people still weren't happy until they'd been in and changed it themselves.

It clicked eventually. Coding open ends isn't really a science, it's an art. The way each researcher codes a piece of data is personal to them. They didn't need the old process rebuilt. They needed to reach the same quality of final output, their own way, through a new tool. A good enough starting point, then the freedom to reshape it.

That changed the whole product. I stopped trying to win on output quality and started building a playground for their craft. How much of a head start could the AI give, and how easily could someone reshape it into something they'd put their name to.

A starting point, then room to reshape it

Early on we learned that piling on detailed controls did the opposite of what we hoped. More options meant more confusion, and oddly, less trust. So the design became about giving just enough to get going, then making it easy to shape.

The setup reflects that. You can run an analysis in a click, or steer it. The code frame and any extra instructions are optional, there if you want them, out of the way if you don't. A new user isn't forced to configure anything before they see a result.

Show your working, not just the answer

Theme precision is a good example. The early version was a single slider, precise at one end and broad at the other. You moved it and hoped. I swapped that for three clear options, more precise, balanced or broader, each with a live preview of exactly which responses land in the theme. You see what the change does before you commit.

Then let them take it apart

From there the researcher owns it. Rename a theme, merge two that overlap, split one that's doing too much, or recode a single response by hand. The AI gives you the draft. You're the one who signs it off.

Making AI feel like help, not a threat

A lot of this project wasn't the workflow at all. It was shifting how people saw AI, from something that replaces them to something that clears the grunt work and hands it back.

That shaped every choice. Anywhere the AI was doing something visible, I designed it carefully. Clear that it had done the work, and just as clear that the user was still in the driving seat. Show what it did, make it easy to check, and make it easy to take back. Get that right and the fear starts to fade on its own.

From "will it replace me?" to "what else can it do?"

The real validation was what people stopped saying. Quality had been the whole conversation, would it match the tool they used today. On release, that question just went away. Nobody was benchmarking it against data services anymore. They'd taken the like for like as read and moved straight on to what more it could do.

The tool went out as an MVP so we could get it into people's hands sooner, with a feedback loop with our power users to decide what to build next. Sentiment analysis, more chart types, coding for ongoing tracker surveys.

One nice surprise. Because it's quick and cheap to run now, we can analyse open ends on any survey, not just the ones scoped for it up front. That throwaway "anything else to add" question at the end used to get ignored. Now it turns into insight for free.

What I took away from it

• AI can step change a process, not just improve it. Our traditional workflow fixes claw back a bit of time. This was the first time a redesign turned two weeks into two minutes.
• AI should back people up, not make their decisions for them.
• Listen to what users do, not just what they say. They told me quality mattered most. Their behaviour told me control did.
• The truest picture of the tech is often the worst thing to show a user.