Can we talk about synthetic users? We, experience designers (or researchers; HCD is a big umbrella), all know they’re a bad idea… but why? And how do we convince our business colleagues without sounding like the whole field is in the “AI is gonna take our jobs” panic?
While slighly bored during what Romesh Ranganathan on BBC2 called “crimbo limbo” I decided to play around with synthetic personas. It was either that or clearing out the garage, don’t judge. Here’re the results. Spoiler, they’re not fantastic.
I ran two small scale experiments. One with qualitative data from interviews (thank you Timothy Price from Newcastle University for making the anonymised transcripts available for download!); the other with no data beyond some basic demographic data and assumptions.
Experiment with data
I created a Custom GPT with 10 interview transcripts as a base. The interviews were around parent’s knowledge of the flu and their thoughts about vaccinations. I asked it to reply as a parent in first person and always present its opinions as its own, using words like “I think” and “I feel”. I then fed it various service concepts ranging from benign (digital service to book vaccinations) to downright scary and unethical (let’s monitor a specific group of people). The persona was overjoyed to use the service, completely ignoring the fascist undertones. Not once did it say anything critical.
On the plus side, it was very faithful to the transcripts and had extra detail to expand on if I asked a follow-up question. Its main concerns were privacy, security, and accessibility.
Conclusion: your design a burnt toast and it’ll still hail it the best thing since sliced bread. I wouldn’t use it for anything beyond basic data retrieval from the transcripts, and even then I’d double check everything.
Experiment with no data
I created a similar Custom GPT with the same system prompt but it was directed to use its general training when coming up with responses. Funnily enough it didn’t make much difference. It thought that a government monitoring service for a specific ethnic group was a splendid idea.
Its main concerns were privacy, security, and accessibility.
Conclusion: way too generic, and oddly similar to the previous bot.
Discussion
It seems that no matter what data you throw into it, the responses will be generic and overly positive. So next time someone at a conference tells you that their company makes synthetic personas powered by a bazillion data points from all across the internet — don’t believe them! Those bazillion data points will not make the persona any better, they’ll just add a bazillion more points of failure, and tell you to think about privacy, security, and accessibility.
There are perfectly valid reasons to have a digital twin system simulating the real world in real time. This is what synthetic users are, really. However, people are infinitely more complex than an electric turbine or even a space rocket. AI might make good estimation and have a bazillion data points to draw upon, but—and it’s a big ‘but’—would you trust AI with your space rocket?
Therefore, my point of view is that synthetic users are shit. AI has a thousand more useful use cases that make the world a better place.
Notes
- Bot with data: https://lnkd.in/ehGK9-xU
- Bot with no data: https://lnkd.in/eNZRk9cB
- System prompt available on request (it’s long)
