Artyom Semenov

Blog

  • On synthetic users

    On synthetic users

    Can we talk about synthetic users? We, experience designers (or researchers; HCD is a big umbrella), all know they’re a bad idea… but why? And how do we convince our business colleagues without sounding like the whole field is in the “AI is gonna take our jobs” panic?

    While slighly bored during what Romesh Ranganathan on BBC2 called “crimbo limbo” I decided to play around with synthetic personas. It was either that or clearing out the garage, don’t judge. Here’re the results. Spoiler, they’re not fantastic.

    I ran two small scale experiments. One with qualitative data from interviews (thank you Timothy Price from Newcastle University for making the anonymised transcripts available for download!); the other with no data beyond some basic demographic data and assumptions.

    Experiment with data

    I created a Custom GPT with 10 interview transcripts as a base. The interviews were around parent’s knowledge of the flu and their thoughts about vaccinations. I asked it to reply as a parent in first person and always present its opinions as its own, using words like “I think” and “I feel”. I then fed it various service concepts ranging from benign (digital service to book vaccinations) to downright scary and unethical (let’s monitor a specific group of people). The persona was overjoyed to use the service, completely ignoring the fascist undertones. Not once did it say anything critical.

    On the plus side, it was very faithful to the transcripts and had extra detail to expand on if I asked a follow-up question. Its main concerns were privacy, security, and accessibility.

    Conclusion: your design a burnt toast and it’ll still hail it the best thing since sliced bread. I wouldn’t use it for anything beyond basic data retrieval from the transcripts, and even then I’d double check everything.

    Experiment with no data

    I created a similar Custom GPT with the same system prompt but it was directed to use its general training when coming up with responses. Funnily enough it didn’t make much difference. It thought that a government monitoring service for a specific ethnic group was a splendid idea.

    Its main concerns were privacy, security, and accessibility.

    Conclusion: way too generic, and oddly similar to the previous bot.

    Discussion

    It seems that no matter what data you throw into it, the responses will be generic and overly positive. So next time someone at a conference tells you that their company makes synthetic personas powered by a bazillion data points from all across the internet — don’t believe them! Those bazillion data points will not make the persona any better, they’ll just add a bazillion more points of failure, and tell you to think about privacy, security, and accessibility.

    There are perfectly valid reasons to have a digital twin system simulating the real world in real time. This is what synthetic users are, really. However, people are infinitely more complex than an electric turbine or even a space rocket. AI might make good estimation and have a bazillion data points to draw upon, but—and it’s a big ‘but’—would you trust AI with your space rocket?

    Therefore, my point of view is that synthetic users are shit. AI has a thousand more useful use cases that make the world a better place.

    Notes

  • Knowledge management in the era of AI

    Knowledge management in the era of AI

    I’ve been thinking about this over the weekend. We spent the last two years feverishly trying out approaches to make research analysis easier for ourselves. First we tried simply dropping in all our interview transcripts into the chat and let it analyse it. When that didn’t work, and honestly resulted in more hallucinations than anyone thought possible, we switched tact and tried to chunk the data. We experimented with aligning transcripts to interview questions. We even, at one point, added interview transcripts to a CSV file, pre-processed it, and created a JSON output to make them more “LLM friendly”. Every time something was missing.

    LLMs are inherently black boxes of magic. Stuff goes in, complicated statistical stuff goes on, AI slop comes out. Sure, they’re useful in their own small way to write stuff that sounds like it was written by AI, or summarise some high level stuff.

    Slight tangent. I recently read some _insights_ created by an LLM model from a bunch of transcripts. Turns out that data scientists and analysts _need_ to process data. Blew my mind.

    Back to our black boxes of magic. The point is, no one, even the developers, really knows what’s going on inside that black box of an LLM. Once the training data is in, that’s it. It goes and does stuff. And that is a problem.

    How can we trust anything the LLM produces if we can’t explain it?

    So here’s my humble contribution to the field of Explainable AI. I present to you the first epistemologically transparent cognitive dashboard:

    Made with Figma Make

    If you’ve ever seen a debating or argument building software, it’s pretty much that. I’m not reinventing the wheel here. The difference is in who the users are. I envision mini-agents building logical strings of arguments to support any claim. Think of it as a cognitive assistant to make research more traceable. It’s AI-powered in a sense that mini-AI-agents are spinning their wheel working on a single piece of evidence at a time.

    You upload your sources, the little swarm of agents go and break your sources into pieces, process them, tokenize them, and build a shared context together. Then you, the user, can interrogate the data, make claims, ask the AI to build argument graphs to support your arguments, and so on. Suddenly we’re using LLMs to help us construct knowledge and logic, rather than mindlessly processing context windows to spit out a hallucination of what the truth might be.

    As ChatGPT sycophantically praised my idea as a paradigm shift in human cognition… yeah… no… I still couldn’t help but think how neat it would be to have traceable argument chains that show how the LLM came to this or that conclusion.

    Perhaps we can use it in research repositories, perhaps in academia, policy, strategy. Perhaps we can stop the decline of critical thinking in schools by saying, “cool, you want to use AI? Great! Bring me an AI written paper with epistemological transparency. Show me how you got the answer!” Wouldn’t it be neat?

    You could dive into sources to inspect citations, contradictions, and epistemic status of a node. You could mess around with all sorts of fun meta-data that your helpful swarm of agents helpfully made for you. It’s not really NotebookLM, or Obsidian. It doesn’t just take your notes and spits out things it knows about them. It’s a structured reasoning tool that makes LLMs less of a black box of magic.

    So… I guess this is a call to action. It would be great to connect with researchers, engineers, just people who care about epistemic transparency of LLMs. Honestly, if I could build this I would use the crap out of it as a research repo and analysis tool.

    Drop me a line! We can get the MVP out in no time 😉

  • Design? Research? Problem solving.

    Design? Research? Problem solving.

    About 10 years ago I went on IDEO’s human-centred design course. It was free, it was easy to grasp, and it gave us the tools we needed to design great products. It’s still around, you can check it out here.

    At the start of the course we learned about the design process: inspiration, ideation, implementation. We learned that first you speak to people. The methods included interviews, surveys, observation, and more. Then you brainstorm some ideas, test them and iterate until you got to a great solution. Finally you implement it, while continuously gathering feedback and tweaking the final solution until it’s “perfect”.

    “Okay,” I thought, “maybe it’s pretty similar to the process I learned about in uni: Discover, Design, Test, Manufacture. Or maybe it’s similar to the process I learned in university: Research, Build, Iterate Aaaaand! It’s pretty similar to our beloved Double Diamond from the Design Council.

    What’s going on?! It’s almost as if design process is universal whether you’re designing a video game character, a magazine cover, a chair, a toothbrush, or a website or service. The point I’m trying to make is that design is design.

    Why then, is the industry so hell bent on dividing researchers from designers?

    Look at any user researcher job on LinkedIn. Here’s one:

    Bachelor’s degree in a relevant field (e.g., Human-Computer Interaction, Psychology, Data Science, Public Health). Advanced degrees are a strong plus.

    And here’s a requirement for a designer:

    Bachelor’s degree in Design, Human-Computer Interaction, Psychology, or a related field; Master’s degree preferred.

    Does that mean that I designer won’t be picked for a research role? And vise versa? That’s just stupid.

    Let me tell you a story. I went to a conference once and met some junior designers from a FAANG that I won’t name. User journey map? Haven’t heard of one. Do you run user interviews? No. What do you do then? We design the screens.

    I could only chuckle nervously and excuse myself.

    Here’s the problem with researchers who don’t design and designers who don’t research. Let’s say you’re a researcher who found some insights. You present them to the design team and they go away to do some designs but they don’t have the full context. It’s like having two halves of the brain that don’t speak to each other. I mean, we try, we communicate, but things ultimately fall through. We have these insight trackers and knowledge repositories, which are undoubtedly useful but don’t solve the core problem: having the full context.

    My best time as a researcher was when I worked with a designer who used to run research sessions himself. He grasped exactly what was needed to be done, helped with the admin, and took some ace notes. Then, I took on some screens to design and helped him in return. We worked as one unit. It was great.

    So the case I’m trying to make here is that designers should be researchers. You don’t need a PhD in Psychology to speak to some people and learn about their problems. Similarly, if you’re a researcher, I highly suggest you pick up some design skills. Ultimately, if you’re a hiring manager and you see a designer apply for a research position, think twice before rejecting them, and vice versa for researchers applying for design positions.

    The portfolio should speak for itself.

  • On empathy in customer service

    On empathy in customer service

    We did some research recently. We asked participants to recount a particularly frustrating experience with customer service. Almost every single one said, “I felt like they didn’t listen.” and something in the vein of, “I need to know that they got my back, that they understand.” Isn’t it quite profound? Don’t you just want to call customer service and have someone tell you that everything is going to be okay?

    Empathy is a very human concept. It’s important for us to know that the other person understands what we’re going through. “Hey, I’m sorry, that sounds frustrating!” and suddenly everything is just this little bit better. So why don’t we have this in customer service? A business function that’s designed to make customers’ lives better?

    Imagine my surprise then when most participants then proceeded to say that they don’t actually don’t care much that they don’t speak to someone as long as the thing they came here to do is done. I understand the sentiment. Sometimes you just want to get a thing done.

    There is nuance of course. Some things you just need to speak to a person about. Especially when it’s something complex or sensitive.

    So the point is this:

    Speaking to a human is important, but not as important as getting shit done.

    Great, Arty, thanks for this pearl of wisdom. So what?

    The so what is that I see a lot of companies make it impossible for customers to get shit done. They add pages and pages of searchable help articles that are as ignorable as they are useless. They add inefficient systems that take customer information only for the customers to have to say it all again when they get to an agent. They make agents life hell with 17 (yes, we heard this in our research too) tools to search for information.

    The so what is that businesses need to prioritise getting shit done over anything else. Name change, renewal questions, basic stuff, it can all be automated. You don’t need an offshore contact centre. You don’t need to spend millions on training. You just need to make it easy for people to achieve their goals, and if that goal is to speak to an agent, then so be it.

    How do we make it easy for people to achieve their goals? That’s the million pound question. Every business is different. Perhaps a talented (hint hint) designer can help you here. You could introduce Agentic AI, you could add a self service flow to make easy changes to the account, you could make your human agents’ lives simpler by automating parts of their workload. The solutions are many, the question is how do we find those solutions.

  • How does AI actually impact our daily lives?

    How does AI actually impact our daily lives?

    If we take away the incessant torrent of AI news, articles, panicked Reddit posts, trashy Tik Tok videos, and other slosh. What’s left? How do you actually use AI in your day to day?

    Let’s start with a proud sample of one — me!

    Last month I used AI to generate some voice overs for a marketing video and an AI avatar tool to put faces to the words. I wasn’t hiding it was AI generated though, it was part of it. I’d say it’s a pretty niche use case.

    Last week I used AI to do deep research for a Request for Proposal for a client. I ain’t spending a week reviewing 140 websites about digital strategies and competitor analysis. I trust the AI to do it for me, even if it’s not 100% perfect. It saved a ton of time and all I had was to read the insights, write some human analysis as to what they mean, and stick them into a presentation.

    This morning I used AI to give me ideas on how to make my oeufs en cocotte less runny when they come out the oven. Before that I generated some character names for my roleplaying game.

    AI seems to have slipped into my day to day pretty seamlessly. I like it. It’s like having a slightly overeager information butler at my fingertips. I don’t Google thins as often anymore, as predicted by some report somewhere. I don’t spend too much time doing desk research. Is it bad? Absolutely not. Does it impact my creativity? Also no. It’s more of a brainstorming partner. I’m making all the decisions. I don’t think I ever saw it come up with a truly original idea. You’d need a human for that.

    But hey, I wouldn’t call myself the typical user. I actually work with AI for a living, kinda. Let’s take another, more robust sample. Let’s ask some real people how they use AI and how it impacts them. Conveniently we did some interviews on just this topic back in March. Here’s what we came up with.

    People use AI to write letters, plan holidays, and generate emails

    Big insight (yes, sarcasm). Most peoples’ use cases kinda align with mine. AI has become a part of our lives in a completely expected and boring way. It removes the minutiae of having to do boring stuff.

    AI can be your lawyer, wedding planner, can help write a complaint that actually gets heard. It opens up all sorts of opportunities. Listen to this. One participant was telling us about a legal battle they had with a company.

    > When I was writing my email to the [company]. I did, like, so that it sounded legally sound. My housemate did put it through a ChatGPT type thing. So I think it’s almost like, interesting to think about the fact that it’s not only like companies who are using it, but, like, people themselves, I imagine, when they’re having complaints.

    Isn’t it great? You have a lawyer in your pocket now! It’ll get a bunch of facts wrong, and hallucinate another bunch, but boy can it write a good complaint.

    People use virtual assistants like Siri, Alexa, and Google Assistant for tasks, reminders, checking weather, and controlling home devices

    Aha! Got you! I never said we’ll be talking about generative AI, did I?

    But seriously, this finding points at an interesting difference between how we—designers, researchers, etc., working in tech—think about AI and how Mrs. Smith from down the road thinks about it. Alexa is still the pinnacle of AI technology for many people.

    Conclusion? Quite a few actually.

    Firstly, you’ll probably need to explain generative AI to your gran when it comes to her Alexa.

    Secondly, when it does, it’s going to be fantastic for older people. We already know that older folk sometimes use Alexa to alleviate loneliness. Imagine if it could talk like a human and provide some emotional support from time to time like ChatGPT does? For this generation the AI revolution won’t be about making hyper-realistic promo videos. It’ll be about a familiar voice becoming that little bit smarter. I quite look forward to this one.

    People use AI to translate and learn languages

    One participant used ChatGPT to surprise his Danish wife. His granddaughter used it to write a poem in the style of William Shakespeare for school and it gave him an idea to do something similar but in Danish. Isn’t it lovely?

    AI is opening a whole new world of languages for us. There are already demos of real-time translation (similar to what Google Translate does), but the added value of learning the language is pretty neat.

    Conclusions? There is something about making localisation easy, perhaps enriching simple translation with an element of learning is a neat idea. Ultimately though, AI is democratizing the connection and lowering the barrier to not just understanding but connecting with other across linguistic divides.

    Some conclusions

    So after sifting through the hype and looking at how we actually use it, what’s the real story here?

    AI has clearly become a part of our lives in a very subtle and non-invasive way. This “Boring Revolution” is actually kinda nice. I’m sick and tired of the AI hype online. How it’s going to take our jobs and disrupt our very way of life… and other dramatic predictions.

    Instead I’m coming to think of it as the great enabler. It’s not about sentient robots taking over or all designers losing their jobs. It’s about people gaining superpowers—like sifting through the noise and getting the information you’re actually searching for, or writing a near-perfect complaint letter, or feeling less lonely.

    This is wonderfully “boring” in a world where disinformation and AI-slosh is taking over.

    N.B.: hey, we’re all aware of the risks, but just in case you aren’t, this “boring” revolution is still very much open to bad actors. Disinformation can become quite insidious and hard to spot. Companies can learn to optimise their content to make sure they appear in AI search results. Marketing departments and bad state actors can manipulate the narrative. Imagine if ChatGPT tells your some fake news? But stop. Just stop. Let’s not think about that for now. Let us enjoy the fuzzy, cosy glow of the ‘boring’ revolution for just a minute longer.

  • A discovery in healthcare

    A discovery in healthcare

    We worked with the UK Health Security Agency to identify opportunities to improve the Measles Surveillance Pathway. Navigating internal politics and ethical considerations, I led a team to plan and conduct a Discovery. We identified £800k in estimated efficiency savings through automation and process improvement.

    I stood up a team to lead a Discovery phase into the Measles Surveillance Pathway at UK Health Security Agency (HSA). Navigating complex organisational and policy landscape our team successfully interviewed 35 Health Protection Team (HPT) members. We produced 75+ insights, 4 user archetypes, fed into a creation of the “to-be” service blueprint.

    Situation

    Measles Surveillance is a complex pathway where multiple teams must work together to produce accurate data about the measles cases in the UK. For this the Health Security Agency, responsible for all health related threats to the UK security, created the current “pathway”. It is supported by multiple regional Health Protection Teams, responsible for day to day health of their respective regions.

    When a measles case is suspected (by a doctor, or a school nurse, or another healthcare practitioner) a report is made to the local HPT. The local HPT investigates, conduct any necessary follow ups (usually in the form of an oral kit sent to the patient), and send the samples off to a lab. If a measles case is confirmed necessary interventions are made.

    Task

    Our team was brought to conduct generative research with HPTs to understand their pain points and needs, and to identify opportunities to increase efficiency and oral fluid kit (OFK) return rates.

    Approach

    We opted for in-depth interviews as the most pragmatic approach. Surveys, contextual inquiries, and focus groups were considered but rejected due to time, budget, or organisational constraints.

    There were several challenges the team had overcome while working in this regulation heavy environment with low UX-maturity.

    1. Organisational resistance

    Due to the decentralised nature of the regional HPTs it wasn’t immediately obvious who to contact about recruiting participants for our interviews; additionally there seemed to be negative sentiment from the HPTs about the governing HSA; a relatively new department that came to replace the old Public Health England.

    To avoid further delays we went to the source. We contacted HPT team leads directly and asked them to attend an introduction meeting with us. During this meeting we explained the nature of the research and what we were trying to achieve.

    While the immediate resistance was overcome, and we got our recruitment emails out to the teams, the negative sentiment remained throughout. We navigated this by making our intentions and the nature of our research clear upfront in the information sheets we produced.

    Relevant skills: managing consent, stakeholder management, relationship building, influencing, advocating for research.

    2. Stakeholder resistance

    Our first draft of the plan was met with scepticism. Stakeholders from clinical and academic backgrounds felt our epistemology was lacking and our approach was not robust enough to give the insights any confidence or weight. These stakeholders were initially not available for conversation and it took some time before we got everyone in the room together.

    During our conversation I had a chance to educated them about UX research and how it’s different. How the scope and the objective of the research differs from academic or clinical research, for example. I steered the conversation away from NHS Ethics Board Reviews and more towards collaborative workshops and solving a challenge together. This was successful in getting buy-in and unblocking our path forward.

    Relevant skills: advocating for research, stakeholder management.

    3. Data protection concerns

    No Data Protection Impact Assessment was conducted prior to the research team coming onboard. This delayed us significantly. However, I worked with the senior leadership and Quality, Risk, Mitigation (QRM) colleagues to get a Data Protection Agreement (DPA) in place.

    Relevant skills: data protection regulations.

    Result

    As a result of my advocacy and strong stakeholder management skills we conducted 35 interviews with HPT members. Participants, once they understood what we were trying to achieve, were helpful and open, providing rich insight into their daily lives, frustrations, and needs. The final report, containing 4 personas, 9 efficiency saving concept designs, and 1 “to-be” service blueprint, was well received by the client and was used to plan an Alpha phase in the future.

  • Marine Licensing Discovery

    Marine Licensing Discovery

    Situation

    A government department’s ambition was to transform its marine licensing service. The complexity of the process, ministerial oversight, and several failed attempts in previous years made this a challenging environment that required careful stakeholder management and ambiguity around recruitment.

    Task

    The team was tasked with conducting a standard Discovery to understand the current process, and identify opportunities, pain points, and behaviours. A report of our recommendations and early concepts was to be presented to the government ministers who would issue budget for the next phase of work.

    Actions

    As a small and agile research team we worked closely with Product, Design, and Delivery, to plan and execute the research in record time.

    To make sense of the departmental context I conducted several in-depth research kick-off workshops that challenges the client to think about their assumptions. These assumptions were then turned into testable hypotheses and success measures.

    We used generative AI to quickly conduct desk research and review 50+ sources for key themes and existing insight.

    We then conducted interviews with a complex network of service users and stakeholders, mapping the process as we went. We produced lean personas, process maps, and ‘as-is’ service blueprints.

    Recruitment included:

    • Stakeholders (e.g., port authorities, scientific organisations, and other government departments) that were recruited using snowball recruitment through word of mouth and existing relationships.
    • Service users (e.g., engineers and building company leaders applying for a licenses) that were recruited using a third party recruiter.

    Result

    We spoke to 10+ service users and 5 stakeholders. The final report, which included 30+ insights and recommendations, along with a “to-be” service blueprint and several concepts and hypotheses for future testing, was presented to the small project team and a clean version was produced for ministerial scrutiny.

    The department was successful at securing the funding and proceeding to a longer Alpha phase.

  • Central Government Alpha

    Central Government Alpha

    Situation

    The client was amidst a complex digital transformation programme. The ambition was to digitise and modernise by automating manual processes, creating robust data processing capabilities, and re-designing their end-to-end service.

    My responsibilities included supporting the client’s head of research and leading a team of 18 client and Deloitte researchers across 4 streams of work, supporting the wider product leadership, and ensuring the services all built to all relevant standards (GOV Service Standard, WCAG).

    The situation within the team was challenging. Specifically:

    • Immense complexity with 50+ user types identified in Discovery resulting in difficulty prioritising effort
    • Large number of user needs with significant overlap between work streams resulting in duplicated work
    • Demotivated teams with low morale, they’ve been at it for two years and the complexity was only rising
    • Unsuccessful digital transformation efforts in the past left stakeholders and users weary and distrustful
    • The participant pool was drying up with internal stakeholders showing signs of fatigue, and external participants coming back for repeat visits impacting the validity of our findings

    Task

    I supported the client’s head of research to create a unified roadmap of user research, ensure the new service is accessible, ensure high quality and reliability of research to withstand ministerial scrutiny and pass the GOV Service Assessment, and augment and upskill the existing teams with knowledge transfers, training, and mentorship.

    Actions

    Introducing OKRs

    From the initial 1:1s with the team and the subsequent full-team retro we identified 8 areas of improvement that were then synthesised into actionable goals and measurable results.

    Standardising research

    I worked with the Product and Delivery leadership to align research activities to the release roadmap across all work streams. My proposed approach was to work in dual-track Agile with the aim of ensuring both evaluative and generative research is considered.

    I also introduced a standard research checklist, report and kick-off templates, and other collateral to ensure uniformity and high quality of outputs.

    Standardising personas, introducing archetypes

    The client’s 50+ user personas were synthesised into 6 high level archetypes in an attempt to simplify and standardise the programme’s approach. We worked with the brand team to create engaging visuals for each archetype and socialised them widely within the department, even printing out posters and hanging them around the office.

    Addressing participant fatigue

    By introducing research a prioritisation matrix we identified that not every feature needs to be researched in depth. Some features with low ambiguity can be evaluated in a leaner manner reducing the overall strain on the limited participant pool. To address this we began onboarding UserZoom to empower the design team to run their own unmoderated usability tests from templates we’ve set up for them.

    The client already had an internal participant panel and we spent some time reviewing and improving it, as well as running several email interviews to understand how participants felt about being on the panel. We wrote a communications guide for research ops colleagues sending out invites to reflect some of this feedback, e.g., “we know it’s been tough but we need you!”

    Made a significant decision that we will not exclude repeat participants, but instead put a timer on how often they can participate. The timer was initially 6 months but was later changed to 2.

    Centralising insight

    I introduced a central research repository and championed the use of Dovetail. While I supported the Head of Research with internal business case and onboarding efforts, I built a research repository in Excel.

    Each insight was paired with its parent work stream, supported by specific evidence, and had a column for any actions that were made based on this insight. I also introduced monthly grooming meetings where each work stream lead updated their actions for the previous month. This approach significantly increased transparency of research, our ability to cross-reference similar data from different work streams, and our ability to track actions therefore evidencing impact.

    Introducing new methods

    I introduced new methods like:

    • The Kano model to improve prioritisation
    • UX-Lite to measure impact of design
    • Rainbow analysis to make analysis fast, fun, and visual

    Improving Culture

    I ran one off-site for the team where we did team building exercises and got to know each other. This was, according to my end of year written feedback, one of the most fun events to happen on the programme.

    I also introduced standard rituals like frequent retros, manuals of me, regular 1:1s, and ‘fun Fridays’ where we would play skribbl.io and other online games together.

    Result

    It’s hard to measure, but I think the team was much happier when I left, as evidenced by written feedback I received:

    A massive thanks to you Arty for being a great UR but an even better human being. I’d like to think I shared with you quite often the positive impact and useful things you taught me in our time working together. You have been, hands down, the most positively impactful Deloitte UR that joined the team and working with you was great fun. — Senior User Researcher

    Or this one:

    Oh my gosh, what value didn’t you add?? You came to us when we desperately needed an injection of new life, new ideas and refreshed approaches, and you certainly brought that. I know the whole team benefited from your expertise and vigour. — Head of Research

    More objectively however:

    • The teams passed 3 Alpha service assessments, receiving “Green” across the board for research related standards
    • We achieved 80% insight to action rate as evidenced in the central repository
    • We raised UX-Lite by 16 points in 12 months (aggregated across all work streams)
  • Cheap energy club

    Cheap energy club

    I led design of an energy switching product. We conducted a comprehensive discovery to understand user needs, facilitated brainstorming workshop, designed the interface, and supervised build. The resulting product had an average SUS of 83 and accommodated 200 switches per month, with 600k active users.

    Discover

    We started with examining existing data from MoneySuperMarket’s energy switching product and conducted extensive desk research to understand what challenges users experience when switching energy. We also collaborated with marketing and data teams to understand wider market forces at play and understand the viability, feasibility, and desirability of the opportunity.

    We supplemented the desk research with a series of in-depth interviews to understand granular user needs and pain points around energy switching. We found that:

    • automated switching services do not take into account preferences beyond “Price” meaning that users feel not in control,
    • automatic switches based on price only sometimes result in poor customer experiences as energy companies chasing low price cannot compete with more sustainable competitors,
    • several participants reported being switched multiple times per year, resulting in frustration and unnecessary stress

    Based on this insight and behavioural characteristics of our sample we created several user archetypes, a log of user needs, and an “as-is” user journey map.

    Relevant skills: Discovery research, user needs, user journey mapping, user archetypes, qualitative analysis, workshop facilitation, quantitative market analysis, innovation management.

    Design

    We ran several workshops to co-design the new user journey. We used a range of methods. For example, in one instance we ran a mini-“design sprint”, which was a 4 hour workshop where we examined research findings, created ‘How Might We’ questions and sketched ideas with Crazy 8s.

    We relentlessly tested user preferences with remote unmoderated card sorting. We learned that control is a strong motivator. People feel in control when they get a say when to switch, they feel reassured when they are switched to a reputable brand, and they feel like winners when they switch to a cheaper deal.

    Relevant skills: Card sorting, HMW questions, Crazy 8s, workshop facilitation, stakeholder management, visioning, design sprint.

    Prototype

    We continued testing and iterating our prototypes. We started with simple paper prototypes (on paper) and moved through the levels of fidelity (Sketch, Axure, then Vue.js) as our understanding of the final solution developed. Most of the testing was unmoderated in UserZoom but we ran quarterly usability benchmarking sessions in person where we tracked usability metrics like time on task and our usability benchmark (System Usability Scale) to provide objective measure of design changes.

    As the cycles of testing and iterating progressed I worked closely with the Performance Analytics team to develop a measurement framework. My passion for usability helped me introduce and integrate the Google HEART framework to correctly attribute UX changes to revenue. The resulting dashboard, which I developed leveraging my skills in Python and understanding of Google Analytics, is still used to this day.

    Relevant skills: Prototyping (Azure, Sketch, HTML/CSS), usability testing, KPI creation, measurement frameworks, Google HEART, Python, performance analytics.

    Develop

    During the development phase we worked in dual-track Agile with the engineers. Designs were created, validated, and finalised one sprint ahead, following a pre-determined and prioritised roadmap in JIRA.

    The final product reached 200 switches in the first month, and had 600k unique users by the end of the quarter. We captured continuous feedback from the dashboard I created, which was connected to a contextual survey on site allowing users a quick and easy way to provide qualitative feedback.

    Relevant skills: contextual surveys, dual-track Agile.