Artyom Semenov

Tag: Research

  • Claude code for user research

    Claude code for user research

    Those who know me, know that I’m pretty hesitant to use AI in my work. As a researcher, I’ve always been concerned with the quality of output. On the other hand I, like most humans I know, would rather not do more than I need to. No, it’s not laziness. Yes, yes it is. Efficiency, rather.

    In any case, we’ve been experimenting with AI x UR for as long as LLMs first came on scene, and always I was “unhurried and unworried” about what they can do. I still am, but over the years I’ve noticed myself delegating _a lot_ of my work to various LLMs.

    Today I thought it was time to revisit a question I thought I’d put to bed sometime last year. Can LLMs help with research beyond drafting plans and summarising information?

    The answer I came to in the summer of 2025 was a resounding “no.” The answer I came to this morning was a resounding “maybe.” I call that progress.

    Let’s dig in.

    From chat to agent

    Most people, including me, use AI as a chat. To look up information, summarise documents, distil, educate, brainstorm.

    The next level up is using AI to build something. Build a website, build a table, a repository, an excel file.

    The final step is to tell AI what you want and to let the agent achieve the outcome in any way it sees fit.

    So I looked into setting up a research pipeline for my little Claude agent.

    First step was to set up Claude in Visual Studio Code. I’m not going to bore you with the “how to” but instead direct you to a lovely video tutorial: https://youtu.be/gHB4JFG9i3k?si=AiZz_f7kbIE-bpeQ

    The set up

    In CLAUDE.md, the brain centre of the entire enterprise, I described the project, gave some context about me, gave it rules to follow, explained the structure, and, since this is a research project, set up some initial codes.

    I then downloaded the sample interview scripts from the Newcastle University website (huge thanks to a Timothy Price, who made them available) and dumped them in a folder.

    The interviews were about flu vaccines and the participants’ (parents of young children) views on them.

    Coding the transcripts

    Once the transcripts were in I added some initial codes to CLAUDE.md and wrote a workflow that would take each transcript in turn and code it. I gave it freedom to create new codes as long as the agent checks in with me before using them. The result was impressive and detailed.

    I didn’t ask it to do this but Claude automatically added IDs to each excerpt, grouped the participants (vaccinated vs. not), and added turns to track things.

    Another thing I was impressed about is the table on the right. When multiple tags exist, it’s now easy to filter and understand where they are.

    Finally, it created sheets for tag summaries, how many times they were observed and more.

    Overall result, quite good.

    Quality control

    I was paranoid about hallucinations and yet I couldn’t reasonably check all 300+ excerpts. Instead I created a simple workflow (well, Claude did, I just asked it to add a quality control in place) that selected 30 random excerpts and made the human researcher do the following:

    1. Find the except in the transcript
    2. Tag the transcript manually,
    3. Check against an initially hidden column whether tags align,
    4. Answer a few questions (e.g., meaning kept after trimming? Y/N),
    5. And add any comments.

    The agent would not proceed without the fields filled in by the researcher.

    On initial scan (I did 10 out of 30) it looked accurate. I disagreed with Claude only once, and once I selected only one tag and it selected 2. It even added a whole new sheet to track the QA and produce a results summary.

    Thematic analysis

    In the next workflow the agent would open the coded transcripts and read each in turn. It would read each excerpt topic by topic, and then tag by tag for the cross-cutting tags. It would then create patterns, but not based on tag counts alone, and produce an Excel file for the researcher to examine.

    The researcher then will be required to read the file and fill out several columns. The agent won’t proceed without this human intervention and decision point.

    Once the researcher’s input is consider everything will be compiled into a single Excel file with the final themes.

    And yet again, I’m quite impressed with the output. The theme file has the themes themselves, their strength, supporting evidence and again, some sheets I didn’t think of. For example, this at-a-glance summary of the themes.

    Or this summary of topics by tags:

    Crucially, and at my request, it left the following columns for the researcher to complete unassisted:

    Reporting

    The final step is to compile a report. This is where things get particularly interesting. My personal view is that reports are boring. Over the years researcher tried everything to make them more engaging with varying degrees of success. We’ve added videos, gifs, “a day in life” and many more storytelling devices. To no avail. The reports get read once and filed away somewhere never to be seen again.

    Producing interactive HTML reports with Claude code, on the other hand, is quite easy and I hope will become the new standard for UR reporting.

    You can do so much with an HTML report. You can stick it into an LLM Wiki to become a knowledge base. You can put it online and share it with anyone. You can turn it into a page or print it out. The possibilities really are quite varied.

    Even better though, if you have a Figma MCP available with your corporate design system. Then every report will be consistent and professional.

    Anyway, enough philosophising. This is the result.

    https://artys.fun/round-1-report.html

    Here’s what I like about this report.

    Clicking on any evidence in the insight takes you down to the original excerpt. Traceability!

    Clicking on a theme expands it. This allows people to interact with the data.

    This gorgeous evidence explorer area with a search box, filters, and fun interactive elements.

    Discussion

    I am impressed.

    Things have moved on from the days of sticking transcripts into ChatGPT and asking it to find themes. We can now build mini-repositories of knowledge for every project we do and share them with stakeholders.

    There are obviously some limitations and assumptions. For example, I didn’t have time to do a proper quality check or spend the 2-3 hours (recommended by Claude!) on the thematic analysis.

    Why your job is safe

    This is so important it deserves its own heading. I want to make it super clear. Someone who is not an expert in their field would not be able to produce the same output.

    It was me who put in the researcher interventions. It was me who added the QA step. It was me who wrote the instruction to catch any non-anonymised data. It was me who put in countless quality controls and tweaks in the workflows that allowed the model to be so detailed and accurate… and most importantly trustworthy.

    If the old adage of “shit in shit out” is true then your job is definitely safe. To go x10 faster and better and bigger with AI, you still need the expertise, taste, and human creativity to make it happen.

    Next steps

    This is a proof of concept only but I can’t wait to find an opportunity to use it on a real project.

    If you’re curious, check out the full project on my GitHub: https://github.com/ArtySemenov/ur_agent

  • Benchmarking AI usability

    Benchmarking AI usability

    I’ve been working on something fun that I want to share with the design community. A ‘quick and dirty’ usability score for AI I’m calling the AI Experience Score (AES)—not to be confused with the Advanced Encryption Standard, the name is still WIP—inspired by System Usability Scale and all the evaluative scores that came since.

    Back in March 2024 I did a some desk research to see how people build amazing AI products. I looked at HCI papers around the topic of AI, agentic design, and developing human-machine interactions. What I found was a distinct lack of specific guidance or methods. How do we run usability testing with AI interfaces? How do we measure how well we did? No one had the answer. So I set out to find a way to objectively measure how well an AI solution is performing.

    Eventually I came across the original System Usability Scale paper and thought, “hey, I can follow the same process and make something useful!”.

    With the help of many colleagues, I put together a basic questionnaire based on the SUS and UX-Lite—yes, Jeff Sauro is a bit of a hero of mine—and tested it across two rounds of user testing. I tweaked the wording of the questions in between to align with the 5 key principles of Human-Centred AI:

    •     Usefulness
    •     Ease of Use
    •     Trustworthiness
    •     Controllability
    •     Empowerment

    Xu also mentions Scalability and Sustainability, but I deemed these things to be decided at model level rather than a specific AI interface.

    How to use

    The questions are (5-point Likert; Strongly Disagree – Strongly Agree):

    •     The agent’s capabilities match my needs
    •     It was easy to achieve what I wanted using the agent
    •     I trust the agent’s responses
    •     Using the agent enhances my own capabilities
    •     I can consistently get the answers I want to my questions

    This formula gives a final score out of 100:

    (((∑Q1–5)-5)*(100/25))+10

    (Sum the scores, subtract 5, multiply by 4 and add 10 to the result)

    Administer this summative questionnaire after several scenarios at the end of a usability testing session or add this to a contextual survey on your website to get a longitudinal view of the score.

    Validating the score

    Tested and iterated across 2 rounds of testing the final scaled score was reliable (Cormbach’s alpha = .88) and correlated with NPS (r=.80, p=<.001, n=36) and CSAT (r=.92, p=<.001, n=18).

    Though the results are encouraging there are obvious limitations. I would really love for the UX community to test the score on a real product with larger samples and tell me how it went. I’m also keen to hear ideas and feedback on how to improve it.

  • On synthetic users

    On synthetic users

    Can we talk about synthetic users? We, experience designers (or researchers; HCD is a big umbrella), all know they’re a bad idea… but why? And how do we convince our business colleagues without sounding like the whole field is in the “AI is gonna take our jobs” panic?

    While slighly bored during what Romesh Ranganathan on BBC2 called “crimbo limbo” I decided to play around with synthetic personas. It was either that or clearing out the garage, don’t judge. Here’re the results. Spoiler, they’re not fantastic.

    I ran two small scale experiments. One with qualitative data from interviews (thank you Timothy Price from Newcastle University for making the anonymised transcripts available for download!); the other with no data beyond some basic demographic data and assumptions.

    Experiment with data

    I created a Custom GPT with 10 interview transcripts as a base. The interviews were around parent’s knowledge of the flu and their thoughts about vaccinations. I asked it to reply as a parent in first person and always present its opinions as its own, using words like “I think” and “I feel”. I then fed it various service concepts ranging from benign (digital service to book vaccinations) to downright scary and unethical (let’s monitor a specific group of people). The persona was overjoyed to use the service, completely ignoring the fascist undertones. Not once did it say anything critical.

    On the plus side, it was very faithful to the transcripts and had extra detail to expand on if I asked a follow-up question. Its main concerns were privacy, security, and accessibility.

    Conclusion: your design a burnt toast and it’ll still hail it the best thing since sliced bread. I wouldn’t use it for anything beyond basic data retrieval from the transcripts, and even then I’d double check everything.

    Experiment with no data

    I created a similar Custom GPT with the same system prompt but it was directed to use its general training when coming up with responses. Funnily enough it didn’t make much difference. It thought that a government monitoring service for a specific ethnic group was a splendid idea.

    Its main concerns were privacy, security, and accessibility.

    Conclusion: way too generic, and oddly similar to the previous bot.

    Discussion

    It seems that no matter what data you throw into it, the responses will be generic and overly positive. So next time someone at a conference tells you that their company makes synthetic personas powered by a bazillion data points from all across the internet — don’t believe them! Those bazillion data points will not make the persona any better, they’ll just add a bazillion more points of failure, and tell you to think about privacy, security, and accessibility.

    There are perfectly valid reasons to have a digital twin system simulating the real world in real time. This is what synthetic users are, really. However, people are infinitely more complex than an electric turbine or even a space rocket. AI might make good estimation and have a bazillion data points to draw upon, but—and it’s a big ‘but’—would you trust AI with your space rocket?

    Therefore, my point of view is that synthetic users are shit. AI has a thousand more useful use cases that make the world a better place.

    Notes

  • Knowledge management in the era of AI

    Knowledge management in the era of AI

    I’ve been thinking about this over the weekend. We spent the last two years feverishly trying out approaches to make research analysis easier for ourselves. First we tried simply dropping in all our interview transcripts into the chat and let it analyse it. When that didn’t work, and honestly resulted in more hallucinations than anyone thought possible, we switched tact and tried to chunk the data. We experimented with aligning transcripts to interview questions. We even, at one point, added interview transcripts to a CSV file, pre-processed it, and created a JSON output to make them more “LLM friendly”. Every time something was missing.

    LLMs are inherently black boxes of magic. Stuff goes in, complicated statistical stuff goes on, AI slop comes out. Sure, they’re useful in their own small way to write stuff that sounds like it was written by AI, or summarise some high level stuff.

    Slight tangent. I recently read some _insights_ created by an LLM model from a bunch of transcripts. Turns out that data scientists and analysts _need_ to process data. Blew my mind.

    Back to our black boxes of magic. The point is, no one, even the developers, really knows what’s going on inside that black box of an LLM. Once the training data is in, that’s it. It goes and does stuff. And that is a problem.

    How can we trust anything the LLM produces if we can’t explain it?

    So here’s my humble contribution to the field of Explainable AI. I present to you the first epistemologically transparent cognitive dashboard:

    Made with Figma Make

    If you’ve ever seen a debating or argument building software, it’s pretty much that. I’m not reinventing the wheel here. The difference is in who the users are. I envision mini-agents building logical strings of arguments to support any claim. Think of it as a cognitive assistant to make research more traceable. It’s AI-powered in a sense that mini-AI-agents are spinning their wheel working on a single piece of evidence at a time.

    You upload your sources, the little swarm of agents go and break your sources into pieces, process them, tokenize them, and build a shared context together. Then you, the user, can interrogate the data, make claims, ask the AI to build argument graphs to support your arguments, and so on. Suddenly we’re using LLMs to help us construct knowledge and logic, rather than mindlessly processing context windows to spit out a hallucination of what the truth might be.

    As ChatGPT sycophantically praised my idea as a paradigm shift in human cognition… yeah… no… I still couldn’t help but think how neat it would be to have traceable argument chains that show how the LLM came to this or that conclusion.

    Perhaps we can use it in research repositories, perhaps in academia, policy, strategy. Perhaps we can stop the decline of critical thinking in schools by saying, “cool, you want to use AI? Great! Bring me an AI written paper with epistemological transparency. Show me how you got the answer!” Wouldn’t it be neat?

    You could dive into sources to inspect citations, contradictions, and epistemic status of a node. You could mess around with all sorts of fun meta-data that your helpful swarm of agents helpfully made for you. It’s not really NotebookLM, or Obsidian. It doesn’t just take your notes and spits out things it knows about them. It’s a structured reasoning tool that makes LLMs less of a black box of magic.

    So… I guess this is a call to action. It would be great to connect with researchers, engineers, just people who care about epistemic transparency of LLMs. Honestly, if I could build this I would use the crap out of it as a research repo and analysis tool.

    Drop me a line! We can get the MVP out in no time 😉

  • Design? Research? Problem solving.

    Design? Research? Problem solving.

    About 10 years ago I went on IDEO’s human-centred design course. It was free, it was easy to grasp, and it gave us the tools we needed to design great products. It’s still around, you can check it out here.

    At the start of the course we learned about the design process: inspiration, ideation, implementation. We learned that first you speak to people. The methods included interviews, surveys, observation, and more. Then you brainstorm some ideas, test them and iterate until you got to a great solution. Finally you implement it, while continuously gathering feedback and tweaking the final solution until it’s “perfect”.

    “Okay,” I thought, “maybe it’s pretty similar to the process I learned about in uni: Discover, Design, Test, Manufacture. Or maybe it’s similar to the process I learned in university: Research, Build, Iterate Aaaaand! It’s pretty similar to our beloved Double Diamond from the Design Council.

    What’s going on?! It’s almost as if design process is universal whether you’re designing a video game character, a magazine cover, a chair, a toothbrush, or a website or service. The point I’m trying to make is that design is design.

    Why then, is the industry so hell bent on dividing researchers from designers?

    Look at any user researcher job on LinkedIn. Here’s one:

    Bachelor’s degree in a relevant field (e.g., Human-Computer Interaction, Psychology, Data Science, Public Health). Advanced degrees are a strong plus.

    And here’s a requirement for a designer:

    Bachelor’s degree in Design, Human-Computer Interaction, Psychology, or a related field; Master’s degree preferred.

    Does that mean that I designer won’t be picked for a research role? And vise versa? That’s just stupid.

    Let me tell you a story. I went to a conference once and met some junior designers from a FAANG that I won’t name. User journey map? Haven’t heard of one. Do you run user interviews? No. What do you do then? We design the screens.

    I could only chuckle nervously and excuse myself.

    Here’s the problem with researchers who don’t design and designers who don’t research. Let’s say you’re a researcher who found some insights. You present them to the design team and they go away to do some designs but they don’t have the full context. It’s like having two halves of the brain that don’t speak to each other. I mean, we try, we communicate, but things ultimately fall through. We have these insight trackers and knowledge repositories, which are undoubtedly useful but don’t solve the core problem: having the full context.

    My best time as a researcher was when I worked with a designer who used to run research sessions himself. He grasped exactly what was needed to be done, helped with the admin, and took some ace notes. Then, I took on some screens to design and helped him in return. We worked as one unit. It was great.

    So the case I’m trying to make here is that designers should be researchers. You don’t need a PhD in Psychology to speak to some people and learn about their problems. Similarly, if you’re a researcher, I highly suggest you pick up some design skills. Ultimately, if you’re a hiring manager and you see a designer apply for a research position, think twice before rejecting them, and vice versa for researchers applying for design positions.

    The portfolio should speak for itself.

  • How does AI actually impact our daily lives?

    How does AI actually impact our daily lives?

    If we take away the incessant torrent of AI news, articles, panicked Reddit posts, trashy Tik Tok videos, and other slosh. What’s left? How do you actually use AI in your day to day?

    Let’s start with a proud sample of one — me!

    Last month I used AI to generate some voice overs for a marketing video and an AI avatar tool to put faces to the words. I wasn’t hiding it was AI generated though, it was part of it. I’d say it’s a pretty niche use case.

    Last week I used AI to do deep research for a Request for Proposal for a client. I ain’t spending a week reviewing 140 websites about digital strategies and competitor analysis. I trust the AI to do it for me, even if it’s not 100% perfect. It saved a ton of time and all I had was to read the insights, write some human analysis as to what they mean, and stick them into a presentation.

    This morning I used AI to give me ideas on how to make my oeufs en cocotte less runny when they come out the oven. Before that I generated some character names for my roleplaying game.

    AI seems to have slipped into my day to day pretty seamlessly. I like it. It’s like having a slightly overeager information butler at my fingertips. I don’t Google thins as often anymore, as predicted by some report somewhere. I don’t spend too much time doing desk research. Is it bad? Absolutely not. Does it impact my creativity? Also no. It’s more of a brainstorming partner. I’m making all the decisions. I don’t think I ever saw it come up with a truly original idea. You’d need a human for that.

    But hey, I wouldn’t call myself the typical user. I actually work with AI for a living, kinda. Let’s take another, more robust sample. Let’s ask some real people how they use AI and how it impacts them. Conveniently we did some interviews on just this topic back in March. Here’s what we came up with.

    People use AI to write letters, plan holidays, and generate emails

    Big insight (yes, sarcasm). Most peoples’ use cases kinda align with mine. AI has become a part of our lives in a completely expected and boring way. It removes the minutiae of having to do boring stuff.

    AI can be your lawyer, wedding planner, can help write a complaint that actually gets heard. It opens up all sorts of opportunities. Listen to this. One participant was telling us about a legal battle they had with a company.

    > When I was writing my email to the [company]. I did, like, so that it sounded legally sound. My housemate did put it through a ChatGPT type thing. So I think it’s almost like, interesting to think about the fact that it’s not only like companies who are using it, but, like, people themselves, I imagine, when they’re having complaints.

    Isn’t it great? You have a lawyer in your pocket now! It’ll get a bunch of facts wrong, and hallucinate another bunch, but boy can it write a good complaint.

    People use virtual assistants like Siri, Alexa, and Google Assistant for tasks, reminders, checking weather, and controlling home devices

    Aha! Got you! I never said we’ll be talking about generative AI, did I?

    But seriously, this finding points at an interesting difference between how we—designers, researchers, etc., working in tech—think about AI and how Mrs. Smith from down the road thinks about it. Alexa is still the pinnacle of AI technology for many people.

    Conclusion? Quite a few actually.

    Firstly, you’ll probably need to explain generative AI to your gran when it comes to her Alexa.

    Secondly, when it does, it’s going to be fantastic for older people. We already know that older folk sometimes use Alexa to alleviate loneliness. Imagine if it could talk like a human and provide some emotional support from time to time like ChatGPT does? For this generation the AI revolution won’t be about making hyper-realistic promo videos. It’ll be about a familiar voice becoming that little bit smarter. I quite look forward to this one.

    People use AI to translate and learn languages

    One participant used ChatGPT to surprise his Danish wife. His granddaughter used it to write a poem in the style of William Shakespeare for school and it gave him an idea to do something similar but in Danish. Isn’t it lovely?

    AI is opening a whole new world of languages for us. There are already demos of real-time translation (similar to what Google Translate does), but the added value of learning the language is pretty neat.

    Conclusions? There is something about making localisation easy, perhaps enriching simple translation with an element of learning is a neat idea. Ultimately though, AI is democratizing the connection and lowering the barrier to not just understanding but connecting with other across linguistic divides.

    Some conclusions

    So after sifting through the hype and looking at how we actually use it, what’s the real story here?

    AI has clearly become a part of our lives in a very subtle and non-invasive way. This “Boring Revolution” is actually kinda nice. I’m sick and tired of the AI hype online. How it’s going to take our jobs and disrupt our very way of life… and other dramatic predictions.

    Instead I’m coming to think of it as the great enabler. It’s not about sentient robots taking over or all designers losing their jobs. It’s about people gaining superpowers—like sifting through the noise and getting the information you’re actually searching for, or writing a near-perfect complaint letter, or feeling less lonely.

    This is wonderfully “boring” in a world where disinformation and AI-slosh is taking over.

    N.B.: hey, we’re all aware of the risks, but just in case you aren’t, this “boring” revolution is still very much open to bad actors. Disinformation can become quite insidious and hard to spot. Companies can learn to optimise their content to make sure they appear in AI search results. Marketing departments and bad state actors can manipulate the narrative. Imagine if ChatGPT tells your some fake news? But stop. Just stop. Let’s not think about that for now. Let us enjoy the fuzzy, cosy glow of the ‘boring’ revolution for just a minute longer.

  • A discovery in healthcare

    A discovery in healthcare

    We worked with the UK Health Security Agency to identify opportunities to improve the Measles Surveillance Pathway. Navigating internal politics and ethical considerations, I led a team to plan and conduct a Discovery. We identified £800k in estimated efficiency savings through automation and process improvement.

    I stood up a team to lead a Discovery phase into the Measles Surveillance Pathway at UK Health Security Agency (HSA). Navigating complex organisational and policy landscape our team successfully interviewed 35 Health Protection Team (HPT) members. We produced 75+ insights, 4 user archetypes, fed into a creation of the “to-be” service blueprint.

    Situation

    Measles Surveillance is a complex pathway where multiple teams must work together to produce accurate data about the measles cases in the UK. For this the Health Security Agency, responsible for all health related threats to the UK security, created the current “pathway”. It is supported by multiple regional Health Protection Teams, responsible for day to day health of their respective regions.

    When a measles case is suspected (by a doctor, or a school nurse, or another healthcare practitioner) a report is made to the local HPT. The local HPT investigates, conduct any necessary follow ups (usually in the form of an oral kit sent to the patient), and send the samples off to a lab. If a measles case is confirmed necessary interventions are made.

    Task

    Our team was brought to conduct generative research with HPTs to understand their pain points and needs, and to identify opportunities to increase efficiency and oral fluid kit (OFK) return rates.

    Approach

    We opted for in-depth interviews as the most pragmatic approach. Surveys, contextual inquiries, and focus groups were considered but rejected due to time, budget, or organisational constraints.

    There were several challenges the team had overcome while working in this regulation heavy environment with low UX-maturity.

    1. Organisational resistance

    Due to the decentralised nature of the regional HPTs it wasn’t immediately obvious who to contact about recruiting participants for our interviews; additionally there seemed to be negative sentiment from the HPTs about the governing HSA; a relatively new department that came to replace the old Public Health England.

    To avoid further delays we went to the source. We contacted HPT team leads directly and asked them to attend an introduction meeting with us. During this meeting we explained the nature of the research and what we were trying to achieve.

    While the immediate resistance was overcome, and we got our recruitment emails out to the teams, the negative sentiment remained throughout. We navigated this by making our intentions and the nature of our research clear upfront in the information sheets we produced.

    Relevant skills: managing consent, stakeholder management, relationship building, influencing, advocating for research.

    2. Stakeholder resistance

    Our first draft of the plan was met with scepticism. Stakeholders from clinical and academic backgrounds felt our epistemology was lacking and our approach was not robust enough to give the insights any confidence or weight. These stakeholders were initially not available for conversation and it took some time before we got everyone in the room together.

    During our conversation I had a chance to educated them about UX research and how it’s different. How the scope and the objective of the research differs from academic or clinical research, for example. I steered the conversation away from NHS Ethics Board Reviews and more towards collaborative workshops and solving a challenge together. This was successful in getting buy-in and unblocking our path forward.

    Relevant skills: advocating for research, stakeholder management.

    3. Data protection concerns

    No Data Protection Impact Assessment was conducted prior to the research team coming onboard. This delayed us significantly. However, I worked with the senior leadership and Quality, Risk, Mitigation (QRM) colleagues to get a Data Protection Agreement (DPA) in place.

    Relevant skills: data protection regulations.

    Result

    As a result of my advocacy and strong stakeholder management skills we conducted 35 interviews with HPT members. Participants, once they understood what we were trying to achieve, were helpful and open, providing rich insight into their daily lives, frustrations, and needs. The final report, containing 4 personas, 9 efficiency saving concept designs, and 1 “to-be” service blueprint, was well received by the client and was used to plan an Alpha phase in the future.

  • Marine Licensing Discovery

    Marine Licensing Discovery

    Situation

    A government department’s ambition was to transform its marine licensing service. The complexity of the process, ministerial oversight, and several failed attempts in previous years made this a challenging environment that required careful stakeholder management and ambiguity around recruitment.

    Task

    The team was tasked with conducting a standard Discovery to understand the current process, and identify opportunities, pain points, and behaviours. A report of our recommendations and early concepts was to be presented to the government ministers who would issue budget for the next phase of work.

    Actions

    As a small and agile research team we worked closely with Product, Design, and Delivery, to plan and execute the research in record time.

    To make sense of the departmental context I conducted several in-depth research kick-off workshops that challenges the client to think about their assumptions. These assumptions were then turned into testable hypotheses and success measures.

    We used generative AI to quickly conduct desk research and review 50+ sources for key themes and existing insight.

    We then conducted interviews with a complex network of service users and stakeholders, mapping the process as we went. We produced lean personas, process maps, and ‘as-is’ service blueprints.

    Recruitment included:

    • Stakeholders (e.g., port authorities, scientific organisations, and other government departments) that were recruited using snowball recruitment through word of mouth and existing relationships.
    • Service users (e.g., engineers and building company leaders applying for a licenses) that were recruited using a third party recruiter.

    Result

    We spoke to 10+ service users and 5 stakeholders. The final report, which included 30+ insights and recommendations, along with a “to-be” service blueprint and several concepts and hypotheses for future testing, was presented to the small project team and a clean version was produced for ministerial scrutiny.

    The department was successful at securing the funding and proceeding to a longer Alpha phase.

  • Central Government Alpha

    Central Government Alpha

    Situation

    The client was amidst a complex digital transformation programme. The ambition was to digitise and modernise by automating manual processes, creating robust data processing capabilities, and re-designing their end-to-end service.

    My responsibilities included supporting the client’s head of research and leading a team of 18 client and Deloitte researchers across 4 streams of work, supporting the wider product leadership, and ensuring the services all built to all relevant standards (GOV Service Standard, WCAG).

    The situation within the team was challenging. Specifically:

    • Immense complexity with 50+ user types identified in Discovery resulting in difficulty prioritising effort
    • Large number of user needs with significant overlap between work streams resulting in duplicated work
    • Demotivated teams with low morale, they’ve been at it for two years and the complexity was only rising
    • Unsuccessful digital transformation efforts in the past left stakeholders and users weary and distrustful
    • The participant pool was drying up with internal stakeholders showing signs of fatigue, and external participants coming back for repeat visits impacting the validity of our findings

    Task

    I supported the client’s head of research to create a unified roadmap of user research, ensure the new service is accessible, ensure high quality and reliability of research to withstand ministerial scrutiny and pass the GOV Service Assessment, and augment and upskill the existing teams with knowledge transfers, training, and mentorship.

    Actions

    Introducing OKRs

    From the initial 1:1s with the team and the subsequent full-team retro we identified 8 areas of improvement that were then synthesised into actionable goals and measurable results.

    Standardising research

    I worked with the Product and Delivery leadership to align research activities to the release roadmap across all work streams. My proposed approach was to work in dual-track Agile with the aim of ensuring both evaluative and generative research is considered.

    I also introduced a standard research checklist, report and kick-off templates, and other collateral to ensure uniformity and high quality of outputs.

    Standardising personas, introducing archetypes

    The client’s 50+ user personas were synthesised into 6 high level archetypes in an attempt to simplify and standardise the programme’s approach. We worked with the brand team to create engaging visuals for each archetype and socialised them widely within the department, even printing out posters and hanging them around the office.

    Addressing participant fatigue

    By introducing research a prioritisation matrix we identified that not every feature needs to be researched in depth. Some features with low ambiguity can be evaluated in a leaner manner reducing the overall strain on the limited participant pool. To address this we began onboarding UserZoom to empower the design team to run their own unmoderated usability tests from templates we’ve set up for them.

    The client already had an internal participant panel and we spent some time reviewing and improving it, as well as running several email interviews to understand how participants felt about being on the panel. We wrote a communications guide for research ops colleagues sending out invites to reflect some of this feedback, e.g., “we know it’s been tough but we need you!”

    Made a significant decision that we will not exclude repeat participants, but instead put a timer on how often they can participate. The timer was initially 6 months but was later changed to 2.

    Centralising insight

    I introduced a central research repository and championed the use of Dovetail. While I supported the Head of Research with internal business case and onboarding efforts, I built a research repository in Excel.

    Each insight was paired with its parent work stream, supported by specific evidence, and had a column for any actions that were made based on this insight. I also introduced monthly grooming meetings where each work stream lead updated their actions for the previous month. This approach significantly increased transparency of research, our ability to cross-reference similar data from different work streams, and our ability to track actions therefore evidencing impact.

    Introducing new methods

    I introduced new methods like:

    • The Kano model to improve prioritisation
    • UX-Lite to measure impact of design
    • Rainbow analysis to make analysis fast, fun, and visual

    Improving Culture

    I ran one off-site for the team where we did team building exercises and got to know each other. This was, according to my end of year written feedback, one of the most fun events to happen on the programme.

    I also introduced standard rituals like frequent retros, manuals of me, regular 1:1s, and ‘fun Fridays’ where we would play skribbl.io and other online games together.

    Result

    It’s hard to measure, but I think the team was much happier when I left, as evidenced by written feedback I received:

    A massive thanks to you Arty for being a great UR but an even better human being. I’d like to think I shared with you quite often the positive impact and useful things you taught me in our time working together. You have been, hands down, the most positively impactful Deloitte UR that joined the team and working with you was great fun. — Senior User Researcher

    Or this one:

    Oh my gosh, what value didn’t you add?? You came to us when we desperately needed an injection of new life, new ideas and refreshed approaches, and you certainly brought that. I know the whole team benefited from your expertise and vigour. — Head of Research

    More objectively however:

    • The teams passed 3 Alpha service assessments, receiving “Green” across the board for research related standards
    • We achieved 80% insight to action rate as evidenced in the central repository
    • We raised UX-Lite by 16 points in 12 months (aggregated across all work streams)