How do you check out reactions to a new feature in your software application without disturbing your regular users? For many years, the answer has been to turn to companies like UserTesting, which curates a massive pool of 7.6 million testers across the globe for just this purpose. 

Now the company has found a broader role thanks to AI, through making its community more accessible for a range of purposes — including the growing demand to train and validate AI responses. To signal that wider mission, it has changed its name this week to Auros, while retaining the product brands of User Testing and User Interviews under the umbrella of the new corporate identity.

The old name is less appropriate to fast-growing use cases such as market research and AI training, explains Eric Johnson, CEO of Auros:

This whole world of AI training, it's not user testing, right? We're providing humans to help evaluate these models and agents, and things that these folks are building... We will use the word human a lot, human intelligence. We will heavily tie that to AI, and [what] you will not see in there is the word user.

The idea of going to an external panel of testers rather than simply recruiting an early access sample within the existing user base is surprisingly common practice. The company's customer base mostly comprises large enterprises in industries such as banking, airlines and food, with only around a fifth coming from the tech industry. Traditionally, these organizations have used the company's community to get feedback as they're building or updating digital apps, experiences and content. Now the use cases are broadening out, with product designers and engineers wanting to get insights much earlier in the research and development process, and now the entirely new field of training, evaluating and validating the outputs of LLMs and agents. Johnson comments:

The difference in this company from what it was, from two years ago to what we now can go do, is massively different. We went from being a niche UX research tool company to this broader provider in a world where AI is creating so much need for information and insights, and that we have the best ability to do it.

The breadth of the Auros community, not only demographically but also across different industries and job roles, is a particular asset, especially for more esoteric use cases. He gives an example:

Some of those B2B use cases are actually the most valuable, because those are some of the cases where it's hardest to find feedback. If you are, say, a healthcare company or a medical device maker, and you're doing something digital, and your customer is not an average consumer — it is a bespoke type of doctor, it's a cancer doctor — we have the ability to go find those people, and then we have all these mechanisms for them to give feedback.

As well as providing a new source of demand for the company's services, AI has also been a significant catalyst in making those services more accessible to a wider range of roles within organizations, by automating many of the processes that previously needed a UX specialist to set up. He explains:

We're now allowing people like product managers and designers just to go into a chat experience and ask the questions that they're looking for help on, and then our experience effectively goes into our platform. The agent goes in there, designs a different test, finds the right audience, and comes back with the insight. That allows that professional, who in the past would have gone and asked a researcher, to basically get the insight themselves, and that's obviously much more efficient.

The recent launch of an MCP server means that a customer can now invoke the interview and feedback process directly from an agent such as ChatGPT, Claude, Gemini or Figma Make, simply by asking the questions they want insights on without leaving the AI workflow. Those and other agents are also blurring the traditional boundaries between specialist roles in organizations, so that product designers and marketing teams are more likely to want to get feedback. 

Another factor driving demand is that agents are speeding up the software development life cycle, as well as enabling more choice in the user experience, so that there are so many more elements that need testing. A further source of demand is that AI automation means that the capacity to build product is much less of a bottleneck, and now companies have to decide between several different options. Johnson explains:

Sometimes someone's interacting with an agent. Sometimes they're interacting with a dynamic UI. Sometimes they're interacting with a static UI. If you think of all the places where you need feedback, the need for feedback has gotten exponentially larger. So what's exciting for us is that when I got here [two years ago], we were this very niche market company, and that market was getting harder — but then the broader set of people who became builders got way bigger, the velocity they work at got way higher, and the hardest part about it has become, what do I build? That's what we provide input on, because we are the world's best company at going and finding the right people to give you that input.

Now the company is prepping a new service for evaluating AI experiences that goes beyond aspects such as accuracy, efficiency and safety to test human relationship elements like trust and affinity. The new AI Relationship Quality (ARQ) benchmark will debut next week.

Underneath the surface, all of the company's acquired expertise in constructing questions, finding the right mix of interviewees, and assembling the findings, comes into play. He explains:

We have a lot of methodology and a lot of options on how you get the feedback, which... result in much higher quality feedback. Part of the problem when you get people just doing their direct, more 'guerrilla' methods, the feedback may not be statistically significant. They may not be asking the questions properly. They may be leaving things out. When they leverage our full capability set, they're getting the benefit of all our experience. We're helping the customer do these things better.

The ability to analyze video interviews rather than simply look at the result of A/B testing can also yield valuable insights. He goes on:

A lot of what the world has historically done is, they create multiple different versions or experiences. They let people do them, and then they just look at the raw data around what happened. But the problem is, they don't know why those things were happening.

When you watch a customer doing it on video, and you hear them thinking through what they're doing, then you know the why. Sometimes what happens is, even if you had, say three variants, and one was clearly better than the other, you never know why, and you don't even know how to make that one better. But when you watch them, you hear them explain it, and then you look, 'Oh my gosh,' and it makes you think of another idea, and then you create the next. You can make the thing even better.

In all the discussions about whether AI will replace humans, the role of humans as the ultimate consumers of what agents produce is often forgotten. But as the pace of both software development and AI sophistication accelerates, there's a growing need for actual humans to test the outcomes, not just for quality but also for more subjective criteria in a range of different contexts, some of them highly specialized. And as the ability to produce new applications and functionality becomes less constrained, then being able to research the potential demand for these new capabilities becomes more important. AI tester is one of many new job roles for people to do that we never realized we'd need before AI came along, but now there's already more than seven million of them and growing.