Side knows playtesting. We’ve been at this for over a decade, while leading the charge in QA testing for over 30 years. And we’ve seen game companies struggle to make sense of 50 players’ feedback – let alone 1000, or even 100,000. All that data seems nice, but what do you do with it? How do you create actionable insights from masses of player feedback to improve your game?
That’s exactly the challenge we tackled when we developed our AI-powered playtesting analysis system. As the only company in the world delivering mass-scale playtests of up to 100,000 users, as part of the Razer Cortex Playtest Program - Powered by Side, we built a series of AI tools to organize and process overwhelming feedback volumes into clear development priorities. This technical brief explains our approach to solving the big data challenge in playtesting.
Language Analysis with LLMs
The first step of Side’s proprietary data analysis AI is to leverage large-language models such as OpenAI, Gemini, and Llama. We run all text-based player feedback through these LLMs to extract the most common likes and dislikes that players report. Using LLMs enables us to extract the meaning of what players are saying, regardless of how they say it. This allows similar text-based feedback to be combined to find the top insights from masses of player feedback.
Here’s how this works in practice. Imagine three players provide three different things they liked about a game:
- Player 1: “I liked the animation of the gun firing. It’s crisp and looks cool.”
- Player 2: “The art of the game looks incredible, especially while moving around.”
- Player 3: “The game runs smoothly and the characters feel lifelike.”
At first glance, it might seem like three completely different observations, but our LLM-based AI notices a common thread and synthesizes them into a unified insight, counting three data points towards a common like: “Players are finding the animation and movement in the game to be smooth and beautiful.”
While it might not seem like much, this kind of insight, gleaned from analyzing 1,000 or 100,000 playtesters over multiple hours of gameplay, can really make the difference in understanding what players most commonly like about the game and what most players most commonly dislike.
Adding Game Context
But it’s not enough to simply run text through an LLM analyzer – we must also feed it context about the game. Without knowledge of the game's genre, characters, setting, and core mechanics, it would be difficult to put player feedback into context. For example, players saying “the ships were too small” could have two different meanings for a space game versus a sailing game.
Side’s AI systems automatically add context about the game when analyzing the player feedback provided, thus ensuring that feedback is interpreted accurately and avoiding misleading conclusions.
Synthesizing Data Across Custom Survey Fields
Not every field in a player feedback survey is standard or even text based. Some use multiple choice, others are “yes/no” questions, and some ask about demographics or computer specifications. Regardless of the custom field that a game might add to our player survey, Side’s AI uses two approaches to analyze it.
First, we convert responses into a text-based format. For example, asking players to rate how well they liked the game on a scale of 1 to 5 becomes a text feedback field describing: “The player rated the game 3 out of 5.” This allows the LLM analyzer above to work its magic and extract the most common answers.
Second, we perform automatic basic statistical analysis on all non-text fields. For example, the average value, median, and mode. This statistical data becomes very useful in the next two stages of analysis: correlation and segmentation analysis.
Identifying Relationships
Any field that’s not a text-only field, such as multiple choice or “yes/no”, can also be treated in a numerical way for correlation analysis. Our system can then analyze and report on both positive and negative correlations between any two fields.
As part of the Razer Cortex Playtest Program – Powered by Side, we’re able to track player game time, too. This data becomes a key field for identifying correlations between players’ actual playtime and all other fields.
For example, we perform correlation between how long a player played and their average scores and sentiment, leading to insights about playtime affecting how much a player likes a game or not. In another example, we also create cohorts of players based on their playtime to analyze players who played only a little bit compared to those who played a lot – allowing for further segmentation as needed.
Analyzing Player Segments
While the above LLM and correlation analysis is helpful, sometimes we just want to examine data from a specific segment. For instance, you may request an analysis that focuses only on women between 18 and 25 years old. In cases like these, we can generate a second report based exclusively on the data provided by players matching those criteria, while applying the same comprehensive approach to reveal segment-specific insights.
Providing Actionable Insights for Improvement
Once all the data is analyzed and gathered into a feedback report, there is one last critical step that Side’s AI system performs. Using the context about the game, the correlation analysis, the AI-based language analysis, and the custom fields, Side’s AI combines all the feedback and provides specific suggestions to improve your game.
Not only does our pre-trained AI suggest ideas for the specific areas where players reported issues or dislikes (for both text and non-text fields), it also reports on things that players loved and how to highlight them even more. And since these insights can be given for any possible desired segment, game studios can then tailor their improvement plans based on their target demographic in the next iteration of the game.
Dealing with Bugs
Of course, not all feedback relates to design choices – bug reporting remains one of the biggest challenges when scaling playtesting. Whether we’re talking about 1,000 or 100,000 playtesters, a single gameplay issue in a large-scale test could lead to thousands of individual reports, each describing the same problem.
Side’s bug report analysis and deduplication tool was engineered to tackle this challenge head-on. Our tool relies on LLM technology to find duplicate bugs even when the wording of the bug is not identical – automatically extracting bugs from player comments, formatting them, and deduplicating them so that QA experts can review them. This deduplication feature is tunable to be more or less conservative for the purpose of preventing accidental duplicate tagging (false positives).
While our tool is efficient at processing this data, it’s still essential to review the results, which is where Side’s QA specialists come in.
Human-in-the-Loop
AI plays a central role in our playtesting system; however, a reliable and accurate tech-based process still requires human oversight. For playtesting, this happens in two ways.
First, our team of QA experts review the AI’s analysis to ensure it makes sense, complies with our privacy and security guidelines, and delivers the high-value insights that mass playtesting promises.
Second, as an additional layer if our clients require it, all bugs that are detected during the playtest can be given to our expert QA teams spanning the globe for further processing. Our teams ensure the bug is real, is reproducible, is written clearly with reproduction steps, has a video attached, and is correctly filed in your bug ticket system.
This human-in-the-loop approach means the feedback is even more precise, so you can plan and execute your next development phase with greater efficiency.
Privacy and Security
Data security is critical when dealing with your intellectual property and private information, and it’s fundamental to our playtesting platform. We follow ISO-27001 policies and procedures to ensure all data that is collected is kept safe, private, and encrypted.
Our playtest system's security measures include:
- Data encryption at rest and in flight
- Regular penetration and code security testing
- Enterprise-grade security software and techniques that keep all servers patched and secure
- Using LLM models that we fully control which keep data siloed and out of training datasets
- Full GDPR compliance
- Removal and anonymization of any PII (Personally Identifiable Information) that should not be kept in databases
This security-first approach ensures your data remains protected throughout the entire AI analysis pipeline.
Quotes
“While AI processes thousands of responses and finds patterns, human QA experts review all analysis for accuracy and actionable value.”
The Bottom Line
The Razer Cortex Playtest Program, powered by Side’s AI system, is completely changing the landscape of game playtesting. The future of playtesting is no longer limited to just 10 or 50 playtesters in a room, but instead opens the possibility for 1,000, 10,000, or even 100,000 or more, all playing and reporting bugs at the same time.
This capacity to process large volumes of data gives studios the ability to make much better development decisions for their game – without having to spend an exponentially greater amount of time or money. Instead of dealing with limited or overwhelming feedback, studios get comprehensive insights that guide confident development choices.
Want to explore how AI-powered player feedback and analysis can fit into your development pipeline? Get in touch to see how our playtest program can be customized to help you level up your game.