Contents
In previous blog posts, we discussed prioritizing your content moderation efforts and which methods to use. But even though your content moderation process is now in full swing, you shouldn’t put your feet up. Now it is time to analyze your efforts and improve your content moderation.
It can, however, be hard to decide which KPIs to focus on in the stats jungle, and it is easy to get overwhelmed.
Fortunately, through our years of content moderation, we have learned enough to list the most important data to track. The list is not exhaustive but covers the top KPIs you must focus on when doing online content moderation.
Before we start, it is important to note that this article covers KPIs relevant to improving the moderation process and your product. We will not cover agent and team performance stats, like the number of ads reviewed per hour/agent.
How to automate up to 80% of your content moderation
Get exclusive insights and guidance on tackling your content moderation challenges with Implio. Moderation expert and Besedo customer success manager Alessio Coco will take you through our AI capabilities, automated filters, and manual interface in the tool.
Actionable and accessible metrics
The first lesson in the school of metrics is focus. It’s easy to get lost in analytics and create a cockpit of graphs just because you can. But what are you going to use it for? Metrics should be actionable and foster improvements. But it’s not just for you. Everyone in the organization needs access to the metrics required to make informed decisions and contribute toward company goals.
1. Publish time
Measuring publish time will tell you how long a seller has to wait for his or her content to be visible to potential buyers.
You will want this number to be as low as possible, which means a better seller experience. With a long publish time, you risk an increase in duplicate postings, customers leaving for the competition, and an increase in customer support tickets.
How to track:
(Time of content going live on your site) – (Time user submits content) = publish time
Why you should track this:
Automation, post-moderation, and machine learning will usually help you keep this number low. At the same time, you will have a longer time to site if you rely solely on manual moderation.
Ensure you have the right systems in place and fine-tune your automation tool and processes to minimize the manual queues.
2. Average lifetime of reviewed scams
With this metric, you are tracking how long, on average, scams are live on your site.
Keeping an eye on this metric will give you a good indicator of how good your team is at quickly identifying and removing scams from your site. The higher this number is, the bigger the risk of your users getting scammed. In a sense, this metric is tightly connected to monitoring user experience.
How to track:
(Time of Refusal) – (Time of Publishing) = Lifetime of reviewed scam.
Why you should track this:
Use this metric to identify the categories that are most likely to contain scams, and based on this, create a post-moderation action plan.
3. Refusal rate (and reason)
The refusal rate is a great indicator of the seller’s experience. It should be as low as possible while maintaining the quality you want for your site.
You also need to track refusal reasons so that you can provide your users with a clear explanation which helps educate them and, over time, improves the quality of their listings.
A low refusal rate means that it’s easy for a seller to get their ad published on your site.
On the other hand, a high refusal rate reflects a negative user experience for sellers, as getting their content out to potential buyers is not a smooth process.
How to track:
How you track this number will depend on the tool you use for moderation. A good moderation tool will track and store info whenever you refuse an ad and give you data like the most common rejection reasons.
Why you should track this:
If you have a high refusal rate, you should look at how seller needs and your site policies can be better aligned without jeopardizing the quality of your site. Refusal reasons will help you dig into the underlying issue and better understand why content is being rejected.
Remember that a low refusal rate could indicate an issue with your moderation rules. Still, it could also mean that your moderators or automation solution is letting too much through.
4. Refusal rate (and reason) per category
There are many interesting conclusions to be drawn from refusal rates per category. Many will be very specific to your site and the way it is structured, but a few more general ones would be:
- Are ads in certain categories more likely to contain unwanted content (high-risk categories)?
- Are rules for certain categories confusing to your customer?
- Are some categories where the refusal rate is so low that moderation is less urgent (low-risk categories)?
How to track:
How to track the refusal rate per category will again depend on your moderation tool.
Why you should track this:
Use this metric to assign your moderation strategies more strategically. A category with very few refusals might be low-risk, where manual moderation could be substituted with automation at minimal risk.
If rule confusion is causing a high refusal rate, you will want to educate your customers better, change the ad submission process, or relax your requirements.
5. Reasons for reported content
Report reasons will give you indicators of multiple things:
- What type of unwanted content is slipping through the moderation measures you have in place (if you have any)?
- Which type of unwanted content are your users most bothered by?
- Is your site having issues with a specific type of content?
How to track:
Report reasons may be tracked through your customer support or moderation tool. There are two ways of collecting the data for this KPI: One is to let the customer decide the reason when reporting through a pre-determined drop-down menu. The other is to let the agents label it when handling the report.
You can obviously combine the two and let the agent change the reason if it is deemed inaccurate.
Note: One thing to consider when setting up the report function is to have report reasons rotate place to ensure that false data from users blindly picking the first reason has a limited impact.
Why you should track this:
The knowledge gained from report reasons can be used to adjust filters or your moderation processes. You can also use it to focus more resources on the type of content your user base is most vigilant about to ensure your moderation efforts are aligned with your users’ interests.
6. Flag percentage
Flag percentage looks at the number of flagged content vs. published content. This number should be low to indicate that you catch unwanted content before it goes live on your site for your users to see.
A high number here can either indicate that your moderators/automation is not doing a good enough job.
It could also indicate that your policies are not aligned with your users’ expectations. Users might be flagging content that is in line with your policies but that they find offensive for some reason. This can happen when your rules are not communicated or are too complicated.
How to track:
(Number of flagged ads) / (Number of published ads) = Flag percentage.
Why you should track this:
To fully understand this metric, you must combine it with the insights gained from the report.
Your first step should be to find out whether the reports align with your site policies and if you might have to align your policies with your users’ expectations or find a way to educate your users better.
Suppose an underperforming moderation team or automation solution causes a high number. In that case, it is time to invest in training; in the case of automation, you should adjust and update filters. Make sure your moderators have access to their performance stats so they can take ownership of their improvement journeys.
Summary and key takeaways
6 KPIs might seem like too few data points, and your unique business may require more metrics than these. However, 6 relevant and actionable metrics you actively use to improve continuously are much better than the 100 KPIs you track because you can. In the end, focusing on the KPIs that help you reach your goals will make you successful, and setting up tracking for these 6 data points is a good start.
There is plenty of more good stuff on our blog
See allSee all articlesThis is Besedo
Global, full-service leader in content moderation
We provide automated and manual moderation for online marketplaces, online dating, sharing economy, gaming, communities and social media.
Share this article
