Marketplace AI

6 Actionable Metrics To Improve Your Content Moderation


In previous blog posts, we discussed prioritizing your content moderation efforts and which methods to use. But even though your content moderation process is now in full swing, you shouldn’t put your feet up. Now it is time to analyze your efforts and improve your content moderation.

It can, however, be hard to decide which KPIs to focus on in the stats jungle, and it is easy to get overwhelmed.

Fortunately, through our years of content moderation, we have learned enough to list the most important data to track. The list is not exhaustive but covers the top KPIs you must focus on when doing online content moderation.

Before we get started, it is important to note that this article covers KPIs relevant to improving the moderation process and your product. We will not cover stats related to the agent and team performance, like the number of ads reviewed per hour/per agent.

How to automate up to 80% of your content moderation

Get exclusive insights and guidance on tackling your content moderation challenges with Implio. Moderation expert and Besedo customer success manager Alessio Coco will take you through our AI capabilities, automated filters, and manual interface in the tool.

Watch the webinar

Webinar - Optimize ROI of moderation

Actionable and accessible metrics

The first lesson in the school of metrics is focus. It’s easy to get lost in the space of analytics and create a cockpit of graphs just because you can. But what are you going to use it for? Metrics should be actionable and foster improvements. But it’s not just for you. Everyone in the organization needs access to the metrics required to make informed decisions and contribute toward company goals.

1. Publish time

Measuring publish time will tell you how long a seller has to wait for his or her content to be visible to potential buyers.

You will want this number to be as low as possible, which means a better seller experience. With a long publish time, you risk an increase in duplicate postings, customers leaving for the competition, and an increase in customer support tickets.

How to track:

(Time of content going live on your site) – (Time user submits content) = publish time

Why you should track this:

Automation, post-moderation, and machine learning will usually help you keep this number low, while you will have a longer time to site if you are relying solely on manual moderation.

Ensure you have the right systems in place and fine-tune your automation tool and processes to minimize the manual queues.

2. Average lifetime of reviewed scams

With this metric, you are tracking how long, on average, scams are live on your site.

Keeping an eye on this metric will give you a good indicator of how good your team is at quickly identifying and removing scams from your site. The higher this number is, the bigger the risk of your users getting scammed. In a sense, this metric is tightly connected to monitoring user experience.

How to track:

(Time of Refusal) – (Time of Publishing) = Lifetime of reviewed scam.

Why you should track this:

Use this metric to identify the categories that are most likely to contain scams, and based on this, create a post-moderation action plan.

3. Refusal rate (and reason)

The refusal rate is a great indicator of the seller’s experience. It should be as low as possible while maintaining the level of quality you want for your site.

You also need to track refusal reasons so that you can provide your users with a clear explanation which helps educate them and, over time, improves the quality of their listings.

A low refusal rate means that it’s easy for a seller to get their ad published on your site.

On the other hand, a high refusal rate reflects a negative user experience for sellers, as getting their content out to potential buyers is not a smooth process.

How to track:

How you track this number will depend on the tool you use for moderation. A good moderation tool will track and store info every time you refuse an ad and give you data like the most common rejection reasons.

Why you should track this:

If you have a high refusal rate, you should look at how seller needs and your site policies can be better aligned without jeopardizing the quality of your site. Refusal reasons will help you dig into the underlying issue and better understand why content is being rejected.

Remember that a very low refusal rate could indicate an issue with your moderation rules, but it could also mean that your moderators or automation solution is letting too much through.

4. Refusal rate (and reason) per category

There are many interesting conclusions to be drawn from refusal rates per category. Many will be very specific to your site and the way it is structured, but a few more general ones would be:

How to track:

How to track the refusal rate per category will again depend on your moderation tool.

Why you should track this:

Use this metric to assign your moderation strategies more strategically. A category with very few refusals might be low-risk, where manual moderation could be substituted with automation at minimal risk.

If rule confusion is causing a high refusal rate, you will want to educate your customers better, change the ad submission process, or relax your requirements.

5. Reasons for reported content

Report reasons will give you indicators of multiple things:

How to track:

Report reasons may be tracked through your customer support or moderation tool. There are two ways of collecting the data for this KPI: One is to let the customer decide the reason when reporting through a pre-determined drop-down menu. The other is to let the agents label it when handling the report.

You can obviously combine the two and let the agent change the reason if it is deemed inaccurate.

Note: One thing to consider when setting up the report function is to have report reasons rotate place to ensure that false data from users blindly picking the first reason has a limited impact.

Why you should track this:

The knowledge gained from report reasons can be used to adjust filters or your moderation processes. You can also use it to focus more resources on the type of content your user base is most vigilant about to ensure your moderation efforts are aligned with your users’ interests.

6. Flag percentage

Flag percentage looks at the number of flagged content vs. published content. This number should be low to indicate that you catch unwanted content before it goes live on your site for your users to see.

A high number here can either indicate that your moderators/automation is not doing a good enough job.

It could also indicate that your policies are not aligned with your users’ expectations. Users might be flagging content that is in line with your policies but that they find offensive for some reason. This can happen when your rules are not clearly communicated or when they are too complicated.

How to track:

(Number of flagged ads) / (Number of published ads) = Flag percentage.

Why you should track this:

To fully understand this metric, you must combine it with the insights gained from the report.

Your first step should be to find out whether the reports align with your site policies and if you might have to align your policies with your users’ expectations or find a way to educate your users better.

If a high number is caused by an underperforming moderation team or automation solution, it is time to invest in training, and in the case of automation, you should adjust and update filters. Make sure your moderators have access to their performance stats so they can take ownership of their improvement journeys.

Summary and key takeaways

6 KPIs might seem like too few data points, and your unique business may require more metrics than these. However, 6 relevant and actionable metrics you actively use to improve continuously are much better than the 100 KPIs you just track because you can. In the end, focusing on the KPIs that help you reach your goals will make you successful, and setting up tracking for these 6 data points is a good start.

This is Besedo

Global, full-service leader in content moderation

We provide automated and manual moderation for online marketplaces, online dating, sharing economy, gaming, communities and social media.

Form background