Marketplace AI

6 Actionable Metrics That Will Improve Your Content Moderation

Mar 17 2016

In previous blog posts, we have talked about how to prioritize your moderation efforts and which methods to use. But even though your moderation process is now in full swing, you shouldn’t put your feet up. Now it is time to analyze your efforts and optimize based on the incoming data. It can, however, be hard to decide which KPIs to focus on in the stats jungle and it is easy to get overwhelmed.

Fortunately, through our years of moderation we have learned enough to make a list of the most important data to track. The list is not exhaustive but covers the top KPIs that you need to focus on when doing online content moderation.

Before we get started it is important to note that this article is covering KPIs relevant to improving the moderation process and your product. We will as such not cover stats related to agent and team performance like number of ads reviewed per hour/per agent.

How you can automate up to 80% of your content moderation

Get exclusive insights and guidance on how to tackle your content moderation challenges with Implio. Moderation expert and Besedo customer success manager, Alessio Coco, will take you through our AI capabilities, automated filters and manual interface in the tool.

Webinar - Optimize ROI of moderation

Watch the Webinar

Actionable and Accessible Metrics

The first lesson in the school of metrics is focus. It’s easy to get lost in the space of analytics and create a cockpit of graphs just because you can. But what are you going to use it for? Metrics should be actionable and foster improvements. But it’s not just for you. Everyone in the organization needs access to the metrics they require to take informed decisions and contribute towards company goals.

1. Publish Time

Measuring publish time will tell you how long a seller has to wait for his or her content to be visible to potential buyers.

You will want this number to be as low as possible as that means a better experience for sellers. With a long publish time you risk an increase in duplicate postings, customers leaving for the competition and an increase in customer support tickets.

How to track:

(Time of content going live on your site) – (Time user submits content) = publish time

Why you should track this:

Automation, post-moderation and machine-learning will usually help you keep this number low, while you will have a longer time to site if you are relying solely on manual moderation.

Make sure you have the right systems in place and fine-tune your automation tool and processes to minimize the manual queues.

2. Average Lifetime of Reviewed Scams

With this metric you are tracking how long scams are live on your site on an average.

Keeping an eye on this metric will give you a good indicator of how good your team is at quickly identifying and removing scams from your site. The higher this number is, the bigger the risk of your users getting scammed. In a sense this metric is tightly connected to monitoring user experience.

How to track:

(Time of Refusal) – (Time of Publishing) = Lifetime of reviewed scam.

Why you should track this:

Use this metric to identify the categories that are most likely to containg scams, and based on this create a post-moderation action plan.

Learn how to moderate without censoring

Why moderating content without censoring users demands consistent, transparent policies.


3. Refusal Rate (and reason)

The refusal rate is a great indicator of the seller experience. It should be as low as possible while maintaining the level of quality you want for your site.

You also need to track refusal reasons so that you can provide your users with a clear explanation which helps educate them and over time improves the quality of their listings.

A low refusal rate means that it’s easy for a seller to get their ad published on your site.

A high refusal rate on the other hand reflects a negative user experience for sellers as getting their content out to potential buyers is not a smooth process.

How to track:

The way you track this number will depend on the tool you are using for moderation. A good moderation tool will track and store info every time you refuse an ad and give you data like the most common rejection reasons.

Why you should track this:

If you have a high refusal rate you should look at how seller needs and your site policies can be better aligned without jeopardizing the quality of your site. Refusal reasons will help you dig into the underlying issue and better understand why content is being rejected.

Keep in mind though that a very low refusal rate could indicate an issue with your moderation rules, but it could also mean that your moderators or automation solution is letting too much through.

4. Refusal Rate (and Reason) per Category

There are many interesting conclusions to be drawn from refusal rates per category. Many will be very specific to your site and the way it is structured, but a few more general ones would be:

How to track:

How to go about tracking refusal rate per category is again going to depend on your moderation tool.

Why you should track this:

Use this metric to assign your moderation strategies more strategically. A category with very few refusals might be a low risk category where manual moderation could be substituted with automation at a minimal risk.

If rule confusion is causing a high refusal rate, you will want to look at either educating your customers better, changing the ad submission process or relaxing your requirements.

5. Reasons for Reported Content

Report reasons will give you indicators of multiple things:

How to track:

Report reasons may be tracked through your customer support tool or your moderation tool. There are two ways of collecting the data for this kpi: One is to let the customer decide the reason when reporting through a pre-determined drop-down menu. The other is to let the agents label it when handling the report. You can obviously combine the two and let the agent change the reason if it is deemed inaccurate.

NOTE: One thing to consider when setting up the report function is to have report reasons rotate place in order to ensure that false data from user blindly picking the first reason has limited impact.

Why you should track this:

The knowledge gained from report reasons can be used to adjust filters or your moderation processes. You can also use it to focus more resources on the type of content your user base is most vigilant about to ensure your moderation efforts are aligned with the interests of your users.

6. Flag Percentage

Flag percentage looks at the number of flagged content vs. published content. This number should be low to indicate that you catch unwanted content before it goes live on your site for your users to see.

A high number here can either indicate that your moderators/automation is not doing a good enough job.

It could also indicate that your policies are not aligned with the expectations of your users. Users might be flagging content that is actually in line with your policies, but that they for some reason find offensive. This can happen when your rules are not clearly communicated or when they are too complicated.

How to track:

(Number of flagged ads) / (Number of published ads) = Flag percentage.

Why you should track this:

In order to fully understand this metric you need to combine it with the insights gained from report reasons.

Your first step should be to find out whether the reports are in line with your site policies and if you might either have to align your policies with your users’ expectations or find a way to educate your users better.

If a high number is caused by an underperforming moderation team or automation solution, it is time to invest in training, and in the case of automation you should adjust and update filters. Make sure your moderators have access to their own performance stats so they can take ownership of their own improvement journeys.

6 KPIs might seem like too few data points and your unique business may require more metrics than these. However, 6 relevant and actionably metrics that you actively use to continuously improve, is much better than 100 KPIs that you just track because you can. In the end, a focus on the KPIs that actually help you reach your goals is what will make you successful, and setting up tracking for these 6 data points is a good start.

This is Besedo

Global, full-service leader in content moderation

We provide automated and manual moderation for online marketplaces, online dating, sharing economy, gaming, communities and social media.

Form background