Marketplace AI Custom solution

AI Moderation: Get to Know the Basic Concepts

Here at the end of 2016, no one is in any doubt that artificial intelligence (AI) and automation are key components in delivering quality services efficiently.

But despite the hype around machine learning (ML) and AI, the technologies are still very young. This means that while everyone agrees they are invaluable tools, most people don’t fully understand all the terms and concepts related to ML and AI. This is especially true when looking at specialized use cases like AI moderation.

As such, we figured we would shed a bit of light on some of the important concepts to know and understand when it comes to machine learning when applied to moderating content.

Automation Rate

Automation rate is the percentage of your total incoming ad volume that can be automated. It’s closely related to your accuracy levels (the lower accuracy you demand, the higher automation rate you can have). If out of 100 documents the AI feels it can take a decision on 99, then you have an automation rate of 99%. The remaining 1 document will be sent to manual moderation.

Accuracy Level

The accuracy level of your AI measures how often it takes the correct decision when moderating. If out of 100 jobs, the AI takes the right decision on 95, then your accuracy level is 95%. The accuracy level you have for manual moderation can be a good indicator of what you should aim for with automation. With automation, it’s possible to have very high accuracy levels, but the higher demands you have for accuracy, the lower your automation rate will be and more content will be funneled to manual moderation.

Confidence Markers

We use the confidence markers as a way to find the optimal balance between accuracy level and automation rate. Let’s say the AI has to take a decision between rejecting or approving a piece of content. Both options will be assigned a probability score between 0 and 1. If the scores are far apart the confidence will be high, but if the scores are very close we define the confidence as low. For example, a piece of content with 0.15 probability to reject and 0.85 to approve, would have high confidence because the numbers are far apart, but if both scores were 0.5 the confidence would be low since the scores are tied. The same would be the case for 0.45 reject and 0.55 approve. With scores this close it is hard to say with confidence that one option is the correct one.

In order for the AI to understand what content can be automated you have to define a confidence threshold.  Any result with a confidence value smaller than the threshold is considered uncertain and sent for manual review. For each model we create, we will define a unique threshold.

Model Building

While machine learning as the word implies, will learn on its own, talented engineers/data scientists need to build the model that it operates from. For each problem you want the AI to solve, you need to build a unique machine learning model. For example: If you want to look for scams you need one model, but if you want to look for swearing as well and you want to know the specific reason something was rejected (scam or swearing) you will need to build a separate model for both scams and swearing.

Part of building a model is training it using big data sets from which it learns patterns. There are different ways of approaching this, but here we will cover generic and tailored models.

A generic model for content moderation will have been trained on a data set (often synthetic) with the aim to be broadly applicable to many different sites, but only solving very generic and basic issues.

Tailored models, on the other hand, are trained on datasets unique to a specific site. Here the model will be sensitive to the particular rules and processes of that distinct site. Unless the rules and language of another site were 100% identical it is unlikely the tailored model could be applied to another site, however for the site the model was developed for it deliver much better accuracy levels and allow for a way higher automation rate than a generic model can ever hope to achieve.

Learn how to moderate without censoring

Why moderating content without censoring users demands consistent, transparent policies.


Retraining Models

Retraining an algorithm means overwriting the current decision patterns by using new and better data to achieve the desired ones. This is often done when site policies are changed. An example could be that now that cannabis is legal in several states in the US, sites in those states might want to move from rejecting to accepting cannabis. To allow this, the machine learning algorithm needs to be fed new data where the decision on listings related to cannabis is “accept” rather than “reject”.

NOK Recall

Your NOK(not OK) recall rate is showing how much unwanted content that is correctly identified. A high NOK recall rate means that the AI model correctly blocks the majority of unwanted content.

Let’s say that we have 100 listings, 90 of which are OK and 10 that are Not OK. The AI model managed to correctly identify 5 Not OK listings. This means that 5 Not OK listings were falsely approved by the AI model. In this scenario, the recall rate would be 50% (5/5+5).

NOK Precision

NOK precision measures how precise the model is when refusing bad content. A high NOK precision means that the model will rarely refuse good content. For example, if there are 100 content pieces and the machine determine 10 of them to be scams, but only 8 of them were actually scams, then the algorithm has a precision of 80% (8/8+2).

These are just a few of the concepts tied to machine learning when applied to content moderation, but when you know and understand these, then you have a pretty good foundation for understanding what kind of AI moderation setup you want and what it might entail.

If you have more questions or if you are just curious about knowing more about machine learning for content moderation, you are welcome to reach out. We are happy to expand on the topic!

This is Besedo

Global, full-service leader in content moderation

We provide automated and manual moderation for online marketplaces, online dating, sharing economy, gaming, communities and social media.

Form background