So how do you calculate the price of custom content moderation with AI? At Besedo we look at it from a number of angles: Volumes, complexity of moderation actions needed and languages. We build and tailor something bespoke for each client that we do not share with anyone else.

If you work in content moderation for a classifieds site or an online marketplace, you’ll have probably heard lots of talk about machine learning and tailored AI. No doubt you’ll have wondered about its features, cost, and value.

As a content moderation service provider we’d gladly shout out positive things about tailored AI all day long (!), but we also wanted to give some background into why we believe it works, to shed some light on alternatives, and give some insight into costs.

Cost and ROI comparison between tailored AI and generic ML models

In a nutshell, tailored AI is a machine learning algorithm that’s created using clients structured and labeled data. By inputting this data, you can teach your AI to learn very specific moderation patterns. It can handle complexity, is self-learning, will give you a much higher accuracy rate, and higher automation levels. It’s much more meaningful and offers better results than generic alternatives, which are less reliable and error-prone.

At Besedo for instance with tailored models, we have accomplished automation rates of up to 90% with an accuracy level of up to 99% accuracy. That would not be possible using generic one size fits all models.

Generic AI, while useful when moderating something fixed – like language – can’t handle specific challenges, as it doesn’t learn in the same way as tailored AI. Say you want to set moderation criteria for profile pictures on a dating site. There are lots of things you need to do: ensure users are over 18, censor nudity, make sure there’s a face visible, that no weapons are shown, and that each picture is good quality. These are the requirements of a specific platform. Using several different generic AI models to try and moderate these criteria won’t work as well as a single tailored AI can. But you could always build your own model, right?

While it might seem simple and less expensive to build your own tailored AI, it often ends up as a costly distraction. Companies can spend years pouring in resources into a setup and still never get it exactly right. Creating powerful machine learning moderation models isn’t just a matter of putting a couple of developers on the task. It requires data scientists and semantic experts to make sure the AI keeps learning and performing better. Considering the ongoing cost and complexity, why create your own content moderation algorithm when there are expert companies offering tailormade solutions? Unless you are a huge company with very specific needs you wouldn’t develop your own helpdesk or customer service tool. Then why go that route with content moderation?

The price of AI moderation

So how do you calculate the price of a tailored AI? At Besedo we look at it from a number of angles: Volumes, complexity of moderation actions needed and languages. We build something bespoke for each client that we do not share with anyone else. There is a setup fee to create an AI model for the client, which involves learning from available client data to build a specific model; monthly moderation fees, which we base on the projected volumes (starting at a minimum of 200,000 moderated items per month). The monthly moderation fee covers hosting, software licenses, and maintenance. Apart from this, there is a monthly professional fee, which includes updates, new performance improvements and updates of rules to ensure that your automation rate and performance is always improving. Finally, we have a fixed support fee that gives you 24/7 technical support.

A lot goes into creating a tailored AI, but it is still a lot more cost-effective than manual moderation, especially over time: and is far less expensive than developing your own moderation model. You can’t really compare a tailored approach to a generic one at all since that would be like comparing a chisel to a sledgehammer. You will not get the accuracy you need and will end up wasting money – as well as time and effort.

All factors considered, by our calculations companies that choose tailored AI can save anywhere between 50%-90% on manual moderation pricing alone.  Surely that’s a worthwhile investment of time and money?

Still not convinced? Get in touch!

This is Besedo

Global, full-service leader in content moderation

We provide automated and manual moderation for online marketplaces, online dating, sharing economy, gaming, communities and social media.

Form background

Every forward-looking business is considering how to implement automation strategies throughout their different departments these days. It is not surprising that moderation, which traditionally required a lot of manual work has been one of the first areas to get turned upside down by the emerging automation trend.

But smart business owners also realize that for automation to be efficient, it has to be applied right. Even though there are huge cost-savings to be had by adding automation to your moderation process, you will only reap the monetary benefits if you chose the right approach.

Bill Gates famously said: 

“The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency.”

We agree with Gates and will also add:

“The third rule is that the wrong automation type applied to an efficient or inefficient operation will be very costly both to resources and potentially to brand image as well.”

Why? Because each type of automation has it’s own strengths and weaknesses, which will benefit businesses differently depending on volumes, budget and growth stage.

We will go through these characteristics now.

Filters

Filters were part of the first wave of automation efforts applied by site owners desperate to manage content and keep users safe.
In 2017, filters might seem like an inflexible and rigid approach to moderation, but they are still a very valuable tool in the moderation toolkit both as a cheap entry point and as an additional level working alongside AI to keep your site clean.

How Do Automation Filters Work and Where Are They Most Useful?

If applied and maintained correctly filters can be quite powerful on their own. In Implio, filters or rules as we call them can be set up to be pretty comprehensive. You can design them so they look at price, IP, description or any other data field available. On top of this our tool offers the option to create lists, allowing for easy updates without messing about with the rule structure. Filter automation is great for cases where you want to target very specific information like user data.

Filters can add a relatively inexpensive first layer of defence against scammers and other undesirable users. Filters can be created to target any unwanted content that follows a recurring pattern, which can be precisely described and thus formulated as a rule.

Another benefit with filters is that they are very easy to change and set up. You don’t need huge datasets to train the filters, you just enter a new rule based on your needs and you are ready to go.

The drawback with filters is of course that you continuously have to update them which means you are often in an open arms race with scammers and other users, intent on misusing your site.

Generic Machine Learning Models

Most offerings for automation on the market today, which aren’t filters, fall into this category. Generic machine learning models are algorithms created from general data and targeted at issues that are common to a wide variety of sites.

How Do Generic Machine Learning Models Work and Where Are They Most Useful?

The difference between generic and tailor made machine learning models, is primarily in the datasets used to train the algorithms. Whereas a tailor-made solution will be built from your specific data, a generic model is taught from an artificially created dataset aimed at covering all variables of the problem it is trying to solve.

A generic model meant to deal with swear words would as such be presented with a dataset containing 50% documents full of swear words and 50% which were clean and okay.

The main problem with generic models is, as you can probably imagine, that they have a tendency to be very broad, resulting in low accuracy rates.

One of the challenges in building datasets to train generic machine learning models is that there is no general agreement on what constitutes for instance bad language. Different communities will have different levels of sensitivity towards profanities and sometimes whether a word is bad or not can even be a matter of context. With a generic model you will get an algorithm tuned to what is generally accepted as swearwords and that might not fit your site.

If your community isn’t largely specialised, generic machine learning models can however be good for getting the worst content off your site quickly and at a very affordable price. Just make sure that you are aware of the limitations and comfortable knowing that some bad content will slip through and some genuine posts will get caught. Generic machine learning models are bound to create a lot more false positives than tailor-made algorithms and customized filters.

Tailor-made Machine Learning Models

As opposed to generic models, tailor-made machine learning algorithms are trained using data that is specific to your site.
This allows the model to operate at a very high accuracy level and if we take the example of swearing, you avoid false positives from words that are generally labelled as profanity, but which are acceptable in the context of your community.

To illustrate the point let’s say that you are running a site for medical professionals where they can buy and sell equipment while also discussing their trade and exchange knowledge. On such a site, using words for genitalia wouldn’t be out of place or considered profane. Posts containing those words would however very likely get caught by a generic model. A tailor-made algorithm would know that those posts are generally allowed through, whereas those containing the f-word are still rejected as profane.

How Do Tailor-Made Machine Learning Models Work and Where Are They Most Useful?

When we create tailor-made machine learning models at Besedo we work closely together with our clients to ensure the best possible outcome. The bigger the dataset our client has available, the more accurate a solution we can create for them. Over time with tailor-made solutions we can reach an accuracy of 99% with an automation rate of 90%, something completely unrivalled by either generic models or filter automation.

A tailor-made machine learning solution is able to reach very high accuracy and automation levels, but it does require quite big data volumes. If you run a small site, and are just starting out, you may not have enough valid data to efficiently train the algorithms. In such cases your better choice might be filters or a generic machine learning solution.

Learn how to moderate without censoring

Why moderating content without censoring users demands consistent, transparent policies.

Untitled(Required)

Choosing the Automation Setup That Is Right for You

Now that we have been through the different types of automation solutions, it is time to find out which one fits your needs best.

At this point it probably comes as no surprise that it really comes down to a question of volumes, accuracy and budget.

In the table below we are illustrating elements that impacts the choice of automation type. Based on this matrix you can find a good guideline for which automation type will work best for your specific business.

When Are Tailor-Made Machine Learning Models the Best Option?

If your monthly volumes are 100k or more, then a tailor-made machine learning model is the way to go. You have sufficient volumes so it is possible to create very exact models.

Furthermore, if you are currently moderating your content manually, you will make your money back very fast even with an investment in a tailor-made solution. Tailor-made machine learning models will allow you to manage your content with high accuracy.

When Is Filter Automation the Best Option?

If your volumes are less than 100k and avoiding false positives is very important to you, then filter automation makes sense for your business. You will be able to tweak the wordlists and rules until they match your site rules and you can even apply a level of manual moderation to review items that get caught in the filter to ensure a higher accuracy level.

When Are Generic Machine Learning Models the Best Option?

If your volumes are less than 100k items per month and you can live with an accuracy as low as 75%, generic machine learning models might be a good solution. In the event that you have to choose between no moderation and a generic algorithm, it is generally better to apply some level of moderation as long as the solution doesn’t catch too much content published by genuine users.

It might even be that the best solution for you is a combination of the three in a tailored setup to fit the specific and varied challenges your business faces.

Are you still in doubt about which automation type fits your business best? One of our solution designers will be happy to analyze your business needs with you, to determine what option will benefit you the most.

This is Besedo

Global, full-service leader in content moderation

We provide automated and manual moderation for online marketplaces, online dating, sharing economy, gaming, communities and social media.

Form background

Do you have a machine learning (ML) solution developed in-house, or do you consider building one? We have seen initiatives like this fail time and again. It usually ends up with frustrated quality managers in conflict with developers over prioritization and quality definitions.

This blog post shows you why you should focus your development resources on your core product and consider sourcing a custom made moderation solution.

Assessing Perceived Benefits

Stop! Before you read on, ask yourself one question. Why are you even developing a tool in-house? What are you trying to achieve? Do you believe it to be more cost-efficient? More flexible? Better tailored to your needs? If your answer is yes, you are about to be disappointed. Our extensive experience working with and advising clients on moderation has proven that time and time again the in-house benefits fail to deliver against all expectations. Let’s look at those perceived benefits one by one.

Cost-efficiency

If you think that finding competent developers is difficult and expensive, then wait until you start looking for people with ML capabilities. To create a state of the art Artificial Intelligence (AI) moderation engine you’ll need a team of data scientists and analysts. ML has a lot more to do with statistics than coding and guys with this type of competency are hard to find and come at a premium.

Were you thinking of utilizing your current in-house dev-team? Cross-utilization is a nice dream, but in reality you will find that skilled programmers are not necessarily great machine learning model makers. Even if your current team has the capabilities of creating your ML models, you will find that supporting tools like these tend to get down-prioritized when new features and bugs need to be fixed on your core product. The complaint we hear from quality managers is repeatedly that they end up with lackluster tools that don’t get updated often enough.

With a subpar machine learning moderation solution, you risk more false positives, leading to unhappy customers and more unwanted content slipping through upping the user risk.

If you buy a ready-made product that’s specialized for the needs of your industry, you will in-directly leverage all the experience and knowledge that the development team has acquired throughout the years of building this type of tool. You’re basically buying 100% dedication, competency and ongoing training on new algorithms and better models.

Flexibility

Do you believe that an in-house solution can adapt faster when you need to make changes to the models? Well, probably not! A professional solutions provider will be able to customize the machine learning model and accommodate your needs at least as quickly, as you could if you had your own dedicated in-house team. Your solution partner probably has a better understanding of moderation needs and knows how important speed and flexibility are to keep up the quality of your content. They are dedicated to moderation and won’t risk being tied up with other tasks, which allows for super-quick updates and model tweaks when needed.

Tailor Made

You know your industry and product like the backside of your hand, we won’t argue with that. But honestly spoken, can you call yourself an expert in automated content moderation? An experienced solutions provider has the knowledge and can draw on experience from working with sites of all sizes and growth stages and with content ranging from marketplace ads to dating profiles. At Besedo, for example, we have dedicated the last 15 years’ to spotting and dealing with new and upcoming moderation challenges.

Learn how to moderate without censoring

Why moderating content without censoring users demands consistent, transparent policies.

Untitled(Required)

All-in-one Solution

If we at Besedo were asked to select the one thing that we’re most excited about when it comes to our Machine Learning solution for moderation, it’s the benefit of being part of a bigger moderation ecosystem.

We have combined filter moderation, machine learning and a manual moderation panel in one tool. This means you only ever have to integrate once. There’s no risk of one tool breaking if another is updated and most importantly it allows for a seamless feedback loop from manual moderation to continuously improve the machine learning model.

If you consider building a similar solution in-house, make sure to investigate how long it would it take and if those resources couldn’t be used better improving your core product.

Specialized Machine Learning Moderation Solutions vs. Generic AI Models

Some companies we have spoken to about ML moderation have mentioned that they are trying generic AI solutions to see if they can solve their content challenges. And while we understand the lure, reality is that Machine learning itself isn’t necessarily going to help you. In the end, it must solve the particular challenges your business has. Generic ML solutions are, as the name indicates, generic and almost never live up to the high standards marketplaces need to protect their content.

If we hear that a potential client is testing out generic solutions, we always ask them to get back to us on what results they are getting. So far the unanimous feedback has been that these kind of solutions are not working for them. It is simply too generic.

Let’s take a quick look at some of the issues with generic models.

Generic

As the name states a generic solution will be built around a generic dataset. That might work for very basic challenges, but even something as simple as catching obscene language will be hard with a generic model.  The definition of what profanity is can vary wildly depending on target group and the purpose of the site.

The fact that it is generic means your accuracy level will be lower and your false positives much higher. The end result is that you will still need to do a lot of manual moderation to keep your content quality high and your users safe.

A specialized moderation solution, on the other hand, will be tailor-made using your data, meaning the AI will adhere to the unique rules and policies you have set up for your site.

Inflexible

A generic solution is static and won’t help you quickly adapt to changes in the world or in user behavior. It will also often have a very specific use and it is unlikely it will be able to solve all your moderation challenges.

A Car vs an Engine

One of the main issues with generic models is that they can rarely be applied without at least some tweaking. In the end, you will likely end up dedicating a lot more resources than you originally anticipated just getting the generic machine learning model to work reasonably.

When you’re looking for a car, you buy a car, not an engine to build a car. In the end, a tailor-made machine learning solution often ends up being more cost-effective, the time to market is much shorter and it has a lower maintenance cost.

So you see? Either way you look at it, if you are serious about moderation (and when you are trying to disrupt the market you should be), then you really should go with a tailor-made machine learning solution created by experts in the field.

This is Besedo

Global, full-service leader in content moderation

We provide automated and manual moderation for online marketplaces, online dating, sharing economy, gaming, communities and social media.

Form background