User safety is key for all online platforms, particularly when you’re dealing with vulnerable youngsters. Moderating can be challenging and getting the balance between censorship and safety right can be hard.

We sat down with industry veteran and founder of Friendbase; Deborah Lygonis, to discuss the experience she’s gained from developing and running a virtual world for teens.

Deborah Lygonis - Founder of Friendbase

Interviewer: Hi Deborah. Could you please give us a short introduction to yourself?

Deborah: My name is Deborah Lygonis and I am a serial entrepreneur. I have started and run several businesses over the years, mainly within the software and gaming sector, but also e-health and other tech. I love tech and I’m passionate about startups and entrepreneurship. I also work as a coach and mentor for entrepreneurs within what’s called the European Space Agency Business Incubator; The ESA BIC, and for a foundation called Entrepreneurs Without Borders.

So, we put together a mockup of an Android, IOS, Web version and put it out there to see if that was something that today’s young people would like.

Interviewer: Definitely! What’s in the future for Friendbase? Where are you in two years?

Deborah: Where are we now? We’re now raising funds, because what we’ve seen is that we have a very, very loyal member base and they are wanting to invite more of their friends. And I think that with very, very little work, we can get the platform on a really interesting growth path.

Friendbase is a virtual world for teens where they can chat, play games and also design their looks and spaces. Now we’re also moving towards Ed tech in the way that we’ll be introducing quizzes that are both for fun but also have learning elements in them.

Interviewer: That sounds awesome. What would you say is the main challenge when it comes to running cross-platform online community and specifically one that caters to teens?

Deborah: There are a lot of challenges with startups in general, but also, of course, running an online community. One challenge is when you have people that meet each other in the forms of Avatar and written chat and they have different personalities and different backgrounds that can cause them to clash. The thing is that when you write in a chat, the nuances in the language don’t come through as opposed to when you have a conversation face to face. It’s really very hard to judge, the small subtleties in language and that can lead to misunderstandings.

Add to that as well that there are lots of different nationalities online. That in itself can lead to misunderstandings because they don’t speak the same language.

What starts off as a friendly conversation can actually rapidly deteriorate and end up in a conflict just because of these misunderstandings. That is a challenge, but that’s a general challenge, I think, with written social interactions.

Interviewer: Just so we understand how Friendsbase work. Do you have one to one chat, one to many chats or group chats? How does it work?

Deborah: The setup is that we can have up to 20 avatars in one space. No more, because then it will get too cluttered on the small phone screens. So, you can have group chats. I mean, you see the avatars and then they have a text bubble as they write so that it can be several people in one conversation.

Interviewer: Do you have the opportunity for groups of friends to form and join the same kind of space together?

Deborah: Yes. Each member has its own space. They can also invite and open up their space for other friends.

Interviewer: And in that regard. What you often see in the real world with team dynamics is that there is a group of friends and there is the popular people in that group. And then one person who maybe is a little bit an outsider, who will at times be bullied by the rest of the group. Do you see people ganging up on each other sometimes?

Deborah: I haven’t seen groups of people ganging up on one individual. It’s more the other way around. There are individuals that are out to cause havoc and who are just online to be toxic.

Interviewer: That means that you have in general, you have a really nice and good user base. But then there’s the rotten fruits that come in from time to time.

Deborah: That is what it is like today. We are still fairly early stage, though, when it comes to the amount of users. So I would expect this to change over time. And this is something that we’re prepared for. We added safety tools at a really early stage to be able to learn how to handle issues like this and also how to moderate the platform when incidents occur. So, I think that even though that we don’t have that type of ganging up on each other at the moment, I would expect that to happen in the future.

Interviewer: But it sounds like you’re prepared for it. Now you’ve made a really nice segue into my next question; What is the main motivation challenges you experienced running Friendbase? What are the main challenges right now and what do you expect you will have to handle later on?

Deborah: I think that a challenge in itself for all social platforms is to set the bar on what is acceptable and not.
Our target group are mid teens and up. So we don’t expect young children to be on Friendsbase. We feel that if we made a social world for young children, then we’d need to have a completely different set of regulations, more controlled regulations, rather than when it is teenagers and upwards.
However, that demographic is also very vulnerable. So, of course, there has to be some sort of measurement in place. The challenge is to determine, at what level do you want to put the safety bar and also how can you tell the difference between what is banter between friends and when it sort of flips over to actually be toxic or bullying? That’s something that is really, really hard to differ between. And I think that if you work with chat filters, then you have to have some sort of additional reporting system for when maybe the filters don’t manage this challenge. The filter is only a filter and can’t determine between the two. So that’s one challenge. It’s also complex to enforce the rules that are in place to protect the users without being perceived as controlling or patronizing.
At the moment, we also have a challenge in that we have users that come back solely for the purpose to cause havoc and create a toxic environment. We track them down and we ban their accounts, but it’s a continuous process.
That is something that should it escalate over time it will become increasingly time consuming. That’s why it’s really, really important for us to have tools in place so that it doesn’t have to be moderated manually. That will just take too much resource and time.
Of course, you have the even darker side of the internet; sexual predators that are out to groom vulnerable youngsters and to get them to maybe move over to a different platform where they can be used in a way that is extremely negative.
That’s something that is difficult to handle. But today, thanks to artificial intelligence and again, amazing toolsets out there. There are attempts to look at speech patterns and try and identify that sort of behavior. And there it’s also really great to have your own tool sets where the user can actually report someone if they feel threatened or if they feel that someone’s really creepy.

Interviewer: When you have returning users who have made it their goal to attack the platform, in a malicious way, do you see that it’s the same people returning based on their IP or the way that they talk?

Deborah: It’s not always possible to see it based on their IP because they use different ways of logging in. However, given their behavior, we can quickly identify them. And we have a group of ambassadors as well online on Friendbase that help us. On top of that we have a chat filter which can red flag certain behavior. So that helps as well.

There are a group that come back over and over again and for some mysterious reason they always use the same username. So they’re not that hard to identify. That group is actually easier to control than a group which has a different motive on why they are online and why they are trying to target youngsters. The toxic ones that are just there because they think it’s fun to behave badly. It’s easy to find them and close down their accounts.

Interviewer: We already touched upon this, but what would you say is the hardest moderation challenge to solve for you right now?

Deborah: The hardest moderation challenge to solve is, of course, finding the people who are deliberately out to target lonely youngsters that hunger for social contact. The whole grooming issue online is a problem. We are constantly trying to find new toolsets and encourage our users to contact us if there’s something that doesn’t feel right. So grooming is something that we’re very, very much aware of. If we happen to shut down someone’s account by mistake for a couple of hours, they’re most welcome to come to us and ask why. But we’d rather be safe than sorry when it comes to this kind of behavior. However, it is hard to track because it can be so very, very subtle in the beginning.

Interviewer: Friendsbase has been around for a while now. Are there any challenges that has changed or increased in occurrence over the years? And if yes. How?

Deborah: Actually, not really. I think the difference is in our own behavior as we are so much more aware of how we can solve different problems.

Bullying has been around for years. Free Internet as well. Sexual harassment of youngsters and between adults, of course, has also been around for years. It’s nothing new. I mean, the Internet is a fantastic place to be. It democratizes learning. You have access to the world and knowledge and entertainment.
It’s amazing.
But there is a dark side to it. From a bullying perspective you have the fact that previously, if you were bullied at school, you could go home or you could go to your social group somewhere else and you would have somewhere where you would feel safe.

When it’s online, it’s 24/7.

And it is relentless when it comes to the whole, child abuse part. Of course, it existed before as well. But now with the Internet, perpetrators can find groups that have the same desires as themselves and somehow together they can convince themselves as a group that it’s more acceptable. Which is awful. So that is the bad part of the net.

So, when you ask: Have the challenges changed or increased since we started Friendbase? No, not really. But what has changed is the attitude of how important it is to actually address these issues. When we started the company in 2013. We didn’t really talk that much about safety tools. I mean, we talked about should we have whitelist or a blacklist, the words. It was more on that level. But today most social platforms, they have moderation, they have toolsets, they have guidelines and policies and so forth.

So, I think that we who work with online communities as a whole have evolved a lot over the past years.

Interviewer: Yeah, I would say today in 2020, you probably wouldn’t be able to launch a social community or platform without launching with some sort of moderation tools and well-defined guidelines.

Deborah: I think you’re right. Several years ago, I did the pitch where we were talking about online safety and tools of moderation and were completely slaughtered. What we were told was that being good online or this whole be cool to be kind is going to stop our growth. It’s much better to let it all run rampant and then it will grow much faster. I don’t think anyone would say something like that today. So that’s a huge shift in mindset. Which is great. We welcome it.

Interviewer: That’s a fantastic story. You’ve been in this industry so long; you’ve seen this change. I find it fascinating that just seven years ago when you said I want to protect my users, people laughed at you. And now people would laugh at you if you said, I’m gonna go live without it.

Deborah: I know. Can you imagine going on stage today saying that I don’t care about safety? I mean, people would be so shocked.

Interviewer: You said before when we talked about the main challenges if you experienced growth, you’d need to change your approach to moderation and automate more in order to just keep up?

Deborah: Yes, definitely. We try and stay on top of what toolsets are out there.

We build in our own functionality, such as muting users. So, if someone is harassing you, you can mute them so that you can’t see what they’re writing. Small changes like that, we can do ourselves, which will be helpful.

Something I’d like to see more and that we’ve actually designed a research project around is to not only detect and ban bad behavior, but to encourage good behavior.
Because that in itself will also create a more positive environment.

That’s something that we’re really excited about, to work with people that are experts within gamification and natural language processing to see how can we create tool sets where we can encourage good behavior and see what we can do. Maybe we can start deflecting a conversation that is obviously on its way to going seriously wrong. It could be so simple as a small time delay when somebody writes something really toxic with a pop up saying: “Do you really want to say this?”. To just make someone think once more.

This is something that we’re looking into. It’s super interesting. And I hear there’s a couple of companies just the last few months that are also talking about creating tool sets for something like this. So, I think it’s going to be a really, really interesting development over the coming years.

Learn how to moderate without censoring

Why moderating content without censoring users demands consistent, transparent policies.


Interviewer: It sounds like safety is very important to Friendbase. Why is that?

Deborah:  Why is that? Quite early on, we who work in the company discussed what our core values should be. And one of the core values we decided upon is inclusion. Everybody is welcome. And for everyone to feel welcome. You have to have a welcoming atmosphere.

When you continue along that line of thought, then obviously you come to the point where, OK, if everyone’s going to be welcome and you want it to be a friendly space, then somewhere you’re going to have to stop toxic behavior. So, for us safety, it’s just part of our core values.

And also, I have a teenage daughter who loves gaming. I’ve seen how platforms behave. She’s part of groups that interact with each other online. I just feel that there must be a way of doing things better. It’s as simple as that. We can do better than this, letting it be super toxic. And there are some amazing people out there working with fantastic toolsets. There are some fantastic platforms and social games out there that also work in the same sort of direction as we do. It’s really great.

And you know what? To be quite honest, I think that there have been several case studies where it’s proven as well from a business perspective that you have a longer retention and a higher profitability when you can keep your user online for a longer time. So, you know, in itself, from a business sense, it also makes perfect sense to work in a way where you keep your user as long as possible.

Interviewer: You have tons and tons of experience obviously with startups and social platforms. If you were to give a piece of advice to someone who is running a similar service to Friendbase or even who are thinking about starting one, what would that be?

Deborah:  It would be, first of all, to determine what level of safety you want to have, depending on your user group. Obviously, the younger demographic you have, the more safety tools you must ensure that you have in place. Also, not to build everything yourself. Especially if you’re working on an international market with many languages. Just to be able to filter many languages and in a decent way is a huge undertaking. If you think that you’re going to be able to hack together something yourself, it’s not that easy. It’s better to work with a tool or a company that has that as their core business because they will constantly be working with the state of the art solutions.

So better to liaise with switched on companies that already work with this as their main reason for being. I think that’s important. And then, of course, add your own easy to report system, easy to communicate with your user’s system so that you have sort of a double layer.

I mean, I’ve seen several different companies that work now with different moderation tools and chat filters and so forth. Many of them they do stellar work. And it’s important at the end of the day because if anything really, really bad would happen, then you’re just finished as a business. It’s as simple as that. The last thing you would want is to have someone knock on your door and shut you down because something’s happened online in your platform.

Deborah: So, yeah, our our aim is to become one of the big global players. It’s exciting times ahead.

Interviewer: For sure. Any closing remarks? Any statements you want to get out there from a personal point of view or from Friendbase?

Deborah: The Internet is a great place to be because there’s so much you can learn. You can meet so many interesting people. But, there is a dark side as well. And you have to be aware of it. Just by being a little bit street smart online people can keep themselves safe. And we’re getting there. People are learning. Schools have it in their curriculum, social platforms try to teach users how to behave. So slowly but surely, we’re getting there.

Friendbase is currently looking for more investors. If you are interested reach out to Deborah Lygonis.

If you need help with content moderation get in touch with Besedo.

This is Besedo

Global, full-service leader in content moderation

We provide automated and manual moderation for online marketplaces, online dating, sharing economy, gaming, communities and social media.

Form background

The outbreak of COVID-19 or Coronavirus has thrown people all over the world into fear and panic for their health and economic situation. Many have been flocking to stores to stock up on some essentials, emptying the shelves one by one. Scammers are taking advantage of the situation by maliciously playing on people’s fear. They’re targeting items that are hard to find in stores and make the internet – and especially online marketplaces – their hunting ground, to exploit desperate and vulnerable individuals and businesses. Price gouging – or charging unfairly high prices – fake medicine or non-existent loans are all ways scammers try to exploit marketplace users.

In this worldwide crisis, now is a great time for marketplaces to step up and show social responsibility by making sure that vulnerable individuals don’t fall victim to corona related scams and that malicious actors can’t gain on stockpiling and selling medical equipment sorely needed by nurses and doctors fighting to save lives.

Since the start of the Covid-19 epidemic we’ve worked closely with our clients to update moderation coverage to include Coronavirus related scams and have helped them put in place new rules and policies.

We know that all marketplaces currently will be struggling to get on top of the situation and to help we’ve decided to share some best practices to handle moderation during the epidemic.

Here are our recommendations on how to tackle the Covid-19 crisis to protect your users, your brand and retain the trust users have in your platform.

Refusal of coronavirus related items

Ever since the outbreak started, ill-intentioned individuals have made the price of some items spike to unusually high rates. Many brands have already taken the responsible step of refusing certain items they wouldn’t usually reject, and some have set bulk-buying restrictions (just like some supermarkets have done) on ethical and integrity grounds.

Google stopped allowing ads for masks, and many other businesses have restricted the sale or price of certain items. Amazon removed thousands of listings for hand sanitizer, wipes and face masks and has suspended hundreds of sellers for price gouging. Similarly, eBay banned all sales of hand sanitizer, disinfecting wipes and healthcare masks on its US platform and announced it would remove any listings mentioning Covid-19 or the Coronavirus except for books.

In our day to day work with moderation for clients all over the world we’ve seen a surge of Coronavirus related scams and we’ve developed guidelines based on the examples we’ve seen.

To protect your customers from being scammed or victim of price-gouging and to preserve your user trust, we recommend you refuse ads or set up measures against stockpiling for the following items.

  • Surgical masks and face masks (type ffp1, ffp2, ffp3, etc.) have been scarcely available and have seen their price tag spike dramatically. Overall, advertisements for all kinds of medical equipment associated with the Covid-19 should be refused.
  • Hands sanitizer and disposable gloves are also very prone to being sold by scammers at incredibly high prices. We suggest either banning the ads altogether or setting regular prices on these items.
  • Empty supermarket shelves of toilet paper have caused this usually cheap item to be sold online at extortionate prices, we suggest you monitor and ban these ads accordingly.
  • Any ads with the mention of Coronavirus or Covid-19 in the text should be manually checked to ensure that they aren’t created with malicious intends.
  • The sale of magic medicines pretending to miraculously cure the virus.
  • Depending on the country and its physical distancing measures, ads for home services such as hairdressers, nail technicians and beauticians should be refused.
  • In these uncertain times, scammers have been selling loans or cash online, preying on the most vulnerable. Make sure to look for these scams on your platform.
  • Similarly, scammers have been targeting students talking about interest rates being adjusted.

Learn how to moderate without censoring

Why moderating content without censoring users demands consistent, transparent policies.


Optimize your filters

Ever since the crisis started, scammers have become more sophisticated as days go by, finding loopholes to circumvent security measures. By finding alternative ways to promote their scams, they use different wordings such as Sars-CoV-2 or describing masks by their reference numbers such as 149:2001, A1 2009 etc. Make sure your filters are optimized and your moderators continuously briefed and educated to catch all coronavirus-related ads.

Right now, we suggest that tweak your policies and moderation measures daily to stay ahead of the scammers. As the crisis evolves malicious actors will without doubt continue to find new ways to exploit the situation. As such it’s vital that you pay extra attention to your moderation efforts over the following weeks.

This is Besedo

Global, full-service leader in content moderation

We provide automated and manual moderation for online marketplaces, online dating, sharing economy, gaming, communities and social media.

Form background

Is your site suffering from ‘marketplace leakage’? If so it’s because your customers are sharing their personal details with each other – to avoid paying site fees. But by doing so they also put themselves at risk. Here’s how to make sure your business protects itself from marketplace leakage and those that use it.

Marketplace leakage (also referred to as ‘breakage’) is a real problem for many online businesses. According to Venture Capitalists, Samaipata, the term can be defined as ‘what happens when a buyer and seller agree to circumvent the marketplace and continue transacting outside the platform.’

Broadly speaking, there are several ways in which personal details are shared – via listings, embedded in images, and within one-to-one chats. Information shared typically includes phone numbers, email addresses, WhatsApp details, and money transfer account details.

From a user perspective, it might make sense to try and do so. However, many don’t realize the wider ramifications of marketplace leakage and the negative impact it can have on the platforms they transact on – and on their own businesses.

Let’s look more closely at the impact of sharing personal details online via marketplaces and what can be done to prevent it.

How personal details do damage

As we see it, there are 3 key ways in which sharing personal details can have a negative impact.

1. Conversions

From eBay to Airbnb; Amazon to Fiverr – the vast majority of marketplaces facilitate the trade of goods and services. As a result, a core part of each platform is its payment infrastructure.

But not only do these solutions offer a trusted way for users to transact, they can also be used to collect fees – a percentage paid for using the platform.

In the early days of a platform’s existence, many sites may be available to both buyers and sellers for free – whilst the marketplace is trying to scale and get as many users as possible. However, once it’s reached a certain threshold and networks effects are visible, it’s common for them to begin charging, often through the transaction.

This is often when users – primarily those selling on these sites – will try to circumvent the platform and include their contact details in each post. It might be that they paste their email address in the product description itself, or create an image that has details included within it.

When this occurs, your marketplace loses out on conversions. It’s something that’s easy to overlook and – on the odd occasion – let slide. But in the long-term, activities like this will seriously dent your revenue generation.

2. Retention

One of the major differentiating factors between online marketplaces is whether they’re commoditized or non-commoditized – particularly where service-focused platforms are concerned.

While commoditized service providers are more about getting something specific fixed, delivered, or completed (think Uber or TaskRabbit); non-commoditized providers (eg Airbnb) take into account a number of determining factors – such as location, quality, and available amenities.

Due to the nature of these sorts of services, they are more likely to encourage personal interactions – particularly when repeat transactions with the same vendor are involved. Once trust and reliability are established, there’s little incentive for either party to remain loyal to the platform – meaning conversions are more likely to be forfeited.

Leakage of this nature was partly to blame for the demise of Homejoy – an on-demand home services recruitment platform. The nature of the work involved increased the likelihood of recurring transactions. However, it transpired that the features facilitated by the site – in-person contact, location proximity, and reliable workmanship – were of greater value than the incentives offered by using the site itself in many cases.

As a result, more and more transactions began happening outside of the marketplace; meaning that the site lost out on recurring revenues.

3. User safety

Losing control of the conversation and having users operate outside of your marketplace, increases the risk of them being scammed.

This is particularly prevalent in online dating, where even experienced site users can be duped into providing their personal details to another ‘lonely heart’ in order to take the conversation in a ‘different direction’.

eHarmony offers some great advice on what users should be wary of, but the general rule of thumb is to never disclose personal details of any kind until a significant level of trust between users has been established.

While similar rules apply to online marketplace users too, some telltale signs of a scammer are requests for alternative payment methods – such as bank or money transfers, or even checks.

An urgency to trade outside of the marketplace itself is also a sign to be aware of. So it’s important to advise your users to be cautious of traders that share their personal details. Also, make a point of telling them to be wary of vendors who are ‘unable’ to speak directly to them – those who request funds before any arrangements have been made.

In all cases, marketplaces that don’t monitor and prevent this kind of activity put their customers at risk. And if their transaction is taken away from your site, they forfeit the protection and assurances your online marketplace provides.

But unless your users understand the value and security of your platform, they’ll continue to pursue conversations off your site and expose themselves to potential scammers.

Learn how to moderate without censoring

Why moderating content without censoring users demands consistent, transparent policies.


Preventing marketplace leakage

The best way to overcome these issues and prevent marketplace leakage is to do all you can as a marketplace owner to keep buyer-seller conversations on your site and reinforce why it’s in their (and to some extent your) interest not to share personal details and remain on your platform.

There are several ways to do this.

Stronger communication

The stronger the communication channels are within your platform, the less incentive there is for customers to navigate away from your site.

From eBay and Airbnb’s messaging functionality (which look and feel like email servers) to one-to-one chat platforms (similar to Facebook Messenger or WhatsApp), or even on-site reviews and ratings; the more user-friendly and transparent you make conversations between different parties, the greater the likelihood they’ll remain on your site. A point we also highlighted and covered in our webinar about trust building through UX design.

In addition, it’s always worth reinforcing exactly what your marketplace offers users – and reminding them of their place within it. For example, telling them they’re helping build a trust-based peer-to-peer network is a powerful message – one that speaks to each user’s role as part of a like-minded online community.

Provide added value services

If users feel as though there’s no real value to using your site – other than to generate leads or make an occasional purchase – there’s very little chance that you’ll establish any meaningful connection.

The best way to foster user loyalty is to make the experience of using your marketplace a better experience to the alternative. In short, you need to give them a reason to remain on your site.

In addition to safety and security measurements – consider incentives, benefits, and loyalty programs for both vendors and buyers.

Turo, the peer-to-peer car rental site is an example of a company that does this very well – by offering insurance to lenders and travelers: both a perk and a security feature.

In a similar way, eBay’s money-back guarantee and Shieldpay’s ‘escrow’ payment service – which ensures all credible parties get paid; regardless of whether they’re buying or selling – demonstrate marketplaces acting in both customers and their own interests.

Another way in which marketplaces offer better value is through the inclusion of back-end tools, which can help vendors optimize their sales. Consider OpenTable’s booking solution for example. The restaurant reservation platform doesn’t just record bookings and show instant availability; it also helps its customer fill empty seats during quieter services.

Platforms that can see past their initial purpose and focus on their customers’ needs are those that thrive. They offer a holistic, integrated solution that addresses a wider range of pain points. Which is a great way of ensuring they’ll remain loyal to your business; ultimately reducing leakage.

Filter and remove personal details

A relatively straightforward way to prevent marketplace leakages is to monitor and remove any personal details that are posted on your site.

However, this can turn out to become quite the task, especially when the amount of user-generated content increases.

The next logical step here would be to direct efforts towards improving your content moderation. Either improve your manual moderation and expand your team or look at setting up an automated moderation solution.

An automated filter is a great solution to help prevent personal details to be shared, and although the filter creation process can be complex, it’s definitely possible to create highly accurate filters to automatically detect and remove personal details in moderation tools like Implio.

Machine learning AI is another great automated moderation solution that will help with preventing personal details, and much more. Built on your platform-specific data, a tailored AI moderation setup is developed to meet your marketplace’s unique needs. This solution is a great option for online marketplaces that look for a complete customized solution.

Added value and moderation – a mutual benefit

Trust, security, and accountability are the most valuable features that any marketplace or classified sites can offer its users. However, they’re not always the most visible components.

But when they’re parts of a broader benefit – such as optimized user experience or a suite of useful features – the need to share personal details and transact way from a site is mitigated.

That said, shared personal details will always contribute to marketplace leakage. And without the right monitoring and moderation processes in place, it’s impossible for marketplace owners to overcome the challenge of marketplace leakage.

At Besedo, we work with online marketplace and classified sites to help them make the right choices when it comes to safeguarding their businesses and users by removing personal details.

To learn more about how you can prevent personal details form your marketplace, specifically through automated filters, check out our on-demand Filter Creation Masterclass.

This is Besedo

Global, full-service leader in content moderation

We provide automated and manual moderation for online marketplaces, online dating, sharing economy, gaming, communities and social media.

Form background

Creating a filter for moderation is the easy part. Ensuring that a filter produces the desired result is a science. Great accurate content moderation filters don’t just catch what they are set up to find, but also work in a way that minimizes the number of false positives.

With good filters alone our clients have achieved up to 80% automation of their content moderation, but what is the secret to actually building a filter that is both accurate and efficient?

8 steps of filter moderation

Our expert filter manager Kevin Martinez shares his 8 steps on how to build content moderation filters that work.

In this example, we are using drugs as the target of the filter, but the process is the same for building anything from profanity filters to rules aimed at catching ads for endangered species.

1. Mission:
Define the goal of the filter. In this case, we want it to help us prevent illegal drugs from getting posted to our site. As such, we created a filter in Implio and name it “drugs”. It is always advisable to name your filters something descriptive. You are likely going to end up with a bunch of filters so it is good to be able to understand their function at a glance.

2. Local:
Check the laws of the country your site is operating in. Laws can vary widely depending on country and sometimes even by region. In Spain, for instance, you are allowed to sell the growing box for cannabis and cannabis seeds, but not the plant itself.

3. Action:
Decide on the action you want the filter to take. Should It refuse, send to manual moderation or is it just a test filter that shouldn’t take any action other than highlighting the ads that would’ve been caught? In Implio the default action is to automatically accept any ads that don’t match the filter, but you have full control over what happens to content that matches your rules.

4. List:
Create a list of all drug-related keywords (Cocaine, heroin, cannabis etc.). Make sure you also include any slang words your users are likely to use for drugs.

5. Rule:
Now it is time to set up your rule. Make sure that the rule pulls from the list and that you add exceptions to avoid false positives. For example, in step 2 we discovered that selling cannabis seeds are okay. As such we need to make sure our filter excludes “cannabis”+”seeds”.

6. QA:
Once your filter is set up with the list and all relevant exclusions it is time for the first quality check. Upload your data to Implio and review the matches. Are you getting any false positives?

7. Exceptions:
For all false positives make sure that you add exceptions (also called white-listing specific content). Think your exceptions through so you don’t give a blanket white-list that allows unwanted content through.

8. Rinse & repeat:
Once you have added new exceptions run your data through again to quality check your updated filter. Repeat step 6-8 as many times as you have to in order to reach your target quality rate. At Besedo we aim at 95% accuracy as a minimum and for most of our filters we reach higher.

Even though you have the option to refuse content pieces automatically, we do not recommend that unless you are able to reach 100% accuracy level on your filter. (It is quite rare to be able to reach that accuracy level, but an example of a filter that could reach such scores would be an IP related rule where you do not want to allow users from certain IP’s to post).

If you are not 100% certain that all ads matched by the refusal filter should be refused, then you should send matches for manual review instead. Otherwise, you run the risk of ruining your user experience.

If you follow Kevin’s 8 steps you should be well equipped to create your own accurate filters. If you want to improve your abilities in filter management even further, we offer training sessions where our expert filter managers will teach you all about regular expressions and rules crafting through step by step guides and exercises.

Learn how to moderate without censoring

Why moderating content without censoring users demands consistent, transparent policies.


This is Besedo

Global, full-service leader in content moderation

We provide automated and manual moderation for online marketplaces, online dating, sharing economy, gaming, communities and social media.

Form background

We talk weapons, water heaters, challenges of weeding out false positives and how to create accurate filters with Besedo filter manager, Kevin Martinez.

Interviewer: Great to meet you, could you tell us a bit about yourself?
Kevin: I’m Kevin Martinez; originally from Spain, but raised in France, now working out of Besedo’s Malta office. I’ve been with the company for five years. In 2016 I had the honor of setting up Besedo’s first client filter. And we still have the client – so I must have done something right!

Interviewer: Excellent! So, tell us more about what you do.
Kevin: The short answer is ‘I’m a filter manager’. I make sure that our clients’ filters are working as well as they should be – monitoring filter quality across all Besedo assignments.

I manage three filter specialists – two in Colombia and another in Malta. Being from different cultures, speaking different languages, and having a presence in different time zones means we can work with clients across the world.

The longer answer is that I assess decisions that our automated moderation tool Implio has made. Quality checks like these are done at random. I take a sample of content that’s been approved – items that have been filter-rejected and filter-approved – and identify if any mistakes were made. I then learn from these mistakes and make appropriate adjustments to the filter. This way we maintain and improve the accuracy rate of our filters over time.

Quality checks take time, as we’re really thorough. A single one can take half a day! But tracking the quality day-by-day is vital to keeping the filters accurate and it allows us to provide a report with a quality rate for our clients at the end of each month.

Interviewer: That sounds like a complex task… What kind of things are you looking for?
Kevin: Typically, we’re looking for false positives in filters: terms that are correctly filtered according to the criteria set, but aren’t actually prohibited.

Take Italian firearms brand, Beretta, for example. Weapons are prohibited for sale online in some nations, but not in others. So, for many sites a filter rejecting firearms would make sense.

However, there’s another Italian brand called Beretta – but this company manufactures water heaters (!). There’s also a Chevrolet Beretta car, and an American wrestler who goes by Beretta too. The filter can’t distinguish between these as completely different things until we know that they need to be distinguished between. So, lots of research is needed to ensure that, say, a Beretta water heater parts ad isn’t mistakenly rejected from an online marketplace.

A good filter will reduce the time the moderators spend on the content queue and will also reduce the length of time it takes to get a piece of content live on the site. It’s an ongoing process, one that gets better over time: gradually improving automation levels and making the manual moderator’s job a lot easier.

Interviewer: What’s the overall effect of a ‘bad’ filter, then?
Kevin: It depends. If the filter is set up to auto-reject matched words and phrases, it leads to a bad user experience as genuine ads might get rejected (as the case with water heaters illustrated).  If, the filter is set up to send matched content for manual moderation, the automation level decreases. We agree to a certain automation level when we sign a contract with a client, so if there are more items for the manual moderation team to approve; it puts pressure on us to reach our service level agreement.

Interviewer: Which rules are hardest to program into a filter?
Kevin: Scam filters are the most complex to implement; mostly because scams evolve and because scammers are always trying to mimic genuine user behavior. To solve this, we monitor a number of things in order to detect ‘suspicious’ behavior, including email addresses, price discrepancies, specific keywords, IP addresses, payment methods (like PayPal and Western Union) – among other things.

One of the biggest challenges is that on their own, elements like these aren’t suspicious enough to warrant further investigation; so we have to ensure the filter recognizes a combination of them for it to be effective. We perform a lot of research and collaborate closely with clients, to ensure each filter is as accurate as possible.

Interviewer: Sounds like you need a lot of expertise! What does it take to be a good filter manager? 
Kevin: You need to understand how moderation works, and most filter specialists have a good grasp of computer programing (particularly the concept of regular expression) too. But equally you need to have a curious, analytical, and creative mind.

Admittedly, filter quality checks can be a bit repetitive, but they are very important. Being able to investigate, test, and find ways to setup and improve filters is crucial. This means understanding how the filter will interact with words in practice, not just in theory. The most important thing is to have the drive to keep pushing; to find the perfect solution for the client’s needs.

Learn how to moderate without censoring

Why moderating content without censoring users demands consistent, transparent policies.


Interviewer: What do you enjoy the most about your work?
Kevin: I love the beginning of every new project. I help onboard each new client from the very start, setting up the filters and creating a report for them. Each one is different, so lots of investigation is involved as there are different rules to consider: depending on who the client is, what they do, and where they’re based.

As mentioned, rules can differ between countries. For instance, in South America, you don’t need to apply a gender discrimination filter for something like jobs or housing – unthinkable in Europe, which has stringent equality laws.

Each day I look at the quality of the client data by opening a random filter, reviewing at the ads going through that filter and seeing everything’s working correctly. There are many parameters involved, and it involves going over the finer details, but this is the stuff I’m passionate about. I can be quite obsessional about it!

Nothing is impossible. I aim to get the client what they want and will try again and again and find a creative way to deliver it!

Kevin Martinez interview

Kevin Martinez

Kevin is a filter manager at Besedo. He combines creativity, perseverance and in-dept research to create highly accurate filters in the all-in-one moderation tool; Implio.

This is Besedo

Global, full-service leader in content moderation

We provide automated and manual moderation for online marketplaces, online dating, sharing economy, gaming, communities and social media.

Form background

What does it take to build a state-of-the-art Artificial Intelligence content moderation tool? We caught up with Besedo’s semantics expert and computational linguistics engineer, Evgeniya Bantyukova.

Interviewer: Nice to meet you! Tell us a little about yourself.

Evgeniya: I’m Evgeniya and I’m based in Besedo’s Paris office. I’m originally from Russia but I’ve been in France for the past five or so years. I started at ioSquare about a year and a half ago, and have continued to work there as part of Besedo since the two companies merged last year.

Interviewer: What do you do? What is your job title and what does it really mean?

Evgeniya: As a computational linguistics engineer, I guess you could describe me as part linguist and part computer programmer. The work I do bridges the gap between what people search for and post online and the way content is moderated.

I work with semantics. This means I spend a lot of time researching information and looking at the different ways words and phrases are presented and expressed. I also build filters to analyze and identify the information I’ve manually researched. It’s an iterative process of constant refinement that takes time to perfect.

The filters can then be used by us, on behalf of our clients, to identify when a certain piece of text using these terms and phrases is submitted to their site; before it gets posted. The ultimate aim is to ensure that incorrect, defamatory, or just plain rude information doesn’t get posted to our clients’ sites.

Interviewer: What kind of projects have you worked on? Could you give us an example? 

Evgeniya: Sure. Recently I was tasked with creating a filter for profanity terms in several different languages – not just the words themselves, but variations on them, like different ways to spell them or alternative phrasings.

This also involved analyzing them and creating a program or model that could detect their use. There was a lot of data capture and testing involved on millions of data points; which helped ensure the filters we built were as effective as possible.

One thing I’m working on right now is a project tackling fake profiles on dating sites: analyzing scam messages and extracting the expressions and words that are most frequently used. One thing I have discovered in this process is that those posting fake profiles often use sequences of adjectives – words like ‘nice’, ‘honest’, or ‘cool ‘ – so now I’m looking at creating a model that finds profiles that fit that description. That approach on its own would create many false positives, but with discoveries like these we get a much more precise idea of what fake profiles look like, and that helps us create filters that limit the number that go live on our clients’ sites.

Interviewer: How does the work you do feed into AI moderation?
Evgeniya: Crafting filters involves working on a set amount of data. The more data we have, the more accurate we can make our filters. It’s an iterative and human-driven process, but engineered to be very precise.

Filters like these, when used as verification models, can help improve the precision and quality of manual content moderation. And when used in combination with our machine learning/deep learning pipeline, they improve our AI’s overall accuracy and efficiency.

The filters I build are quite generic so they are used as a framework for multiple clients, depending on their moderation needs. And they can be tailored to specific assignment as needed. On top of that and to keep our filters “sharp”, we continuously update them, as language evolves and new trends and words appear.

Interviewer: Do you have any heroes or role models that you admire in your field?

Evgeniya: Well, as you might imagine, role models in computational linguistics are kind of hard to come by. But I’m a big fan of theoretical linguists like Noam Chomsky.

Interviewer: What qualities do you need to succeed in your field?

Evgeniya: I think you need to be genuinely curious about the world in general. Every new trend and phenomenon should interest you as they will result in new tendencies and words and that will impact the filters you are crafting.

You also need to have a knack for languages or at least the structure of how different languages are built.

Finally you need to be openminded and able to stay objective. When working on a profanity filter, it doesn’t help if you are continuously offended. You need to stay neutral and focus on the endgame; keeping people safe online.

This is why I enjoy my job so much, it is very rewarding knowing that you are making a difference – whether that’s ensuring that a site is secure for users or more generally when seeing the positive impact of something you’ve done. Take dating sites for instance; The fact that the work I do can help someone find love, that’s the greatest reward I can think of. I guess I’m something of a hopeless romantic!

Evgeniya Batyukova - portrait

Evgeniya Bantyukova

Evgeniya is a linguistic engineer at Besedo.

She combines her programming and linguistic skills in order to automatically process natural languages.

Her work allows Besedo to build better and more accurate filters and machine learning algorithms.

Learn how to moderate without censoring

Why moderating content without censoring users demands consistent, transparent policies.


This is Besedo

Global, full-service leader in content moderation

We provide automated and manual moderation for online marketplaces, online dating, sharing economy, gaming, communities and social media.

Form background