Why is Trust and Safety so hard? | This Week in Business

This Week in Business is our weekly recap column, a collection of stats and quotes from recent stories presented with a dash of opinion (sometimes more than a dash) and intended to shed light on various trends. Check every Friday for a new entry.

Discord this week rolled out a new approach to tackling toxicity on its platform. This is a problem that has bedeviled online games and communities pretty much since there were online games and communities, so let's see how Discord plans to solve the problem.

QUOTE | "We really want to give people who have had a bad day the chance to change." – Discord senior director of policy Savannah Badalich explains the company's new content moderation policies, which will avoid permanent bans in an attempt to rehabilitate bad actors.

Wow, that sounds great. I wonder if anyone has ever tried this before.

QUOTE | "98% of our players are actually very good. What we're seeing is they all have bad days. That's where your toxicity comes from." – In 2013, Riot Games lead designer of social systems Jeffrey Lin explains why the League of Legends developer tried to reform toxic players instead of banning them.

Uh-huh. And how did that go?

STAT | 81% - The percentage of League of Legends players in the Anti-Defamation League's 2022 Hate and Harassment Online survey who reported experiencing harassment in the game, tied for the fourth worst game among the 20 major titles the ADL looked at. Riot's Valorant was the second worst, with 84% of players reporting harassment.

Spare the rod and spoil the manchild, I guess.

Perhaps more concerning is that this is a change in policy for Discord, which certainly suggests it was more active about banning toxic users in the past, even though we have pretty good evidence it wasn't as active about it as it should have been.

STAT | Over 1,000 – In 2018, the number of Discord servers named with the Nazi shorthand "1488." A source familiar with the company's moderation policy told me that when a reporter informed them about violations on a server named 1488, the company took no action because it was unsure which of the servers using a white supremacist slogan for its name was the one with violations on it.

Keep in mind, this was after Discord had already pledged to get serious about its Nazi problem in the wake of white nationalists using it to organize the 2017 Unite the Right rally in Charlottesville, VA where a counter-protestor was murdered and 19 more people were injured by a 20-year-old remembered for his fondness of racial slurs and Adolf Hitler.

Discord was given a very clear picture of the harms its lax moderation had led to, and pledged to crack down on hate groups as a result. Yet even a year down the road, it still just couldn't bring itself to nuke all 1,000 Nazi servers and their member accounts from orbit, instead letting them all keep going because acting on a reported violation of its policies was like finding a needle in a stack of needles.

To be fair, the emphasis on reform isn't the only change to Discord's safety policies, and some of the other new steps seem good, like better communication as to how users broke rules and content filters and safety alerts enabled by default on teen accounts.

Gaming the system

Now I'm not going to pretend community moderation is an easy gig. It's complicated, and at the scale of an operation like Discord or League of Legends, there are going to be endless edge cases, people testing every line a company draws, and every wrong call (and even some right ones) having the potential to become a big scandal.

In short, it's no fun at all. So naturally, people still make games out of it.

Last week, Copia Gaming, TechDirt's Mike Masnick and Leigh Beadon, and Randy Lubin of Leveraged Play launched Trust & Safety Tycoon, a free web-based game that puts players in the shoes of a content moderator for fictitious social media start-up Yapper. It's a follow-up to a similarly themed game they released earlier this year called Moderator Mayhem.

Both games challenge players to make the moderation decisions for social networks, and aim to convey how seemingly simple decisions can actually be quite complex, or have unintended consequences. In Moderator Mayhem, players are given an ever-mounting stack of user reports and limited time to properly investigate each one before making a determination in line with the network's policies.

In the more complex Trust & Safety Tycoon, users make moderation and policy decisions without time pressure, but with a mandate to juggle Yapper's user growth, ad revenue, moderator speed, and team morale. Fail to adequately manage them and the player is fired and the game is over.

Both are interesting takes on the problem and may enlighten players to the sort of difficult tasks these companies face in community moderation, but they're also expressing a very specific view of the problem.

Take Moderator Mayhem. The big pressure in the game is time, and the player is effectively forced to make decisions with incomplete information. For example, you may have to deal with a report that a user review contains links people can click on to download copyrighted films for free, but you need to waste precious seconds holding a button to check for additional context in order to find out that the review was posted by the copyright holder, and there's not enough time to do that for every report.

While the game does take into account the possibility of AI moderation and the ways that can increase (or even decrease) efficiency, from what I played there was no consideration given for the fact that the workload of a social network's moderators can be made more manageable by hiring more moderators, and time pressures like that exist in most cases because there is simply more moderation that needs to be done than staff able to do it.

As for Trust & Safety Tycoon, I didn't make it terribly far in my first game because I was fired for user growth being too slow. Ad revenue, moderation speed, and team morale were all healthy enough. The second time I tried I did a little bit worse because user moderation became too slow. The third time I tried to speedrun my firing, so it was a bit disheartening when I still hung on nearly as long. (Disheartening but apparently true-to-life, given that Twitter/X somehow still exists.)

Trust & Safety Tycoon is a reflection of the status quo in social networks right now

So Trust & Safety Tycoon is a reflection of the status quo in social networks right now, and how difficult it is to thread the needle and satisfy each of the priorities these companies currently have. And for that reason, it's actually quite a good argument for why these priorities are largely untenable and unsustainable.

And when you get down to it, those priorities are not inherent to a new social network. Those pressures for endless growth and zero-effort moderation are there because the people holding the money insist on them, and the people building these businesses plan on them to keep things running.

So while I appreciate Trust & Safety Tycoon and Moderator Mayhem for their portrayal of the difficult choices facing individuals on the moderation teams for sites like this, I'm a little concerned the takeaway people will have from it is that moderation is difficult so it's unreasonable to ask these companies to get it right.

But these companies essentially chose to have these problems by embracing a business plan maximizing growth without concern for whether they could scale effective moderation efforts alongside it. The problem is one of their own making, so it's only fair they be the ones to solve it. And if they can't solve it and still exist as a viable business, then that might be the universe's way of telling us they shouldn't exist at all.

Trust and Safety is the enemy

The task of effective and healthy moderation becomes much trickier for a Trust and Safety team when you look at it through the lens of the start-up world's values, because much of that world is actively and openly hostile to the very concept.

Venture capital firm Andreesen Horowitz told us that much last week in a "Techno-Optimist Manifesto" posted to its website. It's a long-winded bit of rhetorical contortionism that pushes the limits of how far a human's head can proceed up its own rectum. Horrifying, certainly, but also strangely impressive?

QUOTE | "Our enemies are not bad people – but rather bad ideas. Our present society has been subjected to a mass demoralization campaign for six decades – against technology and against life – under varying names like 'existential risk,' 'sustainability,' [Environmental, Social, and Governance],' 'Sustainable Development Goals,' 'social responsibility,' 'stakeholder capitalism,' 'Precautionary Principle,' 'trust and safety,' 'tech ethics,' 'risk management,' 'de-growth,' 'the limits of growth.'" – Andreesen Horowitz goes to the Nixonian effort of making an enemies list, and puts "trust and safety" on it, right alongside ethics, social responsibility, risk management, and anyone who would conceive of barriers to infinite growth.

I really want to emphasize that this was in a manifesto about how their approach to tech will create a better world and "a far superior way of living, and of being."

Looking through that list of "bad ideas," the common thread is that they are obligations and guard rails, things that say wealthy people chasing a return on investment have to consider the possible negative consequences of their actions on others, and possibly even change those actions as a result.

Andreessen Horowitz's enemies are basically Uncle Ben telling Peter Parker, "With great power comes great responsibility."

In short, Andreessen Horowitz's enemies are basically Uncle Ben telling Peter Parker, "With great power comes great responsibility."

Andreessen Horowitz hates Uncle Ben. Andreessen Horowitz is the robber that shot that old man in the gut and left him to die. In fact, the biggest difference between the robber and Andreessen Horowitz is that the robber didn't have a white-hot personal grudge against Ben. (And also that Andreessen Horowitz would insist on being hailed as a visionary savior and intellectual giant because murdering that old man inspired Peter to become Spider-Man and actually help people.)

It's worth noting that years ago Andreessen Horowitz invested in Facebook, Instagram, Lyft, Airbnb, and Robinhood, and that money came with no small amount of influence over the companies' operations.

Andreessen Horowitz co-founder Marc Andreessen has been on Facebook/Meta's board of directors since 2008. Co-founder Ben Horowitz sat on Lyft's board of directors for four years. Andreessen Horowitz managing partner Jeff Jordan sits on Airbnb's board.

So when you read about Facebook doing nothing to stop its platform helping incite genocide, Lyft hiding safety problems to boost its IPO price, Airbnb's flagrant disregard for local laws and safety standards, and Robinhood's deeply questionable handling of the GameStop stock rally, it's not hard to see Andreessen Horowitz's influence in the utter contempt for the idea that a business should ever pump the brakes on its own endeavor for the safety or well-being of others.

I feel we mention the awful track record of big tech companies surprisingly often in this column. Part of that is because when people do things like "facilitate genocide," they shouldn't be allowed to just wait a news cycle and have everyone forget that, and part of it is because this is a weekly recap column and the mountain of evidence speaking to the myriad harms done by these companies grows on a more-or-less weekly basis.

QUOTE | "Meta has profited from children's pain by intentionally designing its platforms with manipulative features that make children addicted to their platforms while lowering their self-esteem. Social media companies, including Meta, have contributed to a national youth mental health crisis and they must be held accountable." – New York attorney general Letitia James talking about a federal lawsuit filed by 31 states this week accusing the Facebook and Instagram owner of harming young users' mental health through design intended to boost engagement and addiction. Eight more states filed their own suits in state courts this week, and Florida kicked in an extra federal suit alleging Meta misled users about health risks on top of that.

But that's mostly tech stuff, and we usually focus on video games.

Fortunately for our purposes, Andreessen Horowitz also has a pretty significant footprint in gaming, having almost 50 gaming firms in its portfolio, including lucrative exits for Zynga ("I would venture to say it is one of the most evil places I've run into") and Roblox (long-criticized as profiting from the exploitation of children and providing inadequate safeguards for them on its platform).

Naturally, this firm of techies with a visible disdain for the impact their profit-making might have on the rest of the world was also super-excited for blockchain, which has proven to be incredibly wasteful, but not incredibly useful.

STAT | $7.6 billion - The amount of money Andreessen Horowitz has raised for crypto and blockchain start-ups.

Trust and Safety is a hard problem in part because firms like Andreessen Horowitz make it hard. Because they believe in "move fast and break things" even (or perhaps especially) when the things you're breaking belong to other people, or even everybody. Because they believe their wealth is a meritocratic emblem of their intelligence, entitling them to shape society however they see fit while the intellectually stunted little people of the world feebly protest about privacy, or labor rights, or safety precautions, or equality, or any of a million other trivialities while the Techno-Optimists are saving humanity and making the world a better place by helping genius entrepreneurs bring their dreams to life.

Some of the important dreams Andreessen Horowitz has helped bring to life include:

• Custom sneakers designed for video game enthusiasts
• A gig economy e-girl-for-hire site
• Virtual horse racing where the horses are NFTs
• An NFT rental service
• Mail-order razor blades

Surely you can understand how these start-ups should be allowed to chase those world-changing dreams without the shackles of things like ethics, concern for the environment, or trust and safety.

Obviously not all of Andreessen Horowitz's investments are so trivial. But I kind of wish they were because that seems like it might be preferable to having their ethically freewheeling approach dictating the evolution of more important fields. I shudder to think of the influence Andreessen Horowitz has over the more than 80 companies it has backed in the healthcare industry, where profit-driven entities regularly produce obscene outcomes and anything aspiring to infinite growth is correctly identified as a cancer.

That's a lot of time for this column to spend talking not so much about games but about the investors behind a bunch of gaming and gaming-adjacent start-ups.

But when we talk about the difficult problems around trust and safety in the industry today, it's well worth understanding that we have these problems because we have prioritized growth over safety.

That prioritization, the problems that stem from it, the solutions companies are willing to consider to those problems – and even who gets to join the next generation of industry leaders starting new studios – are all influenced by the values of those who hold the purse strings, people who view trust and safety as the enemy.

Contents under pressure

Nintendo is cracking down on people doing free marketing for it. The company this week announced new guidelines for sharing videos and images of its games online so that content creators would know what could risk them piling up copyright strikes or getting their work taken down. As you might have guessed, the guidelines cover a fair bit of ground Nintendo has no business ruling out.

QUOTE | "We reserve the right to object to any content that we believe is unlawful, infringing, inappropriate, or not in line with these Guidelines." – Nintendo in its latest guidelines.

Let's breeze past "unlawful" and "infringing" because while we can debate what qualifies, Nintendo should be on firm ground to object to anything that falls under those descriptions. But "inappropriate" is a real nice blanket term for "anything we don't want." So what's no longer safe to show?

• Emulation – Nintendo says anything showing how to use emulators "and/or" devices that circumvent its security measures is a no-go.
• Unauthorized consoles and/or software – Are you playing on an Analogue Super NT? Are you streaming a homebrew Wii game? How about any of those unlicensed Tengen NES games like Gauntlet, Pac-Man, Shinobi, or Klax? Nintendo hates all of it.
• Data mining – Anything with video, images, or sound that can't be accessed through regular gameplay goes on Nintendo's naughty list
• Mods – Anything involving cheating or unauthorized modification, so tool-assisted speedruns and games with translation patches are grounds for Nintendo to move for a takedown, clearly another step in Nintendo's decades-long crusade to prevent English speakers from ever enjoying Mother 3.

Whether or not Nintendo actually follows through on these threats, they're still a pretty dramatic overreach, and I don't think Nintendo could go to court to get any of this taken down legally. But that's why Nintendo isn't directly going after the people actually doing the "inappropriate" activity.

Instead it's pressuring corporations because they aren't as incentivized to fight that fight, and B2B action doesn't risk providing clarity to any legal gray areas around these activities. As we've said before, when it comes to these matters, the law is effectively whatever Nintendo says it is.

Meta's impaired vision

STAT | $11.5 billion – The operating losses of Meta's Reality Labs AR/VR division through the first nine months of its fiscal year.

STAT | $46.5 billion – The cumulative operating losses of Meta's Reality Labs since 2019.

Oof that's a lot of money. But if they're still chasing this thing so aggressively, it's probably because they can clearly see the turning point, right?

QUOTE | "For Reality Labs, we expect operating losses to increase meaningfully year-over-year due to our ongoing product development efforts in augmented reality/virtual reality and our investments to further scale our ecosystem." – Meta lets investors know 2024 will not be the year it turns this thing around.

That doesn't sound great, particularly when you take into account that Reality Labs' sales have been trending downward for nearly two years now.

STAT | 42% - The year-over-year drop in Reality Labs revenue over the first nine months of 2023.

This month's launch of the Quest 3 should help grow sales in the near-term, but as for those losses… Has any human being ever burned so much money in the pursuit of making fetch happen?

Has any human being ever burned so much money in the pursuit of making fetch happen?

What gives me the most skepticism about VR/AR these days is that the actual experiences being offered have gotten noticeably closer to the standalone, high-powered, light and comfy headset we were always told would be the breakthrough for the technology, but excitement around the field remains modest at best.

Of course, there are plenty of true believers who aren't so bothered by that, and will tell you it's a natural part of the cycle for new technology.

QUOTE | "I'm even more bullish about VR and mixed reality than I ever have been. There are a few people here [at GDC] who are like, 'Oh well, is VR really kicking off?' If you've seen the Gartner hype cycle, there's this huge hype and everyone says it's going to change the world - stuff comes out, it's expensive, it's not got amazing software, we're still learning how to make great VR games, there's this trough of disillusionment." - nDreams founder and CEO Patrick O'Luanaigh tells us there's nothing to be concerned about. In an interview from six-and-a-half years ago.

How long does the disillusionment have to last until we just agree VR is what it is, and that's perfectly fine and good, but it's not the future of everything?

The rest of the week in review

QUOTE | "Today's patch corrects an error where the Cuban flag was incorrectly displayed instead of the Puerto Rican flag. We understand that accurate representation matters, and greatly regret this error. We sincerely apologize and will do better in the future." – Insomniac Games apologizes for putting the wrong country's flag in Miles Morales' home in Spider-Man 2.

STAT | 0% - The chances of a game like Need for Spirit: Drink and Drive Simulator arriving on the Switch with a PEGI 3+ rating if anybody actually looked at it for two seconds instead of relying entirely on the International Age Rating Coalition's method of assigning ratings based on a developer-submitted questionnaire.

QUOTE | "You don't have a market that expects [non-stereotypical work], and we've had this conversation come up with funders; there is a push on Indigenous creatives to go that direction [of stereotypical work]" - Achimostawinan Games' Meagan Byrne says there's pressure for Indigenous creative work to match non-Indigenous preconceptions about what Indigenous work looks like.

STAT | 13% - Xbox content and services revenue grew 13% last quarter thanks to the launch of Starfield, which Microsoft says racked up more than 11 million players and boosted subscriber growth for Game Pass. Xbox hardware revenue still dropped 7% year-over-year.

STAT | 53% - Capcom's sales were up 53% for the first half of the fiscal year, thanks to Street Fighter 6 selling 2.47 million copies, Mega Man Battle Network Legacy Collection Vol 1 and Vol 2 selling 1.42 million, and the Resident Evil 4 remake adding another 1.66 million units to its total.

STAT | 35 – People laid off this week by Hungarian animation studio and Embracer Group subsidiary Digic.

STAT | 5 months – The span between layoffs at PlayStation's Visual Arts support studio, which cut people in May and again last week.

STAT | 20 - PlayStation's Media Molecule studio is looking to lay off about 20 people, or 15% of its workforce.

STAT | 15 – Roblox China cut 15 people at its Shenzhen headquarters and in its US teams due to an "evaluation of the operational structure."

QUOTE | "Many were drawn to their jobs at Experis because of their desire to work in the video game industry, only to be faced with burnout and a struggle to cover basic living expenses." – A representative with the International Association of Machinists and Aerospace Workers (IAMAW) union explains why it's helping game testers at Experis Game Solutions in their push to unionize.