Malign creativity

January 27, 2021 9:22 AM

Kamala Harris abuse campaign shows how trolls evade social media moderation (The Conversation UK) – Alexandra Pavliuc, a researcher in Social Data Science at the Oxford Internet Institute, presents some findings from her and her colleagues’ report on gendered and sexualized disinformation, Malign Creativity: How Gender, Sex, and Lies are Weaponized Against Women Online. Another brief summary available also here – some highlights:

The report presents an “analysis of online conversations about 13 female politicians across six social media platforms, totaling over 336,000 pieces of abusive content shared by over 190,000 users over a two-month period.”

The subjects of the analysis include Vice President Kamala Harris, Senators Susan Collins and Kirsten Gillibrand; US Representatives Alexandria Ocasio-Cortez, Elissa Slotkin, Ilhan Omar, Jaime Herrera Beutler, Elise Stefanik, and Lauren Underwood; Michigan Governor Gretchen Whitmer; as well as world leaders including New Zealand Prime Minister Jacinda Arden, Canada’s Deputy Prime Minister Chrystia Freeland, and UK Secretary of State for the Home Department Priti Patel. Data was collected from Twitter, Reddit, Gab, 4chan, 8kun, and Parler.

During the period of data collection, between September 1st and November 9th, 2020, Vice President Kamala Harris was far and away the most targeted for the 13 subjects, drawing 78% of the total number of recorded instances.

...Importantly, the researchers observe a number of tactics – which they dub “malign creativity”– that abusive users employ to avoid detection by social media platform moderators, such as the use of coded language.

...Fundamentally, the researchers conclude that social media platforms are not taking substantial enough action to police gendered harassment and disinformation, and that their policies are inadequate to the scale and scope of the problem.

posted by bitteschoen (10 comments total)

Online gendered abuse and disinformation is a national security issue. In interviews with women targeted by Russian, Iranian, or Chinese state media, the research team found that gendered tropes formed the basis of disinformation and smear campaigns against them.

This is brilliant framing (and happens to be true). It's hard to come up with a better definition of a deliberate bad actor than state media bots.
posted by benzenedream at 9:29 AM on January 27, 2021 [12 favorites]

I found this concept interesting:

They should also regularly update platform classifiers or keywords to reflect and root out malign creativity, improve automated detection methods, and introduce nudges or friction to discourage users from posting abusive content.

I'm guessing "nudges or friction" is like a pop-up box saying "Hey this could be construed as abusive, do you really want to send?" And then an algorithm that flags accounts that keep having that happen.

But I feel like there are lots of other ways I'm not thinking of.
posted by emjaybee at 9:44 AM on January 27, 2021 [10 favorites]

US Representatives Alexandria Ocasio-Cortez

so there's at least one person in the world capable of writing out her full name
posted by medusa at 10:08 AM on January 27, 2021 [14 favorites]

Sorry, that sounds like drive-by snark. That's not what I intended. I think these articles are interesting, thanks for posting them.
posted by medusa at 10:08 AM on January 27, 2021 [11 favorites]

I'm guessing "nudges or friction" is like a pop-up box saying "Hey this could be construed as abusive, do you really want to send?" And then an algorithm that flags accounts that keep having that happen.

Beyond technical solutions, you have social ones. One of the better ones that I've seen is creating a culture where abuse is not tolerated in the community, by setting up anti-abuse and anti-bad faith rules and enforcing them. When you have a community that doesn't tolerate abuse, abusers tend to stick out, find no support, and are quickly removed. And in time, the threat of being excluded becomes powerful social friction against abuse.
posted by NoxAeternum at 11:19 AM on January 27, 2021 [4 favorites]

Twitter makes abuse effortless, anonymous, frictionless and rewarding. It was designed by people who exist at the top of most pecking orders (yes, young white privileged wealthy heterosexual males).

It is a machine for abusers to “create engagement”, unchecked.

I would love to see steps toward balancing out this unlimited power they’ve given to trolls, abusers, silencers of others.

For more on this see Mike Monteiro’s Broken by Design.
posted by andreinla at 11:36 AM on January 27, 2021 [15 favorites]

US Representatives Alexandria Ocasio-Cortez

so there's at least one person in the world capable of writing out her full name

It was snarking on people like me who do default to AOC (which is a bit of branding that she's leaned into). And then, only writing or hearing "AOC, AOC, AOC" - I do forget her full name. So snark deserved :)
posted by jb at 12:14 PM on January 27, 2021 [2 favorites]

It's okay, Twitter is creating a platform for unpaid moderation that totally won't attract the people that need the most moderation.
posted by krisjohn at 2:31 PM on January 27, 2021 [2 favorites]

The go-to for online trolls whenever certain politicians are mentioned is always predictably the same. It almost feels like bot-triggered comments. Obama: bombing brown people, Hillary : Benghazi or superpredators; etc. With Kamala Harris it seems to be locking up black folks for drug offenses. It does not matter what the story linked to is about; the comment section will always have this.
posted by indianbadger1 at 2:52 PM on January 28, 2021 [2 favorites]

Nina Jankowicz, one of the authors of the study, has written an essay in Wired about it:

Online Harassment Toward Women Is Getting Even More Insidious
From coded memes to deepfake porn, abusive disinformation campaigns are sliding past moderation tools. Platforms, Congress, and employers need to help women fight back.
posted by bitteschoen at 10:00 AM on January 29, 2021 [2 favorites]