A Breakdown of BullyHunter’s False Data Claims

Update: The BullyHunters website has been deleted, sponsors such as Vertagear have separated themselves from the campaign and SteelSeries have distanced themselves from the campaign, stating that the campaign was mainly organised by a company titled FCB Chicago. While a lot of things still remain odd to me (encouraging people to buy headsets to receive information on how to deal with harassment, ZombiUnicorn saying that SteelSeries was more involved than they are claiming etc.), this campaign is undoubtably a failure.

If someone had told me twenty-four hours ago that my request for a citation would have resulted in thousands of notifications and the partial defamation of an exploitative marketing campaign, I would not have believed you. Alas, this is what is currently happening.

If you have been following my Twitter account, I advised people not to tune in to the BullyHunters event as the event appeared quite sinister to me. I will break down why in “1. The Backstory”. If you are already familiar with what is happening, please feel free to skip to “2. The Data” where I will break down their false data claims. I am aware that many people are already writing articles and making videos about the dramatic nature of the BullyHunters campaign, but this article will focus solely on the data used to promote the event.

As usual, there will be a summary at the bottom. Please note that this article will be more editorial in nature than my usual work. If you would like to see my video game psychology work written in a more formal fashion, please feel free to check out any article in my Psychology of Video Games series. Thank you.

1. The Backstory

BullyHunters first came to my attention as an event that would supposedly raise awareness of online harassment against women and how to counteract it. Their approach (killing people speaking poorly of women during a scripted event) was criticised online for both its ineffective nature and its ‘fighting fire with fire’ mantra. The show’s host, ZombiUnicorn, was very active on Twitter leading up to the event, and this is where I saw a certain tweet from her.

I have included the tweet in picture form to make absolutely sure that no one thinks I am paraphrasing. A piece of data used to promote the BullyHunters event was “Around three million women have stopped playing games altogether because of harassment”.

As I have previously explained, being able to analyse or meta-analyse three million women at the very least is a researcher’s dream. I searched for such a study during my lunch break and could not find even a crumb of such a study. Desperate to find out more, I politely asked for a citation.

She refused, telling me to watch the stream for the source. I genuinely did not have time to watch the stream for one source, so I recalled that she had rehearsed for the event. If she rehearsed, she would know the name of the psychologist speaking about the data. Once again, I politely asked for anyone I could contact to discuss a large-scale and amazing study.

I was told to watch the VOD. This is when I became frustrated at the lack of transparency regarding mental health research. It turns out that ZombiUnicorn did indeed have the source and knew where to direct me to, but she insisted I sit through a stream where products would constantly be advertised to me (example).

I wasn’t permitted to get the source for BullyHunter’s promotional claims unless I sat through a stream where gamer gear was advertised to me. As a mental health researcher, this nauseated me and infuriated me. This is when I made the tweet and the story has not stopped developing since.

Two deep ironies lie in this interaction. The first irony is that I did not have to watch the stream to find the ‘source’. The ‘source’ lay at the very bottom of the BullyHunters website, right below a huge SteelSeries advertisement. ZombiUnicorn could have easily told me to check the website for the source, but she insisted I tune in to the stream where products were advertised to me.

The second irony is that the source itself, what I fought to try to achieve, is the epitome of bad data science.

2. The Data

Before I begin, I would very clearly like to reiterate, in ZombiUnicorn’s own words, one of the promotional pieces of data used to gather viewers for BullyHunters.

“Around three million women have stopped playing games altogether because of harassment”.

The source for this is a 2012 survey of 874 participants extrapolated to every single console-owning woman in America. This means that what was previously “around three million women have stopped playing games altogether because of harassment” is “84 women stopped playing one online game because of harassment”. While I deeply feel for these women, this is a false, dishonest and huge stretch in the data in what has been uncovered to be a marketing campaign for SteelSeries headsets. False mental health statistics are being pushed to sell you gamer gear.

From this, I would like to branch off into two paths. 2.1 will discuss the extrapolation of the data, while 2.2 will discuss the data itself from the source.

2.1 Extrapolation

One thing I have received criticism for is my ‘lack of understanding’ of statistics. I have seen some people argue that BullyHunters should be allowed to use this figure because 874 is a decent enough participant pool to extrapolate from.

To this, I have three things to say:

1. You are right, 874 participants is a decent participant pool to extrapolate from.
2. The extrapolation was made incorrectly for hyperbole. What was ‘women quitting one game’ became ‘women being pushed out of gaming entirely’ with the use of “have stopped playing games altogether” from ZombiUnicorn.
3. The worse crime is that the extrapolation, as a whole, was done incorrectly.

Extrapolation is a good thing because things like random sampling allow us to make inferences about patterns of illness and behaviour without sampling the entire population. But when we extrapolate, we make very, very, very clear that we are extrapolating, otherwise we are quite simply lying and reporting fake data.

For example, let’s say I conduct a study in a country with ten million under-18s and I take a sample of 1000. 10% show signs of having a mental health problem. How do I report this?

Correct

• One million young people are at risk of poor mental health in [country], with [research] showing prevalence rates of 10% for poor mental health.

Not Correct

• One million young people in [country] have a mental health disorder.

We cannot say “x people have y” without actually assessing them first. What we can say is that these people are at risk based on prevalence rates from a smaller sample, but this isn’t sexy enough to sell headsets.

We especially cannot conduct this sort of behaviour in mental health research. Properly identifying rates of poor and dangerous behaviour allow us to develop evidence-based interventions to deal with problems and properly allocate resources to dealing with the scale of the issue. My field has exceptionally high standards and I fear that some people do not appreciate this.

Before I move on to the data itself, I would like to state two more things about the extrapolation:

• The extrapolation was applied only to American women. This means that if you are not an American woman, you are not being represented and your experiences are not being heard.
• The extrapolation was applied only to American women who own a console. This is a methodological flaw as not only are female PC gamers left out, but it is arguably easier to harass someone on a PC due to the increased likelihood of having a keyboard at your fingertips.

Now I will move on to the data.

2.2 The Data

The source for the data comes from this survey conducted in 2012. I am sympathetic towards the person because it seems that they simply had a hypothesis they wanted to test and carried out a little online survey to test it. What then happened was an advertising campaign latched on to their findings, hyperbolised the findings and lead me to them.

I respect this person and their pursuit of knowledge, but I would argue that this survey has absolutely no place as a reliable source of information.

• The study did not receive ethical approval from a governing body such as a university. This means that there was no one to examine the contents of the survey before administering it, nobody to challenge her on it, nobody to improve it and nobody to take responsibility for her in the case of academic misconduct.
• As she did not receive ethical approval, she is not subject to auditing to ensure that the study was conducted correctly and materials are stored appropriately.
• There is no way to test whether the data has been falsified.
• The survey was not peer-reviewed or subject to any academic scrutiny.
• There is no contact information to get in touch with the person regarding the data.
• The sampling details are far too scarce. ‘Facebook, Twitter and gaming communities’ does not give enough detail about gaming communities. From further research, I found that the study was posted on Reddit’s r/GirlGamers and then posted to r/Gaming following prompts from GirlGamers. The person then admits this is their first time doing a survey and does not understand how sampling works. This really makes me wonder about the ‘random’ nature of the sampling which is important for extrapolation.
• The only real statistical meat in this study is percentages and knowledge of margins of error. The data is absolutely not utilised to its fullest potential and we get a very basic picture of things.

Two things also hinder the ability to use this data in the way it has been used by BullyHunters. In this study, the dominant gaming platform was PC gaming, yet this data is being applied to female console gamers. The second factor is that the only demographic information we have is age and gender. We cannot tell which countries these people are from, yet this data is being used to tell the story of American females.

I will repeat the link to the survey if you are interested in reading it for yourself, but these limitations should certainly be kept in mind regarding its use in BullyHunters. I usually write articles discussing research methodology and breaking down complex statistics to make them understandable to readers, but this survey is just too basic and there is not enough information on methodology to justify doing this here; it would just be a mundane repetition of what is already written.

Summary

• When I requested the citation for BullyHunter’s claim that three million women have been harassed out of video gaming, I was insistently told to watch a stream full of commercials for gaming products. This looked suspicious to me and I was not happy with BullyHunter’s lack of transparency, so I warned people about this commercialisation of mental health research.
• What I thought was the commercialisation of mental health research turned out to be not only false and inappropriate data science, but the exploitation of mental health to sell SteelSeries headsets.
• The claim that ‘three million women have been harassed out of gaming’ turned out to be 84 women quitting one video game because of poor online interactions. This survey was conducted by someone with no research experience and no data analysis experience. The survey did not receive any sort of ethical approval, cannot be audited, was not peer reviewed, the data cannot be verified and the author cannot be contacted.

If you enjoyed reading this, please consider donating to charities such as Samaritans and the American Foundation for Suicide Prevention to help make the world a better place amidst companies trying to profit from false statistics. Thank you.