Pluckrose et al. Hoax Papers

Fabio Votta
4.10.2018

Here I want to analyze the Rankings of Journals that received hoax
papers by Boghossian, Lindsay and Pluckrose. You can read about the
story
here
and
here
in their own words.

Packages

Loading in some packages.

# Install these packages if you don't have them yet
# devtools::install_github("favstats/tidytemplate")
# install.packages("pacman")
pacman::p_load(tidyverse, rvest, sjrdata, janitor, ggthemes)

Load Data

Some quick webscraping to get the details on the papers created by
Pluckrose et
al.

areo_html <- read_html("https://areomagazine.com/2018/10/02/academic-grievance-studies-and-the-corruption-of-scholarship/")
journals <- areo_html %>% 
  html_nodes("em:nth-child(1) a") %>% 
  html_text()
titles <- areo_html %>% 
  html_nodes("p") %>% 
  html_text() %>% 
  .[str_detect(., "Title: |Title. |Rubbing One Out: ")]  
short_titles <- areo_html %>% 
  html_nodes("p:nth-child(309) strong , hr+ p strong , p:nth-child(67) strong") %>% 
  html_text() %>% 
  .[!(str_detect(., "Part"))] 
status <- areo_html %>% 
  html_nodes("p") %>% 
  html_text() %>% 
  .[str_detect(., "Status: ")]  
areo_data <- tibble(journals, titles, short_titles, status) %>% 
  mutate(titles = str_remove(titles, "Title: |Title. ")) %>% 
  mutate(status = str_remove(status, "Status: "))
tidytemplate::save_it(areo_data)

Scimago Journal & Country Rank Data

Next, I combine the article data with the Scimago Journal & Country
Rank Database to get Journal Rankings.

Merging and Overview

sj_data <- sjrdata::sjr_journals %>% 
  filter(year == 2017) %>% 
  rename(journals = title) %>% 
  ### renaming journals so they match the article
  mutate(journals = case_when(
    journals == "Sexuality and Culture" ~ "Sexuality & Culture",
    journals == "Affilia - Journal of Women and Social Work" ~ "Affilia",
    journals == "Women's Studies International Forum" ~ "Women’s Studies International Forum",
    journals == "Gender, Work and Organization" ~ "Gender, Work, and Organization",
    journals == "Glq" ~ "GLQ: A Journal of Gay and Lesbian Studies",
    T ~ journals
  )) %>% 
  full_join(areo_data)

Let’s take a look the papers and journals, sorted by journal rank. There
are two metrics in the SJR Database that we’ll focus on:

• SCImago Journal Rank (SJR indicator)

It expresses the average number of weighted citations received in the
selected year by the documents published in the selected journal in the
three previous years, –i.e. weighted citations received in year X to
documents published in the journal in years X-1, X-2 and X-3. See
detailed description of SJR
(PDF).

• H Index

The h index expresses the journal’s number of articles (h) that have
received at least h citations. It quantifies both journal scientific
productivity and scientific impact and it is also applicable to
scientists, countries, etc. (see H-index wikipedia
definition).

sj_data %>% 
  filter(!is.na(status)) %>% 
  select(titles, status, journals, rank, sjr, h_index, categories) %>% 
  arrange(desc(sjr)) %>% 
  knitr::kable()

One limitation becomes notable at the end of the table: The journal
Sociology of Race and
Ethnicity
is a very new journal and has no ranking yet. So the paper that was
submitted here is not included in the following plot.

It’s good to have an overview of all the created papers. But how do the
journals that they were submitted to rank in comparison to other
journals in the field? That is what I attempt to show next.

First, I filter the dataset to only include the journals that received
hoax papers and that are tagged as belonging to Gender/Cultural
Studies. Next, I collapse the (pending) status of the papers into the
following categories:

1. Accepted or Published

2. Reject/Revise and Resubmit

3. (Desk) Rejected/Retired

4. Under Review

sj_data <- sj_data %>%  
  filter(str_detect(categories, "Gender Studies|Cultural Studies") | !(is.na(status))) %>% 
  mutate(status = case_when(
    str_detect(status, "Published") ~ "Accepted or Published",
    str_detect(status, "Reject and Resubmit") ~ "Reject/Revise and Resubmit",
    str_detect(status, "Revise and .esubmit") ~ "Reject/Revise and Resubmit",
    str_detect(status, "Accepted") ~ "Accepted or Published",
    str_detect(status, ".ejected") ~ "(Desk) Rejected/Retired",
    str_detect(status, "review") ~ "Under Review",
    str_detect(status, "Retired") ~ "(Desk) Rejected/Retired",
    T ~ "Other Journals"
  )) 
sj_data %>% 
  janitor::tabyl(status) %>% 
  knitr::kable()

Now, it’s time to take a look at how the journals that received hoax
papers compare to other journals in the field.

Scatterplot

sj_data %>% 
  filter(status != "Other Journals") %>% 
  ggplot(aes(h_index, sjr, color = status)) +
  ## grey background
  geom_point(data = sj_data %>% 
               filter(status == "Other Journals"), 
             color = "lightgrey", alpha = 0.6) +
  ggrepel::geom_text_repel(data = sj_data  %>% 
                             filter(status == "Other Journals") %>% 
                             filter(h_index >= 60 | sjr >= 1.7), 
                           aes(label = journals),  show.legend = F, 
                                  color = "lightgrey", alpha = 0.5) +
  geom_point(size = 1.5, shape = 15) +
  scale_color_manual("", values = c(ggthemes::colorblind_pal()(5))) +
  coord_flip() +
  theme_minimal() +
  ggrepel::geom_text_repel(data = sj_data  %>% 
                             filter(status != "Other Journals"), #%>% 
                           aes(label = journals), show.legend = F, force = 4) +
  labs(y = "SCImago Journal Rank (Average Prestige per Article)", 
       x = "Journal h-index", 
       title = "Rank of Journals that received Hoax Papers by Pluckrose, Lindsay and Boghossian",
       subtitle = "Compared to Journals tagged with 'Gender/Cultural Studies'\n",
       caption = "Data: Scimago Journal & Country Rank Database\nAccessed using {sjrdata} package\nfavstats.eu; @favstats") +
  theme(
    legend.text = element_text(size = 10),
    axis.title = element_text(size = 12, face = "bold"),
    plot.title = element_text(size = 18, face = "bold", hjust = 0),
    plot.caption = element_text(size = 12),
    legend.position = "bottom",
    legend.title = element_text(size = 10), 
    #axis.ticks.length = unit(3, "cm")
    ) +
  guides(color = guide_legend(override.aes = list(size = 4)))

tidytemplate::ggsave_it(gender_studies, width = 12, height = 9)

Now, how bad is the damage? Well, the most notable and impactful
journals that published or accepted hoax papers were Sex Roles (with
the paper “An Ethnography of Breastaurant Masculinity”) and Gender,
Place, and Culture (with the paper “Human Reactions to Rape Culture
and Queer Performativity in Urban Dog Parks in Portland, Oregon”).
Other than that, quite a few of the journals that accepted or published
hoax papers are found in the lower left quadrant with comparitively low
impact.

Boxplot

Let’s compare the journal rankings directly now, with the help of box-
and violinplots.

sj_data %>% 
  rename(`SCImago Journal Rank` = sjr,
         `Journal h-index` = h_index) %>% 
  select(status, `SCImago Journal Rank`, `Journal h-index`) %>% 
  gather(key, value, -status) %>% 
  ggplot(aes(status, value, color = status)) +
  geom_violin() +
  geom_boxplot(width = 0.4) +
  scale_color_manual("", values = c(ggthemes::colorblind_pal()(5))) +
  coord_flip() +
  theme_minimal() +
  facet_wrap(~key, scales = "free_x") +
  labs(y = "Journal Rank", 
       x = "", 
       title = "Rank of Journals that received Hoax Papers by Pluckrose, Lindsay and Boghossian",
       subtitle = "Compared to Journals tagged with 'Gender/Cultural Studies'\n",
       caption = "Data: Scimago Journal & Country Rank Database\nAccessed using {sjrdata} package\nfavstats.eu; @favstats") +
  theme(
    panel.spacing = unit(2.4, "lines"),
    legend.text = element_text(size = 8),
    axis.title = element_text(size = 10, face = "bold"),
    plot.title = element_text(size = 14, face = "bold", hjust = 0),
    plot.caption = element_text(size = 10),
    legend.position = "bottom",
    legend.title = element_text(size = 8), 
    ) +
  guides(color = F)

tidytemplate::ggsave_it(gender_studies_boxplots, width = 10, height = 6)

Some interesting patterns are revealed here. Seems like journals that
rejected papers were somewhat better ranked than those that accepted
hoax papers (more evident with the Journal h-index). It’s also worth
noting, that the majority of the other journals in the field of
Gender/Cultural Studies do rank considerably lower than those that
Pluckrose et al. managed to get their papers published in.

So what are some conclusions to draw from this?

What Pluckrose et al. accomplished by getting seven of their papers
accepted in peer-reviewed journals is definetely remarkable. Reading
through the papers, one can’t really come away and think that this is
anything less than sloppy. The journals that fell for the hoax should
look into making the editorial board clean up their standards. As a
quant guy, I particularly cringed at the “statistical analysis” in the
“Agency as an Elephant Test for Feminist Porn” paper.

Having said that, I think the conclusions that the authors draw from
their hoax project are very far fetched. In the video they published
along with the
papers,
they say:

“[…] corruption is pervasive among many disciplines including
women’s and gender studies, feminist studies, race studies,
sexuality studies, fat studies, queer studies, cultural studies and
sociology.”

To claim that entire fields of studies are “corrupted” because of this
incident seems unwarranted to me.

There are enough journals that rejected the hoaxes and papers in other
fields are retracted all the time
because of fraud or other shortcomings. As an example, this
video details an egregious
case in a nanoscience journal where images were blatantly forged,
something that would have been obvious to any non-nanoscientist. I
understand that this might come off as an attempt to whatabout the issue
away. But nobody would come up with the idea that corruption is
pervasive in nanoscience because of this very blatant case of forgery
and I don’t believe anyone should think the same of gender studies and
related fields.