I have recently been following a discussion in my discipline about the peer review process, which led me to this very interesting paper about the history of and alternatives to the peer review process in psychology.
At the same time, I’ve been working with colleagues on a paper about experiential vs. material purchasing styles, for which we have found convergent correlations all suggesting that experiential purchasers are dispositionally motivated towards seeking new, stimulating experiences to promote positive emotion, while material purchasers often seek to avoid negative emotions. This is supported by the fact that, in the YourMorals.org dataset, experiential purchasers report higher levels of openness to experience, lower levels of neuroticism (both measured by the Big Five Personality Inventory), and lower levels of disgust (as measured by the Disgust Scale). The disgust finding does not necessarily fit with the idea that experiential purchasing is related to seeking new experiences, unless one looks at the literature on disgust. In particular, this study theorized about such a relationship and confirmed it by reporting correlations between disgust and big five personality dimensions.
It occurred to me that I could contribute to the original studies’ findings, by examining the same correlations in our dataset, using a more diverse and far larger sample, and perhaps even including some internal cross-validation. The results are summarized in the table below.
Disgust Scale Correlations with Big Five Personality Traits
The main hypothesis of the original study actually dealt with the two robust relationships found in our dataset, specifically that disgust is negatively related to openness to experience and positively related to neuroticism. In all, these two relationships stand out as robust across groups and in both studies. Interestingly, the correlation between openness to experience and disgust is weaker in the two most ‘rational’ groups, edge.org and libertarians, which might be worth pursuing later. Given the smaller sample size and restricted diversity of the original study, I’d be inclined to say that conscientiousness and agreeableness are not robust correlates of disgust, though this could be an effect of the fact that yourmorals.org uses a different measures of Big Five personality traits from the original study.
Can I publish this finding? It’s only correlational and says nothing about causality. It really doesn’t say much that is new, but rather confirms the original study, more or less. Still, the 26 papers which cited the original study would be slightly more improved if they could cite this finding as well, since it’s the same basic study with a different (larger and more diverse) sample. This is where the discussion of the peer review system converges with this analysis. According to this paper, “many natural science fields operate on a norm that submissions should be accepted unless they are patently wrong.” In contrast, psychology papers are often rejected, not because they are wrong, but because they are not interesting or novel enough.
The paper and the listserve discussion bring up many points related to this, but one relevant one to this finding is that it is hard to build a cumulative science when you don’t reward replication, but instead reward novelty. The end result is that you end up with a series of slightly different perspectives on the same subjects, all named differently, where authors are constantly trying to come up with something new rather than building on something existing. This may help academics, but it makes it very difficult for these theories to be used in the real world. Any research on humans is likely flawed in some way. Can anybody do double-blind experiments on representative samples of people with behavioral measures? The public is wisely skeptical of any social science finding as are academics…but the solution might lie in publishing more replications rather than in restricting the publication process toward the mythical goal of the perfect, novel study. No single study proves anything when dealing with research on people. It’s the convergence of lots of studies that might potentially be convincing enough to outsiders.
– Ravi Iyer
ps. if anyone wants to write this up and publish it traditionally, feel free to contact me