Very few days go by without a new article describing the limits of published scientific research. The headline cases are about scientists who plagarize or completely fabricate data. Yet, in my experience, most scientists are actually quite ethical, meticulous, hard-working, and really concerned with finding the truth. Still, non-scientists would likely be surprised to know that a large number of scientific studies are actually false. An Amgen study found that 46 out of 53 studies with ¡®landmark¡¯ findings were unable to be replicated. A team at Bayer found a slightly more optimistic picture where 43 out of 65 studies revealed inconsistencies when tested independently. Scientific journals continue to accept articles based on the novelty and projected impact of the submission, yet simulations illustrate how the bias of journals toward publishing novel results likely leads to an environment where most published results are actually false. My home discipline of psychology is currently doing some soul searching as it¡¯s a relatively open secret that many results are difficult to reproduce such that a systematic reproducibility project is taking place.
Crowdsourcing is, and always has been the solution. Indeed, the phrase at the bottom of Google Scholar, ¡°standing on the shoulders of giants¡±, acknowledges that science has always been about crowdsourcing, as every scholar is collaborating with the scholars before them. Findings are not produced in a vacuum and build upon (or challenge) previous findings. Replication by others, which effectively crowdsources verification of results, is at the heart of the scientific method. It is perhaps a sign of the narcissism of our age that scientists feel compelled to believe that they discover things largely independently, such that they feel compelled to attack when their findings are challenged. Yet a willingness to be wrong about something is essential to learning, as we can¡¯t learn to walk without falling or learn about relationships without heartbreak. When science becomes more about ego, career, and grant money, it naturally becomes less accurate. Insisting that findings be crowdsourced solves this. No single study, paper, or research group can prove anything by themselves.
Crowdsourcing is not simply averaging the opinions of the masses, as those who would argue against that straw man would have you believe. Mathematically, crowdsourcing is about reducing the influence of sources of error and there is a great deal of academic research on this topic. A good crowdsourcing algorithm does not weight all inputs equally, but instead seeks to identify clustered sources of error, which explains why aggregating across people with diverse personalities, perspectives or job functions produces better results. Inputs need to have some signal vs. noise and need to have uncorrelated error. The unfortunate assumption in most research is that error is uncorrelated statistical noise that can be dealt with using statistical tests. Yet error also occurs due to the unconscious biases of researchers, the sheer number of researchers trying to find novel findings, the degrees of freedom that a researcher has in trying to prove their hypothesis, the non-randomness of sampling, and the volume of available statistical tests that a researcher can use. Given all these other sources of error, it is no wonder that many findings are false. A good crowdsourcing algorithm would be weighted such that true results would have to be shown by multiple researchers using multiple methods, multiple samples, multiple statistical tests, and multiple paradigms. This requires crowdsourcing as no single person can do all this, and even if they could, they would still represent a single source of error.
Technology enables crowdsourcing to be conducted far more efficiently, as has been proven by successful science crowdsourcing projects like GalaxyZoo, FoldIt, Seti@Home, and psychology¡¯s reproducibility project. Trends like citizen science, the quantified self, open access publishing, and interdisciplinarity improve the diversity of perspectives which mathematically improves the ability to find truth. Every meta-analysis result and Nate Silver¡¯s success in aggregating polls in the last election take advantage of the mathematical principles that underlie crowdsourcing, specifically the certainty that aggregating across sources of error produces more truth. In our daily lives, we all crowdsource knowledge that we are uncertain about, looking for confirmation from multiple independent sources when we are skeptical. This same skepticism serves scientists well and scientists should embrace being wrong, confident that the broader truth will be revealed when all data is aggregated intelligently and all perspectives are valued. Crowdsourcing is not some new technique that threatens to fundamentally change scientific research. Rather, it is an extension of the collective effort of knowledge aggregation that is the heart of science and scientists should embrace it as such.
– Ravi Iyer
Well put, but I’m curious about where and how you see crowdsourcing being implemented within the scientific community? I feel like colleges and universities are long overdue in coming up with centralized systems for integrating knowledge rather than focusing on nifty little niches that provide a fast track to publication, but as a recent graduate I also understand that colleges and universities are businesses, first and foremost, and therefore tend to promote those who occupy nifty little niches.
That also leads me to wonder why professional associations don’t do a better job of facilitating collaboration through crowdsourcing-based platforms? It seems to me like they would be in the best position to accomplish something like this.
Interesting post. I like this part.
“It is perhaps a sign of the narcissism of our age that scientists feel compelled to believe that they discover things largely independently, such that they feel compelled to attack when their findings are challenged.”
It is very exactly written. I think in the first place you have to explore yourself and then scientific truth.
Professional associations, unfortunately, have very little incentive to change with technology. My guess is that citizen science, data science, and pressure from the outside world will have to force academia to change as there are too many entrenched interests for it to change itself.
How, on an institutional/journal level, do we give editors incentives to valorize studies yielding null results, by publishing them? Researchers who show strong evidence of no evidence should be heros.
I don’t see convenient information so I’ll post this here. A bit of crowdsourcing.
Have you considered “elitism” as a moral foundation?
I’m coming from the perspective of Social Dominance theory here, and elitism looks like a moral response to me, punishing low status people just as an authoritarian would punish nonconformists. If you look at Social Dominance research they show that elitism is heavily group based, in terms of high/low status instead of ingroup/outgroup. Fundamental to its operation is prejudice against low status groups. High SDO people dislike it when low status people try to include themselves in high status groups, and they absolutely hate when low status people band together to restrict the behavior of high status people. They also tend to strongly believe that one’s status is deserved from a moral perspective.
Elitism in the form of Social Dominance seems to be a very well established, evolved psychological construct that acts as the direct opposite of care/fairness based egalitarian morality. High SDO thinking tends to justify the oppression of low status people and prevent care/fairness reactions to such oppression.
SDO theory doesn’t concern itself with morals, but when I think about my real-world experiences with elitist behavior it sure involves a heck of a lot of punishment, and I think it actually fits the criteria for costly, secondary punishment.
Think of the classic example of people who are rude to waiters. This is stupid from a self-interested point of view, because being rude rather than polite in a trivial social interaction gains you nothing. The only result is worse service. It looks a lot like punishing the waiter for being low status, and it looks costly… but from a group perspective it may be an evolutionary benefit.
Think of the elite as a group of people who can benefit by excluding low status people and being uncaring and unfair toward them. Basically, a minority who can monopolize resources for themselves IF they cooperate in being uncaring, unfair, and exclusionary. Any individual member of the elite benefits the most from free riding – letting everyone else enforce elitism, while personally being the “nice guy” who doesn’t actually get in confrontations with low status people. If you’ve spent any time around elitist groups you may have seen that this kind of free riding is actually punished – elitists have a disdain for members of the elite that refuse to join in their criticisms of low status people.
The interesting thing about elitism, which differs from the other moral foundations, is that it’s not directed at highly specific behaviors like “betraying the tribe” or “being unfair to someone.” Elitist bad attitude is regularly directed at low status people for little to no immediate reason, simply as a way to reinforce their low status. However, there is some regularity in what causes people to demonstrate especially high levels of elitist disdain.
1. Elitists do show disdain for low status people who act submissive (like waiters), but they are far more punitive when low status people try to join their group or demand high social status. Basically, they punish people who rise above their station. I’m thinking of the people who are fine with Mexican immigrants as their gardener or maid, but go nuts when a middle class Mexican family moves into the house next door.
2. Elitists don’t like customs and activities which are associated with low status people, and will punish members of the elite for participating in it (think punishments for nerdy activities, for example). At first you might think this is an example of loyalty/authority morals, but very anti-authoritarian libertarians are actually pretty fond of looking down their nose at “low status” activities.
3. Elitists hate rules restricting the elite for the benefit of low status people with the fiery passion of a thousand suns. Affirmative action is the classic example of the worst thing in the universe from their perspective, but in my experience their hatred of environmental and other economic regulation is also intense because it restricts the most successful people. They have a strong moral response to these things. Some of why libertarians don’t like affirmative action could be liberty morals, but frankly a lot of these regulations aren’t enforced very strongly and have zero impact on them personally. The strength of the hostility is more easily explained by costly punishment of low-status people who are being uppity and demanding fairness.
4. Elitists are positively eager to jump on anyone who is complaining that they’ve suffered, and can be seen as having “deserved it” due to their own failure. It can be utterly silly. I remember when Apple Maps came out, every time I saw a blog post complaining about people getting lost from using it there would be a lot of posts heaping scorn on anyone who could possibly get lost from following a map. This didn’t just come from Apple fanboys, either, I’ve seen the same kind of response on many many subjects. It comes from recognizably high SDO people, not recognizably authoritarian/high RWA people. It’s as if they’re actively eager to classify people as having failed due to their own bad personal characteristics, and then attack them for it. People hang around blogs and news sites and do nothing but find victims to blame.
When I compare moral foundations to SDO it looks like care/fairness is associated with low SDO, but high SDO just shows up as “less caring/fair.” This doesn’t feel right to me when I think of my extensive experiences with libertarians. According to Moral Foundations they’re just less moral than everyone else except for the liberty moral. This means libertarians should engage in much less costly punishment of other people, unless there’s obvious oppression going on. They should, in other words, have a very “live and let live” philosophy that doesn’t involve condemning other people.
My real world experience is that libertarians condemn other people ALL THE TIME, in very moralistic terms, including many situations where liberals wouldn’t condemn anybody. The classic liberal vs. libertarian debate is over care/fairness. Sometimes libertarians oppose care/fairness on the basis of liberty, because they view care/fairness punishment as oppressive, but in my experience it’s a lot more common for them to appeal to elitism. When it comes to low status people, the libertarian attitude toward care/fairness responses is not “I don’t care about them, do what you want as long as it doesn’t involve me.” It’s usually a very aggressive argument that they do not deserve it because they are the ones to blame for their own problems. Libertarians are willing to be quite confrontational in blaming low status people for their problems, in other words they’re deploying that blame as a secondary punishment against anyone supporting care/fairness.
Obviously this is a very “un-PC” suggestion. You are going to get high SDO and economic conservatism highly correlated with the elitist foundation, and conservative politics modestly correlating it. Elitism is one of the worst things you can be accused of in American society. Unlike “loyalty” for example, which is mainly about punishing those who don’t conform but can still be seen in a positive light, conservatives will scream bloody murder at anyone who puts in elitism as a moral foundation.
It’s also rather hard to test using the methodologies I’ve seen in moral foundations studies. Elitism looks like it behaves as a moral psychologically, but people do not subjectively experience it as a moral (though that could just be a western cultural blind spot). People rationalizing their elitism often look rather “anti-moral”, arguing that their targets are to blame for their failures as if it goes without saying that one should condemn or belittle those who fail. The Social Dominance construct is very well established but it asks about values, not moral dilemmas. I think it would be very hard to construct a questionnaire which would get people admitting that they’d be rude to the waiter. Peoples’ idea of what is or is not low status can also vary a lot from culture to culture.