My Expert Is Better Than Your Expert

The world in which we live is a complicated place and none of us has time to learn everything there is to know about every topic out there. As such we all must rely on the opinions of others when it comes to a vast many thing in our lives, and where possible we like to turn to those people who are seen as experts in the field and base our opinions, at least in part, on what they have to say. The trouble is that not all experts are created equal and for every expert saying one thing, you can probably find another expert saying the exact opposite. So how are you, as a good skeptic, meant to work out which experts are worth listening to and which experts you should rightly be skeptical of?*

A lot of research has been done into the question of what makes someone an expert by psychologists, philosophers, epistemologists and neuroscientists, to name but a few. Even Plato raised the question way back in the fifth century BC, though alas he did not come up with an answer. And that’s probably because it is not a simple issue with a definitive answer. However, these researchers have come up with several things that you can do that will help you determine whether you are dealing with someone who can be considered a genuine expert or just a crank in experts clothing.

* This is not a purely academic question as the legal systems of many countries are based upon the idea of having non-experts evaluate the claims of experts. Scott Brewer presents an interesting analysis of this problem, noting that you would not rely on a non-mathematician to resolve a dispute between two rival groups of mathematicians debating the possible resolution of Fermat's Last Theorem, so why do we expert non-scientists to correctly evaluate scientific evidence in a court of law? See the reference section for more details.
 
Watch out for the Nobel Effect

The Nobel Prizes are awarded to those who make outstanding breakthroughs, discoveries or advancements in their chosen field. These are people at the very top of their game, people who have put a lifetime worth of work and dedication into complex and highly competitive fields, like physics, chemistry and medicine. A Nobel Prize will open doors you never knew were there, will make your career and will set you head and shoulders above your peers. The winners of these prestigious awards are undoubtedly some of the smartest people on Earth and are rightly recognised as experts in their field, and right there is the first thing you should keep in mind. Once someone wins a Nobel Prize their opinion and advice is often sort on a wide range of topics. This is the Nobel Effect, so named by professor of philosophy and evolutionary biology Massimo Pigliucci and relates to people being asked questions on a wide variety of matters just because they have shown themselves to be experts in one area.

The problem with this is that research shows that expertise is not transferable, even between areas that are closely related. Just because you are an expert in, let’s say, particle physics it does not mean you can speak with authority on matters such as nanotechnology, stem cell research or climate change. Now while there is supporting evidence that suggests the ability to learn is itself a portable skill many of the skills, techniques and methods you learn in one field of enquiry simply do not transfer over to other fields. Perhaps a clearer way to picture this requires us to leave the realm of science for the moment and take a look at the musical arts.

A concert pianist could rightly be classed as an expert when it comes to playing the piano. They have years of experience behind them and have studied hard to reach the position they have now attained. They are not only skilled performers but usually possess a prestigious knowledge of musical theory as well; they are musicians at the top of their game. However, if you were to take their piano from them and give them a violin instead it is unlikely, they would perform any better than an amateur. True they would undoubtedly have a better understanding of music in general, would be able to read music as easily as you can read this and in all likelihood would pick up the violin faster than someone without their background, but they would not be an expert and it would again take them many years of study to reach the level at which they play the piano, if they ever attain it. The point is that the fact that someone is recognised as an expert is not what matters. What matters is that they are an expert in the subject under discussion. Listening to someone with a PhD, or even a Nobel Prize, in chemistry give their opinion on Big Bang cosmology is like listening to a concert pianist try to play the violin, they may appear to know what they are talking about but without direct experience in that field, their opinion should carry no more weight than your own.

​

Examine Your Expert's Arguments And Those Of Their Rivals

Wherever possible you should always examine the arguments and supporting evidence of not only the expert in question, but of their rivals as well. But wait a minute, didn’t I start this piece by saying that the reason we turn to experts in the first place is because we don’t have time to study up on every possible topic ourselves? Well yes, I did and that is an issue. If you are turning to an expert for their opinion, the chances are that you are doing so because you don’t know enough about the subject at hand to reach a conclusion on your own. As such, you are highly unlikely to be able to correctly identify the reasoning behind the conclusions of your expert to ascertain if they are valid or not; neither are you likely to know the counter arguments to these conclusions nor the rebuttals to these counter arguments. Now of course there will be times when you do know enough about the subject at hand to work out which of two experts is more likely to be correct, but when this is not the case what do you do?

Alvin Goldman, professor of philosophy and cognitive science at Rutgers University in New Jersey, suggests that there are two ways by which a novice, or non-expert, can evaluate the arguments of an expert. There is what he calls direct argumentative justification, which is where you know enough about the subject under discussion to evaluate if an expert’s arguments are accurate or not, and indirect argumentative justification, where you use other methods to decide which expert before you are more likely to be correct. Now we have already talked about when the direct approach applies and when it is pretty much useless to you, so let’s focus on the indirect approach. Indirect argumentative justifications really apply best to debates between experts on opposite sides of an issue. Without knowing many details about the topic under debate a novice can use the indirect approach to reach a conclusion as to which of the experts is the one, they should be listening to. This partly involves keeping an eye out for fallacies in the arguments made as well as other indicators that one of the experts may be making unfounded or unscientific claims. The main part of this approach however deals with how the experts on either side of the debate handle the back and forth, argument, counter argument, rebuttal nature of the discussion.

Let’s say we have two alleged experts in debate about a subject you know little about and you are trying to decide which one you should listen to. Jim, our first expert, presents an argument and Bob, our second expert, immediately fires back with a solid rebuttal of that argument as well as reasons why Jim’s evidence, or at least his use of it, is flawed in some way. However, when Bob makes an argument Jim’s responses fail to rebut his claims and he is unable to demonstrate flaws with Bob’s evaluation of the evidence. This would indicate that Bob probably has a better understanding of the subject at hand and is more familiar with the evidence in support of his arguments. Likewise let’s say that Bob’s responses to Jim’s arguments come quickly, are smoothly presented and well thought-out. However, Jim’s replies to Bob seem hesitant, confused and ad hoc in nature. This would imply that not only is Bob strongly aware of his own arguments but that he also has a good understanding of the counter arguments as well, whereas Jim may be encountering some of Bob’s arguments for the first time. Again, from this you could use the indirect method to conclude that Bob is probably the guy to listen to.

But wait, I am sure you have all recognised the big flaw in this approach. Now while this indirect approach focuses on the apparent strength of the arguments presented it also places a lot of weight on the rhetorical skills of the debaters involved. The problem with this is that it doesn’t necessarily give you an accurate picture of the evidence and arguments on either side of the debate. Let’s say that Jim, who came out worse in our debate, is a professor of quantum physics and spends his days in a lab conducting research and writing papers for prestigious science journals. Bob meanwhile is a proponent of quantum healing and spends his days travelling around the country, giving interviews, debating scientists and trying to convince people that if they just want it bad enough, they can get anything they want due to the nature of quantum entanglement.

​

Well clearly, Bob is going to have far more experience in debates than Jim and so is going to come across better. He has probably also encountered the same counter arguments to the points he makes time and again and so has had time to practice and craft his replies. Jim on the other hand finds himself faced with a complete misunderstanding of his chosen field and a barrage of illogical and misconceived ideas that he has never heard before and thus finds himself struggling to know where to start in addressing them. Just because Bob is the better debater it doesn’t mean that he is the expert you should trust, which brings us nicely to our next point.

​

Do Other Experts Agree With Your Expert?

Ok the first thing to make clear is that science is not a popularity contest. Just because loads of people happen to believe in something does not necessarily mean it is true, it is the evidence that really matters. That aside when it comes to ascertaining if you are dealing with a real expert or not a lot can be said for scientific consensus. In the previous point Bob, a far more skilled debater than Jim, came out on top when we used indirect means to decide which of the two, we should be listening to. However, if we were to take a look at the field of quantum physics, we would find that the vast majority of people working in it, especially those who hold PhDs in the subject, agree with Jim and consider Bob to be a crank.

Imagine you get home tonight and find water all over the floor of your kitchen. You call a plumber and he come out, takes a look and tells you that you have a leaky valve that will need replacing. Being a skeptic, you think it best to get a second opinion and so call another plumber to get their opinion… and then just to be sure you call eight more. Ten plumbers have now taken a look at the problem and of those ten nine tell you that you have a leaky valve that will need replacing. The remaining plumber tells you that the problem is with your boiler and that you need to get a new one at a cost of several thousand pounds. You now have two possible explanations as to what is responsible for the water on your kitchen floor. You know nothing about plumbing, that’s why you called ten plumbers in the first place, so which explanation should you go with?

Now I am sure it is completely obvious that it would make far more sense to go with the consensus and replace the valve than to get a new boiler. Nine trained plumbers looked at the problem, evaluated the evidence and independently came to the same conclusion. Yeah ok so one plumber looked at the same evidence and came to a separate conclusion; however, it is far more likely that he is mistaken, or simply trying to con you, than the other nine reaching the same incorrect conclusion. Well exactly the same approach can be used when evaluating the expertise of two people on either side of, say, the man-made global warming debate. Around 97% of climate experts agree that humans are causing global warming and as such, even if you knew nothing else about this issue, you would be justified in throwing you support behind the expert presenting this conclusion rather than the one stating the opposite.

​

Is Your Expert Really An Expert?

Having consulted ten plumbers and having had nine of them tell you that you need to replace a leaky valve you may find yourself questioning the credentials of the one that came to a different conclusion. Is it possible that he is some sort of plumbing genius who saw something the other nine didn’t, or is it more likely that he doesn’t really know what he is talking about or that he is trying to pull a fast one? Heck maybe he is not even a qualified plumber at all.

When it comes to evaluating experts, it pays to check if your expert is in fact an expert to start with. We have already looked at the Nobel effect where you have experts in one field talking about things in a different field, but in that case, you have people who are indeed recognised experts but whom are simply discussing matters outside of their area of expertise. What we are looking at here is whether your expert can even make the claim to be an expert of any type in the first place. You investigate this in a couple of ways, firstly by looking at the credentials the expert holds and secondly by examining what other experts in the field have to say about them.

At first glance checking out the credentials of your expert seems simple enough, but let’s jump back to Jim and Bob for a moment. When you take a look into the credentials of these two experts you quickly find that both of them hold a PhD in quantum mechanics. Well that’s no help then; they both have the same qualification so there is no real way to use this to judge which one is most likely to be the true expert. But let’s take a closer look as not all PhDs, just like experts in general, are created equal. Investigating further, you find that Jim got his PhD at Cambridge University and his bachelors and master’s degrees at Oxford University. That’s pretty impressive; you need stupidly good grades to even get into those universities. On the other hand, Bob got his PhD at an unaccredited correspondence university that, when you do a Google search, turns out to be a shed in someone’s garden. Now being fair to Bob we will assume that he didn’t just purchase his PhD online and did in fact put a lot of work into it, but either way it is hard not to conclude that Shedsville University probably has lower standards than two of the UKs most prestigious academic institutions. Now this may seem like an elitist approach to things, and to an extent it is, but the truth remains that it does indeed take more effort, skill and academic muscle to attain a degree at some universities than it does at others, and this should at least be taken into account in your evaluation. And this ignores the fact that some accreditation's can simply be purchased for a small fee. For example, you can become a professional member of the impressive sounding American Association of Nutritional Consultants for just $60 a year, no qualifications in nutrition required; they will even send you a certificate to hang on your wall. (Dr Ben Goldarce, to show how worthless credentials like this can be, did this for his dead cat Hettie, making her a certified professional member of the American Association of Nutritional Consultants.)

As well as checking out your experts’ credentials, you can also investigate what other people in their field have to say about them. One way to do this is to see how often their work is referenced by others. In science and other areas of academia when you write a paper that you plan to submit for peer review you need to include references to the evidence you are using to support your research, and this generally consists of papers written by others in your field. As a rule of thumb the more often a paper is cited as a reference the more highly it is regarded by others in the field. As such looking at how often your expert’s papers are referenced by others is a good way to get an idea of how positively other experts think of them. And don’t worry, this is easy to do. A quick search on Google Scholar will provide you with a link showing how often any given piece of work has been cited by others. (Note that you may not get a complete picture from Google Scholar, however, it is a great place to start and a lot easier to use than many of the more academic search engines and should be more than enough to point you in the right direction.)
 
The other way to get an idea of what other experts think of your expert is to simply search for what people have to say about them. We already talked about examining the arguments made by both your expert and their rivals but here we are looking more at what people have to say about the expert themselves. Now obviously anyone who has reached a suitably high point in their career to be classed as an expert is probably going to have ruffled a few feathers along the way and if you look hard enough you will be able to find negative things said about anyone. What is important is to look at who is saying what about them. If you find that all the people who you know are recognised experts in the topic at hand view your expert as a crank who doesn’t know what they are talking about then it pays to take note of this. Alternatively, let’s say your expert has conducted research into the dangers of smoking and you find that, while the scientific and medical communities sing  their  praises, the tobacco industry doesn’t have a good word to say about them, well this might make you wonder about the ideological biases of those criticizing them and whether their complaints  are legitimate. Which once more brings us to our next point.

​

Does Your Expert Have Any Prejudicial Biases?

Everyone has biases, in fact they play such a big part in making us who we are that I spend a fair amount of time in the second part of my book, Building Your Skeptical Toolkit, talking about a few of the biases we all have. Your expert, no matter who they are or what side of the issue they are on, will have their own personal biases, for good or ill.  They, just like us, will have their opinions about the world that are shaped, in part, by their personal experiences, their relationships with others, financial incentives, their level and type of education and any philosophical or spiritual beliefs they may have. There is just no getting around the fact that your expert will come to the table loaded down with biases and other influencing interests that will affect how they see things.

In his paper Alvin Goldman used the following example that I think highlights the influence of biases perfectly. In 1999 the Journal of American Medical Association (JAMA) published a paper that looked at the relationship between the results of clinical trials of new oncology drugs and whether they were funded by non-profit organisations or the pharmaceutical companies that produced the drugs. Perhaps unsurprisingly, they found a statistically significant difference between the two, with those trials funded by non-profits finding unfavourable results 38% of the time whereas those funded by the pharmaceutical companies only found unfavourable results 5% of the time. Clearly those people running the trials were biased, in one way or another, by where the funding for the trial was coming from. This is a little alarming, though sadly not at all unexpected.

So, if all experts are going to be biased in some way then what is the point in looking into whether they hold any prejudicial biases or not? I mean we already pretty much know the answer will be yes, they do? Well again not all biases are the same and not all experts allow their biases to influence them to the same degree. Let us once more take a look at Jim and Bob to see what I mean.

Jim, as previously mentioned, is a professor of quantum physics and has spent years working in the field. He is well paid by the university at which he works and has received a number of large grants from non-profit organisations, private companies and even the government in support of his research. He is a recognised name in the quantum physics community and has published numerous papers that have put his thoughts and opinions on the subject out there for the world to see.  He has also written a successful book aimed at the general public on the bizarre nature of the quantum world that has made him a fair amount of money and which the Discovery channel is interested in turning into a documentary series. Jim considers himself a humanist and believes that ultimately everything in the universe is the result of natural forces.

Clearly Jim is not completely unbiased when it comes to the subject of quantum physics. He has given most of his life to the topic and makes his living teaching and researching it. He also has considerable incentives, both financial and personal, for wanting his thoughts on the matter to be correct. He will undoubtedly approach any discussion on the subject from the point of view of both his scientific and personal beliefs and with a mind to the fact that he built his reputation on a specific set of ideas.

But what about Bob? Well Bob runs a business based upon the idea that quantum physics can be used to heal our bodies, give us a better life and help us attain those things that we most desire. He sells numerous products and services based upon these ideas and has also written a number of bestselling books on the subject. He is a widely recognised face both in the alternative medicine community and on day time TV programs, on which he regularly appears to show off his latest product. He uses the sell-out talks that he gives all over the country as a platform from which to promote his all natural, quantum health cures for everything from AIDS to cancer to trapped wind. He is a strong believer in the idea that doctors have a vested interest in keeping us sick and that vaccines cause all manner of illnesses and should be avoided. He believes that the universe has given us everything we need to heal ourselves naturally and that the use of manufactured medicine is a sin.

Like Jim, Bob clearly has his biases. He makes a good deal of money promoting his ideas and his reputation, livelihood and public image are all tied in with his view of quantum mechanics. He has strong financial and ideological reasons for promoting his ideas and on top of this his own feelings of self-worth and spiritual importance depend heavily on his understanding of quantum physics being correct.

Now I have purposefully given Bob more questionable biases that Jim in order to make a point, though in reality it is unlikely things will always be this clear. Whereas both Jim and Bob have financial incentives for the ideas they hold about quantum physics Bobs are far more direct. He makes his money by directly promoting his ideas, whereas Jim is paid to teach and conduct research and not for the specific opinions he holds. Everything Bob does, from writing books to appearing on TV to giving talks, is used to help sell his products while Jim uses his books, papers and public appearances to educate and promote understanding. While Jim does have a vested interest in his ideas being correct if they were to be disproven, it would not be the end of his teaching career and would likely only affect the direction of his future research projects. If he accepted his mistakes and put his support behind the new findings, then he would undoubtedly retain much of his high reputation within the quantum mechanics community. If it became common knowledge that Bob’s ideas were wrong on the other hand it would directly affect how much money he makes and his ability to sell his products, as such he has a much greater investment in the appearance of being right rather than in actually being right.

Where possible you should look at the nature of the biases and interests that influence your expert. In the JAMA paper we saw a prime example of how funding can influence the outcome of clinical trials even within communities that place a premium on scientific rigour and the value of evidence. I am sure it is not hard to accept the possibility that those making a direct profit from the ideas they support may have an interest in supporting them regardless of what the evidence says. Likewise, some in the anti-vaccination movement have deep seated ideological issues that drive their arguments against vaccines even when the scientific evidence is against them. Many of those who strongly disbelieve that any kind of global warming is taking place have religious reasons for doing so, and the same undoubtedly goes for those who do not accept the theory of evolution. In both these cases their opinions on scientific matters are inextricably linked to their religious beliefs and their view of their place in the universe. To them accepting that they may be wrong about these scientific questions would also require them to accept that they may also be wrong about the spiritual matters as well and most are unwilling to take this step.

Biases will directly affect what your expert says and the arguments they make. If someone’s biases appear to be driving their arguments rather than the evidence, then you may have a valid reason for seeking your expertise elsewhere. However, it is important to note that having strong prejudicial biases does not mean someone cannot be an expert or that the arguments they are making are not valid and correct. As with the other points on this list evidence of questionable biases should be seen as an indicator that the expert in question should be approached with skepticism and not as grounds for dismissing them out of hand.

​

Does Your Expert Have A Good Track Record?

This approach is probably the most straight forward and easiest to understand. It reminds me of an old joke. A woman phones a plumber when her washing machine breaks down. The plumber arrives, studies the machine for a few minutes and then proceeds to give it a hard whack with a hammer. The washing machine starts working immediately and the plumber presents the woman with a bill for £200. “Two hundred pounds?” exclaims the woman, “all you did was hit it with the hammer.” So, the plumber writes her out an itemised bill: “Hitting washing machine with a hammer - £5. Knowing where to hit it - £195.”

To see if your expert really an expert is you can look at their results. If the things they claim turn out to be accurate and correct on a consistent and regular basis then you would be justified in concluding that they know they are talking about. Expertise is something you demonstrate rather than something you simply claim. Sticking with plumbers, if every time you had a plumbing related problem you called the same guy and he came out and had it all fixed and working again within the hour you would conclude that he knows what he is doing, even if you yourself were completely unable to tell the difference between a stop cock and a u-bend. The validity of the results demonstrates your plumber’s expertise.

Part of the way science works involves making predictions and then testing them to see if they are accurate. Again, you may not fully understand why they made the prediction they did but once the results are in you can see if they were right or not. It is also important to note the type of predictions being made here. For example, I could make the prediction that “man-made climate change is real” and when the evidence comes in showing that it is (and it is) I could point to this prediction and announce myself to be an expert in climate change. However, this is not the type of prediction we are generally dealing with when it comes to science. I had a 50/50 chance of being right, this does no more to prove my expertise than picking the winners of a few football matches proves that octopuses are  psychic. (If you are not aware of him then check out Paul the “Psychic” Octopus…though technically it should have been Paul the “Precognitive” Octopus as he allegedly told the future rather than read minds - http://en.wikipedia.org/wiki/Paul_the_Octopus). 

 

The sort of predictions you have in science are more along the lines of “if we do X under situation Y we will see result Z to a level of N”. This is a far more specific prediction with many more than two possible outcomes. If your expert is discussing situations that have dozens or even thousands of possible outcomes and is consistently predicting the correct result then, as with your plumber, you would be justified in accepting that maybe they know what they are talking about.

Now of course it is not always possible for us to check out the track record of every so called expert we encounter, at least not there and then, but sometimes if you know the track record of a couple of the recognised experts in the field you can play a game of connections to reach a provisional judgement about the expert before you. For example, let’s say we have taken the time to look into Jim’s track record and found that he had a fairly consistent record of making accurate predictions within his field of quantum physics. Now let’s say we are at a talk given by Toby and we know that Toby both studied under and worked with Jim at one point. We could therefore provisionally conclude that Toby is likely to share many of the thoughts and opinions regarding quantum physics as his mentor and, until we can look into Toby’s record directly, we could thus co-opt Jim’s track record to help us evaluate Toby’s expertise.

At the end of the day none of these points on their own will tell you if you are dealing with someone who should be considered a genuine expert or not. However, by using them together you will get as clear an idea as to whether your expert is legitimate or not as you are going to get without becoming an expert in the field yourself. Check to make sure your expert is talking about the subject in which they have their expertise rather than just pontificating about a topic in which they have no more direct training than yourself. Look at the arguments they are making either using direct means, where you have an understanding of the topic at hand, or indirect methods to evaluate your experts’ level of understanding about the subject. Check to see if their ideas match with the consensus of other people working in the field as well as seeing if they even have the qualifications to back up their claim to be an expert. Look to see what other experts have to say about them and how often their work is referenced. Examine whether your expert has any questionable biases that may have undue influence upon their opinions and conclusions. And lastly look at their track record to see how good they are at getting things right. Do this and you’ll be an expert at spotting experts in no time.
 
References

 

1. Anderegg, W.R.L., Prall, J.W., Harold, J., & Schneider, S.H.  (2010).  Expert credibility in climate change. Proceedings of the National Academy of Sciences of the United States of America, 107, 12107-12109. doi: 10.1073/pnas.1003187107.

2. Brewer, S. (1998). Scientific expert testimony and intellectual due process. Yale Law Journal, 107, 1535-1568.

3. Doran, P.T., & Zimmerman, M.K. (2009). Examining the scientific consensus on climate change. Eos Transactions American Geophysical Union, 90, 22–23. doi: 10.1029/2009EO030002.

4. Ericsson, K.A., & Smith, J. (1991). Towards a general theory of expertise:  Prospects and limits. Cambridge: Cambridge University Press. ISBN: 0-521-40612-9.

5. Friedberg, M., Saffran, B., Stinson, T.J., Nelson, W., & Bennett, C.L. (1999). Evaluation of conflict of interest in economic analyses of new drugs used in oncology. Journal of the American Medical Association, 282, 1453-1457. doi: 10.1001/jama.282.15.1453.

6. Goldacre, B. (2008). Bad Science. London: Fourth Estate. ISBN: 978-0-00-724019.

7. Goldman, A. I. (2001). Experts: Which ones should you trust? Philosophy and Phenomenological Research, 63, 85-110.

8. Pigliucci, M. (2010). Nonsense on stilts: How to tell science from bunk.  Chicago:  The University of Chicago Press. ISBN: 978-0-226-66786-7.