Threats to External Validity

 I must first consider what external validity is, in order to understand which threats apply. I will effectively define the term as 'the degree to which the findings of an experiment or study can be generalized beyond the conditions of the original study'. This definition allows for multiple interpretations within a range; perhaps none are entirely correct, but all have some value.

So, I will list a few threats to external validity. First is Post Hoc Ergo Propter Hoc Fallacy; which translates as 'after this, therefore because of this'. This fallacy occurs when an experimenter assumes that one event must have caused another simply because the two events occurred closely in time or are followed by other events.

Second is the experimenter's fallacy; which occurs when an experimenter places too much confidence in the results of his/her own study. This can be related to confirmation bias.

Third is the problem of external validity. This occurs when an experimenter does not have enough data to generalize results; a related issue is insufficiency of sample size.

Fourth is the problem of time; results from studies are only valid until new information occurs.

Fifth is observer bias; which occurs when a scientist cannot remain neutral and objective while conducting an experiment. Related to this is Experimenter's Bias.

There are two ways that external validity is threatened. One way is when the experiment involves a laboratory environment or special conditions, and these conditions cannot be applied to real-world situations. The other way is by having an unrepresentative sample of subjects in the testing group. Both can cause data collection which will not be representative of real world behaviour.

Take the lab environment example. If you are screening for a rare disease, and your test involves only people who work in an office building at night, then your results will be valid only for that sample group. Of course there is no way to generalise this data from such a small population.

Take the unrepresentative sample group example. If you are testing a new drug, and you only test it on young healthy men, then your results will not be representative of the whole population. The data collected from this small sample group cannot be applied to other populations.

So, it's vital that we collect data under conditions which reflect real-world situations and from a sample group which reflects the general population.

My conclusion is that we should always be careful to ensure that our data reflects real-world situations and that it is representative of the general population.

If you can't tell, I'm trying to argue that any threat of external validity is actually an internal problem. You see, there are many ways in which our studies could be biased or flawed and we simply don't know where they are for certain.

There is no way to know that there aren't flaws in our studies. Even if you have a study with 'perfect' conditions, there's always the chance that it wasn't set up perfectly and we're just not aware of this.

So it's not that there are 'threats to external validity', it's just that we can't know if our studies aren't flawed. It's a problem with philosophy in general, because all knowledge is limited and fallible.

But because of this, we can't really claim that any external studies are flawed. We have to be careful with our language and make sure not to make unjustifiable claims about the validity/invalidity of others' work.

Now, if you want to claim that because of something like the Gettier problem (if you don't know what it is, I can explain), there aren't certain kinds of 'knowledge' or 'justified true belief', then this isn't a threat to external validity.

If you want to claim that there is something wrong with the idea of knowledge, then it's not a threat to external validity. It's just saying that some kinds of knowledge are harder than others.