Common Problems with Psychological Laboratory Tests (article) by Sam Vaknin on AuthorsDen

Psychological laboratory tests suffer from a series of common philosophical, methodological, and design problems.

A. Philosophical and Design Aspects

Ethical – Experiments involve the patient and others. To achieve results, the subjects have to be ignorant of the reasons for the experiments and their aims. Sometimes even the very performance of an experiment has to remain a secret (double blind experiments). Some experiments may involve unpleasant or even traumatic experiences. This is ethically unacceptable.

The Psychological Uncertainty Principle – The initial state of a human subject in an experiment is usually fully established. But both treatment and experimentation influence the subject and render this knowledge irrelevant. The very processes of measurement and observation influence the human subject and transform him or her - as do life's circumstances and vicissitudes.

Uniqueness – Psychological experiments are, therefore, bound to be unique, unrepeatable, cannot be replicated elsewhere and at other times even when they are conducted with the SAME subjects. This is because the subjects are never the same due to the aforementioned psychological uncertainty principle. Repeating the experiments with other subjects adversely affects the scientific value of the results.

The undergeneration of testable hypotheses – Psychology does not generate a sufficient number of hypotheses, which can be subjected to scientific testing. This has to do with the fabulous (=storytelling) nature of psychology. In a way, psychology has affinity with some private languages. It is a form of art and, as such, is self-sufficient and self-contained. If structural, internal constraints are met – a statement is deemed true even if it does not satisfy external scientific requirements.

B. Methodology

1. Many psychological lab tests are not blind. The experimenter is fully aware who among his subjects has the traits and behaviors that the test is supposed to identify and predict. This foreknowledge may give rise to experimenter effects and biases. Thus, when testing for the prevalence and intensity of fear conditioning among psychopaths (e.g., Birbaumer, 2005), the subjects were first diagnosed with psychopathy (using the PCL-R questionnaire) and only then underwent the experiment. Thus, we are left in the dark as to whether the test results (deficient fear conditioning) can actually predict or retrodict psychopathy (i.e., high PCL-R scores and typical life histories).

2. In many cases, the results can be linked to multiple causes. This gives rise to questionable cause fallacies in the interpretation of test outcomes. In the aforementioned example, the vanishingly low pain aversion of psychopaths may have more to do with peer-posturing than with a high tolerance of pain: psychopaths may simply be too embarrassed to "succumb" to pain; any admission of vulnerability is perceived by them as a threat to an omnipotent and grandiose self-image that is sang-froid and, therefore, impervious to pain. It may also be connected to inappropriate affect.

3. Most psychological lab tests involve tiny samples (as few as 3 subjects!) and interrupted time series. The fewer the subjects, the more random and less significant are the results. Type III errors and issues pertaining to the processing of data garnered in interrupted time series are common.

4. The interpretation of test results often verges on metaphysics rather than science. Thus, the Birbaumer test established that subjects who scored high on the PCL-R have different patterns of skin conductance (sweating in anticipation of painful stimuli) and brain activity. It did not substantiate, let alone prove, the existence or absence of specific mental states or psychological constructs.

5. Most lab tests deal with tokens of certain types of phenomena. Again: the fear conditioning (anticipatory aversion) test pertains only to reactions in anticipation of an instance (token) of a certain type of pain. It does not necessarily apply to other types of pain or to other tokens of this type or any other type of pain.

6. Many psychological lab tests give rise to the petitio principii (begging the question) logical fallacy. Again, let us revisit Birbaumer's test. It deals with people whose behavior is designated as "antisocial". But what constitute antisocial traits and conduct? The answer is culture-bound. Not surprisingly, European psychopaths score far lower on the PCL-R than their American counterparts. The very validity of the construct "psychopath" is, therefore, in question: psychopathy seems to be merely what the PCL-R measures!

7. Finally, the "Clockwork Orange" objection: psychological lab tests have frequently been abused by reprehensible regimes for purposes of social control and social engineering.