Indian Journal of Psychological Medicine
Users Online: 890 
  Home | About Us | Editorial Board | Search | Ahead of print | Current Issue | Archives | Instructions | Contact | Advertise | Submission | Login 
Wide layoutNarrow layoutFull screen layoutHome Print this page Email this page Small font sizeDefault font sizeIncrease font size


 
 Table of Contents    
LEARNING CURVE
Year : 2018  |  Volume : 40  |  Issue : 5  |  Page : 498-499  

Internal, external, and ecological validity in research design, conduct, and evaluation


Department of Psychopharmacology, National Institute of Mental Health and Neurosciences, Bengaluru, Karnataka, India

Date of Web Publication10-Sep-2018

Correspondence Address:
Dr. Chittaranjan Andrade
Department of Psychopharmacology, National Institute of Mental Health and Neurosciences, Bengaluru - 560 029, Karnataka
India
Login to access the Email id

Source of Support: None, Conflict of Interest: None


DOI: 10.4103/IJPSYM.IJPSYM_334_18

Rights and Permissions
   Abstract 


Reliability and validity describe desirable psychometric characteristics of research instruments. The concept of validity is also applied to research studies and their findings. Internal validity examines whether the study design, conduct, and analysis answer the research questions without bias. External validity examines whether the study findings can be generalized to other contexts. Ecological validity examines, specifically, whether the study findings can be generalized to real-life settings; thus ecological validity is a subtype of external validity. These concepts are explained using examples so that readers may understand why the consideration of internal, external, and ecological validity is important for designing and conducting studies, and for understanding the merits of published research.

Keywords: CATIE, ecological validity, external validity, internal validity, reliability


How to cite this article:
Andrade C. Internal, external, and ecological validity in research design, conduct, and evaluation. Indian J Psychol Med 2018;40:498-9

How to cite this URL:
Andrade C. Internal, external, and ecological validity in research design, conduct, and evaluation. Indian J Psychol Med [serial online] 2018 [cited 2019 Mar 21];40:498-9. Available from: http://www.ijpm.info/text.asp?2018/40/5/498/240932




   Did Catie Have External Validity? Top


The answer is both yes and no. CATIE[1] was designed as an effectiveness study; that is, a study with relevance to real-world settings. The CATIE findings are relevant to clinical practice in the USA but are of questionable relevance in India. One reason is that, in the USA, where CATIE was conducted, the primary outcome, time to all-cause treatment discontinuation, is substantially patient-influenced, whereas in India, where families supervise treatment, it is largely caregiver-determined. Another and more important reason is that the healthcare delivery system in clinical practice is strikingly different in the two countries. Thus CATIE has good external validity for clinical practice in the USA but not in India.


   Reliability and Validity Top


Reliability and validity are concepts that are applied to instruments such as rating scales and screening tools. Validity describes how well an instrument does what it is supposed to do. For example, does an instrument that screens for depression do so with high sensitivity and specificity? Reliability describes the consistency with which results are obtained. For example, if an instrument that rates the severity of depression is administered to the same patient twice within the span of an hour, are the scores obtained closely similar? Different types of reliability and validity describe desirable psychometric properties of research and clinical instruments.[2],[3] Validity can also be applied to laboratory and clinical studies, and to their findings, as well, as the sections below show.


   Internal Validity Top


Internal validity examines whether the manner in which a study was designed, conducted, and analyzed allows trustworthy answers to the research questions in the study. For example, improper randomization, inadvertent unblinding of patients or raters, excessive use of rescue medication, and missing data can all undermine the fidelity of the results and conclusions of a randomized controlled trial (RCT). That is, the internal validity of the RCT is compromised. Internal validity is based on judgment and is not a computed statistic.

Internal validity examines the extent to which systematic error (bias) is present. Such systematic error can arise through selection bias, performance bias, detection bias, and attrition bias.[4] If internal validity is compromised, it can occasionally be improved, for example, by a modified plan of analysis. However, biases can be often fatal as, for example, if double-blind ratings were not obtained in an RCT.


   External Validity Top


External validity examines whether the findings of a study can be generalized to other contexts.[4] Studies are conducted on samples, and if sampling was random, the sample is representative of the population, and so the results of a study can validly be generalized to the population from which the sample was drawn. But results may not be generalizable to other populations. Thus external validity is poor for studies with sociodemographic restrictions; studies that exclude severely ill and suicidal patients, or patients with personality disorders, substance use disorders, and medical comorbidities; studies that disallow concurrent treatments; and so on. External validity is also limited in short-term studies of patients who need to be treated for months to years. External validity, like internal validity, is based on judgment and is not a computed statistic.


   Ecological Validity Top


Ecological validity examines whether the results of a study can be generalized to real-life settings.[5] How is this different from external validity? External validity asks whether the findings of a study can be generalized to patients with characteristics that are different from those in the study, or patients who are treated in a different way, or patients who are followed up for longer durations. In contrast, ecological validity specifically examines whether the findings of a study can be generalized to naturalistic situations, such as clinical practice in everyday life. Ecological validity is, therefore, a subtype of external validity. The ecological validity of an instrument can be computed as a correlation between ratings obtained with that instrument and an appropriate measure in naturalistic practice or in everyday life. The ecological validity of a study is a judgment and is not a computed statistic.


   Discussion Top


Ecological validity was originally invoked in the context of laboratory studies that required to be generalized to real-life situations.[5] Thus, laboratory studies of the neuropsychological and psychomotor impairments produced by psychotropic drugs have poor ecological validity because what is studied in relaxed, rested, and healthy subjects tested in a controlled environment is very different from demands that stressed patients face in everyday life. In fact, these cognitive and psychomotor tests, especially when based on computerized tasks, have no parallel in everyday life. How much less ecological validity, then, would research in animal models of different neuropsychiatric states have for patients in clinical practice? This explains why drugs that work in animal models often fail in humans.[6]

On a parting note, a good understanding of the concepts of internal, external, and ecological validity is necessary to properly design and conduct studies and to evaluate the merits and applications of published research.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.



 
   References Top

1.
Lieberman JA, Stroup TS. The NIMH-CATIE schizophrenia study: What did we learn? Am J Psychiatry 2011;168:770-5.  Back to cited text no. 1
    
2.
Streiner DL. Statistics commentary series: Commentary #15-reliability. J Clin Psychopharmacol 2016;36:305-7.  Back to cited text no. 2
    
3.
Streiner DL. Statistics commentary series: Commentary no 17-validity. J Clin Psychopharmacol 2016;36:542-4.  Back to cited text no. 3
    
4.
Juni P, Altman DG, Egger M. Assessing the quality of randomized controlled trials. In: Egger M, Smith GD, Altman DG, editors. Systematic Reviews in Health Care: Meta-analysis in Context. London: BMJ Publishing Group; 2001. p. 87-108.  Back to cited text no. 4
    
5.
Lewkowicz DJ. The concept of ecological validity: What are its limitations and is it bad to be invalid. Infancy 2001;2:437-50.  Back to cited text no. 5
    
6.
Andrade C, Sudha S, Venkataraman BV. Herbal treatments for ECS-induced memory deficits: A review of research and a discussion on animal models. J ECT 2000;16:144-56.  Back to cited text no. 6
    




 

Top
 
 
  Search
 
  
    Similar in PUBMED
   Search Pubmed for
   Search in Google Scholar for
 Related articles
    Access Statistics
    Email Alert *
    Add to My List *
* Registration required (free)  

 
  In this article
    Abstract
    Did Catie Have E...
    Reliability and ...
   Internal Validity
   External Validity
   Ecological Validity
   Discussion
    References

 Article Access Statistics
    Viewed442    
    Printed7    
    Emailed0    
    PDF Downloaded28    
    Comments [Add]    

Recommend this journal