SOWK 6341 Discussion – Identifying Validity Threats
Purpose
Experimental designs are the gold standard of research because they help eliminate any other explanation for why an intervention does or does not work. However, experimental designs are difficult in social service settings for many reasons. For purposes of practicality, we often opt for quasi-experimental designs that introduce limitations to our research (threats to validity). While it is nearly impossible to eliminate every threat to validity, there are ways to mitigate those threats.
Create an original discussion post where you identify two potential threats to validity and craft ways to mitigate those threats.
Discussion: Identifying Threats Mental Health Treatment A nonprofit serving children who have witnessed family violence wants to know if their counselors should use cognitive behavioral therapy (CBT) or trauma-informed cognitive behavioral therapy (TFCBT). They have two counselors trained in CBT who have been working at the agency for 15 years each. They also have two new counselors straight from an MSW program who have been trained in TFCBT. The CBT counselors have a current caseload of 20 children each. All new children will be assigned to the TFCBT counselors. When a child is done with therapy, the counselors will ask the parents if their child’s behaviors have gotten better. They will use that information to decide if they should use TFCBT or CBT.
Book: Rubin, A.,
Potential Threats to Validity in Evaluating Mental Health Treatment Approaches
There are a few potential threats to validity that should be addressed when evaluating the effectiveness of cognitive behavioral therapy (CBT) versus trauma-informed cognitive behavioral therapy (TF-CBT) for treating children who have witnessed family violence.
First, history effects could influence the results if external events impact the children during the study period. For example, if a natural disaster occurs and causes additional trauma, the children in both treatment groups may experience worsening behaviors regardless of the therapy approach. To help mitigate this threat, the study timeframe should be kept as short as reasonably possible to limit outside influences.
Second, selection bias is a concern since the existing CBT caseloads will be compared to the new TF-CBT caseloads. Differences in outcomes could partially or fully be due to pre-existing characteristics of the children rather than the therapy itself. To address this, baseline assessments of things like trauma history, behavioral issues, and family dynamics should be conducted and statistical controls used in the analysis (Twamley et al., 2015).
Third, because the parents will be asked to subjectively rate their child’s behavior change, there is potential for expectancy and observer bias effects. Parents may rate behaviors more positively if they expect or want a certain therapy to work better. Using standardized, validated behavior rating scales from multiple observers like parents, teachers, and counselors can help reduce bias (Schoenwald et al., 2011).
In conclusion, employing controls for history effects, selection bias, and observer bias through standardized assessments, statistical analysis techniques, and multi-rater Assessment can help strengthen the internal validity of this study comparing CBT and TF-CBT approaches.
Twamley, E. W., Jeste, D. V., & Lehman, A. F. (2003). Vocational rehabilitation in schizophrenia and other psychotic disorders: a literature review and meta-analysis of randomized controlled trials. The Journal of nervous and mental disease, 191(8), 515–523. https://doi.org/10.1097/01.nmd.0000082232.42506.1d
Schoenwald, S. K., Sheidow, A. J., & Letourneau, E. J. (2004). Toward effective quality assurance in evidence-based practice: links between expert consultation, therapist fidelity, and child outcomes. Journal of clinical child and adolescent psychology : the official journal for the Society of Clinical Child and Adolescent Psychology, American Psychological Association, Division 53, 33(1), 94–104. https://doi.org/10.1207/S15374424JCCP3301_10
Rubin, A., & Babbie, E. (2016). Empowerment series: Research methods for social work. Cengage Learning.
Forte, J. A. (2017). Human behavior and the social environment: Models, metaphors, and maps for applying theoretical perspectives to practice. Cengage Learning.
