Reviewing studies with diverse designs: the development and evaluation of a new tool

The 16-item quality assessment tool (QATSDD) was assessed to determine its reliability and validity when used by health services researchers in the disciplines of psychology, sociology and nursing. Qualitative feedback was also gathered from mixed-methods health researchers regarding the comprehension, content, perceived value and usability of the tool.

Reference to existing widely used quality assessment tools and experts in systematic review confirmed that the components of the tool represented the construct of 'good research technique' being assessed. Face validity was subsequently established through feedback from a sample of nine health researchers. Inter-rater reliability was established through substantial agreement between three reviewers when applying the tool to a set of three research papers (κ = 71.5%), and good to substantial agreement between their scores at time 1 and after a 6-week interval at time 2 confirmed test-retest reliability.