ABA Exam Coach

C.8: Evaluate the validity and reliability of measurement procedures.

Free practice questions for this part of the BCBA, with an explanation for every option.

A school team wants to measure 'class participation' to evaluate a social skills intervention. Observers record only the number of times the student raises his hand. IOA is consistently above 95%. A teacher points out that the student often raises his hand and then says 'never mind,' and rarely contributes to discussion. What is the primary problem?
  1. A. Poor reliability, because IOA does not reflect true participation
    Incorrect. The high IOA shows observers measure hand-raising consistently. Reliability is not the issue.
  2. B. Observer drift, because observers have changed how they apply the definition
    Incorrect. Nothing suggests observers' application of the definition has shifted. Agreement is consistently high.
  3. C. Reactivity, because the student raises his hand more when observed
    Incorrect. The issue is what is being measured, not a change in behavior caused by being observed.
  4. D. Poor validity, because the measure does not capture the behavior that matters for the intervention's purpose
    Correct. Validity means measuring a socially significant behavior relevant to the question. Hand-raising alone does not represent participation.
Two observers use partial-interval recording across 20 intervals. Observer A scores 12 intervals as occurrences, and Observer B scores 11. They agree that the behavior occurred in 9 intervals and that it did not occur in 6 intervals. What is the interval-by-interval IOA?
  1. A. 64%
    Incorrect. 64% is scored-interval IOA: occurrence agreements divided by intervals either observer scored, 9/14.
  2. B. 45%
    Incorrect. 45% divides only the occurrence agreements by all intervals (9/20) and leaves out nonoccurrence agreements.
  3. C. 75%
    Correct. Interval-by-interval IOA = total agreements (9 occurrence + 6 nonoccurrence) / total intervals (20) = 75%.
  4. D. 92%
    Incorrect. 92% is total interval IOA (11/12), which ignores whether observers agreed on the same intervals.
Two RBTs trained together score a client's vocal tics and consistently get 95% IOA. The BCBA scores the same video samples carefully, with replay, and finds that both RBTs miss about one-third of the tics. What is the best conclusion?
  1. A. The definition must be invalid, because accuracy and validity are the same thing.
    Incorrect. Accuracy (match to true values) and validity (measuring the right behavior) are distinct concepts.
  2. B. The BCBA's scoring is unreliable because it disagrees with two observers.
    Incorrect. Video scoring with replay is a better approximation of true values, so the RBTs' data are the ones in question.
  3. C. The data are accurate, because high IOA confirms the RBTs are measuring the true values.
    Incorrect. Observers can agree with each other and still be wrong in the same way, for example through shared training or drift.
  4. D. High IOA shows the observers agree with each other, but their data are not accurate. Agreement does not establish accuracy.
    Correct. Accuracy means agreement with the true values, here estimated by careful video scoring. High IOA indicates believability, not accuracy.

Take the free diagnostic