Estimating Reliability of Tests with Dichotomous and Polytomous Items According to Classical Measurement Theory and Generalizability Theory: A Comparative Study

This study aimed to compare the reliability coefficients of tests with dichotomous and polytomous items according to Classical Test Theory and Generalizability Theory. To achieve this, a compound mathematics ability test was developed in two formats: dichotomous and polytomous, using five estimation methods: Alpha (α), Theta (θ), Raju (G), and Phi (φ). Each test format comprised 30 items designed to assess students' abilities in numbers, algebra, geometry, and statistics, with a sample of 300 students.

Reliability coefficients for the dichotomous items, estimated using Classical Theory, were as follows: α = 0.87, θ = 0.85, and Raju = 0.86. For Generalizability Theory, the coefficients were: G = 0.87 and φ = 0.86. In contrast, reliability coefficients for the polytomous items, according to Classical Theory, were: α = 0.95, θ = 0.92, and Raju = 0.93; for Generalizability Theory, they were G = 0.95 and φ = 0.94.

Comparison results using the Cochran test and the signal-to-noise ratio test indicated a significant difference at α = 0.05, favoring the polytomous format for reliability. Additionally, the findings demonstrated a preference for Generalizability Theory over Classical Theory in estimating reliability coefficients.