A simulation study of a class of nonparametric test statistics: a close look of empirical distribution function-based tests

Kolmogorov–Smirnov (KS) statistic is a non-parametric statistic based on the empirical distribution function. For the one-sample case, it uses the supremum distance between an empirical distribution function (EDF) and a pre-specified cumulative distribution function (CDF). For two-sample case, it measures the maximum of the distance between two EDFs. KS test, as well as other EDF-based tests such as the Anderson-Darling (AD) test and Cramer-von Mises (CvM) test, has been widely used in statistical analysis. To address and compare the performance of these test statistics, we have conducted a simulation study comparing the type I error and power of the KS test, the CvM test, the AD test, and the Chi-squared test. Our study includes both one sample and two sample tests and for both independent and correlated samples. Our study showed that if we do not have prior information about the tested distributions, EDF-based tests are better. However, so long as we have prior information about the tested distribution and the density of two distributions is bell-shaped and we are expecting differences in variance/sparseness, then the Chi-squared test may be more preferable. When correlation exists between tested samples, adjustment on the informative sample size is important and required.