How to do a goodness of fit test for a multinomial experiment

Description

If we have historical values for multiple population proportions, plus more recent samples from those same populations, we may want to compare to see if the proportions appear to have changed. This is called a goodness of fit test for a multinomial experiment. How can we execute it?

Using SciPy, in Python

View this solution alone.

Let’s say we have a dataset with the previous population proportions for four categories. (This is contrived data, but the code below can be used on your actual data.)

We have also taken a more recent sample and found the number of observations from it that belong to each category. We want to determine if the proportions coming from the recent sample are equal to the previous proportions.

SciPy expects that we will have two lists, one with the expected number of observations in each group (from the previous, or hypothesized proportions) and the other with the actual number of observations in each group (from the more recent sample). SciPy also expects that the total number of observations in each list is the same. We’ll create two lists below with the fake data from above, but you can replace them with your real data

1
2
3
Copy
# Replace your data in the next two lines
old_observations = [43, 62, 52, 16]
new_observations = [56, 80, 12, 25]

We set the null hypothesis to be that the proportions of each category from the recent sample are equal to the previous proportions

We choose a value $0\le \alpha \le 1$ as our Type 1 error rate. We’ll let $\alpha$ be 0.05 here.

1
2
3
Copy
# Run the Chi-square test, giving the test statistic and p-value
from scipy import stats
stats.chisquare(f_obs=new_observations, f_exp=old_observations)

1
Power_divergenceResult(statistic=44.98776977898321, pvalue=9.30824439694332e-10)

Our $p$-value is less than $\alpha$, so we have sufficient evidence to reject the null hypothesis. It does appear that the proportion of at least one of the four categories is significantly different now from what it was previously.

Content last modified on 24 July 2023.

See a problem? Tell us or edit the source.

Solution, in R

View this solution alone.

Let’s say we have a dataset with the previous population proportions for four categories. (This is contrived data, but the code below can be used on your actual data.)

We have also taken a more recent sample and found the number of observations from it that belong to each category. We want to determine if the proportions coming from the recent sample are equal to the previous proportions.

R expects that we will have two vectors, one with the expected number of observations in each group (from the previous, or hypothesized proportions) and the other with the actual number of observations in each group (from the more recent sample). R also expects that the total number of observations in each vector is the same. We’ll create two vectors below with the fake data from above, but you can replace them with your real data

1
2
3
4
5
6
Copy
# our fake data:
old.observations <- c(43, 62, 52, 16)
new.observations <- c(56, 80, 12, 25)
# now organized into a data frame:
categories <- c("A", "B", "C", "D")
data <- data.frame(categories, old.observations, new.observations)

We set the null hypothesis to be that the proportions of each category from the recent sample are equal to the previous proportions.

We choose a value $0\le \alpha \le 1$ as our Type 1 error rate. We’ll let $\alpha$ be 0.05 here.

1
2
Copy
# Run the Chi-Square test, giving the test statistic and p-value
chisq.test(data$new.observations, p=data$old.observations, rescale.p=TRUE)

1
2
3
4
	Chi-squared test for given probabilities

data:  data$new.observations
X-squared = 44.988, df = 3, p-value = 9.308e-10

Our $p$-value is less than $\alpha$, so we have sufficient evidence to reject the null hypothesis. It does appear that the proportion of at least one of the four categories is significantly different now from what it was previously.

If instead you provided the population proportions as the old observations, that is, a vector of values that sum to 1, you can omit the rescale.p argument.

Content last modified on 24 July 2023.

See a problem? Tell us or edit the source.

Topics that include this task

• Bentley University MA214

Opportunities

This website does not yet contain a solution for this task in any of the following software packages.

• Excel
• Julia

If you can contribute a solution using any of these pieces of software, see our Contributing page for how to help extend this website.