November 16, 2016

 

FROM: Ryan Summers (Phone: 701-777-3144; Email: ryan.summers@und.edu)

             Shuai “Sam” Wan

TO: STEM Matrix Foundation

RE: February through June 2016 Pre- and Post- Intervention Data Analysis

 

Beginning in late summer of 2016 Shuai “Sam” Wang and Ryan Summers were engaged in the analysis of pre- and post-intervention data collected from Stem Matrix. These data were collected from 9 schools in Waukegan School District 60 during the spring 2016 semester. To complement these data, qualitative responses were also collected from participating teachers and administrators highlighting the observable impact of the STEM Matrix program.

 

At the time they were approached to review the data collected both Ryan and Sam were affiliated with the University of Illinois in Urbana-Champaign in the departments of Curriculum and Instruction and Educational Psychology, respectively. Ryan Summers is now a tenure-track assistant professor of secondary science education at the University of North Dakota. As a contributor to this project Ryan brought his expertise in survey design and delivery, as well as his general knowledge of intervention programs involving science education. Sam Wang is now a research scientist at SRI International in Washington D.C., and the majority of his professional responsibilities involve planning and strategizing for data collection in a wide variety of educational programs and interventions. Sam has an expansive knowledge of statistical procedures and a great deal of experience in appropriately applying this knowledge to meet the research goals of specific programs. 

  

Using the complete data available focusing on 6 of the 9 schools, which included both pre- and post-intervention data for 255 students in the experimental STEM Matrix program and control classroom settings. The content test contained 10 questions on a 4-point multiple choice scale with a single best answer for each prompt. Descriptive and statistical analyses were applied to address the following questions:

1. Did students in the STEM Matrix program perform better on the post-test measure compared to students in the normal classroom setting (experimental vs. control)?

2. Were any differences detected in the performance of the experimental group across the different schools involved?

First, it is important to note that on average students in both groups scored similarly on the pre-test with mean number of correct responses for the control group of 4.02 and experimental group of 4.06 (n=131 and n=124). Using SAS statistical software, Sam concluded that students who participated in the STEM Matrix experimental group performed better than their peers in the control classrooms on the post-test. Student responses to the 10-question measure indicate that students who participated in STEM Matrix scored over 1-point better than their peers. The average STEM Matrix student scored 1.09 points higher than the average control group student, a statistically significant finding (p < .001) representing a large effect. The second question explored did not lead to the identification of any school as having a different effect. In other words, the finding reported above was consistent for the data analyzed from the 6 schools.   

Overall these findings are notable for the first iteration of an intervention. In the future it might be fruitful to explore other factors, and further examine the full extent to which STEM Matrix participants seem to improve. From the qualitative and anecdotal evidences collected from teachers and administrators there appears to be an overwhelming belief that the program encourages positive attitudes toward science and builds self-confidence, in addition to cultivating desirable skills related to critical thinking and small group collaboration.