The Musical Theater Project Demonstrates the Value of Building Evaluation into Programs from Day One

One girl and two boys growl like tigers while wearing smock-style costumes.

Students participate in a Kids Love Musicals! residency. Photo credit: Heather Meeker

When leaders at the Musical Theater Project in Northeast Ohio decided they wanted to expand their Kids Love Musicals! residency program to serve students with disabilities, they were deliberate in their planning. They sought out resources and expertise from peer arts organizations already working with students with disabilities, and they attended professional development sessions on arts and special education topics. As they laid out their expansion plan, they identified program assessment as a priority and sought to include comprehensive evaluation strategies as a part of the new residencies.

With this in mind, Heather Meeker, Executive Director of the Musical Theater Project (TMTP), connected with leaders at the Schubert Center for Child Studies at Case Western Reserve University (CWRU), located nearby in Cleveland, Ohio. “CWRU is interested in being deeply involved in their community, so developing a mutually beneficial research project was of great interest to them,” says Meeker.

The Schubert Center introduced Meeker to psychology professor Sandra Russ and doctoral student Olena Zyga, who agreed to work with TMTP to assess the new residencies. TMTP agreed to support the academics’ work by raising money to pay for student researchers and faculty time, and Meeker says funders have been especially interested in supporting this collaborative assessment.

The Kids Love Musicals! residencies for children with disabilities aim to teach social skills and emotional understanding through the stories and characters from classic American musicals such as The Wizard of Oz, The Jungle Book, and You’re a Good Man, Charlie Brown. The multi-year evaluation project with the Schubert Center seeks to better understand if engaging in the residency program impacted participants’ socioemotional skills, including the ability to make eye contact, engage with others, take turns appropriately, and demonstrate emotional understanding. A secondary goal is to understand whether gains seen during the residency program extend to other environments.

Russ and Zyga created a custom measurement scale for the program, using their expertise in the fields of psychology and play. TMTP initiated their new residencies for students with disabilities, collecting multiple forms of data throughout. Residency sessions were videotaped across multiple school sites and to include a range of student ages and ability levels; the videos were then coded and scored according to the measurement scale. Teachers were also asked to report on the same variables that were being coded in each session for every student, both before the residency program began and after it had finished.

Analysis of the first round of data, which specifically focused on The Wizard of Oz residency, suggests that students who participated in the Kids Love Musicals! program did make gains in eye contact, turn taking, engagement, and symbolic flexibility. These results were recently published in the Journal of Intellectual Disabilities. Meeker is thrilled that their collaboration with the Schubert Center led to the research being shared broadly, both through journal publication and in various conference presentations by her and Zyga.

Four children stand in front of two adults, all wearing curly gold ribbon on their heads and making roaring faces.

Teaching artists work with students in the Kids Love Musicals! residencies. Photo credit: Heather Meeker

The research collaboration between TMTP and the Schubert Center continues post-report publication, including a new round of data collection focused on identifying if similar gains are seen across curriculums presented to students. Specifically, they are asking if children made the same gains while learning The Jungle Book as made while learning The Wizard of Oz. Analysis of this data is currently underway, with initial results suggesting that curriculum differences do not significantly impact the student outcomes. A final phase of data collection, completed at the end of the 2016-2017 academic year, focused on comparing the active residency period with a pre-residency control period.

Given the success of their collaboration with the Schubert Center, Meeker encourages organizations interested in conducting robust program evaluations to consider partnering with a college or university in their own community. “If a project can be designed with the idea that both the organization and university students can benefit from it, a collaboration can really be a win-win situation,” she says.

Of course, Meeker also warns of the hard work and complications that come with conducting a large-scale assessment. She explains, “We had to make peace with the fact that we would not get 100% compliance from teachers in our data collection efforts, and that not all of the data we worked so hard to collect would ultimately be used in the study. We also did not anticipate the delays that sometimes come with working with a university, like waiting for internal review board approvals for everything from project proposals to parent permission forms.”

But the reward for that hard work is great, Meeker says, as their research has clarified so much for TMTP about the program internally. She concludes, “If you are constantly looking to improve your work, then thorough evaluation is crucial. This project has empowered us to do even more with our programming.”