Kris Baranovic
Southeast Missouri State University, Center for Scholarship in Teaching and Learning
The Problem
The initial motivation for implementing Quality Matters at Southeast Missouri State University (Southeast) was the high withdrawal rates that had historically occurred in online courses during a four-week Wintersession between the Fall and Spring semesters. The administration decided to bring Quality Matters to Southeast as a way to increase the structural consistency of the courses offered in Wintersession in an effort to reduce student frustration and increase course completion.
The Initiative
Southeast applied for and was awarded a Funding for Results demonstration grant in 2013. The initial project was to train faculty and internally review high-demand, large enrollment Wintersession courses. Three instructors participated in Quality Matters professional development, and three courses were reviewed. The results were positive; so in 2014 Southeast’s president and Board of Regents mandated that all online courses offered at Southeast meet the Quality Matters Rubric Standards.
With approximately 500 online courses at Southeast, implementation of QM for all courses was a challenging initiative, so the process was spread out over a period of three years. We paid for faculty to take QM professional development and become Certified Peer Reviewers.Then, we constructed a schedule for performing internal reviews of online courses based on the academic term in which they were offered. The first group of courses slotted for review were Wintersession courses. To prepare for the course reviews, faculty were paid to participate in an internal professional development series on QM in small cohorts.
As of December 2017, 500 courses had been internally reviewed at Southeast, with 481 of them meeting QM Standards. While our initial impetus for using Quality Matters was to increase course completion in our Wintersession, an implementation on such a large scale provided an opportunity to examine a raft of user data and analyze the various results. Here are some of the impacts the data revealed:
Student Engagement Increased
Data was collected on student activity in the learning management system from fully online courses in 2014 (prior to QM Review) and from the same courses with the same instructors offered in 2017 (after QM Review). Looking at the results from Fall, Wintersession, Spring, and Summer, there was a 26% increase in student activity in the LMS. (Shown in "Average Actions Per Student By Semester" graph.)
This finding is similar to the increased student engagement found in Quality Matters courses at FIU Online in the 2016 study The Benefits of Quality Matters Certification: What the Analytics Reveal.
Grades Increased
The increase in student activity as measured by student behavior in the LMS, where nearly all of Southeast’s course content resides, is a positive sign due to the correlation between this increased LMS activity and higher student grades. (Shown in "Student Actions" table.)
To break it down, grades have risen from pre- to post-QM implementation/courses meeting QM Standards by the following percentages:
- Fall – 5%
- Wintersession – 5%
- Spring – 2%,
- Summer – 7%
Where did the increase in A’s come from? In Southeast’s case, the percent increase matched the decrease in B’s and C’s in the courses, while DFW grades remained the same. This is an interesting finding and may reflect that the increased clarity (including explanation of access to student support) and consistency in the courses due to the use of QM may be only part of the solution needed to increase course completion in online courses at Southeast.
A Faculty Community Formed Around Online Pedagogy
The mandate to implement QM established an online teaching community at Southeast. There is now a culture of sharing and experimentation across colleges and departments, and faculty have a structural framework for constructing online courses that support student success. Prior to QM being implemented at Southeast, there were some faculty with little to no training in online education. This comment from a faculty member at Southeast sums up the feeling of many:
“I benefited more from the process than I expected, especially from the facilitated interactions with colleagues.”
All of these benefits are the result of an institutional investment of $467.88 per course. At Southeast, in-state students spend just over $700 per course. An increase in student engagement in online courses, an increase in student grades, and a stronger online teaching community, all at the cut-rate cost of allowing one student in each online course at half-price for a few semesters. We feel it was quite the bargain.
What’s Next?
While all the quantitative data makes a compelling argument, an ethnographic study would help round out the analysis. Testimonials from various types of online students would provide anecdotal evidence and bring much needed voices to these results. Perspectives from students in fully online programs would be most valuable, as their entire Southeast experience exists in virtual learning spaces.
Students earning A-grades in online courses require further study. The range of activity in high-performing students is wide, some logging high numbers of actions and some logging a moderate amount. B- and C-students do not produce the same spread in performance. Comparing the number of actions with quality of submitted work would provide more context regarding how actions reflect engagement. Examining the numbers of actions by the same student across several courses would also give insight into how Southeast students work through their online courses.
While increasing student completion of Wintersession courses was the impetus for using Quality Matters, and this increase did not occur, our data showed that course completion for graduate students in online courses increased by as much as 3%. We know that we need to investigate additional activities to complement the use of QM in order to reach our goal of increased course completion both in Wintersession and in undergraduate online courses taught during other academic terms.
Analysis of the data we have been able to collect on our QM implementation has helped us to both understand the impacts that have occurred, as well as additional efforts we need to undertake to provide a quality online learning experience for students and the instructors who teach them.
