Abstract: This pilot study tested whether showing volunteers attending a postcarding event a video about the relative efficacy of campaign field efforts could move them “up the ladder” to engage in more effective activities like phonebanking. Sister District Action Network (SDAN) teamed up with a Sister District team holding 2 different postcarding events on August 26 and August 27, 2019. At one of the events, the team showed a video that Sister District had created about how phonebanking and canvassing were relatively higher ROI activities than postcarding. Volunteers who attended the two events, but who had never done higher stakes volunteering before, were followed until the November 5, 2019 election to see if they attended additional events and if they moved up the ladder of engagement to higher stakes volunteer activities (i.e., direct voter contact activities like phonebanking or canvassing). Results indicate that the intervention did not have an effect on volunteerism. The outcome data was sparse and indicated the need to replicate this study in a larger sample.
Objective: This pilot study explored whether showing volunteers an educational video about the relative value of various volunteer activities could move them “up the ladder of engagement” to more high stakes volunteer activities.
Background: Handwritten postcarding has become a popular volunteer activity over the past few years. However, research indicates that direct voter contact activities like phonebanking are more effective. Therefore, there is a need to move volunteers “up the ladder of engagement.” Little is known about how to move volunteers up the ladder, so SDAN designed a pilot study to investigate how volunteer education around the efficacy of field tactics might affect later volunteerism.
Specifics: SDAN designed this mixed-methods study. One large team in the Western part of the United States that holds regular events partnered with SDAN to complete the experiment. SDAN contacted volunteers on this team via text message to recruit them for the events to increase event attendance. Once recruitment ended, this team provided SDAN with a list of recruits for two postcarding events: one the night of August 26, 2019 and one the night of August 27, 2019. All recruits for the events were asked to complete a sign-up form where they related their prior involvement with Sister District and answered a question about their social anxiety. Only recruits who reported never volunteering or only donating or postcarding before were included in the study. The video was shown to volunteers at the event on the 27th. It was also emailed to attendees the following day. Volunteers from both postcarding events were followed throughout the election cycle to see if they attended more events in 2019 and if so, what those events were (i.e., postcarding or higher stakes volunteering). Events attended were tracked via sign-in forms collected at team events by the team participating in the study. The team then relayed this information to SDAN. The final sample size was 32 volunteers.
Basic takeaways
- First and foremost, the omnibus test for this model is not significant (p = 0.6485). This means that it is statistically inappropriate to draw conclusions from these pilot study results. Further, results for individual variables should not be interpreted as meaningful. In keeping with SDAN’s mission to release all study results to avoid the file drawer problem, planned and exploratory tests are detailed below. However, results should be considered purely educational, not as reliable evidence for or against the use of this educational video intervention.
- In this sample, no one moved up the ladder of engagement. Only 2 volunteers returned for another event and both attended another postcarding event. Therefore, no one in the sample moved up the ladder of engagement to more high stakes activities.
- People in the treatment condition attended fewer events than people in the control condition, but this effect did not approach statistical significance (p = 0.245).
Key Findings
- People in the video intervention group were not more likely to volunteer again than people not in the video intervention group.
- Volunteers who saw the video explaining that the activity they are participating in was relatively lower value for campaigns than phonebanking were no more likely to volunteer again than people who did not see the video (p = 0.245). It is important to note that the omnibus test for this model was not significant, indicating that individual results should not be interpreted as meaningful. Further, this test is not significant, indicating that we are underpowered and the data is too limited in range to detect any effects.
- Social anxiety was unrelated to volunteering again in this sample.
- Social anxiety, which may explain why some volunteers do not want to do higher stakes volunteering with real time conversations with voters, was not a predictor of the number of events attended (p = 0.690). Again, please note that neither the omnibus test nor the test for this variable was statistically significant.
Caveats and considerations
- This an underpowered pilot study. Due to the small sample size, the range of the outcome data was very restricted (only 2 people really had outcome data – i.e., volunteered again), making it difficult to find anything in the results. A larger sample size would be needed to shed more light on the efficacy of this tactic.
- This study used a unique sample of voters. This study not only focused on Sister District volunteers, but on volunteers in a single geographic area. It is unlikely that this sample is representative of the larger progressive volunteer population.
- The non-significant omnibus test means these data should not be used. We reported this data to be transparent, but the tests presented in this blog are not statistically meaningful and should not be interpreted as such.
Contributions and future directions
This study is the first experimental attempt, that SDAN is aware of, to move volunteers up the ladder of engagement to higher stakes volunteer activities. While this study was not successful, it helps to establish expectations for future research, and highlights the difficulty of the task of moving folks up the ladder in the first place. Future research, with larger samples, is needed to determine how effective an educational video intervention would be in increasing commitment to try higher stakes volunteering.
If you’re interested in reading more about this study, a longer report is here.
SDAN’s commitment: It is SDAN’s intention to provide as much context as possible to allow for the nuanced interpretation of our data. SDAN’s convention is to contextualize effects by reporting p values, confidence intervals, and effect sizes for all models tested (these items may be in the longer report linked in the blog). Additionally, SDAN always differentiates between planned and exploratory analyses and a priori and post hoc tests, and reports the results of all planned analyses regardless of statistical significance. If you have questions about these findings please email Mallory.