Abstract: This study tested the trade-offs between contact rate and the number of conversations with two methods of phonebank calling: letting the phone ring through to voicemail or hanging up the phone if it hasn’t been answered in 3 rings (or 5-6 for people 65+). With the help of 2 volunteer phonebankers, SDAN contacted 815 voters across 9 phonebanks where volunteers used the 3 rings (treatment) method half of the time and the ring through to voicemail (control) condition half of the time. Volunteers recorded how many calls they made and how many voters they contacted. As hypothesized, we found that the voicemail condition was superior for contact rate and the 3 rings condition was superior for the number of call attempts made. Both conditions had approximately the same number of answers. This study has limitations, especially that it was done by only 2 volunteers calling on behalf of a small number of state legislative candidates in a small number of states, and therefore should not be generalized widely. However, the results do provide valuable insight into the phone answering behavior of the current targets of state legislative campaigns and suggests that both of these methods have a place, with an efficiency/contact rate trade-off to make in choosing between them. This study should be replicated in a larger sample, in different types of election years, and in different states.
Objective: This study sought to determine how the contact rates and number of call attempts compared across two methods of phonebanking, one that involves waiting until a call or a voicemail picks up, and one that involves hanging up after 3 rings if no one answers the phone (5-6 for callers over 65).
Background: This study was a pseudo-experiment that investigated the 3 rings tactic, a common phonebanking tactic at Sister District where callers hang up after 3 unanswered rings (or 5-6 for voters 65+). The 3 rings approach is considered a trade-off where contact rate is lower but the number of call attempts are higher. This study compares the 3 rings approach with the standard approach, which is ringing through to voicemail if voters don’t answer.
Specifics:
This study was a pseudo-experiment designed by SDAN. It targeted voters Sister District volunteers called during a phonebank for Sister District candidates. SDAN recruited and trained 2 volunteers who were seasoned phonebankers to call according to the rules relevant to each condition (wait until voicemail to hang up in the voicemail condition and hang up after 3 unanswered rings in the 3 rings condition – 5-6 for voters 65+) and to record the number of calls made, as well as the number of answers in each condition’s call time. Volunteers were instructed to call for both conditions during each phonebank and to spend approximately half of their call time calling for each condition. They were also instructed to counterbalance the two conditions such that the condition rules were randomly used in either the first or the second half of the phonebank each time.
Data were collected for both conditions during 9 different phonebanks from 7/26/2022 to 9/7/2022. An answer was considered someone picking up the phone and speaking to the volunteer in any capacity. Overall, 815 voters were attempted for this study, and 89 contacts were made.
Basic Takeaways:
- The callers in the voicemail (control) condition talked to a larger proportion of the voters they called in the voicemail condition than in the 3-rings condition, as expected.
- The contact rate was 11.99% in the voicemail group vs 10.04% in the 3 rings group. This is not a statistically significant difference (p = 0.376) in this underpowered sample.
- The callers in the 3 rings (treatment) condition called more people than the callers in the voicemail condition, as expected.
- Callers called for approximately the same amount of time in each condition, but there were 367 calls made in the voicemail condition and 448 calls made in the 3 rings condition. This means that 54.97% of the calls made were in the 3 rings condition, compared to 45.03% of the calls made in the voicemail condition. This difference was significant in a two proportions test (p < 0.005).
- It is important to note that there were an almost identical number of answers in both conditions (45 answers for 3 rings and 44 answers for voicemail) even though the 3 rings condition included 81 more phone attempts. This means that even though more calls were being made in the 3 rings condition, that did not equate to a larger number of people answering the phone. On the other hand, this data shows no trade-off in the number of people contacted in an equivalent amount of time spent calling, which indicates that there is not much lost when using the 3 rings method and it could help to get through quick passes of lists to find easily reached voters.
- This study is underpowered to find the contact rate difference between conditions significant. Replication in a larger sample would be useful.
Key findings:
The voicemail condition had a higher contact rate than the 3 rings condition, such that a larger proportion of the people called in the voicemail condition answered the phone compared to the 3 rings condition.
- But this difference of almost 2% was not statistically significant (p = 0.378) in this sample.
- It is likely this size difference would be statistically significant in a larger sample.
The 3 rings condition accounted for 9.94% more of the total calls made than the voicemail condition.
- The 3 rings condition accounted for a significantly larger percentage of the total calls made by volunteers in this study (p = 0.0047). This translated to about a 22.07% increase in calls made in the 3 rings condition compared to the voicemail condition. In other words, a lot more call attempts were made in approximately the same amount of call time in the 3 rings condition.
Caveats and Considerations
- Generalizability is limited. This study only used Sister District trained phonebankers, called for Sister District candidates, and calls happened during a narrow window of 2022. Additional tests are needed to determine if these rates and numbers are reliable.
- We are statistically underpowered to detect some effects. The sample size is small at 815, so the difference in contact rates between the two conditions did not rise to statistical significance. This is likely due to sample size, and this effect would likely be statistically significant in a larger sample.
Conclusions and Future Directions
The hypothesis that the contact rate would be higher in the voicemail condition was true, but it was not borne out at a statistically significant level (p = 0.378) in this sample. Similarly, the hypothesis that more calls would be made in the 3 rings condition compared to the voicemail condition was also true (p = 0.0047). Both conditions saw approximately the same number of answers (44 in the voicemail condition and 45 in the 3 rings condition).
These results indicate that there are no statistically significant downsides to using the 3 rings method, though it should be noted that we are underpowered to detect effects and the difference of almost 2% in contact rate would likely be statistically significant in a somewhat larger sample.
At this effect size, the next pilot should be larger to be adequately powered and to determine if the difference in contact rate rises to the level of statistical significance. The current results suggest that both approaches may have different advantages and uses depending on the phone program’s goals.
If you’re interested in reading more about this study, a longer report is here.
SDAN’s commitment: It is SDAN’s intention to provide as much context as possible to allow for the nuanced interpretation of our data. SDAN’s convention is to contextualize effects by reporting p values, confidence intervals, and effect sizes for all models tested (these items may be in the longer report linked in the blog). Additionally, SDAN always differentiates between planned and exploratory analyses and a priori and post hoc tests, and reports the results of all planned analyses regardless of statistical significance. If you have questions about these findings, please email Mallory.
