Abstract: This study tested whether voter turnout increased after voters received a handwritten postcard featuring ‘values-matched messaging,’ such that they highlighted policy votes that their state legislative representatives made in opposition to the voters’ modeled policy positions. The study was a randomized controlled trial run during the Virginia 2021 general election that included a control condition, a social pressure mailer condition (voters received a printed non-issue specific, social pressure mailer), and a postcard condition (voters received a handwritten postcard with issue-specific messaging). In the analysis, the numbers trended in the expected direction, with more voters in the postcard condition voting than voters who did not receive a postcard (p = 0.064).

We used propensity score matching to address a small baseline difference between voters in the study and to generate a meaningful practical effect size. These results indicated that receiving a postcard had a larger than expected effect compared to not receiving anything (0.67%; p = 0.007).

The treatment conditions had a difference in turnout rate of 0.49%, with postcards turning out more voters than social pressure mailers, but the difference was not statistically significant (p = 0.157). Importantly, moderation analyses suggest that the postcard intervention appeared to work better on people with lower predicted turnout scores for odd-year general elections and on people with higher issue support scores. This sample is not widely generalizable, so this tactic should be replicated. However, initial results are promising and merit further exploration.

Objective: This study’s aim was to explore whether sending handwritten postcards to voters educating them about their state legislative representatives’ voting records on issues they care about could boost voter turnout in the 2021 Virginia general election. We also wanted to test how postcards would perform compared to social pressure mailers, a gold-standard voter outreach tactic.

Background: This study was a randomized controlled trial (RCT) that investigated the effects of sending voters issue-matched postcards on voter turnout. It also included a condition with social pressure mailers, considered a gold standard of Get Out the Vote (GOTV) mail, to provide a baseline for an effective GOTV tactic in addition to the control condition, where targets received no communication.

Specifics:

This study was a randomized controlled trial (RCT) designed by SDAN. The study targeted Virginia voters who met the inclusion criteria: living in a district with a Republican incumbent state legislative candidate being challenged by a Democrat, 70+ issue scores on at least one of three issues highlighted in the postcard messages (pro-choice, climate change, gun control), 70+ partisanship scores, and predicted turnout score of 80 or lower. We only sent postcards to voters whose Republican incumbent state legislator had an ‘F’ on the issue highlighted in the message based on legislator report cards (find more information about criteria and targeting in the short report).

A total of 237,987 targets were sorted into message groups (pro-choice, climate change, gun control) and were included in the study. Of those targets, 28,000 people received social pressure mailers, 79,406 received postcards, and the remainder were assigned to the control condition and received no communications. Volunteers completed the postcards according to the voters’ assigned message groups (pro-choice, gun control, and climate change), and sent them in bulk to an in-state partner in Virginia for in-state mailing around October 19-20, 2021. We estimate that they were delivered to homes from October 23-October 27, 2021. After the election, data was matched back to TargetSmart’s voter file to determine if voters targeted in this study voted in the 2021 Virginia general election.

Post-hoc, we found that predicted voter turnout was significantly related to inclusion in the control group. This led us to undertake a statistical procedure known as propensity score matching to adjust estimates of the effects of the treatment groups (postcards, social pressure mail). Only these adjusted estimates are presented here. Please see the short report linked at the end of this blog for more details on the method.

Basic Takeaways:

• People who received a postcard voted at a higher rate than people in the control condition who received nothing. This difference in turnout was approximately 0.67% and was statistically significant (p = 0.007)
• People who received a social pressure mailer voted at a higher rate than people in the control condition. This difference in turnout was approximately 0.41% but was not statistically significant (p = 0.325).
• People who received a postcard voted at a higher rate than people who received a social pressure mailer. This difference in turnout was 0.49%, and while it was not statistically significant (p = 0.157), it approaches marginal significance.
• Our moderation analyses showed that people with lower predicted turnout scores and higher predicted issue scores were more likely to vote after receiving the postcard compared to people with higher predicted turnout scores and lower predicted issue scores.
  • This is important, because it is really critical to find ways to motivate people who are unlikely to vote. This study indicates that this tactic may work particularly well with less likely voters.
• This study suggests that, while not all effects are statistically significant in this study, postcards performed the best for increasing turnout, followed by social pressure mailers, and controls turned out the least.
• While the propensity score matching method we used addressed the predicted turnout score confound, this study should be replicated using either a single issue or with a sample that meets the 70+ score criterion for all message groups.

Key findings:

Results suggest that people who received postcards voted at a higher rate than people who received nothing (controls).

• People who received postcards voted at a rate that was about 0.67% higher than people in the control condition, who received no communication. This difference was statistically significant (p = 0.007). This means that the postcards appear to have had the intended effect of boosting turnout in the general election.

People who received a social pressure mailer voted at a higher rate than people in the control condition. 

• Results suggest that people who received the social pressure mailer turned out to vote at a rate that was approximately 0.41% higher than people in the control condition, but this effect was not statistically significant (p = 0.325).
• However, it is important to note that, due to the nature of propensity score matching, this comparison is statistically underpowered. This essentially means that this effect might be too small to be statistically significant in the sample size for this comparison, but it may be statistically significant in a sample with more people.

People who received a postcard voted at a higher rate than people who received a social pressure mailer. 

• Results suggest that people who received the postcard turned out to vote at a rate that was approximately 0.49% higher than people in the social pressure mail condition, but this effect was not statistically significant (p = 0.157) .
• Again, it is possible that this effect would rise to the level of statistical significance in a larger sample.

Our moderation analyses showed that people with lower predicted turnout scores and higher predicted issue scores were more likely to vote after receiving the postcard compared to people with higher predicted turnout scores and lower predicted issue scores.

• These interaction effects between condition and turnout (p = 0.033) and between condition and issue scores (p = 0.050) were statistically significant.
• There were no significant interaction effects involving social pressure mail.

Caveats and considerations

• Generalizability is limited. There were extensive inclusion criteria in this study that make it difficult to generalize these results to other election contexts or populations. Future studies will help to determine the boundaries of this particular effect.
• We are underpowered to detect some effects. We are generally underpowered to detect social pressure mail effects in the propensity score-matched model. This means that these tests should be replicated in a larger sample of people.
• Predicted turnout was different for the control group. During the course of this study, we found that predicted turnout is closely related to the issues people care about. Because of this, as well as aspects of the study design, the average predicted turnout for the control group was naturally lower than the average predicted turnout in the postcards and social pressure groups. We have statistically compensated for this in the propensity score-matched results. Future tests should avoid this complication by having only one message group, or narrowing criteria to only people who qualify for all message groups.

Contributions and Future Directions:

This study provides evidence that sending voters postcards that communicate messages speaking directly to a voters’ values (or what we call ‘values-matched messaging’), are indeed effective.

Further, the size of the effect observed here is more than twice as large as any postcard effect we have observed in a general election, including other odd-year elections.

Finally, based on the size of the effects, it appears that postcards actually outperformed social pressure mail, which is widely considered to be a gold standard for direct voter mail. While more research is certainly needed to determine if these effects are reliable, this initial test is extremely promising and suggests that values-matched messaging should be explored further.

There are a variety of limitations in the study, including the confounding effect of turnout on the control condition described above. Regression did compensate for this confound with the use of covariates, and we engaged in propensity score matching to generate meaningful practical effect sizes and significance tests of those differences. While this does allow us to explore the results, it is not a replacement for removing the turnout issue altogether in future studies. This can be done in two ways: limiting to a single message group or limiting the sample to only people who qualify for all possible messages.

Further, undertaking propensity score matching meant a decrease in statistical power, which affected comparisons with social pressure mail. In a larger future sample, the difference between social pressure mailers and controls may rise to statistical significance. Finally, this was a very specific sample of voters, living in very specific districts. Future studies should determine if criteria need to be this strict to detect an effect, or if the effect persists with looser targeting.

This study should be replicated to determine if these effects persist. Further, the study should be replicated in a noisier election year to get a better idea of efficacy in an election where voters are receiving more appeals, like a midterm or presidential election. However, the results are extremely promising, particularly with regard to motivating less likely voters.

If you’re interested in reading more about this study, a longer report can be found here.

SDAN’s commitment: It is SDAN’s intention to provide as much context as possible to allow for the nuanced interpretation of our data. SDAN’s convention is to contextualize effects by reporting p values, confidence intervals, and effect sizes for all models tested (these items may be in the longer report linked in the blog). Additionally, SDAN always differentiates between planned and exploratory analyses and a priori and post hoc tests, and reports the results of all planned analyses regardless of statistical significance. If you have questions about these findings please email Mallory.