Abstract: This investigation measured the downstream (general election) effects of sending completely handwritten or partially handwritten postcards with pro-choice messaging to likely pro-choice voters in Arizona ahead of the 2022 primary. In the primary election study, a sample of 60,000 registered AZ voters who met the inclusion criteria (Democratic-leaning, high modeled pro-choice support scores, and voted in the 2020 general election but not the 2016 general election) were randomly assigned to receive either a completely handwritten postcard detailing the 15 week abortion ban passed by the GOP-controlled state government, a partially handwritten postcard with the same message, or no communication. The analysis did not yield any statistically significant effects in the primary election, likely due to being statistically underpowered, but trends indicated that the completely handwritten postcards performed better in terms of voter turnout than both partially handwritten postcards (by 0.52%) and no communication (by 0.39%). This pattern did not persist in the downstream general election effects, where turnout rates across the 3 conditions were essentially the same: the turnout rate among participants in the control group was 0.07 percentage points higher than in the completely handwritten postcard group, and only 0.04 percentage points higher than in the partially handwritten postcard group. There was a marginally significant interaction effect of pro-choice support scores on turnout in the partially handwritten condition—partially handwritten postcards had a larger positive effect on turnout among people with higher levels of issue support (scores closer to 100), compared to those with lower levels of issue support (scores closer to 70). Overall, the results indicate that sending these postcards ahead of the primary election had no significant downstream effects on general election turnout.

Objective: This study analyzed the downstream general election effects from a prior test conducted during the 2022 Arizona primary. The original study explored the question of whether volunteer-written postcards to voters are as effective when they are only partially handwritten as compared to when they are entirely handwritten. 

Background: The initial study used a randomized controlled trial (RCT) designed to compare the effects on turnout of sending completely handwritten or partially handwritten postcards with pro-choice messaging to likely pro-choice voters in Arizona ahead of the 2022 primary. This subsequent analysis sought to determine if there were any downstream effects of having received these postcards (during the primary) on voting in the general election.

Specifics: The original study was an RCT designed by SDAN. The study targeted a randomly selected subset of 60,000 registered Arizona voters who met the inclusion criteria. These voters were randomly assigned to 1 of 3 conditions, resulting in 30,000 voters in the control condition and 15,000 voters in each of the two postcard treatment conditions. 

In the control condition, targets received no communication. In the postcard conditions, targets received a postcard with the same image, the same script, and the same highlighting and marks of emphasis. In the completely handwritten condition, all elements of the postcard were handwritten. In the partially handwritten condition, only the address, the greeting, and the volunteer name were handwritten, and the main message was pre-printed. 

SDAN provided training, detailed instructions, and a script. Volunteers completed the postcards, and we estimate that they were delivered to homes from July 20-22, 2022. We then evaluated their impact on the sample’s turnout rate in the primary election. After the general election, data was matched back to TargetSmart’s voter file to determine if the voters targeted in this study voted in the 2022 Arizona general election (i.e., to measure any downstream effects of the postcards).

Inclusion criteria: The targets were registered Arizona voters who 1) are categorized as “Biden surge” voters (people who voted in the 2020 general election but did not vote in the 2016 general election, and may or may not have voted in the 2018 general election); 2) have modeled pro-choice issue support scores of 70+; 3) have partisanship scores of 80+; and 4) were born on or before October 1, 1998 (i.e., were old enough to have voted in the November 2016 election).

Basic takeaways: 

There were no significant effects of receiving a completely handwritten postcard or a partially handwritten postcard ahead of the primary election on voting in the general election. Receiving either type of postcard during the primary did not affect whether the recipient voted in the general election.

There is no evidence to suggest that the completely or partially handwritten postcards had significantly different effects depending on how likely people were to vote in a midterm general election.

The results indicate that the partially handwritten postcards had a larger positive effect on turnout among people who were highly likely to support abortion rights, compared to people who were more moderately supportive of abortion rights.

Overall, it does not appear that the postcards had any downstream effects in the general election, but there are limitations to this study. It focused on state legislative districts in a single state (Arizona), and targeted a narrowly defined sample. Results from this test should not be widely generalized.

Key Findings: 

In the main model, there were no statistically significant effects of receiving a completely handwritten postcard or a partially handwritten postcard ahead of the primary election on voting in the general election. 

• Both treatments—completely and partially handwritten postcards—were positively associated with voting, but not to a statistically significant level (p = 0.80 and p = 0.59, respectively).

There was no significant moderation effect based on modeled midterm turnout score. 

• There were no statistically significant interaction effects for the completely or partially handwritten treatment conditions and midterm turnout score (p = 0.396, p = 0.955) on voting. This indicates that people’s responses to the postcards did not vary significantly based on how likely they were to vote in the midterm election in the first place.

There is a marginally significant interaction effect between pro-choice issue support scores and treatment condition: among voters who received partially handwritten postcards, the relationship between pro-choice score and voting is positive (p = 0.06).

• Treatment effects in the partially handwritten condition increase overall, though not consistently, as issue scores increase (see Fig. 1).
• We do not see this pattern among voters in the completely handwritten condition; in this case the coefficient on the interaction term is much smaller and actually negative, and it is not statistically significant (p = 0.565).

Caveats and Considerations:

Generalizability is limited. The inclusion criteria used in this study were quite specific, making it difficult to generalize these results to other election contexts or populations. Future studies using different issues and different samples of voters should help us to better understand the effectiveness of partially handwritten vs. completely handwritten communications.

Treatment effects are expected to decay over time. The actual intervention (the postcards) occurred before the primary election, which was months before the general election in November. So, any short-term effects on turnout are likely to have decayed during the months between the primary and the general elections.

Conclusions and Future Directions:

Similar to the original primary election study, there were no meaningful treatment effects—for completely handwritten or partially handwritten postcards—on voting in the general election. These results indicate that, in addition to no effects in the original study, there were also no downstream effects from these postcards. Interestingly, the pattern we observed in the original study (i.e., completely handwritten postcards had the largest positive impact on voting) did not persist; in the general election, the voter turnout rate was highest among people in the control condition.

There was a marginally significant interaction effect between pro-choice issue support scores and treatment condition. The results indicate that partially handwritten postcards had a larger positive effect on turnout among people with higher levels of support for abortion rights (scores closer to 100), compared to those more moderately supportive of abortion rights (scores closer to 70). We do not see this pattern among voters in the completely handwritten condition, where the coefficient on the interaction term was smaller, negative, and not close to being statistically significant. It may be the case that people with the highest support for pro-choice policies were sufficiently motivated by the partially handwritten postcards and did not need the fully personalized, completely handwritten touch to feel compelled to vote.

There are some limitations to this study. The intervention happened during the primary, and this is a downstream investigation of effects happening months after the initial treatment. Any treatment effects are very likely to have decayed during this period of time. The study also focuses on a very specific group of people: Biden surge voters who are strongly pro-choice, Democratic-leaning, and live in Arizona. This indicates that these results should not be widely generalized, but rather serve as an initial data point. Finally, the sample of people in this study voted at a relatively low rate in the 2022 general election (49%), compared to the state as a whole (~63%), suggesting that this sample is not representative of all AZ voters.

Future studies that focus on a comparison between completely and partially handwritten postcards should consider using larger sample sizes in more competitive elections that elicit higher voter turnout. They may also explore different issues (other than abortion rights) with more generalizable populations.

If you’re interested in reading more about this study, a longer report is here.

SDAN’s commitment: It is SDAN’s intention to provide as much context as possible to allow for the nuanced interpretation of our data. SDAN’s convention is to contextualize effects by reporting p values, confidence intervals, and effect sizes for all models tested (these items may be in the longer report linked in the blog). Additionally, SDAN always differentiates between planned and exploratory analyses and a priori and post hoc tests, and reports the results of all planned analyses regardless of statistical significance. If you have questions about these findings please email Jillian.