Abstract: This pseudo-experiment looked at the effect of two different email fundraising messages using either policy or values framing on fundraising conversion rates and donation amounts. Half of a randomly chosen group of the actively subscribed Sister District email list received an email from HQ about redistricting with the values framing and half received an email about redistricting with the policy framing. Both emails solicited donations to the same State Bridges partner organizations. In the analysis, the values email appears to have a higher conversion rate, as well as a generally higher donation amount per person, though neither of these differences are statistically significant (p = 0.216 and p = 0.900, respectively). When compared to the average of other emails sent by Sister District HQ to fundraise for State Bridges, conversion rates were similar. This suggests that valus messaging, even one with minimal values-based language, may be superior to policy-based messaging. This tactic should be replicated in more volunteer groups at different times of the year, especially earlier in the year.

Objective: This study, and a similar study run on Facebook and Twitter, focused on the efficacy of fundraising appeals with either a values or policy framing for generating funds for State Bridges partner organizations.

Background: This study was motivated by a desire to explore whether message framing highlighting the values of the target may be more effective in generating contributions than message framing focused on policies preferred by the target. This study used an A/B test type format to investigate which message framing, values or policy, is superior when fundraising. The test presented here was conducted via email. A second test was run on social media, but did not generate donations. Results of the social media are summarized briefly at the end of the blog.

Specifics: A group of actively subscribed members of the Sister District email list were randomized into two different conditions: a more traditional policy messaging condition (confirmed delivery to 25,836 members) and the values messaging condition (confirmed delivery to 22,765 members). Emails were sent from the Action Network platform at 9am ET on October 12, 2021. On October 14, 2021 at 9am ET, a kicker email was sent to folks who had received but not opened the initial email. On October 25, 2021, two weeks after the initial email send, fundraising data was analyzed. Prior donation status to Sister District and its candidates and causes and previous State Bridges interest was also assessed.

Basic Takeaways:

• People in the values condition donated at a higher rate than policy participants, but this difference was not statistically significant.
  • The conversion rate for people who became donors in the values condition was 0.21% and in the policy condition it was 0.16%. This difference of 0.05% was not statistically significant (p = 0.218).
• The linear regression model also indicates that donation amounts were slightly higher in the values condition, but the p value indicates no relationship (p = 0.900).
  • Further, the median amount in each condition was identical at $50.
• Values based language may work especially well for people who have already “bought in” to the mission of the fundraising appeal services.
  • It appears that donors were overwhelmingly people who had already attended a Bridges event or donated to the Bridges organizations before, or previous donors to Sister District in some capacity.
• The conversion rate in this study is slightly lower than the average conversion rate for previous State Bridges fundraising emails, but this difference isn’t significant..
• We are underpowered to detect a small effect of the size observed, and would need a much larger sample to determine if this difference between targets in the two conditions is practically meaningful. However, this may be related to how close this email was to the election as the conversion rate for this email appears low compared to previous Bridges fundraising emails. State Bridges is not the primary focus of the organization, so we may have seen smaller effects here than we may have seen if we had been fundraising for candidates, which might tend to generate higher conversion rates and donations in general.
• This data suggests that, while not statistically significant in this study, people who received values based messaging were more likely to donate, and are potentially more likely to give a modestly higher donation amount in general. This study should be replicated in a larger sample and a different period of time.

Key findings:

1. Results suggest that people who received the values email were more likely to donate, but this effect is not statistically significant.
   • The regression results indicate that people in the values email condition did donate more after receiving the email than those in the policy condition, but this effect did meet the level of statistical significance (p = 0.218). However, this may be due to the low baseline donation rate of this email, which may have left us underpowered to detect this effect.
2. It also appears there is no difference in donation amounts between the two conditions.
   • The regression results indicate that there is no difference in the donation amounts given by people who received the values email compared to the policy email (p = 0.900). However, again, we may be underpowered to detect effects.
   • There was a single very large ($1,000) donation in the policy condition that is more than twice the amount of the next highest donation amount. With this donation amount removed, the average values condition donation is higher than the average policy donation by $4 ($76.54 vs $72.15).
3. Donors were highly likely to be people who had donated to Sister District previously in some capacity, and people who had expressed previous interest in the State Bridges program.
   • It appears that donors were overwhelmingly people who had already attended a Bridges event or donated to the Bridges organizations before, and this was especially true for the values condition (97.87% of donors compared to 92.68% in the policy condition). But this 5.19% difference between conditions wasn’t statistically significant (p = 0.2438).
   • Further, most donors were people who had previously donated to either State Bridges organizations or Sister District candidates, causes or organization, and again this was especially true for the values condition. In the values condition, 82.98% of donors had previously donated, and in the policy condition 75.61% of donors had previously donated. Again, this difference between conditions wasn’t statistically significant (p = 0.393).
4. The October email conversion rate was lower than the average conversion rate for State Bridges emails.
   • The average conversion rate for the March and August Bridges emails was 0.03% higher compared to the conversion rate for the October email
   • The average donation amount for the October email was lower than the average donation for the August email but higher than the average donation for the March email. Both differences were significant (p > 0.001).
   • Timing may have affected donation amounts as this email was sent during early voting for the 2021 Virginia state legislative and gubernatorial elections.

Caveats and considerations

• Generalizability is limited. The study sample for this test was Sister District volunteers, and the email raised for a specific cause ( State Bridges partners), that is unique to Sister District. For this reason, the generalizability of the results is limited.
• We are underpowered to detect effects. The conversion rate for the emails was low, with a post-hoc power analysis indicating that we are very statistically underpowered to detect effects of this size. In a replication of this study, we would want to increase the sample size or email a group of people expected to have a higher conversion rate.
• People were far more likely to engage if they had expressed previous interest. The vast majority of the donors in the study were people who had already donated to Sister District previously in some capacity, or were people who had already expressed interest in the State Bridges program. This indicates that these message frames, or perhaps these message frames delivered via email or pertaining to this specific test, may not be as effective in recruiting new donors.

Contributions and Future Directions:

There is some support for the hypothesis that emails with a values framing were more successful in converting recipients into donors, however this effect did not rise to statistical significance. Further, while the average donation amount was slightly higher in the values condition compared to the policy condition when the outlier donation is removed from the sample, this effect also did not rise to statistical significance. Finally, conversion rates and donation amounts were not necessarily higher than previous emails, with the October email outperforming the March email but the August email outperforming both the October and March emails. This indicates that more research is needed in a larger sample or in a sample with a higher conversion rate.

There are a variety of limitations in the study, including that we are underpowered to detect effects. Some differences may have been statistically significant with a larger sample, for instance, the 5.19% difference in Bridges previous donorship and attendance among the two conditions. Further, this email was sent at a time in the election cycle where volunteers may have been more focused on candidate fundraising, as it was shortly before the November 2, 2021 election. Finally, State Bridges is not the focus of Sister District as an organization and this may indicate that State Bridges is not a program they care as deeply about as candidate fundraising.

Due to the limitations in this study, especially the statistical power, this study should be replicated to determine if these email frames do indeed perform differently. Further, the study should be conducted at a time significantly before or after the GOTV election period.

If you’re interested in reading more about this study, a longer report can be found here.

Bridges social media test

We ran a similar study on Sister District’s Twitter and Facebook accounts where messages that were either values oriented or policy oriented solicited donations to State Bridges organizations. There were no donations as a result of these tweets or posts, therefore the same hypotheses could not be tested. We instead conducted some exploratory testing around social media engagement metrics.

According to Twitter analytics, the values tweets got 0.71% higher engagement rates than the policy tweets, a difference that approached marginal statistical significance  (p = 0.141). Engagement statistics were not available for Facebook, but the values posts got 0.24% more “likes” than the policy posts. This difference was not statistically significant (p = 0.482). This test should be conducted again to determine if post and tweet wording could elicit actual monetary donations rather than just engagement.

SDAN’s commitment: It is SDAN’s intention to provide as much context as possible to allow for the nuanced interpretation of our data. SDAN’s convention is to contextualize effects by reporting p values, confidence intervals, and effect sizes for all models tested (these items may be in the longer report linked in the blog). Additionally, SDAN always differentiates between planned and exploratory analyses and a priori and post hoc tests, and reports the results of all planned analyses regardless of statistical significance. If you have questions about these findings please email Mallory.