Abstract: Sister District Action Network (SDAN) and the Mississippi NAACP (MS NAACP) partnered to conduct an experiment around GOTV text messaging during the days leading up to the general election in Mississippi on November 5, 2019. The study targeted predominantly Black / African-American voters whose prior voting history indicated they would be unlikely to vote in the 2019 general election. The MS NAACP sent GOTV text messages to half of the voter targets, and then all targets were tracked to determine if they voted in the 2019 general election. The results indicate that people were less likely to vote in the 2019 general election when they received a GOTV text message, and this was especially true for people who received the persuasion message as opposed to the shortened message.

Objective: This experiment explored whether Mississippi voters who received a GOTV text message from the Mississippi NAACP (MS NAACP) asking them to vote in the 2019 general election had a higher rate of turning out to vote than voters who did not receive a text message; and further, whether voters who received a message with persuasion content voted at a higher rate than those who received a shorter / simpler GOTV text.

Background: Text messaging has become a popular political tactic in recent elections, and a good amount of industry research has tested messaging, timing, and warmth of contact (warm contacts are people who gave the organization their information with the knowledge they may be contacted). Less is known about whether text messaging efficacy may vary by the presumed race or regional location of the recipient. Further, industry research generally suggests that GOTV messaging should be information-forward and should minimize persuasion about candidates and parties. However, persuasion messages generally do not focus on why the act of voting itself is important. This study sought to look at the effect of GOTV texting in a relatively high profile odd-year election among Black / African-American low propensity voters, and further, to test if using persuasion messaging around the act of voting itself may boost turnout.

Specifics: SDAN ran this study with MS NAACP. MS NAACP generated a list of registered voters in almost all state legislative districts in Mississippi who were considered non-white, low propensity voters. SDAN matched that list to the TargetSmart voter file to obtain cell phone numbers for as many targets as possible. “Low propensity” voters were defined as voters who had a turnout score of 75 or below for off year elections, who voted in 2016 but not since, who were newly registered, or who had voted in 2008 and 2018 but not in the elections between those years. Further, voters needed to have a cell phone number listed to be included in the sample. This yielded a list of 31,844 voters. The list was randomized and subjects were randomly assigned to two conditions: Control (no text message) and GOTV text (received a text message). MS NAACP volunteers and staff send text messages to people in the GOTV text condition on November 3, 2019.

While sending the messages, the texting platform suggested that a shorter message would increase response rates. As a result, MS NAACP switched to a shorter message that did not include persuasion content. This was not a planned experimental factor, meaning results are exploratory and cannot be generalized, as people were not randomly assigned to conditions, but it does allow for analyses comparing the efficacy of the two different messages.  See Appendix for the text of the messages sent.

It is important to note that 465 people assigned to be text messaged were unable to be messaged for various reasons (e.g., because the number is disconnected, the number was accidentally changed in a record, the number is actually a landline or fax machine and can’t be text messaged, etc). This put 16,387 people in the no message control group in the message type exploratory analysis, while there are only 15,922 people assigned to the no message control condition.

Takeaways:

1) Receiving a text message made people less likely to vote. Voters assigned to the text message condition were less likely to vote in the 2019 general election than voters who did not receive text messages. However, this result was not statistically significant (p = 0.121). This indicates that being assigned to the texting condition had no significant effect on voters’ behavior, although, in practice, was associated with decreased voter turnout compared to the control condition.

  • 2) Both messages performed worse than no message, but the persuasion message was significantly worse. 
    • The persuasion message performed statistically significantly worse than the control condition, meaning that receiving a persuasion text message was associated with voter backlash (p = 0.012). This means that voters who received a persuasion text message were statistically significantly less likely to vote than people who received no text message.
    • The shortened message was also associated with lower voter turnout than the control condition, but it was not statistically significant (p = 0.584), indicating that the shortened message did not motivate voters to the polls but the backlash associated with that message was not statistically meaningful.
    • This finding suggests that neither GOTV text message was effective in motivating voters to vote, and that the persuasion message actually decreased voter turnout compared to not sending a message at all.3) The persuasion message caused significantly more backlash than the short message. When directly comparing the turnout in the persuasion and shortened message groups, there is a significant difference in turnout between the two types of text messages (p = 0.043). This indicates that the persuasion message not only performed worse than no message, it also performed significantly worse than the short message. In other words, voters who received a persuasion text message were statistically significantly less likely to vote than voters who received a short text message.
    • 4) Basic takeaways:
    • In this study, receiving a GOTV text message had a negative impact on voter turnout.
    • The persuasion message led to the lowest voter turnout (10.64%) of all 3 message groups, and had significantly lower voter turnout than the short message group (11.89%). Both groups had lower turnout than the no message control group (12.42%).
    • The difference between the efficacy of not receiving a message and receiving a short message is not statistically significant in this sample.
    • Overall, the data suggest that the GOTV text message did not work to mobilize voters as intended, and may have actually resulted in lower voter turnout in this sample.

Caveats and considerations

  • Study focused on a niche group of voters. This study focused on a very specific group of voters (mostly Black / African-American Mississippi voters who were considered low likelihood voters for an off-year Gubernatorial/state legislative election), and as such these results should not be widely generalized.
  • Off year elections generally have low turnout.  This appears to be true in this sample, as the average voter turnout score was quite low.
  • Different messages were not planned. Because the two types of messages were not randomly assigned conditions, there are limitations in the generalizability of data focusing message type (no message, persuasion text, or short text).
  • Short text message vs control test is underpowered. The study was also slightly underpowered to detect the effect of condition (text message vs no).
  • Nothing in the data suggests this sample was irregular. Covariates like age, gender, and turnout score behaved as expected and do not appear to account for the backlash we observe in this sample.

Contributions and Future Directions:

This study helps to clarify the utility of GOTV text messaging to low propensity Black / African-American voters in Mississippi ahead of an off-year Gubernatorial election. It also helps to clarify the utility of persuasion messaging around voting itself, as opposed to around candidates or parties. The results provide some evidence that GOTV text messaging may not be a good tactic to mobilize these voters. Further GOTV text messaging tests are needed to clarify these effects and to determine if one would expect backlash in a more high profile election, like a Presidential election. A larger group of targets would be needed to be adequately powered to test another non-persuasion short message.

 

If you’re interested in reading more about this study, a longer report is here.

SDAN’s commitment: It is SDAN’s intention to provide as much context as possible to allow for the nuanced interpretation of our data. SDAN’s convention is to contextualize effects by reporting p values, confidence intervals, and effect sizes for all models tested (these items may be in the longer report linked in the blog). Additionally, SDAN always differentiates between planned and exploratory analyses and a priori and post hoc tests, and reports the results of all planned analyses regardless of statistical significance. If you have questions about these findings please email Mallory.