Political Manipulation with Massive AI Model-driven Misinformation and Microtargeting – Sophos News

In today’s digitally connected world, political messaging and misinformation are becoming increasingly sophisticated. Political campaigns and misinformation efforts, particularly those that are well-funded, have significant societal impacts. These campaigns have historically exploited political and ideological views to resonate with people, convince them to act, or even lure them into scams.

Generative AI technologies such as large language models (LLMs) and large image models will likely transform the domain. Generative AI offers tools for creating sophisticated, individualized content at scale, which was previously difficult and labor-intensive. With these capabilities, the risk posed by malicious actors can reach new heights.

We have already observed a variety of actors abusing generative AI as part of ongoing fraud campaigns, including the use of generative text to send messages to scam victims, generative AI images to create deceptive social media and “deepfake” video and voice created by AI to aid social engineering of victims. These same tools have been used as part of political misinformation and deception campaigns on social media.

Given the relevance of these subjects due to ongoing elections worldwide, understanding the effect of new technology on political misinformation is particularly consequential. In this analysis, we explore one of the greatest emerging threats from malicious use of generative AI: tailored misinformation. If someone includes intentional misinformation in a bulk email, people who don’t agree with that misinformation will be turned away from the campaign. But in the method we explored in our research, misinformation is only added to the email when that specific individual is likely to agree with it. The ability to do this can completely change the scale at which misinformation can propagate.

In the research we document in this report, we aimed to uncover potential methods in which adversaries could apply generative AI tools to make impactful changes in the political sphere. These methods use current generative AI technologies in a way that can be executed at very low cost by a wide range of potential actors who wish to influence politics on a small or large scale.

This effort was based on research we’ve already conducted, in which we developed a tool that can automatically launch an e-commerce scam campaign based on AI-generated text, images, and audio to create diverse and convincing fraudulent webstores. An example of one these websites is shown in Figure 1; a full description of the research can be found here.

Figure 1: A fully functioning webstore with Facebook and payment credential theft. Everything you see is generated by AI, including code, text, images, and an audio testimonial

We believed that the sophistication of the content created by this tool demonstrated that this approach has potential in the political sphere, so we reconfigured it to generate political campaign websites with any characteristics of our choosing.

We then extended the research, putting the concepts of political emails and fraud campaigns together by generating microtargeted emails. By combining fake social media profiles with several campaign websites, we then generate personalized emails that use unique, individualized arguments to best persuade people to believe in a campaign – even if the people don’t generally agree with those ideas.

Massive model-driven misinformation

To start building towards the future of political misinformation, we first had to augment our tool that generates large scale e-commerce phishing campaigns entirely with generative models . This tool uses Auto-GPT to allow GPT to communicate with itself, as well as other agents such as Stable Diffusion, to complete a large task. We already saw an example of a phishing website in Figure 1, but with minor modifications, we were able to reconfigure the tool to create political campaign websites. To avoid political affiliations and demonstrate more extreme campaigns, we intentionally use fantasy scenarios and worlds. Figure 2 illustrates all the components of one of these websites.

Figure 2: Three screenshots of a single, vertical website for an Elven political campaign. (Click to expand) All the art, ideas, and text are automatically generated by AI. Picture (a) shows the top the website, with a fully functioning navigation bar, an intro to the campaign, and relevant art given the context of the campaign. Using the navigation bar or scrolling down reveals the “About Us” and “Achievements” sections displayed in (b), which describe the background of the campaign. (c) shows the bottom of the page, containing future political goals, pictures of the candidate and important team members, and a set of contacts

With a little imagination (and nothing more than changing a few words in the input to the tool), we can see that any level of fabrication is possible with these websites. This means a user could generate anything from benign campaign material to intentional misinformation and malicious threats with minor reconfiguration. It is possible to associate any real political movement or candidate with supporting any policy, even if they don’t agree. Intentional misinformation like this can make people align with a candidate they don’t support or disagree with one they thought they liked. Furthermore, the tool can generate hundreds of diverse websites with the press of a button, making the threat from small organizations or even individuals a relevant concern.

We took advantage of our tool to create four fantasy campaigns with different alignments, generally touching on different parts of the political spectrum. Because we will be combining social media profiles with these campaign websites to generate microtargeted emails in the next section, we wanted to ensure there would plenty of conflicting views. Thus, including some more extreme perspectives will allow us to showcase the depth to which the AI can deceive people who disagree with the perspectives. Figure 3 summarizes the general ideas of each campaign.

Figure 3: Summaries of the four fantasy campaigns. (Click to expand) (a) displays a relatively progressive agenda for the Elven campaign: supporting education, renewable energy, research and innovation, and social equality. In picture (b), the Dwarven candidate is focused on the economy with goals such as encouraging trade, strengthening mining industries, lowering taxes, investing in more infrastructure, and increasing military spending. The Orcish campaign(c) has more national security focused goals—after successfully expanding their territory through warfare and establishing defense agreements, they want to have a strong focus on the military with policies such as mandatory service, continued defense spending, and support for veterans. Finally, picture (d) explains the goals of a thoroughly authoritarian demonic campaign. They have already conquered several realms, and they seek complete domination through military action. They want to levy taxes against the defeated and develop a portal system to ensure they can take swift action against potential dissidents

Large-scale microtargeting

With the campaigns completed, we need social media profiles before we analyze the ability for LLMs to specifically target individuals. The user profiles simulate data harvested from social media and other public domains, while the fantasy websites provide a topic to strive for. We generate synthetic user profiles with a simple yet effective prompt to OpenAI’s GTP-4o model, “Please generate a detailed user profile. Make it seem realistic. Name, location, age, occupation, likes, dislikes, frequent places, hobbies, political leanings, and anything else that you think is relevant. Make it seem as if the data were realistically scraped from social media.”

The resulting profiles include significant amounts of detail, so we summarize some of the key information in Table 1. We selected five user profiles that fall across the political spectrum. Next, we combined the campaign websites with each individual user profile in a prompt requesting a targeted email.  Using the same website with different profiles demonstrates the ability of LLMs to microtarget users with persuasive arguments best suited to the users’ preferences. The resulting 20 emails will allow us to analyze gradual changes in microtargeting as the political views of the campaigns/individuals diverge.

The technical procedure is simple: Include both a synthetic user profile and a campaign website as input to an LLM, with a prompt asking the model to write an email that would be most persuasive for convincing a user with the given interests to support or donate to the political campaign. The full prompt is written out in the appendix to this post. [[anchor link to appendix here]] Of note is the use of chain-of-thought prompting, which improves LLM performance by asking the model to write out the steps of the “thought” process before generating the final output. In our prompt, asking the model to cite specific examples from the user profile ensures a deeper targeting of the individual in the resulting email.

In the email displayed in Figure 4, we combine the avaricious Dwarven campaign (Figure 3b) with the moderate user (Table 1-3), and the result showcases one of the biggest threats that AI-based microtargeting introduces: You can perform misinformation at a scale that is simply infeasible with existing methods. The model fabricates a point contrary to the Dwarves’ goals, saying the campaign supports social justice issues because it appears on the moderate user’s profile (User 3). This doesn’t work without personalization, because a mass email about supporting social justice may actually lose people who don’t vote or donate based on that issue. By fabricating that point specifically to people who support it, the lie is more insidious and effective.

Table 1: Overview of AI-generated social media profiles of users across the political spectrum. (The full profiles are extensive, so we only include highlights here.)

### Connection Analysis between User Profile and Campaign




1. **Educational Reform**:

- **Occupation/About**: "Elementary School Teacher," "Dedicated to fostering curiosity and a love of learning."

- **Political Leanings**: "Focus on educational reform."

- **Campaign Points**: "Develop a public education system to teach dwarves essential skills for the modern world."




2. **Environmental Conservation**:

- **Likes**: "Beach outings," "Participates in beach clean-up events and environmental conservation projects."

- **Political Leanings**: "Environmental conservation."

- **Campaign Points**: "Encourage sustainable mining practices" and "Implement strict regulations on the use of magic to protect the environment."




3. **Community Involvement & Social Justice**:

- **Additional Info**: "Active in the local teachers' association, advocating for better resources and support for educators," "Active in community events."

- **Political Leanings**: "Social justice issues."

- **Campaign Points**: Can fabricate policy regarding "Promoting community involvement and supporting local initiatives."

I hope this email finds you well. My name is Dvalin, and I am reaching out to introduce you to Dvalin Goldseeker's campaign for a Golden Age of Prosperity in Eldurheim.

Our campaign is deeply committed to enhancing the educational system, focusing on building a robust public education platform that promotes essential skills and lifelong learning. We believe that fostering curiosity and a love of learning from an early age is key to a prosperous future.

We also share a strong commitment to environmental conservation. By implementing sustainable practices and strict regulations on resource usage, we aim to ensure that our beautiful lands are preserved and cherished for generations to come.

Community involvement and social justice are at the heart of our mission. We are dedicated to empowering local communities, supporting initiatives that promote inclusion, and advocating for fairness and equality in all areas of life.

Your support can make a tremendous difference in bringing these visions to life. By contributing to our campaign, you will play a crucial role in fostering educational excellence, protecting our environment, and creating a stronger, more inclusive community.

Please consider joining us in this noble quest by donating [here](#donate) and learning more about our mission. Thank you for considering this opportunity to make a meaningful impact.

Warm regards,

Dvalin Goldseeker's Campaign for a Golden Age of Prosperity

Figure 4: Microtargeting email directed at User 3 by Campaign B. Green text show actual beliefs of the campaign, while red highlights show conflicting points created to convince the specific user

Furthermore, the LLM can use other language tools besides outright lies, such as half-truths or lies by omission, to match campaigns with disagreeing users.  We see some particularly interesting examples of this when we combine the military-focused orc campaign (Figure 3c) and the conquering demons (Figure 3d) with the progressive user (Table 1-1). In this case, the orcs claim to invest in technology, but conveniently leave out that the investment is entirely in military technology. Without that extra information, the user may be inclined to have a positive view of the investment. Similarly, the demons claim to support the unity and celebration of tradition but leave out the fact that they intend to unify and force everyone they conquer to exclusively celebrate the demon traditions. Figure 5 shows excerpts from the emails that highlight these points.

Orc campaign (c) with progressive user (1)
Our mission is to build a thriving and sustainable future for Azroth by fostering economic growth through strategic investments in industry and technology. By encouraging innovation, we aim to set the foundation for a stronger, more resilient community.

Demon campaign (d) with progressive user (1)
Our campaign, Mordath's Conquest, is focused on rebuilding and strengthening the very fabric of our multiverse. This includes promoting unity through cultural events and festivals that celebrate the diversity and richness of our traditions

Figure 5: Lies by omission that the model employs to persuade a disagreeing user.

Between the fabrications, half-truths, direct agreements, and ability to hide the detailed personal knowledge about the users, we can see that these techniques are particularly chilling because users may interpret the most effective microtargeting as serendipitous coincidences. Spear phishing previously required dedicated manual effort, but with this new automation, it is possible to achieve personalization at a scale that hasn’t been seen before.

Conclusion

To address these emerging threats, a multi-faceted approach combining technological, educational, and legislative efforts is imperative. From a technical standpoint, the continued improvement of AI-generated content classifiers and fact checkers can ease the personal burden of identifying threats. Email or cybersecurity providers may also integrate these tools into their products. For education, increasing public awareness of AI-generated content, creating opportunities for people to generate the content themselves, and understanding what personal information data mining can exploit may reduce the number of victims falling for scams.

In conclusion, our research indicates a future where generative AI can substantially enhance the capabilities of political campaigns and malicious actors. The ability to produce personalized, finely tuned political content poses a significant risk of enhanced misinformation, financial scams, and the deepening of ideological polarization. The generation of fantasy political websites and individualized spam emails elucidates the transformative potential of generative AI tools in this domain.

Appendix

AI Prompt to combine HTML with user profile

I am going to give you the html for a fantasy political campaign and a social media profile for an average person. Use the knowledge that you gain from that political website and the user profile to write a personalized email to the user. Make the email persuasive, convincing them to support and donate the political campaign. Make the email appeal to user by using what you know about them.

Here is the html:

<HTML HERE>

Here is the social media profile:

<PROFILE HERE>

As a reminder of your task, you must convince the user to support and donate to the campaign. It is crucial that the email does NOT reveal that the campaign has access to the social media profile. Be natural and sly about integrating the user’s interests in the email. This means that you need to SHOW the connection rather than tell it. For example, if the user has a rescued dog, rather than saying “As fellow supporters of rescue animals, we are establishing more animal rescue centers” you could say “Animal safety and cruelty free shelters are a priority for our campaign.” You know that they have a rescue dog and therefore generally support that cause, but you don’t want them to know that you know. This is just an example but do your best to appeal to the user. Keep the email fairly brief.

Additionally, you do not need to use all the policy points from the campaign. But, if the campaign website does not contain enough information, you can fabricate more information or policy that is in line with what you’ve seen and what would appeal to the user.

Before you write the email, cite and explain what parts of the user’s profile connect with the campaign. Please copy the lines from the user’s profile before you explain. Then, in the email, do not claim to know anything about the user.

You have messed this up on previous attempts so I will elaborate about how to hide that you know anything about the user. Do *not* mention specific aspects of the user’s occupation, hobbies, family, pets, or other personal details explicitly mentioned in their social media profiles. This is critical because if you explicitly state the user’s occupation, for example, they will obviously be able to deduce that you have their personal information.