I have known Emma for over a month. She has contacted me four times and follows me after I post an article. Although she claims to live in Shoreditch each time, she seems to have multiple personalities. Her end goal appears to be to generate revenue from her OnlyFans page, which she advertises.
To draw in customers, she shares photos of herself in
revealing clothing. She is attractive and has a notably large backside, if
that's your preference. The source of these images is uncertain, but they are
likely generated with highly realistic synthetic faces, bodies, or bios created
by language models to mimic real users. These accounts are primarily involved
in coordinated influence campaigns, crypto scams, spam, and political boosting.
Research shows that such accounts often operate in groups and tend to have
fewer followers.
To understand how these accounts operate, particularly given
the advanced AI technology involved, visual and behavioural cues are crucial.
Emma’s fake accounts seldom post original or varied content. Other fake
profiles often act as reply-guys, spam affiliate links, promote schemes like
"get-rich-quick,' cryptocurrency scams, or use generic language similar to
ChatGPT. These profiles usually follow thousands of users but have very few
followers.
Although Emma the bot dismisses her work as trivial and
insists she's real, the rise of AI-generated fake identities and synthetic bot
networks poses more than just a technical challenge. It exposes a deeper
problem rooted in capitalism's social dynamics. This issue significantly
affects humans as aware, social beings and could be harmful. To fully
understand this, we must link it to capitalism's long history of using
technology to benefit the ruling class, as well as the wider social crisis
capitalism has induced in human awareness and community.
I identified Emm’s game early, but for others less aware, a
collapse of Shared Reality can threaten mental health. One of the most damaging
effects of AI-created fake profiles is what can be called an epistemic crisis—a
systematic breakdown in an individual's capacity to tell reality from
falsehood. This problem isn't novel under capitalism; historically, the ruling
class has sustained control via ideological mystification.
However, AI bots operating on an industrial scale mark a
significant advancement. When someone cannot be sure if their online
interlocutor is human or machine, if the consensus they see reflects genuine
public opinion or a synthetic effort, and if the emotional connection they feel
is with a real person or an algorithm the very basis of rational social discourse
begins to break down..
This has profoundly corrosive effects on individuals. The
natural human response to an environment saturated with deception and
manipulation is a generalized suspicion, not merely of bots but of everyone.
When you cannot reliably distinguish the genuine from the fake, you begin to
distrust all online interactions. This cynicism is, in many ways, a rational
adaptation to an irrational environment, but it carries an enormous
psychological cost. It deepens social atomisation, makes solidarity harder to
build, and breeds a pervasive sense of isolation and powerlessness. People
retreat from engagement, or are drawn into filter bubbles where algorithmic
amplification — often driven by bot networks — creates false communities built
around manufactured outrage.
In this already fragmented social landscape, the emergence
of AI-generated fake social environments often results in predictable harms.
Young people, still forming their social identities and seeking validation
through peer interactions, are particularly vulnerable. When online communities
are dominated by artificial personas created to provoke engagement, outrage, or
emotional reliance, authentic developmental progress is hindered. The
fundamental human ability to form genuine relationships—based on mutual
vulnerability, genuine uncertainty, and meaningful stakes—is jeopardised when
these interactions occur mainly with machines rather than real individuals.
