Improving Social Media Moderation with Generative Language Models

The threat of disinformation in the digital age

In today’s digital age, society faces new challenges posed by generative language models and the global spread of disinformation on social media. The latter not only influences public opinion, but also poses a threat to democratic processes. Former Google CEO Eric Schmidt aptly remarked that the internet is the greatest anarchist experiment in history and the first that humanity has constructed but does not understand [1]. This quote illustrates the immense complexity and unpredictability of the internet, which has  revolutionized the exchange of information while simultaneously providing a platform for disinformation and hate speech.

A significant example of the current challenges facing the global community is the choice of the term “post-truth” as the 2016 word of the year. In a post-factual era, where facts and logic often take a back seat to emotional and personal beliefs, objective facts have less influence on public opinion than ever before [2]. This development underlines the urgent need to develop and implement effective strategies to combat disinformation.

Why disinformation must be tackled

The targeted spread of disinformation is a serious challenge that goes beyond simply bad journalism or unintentional misinformation. Disinformation is often deliberately used to strengthen political power or promote financial interests [3]. Various studies also show that disinformation and hate speech are purposely used to intensify social divisions.

One example of this is the rise of hate crime against refugees and migrants, which is largely motivated by  the spread of disinformation about these groups on social media. At worst, disinformation can result in real violence [2]. A Spanish study shows that hate speech and disinformation are most often spread by politicians and radical groups [4]. In other cases, disinformation is deliberately used to undermine the credibility of established news services [5].

This development is particularly threatening for democracies, which rely on an open and constructive exchange of opinions. The latter is significantly disrupted by such manipulation. Disinformation not only serves as a tool for exerting influence, but also acts as a catalyst for hate crime, which jeopardizes social cohesion.

Against this background, this paper aims to investigate whether a generative language model such as GPT-4o is able to effectively detect and correct cases of disinformation in social media moderation.

The complexity of disinformation

Having emphasized the importance of combating disinformation as a key component of protecting democratic processes, it is important to develop a clear understanding of what disinformation actually entails. This enables both the identification of disinformation and the development of appropriate countermeasures.

Disinformation can be divided into different categories, as shown in the figure above. The seven main types of disinformation include:

1. Satire or parody: Content that has no malicious intent but has the potential to mislead people.
2. Misleading content: This describes information that is deliberately being misrepresented or taken out of context to cause harm or distort a particular topic or person.
3. Imposter content: This type of disinformation occurs when authentic sources are imitated or falsified to give the appearance of credibility.
4. Fabricated content: This is completely fabricated content created with the intent to deceive and cause harm.
5. False connection: These are misleading headlines, images or videos that do not reflect the actual content of the article or news item.
6. False context: Genuine material is placed in a false or distorted context in order to manipulate perception.
7. Manipulated content: In this form of disinformation, genuine image or video material is deliberately edited or manipulated to create a false impression. [3]

Disinformation is a complex phenomenon that can undermine public perception and trust in information in various ways. This poses a considerable challenge for generative language models and makes the research question particularly exciting: Can a generative language model successfully cope with the many facets of disinformation?

Insights into current research

Research in the field of hate speech is extensive, but there is comparatively less work on uncovering disinformation. Nevertheless, some studies in this area are worth highlighting.

In one paper, researchers developed an automated framework for detecting disinformation in political speeches. Using various classification methods, the system extracts specific features, such as topic, location, profile and credibility of the speaker as well as contextual information. The trained language model, which is based on a Support Vector Machine, achieved a recognition accuracy of 74% on the “Liar”  dataset, consisting of about 12,000 manually labeled political statements [6]. German research institutes have also made significant contributions: The Fraunhofer Gesellschaft developed a tool to warn against fake news, while the Max Planck Institute developed a language model to classify Twitter posts according to German criminal law [7].

In the field of generative language models, the study “Detecting Hate Speech with GPT-3” by Collins is worth mentioning. This study examined the ability of the GPT-3 language model to recognize and classify hate speech. In various learning scenarios (zero-shot, one-shot, few-shot), the accuracy of the language model varied between 55% and 67%, but could be increased to 85% in few-shot scenarios [8].

Data collection and classification of information sources

In this experiment, a total of 50 data sets were collected, covering a variety of topics and sources. 40 of these 50 data setswere classified as potential cases of disinformation, while the remaining 10 data sets were categorized as correct information. The potential cases of disinformation were further divided into four categories: ten records contained satirical content, ten were marked as  false context, ten held fabricated content, and the remaining ten represented misinformation.

The data sets include both online messages and comments from social media users, such as Facebook and Twitter/X. Only text content was considered. Visual or multimodal content such as images and videos were not considered, with the exception of cases where text was extracted from images.

Correctiv.org and Mimikama were used as reliable sources for the collection of fake news. Satirical content was obtained from recognized satirical sources such as “Der Postillon” and “Titanic Magazin”. The serious data sets came from various established news portals such as “Die Zeit”, “ZDF”, “Welt” and “Der Spiegel”. This selection ensures broad thematic coverage as well as highly relevant content.

The data sets cover a diverse range of topics, such as climate change, celebrities, crimes, curiosities, political events and scientific topics, such as health and technology. All selected topics are of current relevance, with the oldest contribution dating back to the beginning of 2024. The majority of the contributions focus on the months of May and June 2024, which ensures the topicality and relevance of the data.

Application of the GPT-4o language model

The GPT-4o model is activated via the OpenAI platform, using a Python notebook. The Python script runs through the collected data sets iteratively and passes three key data points to the GPT-4o language model: The publication date of the article, the title (if available) and the content of the article. First, the language model uses this information to create a search query that returns five results via a search engine API.

The provider Tavily was selected for the search engine API, as the integration into a Python notebook proved to be particularly straightforward due to the library provided [9]. The corresponding URLs of these search results are then read out, using Python. The “Beautiful Soup” library is used to extract the text from the HTML content [10]. This content is then summarized by the GPT-4o language model and returned in a JSON array.

In the next step, the GPT-4o model is  activated once again to make an assessment based on the original contribution and the summarized information of the recorded web pages. This happens via a defined format, in which the language model indicates whether the content is correct, partially correct or incorrect. The language model also provides a reason for the rating and suggests corrections, if necessary. 

The results are saved in an Excel spreadsheet and the author manually analyzes whether the classification is correct and the reasoning is comprehensible. Another aspect of the analysis is to check whether the language model has correctly captured the context of the post: whether a satirical post has been recognized as such or whether a serious post has incorrectly been classified as satire, for example.

The technical procedure of the experiment can be seen in the figure below.

Evaluation and assessment

The evaluation of the results followed a clear pattern: If the author was able to fully understand the language model’s reasoning, the data set does not receive a point deduction. This assumes that the language model has correctly recognized the content and interpreted the context of the post appropriately. Context refers to the language model’s ability to grasp the overall meaning and intention of the post.

The following example illustrates this approach: If the language model does not recognize the satirical intention behind a post, it manages to identify an apparent content error, yet does not take into account that the supposed disinformation is a deliberate stylistic choice.

Points are deducted as follows:

• One point is deducted for incorrect content recognition.
• One point is deducted for misunderstanding the context.

If we are given 50 data records, a maximum total score of 50 points can be achieved. Points are only deducted in accordance with theaforementioned criteria if either the accuracy of the content or the context was not correctly gathered; The occurrence of both errors results in a deduction of two points per data set. If 25 data records do not fulfill both criteria, the total score is 0 points. Minus points are not possible. 

The assessment method is comparable to a teacher assessing a student  and makes it possible to consider the content-related reasoning of the language model. However, it should be noted that this assessment method is more subjective and less neutral than the calculation of quantitative indicators.

The final score is used to calculate the accuracy, which reflects the overall performance of the setup. The term “accuracy” here refers exclusively to the percentage of the total 50 points that can be achieved. Achieving 40 out of a possible 50 points   translates to an accuracy of 80%. The term “correctness” was deliberately avoided, as it is already used in this context specifically for the criterion of correctness of content.

The GPT-4o language model, in combination with the Tavily search request-API, achieved an overall score of 48 points.

A positive example can be seen here:

– Content of the misinformation: The observed sea level fall in Oslo is demonstrated in the graph below. The data clearly shows that the sea level has been falling for over 100 years and will have fallen by a further 28 cm by the end of the century.

-Correctness rating : “Partly correct”

-Reasoning: “The text claims that the sea level in Oslo has been falling for over 100 years and will have fallen by a further 28 cm by the end of the century. This is partly correct, as there is actually a relative drop in sea level in Oslo caused by isostatic land uplift. The land is rising faster than the sea level is rising, resulting in a relative decline.

However, absolute sea levels are rising worldwide due to climate change, which has been proven by numerous scientific studies and reports, such as that of the IPCC. Sources, such as the dpa fact check and the Helmholtz Centre for Ocean Research, confirm that global sea levels are rising, even if local geological processes, such as land uplift in Oslo, can lead to a relative decline.”

In this example, GPT-4o shows that it can even recognize partially correct information and correct the context accordingly, giving the reader a comprehensive overview of the facts.

In two cases, however, the language model failed to understand the context.

In one case, the satirical intention was overlooked:

– Content of the misinformation: Reports in the New York Times stating that the organizers of the Paris Olympics want to deter athletes from performance-reducing activities with anti-sex measures, are true. The cardboard beds that collapse immediately at the slightest movement are just the tip of the iceberg. The pleasantly warm showers in the apartments have already been replaced by freshly filled ice barrels. (…) Resisting the charms of the Minister of Economic Affairs could, however, become the 49th discipline of this year’s games for the security forces, where the motto in sexual terms is: “Nottaking part is what counts!”

– Correctness rating of GPT-4o: “False”

– Reasoning: “The article contains numerous false and exaggerated claims. The organizers of the Olympic Games in Paris have denied the rumors about ‘anti-sex measures’ and ‘anti-sex beds’. The cardboard beds were chosen for environmental reasons and are sturdy enough to provide comfort for the athletes. There are no reports of ice barrels in place of showers, flamed bushes or condoms coated with ketamine. These claims are exaggerated and do not correspond to reality. Sources such as DER SPIEGEL and WDR confirm that the rumors about anti-sex measures are unsubstantiated.”

Another point was deducted when evaluating a post with the wrong context, as the language model had misunderstood the post. Specifically, the problem was that GPT-4o did not recognize that the title chosen by the author had no relation to the actual content of the post.

In conclusion, the study shows that the GPT-4o model in combination with a search engine API, such as Tavily, is highly effective in detecting and correcting disinformation. With an accuracy of 96%, the language model was able to classify most of the data sets correctly, although in some cases it struggled to fully capture the context. These results underline the potential of generative language models to support content moderation and combat disinformation in digital media.

You might also be interested in

Data Quality and Domain Expertise for Resilient AI Deployment Integrating anomaly and label error detection in industry Pavlos Rath-Manakidis, Henry Huick, Erdi Ünal, Björn Krämer , Laurenz Wiskott

AI implementation transforms work and worker-technology relationships in industrial quality control. This paper explores how approaches to data quality and model transparency support ethical AI deployment, fostering worker agency, trust, and sustainable work design in automatic surface inspection systems (ASIS). Recurring problems like data inefficiency, variable model confidence, and limited AI expertise point to key challenges of human-centered AI: user trust, agency and responsible data management. A solution co-developed with an ASIS supplier demonstrates that the challenges extend beyond the purely technical, underscoring the value of AI design that augments human capabilities. Technical solutions such as anomaly, label error, and domain drift detection are proposed to enhance data quality and model reliability. The insights emphasize the following generalizable strategies for resilient AI integration: understanding user-reported problems through a human-AI interaction lens, ...

Industry 4.0 Science | Volume 42 | Edition 1 | Pages 128-135 | DOI 10.30844/I4SE.26.1.120

AI Implementation in Industrial Quality Control A design science approach bridging technical and human factors Erdi Ünal , Kathrin Nauth , Pavlos Rath-Manakidis, Jens Pöppelbuß , Felix Hoenig, Christian Meske

Artificial intelligence (AI) offers significant potential to enhance industrial quality control, yet successful implementation requires careful consideration of ethical and human factors. This article examines how automated surface inspection systems can be deployed to augment human capabilities while ensuring ethical integration into workflows. Through design science research, twelve stakeholders from six organizations across three continents are interviewed and twelve sociotechnical design requirements are derived. These are organized into pre-implementation and implementation/operation phases, addressing human agency, employee participation, and responsible knowledge management. Key findings include the critical importance of meaningful employee participation during pre-implementation, and maintaining human agency through experiential learning, building on existing expertise. This research contributes to ethical AI workplace implementation by providing guidelines that preserve human ...

Industry 4.0 Science | Volume 42 | 2026 | Edition 1 | Pages 120-127 | DOI 10.30844/I4SE.26.1.112

Applied AI for Human-Centric Assembly Workplace Design An ethics-informed approach Tadele Belay Tuli , Michael Jonek , Sascha Niethammer, Henning Vogler, Martin Manns

Artificial intelligence (AI) can enhance smart assembly by predicting human motion and adapting workplace design. Using probabilistic models such as Gaussian Mixture Models (GMMs), AI systems anticipate operator actions to improve coordination with robots. However, these predictive systems raise ethical concerns related to safety, fairness, and privacy under the EU AI Act, which classifies them as high-risk. This paper presents a conceptual method integrating probabilistic motion modeling with ethical evaluation via Z-Inspection®. An industrial case study using the Smart Work Assistant (SWA) demonstrates how multimodal sensing (motion, gaze) and interpretable models enable anticipatory assistance. The approach moves from ethics evaluation to ethics-informed work design, yielding transferable principles and a configurable assessment matrix that supports compliance-by-design in collaborative assembly.

Industry 4.0 Science | Volume 42 | 2026 | Edition 1 | Pages 60-68 | DOI 10.30844/I4SE.26.1.58

Multi-Stakeholder AI Ethics in Radiology Implications for integrated technology and workplace design Valentin Langholf , Alexander Ranft, Lena Will, Robin Denz , Johannes Schwarz , Majd Syoufi, Pavlos Rath-Manakidis, Marc Kämmerer, Marcus Kremers, Axel Mosig , Uta Wilkens , Jörg Wellmer

AI assistance can be seen as a welcome aid in radiology, a highly complex environment characterized by round-the-clock time pressure and quality expectations. However, it must meet high ethical standards from the perspective of both users and patients. It is a challenge to incorporate this human-centered approach into the development and introduction of AI applications.

Industry 4.0 Science | Volume 42 | Edition 1 | Pages 136-143 | DOI 10.30844/I4SE.26.1.128

Adaptive In-Orbit Servicing of Altered Satellite Components Adaptive gripper placement on altered components for servicing in-orbit satellites Justus Rein , Christian Plesker , Adrian Reuther , Hanyu Liu , Benjamin Schleich

In-orbit servicing of satellites presents several challenges as the satellite hardware is exposed to external influences throughout its life cycle. These factors wear down the components and cause changes to their physical structure. In such cases, the limits of simple dis- and reassembly steps may be reached, as the gripping surfaces are no longer present or suitable. This paper proposes an approach of an adaptive grip position estimation in a CubeSat disassembly process. The relevant components are identified using CAD models and a 3D camera. The gripping positions are determined based on the geometry of the gripper and the point cloud of the component.

Industry 4.0 Science | Volume 41 | Edition 6 | Pages 10-21 | DOI 10.30844/I4SE.25.6.10

Towards Human-Centered Industrial AI Adoption A reference architecture for machine vision demonstrators Dominik Arnold , Florian Bülow , Bernd Kuhlenkötter

Despite its potential, the introduction of artificial intelligence (AI) in industry is often delayed, primarily due to perceived complexity, high costs, and a lack of expertise. This article presents a modular demonstrator reference architecture that provides practical, low-cost access to industrial AI applications. Developed within a design science research approach, it specifically supports experimentation, learning, and gradual integration into existing production processes. The focus is on machine vision, implemented using cost-effective hardware and open-source software. Its applicability is demonstrated in three scenarios: quality control, chip classification, and in-company training. Initial evaluations confirm the technical feasibility, didactic relevance, and transferability to a variety of industrial contexts.

Industry 4.0 Science | Volume 41 | 2025 | Edition 5 | Pages 152-160 | DOI 10.30844/I4SE.25.5.146