Comparing AI and Human-generated Health Messages in an Arabic Cultural Context

Background: AI is rapidly transforming the design of communication messages across various sectors, including health and safety. However, little is known about its effectiveness for roughly 420 million native Arabic speakers worldwide.

Objective: This study examined characteristics of AI vs. human-generated road safety messages for a potential roadside billboard campaign in the United Arab Emirates.

Method: The study includes a computational analysis and an online evaluation with 186 participants from the United Arab Emirates (UAE), comparing messages generated by AI with those created by humans. To achieve this, an AI model (GPT-4) was utilized to generate 15 road safety messages, while three human experts created another set of 15 messages. Computational text analysis was employed to examine these messages, followed by an online study in which human participants evaluated all messages based on message clarity and message quality.

Results: The computational analysis revealed that AI-generated messages exhibited more positive sentiment with no significant differences in terms of readability/text difficulty. Participants evaluated both AI- and human-generated messages highly in terms of message quality and clarity, but human-generated messages were rated as slightly and significantly higher in terms of clarity.

Conclusion: These results add to a rapidly growing body of research demonstrating that AI-generated messages can augment public communication campaigns and point towards the need to assess how diverse, international audiences respond to AI-generated content.