HIV prevention messages selected using machine learning from existing social media content were found to be appealing to gay and bisexual men and public health departments in the United States, according to a recent study in the journal PNAS Nexus.
This research, conducted by Dr. Man-pui Sally Chan at the University of Pennsylvania and colleagues, indicates that automated health promotion campaigns requiring fewer resources could become part of HIV prevention going forward.
Background
Traditional health promotion campaigns are typically designed and disseminated by public health departments. These campaigns tend to be expensive, messages may be selected in an unsystematic way, and their effectiveness is rarely evaluated.
Social media platforms provide a wealth of health promotion information generated by both expert and non-expert users. These posts are accessed frequently and reflect current views on HIV prevention – much of which may directly influence the behaviour of users to get tested for HIV or start taking PrEP, for instance.
However, for messages to influence behaviour change, they need to provide clear information that helps with setting a behavioural intention, ultimately leading to a specific health action.
Researchers used machine learning to classify existing social media messages and develop what they term ‘living health promotion campaigns.’ These messages contained the necessary elements required for behaviour change – they would encourage action for the viewer. The researchers then evaluated the effectiveness of these messages using a key population experiment with gay and bisexual men, and a field experiment with public health departments.
“Social media platforms offer a living repository of messages that are generated by the community every day,” the researchers say. “This repository allowed us to produce the first automated, adaptive system to automatically gather and recommend HIV prevention and testing messages for counties in the United States.”
The study
The study was carried out between 2022 and 2023, with the first phase involving computational processes to automate the gathering and selection of actionable HIV prevention messages. A message was considered actionable if it encouraged a user to take an action, such as getting tested for HIV; if it provided relevant information, such as where to find an HIV testing centre; or if it indicated that other people were acting in a way to prevent HIV.
To achieve this, researchers used large training and validation datasets of Twitter posts on HIV prevention from 2010-2018, based on keyword searches using hashtags and from expert accounts, such as those of researchers, government and HIV organisations. Researchers coded posts for actionability. From a training set of 13,314 messages, only 607 were found to be actionable – highlighting the potential benefit of automation for speeding up message selection.
This is where machine learning came in: the training set was used to develop a classification model that could identify actionable vs. non-actionable messages. Messages were also filtered based on factors such as HIV prevention, relevance to gay and bisexual men, and US-specific content. A bi-directional long-short-term memory network (bi-LSTM) model performed best at selecting actionable messages.
After ranking the top 200 messages based on actionability and engagement metrics – such as like and repost data – researchers vetted the messages to ensure they were consistent with the US Centers for Disease Control and Prevention (CDC) recommendations. For instance, a message promoting abstinence only to prevent HIV transmission would not be selected, despite being actionable.
In the second phase, researchers tested three sets of 12 messages with 260 gay and bisexual men in an online experiment. Of these men, 63% were White, 17% Black. This was a well-educated sample, with over 85% having college education. The first two sets of messages were selected using HIV keyword hashtag searches and from expert accounts. Researchers used the bi-LSTM classifier to determine how actionable each message was. Messages were then ranked in terms of both actionability and engagement and any with explicit content, profanity and specific event details were removed. Of these, 12 were vetted by a researcher to ensure that they were CDC-compliant, while another 12 actionable messages were not vetted. The third set was a control set of messages, selected from the HIV hashtag keyword search, but not vetted, classified or ranked for actionability.
Men were asked their perceptions of how actionable messages were generally, how accurate and relevant the messages were for them, with an item included to measure how likely it was that the message would result in behaviour change. Researchers also asked how likely the men were to share the message if they saw it on social media.
The final phase of the study included a field experiment with 19 public health departments and community-based organisations across 42 counties in the US. After researchers ranked posts in terms of actionability and engagement, they used the vetting process described above to select the experimental messages. Control posts were selected randomly from remaining ranked posts. Over eight weeks, researchers sent out daily message recommendations to each agency; they were asked to post between five and ten unique messages from a pool of approximately 183 messages a week but could choose what they wanted to post. In total, the researchers recommended 10,151 messages, including 5,154 experimental messages and 4,997 control messages to investigate which messages agencies would post more frequently and engagement metrics. Additionally, agencies were sent pre- and post-study questionnaires and offered social media support, such as a social media assistant and social media management software, to assist with message selection and posting. Agencies and assistants had no knowledge of the study hypotheses, and which were experimental versus control messages.
Results
Among gay and bisexual men, the experimental messages selected by the classifier were perceived to be significantly more actionable, appropriate for gay and bisexual men, accurate, personally relevant and effective than the control messages. The men also indicated that they would be more likely to share the experimental messages. Those chosen by the classifier did better, regardless of whether messages were vetted by a human or not. However, vetted messages performed better than those that were not vetted.
For instance, when asked about how effective they perceived messages to be (“This message has the potential to change my behaviour”), men gave a higher average rating on a seven-point scale to the actionable vetted messages (4.74), than either actionable not vetted messages (3.68) or the control messages (3.63).
With the field experiment, researchers found that the experimental messages were selected by agencies much more frequently: in total, 273 times versus only 39 times for the control messages, or seven times more likely. Thus, collectively, the experimental posts also received more replies, reshares, and likes. However, there was no statistical difference in average engagement per post between the experimental and control messages.
On average, agencies reported a statistically significant difference in posting more messages on HIV prevention and testing after the experiment, compared to baseline. While other differences, such as identifying and creating relevant messages to post, were reported, these were not statistically significant when comparing baseline to the end of the experiment.
Conclusion
This approach provides a novel avenue for the selection of actionable community-based health promotion messages from existing sources and can be easily used by public health departments who often do not have the adequate resources required to mount expensive HIV prevention campaigns. However, this approach may require further refinement to make it more contextually specific and adaptive – for instance, offering specific content from users in the Southern US to health departments in that region. This would ensure that local nuances are integrated into the campaigns.
“The proposed system can inform policymakers and government and nongovernment agency officials who seek to promote health on social media,” the researchers concluded.
