Our project team received an NSF RAPID/Collaborative grant in May 2020. We refer to the project as Human-AI Teaming, or HAT for short. The focus of this project is for Big Data Analytics to Enhance Response to the COVID-19 Pandemic. It studies the work between humans and AI machines done in times of crisis. 

To understand the project you first need to know how social media is being used to help emergency responders. Social media data can provide important clues and local knowledge that can help emergency responders better comprehend and capture the evolving nature of many disasters. Yet humans alone cannot grasp the vast data generated by social media, so computers are used to assist the process. Currently, volunteers look through social media, such as Twitter, and identify posts with information that could provide emergency responders with relevant information. These volunteers label the tweet using computers that have AI working behind the scenes. These volunteers are known as the Community Emergency Response Team, aka (CERT) volunteers. CERT volunteer work is very crucial for categorizing information shared online during a crisis. For example, they would tag a tweet about part of a home being blown away during a storm. Later they may find a tweet regarding where it was blown away to and they can share the information with the emergency managers. While partnering humans and machines during crises is very useful, there is very little currently known about how to leverage the skills of humans and machines when they work together (human-machine teaming) to identify meaningful information in social media data.

The goal of this research is to better understand the decisions that human annotators make, and how those contribute to the learning of Artificial Intelligence (AI) models. Human decision-making capabilities are limited; yet, humans still have a unique ability to understand the contextual references that machines cannot recognize. For example, machines would not understand sarcasm. This example demonstrates one of the many values that humans offer in a human-AI teaming context. 

The HAT project advances the field by revealing the unique abilities that both humans and machines bring when working together to comprehend social media patterns during an evolving disaster. The HAT project is designed to 1) better understand the process of real-time decisions that volunteers make when they convert social media data into structured codes the machines (Artificial Intelligence algorithms) can understand, and 2) use this knowledge to improve human-machine teaming. 

This research is focused on achieving these goals through studying how and why CERT volunteers label tweets regarding Covid 19. Volunteers are currently going through tweets, determining if they are relevant, then categorizing them as “risky behavior” or “preventative behavior”. For example, if they see a tweet about a large gathering of people with no masks or socializing in crowds, the tweet would be tagged as “risky behavior”. 

Our team has already conducted over 40 interviews with 15 different CERT volunteers, to better understand the thinking and reasoning of why they labeled tweets the way they did. With this new found understanding of human reasoning (called mental models), we will make  improvements to the AI machines working on the Big Data of Social Media. Thus bettering the response and knowledge of Covid-19 information as it develops in real-time. 

This exciting work is the first of its kind to look into the human side of the human-AI teaming  used during crises. In fact, the team has already written a paper and submitted it to be considered for publication in conference proceedings later this year. The faculty and community partner team members are Keri K. Stephens, Moody College of Communication at The University of Texas at Austin, Amanda L. Hughes, Information Technology & Cybersecurity, at Brigham Young University, Hemant Purohit, Information Science & Technology at George Mason University, and Steve Peterson, Volunteer Coordinator at Montgomery County Community Emergency Response Team.  We also have a host of students working on the project including [add everyone here and include yourself]

Stay up to date on this research by going to the NSF HAT Project tab on our website and stay tuned to hear more about the graduate students’ interesting roles in this project in a blog post to come!