Patient AI

Team Name

Deep End

Timeline

Fall 2023 – Spring 2024

Students

• Sammy Mahmoud
• Phu Nguyen
• Hananeel Pankaj
• Hanani Pankaj
• Abdur Rehman

Sponsor

Shawn Gieser | Senior Lecturer at UT Arlington

Abstract

This project is a nurse-patient simulation in which the user will assume the role of a nurse and the language model a patient. The nurse will interact with the patient by talking to it, and the patient will respond accordingly. This project is focused primarily on improving the communication skills of the nurse. The project has three main components: speech-to-text, a language model, and text-to-speech. The user’s voice will be recorded upon speaking into the microphone. The recorded voice will be converted to text, and this text will be passed to the language model. The model will return a response, and this response will be converted to speech for the user to hear. Upon hearing the model’s response, the user will respond to the patient, creating a loop. This, in effect, will simulate a conversation between the nurse and patient.

Background

One of the ways nursing students are taught to communicate with patients is through standardized patient (paid actors) encounters. While this allows the nursing student to interact with a “patient”, it does not provide the atmosphere of reality: the standardized patients follow a script. Moreover, these encounters do not provide nursing students with the necessary skills, specifically regarding communication. When dealing with critically ill patients, nursing students lack the necessary communication skills to interact with such patients¹. Additionally, patients often overlook the importance of nurses². It is assumed that nursing students are not taught how to deal with such patients, as they were trained in an environment where the patient is cooperative.

Project Requirements

1. The AI should be coherent and consistent.
2. The user should be able to talk and then hear a response from the application.
3. The speech-to-text component should be able to recognize English words when spoken with an accent.
4. The AI Patient should be quick enough to respond to mimic an adequately normal conversation.
5. The AI Patient should not respond with traumatizing responses, i.e., responses that are hurtful (while this simulation is be a facsimile of the real world, the goal is not to help nurses handle insults from patients).
6. The GUI should display a patient to give the semblance of talking with an actual patient.
7. The AI patient should not respond saying that it is an artificial intelligence or a language model. Instead, the responses should be limited to what an actual patient would say.
8. The GUI should be robust enough to make the application easy to use.
9. The project can only run on a device with a GPU, as it would speed up the creation of the patient video.
10. There should be a variety of patient templates to choose from, such that each iteration of the program should display a different patient. In addition to appearances, the voices too should vary.

System Overview

The Patient AI project consists of four layers. The necessary inputs are passed in from either the user to a layer or from layer to layer.

1. The Program Setup Layer imports all the necessary imports and start the application in a web browser.
2. The STT (Speech-to-Text) Layer gets the audio from the user and converts it to text. This layer outputs the converted text.
3. The AI Layer gets the output of STT Layer and passes it into the language model. The language model, acting as the patient, will output its respective response. This layer will output that response.
4. The Output Layer gets the output of the AI Layer. It then converts the patient’s response into audio and merges it with a predefined video of the patient. This new video will be displayed to the screen.

Results

Future Work

1. The refresh button, when clicked, clears the visual components on the screen but does not clear the conversation. So, the conversation prior to clicking the refresh button will still be remembered. Thus, the refresh button should also restart the conversation.
2. The GUI has a delete message function, where the user can delete messages individually. However, this only removes the message visually: it does not remove that message from the language model’s history. In the future, the delete message function should apply to the language model’s history as well. This feature is added in case the user misspoke and/or the program ‘hears’ the wrong thing.
3. In the future, the GUI will have a button to terminate the program. That way, the user will not have to close the browser and do ctrl+c on the terminal to completely end the program.
4. As of right now, there are two patients: male and female. In the future, we hope to incorporate patients of different races and, perhaps, body type as well (e.g., obese, skinny, average).
5. The GUI only uses one colour palette. In the future, we want to incorporate a variety of colour palettes. This can make the interface more user-friendly: some may prefer light colours; others may prefer dark colours; some colours can be hard to see for certain people.
6. The GUI only uses one font size. In the future, we want to add the ability to change the font size to make the text more viewable for our audience.
7. Although the patient / AI model does not output profanity, the nurse can design the prompt to get the patient to do so. While profanity is used by patients, this suggests that the nurse can cleverly design a prompt to get the patient to do something inappropriate, like using profanity. Thus, in the future, we want to add some feature in the back-end that processes the nurse’s prompts to determine if it is what a proper nurse would say.

Project Files

Project Charter (link)

System Requirements Specification (link)

Architectural Design Specification (link)

Detailed Design Specification (link)

Poster (link)

References

1. Handan Eren. Nursing students’ difficulties in determining the care needs of patients on clinical practice: A qualitative descriptive study. International Journal of Caring Sciences, 14(2):1291â1299.
2. Judit Sánchez Expósito, et al. Ensuring relational competency in critical care: Importance of nursing students’ communication skills. Intensive and Critical Care Nursing, 2018.