Interviewer: Could you please introduce yourself?
Ali Sarvghad: My name is Ali Sarvghad and I'm a research assistant professor here at UMass. I've been here for close to 5 years, and I work in the area that we call human-computer interaction. More specifically, what I do is data visualization, which is how we transfer data to different kinds of interactive visuals that then enable people to derive insights and new knowledge, and, in some cases, are able to make decisions.
Data visualization is a big field of research, and people work on different problems. So what I refer to as data visualization an interactivate map would be an example. It is usually a product of a complex process that involves collecting data, then cleaning, wrangling, transferring whatever they need to do on the data to prepare it for visualization. Then it goes through different kinds of modeling for visualization. Then you have visuals and people will use those visualizations depending on who those people are for different reasons. Personally, I'm very interested in the human component. Like how people perceive graphical information, and how their perception works. For example, perceptual thresholds, if I show you some graphics how fast we can detect something. So that is one kind of research related which we call graphical perception. And also I work on making visualizations. Specifically, making visualizations more available and accessible to people beyond experts. What I do is typically for experts. How do we, for example, support collaborative visual analytics on large interactive surfaces. But then I also do work that focuses on people specifically. And when I say people, I mean non-experts, not just experts on different types of problems. And I guess that is the part of my work that most overlaps with PIT is making this available to a wider range of people.
Interviewer: Thinking about the intersection of your interests and PIT, when you think about public interest technology, what are the kinds of things or motivations in your own work that come to mind?
Ali Sarvghad: So I'm reflecting on what I do to answer that question. I try to lower the access to technology itself and make it more available to a wider range of audiences. That is one of the ways that I believe that what I do intersects with this idea of PIT and making technology. And that's a very human-centered design process that then you have to engage with them, to try to understand their requirements, the problems, what the issues are, the challenges, and then learn from that to transfer that into the design of the technology. So what I do again, is data visualization, enabling people to use visualizations to make data-driven decisions.
This specific project, interestingly enough, it's not about data visualization yet. So the PIT project that we are still working on is about Parkinson's patients. And the reason that I actually told you a little bit about the pipeline is that in order to produce visualizations and analytics on top of it that fits people, then you need to know the people and their requirements. Also you need to make sure that you collect data that is representative, inclusive, and that is not noisy and such. So the quality of the data, the amount of the data, the kind of the data that you collect matters.
Now, how that relates to this project. Two years ago, we did a different project that involved Parkinson's patients and their caregivers. And these were like interview studies. And one of the things that we learned is that they struggle with basically keeping track of their health information, activity, and such in between their doctors visits. So that sparked this idea of, how can we help them to collect this data and then use that data to help their caregivers and healthcare providers to be more efficient and effective, and all that. Then we started thinking that, usually, Parkinson’s tends to happen in people that are older adults. And they may have physical challenges like tremors that they experience sometimes, that can make it difficult for them to sit in front of the computer to type. So we looked into how people do what they call journaling. One is pen and paper, but it’s not really effective, it has its own pros and cons. Then there are digital journaling tools that again usually have to use a physical keyboard to type. And also, they are static in the sense that all it does is just receive and record what you say. So, considering all of that, we're thinking about how we can use these recent advancements that we see in large language and vision models. What do these people need and what data needs to be collected to put something together that helps them to facilitate this process of journaling, and also take it to the next level and tell you how?
So this project is, as I said, Parkinson's patients we call it a voice based or voice enabled conversation, generally meaning that they talk to a microphone. Patients can talk to it like, “I want to start journaling,” and the system captures what they say. But one of the very interesting and novel aspects of what we designed is that actually it is capable of probing, meaning that depending on what you say, it can ask medically relevant questions. A really simple example: if you say, “my tremors were worse this morning,” then it's going to ask you about your medication intake. We have a collaborator who is a Parkinson's specialist and we designed this set of follow-up questions and probes based on identifying what is the issue that the patient was reporting at the moment, and then, based on that, we ask a number of follow up questions. So why do we do that? The idea is that more basic voice recordings are static and we call this a dynamic model that makes these probes to collect more information. So when we put this all together, and you provide this to the healthcare provider, they will have a much richer understanding of what these people experience. What are their issues? What are their challenges?
Our question was basically, what is missing at the moment? This is missing. And we wanted to basically lower the barrier to access. Make it easier. Because talking to the system using natural conversation is more intuitive, and it happens more organically rather than when you want to type something. So it makes it easier, it makes it more accessible, and also the part that is capable of asking follow up questions benefits the patients. Also whoever is involved caring for them has more data to work on. So this is the first component of this project. And this is kind of an ambitious project. So when we are done with this, we're going to move on to building the analytics on top of this data collection. And we are thinking of basically building different visual analytics—that's the term that we use, both for patients, so they can look and see how things are changing, and for the healthcare providers. That would be like the entire ecosystem of this project. So in short this project is basically utilizing this AI machine learning to make the tool smarter, more efficient, and more effective at collecting information from people, which is much easier to use than typing out.
Interviewer: It sounds like it also is just a significantly more humane or human way to go about this kind of interaction as well.
Ali Sarvghad: Absolutely. Designing the conversation is very important, especially for patients, or a group that is considered vulnerable. We also work on personalization of the system. So when person A talks to the system, it's not going to be exactly the same as when person B is. When talking to the system, the follow up questions that we're going to ask for the medical part are pretty much the same, but the delivery of the conversation will not be. We use a name to refer to them, and based on their medical profile and the history of the conversation with this tool, in the current journaling session we personalize different aspects. So it makes it more relatable and more of this feeling that, “oh, this system is talking to me.” And there is research that shows that this opens up a lot of positives in the user experience, and that they even provide more information, when they feel that kind of connection with the system.
Interviewer: Sure, I think that makes a lot of sense, even from the perspective of if you've gone to the doctor's office with a specialist you've never met before, and it seems like they are a little tense that day. You aren't gonna be as open about your issue as you are with your personal doctor. So I have to ask one question, given that this is, of course, a PIT related project. You've mentioned, of course, the use of these machine learning models in the vein of personalization. I'm curious if you can hint at the kinds of steps, if any, are being taken to mitigate the kinds of biases and things like that that we know are a part of the training of these large language models. If we pull them off the shelf.
Ali Sarvghad: Okay, good question. So the short answer is nothing right now. Because, the research questions we mostly focus on are basically on the utility and feasibility of this approach. That is very important. And we didn't look into biases, but we worked on, of course, security and privacy, so that all the data that we collect are anonymized and they are all kept on HIPAA compliant Amazon servers. So security and privacy, of course, are very important. But the research questions that we're asking at this point are like, “is this idea gonna work”, and “to what degree it's gonna work”. We've tried training our models based on some data that we had within the user study. And well it actually failed and we learned many things about the computational side and the human side of this system. When we did the pilots with patients, and we built the tool with the prototype and tested it, I had them do this journaling on a daily basis, more than once a day for 2 weeks. And we collected that data and we noticed that many things didn't work in terms of what we called intent identification. Which is things like, are you talking about tremors? Are you talking about pain? Because people use a really wide variety of ways to talk about their experiences. So that's a hard problem by itself. But then we learned something that we didn't expect in terms of their reporting behavior. We thought as users interact with this sort of system, they would have conversations as essentially single sentences back and forth. However, people suddenly started to give these blobs, like whole paragraphs. And that made actually understanding the intent of what they just said much more difficult. Both for understanding “What's the issue?”, and how to follow up, what would be the proper question to ask? And these are the things that we were focusing on and learning as we are doing this.
Ok, so why am I telling you this and how does it go back to what you asked me. We need a huge, huge, huge amount of data to train something that's gonna actually work properly based on what I just told you. So we switched back to GPT-4. And now in the new version, here, we're just using GPT. 4. But behavior might be biased or not work, as you know, in terms of if it gets everything correct and all that. But, again, that's something that we are pushing a little further back. If we use a model that is large enough, it can handle basically this complex behavior that people show. And then we'll get to when we want to deploy this perhaps in the future we would look into if this one are the biases that could basically happen in terms of I don't know, off the top of my head, I cannot think of what sort of biases could happen in terms of system behavior and the data that is trained on but that's something that that we should definitely look into, that makes sense.
Interviewer: I can think of. Well, one, I think an important part that you mentioned earlier on that made me very happy is you have the kind of medical side of it scripted out already. That makes a lot of sense. The last thing you would want is ChatGPT to hallucinate some medically unmeaningful data.
Ali Sarvghad: Oh yes, you are actually bringing up a very good point. In the pilots we noticed that sometimes it (GPT-3) started giving medical advice during the patient's conversation with the system. We didn’t use GPT primarily, but it worked as the fallback for when our model that we trained couldn't identify the user’s intent. But, because of these results we built these sort of basically precautions into the system. That the system should not give any medical advice or anything like that. And it's just asking, trying to understand what people are talking about, and then asking the questions from a list of questions that we know are the right questions to ask. And that is how it basically is supposed to operate.
Interviewer: And then another thing I can think of is, of course, given it's an inherently few-shot data driven paradigm for this kind of personalization component, I imagine that there's some pre-training involved for that. The kind of diversity within that sample size is going to matter a lot from a PIT standpoint, and it would be very valuable to think through very carefully with respect to the types of demographics that are interacting with the system. Because, you know, for example, people have different vernaculars. So it'd be very interesting to see what happens if a vernacular that wasn't a part of the training set were then introduced to the system, and how it would respond
Ali Sarvghad: Absolutely, absolutely, yes, these are very important questions. And we definitely will work on basically these issues when we are past this preliminary phase of feasibility. And you know, usability and utility testing is over. That makes sense.
Interviewer: You've talked about this desire to lower the access barrier to technology. And so I'm wondering if any of your past experiences or your background has been a motivating force for you in this work?
Ali Sarvghad: So I'm really interested in the human component of visualization research. Over the 10 years I've been doing this for, I've started to learn that there is a focus on specific problems and on a specific set of users, like the experts that I’ve mentioned. So in my realm, you see a lot of research targeting specific groups with specific skill sets and abilities, and while you can see that there are opportunities for people outside of this specific set of experts, not many people are looking into these problems, and that's when I started to notice that there are problems that we are ignoring. That is how I started thinking about and getting more and more interested in expanding the reach and accessibility of data visualization and analytics in order to benefit other groups. Essentially making what I call personal, responsible data visualization, which is data visualization that tries to lower the barrier to accessibility to expand the reach of data visualization to a wider set of audiences.
So when thinking about accessibility and inclusivity, I consider how we can take something that already works perfectly well but for this very specific group, and make it more accessible. One of the projects that we did a few years ago was about how we can increase the inclusivity of data collected from the public. So we modified iClickers and we took them to the town hall meetings here in Amherst. Many people are usually not comfortable in these meetings. They go to these town hall meetings, take the time to be there, but most of the time they are silent for many different reasons—they don't like confrontation, they are shy, or whatever. So we gave these iClickers to the people, and they were labeled, and people could use them as the discussions were going on. And we found out that when we did that, actually, a lot of these people are starting to provide feedback. And we collected a lot more data from the audience, from the participants of town hall and public engagement meetings in a physical public engagement venue. And this shows that we can take a technology that already exists and sort of repurpose it and use it for inclusive data collection. So inclusivity is very important in thinking about how we can make this more accessible, and how we can make these tools fit the specific requirements of this group that we have not necessarily focused on, such as older adults and maybe even kids at school.
Interviewer: Were consideration of inclusivity and accessibility part of your education and training?
Ali Sarvghad: No, not not my field. Maybe it's because it is a relatively new field. In recent years there is more and more work that talks about the ethical considerations of data visualization, visual analytics, data collection, etc. So I guess it's still kind of emerging. When I was doing my graduate studies, there was not much discussion and awareness of this, as this was not really mainstream, but it's definitely gaining attention and is definitely growing more and more important for sure.
Interviewer: If other faculty members are interested in trying to incorporate these kinds of PIT-related principles into their work, how and where should they start? How did you start to get involved with PIT as an organization?
Ali Sarvghad: To be honest with you, I received an email from PIT@UMass that said, “we are taking proposals and if you have anything that you think that fits our goals then share it with us,” and I did and they liked it. So that was how I actually came across PIT. Well, that's a model that's still useful and valuable, that basically spreads information about what's going on, like this exhibit. And the more that people know about this as well, they would look into PIT itself, but also the idea of PIT and advocate for what it is. I talked about inclusivity and accessibility and I'm sure if you talk to other people they will talk about other facets. I'm sure other CICS faculty members are doing projects that would fit the definition of PIT, but maybe they're not aware of it. They don't yet think in PIT terms about the project.