Relatio

CSC 212 Project Proposal

Jillian Penfield, Hannah Simons, Gabby Stillman, Ellie Yu

Problem Statement

Everyday, about 23 billion text messages are sent worldwide. People use texting for all sorts of relationships, including personal, professional, and groups. However, there is no tool to measure the quality and health of those relationships over time. Relatio aims to tackle this problem by creating a Messenger plug-in that would analyze and provide feedback about the health of the relationships. Metrics such as quantity, tone, capital letter usage and number of swear words could help indicate what could be improved about the relationships’ communication style, as well as flag any signs of digital abuse. Digital abuse is any attempt for a person to demean, control or bully another through text, email and direct messages. This is a serious problem, which is often related to poor self-esteem and unhealthy relationships.

In this project, Relatio aims to improve relationships by providing easy to understand statistics over time which charts the digital communication in relationships and ways to get support, as shown in Figure 1. Since text messages are used to augment the communication that occurs in person, they can help identify patterns, habits and attitudes of which participants may not spend enough time reflecting. In this way, users will be empowered to take control of their relationships and be given feedback on their outgoing and incoming communication. In the best case scenario, this will help them be more mindful and work to improve their relationships, and in the worse case scenario, this will help them identify potential signs of an abusive relationship or friendship. On both ends of the spectrum, users are empowered to make sense of and highlight data points in their relationship that they or their partner may have not considered themselves.

Figure 1- Relatio’s General Outline

All of the features work to support the overall goal of improving relationships and identifying problematic patterns. “At-A-Glance” will show changes in different metrics, which will allow users to get a feel for trends that may not be noticeable day to day. “Self Check-In” allows a user to rate their own experiences, which will supplement the analytics by capturing how different interactions make the user feel. “Documents” and “Get Help” are both there in the case that the user feels unsafe or threatened in the relationship. “Documents” allows a user to collect screenshots of threatening behavior overtime, as in a legal setting, their case will need this to be taken seriously. The “Get Help” section will use the Google Maps API to connect the user with emergency resources nearby. Both of these two sections are “My-Eyes-Only,” which means they would need to enter the correct password to use them, which ensures privacy.

Current State of the Art

The two biggest advancements in this space of relationship analysis are by a Data Scientist, Guy Tsror, who analyzed a year’s worth of texts from his own relationship, and a startup called Mei, which has published the first app to integrate AI into relationship advice and analytics. Tsror, whose work is published on Medium, takes a casual approach, as he explains it was something he did to celebrate his one year anniversary. He utilizes word clouds, line graphs, and charts to show a variety of statistics about his relationship; however, none of them seem to draw anything other than surface-level conclusions, which he readily concedes in the article. The other, Mei, is a messaging app that uses artificial intelligence to give feedback and advice on relationships. Other features of Mei include real-time relationship advice and reply suggestions, scheduled send for text, custom folders and labels, and some analytics. The custom folder idea could be expanded to our use case of detecting toxic communications, in which the user can have a hidden folder where they collect any threatening screenshots. In its current state, to use this app, Android users have to make Mei their primary SMS service and switch from the default one. In our project, we aim to build on top of existing messaging interfaces as a plug-in, to prevent users from having to switch to a new service. We intend to expand on this work by providing relationship analysis in a more accessible setting, specifically the Facebook Messenger service, so it is cross-platform and does not require overriding any phone’s default SMS settings. Additionally, our focus will concentrate on analytics of relationships over-time, and will empower the user to see these patterns and changes as they occur via custom visualization tools.

Need Finding

A combination of surveys and interviews will be used for need-finding, and each helps us have a well-rounded set of information about our potential users. First of all, to get a broad overview of peoples’ opinions about online messaging and their relationships, a survey will be widely distributed across social media platforms, including class pages and clubs to get a diverse sample. A survey is useful because it is easy to distribute, and allows us to get results relatively quickly. However, we cannot rely solely on a survey for conducting product research because all of the information is self-reported. Especially in a setting like ours in which we are gaining feedback on how a messaging analyzer could improve relationships, it is important to acknowledge that self-reported data will be subject to the social desirability effect. This is both because it is rare for people to want to admit that they have poor or toxic habits when it comes to messaging and because often people are not actually aware of this behavior themselves.

After the surveys, we will do some interviews with current users of the Facebook Messenger app to determine their habits and attitudes towards the effect of messaging on their in-person relationships. These will be open-ended based on what the interviewee brings up, but aim to cover how they currently use Messenger, the effectiveness of messaging versus in-person conversation, and how they perceive and interpret messages. One limitation of these interviews is that due to the personal nature of our topic, some participants might feel awkward discussing their private relationships with a researcher. The best way to combat this is to build rapport from the beginning, make sure the users understand what information we are looking for and how we will use it, and follow strict confidentiality with all participants. Additionally, the survey will not track any identifying details, so they are always free to leave more comments in the survey should the need arise.

The biggest recruitment tool, especially in this world circumstance in which people are encouraged to stay home, will be social media. This looks like each of the researchers sharing the survey with their own personal networks, their student organizations, and their class pages. Between the four of us, we have four different majors, student organizations, and hometowns to tap into, which should give us a wide range of geographic, ethnic, and academic backgrounds. While participants in the surveys or interviews will not necessarily benefit from completing the survey or doing the interview, they could be used in the future as beta testers, who would have the potential to use our product in its early stages and gain some key insight into their own relationships, which would be something positive for them. With that in mind, we could use that information to recruit people who are interested in self-discovery, reflection, and potentially improving their relationships.

A sample survey can be found here. Note that this is a copy of the survey which is actually distributed to allow you to view it while still protecting users’ answers from public view.

Prototyping

We will go through 3 prototyping iterations, each increasing in fidelity in both breadth and depth. Having multiple iterations of the prototype will be helpful for us to identify the potential gaps or issues that will emerge in the design process. To achieve our project goals, we brainstormed the following list of key features to be implemented throughout prototyping:

At-a-Glance Status Report: Based on the information collected and analyzed from chat history, Relatio would provide users with plot feedback on their relationship status over day, week, and month. It mainly measures relationship metrics including level of trust, respect, aggression, compromise, responsiveness, and understanding. These summary reports could help users better understand their stage of relationship and more importantly identify any signs of digital abuse. It is visible only to the user, ideally with password protection, to keep privacy and confidentiality.

Self-Reporting Check-in: Self-report is another feature that Relatio provides. It is a safe space for users to channel their thoughts and stay mindful of how certain behaviors in their relationships make them feel and affirmations they deserve love. Additionally, putting time into self-reflection and journaling is an effective and powerful tool to help them relieve their anxiety and stress in an unhealthy relationship.

Documentation: Relatio also saves any threatening texts, actions and evidence in one private and disguised area that is only accessible to the user. This could be useful for legal and personal evidence. For example, if users would like to seek help and support from hotlines or care resources, they are able to directly show the documentation and address their issues more effectively.

Get Help: Relatio would help users connect with local shelters or national hotlines using their APIs in a timely and efficient way. The plug-in comes with google map integration to show the closest emergency resources available to users.

Figure 2. Prototype Iteration Plan

Implementation

In order to create Relatio, we have created an in depth technical plan including various features. After team research and many discussions, we have decided the best route of development is a Google Chrome extension. Our extension will have the ability to be installed on any Chrome browser and with just a click analyze the relationship of the user and whoever they are currently messaging on Facebook Messenger. The user will need to have Facebook Messenger open to the person they’d like to analyze their relationship with including a completed conversation. The user will then be able to click “analyze” which will trigger our backend Javascript plugin IBM Watson to give the user relationship metrics including trust, aggression, respect, compromise, understanding, and responsiveness. We will also offer explanations of each of these metrics via an information button so that the user can properly understand what their metrics mean. We will also offer a “Self Check-In” for the user to compare to our analysis as well as a “Get Help” section if the user finds themself in an abusive relationship. Though a simple front-end application, the back-end will be extensive to provide accurate and useful results to the user about their current relationship. This application will have a front-end of HTML with a backend of Javascript as most Google Chrome Extensions are made in that format. The following images depict an example of what the extension could look like in our final product.

Figure 4. Project Concept

Of course, we are sure to run into road bumps and technical challenges. None of us have recently created a Chrome extension so there will definitely be a learning curve in that aspect. We are also expecting to run into issues with grabbing the data from the user’s messages but are hoping Chrome extensions vast development guide will help us with that. On top of that, linking multiple plugins together both front-end and back-end always poses a challenge but we are confident that our team can handle it. Our design definitely has a few trade-offs as ideally this app would be an iMessage plugin since iMessage is more popular than Facebook Messenger, and iPhones are more accessible than laptops. However, Apple does not allow plugins on iMessage and is less flexible with its developers. Given the time constraint and the background of web development knowledge in our group, we feel a Google Chrome extension will be the best route. Ultimately, our goal is to allow users to check the health of their relationship and if we can make that happen on any device it will be a success in our eyes.

User Study / Evaluation

To test the usability of Relatio, we decided to conduct A/B testing to compare our two upcoming proposed solutions and help to shape the final plug-in. Our hypothesis is that version A is easier to use than version B in terms of navigation. More specifically, we will focus on manipulating the independent variable: navigation to measure the dependent variable ease of use to complete a task. The baseline or control condition is version A, which is our primary design that we plan to implement; version B is the challenger which alters the navigation of version A. For participants who will be involved in the study, we will run the evaluation with a combination of within-subjects and between-subjects so that we can include counterbalancing to avoid ordering effects. In terms of the participant profile, we plan to recruit a total of 10 participants (Note: 10 is the minimum number of user study, we will strive to recruit as more participants as we could due to the pandemic situation), who will be randomly divided in half and allocated to group A and group B. The desirable participants should be familiar with using Facebook Messenger, having access to Chrome extension and in an ongoing relationship. Due to the current COVID-19 situation, we plan to recruit from family and friends who will meet the before-evaluation criteria to help us evaluate virtually through observing them live. How we plan to conduct the testing is to have group A (n = 5 participants) first test version A and then version B, and then have group B (n = 5 participants) test the conditions in the reverse order. Please see the diagram of the usability testing design below:

If we have more time allowed, we also plan on conducting an experimental design of longitudinal study in order to understand effectiveness and ensure quality of the finished product. Our hypothesis is that Relatio will help users take better control of their relationships through sentiment analysis and documentation than the control condition-- conventional approaches such as talking to family or friends about relationship issues. The null hypothesis is that there is no difference between Relatio and the conventional approach, and vice versa for the alternative hypothesis. Similar as above, we will run the experiment again with both within-subjects and between-subjects to avoid ordering bias. Most of the experimental design is the same as the usability testing described from above except that this experiment is longitudinal, which means we measure the impact by asking feedback from the same group of participants every week with a short questionnaire. We believe that one of the crucial metrics that would make Relatio better than what exists today is usefulness. With over-time objective feedback on relationship status, users could better track the health of their relationships or identify any trends of digital abuse. Another important metric is accessibility. With Relatio, users can have easy access to the plugin and no longer need to override any phone’s default SMS settings, switch to a new device, or download a new app. One last metric is resource. Unlike other existing ones, Relatio is able to connect the users with online and offline resources that could be helpful in improving their relationship status.

According to Murphy’s law, anything that can go wrong will go wrong. Keeping that note in mind, our team will strive to plan ahead and address any potential issues accordingly and timely. In case the outcomes do not come as expected, our fallback plan is to create at least two versions of design and conduct usability testing for ensuring functionality and quality. This way, we are able to meet the project timeline with adequate preparation and less stress. One of most essential features of Relatio is to generate the at-a-glance report. Our team will do our best in implementing this feature in Google Chrome extension with extensive research and guidance from our instructor and teaching assistants. However, if the feature does not work at all as intended, we might want to move onto a new text screenshot website platform where users could upload screenshots of their chat history and the back-end of the platform would generate analysis reports and provide resources from there. Overall, this alternative solution would help meet our project goal to allow users to check the health of their relationships.

Figure 3. Timeline and Deliverable

Alternative Solution

In order to make our project work, we will need a working Google Chrome extension that can properly grab the messages from Facebook Messenger. If this proposed solution does not work out or becomes infeasible, we will create a web based app instead. We would have to incorporate some sort of way to analyze the user’s conversation such as a text box for copy and pasting of messages or a screenshot upload to extract the text from. This obviously is not as seamless as our original plan as it would be a lot more tedious for the user to input their data but would still ultimately allow the user to receive a relationship analysis.

We also must be extremely careful in our project to not report to user’s that their relationship is extremely unhealthy when it may not be. Just as we saw in class, the facial emotions of tennis players were difficult to predict just as conversational emotion will be. In order to be as accurate as possible, this will require the use of specific plugins previously mentioned as well as lots of testing data. On testing, we will be sure to include a disclaimer that our extension is still in beta testing and results may not be completely accurate just yet. We will give the tester the option to report if they believe their relationship analysis was incorrect and explain why so that we can learn from our mistakes. There is always the chance that a user may have an unhealthy conversation and not an unhealthy relationship so we will ensure our extension clearly states that the given data is based on only their Facebook Messenger conversations. We hope to make our extension accurate enough to correctly give constructive feedback to the user in order to improve their relationship.

Of course, we are sure there will be setbacks we run into that we will be able to overcome. We envision issues pertaining to the Google Chrome extension creation, the Facebook Messenger content extraction, and the conversation analysis plugins. However, with the varying and unique technical backgrounds of our team members we will persevere with the knowledge of each other and the help of the vast documentation online for Google Chrome Extension and conversation analysis.