Research

Office of Evaluation Sciences

Understanding rural small businesses in the federal marketplace

Understanding small manufacturers in the federal marketplace

Requiring electronic manifests of hazardous waste generated by EPA-led cleanup sites

Evaluating the effect of changes to the Low Income Home Energy Assistance Program

GSA Evaluation Division

Remote Work in the Federal Government

Comparing Outcomes Between Remote and Teleworking Federal Employees

Ratnalekha V. N. Viswanadham1, Hayley M. Belli 2, Saul B. Blecker 2, Christina Wong 3, Tiffany Rose Martinez 2, Andrea B. Troxel 2, Devin M. Mann 2, 3, 4

1 Department of Pediatrics, NYU Grossman School of Medicine
2 Department of Population Health, NYU Grossman School of Medicine
3 MCIT, NYU Langone Health

4 Department of Medicine, NYU Grossman School of Medicine

5 Direction for Research, Studies, Evaluation, and Statistics (DREES) at the Ministry of Health in France

NIH Grant R21/R33 AG057382

We use comments in the EHR left by physicians to evaluate the barriers to adhering to Choosing Wisely guidelines that promotes reducing diabetes medication prescribing in older adults to avoid hypoglycemia..

(SUPPLEMENT)

Ratnalekha V. N. Viswanadham1, Yuhan (Betty) Cui 1, Priyanka Solanki 2, 3, Nicole Redfern 1,3, Amelia R. Shunk4, Angela Mastrianni 1, Defne L. Levine1, Devin M. Mann 1, 2, 3, Safiya I. Richardson* 1, 2, 3

1 Department of Population Health, NYU Grossman School of Medicine, New York, NY

2 Department of Medicine, NYU Langone Health, New York, NY

3 Medical Center Information Technology, Department of Health Informatics, NYU Langone Health,

New York, NY

4 Tulane University School of Medicine, New Orleans, LA

Linking EHR use to care quality offers insights for interventions to improve guideline adherence and close care gaps. We examined how EHR metadata can measure cognitive load in primary care providers during statin prescribing and identify points of cognitive load in the EHR workflow. EHR primary care encounter data from a large academic health system in 2024 were retrospectively extracted. We identified adult patients who met the criteria for statin initiation and calculated their ASCVD risk scores. Cognitive load metrics were derived from EHR metadata. Logistic regressions evaluate associations between cognitive load and statin initiation, adjusting for patient covariates and provider fixed effects. Gradient-boosted forests and SHAP values identified key EHR events and cognitive load associated with the initiation of statin therapy. Longer encounter duration increased the likelihood of statin initiation, whereas more time spent per EHR event decreased it. Non-linear effects were observed for loop count and distinct event count: the probability of initiation decreased with increasing loop counts up to 93.9 loops, then increased beyond this threshold. For distinct events, the initiation probability increased up to approximately 18 events and declined at higher counts. In a gradient-boosted decision tree model, average time per event was the strongest predictor (72.2% relative contribution). Additional positive predictors included the time spent reviewing lab results and on suggested medication order sets. Modifying the order list and looping back to it were negatively associated with statin initiation. EHR metadata can associate cognitive load with appropriate clinical behavior, finding nonlinear relations between cognitive load and statin initiation rates. This work highlights the need to optimize EHR systems to reduce cognitive burden and support clinical decision-making. Connecting cognitive load to prescribing behavior gives insight into how workflow adjustments and enhanced decision support can improve adherence to guidelines and patient care.

John Michael Meddar1, Ratnalekha Viswanadham1, Defne Levine1, Kendra Willis2, Tiffany Rose Martinez1, Noah Choi, Jackson Douglas, Katharine Lawrence2

1 Department of Population Health, NYU Grossman School of Medicine

2 Weill Cornell Medical School

3 Department of Medicine, NYU Grossman School of Medicine

The COVID-19 pandemic accelerated the adoption of telemedicine in the U.S, which negatively impacted racial/ethnic minorities who struggled with unequal access and use of digital technologies. In this study, we analyzed literature from the pandemic, examining unequal use of telemedicine among racial/ethnic minority groups. Seventy-seven studies met our inclusion criteria, and findings were summarized based on the reported frequency of telemedicine use or proportional use compared to non-Hispanic Whites. We compared telemedicine use across the following visit types: video, telephone, and in-person visits. Our findings showed a greater number of studies reported increased telemedicine (video and/or telephone) use among non-Hispanic Black and Hispanic populations, while more studies reported decreased use among Asians and other minority populations. We also examined factors that influenced telemedicine adoption across studies. Identified factors included access to digital technologies, digital literacy, providers’ racial bias and attitudes towards telemedicine, institutional readiness to adopt and effectively use telemedicine, community outreach, and policies influencing digital expansion and telemedicine reimbursements. Our findings highlight uneven patterns of telemedicine use among minority populations and a multitude of factors that influence these patterns. Effective strategies are needed across various levels of healthcare delivery to improve the digital inclusion of minority populations

Daniel Tawfik; Adam Rule; Aram Alexanian; Dori Cross; A Jay Holmgren; Sunny S. Lou; Eugenia McPeek Hinz; Christian Rose; Ratnalekha V. N. Viswanadham; Rebecca G. Mishuris; Jorge M. Rodríguez-Fernández; Eric Ford; Sarah Florig; Christine A. Sinsky; Nate Apathy

This article aims to introduce emerging measurement domains made feasible through electronic health record (EHR) use metadata, to inform the changing landscape of healthcare delivery. We reviewed domains in which EHR metadata may be used to measure healthcare delivery, outlining a framework for evaluating measures based on desirability, feasibility, and viability. We argue that EHR use metadata may be leveraged to develop and operationalize novel measures in the domains of Team Structure and Dynamics, Workflows, and Cognitive Environment, to provide a clearer understanding of modern healthcare delivery. By enabling measures that can be used to inform the next generation of healthcare delivery, EHR use metadata may be used to improve the quality of patient care and support clinician well-being. Careful attention is needed to ensure these measures are desirable, feasible, and viable.

Danissa Rodriguez1, Ji Chen1, Ratnalekha Viswanadham2 , Katharine Lawrence1, Devin Mann1, 3

1 Department of Population Health, NYU Grossman School of Medicine
2 Department of Pediatrics, NYU Grossman School of Medicine
3 Department of Medicine, NYU Grossman School of Medicine

• 2023: Poster, Society of General Internal Medicine Annual Meeting, Aurora, CO

Digital diabetes prevention programs (dDPPs) are effective “digital prescriptions” but suffer from high attrition rates and program non-completion. To address this, we developed PAMS, a personalized automatic messaging system that leverages SMS and data integration into clinical workflows to enhance patient-provider communication and increase engagement with a dDPP. Preliminary data showed positive results. Further investigation is needed to evaluate the role of machine learning in developing a more personalized version of PAMS with tailored support technology to boost engagement. This study leverages machine learning to develop digital engagement phenotypes of dDPP users and assess its accuracy in predicting engagement with dDPP activities. Learning from this research will be used as part of a PAMS optimization process to increase PAMS personalization by incorporating engagement prediction and digital phenotyping. To prove the feasibility of using dDPP user-collected data to build a machine learning model able to predict engagement and contribute to identifying digital engagement phenotypes. To describe methods for developing the machine learning models using our dDPP datasets and present preliminary results. To present preliminary data on user profiling based using the output of our machine learning results. Using the gradient-boosted forest model, we predicted engagement in four key dDPP individual activities (physical activity, lessons, social activity, and weigh-ins) and general activity (engagement in any activity) based on previous short-term and long-term activity in the application. The area under the ROC curve (AUROC) and the Area under the Precision-Recall curve (AUPRC) metrics determined model performance. Shapley values reflected the feature importance of the models and determined what variables were used for user profiling with latent profile analysis. We developed two models using weekly and daily DPP datasets (328,821 and 704,242 records, respectively) that yielded predictive accuracies above 90%. Although both models were highly accurate, the daily model better fit our research plan because it predicted daily changes in individual activities, crucial information to be used when creating the “digital phenotypes.” Additionally, to better understand the variables contributing to the model predictor, we calculated the Shapley values for both models to identify the features with the highest contribution to model fit; results suggested that engagement with any activity in the dDPP in the last seven days had the most predictive power. We profiled users with latent profile analysis after two weeks of engagement (BIC = -3222.46) with the dDPP and identified six profiles of users, including those with high engagement, minimal engagement, and attrition. Preliminary results demonstrate the feasibility of applying ML methods with predicting power as an acceptable mechanism to tailor and optimize messaging interventions to support patient engagement and adherence to digital prescriptions. The results enable future optimization of our existing messaging platform and expansion of this methodology to other clinical domains.

Clinical Trial: https://www.clinicaltrials.gov/ct2/show/NCT04773834 

Ratnalekha Viswanadham1

1 Department of Pediatrics, New York University Grossman School of Medicine

Designing the next mass vaccination campaign will require ideas and techniques from behavioral economics. History and current events have shown how failing to consider features of human behavior, from individual biases in information processing to the relationships between people and policymakers, frustrated public health efforts. A campaign should consider how to deliver processes and information necessary to help people decide to vaccinate. Such campaigns need to employ “nudges” and remove “sludge” to influence decisions directly. Strategies must be both acute and systemic to the current approach to public health.

Press: The Communication Initiative Network, AcAudio

Chapter in The State of Mind of Economics. Editors: Alan Kirman and Miriam Teschel

Authors: Hilke Plassmann1, 2 and Ratnalekha Viswanadham1, 2

1INSEAD, Fontainebleau, France
2Control-Interoception-Affect (CIA) team, Institut du Cerveau et de la Moelle Épinière (ICM), INSERM UMR 1127, CNRS UMR 7225, Sorbonne Université, Paris, France

We discuss the implications of the neuroeconomics research underlying food choice on policy interventions.

pdf here

Ratnalekha Viswanadham, Yuhan (Betty) Cui, Amelia Shunk, Nicole Redfern

1INSEAD, Fontainebleau, France
2Control-Interoception-Affect (CIA) team, Institut du Cerveau et de la Moelle Épinière (ICM), INSERM UMR 1127, CNRS UMR 7225, Sorbonne Université, Paris, France

• 2024: Poster, American Medical Informatics Association's Annual Conference

(in preparation for  Frontiers in Nutrition)

Ratnalekha Viswanadham 1, 2 and Paulo Albuquerque1

1INSEAD, Fontainebleau, France
2Control-Interoception-Affect (CIA) team, Institut du Cerveau et de la Moelle Épinière (ICM), INSERM UMR 1127, CNRS UMR 7225, Sorbonne Université, Paris, France

• 2021: Talk at INFORMS Marketing Science Institute (virtual), HEC Paris Marketing Research Camp (co-author)

We measure the heterogeneity in the effects of two pre-selected front-of-pack (FOP) nutrition labels on food product choices in real-life grocery shopping settings. Previous literature has measured the average effect of FOP labels but does not explain in what conditions these labels work better or worse. Using a large data set of more than 1.1 million purchases resulting from over 200,000 labels placed in two food categories, we quantify how the effect of FOP labels varies with (1) consumer basket characteristics; (2) store characteristics; and (3) product characteristics. We find that FOP labels have the strongest response by consumers that buy the most expensive products, who are very likely the higher-income consumers, creating even more inequality in terms of the healthiness of food choices between rich and poor consumers. We also show evidence that retailers, with their assortment and pricing choices, have a strong impact on the effectiveness of the FOP labels.

pdf here (Last updated 25 May 2021) 

(reanalysis)

Ratnalekha Viswanadham1,2, Yann Cornil3, Pierre Chandon1, Liane Schmidt2, Christine Poitou4, Michele Chabert4, Judith Aron-Wisnewsky4, Karine Clément4 & Hilke Plassmann1, 2

1INSEAD, Fontainebleau, France
2Control-Interoception-Affect (CIA) team, Institut du Cerveau et de la Moelle Épinière (ICM), INSERM UMR 1127, CNRS UMR 7225, Sorbonne Université, Paris, France,
3Sauder School of Business, University of British Columbia, Vancouver, Canada
4Institute Cardiométabolism & Nutrition, Université Pierre et Marie Curie Paris VI, Paris, France

• • 2019: ICM Internal Retreat, INSEAD Thesis180 Competition (First Place), SfNE

  • 2018: SCP, Marketing Science, INSEAD-Wharton Doctoral Consortium, ACR

  • 2017: SfNE, OECD State of Mind in Economics, SJDM

  • 2016: SfNE

We investigate how bariatric surgery impacts delay discounting (DD), its temporal stability, and its domain differences for food and money. DD for money has been identified as a potential metric of impulsivity in addictions including eating disorders and obesity. Current literature provides little evidence of a causal effect of DD through body weight. DD has not been investigated in morbid obesity, and it is unclear whether obesity-related impulsivity is domain-specific.

In Study 1, obese and lean participants made a series of indifference point choices for chocolates and Euros. The obese were recruited from patients qualifying for bariatric surgery and were seen before surgery and twice afterward (3 & 12 months). Lean controls repeated the task 6 months later. Obese patients pre-surgery asked for more chocolates to wait one week compared to lean controls. The effect was attenuated post-surgery among obese patients but remained stable over time among lean subjects, shown by the significant interaction effect between participant group and time. We found no difference between obese and lean subjects for monetary rewards and no change post-surgery.

In Study 2, lean subjects and obese patients from the same hospital pool completed an incentive-compatible, DD task consisting of choosing between smaller-sooner and larger-later rewards for money and monetarily equivalent chocolate rewards. Both groups repeated the task 6 months later, obese patients after bariatric surgery. We estimated the ASAP model and compared differences in its discount parameter between groups and over time. Again, obese patients exhibited greater impatience for food rewards than lean individuals; this difference was most pronounced before surgery. A planned contrast on the interaction between participant groups and time for the difference in DD before and after surgery showed that patients became less impatient after the surgery. We found no such effects for money.

In Study 3, we eliminate alternative explanations to DD differences across reward domains. Participants of lean, overweight, or obese BMIs repeat DD tasks similar to those of Study 2 but with various reward types. Participants discount less for monetary rewards compared to any food rewards, independent of weight status. DD was consistent in unhealthy and healthy food choices in choices between quantities. In food choices based on quality (less tasty food now or tastier food later with fixed quantities), overweight and obese participants make significantly more impulsive choices than lean participants. Participants do discount future rewards more for food than money, regardless of trade-off type like food or quantity versus quality.

(data collection resumed after delays from France pension strikes and COVID-19)

Ratnalekha Viswanadham1,2, Redwan Maatoug1, Nicolas Langbour3; Ghina Harika-Germaneau3, Bruno Millet1 and Hilke Plassmann1, 2

1INSEAD, Fontainebleau, France
2Control-Interoception-Affect (CIA) team, Institut du Cerveau et de la Moelle Épinière (ICM), INSERM UMR 1127, CNRS UMR 7225, Sorbonne Université, Paris, France
3Center Hospitalier Henri Laborit, Poitiers, France

  Every day, people decide between immediate but temporary gratification and delayed but possibly more rewarding outcomes. Self-control, or the ability to regulate one’s actions, thoughts, and impulses in the face of such conflicts in pursuit of higher goals; helps individuals make financial decisions, choose foods, and (in light of the global COVID-19 pandemic), follow restrictions on daily life. Higher individual self-control has been correlated positively with better grades in school, better psychological adjustment during adversity, decreased binging on food and addictive substances, better social relationships, and more positive emotional responses (Tangney et al. 2004). It remains unclear how self-control produces these outcomes; understanding the mechanism can reveal what happens within people with insufficient self-control and guide strategies for improving human behavior. Our main question is to determine how changing the ability to assess the value of different choices impacts self-control.  

  Self-control relies on a system of interconnected processes that involve working memory, executive function, and value integration. Numerous studies have shown how manipulating activity in the dorsolateral prefrontal cortex (dlPFC) changes how the brain’s valuation system (BVS) weighs important factors in value-based decision-making, as both regions are involved in working memory and executive function (Camus et al. 2009; Tusche and Hutcherson 2018). However, directly manipulating the BVS, namely in the orbitofrontal cortex (OFC), is extremely difficult without either creating lesions in animal subjects such as monkeys or studying the effects of previous brain damage or neurological conditions in patients (Fellows 2006; Noonan et al. 2017).

  We plan to study obsessive-compulsion disorder (OCD) and its non-invasive brain stimulation therapy as natural perturbations of brain activity in the OFC to investigate neurological processes involved in self-control. We are fortunate to receive access to patients in an OCD outpatient clinic receiving inhibitory repetitive transcranial stimulation (rTMS) to the OFC to run a set of economic experiments. OCD arises from cognitive control and decision process deficits as seen in its characteristic signs of ritualistic organizations, behaviors, and obsessions (Nestadt et al. 2016). However, the exact psychological and neuroscientific mechanisms underlying these deficits remain ambiguous. Past studies have used imaging and brain connectivity measurements to observe that OCD patients exhibit heightened activity in the OFC at resting state and when presented with compulsive control behavior triggers, so we hypothesize that studying OCD patients can reveal how the OFC contributes to self-control. 

  Our study bridges the gap between the biological basis of self-control and the day-to-day mental processes that guide how a consumer makes choices. Our experiments can reveal the biological mechanisms underlying self-control and its interrelated systems. Because we work directly with patients, our results would provide further proof of the efficacy of an OCD therapy beyond the evidence provided by clinical measures and assessments. 

• 2014: Haas School of Business, UC Berkeley (under Ming Hsu, Eric Set, and Ignacio Saez): I tested a post-hoc Bayesian model reduction process to identify significant dopaminergic genetic variations that correlated with learning behavior in a gambling task. 

• 2013-2014: Caltech (under Antonio Rangel and John Clithero): I built upon a coordinate-based meta-analysis of brain regions that are commonly involved in value-based and perceptual decision-making processes.

  • 2014: NCRC, Harvard University

  • 2013: Doris S. Perpall Speaking Competition Semifinalist, SCCUR

• 2012: NASA-Jet Propulsion Laboratory (under Benjamin Holt): Seasonality and In-Situ Ice Thickness in the Arctic Circle 

• 2010: Stanford University School of Medicine (under Minnie Sarwal and Purvesh Khatri): I curated laboratory and publicly available clinical data to begin a large-scale meta-analysis to help identify common microRNA markers that can predict acceptance of solid organ xenografts.

(preprint; submitted to PLOS Digital Health)

Ratnalekha Viswanadham1, Simon Jones2, Kellie Owens2, Safiya Richardson1, 2

1 Department of Pediatrics, NYU Grossman School of Medicine
2 Department of Medicine, NYU Grossman School of Medicine

• 2023: Poster, Society of General Internal Medicine Annual Meeting, Aurora, CO; Talk, AcademyHealth Annual Research Meeting, Seattle, WA.
  
  

Digital tools meant to standardize care can unintentionally contribute to which patients receive care. We investigate whether providers’ use of these tools relates to a patient’s identity or is influenced by the types of tools available in primary care. Using electronic health record data from a large urban health system, we find that provider responses to alerts are shaped not only by patient identity but also by the alert's nature. Direct effects suggest that providers may engage differently with CDS depending on patient demographics, pointing to potential interpersonal bias. Indirect effects show that certain patient groups are more or less likely to receive specific alerts, indicating embedded algorithmic or systemic bias. These findings underscore the importance of evaluating both provider behavior and CDS tool design when assessing equity in digital health. Even when providers respond consistently, unequal alert exposure can produce inequitable outcomes. Our results underscore the need for more transparent, equity-aware CDS design and implementation strategies that account for both human and technological sources of bias.