Predictive Modeling of Opioid Prescription Fraud by CMS Providers

Natural and synthetic opioids are powerful and highly effective prescription medicines used by doctors / health care providers in the treatment of moderate to severe pain in patients. Although useful as an approach to pain management, opioids have also shown to be dangerous as they can lead to addiction, overdoses, and even death. With a reported 46,000 opioid-related deaths in 2017 (an average of around 130 people per day) and an estimated 11.4 million people reportedly misusing prescriptions¹, the over-prescription of opioids presents a sizeable challenge to the healthcare system and its’ patients in the United States.

As last publicly reported by the Centers for Disease Control (CDC) in 2017, the opioid prescription rate per 100 people in the United States was 58.7, which represents a significant decline from its peak of 81.3 in 2012². While experiencing a decline on the national level, on a local / community-level the popularity opioids has fostered an environment where providers and organizations can financially benefit from the over-prescription and misuse of the drugs for the treatment of pain.

PART 1 - Problem Definition

The predictive model will be based upon analysis of the following sources of relevant data:

● Department of Justice Press Releases (2010-2020)

The Department of Justice issues regular press releases pertaining to individuals and organizations which have been formally charged, plead guilty to, or convicted of fraudulent activity. This project will focus on press releases identified as “Health Care Fraud” and reference opioids or opioid-related terms (such as specific drug names) for identification of provider / organization names.

● National Provider Identifier (NPI) Registry

The CMS National Plan and Provider Enumeration System (NPPES) contains a registry of all Medicare / Medicaid eligible providers. This project will utilize NPIs as a mechanism to accurately trace fraudulent provider / organization names to the records contained within the Medicare Provider Utilization and Payment Dataset.

● Medicare Provider Utilization and Payment Data: Part D Prescriber Summary Tables (2013-2017)

The Public Use Files (PUF) contain information on prescription drugs prescribed by providers which have been paid for using the Medicare Part D Prescription Drug Program. The data is based upon the CMS Chronic Conditions Data Warehouse with provider-based prescription records associated with Medicare Advantage Prescription Drug (MAPD) plans and Prescription Drug Plans (PDP). Records are aggregated on a per year basis by unique National Provider Identifiers (NPI) and include the following categories of data: total number of prescriptions dispensed, total drug cost, beneficiary characteristics, and drug categories. This data will be utilized in my project to determine provided-based prescription trends.

● Physician Compare 2017 Individual Eligible Clinician Public Reporting - Overall MIPS Performance

The Physician Compare file contains the Merit-Based Incentive Payment System (MIPS) final scores and performance category scores for all participating clinicians in 2017. This data will be utilized in my project to determine likely legitimate opioid providers.

PART 2 - Existing Research / Initial Exploratory Data Analysis

Existing Research Studies

Two existing research studies were identified which are directly relevant to my projects’ main goal of developing a predictive model to determine the likelihood of opioid-related fraud being perpetrated by a Medicare Part D Provider. First, the US Department of Health and Human Services Office of Inspector General (HHS-OIG) conducted a study that evaluated the prescription drug events records from 2016 with the goal of protecting beneficiaries from the adverse effect of opioid abuse³. The study’s three main components were: an analysis of opioid dispensation and spending (utilization) across all beneficiaries; individual beneficiary usage accounting for prescription strength and treatment duration; and identification of providers which exhibited extreme prescription patterns.

Key results of this study include that one-third of Medicare Part D beneficiaries received at least one opioid prescription; approximately ½ million beneficiaries received large opioid amounts; and around 400 providers exhibited potentially questionable prescription practices.

The second identified research effort is a CMS developed interactive mapping tool which enables users to perform comparisons of Medicare Part D opioid prescriptions on a national-level scale⁴. The tool allows users to explore claim totals and prescribing rates for both opioid and long-acting opioid prescriptions from 2013 through 2017. Percentage change is also determined and displayed for the entire timeframe to assist in visualization of trends at both the state and county levels.

Although in alignment with the referenced studies, the goal of this project would expand upon scope to include an analysis based on known cases of opioid prescription related fraud. By including this aspect, this project endeavors to establish a predictive capability to actively identify suspect providers.

Initial Exploratory Data Analysis

The Medicare Provider Utilization and Payment Data: Part D Prescriber Summary Tables⁵ for 2013 through 2017 were downloaded and analyzed utilizing Python within JupyterLabs. The resulting combined dataframe contained 84 source columns / features⁶ with an additional column added during load for calendar year, which is identified in the filename. There exists one record for each provider with a registered NPI for each year within the timeframe publicly available, which results in approximately 5.5 million source records.

Each record contains four categories of data pertaining to the provider within that specified calendar year. These categories include details on the following: provider registration, applicable Medicare Program, prescription drug utilization, and beneficiary demographics / health characteristics.

Some key outcomes of initial exploratory data analysis are:

PART 3 - Final EDA / Predictive Model Preparation

Developed Data Pipeline for Fraudulent Provider Identification

Entity / Location Extraction: Dataset of each scraped opioid-related DOJ Press Release text was cleansed and spaCy (NLP Python Library) was utilized to extract entities from the text. The spaCy library supports extraction of numerous entities, for my purposes the entities extracted included: ORG (organization), PERSON (names of people), GPE (country, city, state), and LOC (non-GPE location).

Performed Final EDA of Part D Prescriber Summary Table Data

Likely Legitimate Provider Determination: Analyzed opioid provider dataset to identify averages and outliers in terms of opioid and long-acting opioid characteristics (Beneficiary Count, Claim Count, Prescription Rate, Drug Cost, and Day Supply). Of note, the assumptions necessary to determine likely legitimate providers was not initially taken into consideration but is necessary to properly classify providers in the labeled dataset. This determination will be revised prior to execution of predictive models in Part 4.

Predictive Modeling Preparation

CMS Part D Opioid Provider Labeled Dataset Creation: Combined Opioid Provider Dataset with classified listing of fraudulent and likely legitimate providers determined in prior steps to compile labeled data. Classifiers for remaining providers will be left blank to indicate “unknown” status.

Model Construction: Based upon initial research and evaluation of the scikit-learn algorithm cheat sheet , focus will be placed on implementation of Linear Support Vector Classification (SVC) and Naive Bayes. The analysis will be performed using Orange for rank and feature selection, split of data between testing and training, and algorithm execution. Other algorithms will also be explored and evaluated in terms of applicability and accuracy to produce the most statistically significant results.

PART 4 - Execution, Interpretation, and Results

Final Labeled Dataset Adjustments

Before executing the prediction modeling and interpretation of results, several adjustments were needed to finalize the labeled dataset of opioid providers. First, the Fraudulent Provider Identification Data Pipeline was modified to address instances where multiple NPIs were identified during the resolution process. Unfortunately while completing this effort, it was determined several of the 120 DOJ Press Releases on which data mining was originally performed were not fraud-related. The press releases in question were policy-based and thus resulted in an inflated number of fraudulent NPIs identified in Part 3.

To address this issue, the data pipeline was modified to include extraction of identified verbs within the text. The additional extracted data was used to filter out press releases which were unlikely fraud related (see Appendix A for verb listing). This resulted in removal of a subset of extracted entities associated with nine press releases originally included in Part 3.

To resolve the instances of multiple NPIs being returned based on the remaining identified names, the extracted location processing was enhanced to include city-like names (see Appendix A for city filtering terms). The extracted city names were then included in the NPPES API call to narrow results. After implementation of these adjustments, removal of duplicates, and final manual verification of inclusion, the following table outlines the achieved results of the fraudulent provider data pipeline efforts:

The second labeled dataset adjustment needed was identification of likely legitimate CMS Part D Opioid Providers. To determine a subset of CMS Part D Providers which were likely to be acting legitimately in prescription practices, a new dataset pertaining to the CMS Quality Payment Program (QPP) was identified and incorporated. The dataset pertains to the Merit-based Incentive Payment System (MIPS), which is one component of the QPP that rewards participant clinicians with payment adjustments based on their cost efficiency, quality of care, and health outcomes⁷.

Since participation in MIPS is voluntary, adds additional scrutiny to a providers practices, and can either positively or negatively affect payments from CMS, it is highly unlikely a fraudulent provider would participate. Thus, for the purposes of identifying likely legitimate opioid providers for this project, the published MIPS Final Scores for 2017 where providers received a perfect score (100) were merged with the opioid provider dataset. No instances of providers identified as fraudulent were identified in the filtered MIPS-related data.

The table below details the final composition of the CMS Part D Opioid Provider Labeled Dataset utilized in predictive modelling efforts.

Predictive Modeling

Based upon the composition of the final CMS Part D Opioid Provider Labeled Dataset, a semi-supervised machine learning approach was determined to be the best option. Semi-supervised learning falls between supervised (with labeled training data) and unsupervised (with no labeled training data) approaches and provides a practical method to improve learning accuracy when only a small amount of labeled data is available⁸.

To utilize the semi-supervised learning approach for the project, pseudo-labeling was employed to assign approximate labels to the unlabeled portion of the dataset⁹. First, the labeled portion of the dataset was separated and used to train an initial model. The resulting initial model was then used against the remaining dataset to predict fraudulent labels for the previously unlabeled data. Finally, the two datasets were merged and used to create the final predictive model.

Results

The semi-supervised learning approach was implemented within the Orange3 Data Analysis Environment (see Appendix B to view entire prediction model workflow), the following results were observed for each step of the process:

• Labeled Provider Modeling: After filtering the opioid provider dataset for the 38,314 labeled providers, the Information Gain Ratio algorithm was used to determine the top 10 impactful features. These features included (in descending order of importance): la_opioid_bene_count, la_opioid_drug_cost, average_age_of_beneficiaries, opioid_presciber_rate, la_opioid_claim_count, beneficiary_race_nat_ind_count, antipsych_bene_count_ge65, benecificiary_age_less_65_count, opioid_claim_count.

• Pseudo-Labeling of Unlabeled Providers: The Naive Bayes Model generated above was then applied to the remaining 666,149 unlabeled providers. This pseudo-labeling process resulted in creation of approximate / predicted labels for the previously unknown classifiers.

• Combined Dataset: The labeled and pseudo-labeled datasets were then concatenated to produce the final fully labeled opioid provider dataset.

• Fraudulent Provider Prediction Modeling: The fully labeled opioid provider dataset was then sampled using two approaches with the following results:

  • 70/30 Split - The table below display the outcomes of the predictive models based on the 70/30 sampling split. Although Neural Network did perform slightly better, the low number of fraudulent labeled providers lead to the identification of no instances of a fraudulent provider. Thus, only the confusion matrix for Naive Bayes was included, and depicted in the figure below.

• • Cross Validation (10 Folds) - The table below display the outcomes of the predictive models based on cross validation (10 folds) sampling. Neural Network performed the same and Naive Bayes improved slightly.

Conclusion

Although the semi-supervised learning approach with pseudo-labeling yielded results the model prediction was overall hampered by the small size of identified fraudulent providers. I believe with additional data sources and enhancements to the NLP analysis, the results of the fraudulent data pipeline would improve. In conjunction with the process implemented to create the CMS Part D Opioid Provider Labeled Dataset Process, this would position the developed prediction workflow to produce more actionable results.

Lessons Learned

• Early data mining assumption that simple extraction of names and locations from the filtered set of well-structured DOJ Press Releases would not yield a significant amount of falsely identified fraudulent providers was inaccurate. Although lawyers, prosecutors, etc. did not have registered NPIs, the commonality of extracted names amongst all identified locations lead to a significant number of misidentifications since position in text was not considered. Also, a number of press releases were policy or informational based which added to misidentification.

• Basic utilization of the spaCy NLP Python Library to extract entities, locations, and parts of speech was powerful and relatively straightforward. Contextual analysis of the press releases and identification of persons of interest and associated locations was quite challenging. This resulted in use of only rudimentary NLP analysis and contributed to false identification of fraudulent providers.

• All necessary components of the opioid provider labeled dataset were not considered at project on-set. Identification of likely legitimate opioid providers was initially overlooked and lead to delays in the predictive modeling preparation steps.

Future Considerations

• Identify additional data sources for use in identification of fraudulent opioid providers to increase the number of labeled instances and improve prediction model / results.

• Improve NLP to consider an entities position within the text to reduce misidentification and incorporate better contextual processing of text to avoid data mining of informational-only and policy-based press releases.

• Improve determination of likely legitimate opioid providers beyond use of MIPS Final Scoring to include additional datasets.

• Incorporate time component into analysis to evaluate opioid providers trends throughout the span of publicly available data (2013-2017).

• Extract date and timeframes from DOJ Press Releases to better map provider fraud to specific years and provider more granular analysis.

• Improve Fraudulent Providers Data Mining Pipeline to include determination of providers that have been charged multiple times and develop a weighting mechanism so they will be more strongly classified.

• Incorporate feature engineering to improve prediction results.

References

¹ CMS, “CMS Roadmap: Strategy to Fight the Opioid Crisis.” April 2020. Retrieved from https://www.cms.gov/About-CMS/Agency-Information/Emergency/Downloads/Opioid-epidemic-roadmap.pdf on 2020-02-12.

² CDC, “U.S. Opioid Prescribing Rate Maps.” March 2020. Retrieved from https://www.cdc.gov/drugoverdose/maps/rxrate-maps.html on 2020-02-12.

³ HHS-OIG, “Opioids in Medicare Part D: Concerns about Extreme Use and Questionable Prescribing”, OEI-02-17-00250, July 2017. Retrieved from https://oig.hhs.gov/oei/reports/oei-02-17-00250.asp on 2020-02-24.

⁴ CMS, “Medicare Part D Opioid Prescribing Mapping Tool / Methodology”, April 2019. Retrieved from https://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/Medicare-Provider-Charge-Data/Downloads/Opioid_Methodology.pdf on 2020-02-24.

⁵ CMS, “Medicare Provider Utilization and Payment Data: Part D Prescriber Summary Tables” available at https://data.cms.gov/browse?category=Medicare+-+Part+D&sortBy=alpha&tags=provider+summary&utf8=%E2%9C%93

⁶ CMS, “Medicare Fee-For Service Provider Utilization & Payment Data Part D Prescriber Public Use File: A Methodological Overview”, April 2019. Retrieved from https://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/Medicare-Provider-Charge-Data/Downloads/Prescriber_Methods.pdf on 2020-02-20.

⁷ CMS, “Quality Payment Program: MIPS Overview.” Retrieved from https://qpp.cms.gov/mips/overview on 2020-04-25.

⁸ Wikipedia contributors. "Semi-supervised learning." Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 2020-04-19. Retrieved from https://en.wikipedia.org/wiki/Semi-supervised_learning on 2020-05-09.

⁹ Jain, Shubham. “Introduction to Pseudo-Labelling : A Semi-Supervised learning technique.” Analytics Vidhya. 2017-09-21. Retrieved from https://www.analyticsvidhya.com/blog/2017/09/pseudo-labelling-semi-supervised-learning-technique/ on 2020-05-09.