The ultimate measure of doctors’ medical education is the quality of care they provide to their patients. But how do you connect the two? How do medical educators know when their curriculum is making a positive impact where it matters most: in practice? These questions have challenged medical school and residency leadership for as long as medical education has existed. At NYU School of Medicine, the curriculum has been revamped numerous times over the years to reflect new insights into how care should be delivered, most recently with the introduction of the Curriculum for the 21st Century, which emphasizes informatics, among other innovations. Residency training, too, has been continually updated at NYU Langone Health.

“We finally have a trove of data that will enable us to answer some fundamental questions, and we’re going to take full advantage of it.”—Marc M. Triola, MD, Director of IIME, Associate Dean for Educational Informatics

But frustratingly even for highly data-driven institutions such as NYU Langone, confirming which changes in education and training actually translate to better care has been more a matter of judgment than solid numbers. “We do surveys among our graduates, but very little data comes back to us,” says Marc M. Triola, MD, associate dean for educational informatics and associate professor of medicine at NYU Langone, where he is also the founding director of the Institute for Innovations in Medical Education (IIME).

That’s about to change. Dr. Triola has been heading up a groundbreaking effort to use big data to bridge the gap between medical education and patient outcomes. “We finally have a trove of data that will enable us to answer some fundamental questions,” he says, “and we’re going to take full advantage of it.”

Gaining Broad Access to Patient-Care Data

The turning point came in 2013 when the federal government began granting public access to hundreds of health data sets from the Centers for Medicare and Medicaid Services (CMS). This windfall of information offered valuable insights into myriad facets of patient care, from tests and diagnoses to costs and outcomes—especially when coupled with similar treatment data on non-CMS patients made available by several states, including New York.

Still, there was a hitch when it came to tying that data back to medical school: The government data are linked to physicians not by their names, but by their National Provider Identifier, or NPI, a 10-digit number assigned to every practicing physician. That final piece of the puzzle fell into place for Dr. Triola’s team in 2015 when the American Medical Association (AMA) provided access to its database of NPIs, along with a $1 million grant to support the team’s work. “We’ve been wondering for years how changing the way we train physicians can make a difference,” says Susan Skochelak, MD, the AMA’s group vice president for medical education. “Marc has been wonderful at conceptualizing how big data can help tell us that.”

Since then, Dr. Triola has identified the detailed practice data of some 8,500 graduates of NYU School of Medicine, and about 12,000 graduates of NYU Langone’s residency training programs. The data set is capable of shedding light on any number of queries about how instituting curriculum changes impacts medical practice. In a sense, they are the types of questions that Dr. Triola was born to tackle.

“I was raised in a statistics family,” he explains. “My parents were both teachers, and my father wrote a popular textbook on statistics.”

Connecting Curriculum Reforms to Physician Practice

For the past year, Dr. Triola and his team have been focused on the massive challenge of organizing and warehousing the disparate databases. However, as an early proof of concept, the team has already published data showing how curriculum reforms at NYU School of Medicine influenced the prescribing behavior of its graduates. One query looked at prescriptions for benzodiazepine-based anti-anxiety medicines, which have come to be seen as risky for patients over the age of 65. While the data show that prescription rates declined fairly smoothly year over year among physicians in general, the results were quite different for NYU School of Medicine graduates. “We clearly see two distinctive drops in prescription rates that coincide with two separate reforms to our pharmacology curriculum,” says Dr. Triola.

That insight might seem obvious—one expects graduates to follow their training—but curriculum and residency program changes don’t necessarily translate to practice changes over the long haul, Dr. Triola points out. In the years after medical school, when practice data accumulate, graduates can be influenced by their residency, their fellowships, the systems they practice within, their colleagues, and other factors. These influences all serve as potential confounders when trying to isolate correlations between education and practice data.

The good news, Dr. Triola notes, is that in many cases, these variables can be controlled for by comparing NYU School of Medicine graduates with graduates of other schools who end up practicing in the same settings. “That should allow us to see whether physicians continue to bear the imprint of the school they attended, or whether that imprint is largely erased over time by the healthcare system,” he says.

In the meantime, Dr. Triola’s insights into prescription trends not only validate an important curriculum change, but also suggest that broader changes in medicine can be quickly incorporated into medical school training. The latter point particularly resonates with the AMA. “We think there’s been a gap between what’s taught in medical school and how medicine has been shifting to working in teams, to working within healthcare systems, and to focusing on outcomes and patient safety,” says Dr. Skochelak.

“We want to know what can be done in education to close that gap.” To answer that particular question, Dr. Triola says his team plans to look at whether recent curriculum changes that focus on improving how physicians work in teams can be tied to improvements in patient outcomes. If so, those elements might be expanded; if not, different approaches to teaching teamwork might be in order.

Data May Change the Future of Residency Training

Another important area of inquiry is the ongoing shift in surgery from inpatient to outpatient settings. “Our data could give real visibility to a possible need to move some residency training from the hospital to clinics and ambulatory facilities,” Dr. Triola says. Such a hypothetical change would go against a long tradition of embedding residencies firmly in hospitals, he notes, but if the data indicate the shift would improve patient outcomes, there might be a compelling case for enacting it. Likewise, notes Dr. Triola, a potentially data-driven target for the medical school curriculum might be the way students learn to help patients with smoking cessation, hypertension control, and other medical-intervention “hot spots” that can have a major impact on patient health at a relatively minor cost. Any curriculum change that can be linked to even a small improvement in those hot spots might benefit healthcare enormously, and Dr. Triola and his team will be looking for those links.

Eager to glean insights from the new data warehouse is Joan F. Cangiarella, MD, associate dean for education, faculty, and academic affairs, and director of NYU School of Medicine’s pioneering three-year MD degree pathway program. “We’d like to study comparisons between the three-year and four-year programs,” says Dr. Cangiarella. “Will the graduates’ residency and practice metrics show that there are differences in professionalism, maturity, and development? Without good data, it’s very hard to assess that.”

Dr. Triola contends that for these questions, among many others, the data might indicate that even small changes to education and training could end up making big differences in the healthcare system. “That’s the whole point of medical education,” he says. “But now we can achieve it with an evidence-based feedback loop.” While his team’s efforts will be focused on NYU School of Medicine’s medical and residents, he notes that it’s hard to overestimate the impact of those graduates on United States healthcare, given that collectively they treat millions of patients a year.

Since embarking on this project three years ago, Dr. Triola has seen other medical schools set up similar big-data efforts. The AMA has given grants to 10 other schools and is also sharing its NPI data with 4 of them.

“When other schools see what NYU School of Medicine is doing with this data to try to improve healthcare, they come to us to see if they can try to do the same thing,” says Dr. Skochelak.

“NYU School of Medicine has really been at the forefront of applying informatics to medicine.”

Dr. Triola points out that it’s important to be wary of taking the dependence on data too far. The risk, he explains, might be that schools keep expanding those elements of education that can be clearly linked by data to improved outcomes and value while shrinking those more personal elements that pay off in ways that aren’t easily quantified by numbers— aspects such as instilling passion, dedication, and integrity. “Our educators have first and foremost a human relationship with their students, not a data-driven relationship,” Dr. Triola says. “Instead of replacing that relationship, the data can and should be used to make it richer.”