Why Pharmaceutical Risk Management Is in Deep Trouble

The ICH Q9 guidelines finalized in 2005 called for pharmaceutical firms to use a risk-based approach to the specification, design, and verification of manufacturing systems having the potential to affect product quality and patient safety. In 2008, ICH Q10 added that the design of the pharmaceutical quality system should incorporate risk management and risk-based audits.

Pharmaceutical firms had little background in the relevant areas of risk-management. Early troubles the industry faced in applying risk tools developed in other industries are well documented. Potential benefits of proactive risk management include reduction in regulatory oversight and associated costs, reduced cost from discrepant materials, reduced batch-failure rates, and a safer product. Because risk management, in theory, is present in the earliest stages of product and process design,  it can, in theory, raise profitability while improving patient safety.

Such theoretical benefits of good risk management are in fact realized by firms in other industries. In commercial aviation, probabilistic risk analysis is the means by which redundancy is allocated to systems to achieve a balanced  design – the minimum weight and cost consistent with performance requirements. In this sense, good risk analysis is a competitive edge.

From 2010 to 2015, Class 1 to 3 FDA recall events ranged from 8000 to 9500 per year, with an average of 17 injunctions per year. FDA warnings rose steadily from 673 in 2010 to 17,232 in 2015. FDA warning letters specifically identifying missing or faulty risk assessments have also steadily increased, with 53 in 2015, and 83 so far this year, based on my count from the FDA databases.

FDA warnings 2010-2015

It is not merely foreign CMOs that receive warnings identifying defective risk assessments. Abbott, Baxter, Johnson & Johnson, Merck, Sanofi and Teva are in the list.

The high rate of out-of-spec and contamination-recalls seen in the FDA data clearly points to low hanging fruit for risk assessments. These issues are cited in ICH Q9 and 10 as examples of areas where proactive risk management serves all stakeholders by preventing costly recalls. Given the occurence rate in 2015, it’s obvious that a decade of risk management in pharma can’t be declared a major success. In fact, we seem to be losing ground. So what’s going on here, and why hasn’t pharma seen results like those of commercial aviation?

One likely reason stems from evolution of the FDA itself. The FDA predates most of drug manufacture. For decades it has regulated manufacturing, marketing, distribution, safety and efficacy  of drugs and medical devices (among other things) down to the level of raw materials, including inspection of facilities. With obvious benefits to consumers, this role has had the detrimental side effect of maturation of an entire industry where risk management and safety are equated with regulatory compliance by drug makers. That is, there’s tendency to view risk management as something that is imposed by regulators, from the outside, rather than being an integral value-add.

The FAA, by contrast, was born (1958) into an already huge aircraft industry. At that time a precedent for delegating authority to private persons had already been established by the Civil Aviation Act. Knowing the FAA lacked the resources to regulate manufacturing to a level of detail like that in the FDA, it sought to foster a culture of risk in aircraft builders, and succeeded in doing so through a variety of means including  expansion of  the industry participation in certifying aircraft. This included a designated-engineering-rep program in which aircraft engineers are duty-bound delegates of the FAA.

Further, except for the most basic, high-level standards, engineering design and safety requirements are developed by manufacturers and professional organizations, not the FAA. The FAA’s mandate to builders for risk management was basically to come up with the requirements and show the FAA how they intended to meet them. Risk management is therefore integrated into design, not just QA and certification. The contrasting risk cultures of the aviation and pharmaceutical industries is the subject of my current research in history of science and technology at UC Berkeley. More on that topic in future posts.

Changing culture takes time and likely needs an enterprise-level effort. But a much more immediate opportunity for the benefits envisioned in ICH Q9 exists directly at the level of the actual practice of risk assessment.

My perspective is shaped by three decades of risk analysis in aviation, chemical refinement, nuclear power, mountaineering and firefighting equipment, ERM, and project risk. From this perspective, and evidence from direct experience in pharma combined with material found in the FDA databases, I find the quality of pharmaceutical risk assessment and training to be simply appalling.

While ICH Q9 mentions, just as examples, using PHA/FHA (functional hazard analysis), hazard operability analysis, HACCP, FMEA (failure mode effects analysis), probabilistic safety analysis and fault trees at appropriate levels and project phases, one rarely sees anything but FMEAs performed in a mechanistic manner with the mindset that a required document (“the FMEA form”) is being completed.

Setting aside, for now, the points that FMEA was not intended by its originators to be a risk analysis tool and is not used as such in aerospace (for reasons discussed here, including inability to capture human error and external contributors), I sense that the job of completing FMEAs is often relegated to project managers who are ill-equipped for it and lack access to subject matter experts. Further injury is done here by the dreadfully poor conception of FMEA seen in the Project Management Institute’s (PMI) training materials inflicted on pharma Project Managers. But other training available to pharma employees in risk assessment is similarly weak.

Some examples might be useful. In the last two months, I’ve attended a seminar and two webinars I found on LinkedIn, all explicitly targeting pharma. In them I learned, for example, that the disadvantage to using FMEAs is that they require complex mathematics. I have no clue what the speaker meant by this. Maybe it was a reference to RPN calculation, an approach strongly opposed by aviation, nuclear, INCOSE and NAVAIR – for reasons I’ll cover later – which requires multiplying three numbers together?

I learned that FMEAs are also known as fault trees (can anyone claiming this have any relevant experience in the field?), and that bow tie (Ishikawa) “analysis” is heavily used in aerospace. Ishikawa is a brainstorming method, not risk analysis, as noted by Vesely 40+ years ago, and it is never (ever) used as a risk tool in aerospace. I learned that another disadvantage of FMEAs is that you can waste a lot of time working on risks with low probabilities. The speaker seemed unaware that low-probability, high-cost hazards are really what risk analysis is about; you’re not wasting your time there! If the “risks” are high-probability events, like convenience-store theft, we call them expenses, not risks. I learned in this training that heat maps represent sound reasoning. These last two points were made by an instructor billed as a strategic management consultant and head of a pharmaceutical risk-management software firm.

None of these presentations mentioned functional hazard analysis, business impact analysis, or any related tool. FHA (my previous post) is a gaping hole in pharmaceutical risk assessment, missing in safety, market, reputation, and every other kind of risk a pharma firm faces.

Most annoying to me personally is the fact that the above seminars, like every one I’ve attended in the past, served up aerospace risk assessment as an exemplar. Pharma should learn mature risk analysis techniques and culture from aviation, not just show photos of aircraft on presentation slides. In no other industry but commercial aviation has something so inherently dangerous been made so safe, by any definition of safety. Aviation achieved this (1000-fold reduction in fatality rate) not through component quality, but by integrating risk into the core of the business. Aviation risk managers’ jaws hit the floor when I show them “risk assessments” (i,e., FMEAs) from pharma projects.

One thing obviously lacking here is simple analytic rigor. That is, if we’re going to do risk assessment, let’s try to do it right. The pharmaceutical industry obviously has some of the best scientific minds around, so one would expect it to understand the value of knowledge, diligence, and their correct application. So perhaps the root of its defective execution of risk management is in fact the underdeveloped risk culture mentioned above.

The opportunity here is immense. By cleaning up their risk act, pharmaceutical firms could reap the rewards intended by ICH Q9 a decade ago and cut our ballooning regulatory expenses. Leave a comment or reach me via the About link above to discuss this further.

 –  –  –


In the San Francisco Bay area?

If so, consider joining us in a newly formed Risk Management meetup group.

Risk assessment, risk analysis, and risk management have evolved nearly independently in a number of industries. This group aims to cross-pollinate, compare and contrast the methods and concepts of diverse areas of risk including enterprise risk (ERM), project risk, safety, product reliability, aerospace and nuclear, financial and credit risk, market, data and reputation risk, etc.

This meetup will build community among risk professionals – internal auditors and practitioners, external consultants, job seekers, and students – by providing forums and events that showcase leading-edge trends, case studies, and best practices in our profession, with a focus on practical application and advancing the state of the art.

If you are in the bay area, please join us, and let us know your preferences for meeting times.

https://www.meetup.com/San-Francisco-Risk-Managers/

2 thoughts on “Why Pharmaceutical Risk Management Is in Deep Trouble

  1. Bill, great article. I think part of the risk culture problem in Pharma is the paralysis caused in no small part by Six Sigma, which heavily pushes FMEA and RPN. Six Sigma is widely adopted in Pharma and those who go through the training think that they know how to do risk analysis after learning how to multiply S x O x D. It’s almost a religion. At least one Six Sigma reference I’m aware of does claim that FMEA+RPN is used in aerospace and cites Boeing as an example. I don’t have any independent confirmation of that.

    To be clear, though, some SIx Sigma practitioners are actively pushing other risk methods such as FTA, but not FHA that I am aware of. Also, I think that risk assessments in Pharma are now, in general, group activities where the project manager no longer makes these decisions in isolation. Usually a group of subject matter experts are directly involved. Otherwise, yes, pharma is not doing so well with implementing quality risk management.

    Those lone voices crying out in the wilderness of pharmaceutical risk management, pushing for mathematical rigor and the abandonment of the nonsense that is RPN, are mostly ignored by their management. Why? Because compliance trumps science, unfortunately. Manufacturing wants to make sure QA is satisfied, and QA has no reason to abandon current methods. Why don’t they? Because there is no cause-effect relationship that can be discerned from the application of current risk management methods (e.g., FMEA+RPN) and resulting manufacturing quality. As Doug Hubbard says, we fool ourselves thinking that our risk management methods are working only because we have no means of quantifying the relationship between the nonsensical RPN or qualitative rankings (e.g., high, medium, low) and rate of failures. We don’t know that our risk management is working, or not working! We have no way of knowing. But as you pointed out, the rise in FDA warnings over time seems to suggest that we’re not doing so well.

    Unfortunately, in order for those lone voices urging greater rigor to be taken seriously by management, they need to show that the more rigorous methods (which also require more resources to effectively execute) will have a quantitative benefit, and that relationship may only be seen over the course of many years. With such a long period between cause and effect, its almost futile to push for better risk management methods. One would advance his/her career faster by pursuing other improvement initiatives. Hence, we’re stuck.

    Like

    1. Given that the PMI embraces FMEA with RPN, it is likely that Boeing uses RPN for something in the realm of project mgmt, but not in the context of aircraft design and certification.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s