Using Production And Postmarket Data To Validate FMEA Assumptions

Failure mode and effects analysis (FMEA) risk priority number (RPN) scores have traditionally been used to quantify risks for users, designs, and processes. Recently, action priority (AP) tables were introduced as an alternative method to compute relative risk. 

One common issue with regulatory agencies and certification bodies regarding FMEA is that organizations are not utilizing production and postmarket surveillance data to validate the probability of occurrence and probability of detection assumptions, which leads to inspectional observations and audit findings.

Risk Ranking

FMEA helps quantify and prioritize risk using criticality/severity, occurrence, and detection ratings that are used to determine the level of risk. Criticality/severity (S) is a measure of the seriousness of the possible consequences of a hazard. Probability of occurrence (O) is the likelihood that a cause will occur and that it will induce the detrimental consequences of the associated hazard. Detection (D) is the probability that controls (design, in-process, inspection, alert/warning) will eliminate, mitigate, or catch the defect and prevent escapes.

The output of an FMEA is the relative risk for each failure mode, which is used to rank the failure modes on a relative risk basis. There are two methods of rating the risk: RPN and AP tables.

The RPN output of an FMEA is a relative risk score for each failure mode, which is used to rank the failure modes on a relative risk basis. The scoring system typically used for S, O, and D is from 1 to 5, with 1 considered low risk and 5 high risk. The RPN score is calculated by multiplying the scores for S, O, and D. 

Table 1 – Example Criticality/Severity, Occurrence, and Detection Ratings (RPN Method) (See Appendix I – Example RPN Action Criteria and Appendix II – Example Criticality/Severity and Occurrence Action Requirements.)

The AP table assigns one of three suggested rankings for each action based upon the S, O, and D values. The AP tables assign risk weighted first on S, then O, and finally D. Table 2 provides example S, O, and D ratings for the AP method.

The AP rankings are as follows:

• High Priority: The FMEA team must identify appropriate actions or improve the prevention or detection controls or reduce the probability of occurrence.
• Medium Priority: The FMEA team should identify an appropriate action or improve the prevention or detection controls or reduce the probability of occurrence.
• Low Priority: The FMEA team could improve upon the prevention and detection rankings or reduce the probability of occurrence.

Table 2 – Example Criticality/Severity, Occurrence, and Detection Ratings AP Method (See Appendix III – Example AP Table)

Probability Of Occurrence

Production data, specifically nonconformance data, can be used for validating the probability of occurrence assumptions. Nonconformance data can be thought of as the probability of inspection and verification systems catching failure modes internally. But the true probability of occurrence is the sum of the internal failures (nonconformances) and external failure data (complaints) or escapes of nonconformances.

Probability Of Detection

Complaint data can be used for validating the probability of detection assumptions. The probability of detection can also be thought of as the probability of inspection and verification systems allowing an escape of a failure mode.

Organizing Data

The key is to have the nonconformance and complaint systems organized with logical failure modes that are readily identifiable. The data should be easy to collect, quantify, and analyze. The more sophisticated and complex the products and the organization, the more sophisticated and complex the monitoring and measuring systems should be implemented. The examples shown here are simple counts per year; however, normalized rates per units produced or sold per month are probably better as normalized to account for fluctuations in production and sales.

Example failure modes could include defects involving:

• Labeling
• Packaging
• Cosmetics
• Dimensional
• Functional
• Other    

It should be obvious that each of the failure modes identified here can be subdivided further. I strongly suggest that there should be no more than five to 10 top-level failure modes to help facilitate organizing nonconformances and complaints. Each of the top-level failure modes can then be further subdivided, as necessary. Table 3 provides an example of a top-level failure mode for labeling and possible sub-level categories.

Table 3 – Example top-level failure mode for labeling and possible sub-level categories for labeling

Traditional FMEA Example

A team has developed an FMEA for labeling with three failure modes, as shown in Figure 1. Note the numbers in parentheses are the total number of failures/escapes assumed per year. After one year, the FMEA team collected and summarized the internal nonconformance and external complaint data to validate the initial assumptions of probability of occurrence and probability of detection. The summarized data is shown in Figure 1. See Table 1 for FMEA ratings.

Figure 1 – Traditional FMEA assumptions

For failure mode L10 Wrong Label, the probability of occurrence was estimated to be 1 (three failures per year), but the actual number of failures for the year was 11 (nonconformances + complaints). Therefore, the probability of occurrence must be increased from 1 (three failures per year) to 3 (30 failures per year) to reflect the actual data. The probability of detection was estimated to be 5 (300 failures per year). Therefore, the probability of detection can reduce from 5 (300 failures per year) to 3 (30 failures per year) to reflect the actual data of nine reported complaints.

For failure mode L11 No Label, the probability of occurrence was estimated to be 1 (three failures per year), and the actual number of failures for the year was 16 (nonconformances + complaints). Therefore, the probability of occurrence must be increased from 1 (three failures per year) to 3 (30 failures per year) to reflect the actual data. The probability of detection was estimated to be 3 (30 failures per year). Therefore, the probability of detection assumption is valid, as there were 16 reported complaints.

For failure mode L12 Illegible Labels, the probability of occurrence was estimated to be 3 (30 failures per year), and the actual number of failures for the year was 33 (nonconformances + complaints). Therefore, the probability of occurrence must be increased from 3 (30 failures per year) to 5 (300 failures per year) to reflect the actual data. The probability of detection was estimated to be 3 (30 failures per year). Therefore, the probability of detection assumption is valid, as there were 18 reported complaints.

Figure 2 – Updated traditional FMEA

FMEA Action Priority (AP) Method Example

A team has developed an FMEA for labeling as shown in Figure 3 for labeling with three failure modes. Note the numbers in parentheses are the total number of failures/escapes assumed per year. After one year, the FMEA team collected and summarized the internal nonconformance and external complaint data to validate the initial assumptions of probability of occurrence and probability of detection. The summarized data is shown in Figure 3. See Table 2 for FMEA ratings.

Figure 3 – Action priority (AP) FMEA assumptions

For failure mode L10 Wrong Label, the probability of occurrence was estimated to be remote (three failures per year), but the actual number of failures for the year was 11 (nonconformances + complaints). Therefore, the probability of occurrence must be increased from remote (three failures per year) to moderate (30 failures per year) to reflect the actual data. The probability of detection was estimated to be slight (300 failures per year). Therefore, the probability of detection can be reduced from slight (300 failures per year) to adequate (30 failures per year) to reflect the actual data of nine reported complaints.

For failure mode L11 No Label, the probability of occurrence was estimated to be remote (three failures per year), and the actual number of failures for the year was 16 (nonconformances + complaints). Therefore, the probability of occurrence must be increased from remote (three failures per year) to moderate (30 failures per year) to reflect the actual data. The probability of detection was estimated to be excellent (three failures per year). Therefore, the probability of detection assumption is valid, as there were no reported complaints.

For failure mode L12 Illegible Labels, the probability of occurrence was estimated to be moderate (30 failures per year), and the actual number of failures for the year was 17 (nonconformances + complaints). Therefore, the probability of occurrence assumption is valid. The probability of detection was estimated to be adequate (30 failures per year). Therefore, the probability of detection can be reduced from adequate (30 failures per year) to excellent (three failures per year) to reflect the actual data of two reported complaints.

Figure 4 – Updated action priority (AP) FMEA assumptions

Conclusion

FMEAs should never be one and done. It is time to ensure that your FMEA process benefits from the use of production and postmarket surveillance data. Using production and postmarket surveillance data can help your organization design and manufacture safer products and demonstrate compliance with international standards, guidances, and regulatory requirements.

Ensure the production and postmarket surveillance data is easy to collect, quantify, and analyze using logical failure modes that are readily identifiable.

I cannot emphasize enough the importance of establishing procedures (documenting) to manage the tools and methods used. Best practice includes providing the rationale for your organization’s use of risk management tools and activities.  The requirements and risk management tools presented in this article can and should be utilized based upon industry practice, guidance documents, and regulatory requirements.

References:

1. FMEA Handbook 1^st Ed, 2019, Automotive Industry Group (AIAG) and the Verband der Automobilindustrie (VDA), Southfield, MI.
2. Durivage, Mark A., “Is It Time To Say Goodbye To FMEA Risk Priority Number (RPN) Scores?” https://www.pharmaceuticalonline.com/doc/is-it-time-to-say-goodbye-to-fmea-risk-priority-number-rpn-scores-0001

Appendices:

Appendix I – Example RPN Action Criteria

Appendix II – Example Criticality/Severity and Occurrence Action Requirements

*ALARP - As Low as Reasonably Possible

Appendix III – Example Action Priority (AP) Table

About The Author:

Mark Allen Durivage has worked as a practitioner, educator, consultant, and author. He is managing principal consultant at Quality Systems Compliance LLC, an ASQ Fellow and SRE Fellow. Durivage primarily works with companies in the FDA regulated industries (medical devices, human tissue, animal tissue, and pharmaceuticals), focusing on quality management system implementation, integration, updates, and training. Additionally, he assists companies by providing internal and external audit support as well as FDA 483 and warning letter response and remediation services.He earned a BAS in computer aided machining from Siena Heights University and an MS in quality management from Eastern Michigan University. He holds several certifications including CRE, CQE, CQA, CSSBB, RAC (Global), and CTBS. He has written several books available through ASQ Quality Press, published articles in Quality Progress, and is a frequent contributor to Life Science Connect. You can reach him at mark.durivage@qscompliance.com and connect with him on LinkedIn.