Using Production And Postmarket Data To Validate FMEA Assumptions
By Mark Durivage, Quality Systems Compliance LLC
Failure mode and effects analysis (FMEA) risk priority number (RPN) scores have traditionally been used to quantify risks for users, designs, and processes. Recently, action priority (AP) tables were introduced as an alternative method to compute relative risk.
One common issue with regulatory agencies and certification bodies regarding FMEA is that organizations are not utilizing production and postmarket surveillance data to validate the probability of occurrence and probability of detection assumptions, which leads to inspectional observations and audit findings.
Risk Ranking
FMEA helps quantify and prioritize risk using criticality/severity, occurrence, and detection ratings that are used to determine the level of risk. Criticality/severity (S) is a measure of the seriousness of the possible consequences of a hazard. Probability of occurrence (O) is the likelihood that a cause will occur and that it will induce the detrimental consequences of the associated hazard. Detection (D) is the probability that controls (design, inprocess, inspection, alert/warning) will eliminate, mitigate, or catch the defect and prevent escapes.
The output of an FMEA is the relative risk for each failure mode, which is used to rank the failure modes on a relative risk basis. There are two methods of rating the risk: RPN and AP tables.
The RPN output of an FMEA is a relative risk score for each failure mode, which is used to rank the failure modes on a relative risk basis. The scoring system typically used for S, O, and D is from 1 to 5, with 1 considered low risk and 5 high risk. The RPN score is calculated by multiplying the scores for S, O, and D.
Criticality/Severity Definition 

5 
Catastrophic 
3 
Serious 
1 
Negligible 
Occurrence 

5 
Certain; 300 failures / year 
3 
Occasional; 30 failures / year 
1 
Remote; 3 failures / year 
Detection 

5 
Undetectable; 300 escapes / year 
3 
Moderate; 30 escapes / year 
1 
Excellent; 3 escapes / year 
Table 1 – Example Criticality/Severity, Occurrence, and Detection Ratings (RPN Method) (See Appendix I – Example RPN Action Criteria and Appendix II – Example Criticality/Severity and Occurrence Action Requirements.)
The AP table assigns one of three suggested rankings for each action based upon the S, O, and D values. The AP tables assign risk weighted first on S, then O, and finally D. Table 2 provides example S, O, and D ratings for the AP method.
The AP rankings are as follows:
 High Priority: The FMEA team must identify appropriate actions or improve the prevention or detection controls or reduce the probability of occurrence.
 Medium Priority: The FMEA team should identify an appropriate action or improve the prevention or detection controls or reduce the probability of occurrence.
 Low Priority: The FMEA team could improve upon the prevention and detection rankings or reduce the probability of occurrence.
Criticality/Severity Definition 
Catastrophic 
Moderate 
Minor 
Occurrence 
Certain; 300 failures / year 
Moderate; 30 failures / year 
Remote; 3 failures / year 
Detection 
Slight; 300 escapes / year 
Adequate; 30 escapes / year 
Excellent; 3 escapes / year 
Table 2 – Example Criticality/Severity, Occurrence, and Detection Ratings AP Method (See Appendix III – Example AP Table)
Probability Of Occurrence
Production data, specifically nonconformance data, can be used for validating the probability of occurrence assumptions. Nonconformance data can be thought of as the probability of inspection and verification systems catching failure modes internally. But the true probability of occurrence is the sum of the internal failures (nonconformances) and external failure data (complaints) or escapes of nonconformances.
Probability Of Detection
Complaint data can be used for validating the probability of detection assumptions. The probability of detection can also be thought of as the probability of inspection and verification systems allowing an escape of a failure mode.
Organizing Data
The key is to have the nonconformance and complaint systems organized with logical failure modes that are readily identifiable. The data should be easy to collect, quantify, and analyze. The more sophisticated and complex the products and the organization, the more sophisticated and complex the monitoring and measuring systems should be implemented. The examples shown here are simple counts per year; however, normalized rates per units produced or sold per month are probably better as normalized to account for fluctuations in production and sales.
Example failure modes could include defects involving:
 Labeling
 Packaging
 Cosmetics
 Dimensional
 Functional
 Other
It should be obvious that each of the failure modes identified here can be subdivided further. I strongly suggest that there should be no more than five to 10 toplevel failure modes to help facilitate organizing nonconformances and complaints. Each of the toplevel failure modes can then be further subdivided, as necessary. Table 3 provides an example of a toplevel failure mode for labeling and possible sublevel categories.
L1 Labeling 
L10 Wrong Label 
L11 No Label 
L12 Illegible Label 
Table 3 – Example toplevel failure mode for labeling and possible sublevel categories for labeling
Traditional FMEA Example
A team has developed an FMEA for labeling with three failure modes, as shown in Figure 1. Note the numbers in parentheses are the total number of failures/escapes assumed per year. After one year, the FMEA team collected and summarized the internal nonconformance and external complaint data to validate the initial assumptions of probability of occurrence and probability of detection. The summarized data is shown in Figure 1. See Table 1 for FMEA ratings.
Failure 
S 
O 
D 
RPN 
Nonconformances 
Complaints 
Total 
L10 Wrong Label 
3 
1 (3) 
5 (300) 
15 
2 
9 
11 
L11 No Label 
3 
1 (3) 
3 (30) 
9 
0 
16 
16 
L12 Illegible Labels 
3 
3 (30) 
3 (30) 
27 
15 
18 
33 
Original Assumptions 
Actual 
Figure 1 – Traditional FMEA assumptions
For failure mode L10 Wrong Label, the probability of occurrence was estimated to be 1 (three failures per year), but the actual number of failures for the year was 11 (nonconformances + complaints). Therefore, the probability of occurrence must be increased from 1 (three failures per year) to 3 (30 failures per year) to reflect the actual data. The probability of detection was estimated to be 5 (300 failures per year). Therefore, the probability of detection can reduce from 5 (300 failures per year) to 3 (30 failures per year) to reflect the actual data of nine reported complaints.
For failure mode L11 No Label, the probability of occurrence was estimated to be 1 (three failures per year), and the actual number of failures for the year was 16 (nonconformances + complaints). Therefore, the probability of occurrence must be increased from 1 (three failures per year) to 3 (30 failures per year) to reflect the actual data. The probability of detection was estimated to be 3 (30 failures per year). Therefore, the probability of detection assumption is valid, as there were 16 reported complaints.
For failure mode L12 Illegible Labels, the probability of occurrence was estimated to be 3 (30 failures per year), and the actual number of failures for the year was 33 (nonconformances + complaints). Therefore, the probability of occurrence must be increased from 3 (30 failures per year) to 5 (300 failures per year) to reflect the actual data. The probability of detection was estimated to be 3 (30 failures per year). Therefore, the probability of detection assumption is valid, as there were 18 reported complaints.
Failure 
S 
O 
D 
RPN 
Risk 
Comments 
L10 Wrong Label 
3 
3 (30) 
3 (30) 
27 
↑ 
Additional mitigations should be implemented. Consider an automated vision inspection system. 
L11 No Label 
3 
3 (30) 
3 (30) 
27 
↑ 
Additional mitigations should be implemented. Consider an automated vision inspection system. 
L12 Illegible Labels 
3 
5 (300) 
3 (30) 
45 
↑ 
Additional mitigations should be implemented. Consider an automated vision inspection system. 
Updated Assumptions 


Figure 2 – Updated traditional FMEA
FMEA Action Priority (AP) Method Example
A team has developed an FMEA for labeling as shown in Figure 3 for labeling with three failure modes. Note the numbers in parentheses are the total number of failures/escapes assumed per year. After one year, the FMEA team collected and summarized the internal nonconformance and external complaint data to validate the initial assumptions of probability of occurrence and probability of detection. The summarized data is shown in Figure 3. See Table 2 for FMEA ratings.
Failure 
S 
O 
D 
AP 
Nonconformances 
Complaints 
Total 
L10 Wrong Label 
Moderate 
Remote (3) 
Slight (300) 
Medium 
2 
9 
11 
L11 No Label 
Moderate 
Remote (3) 
Excellent (3) 
Low 
16 
0 
16 
L12 Illegible Labels 
Moderate 
Moderate (30) 
Adequate (30) 
High 
15 
2 
17 
Original Assumptions 
Actual 
Figure 3 – Action priority (AP) FMEA assumptions
For failure mode L10 Wrong Label, the probability of occurrence was estimated to be remote (three failures per year), but the actual number of failures for the year was 11 (nonconformances + complaints). Therefore, the probability of occurrence must be increased from remote (three failures per year) to moderate (30 failures per year) to reflect the actual data. The probability of detection was estimated to be slight (300 failures per year). Therefore, the probability of detection can be reduced from slight (300 failures per year) to adequate (30 failures per year) to reflect the actual data of nine reported complaints.
For failure mode L11 No Label, the probability of occurrence was estimated to be remote (three failures per year), and the actual number of failures for the year was 16 (nonconformances + complaints). Therefore, the probability of occurrence must be increased from remote (three failures per year) to moderate (30 failures per year) to reflect the actual data. The probability of detection was estimated to be excellent (three failures per year). Therefore, the probability of detection assumption is valid, as there were no reported complaints.
For failure mode L12 Illegible Labels, the probability of occurrence was estimated to be moderate (30 failures per year), and the actual number of failures for the year was 17 (nonconformances + complaints). Therefore, the probability of occurrence assumption is valid. The probability of detection was estimated to be adequate (30 failures per year). Therefore, the probability of detection can be reduced from adequate (30 failures per year) to excellent (three failures per year) to reflect the actual data of two reported complaints.
Failure 
S 
O 
D 
AP 
Risk 
Comments 
L10 Wrong Label 
Moderate 
Moderate (30) 
Adequate (30) 
High 
↑ 
Additional mitigations should be implemented. Consider an automated vision inspection system. 
L11 No Label 
Moderate 
Moderate (30) 
Excellent (3) 
Medium 
↑ 
Additional mitigations should be implemented. Consider an automated vision inspection system. 
L12 Illegible Labels 
Moderate 
Moderate (30) 
Excellent (3) 
Medium 
↓ 
Evaluate ways to reduce risk. 
Updated Assumptions 


Figure 4 – Updated action priority (AP) FMEA assumptions
Conclusion
FMEAs should never be one and done. It is time to ensure that your FMEA process benefits from the use of production and postmarket surveillance data. Using production and postmarket surveillance data can help your organization design and manufacture safer products and demonstrate compliance with international standards, guidances, and regulatory requirements.
Ensure the production and postmarket surveillance data is easy to collect, quantify, and analyze using logical failure modes that are readily identifiable.
I cannot emphasize enough the importance of establishing procedures (documenting) to manage the tools and methods used. Best practice includes providing the rationale for your organization’s use of risk management tools and activities. The requirements and risk management tools presented in this article can and should be utilized based upon industry practice, guidance documents, and regulatory requirements.
References:
 FMEA Handbook 1^{st} Ed, 2019, Automotive Industry Group (AIAG) and the Verband der Automobilindustrie (VDA), Southfield, MI.
 Durivage, Mark A., “Is It Time To Say Goodbye To FMEA Risk Priority Number (RPN) Scores?” https://www.pharmaceuticalonline.com/doc/isittimetosaygoodbyetofmeariskprioritynumberrpnscores0001
Appendices:
Appendix I – Example RPN Action Criteria
RPN 
Risk Acceptability 
Action 
1 to 14 
Low 
Although low risk, continue mitigation process as far as possible 
15 to 29 
Tolerable 
This should only be revisited in future design review if corrective action enhances the reliability or product appeal. 
30 to 49 
Undesirable 
Risk is acceptable only if it cannot be further mitigated by organizational or technological solutions which do not reduce the clinical/functional utility of the product. 
Above 49 
Intolerable 
Risk should be eliminated or reduced by protective measures. Justification required for risk that is accepted. 
Appendix II – Example Criticality/Severity and Occurrence Action Requirements



Criticality/Severity 


Catastrophic 
Serious 
Negligible 


Occurrence 

5 
3 
1 

Certain 
5 
Unacceptable 
Unacceptable 
ALARP 

Occasional 
3 
Unacceptable 
ALARP 
ALARP 

Unlikely 
1 
ALARP 
ALARP 
Acceptable 

*ALARP  As Low as Reasonably Possible
Appendix III – Example Action Priority (AP) Table
Severity Criticality 
Occurrence 
Detection 
Action Priority 
Catastrophic 
Certain 
Slight 
High 
Adequate 
High 

Excellent 
High 

Moderate 
Slight 
High 

Adequate 
High 

Excellent 
High 

Remote 
Slight 
High 

Adequate 
High 

Excellent 
Medium 

Moderate 
Certain 
Slight 
High 
Adequate 
High 

Excellent 
Medium 

Moderate 
Slight 
High 

Adequate 
High 

Excellent 
Medium 

Remote 
Slight 
Medium 

Adequate 
Medium 

Excellent 
Low 

Minor 
Certain 
Slight 
Medium 
Adequate 
Medium 

Excellent 
Low 

Moderate 
Slight 
Low 

Adequate 
Low 

Excellent 
Low 

Remote 
Slight 
Low 

Adequate 
Low 

Excellent 
Low 
About The Author:
Mark Allen Durivage has worked as a practitioner, educator, consultant, and author. He is managing principal consultant at Quality Systems Compliance LLC, an ASQ Fellow and SRE Fellow. Durivage primarily works with companies in the FDA regulated industries (medical devices, human tissue, animal tissue, and pharmaceuticals), focusing on quality management system implementation, integration, updates, and training. Additionally, he assists companies by providing internal and external audit support as well as FDA 483 and warning letter response and remediation services.He earned a BAS in computer aided machining from Siena Heights University and an MS in quality management from Eastern Michigan University. He holds several certifications including CRE, CQE, CQA, CSSBB, RAC (Global), and CTBS. He has written several books available through ASQ Quality Press, published articles in Quality Progress, and is a frequent contributor to Life Science Connect. You can reach him at mark.durivage@qscompliance.com and connect with him on LinkedIn.