Forensic Science Assessments: A Quality and Gap Analysis (July 11, 2017)

This is a “plain language” summary of an up-to-date assessment of the scientific validity of arson investigation. (The full report is available at this link; the assignment is to read pages 11-13.)

Project Findings and Outcomes

The recommendations emerging from this report promise to be influential in the advance of fire and fire debris analysis. The report offers an agenda encouraging the injection of rigorous scientific methods and standards into efforts to better understand the strengths and weaknesses of the current scientific bases of fire investigation. The report can be used by funders, public and private, to develop “calls for proposals” that will generate investigator initiated research into questions touching, directly or indirectly, on fire investigation. This research ideally will come from a broader range of disciplines and institutions than have traditionally been engaged in the study of fires. For example, this and other forensic studies can benefit from contributions by physicists, psychologists, statisticians, and chemists. We also hope that this and future reports, and the increase in funding it may encourage, will lead to more collaborative research by teams of forensic and academic scientists and engineers.

This report should also help other key actors in the criminal justice system—law enforcement, attorneys, and judges—better understand the limits of the current scientific underpinnings of fire analysis. For instance, as judges gain a better understanding of the current state of knowledge regarding fires they will be better able to fulfill their gatekeeping responsibilities, thus bringing a “culture of science” into legal decision-making.

Fire Science

Fire investigation has been the target of well-founded skepticism as relatively newer investigative assumptions and practices are replacing those that historically were accepted “wisdom” in the field. However, not everyone is ready to abandon the standards they have relied on for their entire careers. “Police and fire investigators across the country were often slow to accept advances in fire science that called into question what had long been considered unmistakable evidence of arson” and in “many jurisdictions, those [new] rules were slow to take hold, as veteran investigators clung to what are now considered disproven theories.” In fact, some investigators have been “openly hostile to the updated science” (Mills, 2015).

As advances in fire science are being made, investigators and attorneys are revisiting old cases. What they are finding is chilling. John Lentini (one of the authors of this report) estimates that “a couple hundred” people are wrongly imprisoned because of methods used in fire investigation that have not been empirically validated. According to the National Registry of Exonerations, 63 individuals convicted of arson have already been exonerated since 1991 (http://www.exonerationregistry.org/). Michael Keller, an electrical engineer at a research facility supported by the Bureau of Alcohol, Tobacco, Firearms and Explosives, has investigated three potential capital cases in which the fires were eventually found to be accidental (Bui, 2015). We will never know for sure how many people have been wrongfully convicted based on the aforementioned methods and practices. Likewise, we will never know how many criminal arsonists have not been indicted because of inadequate forensic tools. This report sets forth an evaluation of research on fire scene investigation and fire debris analysis, and we hope, will be 6 read by judges, lawyers for the prosecution and defense, law enforcement officers, laboratory scientists, policymakers, funding agencies, and fire investigation practitioners. It also points to future directions that much-needed research should take.

CONCLUSIONS & RECOMMENDATIONS

FIRE SCENE INVESTIGATION

The conclusions presented in this section of the report are drawn from the two technical sections that follow (fire scene investigation and fire debris analysis). Recommendations are tied to the conclusions. As with the technical sections that follow, the Conclusions and Recommendations are divided between fire scene investigation and fire debris analysis. Each conclusion is accompanied by page numbers to guide the reader to the location in the technical section that supports it. The 25 recommendations can be the basis for further research by fire scientists, as well as scientists from other disciplines in academia.

Origin and Cause Determination

Conclusions

• Fire has been extensively studied, but the complex chemical and physical processes involved are still not fully understood. While much is known about the behavior of fires in building enclosures, using that knowledge to determine where a particular fire started and what caused it remains very challenging. An incorrect determination of a fire’s origin generally leads to an incorrect determination of its cause (pp. 13, 18).
• Determining the origin of a pre-flashover fire is generally a straightforward exercise based on burn pattern analysis. However, determining the origin of post-flashover fires is more difficult because post-flashover fires may create new ventilation-generated burn patterns while obscuring pre-existing burn patterns. Moreover, the longer a fire burns in a fully involved condition, the more difficult is the determination of the correct area of origin (pp. 13, 17, 19).
• The speed with which a fire spreads in residences with contemporary furnishings cannot generally be used alone to classify a fire as accidental or incendiary because fires involving modern upholstered furnishings tend to burn faster than those used in older furnishings, regardless of the cause of the fire (p. 19).
• Computer-based deterministic fire models can be used as one means of testing different hypotheses regarding the origin and development of a fire, but such models cannot generally be used alone to determine the cause of a fire. Uncertainties exist concerning these models when they are applied to fire cause determination (p. 23).

Recommendations

1. To improve the analysis of a fire’s origin and cause, tests should be run in both reduced and full scale, using multiple compartments and multiple openings, fully documenting the aftermath; with the burning of different materials under a range of realistic fire conditions; and by lighting fires in identically constructed compartments. These tests should be scientifically instrumented so that information, such as temperature at various layers of the room and radiant heat fluxes are measured. A major consideration in deciding to conduct this research is the high cost of burning compartment test rooms. However, by using reduced scale testing, the cost can be lowered and may provide the same critical values related to heat as a full-scale test.
2. When a physical fire test is conducted, the fire scenario being tested should also be simulated with a deterministic fire model to evaluate the accuracy of the model and to better understand uncertainties associated with the model. Based on the data obtained through such comparisons, the computer-based deterministic fire models can be continually refined to produce more accurate results, and over time may find an expanded role as a useful tool in actual investigations.

Locating Ignitable Liquid Residues in Fire Debris

Conclusions

• A well-trained canine detection team is the current “gold standard” for locating samples at the fire scene that may test positive for ignitable liquid residues (ILRs) in the fire debris analysis laboratory. Canines are advantageous because they provide immediate feedback and are mobile, allowing them to search a large space in a very short time. However, canine alerts should not be relied upon unless confirmed by laboratory analysis (pp. 18, 30).
• Many substances are produced during a fire, but investigators are currently in a position to look only for evidence of ignitable liquids, for which there is currently validated instrumentation. Other chemical markers are of little value, because there is no way to determine when they may have been created during a fire (p. 23).

Recommendations

3. Testimony that relies on canine alerts only, without supporting laboratory results, should not be used in court proceedings.
4. New technologies, as well as additional training aids and research on new methods need to be developed for measuring canine performance that could enhance their effectiveness.
5. Comparative research assessing the effectiveness of technologically more innovative field tools against the effectiveness of canine use should be a research priority.

Reliability and Validity

Conclusion

• Little is known about the consistency and accuracy of conclusions among experienced investigators when presented with the same data (p. 25).

Recommendations

6. The reliability of conclusions when fire investigators are presented with similar data of fire origin and cause should be studied. This will allow the calculation of both error rates and the reliability of investigators’ conclusions. This exercise should be repeated over several fire scenarios to help determine what types of fire scenes elicit few disagreements and what types elicit many. These tests of reliability will provide feedback on decision points that cause divergent findings among investigators.
7. Not only should the reliability of investigators’ conclusions or diagnoses be established, but research should also be done on the validity of those conclusions or diagnoses. The use of “test fires” (described in recommendation #1) will help to establish a “ground truth” against which the validity of investigators’ conclusions can be assessed. Such tests will also create crucial knowledge about which cues or diagnostics are genuinely associated with various fire characteristics. 8. The data generated by the research on reliability and validity should be incorporated into a database that could be used to develop standards for identifying the origin and cause of fires and serve as a resource for the education and training of fire investigators.

Cognitive Bias

Conclusions

• Interpretation of the evidence regarding origin and cause is often subjective and depends to a significant degree on human cognitive factors (p. 13).
• Evidence from other domains, as well as within forensic science, suggests that there are practical ways to mitigate and minimize bias (such as, Linear Sequential Unmasking). The aim of such procedures is to maximize the independence of mind of forensic examiners to be as bias free as possible in both the identification of relevant evidence and the conclusion about what the evidence shows about a fire’s origin and cause. In this context, the distinction between bias and relevance should be noted. For example, eyewitness testimony although potentially biasing, can also be relevant in some instances. One of the recommendations from the OSAC (Organization of Scientific Area Committees) Fire and Explosion Investigation Subcommittee suggests research in this area (p. 26).

Recommendations

9. Given what is known about the role of cognitive bias in interpretation and decision making, the work by fire scene investigators should be separated from other components of the fire investigation. Those who gather and prepare evidence should focus on scientific analysis and be as neutral as possible in deciding what evidence to collect and how to interpret it. This would help to minimize bias that might affect fire scene investigation. In some jurisdictions, this will be cost prohibitive, but in those jurisdictions where this is not the case, this recommendation should be followed.
10. Case management interventions should be adopted that shield fire scene investigators from information irrelevant, but potentially biasing, for assessing the scientific evidence critical to determining fire origin and cause (see the National Commission on Forensic Science document “Ensuring That Forensic Science Is Based Upon Task-Relevant Information”).
11. There should be policies and procedures that clarify what is or is not relevant for fire investigators to know at each stage of an investigation, in order to reduce the possibility of bias.
12. Forensic laboratories and fire scene investigators’ professional societies should adopt policies and procedures to help implement the recommended changes in case management. Policies and procedures should reflect what is best for helping fire investigators reach accurate scientific conclusions without regard for the convenience of the labs or others associated with the investigation, such as law enforcement.
13. In general scientific practices, bias is often handled by “blinded procedures,” whereby researchers are unaware of information irrelevant to their task. This should be the gold standard for fire scene investigation as well. The literature on this topic discussed in the 10 technical report includes specific suggestions for accomplishing this challenging task in the context of forensic science.
14. The implementation of the recommendations for minimizing the influence of cognitive bias should be accompanied by monitoring and evaluation to assess their impact and, where appropriate, lead to modifications.

Education, Certification, and Experience

Conclusions

• There are insufficient educational and proficiency testing requirements for fire scene investigators (p. 22).
• There is currently no scientific basis for concluding that the accuracy of certified fire investigators, in particular, is better than the accuracy of non-certified fire investigators (p. 22).

Recommendations

15. Education and training for fire investigators should cover the issues of human cognition and cognitive bias, as well as what has been discovered in the reliability and validity testing discussed in recommendations #6 and #7. The training could be done through live training simulations as well as online with photographs and videos or other depictions of test fires in which the origin and cause are known with certainty. Testing materials could employ the same technique with new fire depictions not used in the training materials.
16. The effects of education, training, and certification on fire investigators’ ability to determine fire origin and cause should be further studied.