The PEOPLE, Petitioner,
v.
The SUPERIOR COURT of Los Angeles County, Respondent;
Martell (CHUBBS), Real Party in Interest.

B258569

Filed January 9, 2015

Attorneys and Law Firms

Jackie Lacey, District Attorney, Roberta Schwartz and Matthew Brown, Deputy District Attorneys, for Petitioner.

No appearance for Respondent.

Angelyn Gates for Real Party in Interest.

Opinion

WILLHITE, J.

Real party in interest Martell Chubbs was charged in a November 28, 2012 information with the murder of Shelley H. in 1977 (Pen.Code, § 187, subd. (a)).[FN1] The charge was filed after a DNA sample from the victim was found to be a match for Chubbs. The People petition for a writ of mandate to overturn the order of the superior court compelling the disclosure of a computer source code for software, TrueAllele Casework (TrueAllele), which was used in the DNA analysis. The People contend that the source code is a protected trade secret of the creator and owner of the software, Mark W. Perlin, and his company, Cybergenetics. We grant the petition.

FACTUAL AND PROCEDURAL BACKGROUND

Preliminary Hearing Evidence[FN2]

In December 1977, Long Beach Police Department officers found the 17–year–old victim in her Long Beach apartment. She was lying on the end of the bed with her feet touching the ground and with an electrical wire tied around her neck. During an autopsy, swabs were taken from the victim's vagina and smeared onto slides.

In June 2011, as part of a cold case investigation, Sorenson Forensics (Sorenson) conducted a DNA test on the vaginal swabs from the victim. Sorenson generated a DNA report that indicated three contributors to the DNA: a major sperm DNA profile attributable to an unidentified male, a minor sperm DNA profile, and a partial DNA profile attributable to the victim. Sorenson excluded the victim's husband, Nolan Hankins, as the source of the major sperm DNA profile.[FN3]

Although the record before us does not include the basis for the arrest, Long Beach Police Department detectives arrested Chubbs in August 2012. Chubbs confirmed that he lived in Long Beach in the 1970s.

In September 2012, Sorenson compared the DNA profile of Chubbs, an African–American, to the major sperm DNA profile and found a match. The frequency of the profile occurrence in the general population was determined to be one in approximately 10,000 for African Americans.

At the preliminary hearing in November 2012, Chubbs was held to answer for one count of murder. The information charged Chubbs with one count of murder and alleged six prior convictions of serious felonies (§ 667, subd. (a)(1)) that also qualified as strikes under the Three Strikes law (§§ 667, subds. (b)-(i), 1170.12, subds. (a)-(d)). In January 2013, Chubbs pleaded not guilty to the murder charge.

As part of trial preparation, in September 2013, the People sent the victim's vaginal slide to Cybergenetics' lab in Pittsburgh, Pennsylvania for further testing. Cybergenetics prepared a supplemental report, explaining that it had used its TrueAllele software to “infer possible DNA contributor genotypes from the samples,” then compared the evidence genotypes to the reference genotypes (which included Chubbs' and Hankins' genotypes) to compute likelihood ratio DNA match statistics. “TrueAllele assumed that the evidence sample data ... contained two or three contributors, and objectively inferred evidence genotypes solely from these data.” Perlin concluded in the supplemental report that the DNA match between the vaginal sperm sample and Chubbs is “1.62 quintillion times more probable than a coincidental match to an unrelated Black person.” Perlin also concluded that the DNA match with Hankins was “2.82 million times more probable than a coincidental match to an unrelated Black person.”

Defense Discovery Efforts

In November 2013, Chubbs made his third informal discovery request, which included the request at issue here, for Cybergenetics' source codes for TrueAllele. In January 2014, Chubbs filed a motion to compel discovery that included the request for Cybergenetics' source codes. Defense counsel cited statements in Cybergenetics' supplemental report indicating that TrueAllele made assumptions and inferences in computing its DNA match statistics. According to defense counsel, the TrueAllele program was “brand new” and had not been the subject of a Kelly hearing, and without the source codes there would be no way to cross examine Perlin about the efficacy and accuracy of the program.[FN4]

The defense received several discovery items related to Cybergenetics and TrueAllele, including the following: the September 2013 supplemental report, a November 2013 case packet by Cybergenetics, published articles by Perlin regarding DNA analysis and the TrueAllele software, a data disc from Sorenson, TrueAllele manuals from March 2014, a data disc from Cybergenetics, and a PowerPoint presentation to be used by Perlin. However, the dispute here focuses on the source codes for TrueAllele, which were not produced.

On January 15, 2014, the People filed an opposition to the motion to compel discovery, arguing that the defense was not entitled to a discovery order because the People had voluntarily complied with their discovery obligations, citing section 1054.5, subdivision (a).[FN5] As pertinent here, the People explained that they requested the source code from Cybergenetics, but Cybergenetics did not turn it over because it is a trade secret. The People argued that disclosure of the source code would be “financially devastating” to Cybergenetics.

The People stated in their opposition that, although Cybergenetics is unwilling to disclose its source code, it “is willing to conduct additional TrueAllele testing on a limited set of defense-provided data to further defense understanding of the system, its operation and its reliability. Cybergenetics is also willing to meet with defense experts (in person or via an Internet meeting) to show them the results in this case, and explain to them on a TrueAllele computer how the system operates, though Cybergenetics cannot provide [the] defense with a[n] executable version of the TrueAllele casework system which costs $60,000.”

Chubbs then filed an application for an out-of-state subpoena duces tecum, seeking the source codes for the TrueAllele software. He argued that the source codes were essential to his defense because the DNA evidence from the vaginal slide was the only evidence against him. He pointed out the discrepancy between the random match probability calculated by Sorenson (1 in approximately 10,000) and the likelihood ratio calculated by Cybergenetics (1.62 quintillion times more probable than a coincidental match) to argue that the source codes were necessary to cross-examine Perlin about the accuracy of TrueAllele.

In a declaration submitted with the application, defense counsel stated that forensic experts and other attorneys who work with DNA evidence advised her to obtain the source codes for TrueAllele. She stated that “other experts in the field have developed a similar software program as TrueAllele for which their source codes are open for public review.” Defense counsel further stated that Allan Jamieson, an expert in DNA analysis who had experience with TrueAllele, told her that she could not properly defend against the TrueAllele results without the source codes.

Jamieson stated in his declaration that “access to this code is the only satisfactory and professionally recommended way to fully consider the validity of the TrueAllele analysis” in this case. He stated that “[o]ther analysts who have developed computer-assisted DNA comparison software ... do not hide their source codes” and instead make them freely available, which allows others to fully review and verify the reliability of the method and results in any given case.

Motion to Quash

On May 16, 2014, the People filed a motion to quash the subpoena duces tecum. Contrary to its earlier argument in its opposition to the motion to compel discovery, the People now argued that Cybergenetics was not a third party to the investigation but instead was an investigatory agency within the meaning of section 1054.5, subdivision (a).

The trial court denied the People's motion to quash and issued a certificate for an out-of-state subpoena, ordering Perlin to produce the source codes. On June 16, 2014, a Pennsylvania court issued an order directing compliance with the subpoena duces tecum. The Pennsylvania court reasoned that Perlin was a material witness, and the means by which he arrived at his opinions likewise was material. The court thus ordered Perlin to appear with the source codes and stated that any issue regarding the disclosure of trade secrets should be determined by the California court.

The People moved to quash the subpoena duces tecum. The People argued that the materials are a trade secret, that Chubbs has not established the source codes are material or necessary, and that the discovery is not permitted by section 1054, subdivision (e).[FN6]

On June 24, 2014, the trial court issued and held a body attachment for Perlin based on his failure to appear pursuant to the subpoena duces tecum. The People subsequently withdrew the contention that Perlin was an expert employed by the prosecution pursuant to subdivision (a) of section 1054.5, noting that Perlin had retained private counsel regarding the trade secret privilege.

Perlin, represented by California counsel, submitted a brief in support of his assertion of the trade secret privilege. The People filed a motion to reconsider the court's May 16 order denying the People's motion to quash the subpoena duces tecum, and for an order granting the motion and quashing the body attachment held for Perlin.

At a July 29, 2014 hearing, the court ruled the source codes are not necessary pursuant to Kelly/Frye, but that Chubbs' right to confront and cross-examine witnesses required the production of the source codes. The prosecutor again invoked the trade secret privilege on Perlin's behalf. The court found that nondisclosure of the source codes does “work injustice” in the sense that it denies Chubbs a right to confront and cross-examine witnesses (Evid.Code, § 1060), and that a protective order can protect Perlin's interest. The court indicated that it would follow the procedure set forth in Evidence Code sections 1061 and 1062, by issuing a protective order and, if needed, excluding the public from the proceedings. The court held the body attachment for Perlin until August 26 and ordered the prosecution to provide the source codes on that date.

On August 26, 2014, the court deemed the TrueAllele source code a trade secret for purposes of the trial. Perlin brought an encrypted form of the source code. However, before turning over the source code, the prosecution raised the issue of a protective order. The court explained that although it would grant a protective order to minimize disclosure of the source code, the source code would be revealed to a certain extent at trial. The People subsequently did not proffer a protective order, but instead refused to turn over the source code. Defense counsel requested the exclusion of the TrueAllele results at trial. The court granted the request based on Chubbs' rights under the confrontation clause and the fact Perlin was to be a main prosecution witness against Chubb.

The People petitioned for a writ of mandate to this court. We issued an alternative writ of mandate ordering the superior court to vacate the July 29 and August 26, 2014 orders compelling the disclosure of the computer source codes, or to show cause why a peremptory writ of mandate should not issue.[FN7] The superior court did not vacate its ruling, and the matter is now before us.

DISCUSSION

The People contend that the trial court improperly applied the trade secret privilege and that Chubbs failed to make a prima facie showing sufficient to overcome the privilege.[FN8] “The court's ruling on a discovery motion is subject to review for abuse of discretion. [Citation.]” (People v. Jenkins (2000) 22 Cal.4th 900, 953.) “A trial court has abused its discretion in determining the applicability of a privilege when it utilizes the wrong legal standards to resolve the particular issue presented. [Citation.]” (Seahaus La Jolla Owners Assn. v. Superior Court (2014) 224 Cal.App.4th 754, 766.)

We begin by setting forth the statutes and law regarding the trade secret privilege.

I. Trade Secret Privilege

Evidence Code section 1060 provides: “If he or his agent or employee claims the privilege, the owner of a trade secret has a privilege to refuse to disclose the secret, and to prevent another from disclosing it, if the allowance of the privilege will not tend to conceal fraud or otherwise work injustice.” In the instant case, it is undisputed that the source codes in issue constitute a trade secret. (See Evid.Code, § 1062, subd. (a) [for purposes of Evidence Code sections 1061, 1062, and 1063, which apply to criminal cases, “[t]rade secret” is defined in Civil Code section 3426.1 or Penal Code section 499c, subdivision (a)(9) ].)

In civil cases, a burden-shifting procedure is used to evaluate assertion of the trade secret privilege. Based on the language and legislative history of Evidence Code section 1060, the court in Bridgestone/Firestone, Inc. v. Superior Court (1992) 7 Cal.App.4th 1384 (Bridgestone) held that “the party claiming the [trade secret] privilege has the burden of establishing its existence. [Citations.] Thereafter, the party seeking discovery must make a prima facie, particularized showing that the information sought is relevant and necessary to the proof of, or defense against, a material element of one or more causes of action presented in the case, and that it is reasonable to conclude that the information sought is essential to a fair resolution of the lawsuit. It is then up to the holder of the privilege to demonstrate any claimed disadvantages of a protective order.” (Id. at p. 1393; see also Raymond Handling Concepts Corp. v. Superior Court (1995) 39 Cal.App.4th 584, 590–591 [relying on the procedure enunciated in Bridgestone and concluding that the information was discoverable and that the trial court did not abuse its discretion in entering a protective order].)

Chubbs contends that when a defendant in a criminal case seeks disclosure of an item meeting the definition of a trade secret, Evidence Code section 1060 does not permit the owner of the trade secret to refuse to disclose. Rather, according to Chubbs, Evidence Code sections 1061 and 1062 supersede section 1060, and authorize only (on a proper showing) the remedy of a protective order (§ 1062) and exclusion of the public from portions of the trial at which a trade secret might be revealed (§ 1062). Evidence Code section 1061 provides: “In addition to Section 1062, the following procedure shall apply whenever the owner of a trade secret wishes to assert his or her trade secret privilege, as provided in Section 1060, during a criminal proceeding.” (Evid.Code, § 1061, subd. (b).) The statute then sets forth a procedure under which the holder of the trade secret privilege or an authorized representative may move for a protective order (Evid.Code, § 1061, subd. (b)(1)), any party to the proceeding may oppose the motion (id., subd. (b)(2)), and the court, on a finding “that a trade secret may be disclosed ... unless a protective order is issued and that the issuance of a protective order would not conceal a fraud or work an injustice, ... issue[s] a protective order limiting the use and dissemination of the trade secret” (id., subd. (b)(4)).[FN9] Evidence Code section 1062 similarly provides a procedure under which “the court, upon motion of the owner of a trade secret, or upon motion by the People with the consent of the owner, may exclude the public from any portion of a criminal proceeding where the proponent of closure has demonstrated a substantial probability that the trade secret would otherwise be disclosed to the public during that proceeding and a substantial probability that the disclosure would cause serious harm to the owner of the secret, and where the court finds that there is no overriding public interest in an open proceeding. No evidence, however, shall be excluded during a criminal proceeding pursuant to this section if it would conceal a fraud, work an injustice, or deprive the People or the defendant of a fair trial.” (Evid.Code, § 1062, subd. (a).)[FN10]

Although Chubbs does not expressly acknowledge it, an implicit premise of his contention seeking disclosure of the source codes is that a criminal defendant need not make a prima facie showing of the relevance and necessity of the trade secret before disclosure occurs. Rather, upon a defense request for material that qualifies as a trade secret, the holder of the trade secret privilege cannot object on the ground that no showing of relevance and necessity has been made. To the contrary, the privilege holder's only remedies, even for material as to which there is no relevance and necessity, are to seek a protective order limiting the terms of disclosure (but not precluding disclosure) under Evidence Code section 1061, and closing the proceedings at which the trade secret might be disclosed under Evidence Code section 1062.

We decline to read Evidence Code sections 1061 and 1062 in such a manner. In short, it makes no sense to require the holder of a trade secret privilege to submit to disclosure of the trade secret, even subject to a protective order and the closing of certain proceedings, without a showing that the trade secret is relevant and necessary to the defense. (See People v. Superior Court (Barrett ) (2000) 80 Cal.App.4th 1305, 1318 [“A criminal defendant has a right to discovery by a subpoena duces tecum of third party records on a showing of good cause—that is, specific facts justifying discovery.”].) We thus conclude that the test for trade secret disclosure adopted in Bridgestone —a prima facie, particularized showing that the source code is relevant and necessary to the defense—is required for Chubbs to require disclosure of the source codes.

The trial court here correctly began with a determination under Evidence Code section 1060 that the source code is a trade secret and then moved to the issue of a protective order pursuant to Evidence Code section 1061. However, because we find that Chubbs did not meet his prima facie burden for disclosure, we conclude that the People should not have been compelled to produce the source code, whether or not subject to a protective order.

II. Chubbs' Evidence Regarding Necessity of Source Code

Chubbs submitted declarations from defense counsel and Jamieson to support his contention that the source code is essential to his defense.[FN11] Defense counsel relied on the fact that the DNA evidence was the only evidence connecting Chubbs to the victim.

Defense counsel further declared that without the source code, “there is no way for my expert to determine what assumptions, among other things, have been made and if they are appropriate in this particular case.” In her application for the subpoena duces tecum, she declared that her DNA experts, another forensic DNA consultant (who has helped develop a program similar to TrueAllele with source codes open for public review), and unidentified attorneys “who focus on DNA evidence regarding TrueAllele” advised her to request the source codes and pseudo source codes.

In Jamieson's declaration, he claimed ten years of forensic experience, as well as familiarity with TrueAllele's “claimed methodology and use” and “experience” in court with TrueAllele. He opined that “access to this code is the only satisfactory and professionally recommended way to fully consider the validity of the TrueAllele analysis” in this case. He claimed that others who have developed computer-assisted DNA comparison software “do not hide their source codes” and instead make them freely available, which allows others to fully review and verify the reliability of the method and results in any given case.

In considering whether Chubbs has made a prima facie showing of the necessity of the source code to his defense, we consider not only Chubbs' evidence but also Perlin's declaration, which was submitted in support of the People's motion to quash the subpoena duces tecum. (See Bridgestone, supra, 7 Cal.App.4th at p. 1395 [“while the burden of making a prima facie showing of the particularized need for a trade secret is on the party seeking discovery, the trial court need not ignore evidence presented by the opposing party on the question whether the information sought is a trade secret”].)

Perlin explained that TrueAllele is useful when uncertainty in DNA analysis arises, such as when two or more people contribute to the evidence, and it decreases uncertainty by comparing information to a suspect. TrueAllele is “Cybergenetics' computer implementation of [a] two-step DNA identification inference approach.” This process involves, first, “objectively inferring genotypes from evidence data, accounting for allele pair uncertainty using probability,” and “subsequently matching genotypes, comparing evidence with a suspect relative to a population, to express the strength of association using probability.”

Perlin declared that TrueAllele is widely accepted, having been used in approximately 200 criminal cases in courts in California, Pennsylvania, and Virginia, and it has been subjected to numerous validation studies, five of which were published in peer-reviewed scientific journals.

Perlin explained that software source code is the programming language used to write a computer program. The source code “details step-by-step human-readable instructions that describe to the computer and programmers how the program operates,” and is “translated into computer-readable ‘executable’ software.” He stated that TrueAllele has about 170,000 lines of computer source code and opined that reading through the source code would not yield meaningful information.

As to the proprietary nature of the source code, Perlin explained that others “can easily copy a computer program if they have its source code,” which “contains the software design, engineering know-how, and the algorithmic implementation of the entire computer program.” Cybergenetics has invested millions of dollars over 20 years to develop TrueAllele, which it offers to crime labs for a base license fee of $60,000.

Perlin differentiated TrueAllele from the open source DNA analysis software programs referenced in the declarations of defense counsel and Jamieson, stating that open source programs “typically are not validated prior to release, because the process of perfecting software is costly.” In addition, open source forensic programs “tend to be relatively short programs consisting of several hundreds of lines of code,” in contrast to the 170,000 lines of code in TrueAllele.

Cybergenetics accordingly has never disclosed the source code to anyone outside the company and does not distribute it to businesses or government agencies that license the software. Cybergenetics does, however, disclose TrueAllele's methodology and its “underlying mathematical model” to enable others to understand its genotype modeling mechanism. The company “provides opposing experts the opportunity to review the TrueAllele process, examine results, and ask questions.”

Cybergenetics keeps the source code secret because of the “highly competitive commercial environment” in which it operates. Perlin declared that Cybergenetics' competitors are interested in replicating TrueAllele and that disclosure of the source code would enable its competitors to copy the product, causing the company irreparable harm. Perlin believed that source code is not revealed for other commercial forensic DNA software because the source code is not needed to assess the software programs' reliability.

Jamieson's general statement that a criminal defendant cannot “receive a diligent and fair verification of a DNA testing or analysis method” without the source codes does not address Perlin's explanations of what the source code actually is and why it is not needed to test the methodology or reliability of TrueAllele's analysis. Jamieson also generally states that access to the source code is the only way to consider the validity of the TrueAllele analysis in Chubbs' case, but he does not explain how access to the source code would allow him to test the reliability of TrueAllele's analysis. (See Bridgestone, supra, 7 Cal.App.4th at p. 1396 [“nowhere did [the real parties' expert] describe with any precision how or why the [trade secret] formulas were a predicate to his ability to reach conclusions in the case”].)

Similarly, defense counsel generally states in her declaration that others have told her she needs to request the source codes and that there is “no way [she] can properly prepare to defend against the TrueAllele results without the source codes and pseudo source codes.” However, these general declarations do not address why the source code is needed to review the reliability of the TrueAllele analysis, how the source code would be used to review the TrueAllele results, or what could be revealed by the source codes that would be useful to Chubbs' defense. Indeed, her declarations regarding TrueAllele's methodology, inferences, and reliance on the likelihood ratio rather than the random match probability illustrate her understanding of TrueAllele and thus undercut her argument that the source codes are necessary to understanding TrueAllele. This is particularly true in light of the fact that defense counsel received the patent documents regarding TrueAllele, numerous published articles regarding TrueAllele, and TrueAllele operating manuals. Further supporting the position that the source code is not necessary to an understanding of TrueAllele is Perlin's statement in his declaration that Cybergenetics discloses to opposing experts TrueAllele's methodology, how it applies its method to the data, and how the software works. The supplemental report prepared by Cybergenetics also explained the assumptions made by TrueAllele in its analysis. The vague statements by defense counsel and Jamieson do not describe in any way how the source code would have any bearing on the reliability of the analysis.

In his declaration, Perlin cited Commonwealth v. Foley (Pa.Super.2012) 38 A.3d 882, in which the Superior Court of Pennsylvania held that the trial court did not abuse its discretion in admitting Perlin's DNA-related testimony. (Id. at p. 890.) Although this out-of-state case does not carry precedential weight, we agree with its conclusion that access to TrueAllele's source code is not necessary to judge the software's reliability. Similar to Chubbs' case, Perlin's estimate of the probability of a DNA match to the defendant in Foley was much higher (1 in 189 billion) than the estimates of the other scientific experts (1 in 13,000 and 1 in 23 million). (See id. at p. 887.) As pertinent here, the Pennsylvania court rejected the defendant's argument that Perlin's testimony should have been excluded, reasoning that “scientists can validate the reliability of a computerized process even if the ‘source code’ underlying that process is not available to the public. TrueAllele is proprietary software; it would not be possible to market TrueAllele if it were available for free. [Citation.]” (Id. at p. 889.) The court further reasoned that TrueAllele “has been tested and validated in peer-reviewed studies,” citing several papers that “were published in peer-reviewed journals” and thus “reviewed by other scholars in the field.” (Id. at pp. 889–890.)

“[I]t is not enough that a trade secret might be useful to real parties.” (Bridgestone, supra, 7 Cal.App.4th at p. 1395.) Instead, “the party seeking discovery must make a prima facie, particularized showing that the information sought is relevant and necessary to the proof of, or defense against, a material element of one or more causes of action presented in the case, and that it is reasonable to conclude that the information sought is essential to a fair resolution of the lawsuit.” (Id. at p. 1393.) Chubbs has received extensive information regarding TrueAllele's methodology and underlying assumptions, but he has not demonstrated how TrueAllele's source code is necessary to his ability to test the reliability of its results. We therefore conclude that Chubbs has not made a prima facie showing of the particularized need for TrueAllele's source code.

III. Right to Confront Witnesses

The trial court relied on Chubbs' constitutional right to confrontation to conclude that the People were required to produce the source code. However, our state supreme court has stated, “invocation of the confrontation or compulsory process clauses in a claim involving pretrial discovery ‘is on weak footing’ because it is unclear whether or to what extent those constitutional guarantees grant pretrial discovery rights to a defendant. [Citations.]” (People v. Clark (2011) 52 Cal.4th 856, 982–983; see also People v. Hammon (1997) 15 Cal.4th 1117, 1126 [examining United States Supreme Court precedent and concluding, “it is not at all clear ‘whether or to what extent the confrontation or compulsory process clauses of the Sixth Amendment grant pretrial discovery rights to the accused’ ”] (Hammon ).)

“In [Hammon ], the Supreme Court held the trial court properly quashed a subpoena duces tecum the defendant served on psychotherapists treating the alleged victim without first conducting an in camera review of the material. ‘[R]eject[ing the] defendant's claim that pretrial access to such information was necessary to vindicate his federal constitutional rights to confront and cross-examine the complaining witness at trial or to receive a fair trial’ [citation], Hammon held ‘the trial court was not required, at the pretrial stage of the proceedings, to review or grant discovery of privileged information in the hands of third party psychotherapy providers’ [citation].” (People v. Petronella (2013) 218 Cal.App.4th 945, 958 (Petronella ).)

Hammon reasoned that United States Supreme Court precedent addressing a criminal defendant's right under the confrontation clause to information protected by state-created evidentiary privileges applied to a defendant's trial rights, not pretrial rights. (Hammon, supra, 15 Cal.4th at pp. 1123–1127.) The court further reasoned that, “[w]hen a defendant proposes to impeach a critical prosecution witness with questions that call for privileged information, the trial court may be called upon ... to balance the defendant's need for cross-examination and the state policies the privilege is intended to serve. [Citation.] Before trial, the court typically will not have sufficient information to conduct this inquiry; hence, if pretrial disclosure is permitted, a serious risk arises that privileged material will be disclosed unnecessarily.” (Id. at p. 1127.) The court thus “decline[d] to extend the defendant's Sixth Amendment rights of confrontation and cross-examination to authorize pretrial disclosure of privileged information.” (Id. at p. 1128.)

Similarly, Petronella concluded that the trial court's pretrial ruling upholding a privilege claim against the defendant's subpoena did not violate the defendant's constitutional rights to confrontation and due process. (Petronella, supra, 218 Cal.App.4th at pp. 958–959.) Pursuant to Hammon and Petronella, we conclude that Chubbs' right to confrontation does not apply to pretrial discovery of the source code, which is privileged information.

Chubbs relies on the concurring and dissenting opinions in Pennsylvania v. Ritchie (1987) 480 U.S. 39 (Ritchie ) to argue that the confrontation clause applies to pretrial discovery. However, Hammon specifically addressed Ritchie in concluding that the Sixth Amendment right to confrontation did not confer a right to discover privileged information before trial. (Hammon, supra, 15 Cal.4th at pp. 1125–1127.) We therefore conclude that the trial court abused its discretion in relying on the confrontation clause to order disclosure of the TrueAllele source codes.

DISPOSITION

Let a peremptory writ of mandate issue directing respondent court to vacate its order compelling disclosure of the source code, and to issue a new order denying the motion to compel discovery.

Footnotes

[FN1] All unspecified statutory references are to the Penal Code.

[FN2] The People have not included the transcript of the preliminary hearing, instead relying on a declaration from the deputy district attorney who appeared at the preliminary hearing, summarizing the evidence.

[FN3] Hankins is sometimes referred to in the record as the victim's boyfriend, rather than her husband.

[FN4] The three-pronged test established in People v. Kelly (1976) 17 Cal.3d 24 “provides a framework within which courts can analyze the reliability of expert testimony based on new or novel scientific methods or techniques.” (People v. Lucas (2014) 60 Cal.4th 153, 223.) “ ‘The first prong requires proof that the technique is generally accepted as reliable in the relevant scientific community. [Citation.] The second prong requires proof that the witness testifying about the technique and its application is a properly qualified expert on the subject. [Citation.] The third prong requires proof that the person performing the test in the particular case used correct scientific procedures.’ [Citation.]” (Id. at p. 223, fn. 31.) The test is also known as the Kelly/Frye test. (In re Jordan R. (2012) 205 Cal.App.4th 111, 115, fn. 3; see Frye v. U.S. (D.C.Cir.1923) 293 F. 1013.)

[FN5] The statute provides: “No order requiring discovery shall be made in criminal cases except as provided in this chapter. This chapter shall be the only means by which the defendant may compel the disclosure or production of information from prosecuting attorneys, law enforcement agencies which investigated or prepared the case against the defendant, or any other persons or agencies which the prosecuting attorney or investigating agency may have employed to assist them in performing their duties.” (§ 1054.5, subd. (a).)

[FN6] Section 1054 states that the chapter on discovery “shall be interpreted to give effect to all of the following purposes,” including “[t]o provide that no discovery shall occur in criminal cases except as provided by this chapter, other express statutory provisions, or as mandated by the Constitution of the United States.” (§ 1054, subd. (e).)

[FN7] We further issued a temporary stay of the trial.

[FN8] We disagree with Chubbs' contention that the trial court's ruling was an evidentiary ruling not subject to writ review. Although the trial court's ultimate ruling was to exclude the TrueAllele evidence, this was based on the People's refusal to disclose the source codes. “ ‘Extraordinary review of a discovery order will be granted when a ruling threatens immediate harm, such as loss of a privilege against disclosure, for which there is no other adequate remedy. [Citation.] ... “ ‘[W]here the petitioner seeks relief from a discovery order that may undermine a privilege, we review the trial court's order by way of extraordinary writ. [Citation.]’ ” [Citation.]’ [Citation.]” (Doe v. Superior Court (2011) 194 Cal.App.4th 750, 754; see also § 1512, subd. (a) [authorizing the people to seek review of an order granting a defendant's motion for discovery by a petition for a writ of mandate]; People v. Superior Court (Mouchaourab ) (2000) 78 Cal.App.4th 403, 413 [“writ review is appropriate when the petitioner ‘seeks relief from a discovery order which may undermine a privilege, because appellate remedies are not adequate once the privileged information has been disclosed’ ”].)

[FN9] Evidence Code section 1061 provides in relevant part: “(b) In addition to Section 1062, the following procedure shall apply whenever the owner of a trade secret wishes to assert his or her trade secret privilege, as provided in Section 1060, during a criminal proceeding:

“(1) The owner of the trade secret shall file a motion for a protective order, or the people may file the motion on the owner's behalf and with the owner's permission. The motion shall include an affidavit based upon personal knowledge listing the affiant's qualifications to give an opinion concerning the trade secret at issue, identifying, without revealing, the alleged trade secret and articles which disclose the secret, and presenting evidence that the secret qualifies as a trade secret under either subdivision (d) of Section 3426.1 of the Civil Code or paragraph (9) of subdivision (a) of Section 499c of the Penal Code. The motion and affidavit shall be served on all parties in the proceeding.

“(2) Any party in the proceeding may oppose the request for the protective order by submitting affidavits based upon the affiant's personal knowledge. The affidavits shall be filed under seal, but shall be provided to the owner of the trade secret and to all parties in the proceeding. Neither the owner of the trade secret nor any party in the proceeding may disclose the affidavit to persons other than to counsel of record without prior court approval.

“(3) The movant shall, by a preponderance of the evidence, show that the issuance of a protective order is proper. The court may rule on the request without holding an evidentiary hearing. However, in its discretion, the court may choose to hold an in camera evidentiary hearing concerning disputed articles with only the owner of the trade secret, the people's representative, the defendant, and defendant's counsel present. If the court holds such a hearing, the parties' right to examine witnesses shall not be used to obtain discovery, but shall be directed solely toward the question of whether the alleged trade secret qualifies for protection.

“(4) If the court finds that a trade secret may be disclosed during any criminal proceeding unless a protective order is issued and that the issuance of a protective order would not conceal a fraud or work an injustice, the court shall issue a protective order limiting the use and dissemination of the trade secret, including, but not limited to, articles disclosing that secret. The protective order may, in the court's discretion, include the following provisions:

“(A) That the trade secret may be disseminated only to counsel for the parties, including their associate attorneys, paralegals, and investigators, and to law enforcement officials or clerical officials.

“(B) That the defendant may view the secret only in the presence of his or her counsel, or if not in the presence of his or her counsel, at counsel's offices.

“(C) That any party seeking to show the trade secret, or articles containing the trade secret, to any person not designated by the protective order shall first obtain court approval to do so:

“(i) The court may require that the person receiving the trade secret do so only in the presence of counsel for the party requesting approval.

“(ii) The court may require the person receiving the trade secret to sign a copy of the protective order and to agree to be bound by its terms. The order may include a provision recognizing the owner of the trade secret to be a third-party beneficiary of that agreement.

“(iii) The court may require a party seeking disclosure to an expert to provide that expert's name, employment history, and any other relevant information to the court for examination. The court shall accept that information under seal, and the information shall not be disclosed by any court except upon termination of the action and upon a showing of good cause to believe the secret has been disseminated by a court-approved expert. The court shall evaluate the expert and determine whether the expert poses a discernible risk of disclosure. The court shall withhold approval if the expert's economic interests place the expert in a competitive position with the victim, unless no other experts are available. The court may interview the expert in camera in aid of its ruling. If the court rejects the expert, it shall state its reasons for doing so on the record and a transcript of those reasons shall be prepared and sealed.

“(D) That no articles disclosing the trade secret shall be filed or otherwise made a part of the court record available to the public without approval of the court and prior notice to the owner of the secret. The owner of the secret may give either party permission to accept the notice on the owner's behalf.

“(E) Other orders as the court deems necessary to protect the integrity of the trade secret.

“(c) A ruling granting or denying a motion for a protective order filed pursuant to subdivision (b) shall not be construed as a determination that the alleged trade secret is or is not a trade secret as defined by subdivision (d) of Section 3426.1 of the Civil Code or paragraph (9) of subdivision (a) of Section 499c of the Penal Code. Such a ruling shall not have any effect on any civil litigation.”

[FN10] Section 1062 provides in full:

“(a) Notwithstanding any other provision of law, in a criminal case, the court, upon motion of the owner of a trade secret, or upon motion by the People with the consent of the owner, may exclude the public from any portion of a criminal proceeding where the proponent of closure has demonstrated a substantial probability that the trade secret would otherwise be disclosed to the public during that proceeding and a substantial probability that the disclosure would cause serious harm to the owner of the secret, and where the court finds that there is no overriding public interest in an open proceeding. No evidence, however, shall be excluded during a criminal proceeding pursuant to this section if it would conceal a fraud, work an injustice, or deprive the People or the defendant of a fair trial.

“(b) The motion made pursuant to subdivision (a) shall identify, without revealing, the trade secrets which would otherwise be disclosed to the public. A showing made pursuant to subdivision (a) shall be made during an in camera hearing with only the owner of the trade secret, the People's representative, the defendant, and defendant's counsel present. A court reporter shall be present during the hearing. Any transcription of the proceedings at the in camera hearing, as well as any articles presented at that hearing, shall be ordered sealed by the court and only a court may allow access to its contents upon a showing of good cause. The court, in ruling upon the motion made pursuant to subdivision (a), may consider testimony presented or affidavits filed in any proceeding held in that action.

“(c) If, after the in camera hearing described in subdivision (b), the court determines that exclusion of trade secret information from the public is appropriate, the court shall close only that portion of the criminal proceeding necessary to prevent disclosure of the trade secret. Before granting the motion, however, the court shall find and state for the record that the moving party has met its burden pursuant to subdivision (b), and that the closure of that portion of the proceeding will not deprive the People or the defendant of a fair trial.

“(d) The owner of the trade secret, the People, or the defendant may seek relief from a ruling denying or granting closure by petitioning a higher court for extraordinary relief.

“(e) Whenever the court closes a portion of a criminal proceeding pursuant to this section, a transcript of that closed proceeding shall be made available to the public as soon as practicable. The court shall redact any information qualifying as a trade secret before making that transcript available.

“(f) The court, subject to Section 867 of the Penal Code, may allow witnesses who are bound by a protective order entered in the criminal proceeding protecting trade secrets, pursuant to Section 1061, to remain within the courtroom during the closed portion of the proceeding.”

[FN11] Chubbs attached to his return a declaration from Dr. Travis Doom regarding the necessity of the source codes. We decline to consider the declaration because it was not submitted to the trial court. (See Pomona Valley Hospital Medical Center v. Superior Court (2013) 213 Cal.App.4th 828, 835, fn. 5 [“Writ review does not provide for consideration of evidence not before respondent court at the time of its ruling.”].)

STATE of New Jersey, Plaintiff-Respondent,
v.
Corey PICKETT, Defendant-Appellant.

DOCKET NO. A-4207-19T4

Argued January 19, 2021
Decided February 3, 2021

Attorneys and Law Firms

Tamar Y. Lerer, Assistant Deputy Public Defender, argued the cause for appellant (Joseph E. Krakora, Public Defender, attorney; Tamar Y. Lerer, of counsel and on the briefs).

Stephanie Davis Elson, Assistant Prosecutor, argued the cause for respondent (Esther Suarez, Hudson County Prosecutor, attorney; Stephanie Davis Elson, of counsel and on the briefs).

Amanda G. Schwartz, Deputy Attorney General, argued the cause for amicus curiae Attorney General of New Jersey (Gurbir S. Grewal, Attorney General, attorney; Amanda G. Schwartz, of counsel and on the brief).

Karen Thompson argued the cause for amicus curiae American Civil Liberties Union of New Jersey and Electronic Frontier Foundation (American Civil Liberties Union of New Jersey, Kit Walsh (Electronic Frontier Foundation) of the California and Massachusetts bars, admitted pro hac vice, and Hannah Zhao (Electronic Frontier Foundation) of the New York bar, admitted pro hac vice, attorneys; Karen Thompson, Alexander Shalom, Jeanne LoCicero, Kit Walsh and Hannah Zhao, on the joint brief).

Christopher D. Adams, Holmdel, argued the cause for amicus curiae The Association of Criminal Defense Lawyers of New Jersey (Greenbaum, Rowe, Smith & Davis LLP, attorneys; Christopher D. Adams, of counsel and on the brief; Abdus-Sami M. Jameel, on the brief).

Dana M. Delger (Innocence Project Inc.) of the New York bar, New York, NY admitted pro hac vice, argued the cause for amicus curiae The Innocence Project (Dana M. Delger (Innocence Project Inc.) of the New York bar, admitted pro hac vice, Mazraani & Liguori, LLP, Michael A. Albert (Wolf, Greenfield & Sacks, P.C.) of the Massachusetts bar, admitted pro hac vice, and Anant K. Saraswat (Wolf, Greenfield & Sacks, P.C.) of the Massachusetts bar, Boston, MA admitted pro hac vice, attorneys; Dana M. Delger, Joseph M. Mazraani, North Brunswick, Michael A. Albert and Anant K. Saraswat, on the brief).

Dino L. LaVerghetta (Sidley Austin LLP), of the District of Columbia and New York bars, Washington, DC, admitted pro hac vice, argued the cause for amici curiae Drs. Mats Heimdahl and Jeanna Matthews (Coughlin Duffy LLP, Dino L. LaVerghetta, (Sidley Austin LLP) of the District of Columbia and Virginia bars, admitted pro hac vice, and Iain C. Armstrong (Sidley Austin LLP) of the District of Columbia and Virginia bars, admitted pro hac vice, attorneys; Dino L. LaVerghetta, Iain C. Armstrong, Washington, DC, Matthew Hopkins, Morristown, and Mark K. Silver, Washington, DC, on the brief).

J. David Pollock, attorney for amicus curiae The Legal Aid Society.

Singer & Fedun, LLC and Kendra K. Albert (Cyberlaw Clinic, Harvard Law School) of the Massachusetts bar, admitted pro hac vice, attorneys for amicus curiae Upturn, Inc. (William Singer and Kendra K. Albert, on the brief).

Before Judges Fasciale, Rothstadt and Susswein.

Opinion

The opinion of the court was delivered by

FASCIALE, P.J.A.D.

In this case of first impression addressing the proliferation of forensic evidentiary technology in criminal prosecutions, we must determine whether defendant is entitled to trade secrets of a private company for the sole purpose of challenging at a Frye1 hearing the reliability of the science underlying novel DNA analysis software and expert testimony. At the hearing, the State produced an expert who relied on his company's complex probabilistic genotyping software program to testify that defendant's DNA was present, thereby connecting defendant to a murder and other crimes. Before cross-examination of the expert, the judge denied defendant access to the trade secrets, which include the software's source code and related documentation.

This is the first appeal in New Jersey addressing the science underlying the proffered testimony by the State's expert, who designed, utilized, and relied upon TrueAllele, the program at issue. TrueAllele is technology not yet used or tested in New Jersey; it is designed to address intricate interpretational challenges of testing low levels or complex mixtures of DNA. TrueAllele's computer software utilizes and implements an elaborate mathematical model to estimate the statistical probability that a particular individual's DNA is consistent with data from a given sample, as compared with genetic material from another, unrelated individual from the broader relevant population. For this reason, TrueAllele, and other probabilistic genotyping software, marks a profound shift in DNA forensics.

TrueAllele's software integrates multiple scientific disciplines. At issue here—in determining the reliability of TrueAllele—is whether defendant is entitled to the trade secrets to cross-examine the State's expert at the Frye hearing to challenge whether his testimony has gained general acceptance within the computer science community, which is one of the disciplines. The defense expert's access to the proprietary information is directly relevant to that question and would allow that expert to independently test whether the evidentiary software operates as intended. Without that opportunity, defendant is relegated to blindly accepting the company's assertions as to its reliability. And importantly, the judge would be unable to reach an informed reliability determination at the Frye hearing as part of his gatekeeping function.

Hiding the source code is not the answer. The solution is producing it under a protective order. Doing so safeguards the company's intellectual property rights and defendant's constitutional liberty interest alike. Intellectual property law aims to prevent business competitors from stealing confidential commercial information in the marketplace; it was never meant to justify concealing relevant information from parties to a criminal prosecution in the context of a Frye hearing.

Requiring access to trade secrets in criminal cases is not new in New Jersey. In State v. Chun, 194 N.J. 54, 64, 66, 68-70, 943 A.2d 114 (2008), our Supreme Court ordered Draeger Safety Diagnostics Inc. (Draeger), the company that produces the Alcotest 7110 breathalyzer, to disclose its proprietary source code for independent review. Outside objective analysis revealed significant source code errors. Id. at 126-32, 943 A.2d 114.

In other jurisdictions, and directly on point here, courts have also made available under protective orders proprietary information of genotyping software, with noteworthy results. For example, as part of a Daubert2 hearing, a federal judge unsealed the source code of Forensic Statistical Tool (FST), a probabilistic genotyping software that had been developed and used by the New York City Office of Chief Medical Examiner (OCME). In 2017, that review demonstrated the software—employed in thousands of criminal prosecutions—was unreliable, did not work as intended, and had to be eliminated. And in 2015, after TrueAllele's competitor, STRmix, was forced to reveal its source code, analysts discovered coding errors that led to misleading results. The analysis of that proprietary information substantially affected the software's reliability. In appropriate circumstances, especially where civil liberties are on the line, independent source-code review is critical when determining reliability at a Frye hearing. These case studies illustrate that software is not immune from error. Fundamental due process and fairness demand access.

We hold that if the State chooses to utilize an expert who relies on novel probabilistic genotyping software to render DNA testimony, then defendant is entitled to access, under an appropriate protective order, to the software's source code and supporting software development and related documentation—including that pertaining to testing, design, bug reporting, change logs, and program requirements—to challenge the reliability of the software and science underlying that expert's testimony at a Frye hearing, provided defendant first satisfies the burden of demonstrating a particularized need for such discovery. To analyze whether that burden has been met, a trial judge should consider: (1) whether there is a rational basis for ordering a party to attempt to produce the information sought, including the extent to which proffered expert testimony supports the claim for disclosure; (2) the specificity of the information sought; (3) the available means of safeguarding the company's intellectual property, such as issuance of a protective order; and (4) any other relevant factors unique to the facts of the case. Defendant demonstrated particularized need and satisfied his burden.

Importantly, the President's Council of Advisors on Science and Technology (PCAST) emphasized that probabilistic genotyping is in its infancy and such “subjective methods” must be subject to “careful scrutiny.” President's Council of Advisors on Sci. & Tech., Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods 5 (2016) [PCAST Report]. We did that here. Specifically, PCAST found in 2016—and pertinent to questions of reliability—that probabilistic genotyping programs should be independently evaluated to determine whether the methods are scientifically valid and, importantly, whether the software itself correctly implements the methods. Id. at 79. The latter has never been done for TrueAllele. Full independent access in an adversarial system is a prerequisite to meaningful cross-examination of the State's expert at the Frye hearing, and essential to the judge's threshold gatekeeping reliability determination of whether the science underlying the proposed expert testimony has “gained general acceptance in the particular field in which it belongs.” State v. Harvey, 151 N.J. 117, 169, 699 A.2d 596 (1997) (quoting Frye, 293 F. at 1013-14).

We therefore reverse and remand for further proceedings consistent with this opinion.

I.

Just after 10:00 p.m. on April 16, 2017, two police officers traveling in an unmarked vehicle along Ocean Avenue in Jersey City observed two men, later identified as defendant and co-defendant Jonathan Ferrara, approach a group gathered near the intersection with Van Nostrand Avenue, simultaneously raise their handguns, and fire into the crowd. One victim sustained a bullet wound to the head and was pronounced dead at the scene. A second victim, a ten-year-old girl, suffered a non-fatal wound to the abdomen when a bullet entered a vehicle in which she was sitting.

After the shooting, the officers pursued defendant and Ferrara as they fled down a side street with their guns still in hand. The police arrested them within a few blocks of the incident.3 Police found a Colt .45 caliber semi-automatic handgun while retracing Ferrara's path, and recovered a .38 caliber Smith and Wesson revolver and ski mask while retracing defendant's path.

A forensic scientist detected the presence of amylase, a constituent of saliva, on the ski mask, and investigators swabbed the trigger guard, grip, and front sight of both weapons and the magazine of the Colt .45 for DNA evidence. The forensic scientist forwarded the mask and swabs to a laboratory, where analysts determined that the samples from the guns and one from the mask failed to meet the criteria for traditional DNA analysis, but that two specimens from the mask each reflected a mixture of DNA profiles, one with two contributors and the other with three. A comparison with buccal swab samples taken from defendant and Ferrara showed that defendant was the major source contributor for the DNA profiles from both the ski mask specimens conducive to traditional analysis.

Because the remaining samples failed to satisfy the criteria for traditional DNA analysis, the State forwarded the testing data to Cybergenetics Corp. Laboratory (Cybergenetics), a private firm in Pittsburgh, for analysis using its proprietary TrueAllele computer software program. Ferrara could not be identified as a contributor to any of the samples under the statistical analysis, but defendant was identified as a source of the DNA on the Smith and Wesson and the ski mask.

There is a substantial difference between testing DNA utilizing traditional DNA methods and analyzing low levels or complex mixtures of DNA relying on probabilistic genotyping software.

In traditional DNA analysis, DNA is chemically extracted from a biological sample and amplified at a predetermined set of segments, or loci, using polymerase chain reaction (PCR), a technique that replicates the desired segments to generate millions of copies of each. PCAST Report, at 69. The lengths of the resulting fragments are then extrapolated, by comparison with known molecular size standards, from the distance each travels through a polymer solution during a process called capillary electrophoresis. Ibid. The analyst then generates a profile from the pair of lengths measured at each locus—one for each of the genetic variants, or alleles, inherited from each parent—and uses the resulting list of alleles for comparison to known samples. Ibid.

For a single-source sample, identifiable by the presence of at most two distinct fragment lengths—one reflecting each allele—for each locus, the profile may be directly compared with that for a known individual to assess whether the profiles match. Id. at 70. For a simple mixture involving genetic material from two individuals, on the other hand, analysis proceeds in much the same manner, but requires first distinguishing the two separate profiles, either by an imbalance in material rendering one contributor more dominant than the other or by the presence of a known individual's DNA in the mixture, such as is often the case in the sexual assault context. Id. at 70, 73. In conjunction with the simple determination of a match in the list of alleles, human analysts also typically calculate a “random match probability”—a statistic measuring the likelihood that another individual in the relevant population, selected at random, would have the same genotype as the contributor to the sample. Id. at 72-73. The smaller the probability, the more solid the match.

But analysis is more difficult with complex mixtures, particularly where the genetic material involved is small:

Such samples result in a DNA profile that superimposes multiple individual DNA profiles. Interpreting a mixed profile is different for multiple reasons: each individual may contribute two, one or zero alleles at each locus; the alleles may overlap with one another; the peak heights may differ considerably, owing to differences in the amount and state of preservation of the DNA from each source; and the “stutter peaks” that surround alleles (common artifacts of the DNA amplification process) can obscure alleles that are present or suggest alleles that are not present. It is often impossible to tell with certainty which alleles are present in the mixture or how many separate individuals contributed to the mixture, let alone accurately to infer the DNA profile of each individual.

[Id. at 75-76.]

Compounding that problem, analysis of small samples often entails allele “drop-in”—the detection of an allele from a contaminant DNA fragment that was not part of the original sample—or “drop-out”—the failure to detect an allele from DNA belonging to the sample, usually due to insufficiency of the quantity for analysis. John M. Butler, Advanced Topics in Forensic DNA Typing 324-26 (2011). The consequence is that analysis of these samples is inherently more probabilistic and leaves more room for interpretation than for the single-source or simple-mixture samples that have been traditionally subject to DNA testing using the above procedures. PCAST Report, at 76.

The TrueAllele Casework system is one of several software programs developed with the goal of undertaking analysis of these more complex samples in as objective a manner as possible. PCAST Report, at 78-79. Such programs employ probabilistic genotyping, the “use of biological modeling, statistical theory, computer algorithms, and probability distributions,” to “assist,” rather than “replace,” “the DNA analyst in the interpretation of forensic DNA typing results.” Science Working Group on DNA Analysis Methods (SWGDAM), Guidelines for the Validation of Probabilistic Genotyping Systems 2 (June 2015) [SWGDAM Guidelines].

Specifically, the programs use mathematical models and simulations, subject to parameters programmed into the software to account for drop-in or drop-out effects and other issues, id. at 3, to calculate a likelihood ratio—a statistic measuring the probability that a given individual was a contributor to the sample against the probability that another, unrelated individual was the contributor. Justice Ming W. Chin et al., Forensic DNA Evidence § 5.5 (2020). In contrast to the implication for a random match probability, the higher the likelihood ratio, the more solid the match.4

The State requested a Frye hearing, acknowledging that TrueAllele has not yet been found reliable and admissible in New Jersey. The judge commenced the hearing, at which a co-founder of Cybergenetics, Dr. Mark Perlin, testified for two days ending in April 2019. The judge qualified Dr. Perlin as an expert in “the fields of DNA Evidence, Interpretation, and Likelihood Ratio.”

Prior to cross-examination, defendant moved for TrueAllele's software source code and related documentation. Specifically, the defense sought the source code and “all software dependencies such as third-party code libraries, toolboxes, plug-ins, and frameworks,” as well as “[s]oftware engineering and development materials describing the development, deployment, and maintenance” of the code. Defendant challenged the reliability of the probabilistic genotyping program, refusing to blindly accept validation studies involving Dr. Perlin, none of which were, as PCAST called for, independent studies to investigate whether the program's software correctly implemented the underlying probabilistic genotyping methods.

The parties submitted written declarations by experts detailing, among other things, the uncertainty involved in DNA mixture interpretation, the need for verification and validation (V & V) of software engineering, the existence of software engineering failures, and materials relevant to testing probabilistic genotyping software. Defense counsel produced a declaration written by defendant's expert, Nathaniel Adams, a systems engineer retained to address the reliability of the science underlying testimony based on TrueAllele. The State produced a declaration by Dr. Perlin. The parties were apparently satisfied—as was the judge—that the detailed declarations, Dr. Perlin's testimony over two days, and the documentation introduced at the Frye hearing, established a sufficient motion record. We reach that conclusion because the State did not move to require testimony from Mr. Adams or further testimony from Dr. Perlin, or otherwise seek a limited remand for that purpose. In our view, that is not surprising given the detailed record and declarations submitted by the experts addressing the source code.

Mr. Adams has important and extensive experience performing probabilistic genotyping analyses, including undertaking review of source codes. He reviewed “software development materials, including source code, for [the] probabilistic genotyping systems STRmix[ ] and FST used in criminal cases in New York, Illinois, United States, and Australian courts.” He explained:

Since the likelihood calculations are dependent on the statistical models (algorithms) underlying the probabilistic software, and software behaviors affecting the models will necessarily impact the calculated likelihoods and ultimately the reported likelihood ratio. Complex systems such as TrueAllele[ ] involve a hierarchy of models with dozens or hundreds of parameters, each affecting the overall system's behavior.

Mr. Adams pointed out that forensic DNA analysis “lacks formal standards specific to the development and validation of probabilistic genotyping software.”5 But software quality unquestionably depends in part on the “quantity and severity of defects present in the program.” He spelled out that defects cause incorrect and misleading results. One goal of the V & V processes is to assure “appropriate ... methods have been used throughout the software development process and have produced an acceptable product.” V & V involves reviewing software development materials for “correctness, completeness, consistency, and accuracy.” We need not detail every aspect of his declaration; suffice it to say that Mr. Adams provided the judge with an in-depth and thorough basis to grant the motion.

Dr. Perlin submitted a seventy-eight-paragraph declaration, which the State attached to its September 13, 2019 letter opposition brief to the judge. His declaration covered such topics as the role of TrueAllele in DNA analysis; TrueAllele's purported widespread acceptance; whether TrueAllele is reliable; background on the software source code; an explanation for why TrueAllele is a trade secret; the risks of disclosing the source code; and importantly, the reasons for why TrueAllele's source code is not needed. His declaration, therefore, developed the record for the judge's consideration of defendant's discovery motion.

Dr. Perlin explained that the source code details “step-by-step human-readable instructions that describe to the computer and programmers how the program operates.” According to Dr. Perlin, who has degrees in chemistry, mathematics, medicine, and computer science, the source code “contains the software design, engineering know-how, and algorithmic implementation of the entire computer program.” Although the software program itself is patented, its source code is not disclosed in patent documents; instead, Cybergenetics “considers the ... source code to be a trade secret.”6 Dr. Perlin, although he was the State's expert, advocated on behalf of his company that access to the source code would be “immaterial to [a criminal] case,” “[un]reasonable,” and not “in the interests of justice.”

According to Dr. Perlin, TrueAllele's software program consists of approximately 170,000 lines of source code written in MATLAB, a mathematical programming language designed specifically for visualizing and programming numerical algorithms. Dr. Perlin volunteered that it could take hours to decipher only a few dozen lines of the “dense mathematical text” comprising the code, and estimated that it would take a person, reading at a rate of ten lines per hour, about eight and a half years to review the code in its entirety.

Dr. Perlin explained that Cybergenetics operates in a “highly competitive commercial environment.” According to him, at least ten other groups have developed “similar software.” He defended confidentiality by asserting “for-profit companies [like Cybergenetics] generally do not make their source codes available to the public.” Such secrecy gives Cybergenetics a commercial “advantage over its competitors” because they do not know—nor does anyone else—the proprietary code. Once divulged, proprietary trade secrets, Dr. Perlin explained, are “valuable to competitors” and can be “sold for profit.” He declared that ample material for this case had already been provided, including “over thirty validation studies and publications.” His declaration omits reference to his own involvement in those studies, or the participation in the studies of current or former employees of Cybergenetics, and he neglected to acknowledge the lessons learned from STRmix and FST, which were revealed once other courts forced them to make accessible their source codes for independent review under protective orders.

Dr. Perlin explained that Cybergenetics permits testing the software online through cloud computing without having to purchase the product, and makes its methodology, which has long been published, and testing results available for review and questioning, either at its Pittsburgh office or by teleconference. Cybergenetics offered defendant an opportunity for “inspection” of the source code under a severely restrictive non-disclosure agreement (NDA), which limited inspection to an expert witness retained by defendant at a time and place determined by the company, under supervision by a company representative, and video surveillance and recording at all times. According to the NDA, a stand-alone computer that would not accept storage devices would be provided for viewing the source code, and, although the expert could make handwritten notes, the expert would be forbidden from bringing any photographic devices, including smart phones or tablets, into the room, and would be bound to turn any notes over to Cybergenetics. The expert would be broadly bound to accept responsibility for any legal and financial consequences, including a $1,000,000 automatic fine, in the event of a breach and could not “be a developer of competing software products” or “have any (direct or indirect) commercial, research or employment interest in such products.”

Mr. Adams emphasized that several of the restrictions Cybergenetics imposed would undermine an effective review of the source code for purposes of assessing TrueAllele's reliability. Specifically, the prohibitions on taking notes except by hand and on accessing the Internet or any removable storage device would inhibit adequate “documentation of the inspection process and collection of demonstrative materials,” and his inability to compile or execute the source code would be detrimental to any “rigorous ... inspection.” The ban on email communication, meanwhile, would restrict his consultation with defense experts in other relevant fields, such as biology, statistics, or software development, which would be necessary for understanding and evaluating the source code and related proprietary information. Mr. Adams did not believe any expert would agree to the automatic assumption of all liability for a breach.

Defendant's separate proposed protective order provided that the materials would remain confidential and used solely for purposes of preparation of defendant's defense in this matter, that no recipient could “reveal, use, or disclose any part” of it, except in compliance with the restrictions in the order, and that no third-party could be granted access without first agreeing to be bound by the same terms. Defendant's order would forbid any disclosure at all to a consultant or expert who was “the developer of” or who would “have any direct or indirect commercial or employment interest in competing software products.” The source code would be made available in a specified accessible format on a stand-alone computer provided by Cybergenetics for the expert to review and, as necessary, “make inspection notes, use necessary software, [and] create snippets or screen shots of relevant lines of code for use in his/her report.” All materials, moreover, would be filed under seal, and all counsel would be bound to take all “reasonable and appropriate measures to prevent unauthorized disclosure,” with any violation subject to civil and criminal sanction.

In response, Cybergenetics offered to remove some of the conditions for disclosure in its initial agreement, including the requirement for the expert to turn over any notes, but left most in place, most notably the broad acceptance of liability and the prohibition on taking notes or documenting the inspection in any other manner than by pen and paper. But the parties were unable to reach an agreement despite “[e]xtensive communications between the parties.”

Indeed, at oral argument in October 2019, the judge and counsel agreed that an appropriate protective order would accommodate all concerns. To that end, defense counsel produced a sample protective order utilized in Illinois, when a court there ordered STRmix to make its source code accessible for independent review. Although the assistant prosecutor stated this case had a “long and torturous procedural history,” and eventually conceded that the source code had never been independently examined or tested, especially by software scientists, he agreed with the judge that “all [the State] need[ed]” before access was given were “conditions and parameters to protect [Cybergenetics'] proprietary interests.” Oral argument was carried at least seven times during which counsel unsuccessfully discussed negotiating a protective order.

On June 23, 2020, the judge entered an order denying defendant's motion. The judge did not explicitly—for purposes of the discovery motion and his reliability determinations under Frye—address the importance of allowing defendant an opportunity to independently evaluate whether TrueAllele's software correctly implements the probabilistic genotyping methods, as emphasized by PCAST, rather than relegating defendant to blindly accepting that the software operated as intended. The judge omitted reference to whether the software's program contained bugs, glitches, or defects, and if so, whether such problems—untested in an adversarial system—could affect the software's output, which would in turn affect the reliability of TrueAllele. Pointing to unilateral conditions imposed solely by the State, the judge noted, however, that

[t]he State is willing to make the source code available for defense expert review. The State submits the defense expert is welcome to come to the prosecutor's office, view the source code on a provided device, and take notes.

Understandably, the State never contended before the judge that the judge was missing any substantial pertinent information to make an informed decision on defendant's motion for the discovery. We believe that is primarily because the judge had the detailed source-code declarations by the experts, Dr. Perlin's testimony over two days, validation studies and peer-reviewed articles, as well as out-of-state case law addressing the reliability of TrueAllele. Indeed, the State's willingness to permit limited access demonstrates its main objection pertained not to accessing the source code but rather reasonable parameters surrounding inspection. Defendant, meanwhile, maintains the parameters the State has thus far offered are unreasonably burdensome and restrictive.

In July 2020, we granted leave to appeal from the June 23, 2020 order; we later granted amici permission to participate. As we pointed out, and as part of defendant's motion for leave to appeal, and thereafter, the State never requested a limited remand to expand the Frye record with additional testimony by Dr. Perlin or anyone else. The record therefore demonstrates the parties did not in any meaningful way dispute the adequacy of the motion record.

II.

On appeal, defendant argues the following points, which we have partially re-numbered:

POINT I

THE RELIABILITY OF TRUEALLELE CANNOT BE DETERMINED WITHOUT COMPLETE DEFENSE ACCESS TO ITS SOURCE CODE AND THE TOOLS NECESSARY TO INSPECT THAT CODE.

1. TrueAllele Is Dramatically Different Than Traditional DNA Analysis And Its Reliability Has Never Been Established In New Jersey.

2. A Complete Review Of Source Code Is Necessary For A Rigorous Assessment Of TrueAllele's Reliability.

i. Errors in software programs are ubiquitous and often have devastating results. Nothing short of full source-code review can catch and correct these errors.

ii. Errors in the source code of probabilistic genotyping software have been found. There is no reason to assume that TrueAllele, whose source code has never been subject to outside inspection, is immune from these errors.

iii. Validation studies and peer-reviewed articles are not a substitute for source-code review.

3. Defendant Is Entitled To The Source Code And Related Materials Under Our Discovery Rules And Jurisprudence. Any Proprietary Interests Cybergenetics Has Can Be Accommodated By A Protective Order.

4. This Court Should Not Repeat The Mistake Of Other Courts By Failing To Subject TrueAllele To Source-Code Review Before Ruling On Its Admissibility.

5. Disclosure Is Necessary To Preserve The Fairness Of Any Trial In Which TrueAllele May Be Used In The Future.

In his reply letter brief, defendant makes the following additional contentions, which we have re-numbered:

[POINT II]

THE MATERIALS THE DEFENSE SEEKS ARE NECESSARY IN ORDER FOR TRUEALLELE'S RELIABILITY, AND THEREFORE ADMISSIBILITY, TO BE DETERMINED.

1. The State's Conclusory Assertion That The Materials At Issue Are A Trade Secret Does Not Shield These Materials From Disclosure.

2. The State Has Failed To Demonstrate That TrueAllele's Reliability Can Be Assessed Without Access To These Materials.

III.

We have the benefit of extraordinarily thoughtful amici briefs from a multitude of organizations, including the New Jersey Attorney General and other interested entities from around the nation. Before directly analyzing the issues, we detail their positions. Doing so informs our analysis and holding.

(i) New Jersey Attorney General (AG)

The AG asserts that defendant requires the State to prove that TrueAllele is “infallible,” which the AG argues is not required under Frye. The AG argues the State satisfied its burden under Frye by offering three things: testimony by Dr. Perlin; “validation studies and publications”; and opinions from other jurisdictions in which those courts have deemed TrueAllele reliable without independent inspection of the proprietary information. The AG states that access to the source code is therefore unnecessary to determine whether TrueAllele is generally accepted in the scientific community. The AG argues complete general acceptance is not required, and that “any concerns are best served during cross-examination.”

The AG suggests that if this court deems full access is possibly necessary, then we should remand and allow testimony from Dr. Perlin and Mr. Adams about whether access to proprietary information is appropriate. The AG cites State v. Ghigliotty, 463 N.J. Super. 355, 384-85, 232 A.3d 468 (App. Div. 2020), for the proposition that defendant must “provide the [judge] with a rational basis” before allowing reasonable access. According to the AG, defendant failed to do so here.7

At oral argument, the AG conceded the State will not be prejudiced by disclosure of the discovery. The AG argues the State is willing to make the trade secrets available to defendant, but contends defendant is unreasonably unsatisfied with the State's terms of inspection. The AG contends that defense counsel wanted “unsupervised and unrestricted access to proprietary information.” In affording access to the information, the AG asserts that the State “removed many of the typical restrictions required.” The AG states “[a] protective order that offers no protections is not adequate in a competitive market.” As the AG points out, the parties unsuccessfully attempted negotiating terms of such an order.

(ii) The Innocence Project

The Innocence Project maintains that analyzing the source code is critical to determining the reliability of TrueAllele because it would reveal, among other things, errors in coding or output. The Innocence Project underscores these indisputable facts: people write source codes; people make mistakes.

The Innocence Project states that genotyping software is prone to error, as exemplified by the problems associated with STRmix and FST. It is not enough—as the State argues—to allow inspection of articles discussing how TrueAllele is intended to work; without the source code it is impossible to detect errors in implementation. Without access to the source code one cannot identify errors or biases, which the Innocence Project explains are relevant to reliability at the Frye hearing. Although algorithms and models are publicly available, TrueAllele's source code, which the Innocence Project contends is prone to error even when the corresponding algorithms and models may be correct, is known only by individuals at Cybergenetics. At oral argument, the Innocence Project emphasized that, while validation studies are important and programs may find their way into court without them, independent review and the judge's Frye gatekeeping should not be perfunctory.

(iii) Upturn, Inc. (Upturn)

Upturn, an organization seeking to advance equity and justice in the design, governance, and use of technology, points out that TrueAllele's source code has never been independently reviewed, and that such a review is a basic and necessary step in ensuring reliability. Importantly, the version of TrueAllele software utilized in defendant's case postdates every one of the validation studies cited by Cybergenetics and the State. It explains this is critical because subsequent source code variations may introduce new errors not previously present. Undertaking an independent review establishes whether the software is properly implementing the program's design specifications and that the code itself is devoid of bugs, glitches, and defects that could affect the software's output. And equally important is that TrueAllele's source code has never been scrutinized by any party outside of Cybergenetics; therefore, the validation studies produced by the State to date are limited.

Upturn points out that looking at what happened with FST in New York—when a federal judge required OCME to make available the source code for the program, revealing errors—demonstrates the significance of what is at stake. Mr. Adams examined FST's code and discovered two critical problems: the code did not implement FST's methods and models utilized in FST's validation studies, and there were coding errors. The New Jersey Supreme Court did the same thing in Chun, by requiring Draeger to produce its source code. Upturn encourages this court to take the same action to assess TrueAllele's reliability at the Frye hearing.

Upturn maintains that trade secrets should not be prioritized over considerations of justice, especially because production of a for-profit company's trade secret can be reviewed under an appropriate protective order. Upturn relies on N.J.S.A. 2A:84A-26 (rejecting application of trade secret privilege where it “tend[s] to conceal fraud or otherwise work[s] injustice”), and it contends that interpreting evidentiary privileges narrowly, Pierce Cnty. v. Guillen, 537 U.S. 129, 144, 123 S.Ct. 720, 154 L.Ed.2d 610 (2003), provides further support for resisting application of the privilege whenever that would impede justice.

(iv) The Association of Criminal Defense Lawyers of New Jersey (ACDL-NJ)

ACDL-NJ asserts that probabilistic genotyping has not yet been used or tested in New Jersey. Like other amici, ACDL-NJ argues that, given that TrueAllele's leading competitor, STRmix, has produced its source code and conceded its software had errors, rigorous scrutiny of TrueAllele's source code becomes even more compelling. If anything, STRmix's source-code problems reaffirm the basic principle in computer engineering that software is prone to human error.

ACDL-NJ argues the source code is discoverable under New Jersey law. Of course, the United States Constitution and the New Jersey Constitution guarantee a meaningful opportunity to present a complete defense. But ACDL-NJ asserts that New Jersey's robust discovery practices are broader than those of other jurisdictions. For example, Rule 3:13-3(b) provides a non-exhaustive list of discoverable materials. And in Chun, the Court allowed the defendants to analyze the source code of the software that ran the Alcotest, which disclosed two errors that affected the way Alcotest results had been used in prosecutions. 194 N.J. at 94, 943 A.2d 114.

Finally, ACDL-NJ argues that requiring a defense attorney to sign any order that preconditions a defense expert's review of the source code in practical ways—such as in this case—is prohibitive. Doing so impedes counsel's ability to provide an effective defense, which would be free from any conflict of interest. Here, there are enormous problems associated with the State's proposed protective order: defense counsel could only make handwritten notes while looking at hundreds of thousands of lines of code; counsel could not use electronic devices; the only computer available to counsel would be one provided by Cybergenetics; counsel would be under constant supervision; and counsel would be exposed to monetary damages, including fees and costs, as spelled out under the State's terms. Rather, ACDL-NJ contends that the judge should issue an appropriate protective order that protects Cybergenetics' proprietary interests, while simultaneously protecting defendant's liberty interests. ACDL-NJ notes that protective orders have safeguarded trade secrets in high-risk civil litigation for years and can therefore do so here.8

(v) The Legal Aid Society (LAS)

LAS is the primary public defender in New York City. LAS has first-hand experience litigating the admissibility of a proprietary probabilistic genotyping program—FST—including successfully obtaining access to FST's source code, which led to an alarming discovery: significant flaws existed in the software program. Without access to the source code, the defects and glitches in the software would not have been uncovered. The bugs in the program were substantial enough for OCME to cease using FST, which up to that point had been used in thousands of criminal prosecutions over several years.

Like the other amici, LAS emphasizes the extraordinary complexity of probabilistic genotyping. LAS urges us to carefully consider the inherent limitations of the expert testimony, scientific and legal writings, and judicial opinions submitted by the State: none required an examination of TrueAllele's source code. Consequently, LAS implores us to consider the State's submissions with healthy skepticism.

LAS explains that probabilistic genotyping software is intended to address interpretational challenges of testing low levels or complex mixtures of DNA. For example, stochastic effects and artifacts complicate determining genotypes, or DNA profiles: alleles not belonging to true donors appear, they can be distorted, and artifacts appear as real alleles. LAS points out PCAST  emphasized that probabilistic genotyping is in its infancy and must be subject to “careful scrutiny.” PCAST Report, at 79. Specifically, PCAST found in 2016—and pertinent to questions of Frye reliability—that probabilistic genotyping programs should be evaluated to determine “whether the methods are scientifically valid” and importantly, “whether the software correctly implements the methods.” Ibid. And critical to the determination, according to PCAST, is testing by independent entities “not associated with the software developers.” Ibid. LAS states that the only way to determine whether a program operates as intended is to evaluate how the program performs its calculations, which requires access to the source code.

Fortunately, due to its own efforts, LAS points to the case study of FST, troubling that it is, which demonstrates the importance of an independent and full source-code review when a judge makes a threshold reliability determination of whether novel forensic software has achieved general acceptance in the relevant scientific community. The creators of FST fought tooth and nail not to disclose its source code. But after a federal judge denied OCME's motion to quash a subpoena for the source code, a stark discovery was made about the program: FST did not work as promised. FST was utilized in thousands of criminal prosecutions before the discovery was made. OCME announced—after the production of the source code—that it would phase out using FST in criminal prosecutions.

According to LAS, the State's assertion that the source code was not needed in any of TrueAllele's “numerous [prior] nationwide admissibility rulings,” is at best misguided. LAS implores us to carefully examine the premise of that body. LAS contends that what matters is not the number of cited opinions, but rather, the power of the court's reasoning. LAS reminds us, as the New York Court of Appeals recently stated in People v. Williams, 35 N.Y.3d 24, 147 N.E.3d 1131, 1140-42 (2020), that it was a mistake to rely on the repetition of case law to establish reliability; rather, for purposes of a Frye hearing, lower courts were bound to ensure that FST was “supported by those [in the relevant scientific community] with no professional interest in its acceptance.” LAS urges us to heed the lessons of FST and permit full independent access to the source code under a protective order.

(vi) Drs. Mats Heimdahl and Jeanna Matthews

Drs. Heimdahl and Matthews are experts in engineering, testing, and validating computer systems, including forensic evidentiary software. They, together with eight other experts in this specific field that they have identified, argue that reliability of the TrueAllele software cannot be evaluated without full access to “executable source code and related documentation,” something that no one to date has seen. They contend that doing so is not only prudent, but essential to determining whether TrueAllele operates as Cybergenetics claims, which is fundamental to any fair, legitimate, and impartial assessment of reliability.

Drs. Heimdahl and Matthews remind us that software faults are ubiquitous. They argue that even simple software programs are prone to failure, and that an error in any one of the three domains of software engineering—problem identification, algorithm development, and software implementation—undermines the trustworthiness of the science underlying the relevant expert testimony, because the system is consequentially compromised. After providing examples illustrating various errors in more simplistic software, they demonstrated that a greater risk of flaws in more complex programs are likely.

For example, a source-code review revealed at least thirteen STRmix coding faults. Drs. Heimdahl and Matthews argue, in one important example, a miscode impacted sixty criminal cases, requiring new likelihood ratios to be issued in twenty-four of them. These errors were not discovered until the source code was independently examined.

In FST, alarming discoveries were also made. But the findings did not come to light until a federal judge ordered disclosure of FST's source code. Once that occurred, it was uncovered that a “secret function ... was present in the software, tending to overestimate the likelihood of guilt.” And the functioning of the software did not use the “methodology publicly described in sworn testimony and peer-reviewed publications.” These discoveries led to the overturning of a high-profile conviction.

Drs. Heimdahl and Matthews assert that thousands of faults were discovered in the source code of breathalyzer systems. They point out that judges in Massachusetts and New Jersey threw out more than 30,000 breath tests in a twelve-month period. Drs. Heimdahl and Matthews urge us not to ignore these facts.

Drs. Heimdahl and Matthews argue that the testing of TrueAllele is incomplete. Thirty-five of the thirty-six validation studies produced by the State, which were written by or included involvement from current or former employees of Cybergenetics or law enforcement agencies, did not consider the source code, and they were otherwise incomplete because the number of samples tested was relatively small. They note that TrueAllele's software is non-continuous, meaning that correct results for the samples used in the validation studies do not preclude the possibility of erroneous results for others that do not match those samples. Thus, for a reliability determination, independent and full access to the software is required. Supporting software development documentation must be produced, including that pertaining to testing, design, bug reporting, change logs, and program requirements, which will provide a road map to understanding the source code.

(vii) American Civil Liberties Union of New Jersey (ACLU-NJ)

The ACLU-NJ argues that independent review is essential. Questioning Dr. Perlin, reviewing validation studies and peer-reviewed articles in which he or his current or former employees were involved, or relying on out-of-state judicial opinions citing his testimony and those studies misses the importance of objective analysis of the science underlying his forensic testimony. Most importantly, it cannot substitute for independent analysis of the code itself, which would demonstrate whether the software operates as intended.9

IV.

As the New Jersey Supreme Court recently stated, “the Judiciary must ensure that proceedings are fair to both the accused and the victim. Trial judges partly fulfill that responsibility by serving as a gatekeeper. In that role, they must assess whether expert testimony is sufficiently reliable before it can be presented to a jury.” State v. J.L.G., 234 N.J. 265, 307-08, 190 A.3d 442 (2018). When the evidence is labeled as scientific and expert, there is substantial danger that juries will accord excessive weight to testimony that might otherwise be unreliable. Ghigliotty, 463 N.J. Super. at 373, 232 A.3d 468. The law accounts for this eventuality.

To fulfill their gatekeeping responsibility, judges begin by applying N.J.R.E. 702, which states that, “[i]f scientific ... knowledge will assist the trier of fact to understand the evidence or to determine a fact in issue, a witness qualified as an expert by knowledge, skill, experience, training, or education may testify thereto in the form of an opinion or otherwise.” To satisfy this requirement,

the proponent of expert evidence must establish three things: (1) the subject matter of the testimony must be “beyond the ken of the average juror”; (2) the field of inquiry “must be at a state of the art such that an expert's testimony could be sufficiently reliable”; and (3) “the witness must have sufficient expertise to offer the” testimony.

[J.L.G., 234 N.J. at 280, 190 A.3d 442 (quoting State v. Kelly, 97 N.J. 178, 208, 478 A.2d 364 (1984)).]

In general, these prongs “are construed liberally in light of [N.J.R.E.] 702's tilt in favor of the admissibility of expert testimony.” State v. Jenewicz, 193 N.J. 440, 454, 940 A.2d 269 (2008). The first and third prongs are not at issue here; rather, the second prong is. The parties and amici have focused—as do we—on whether defendant is entitled to independently review the source code and related documents pertaining to the reliability prong before cross-examination of Dr. Perlin and before the judge completes his important Frye reliability gatekeeping function.

In criminal cases, the Supreme Court of New Jersey has continued to apply the Frye standard to assess scientific reliability lying beneath the expert testimony.10 J.L.G., 234 N.J. at 280, 190 A.3d 442. The Frye test requires trial judges to determine whether the particular science underlying the proposed expert testimony has “gained general acceptance in the particular field in which it belongs.” Frye, 293 F. at 1014; accord J.L.G., 234 N.J. at 280, 190 A.3d 442; Harvey, 151 N.J. at 169, 699 A.2d 596; see also State v. Torres, 183 N.J. 554, 568, 874 A.2d 1084 (2005). “Although we look for wide support within the relevant scientific community, complete agreement is not required for evidence to be admitted.” J.L.G., 234 N.J. at 281, 190 A.3d 442. Importantly—like here—there might be more than one scientific community to consider. Thus, to assess the reliability prong of N.J.R.E. 702, and relevant to the issues on appeal, the judge should consider—as to general acceptance in the scientific community—whether Cybergenetics' TrueAllele probabilistic genotyping computer program is scientifically valid and importantly, whether the source code itself correctly implements the methods. But to do that raises the question of whether defendant is first entitled to discovery of the proprietary information he seeks, which brings us to our legal analysis.

V.

The right to a fair trial is fundamental and guaranteed pursuant to the Fifth and Sixth Amendments of the United States Constitution, as well as the New Jersey Constitution. U.S. Const. amend. V, VI; N.J. Const. art. I, ¶ 10. Our Constitutions also ensure criminal defendants “a meaningful opportunity to present a complete defense.” State v. Garron, 177 N.J. 147, 168, 827 A.2d 243 (2003) (quoting Crane v. Kentucky, 476 U.S. 683, 690, 106 S.Ct. 2142, 90 L.Ed.2d 636 (1986)). These fundamental legal rights are the hallmark of our judicial process, a process which technology has recently heavily impacted. Forensic evidentiary computer software itself generates expert evidence, and the New Jersey Rules of Evidence enable the introduction and, consequently, cross-examination of expert witnesses. N.J.R.E. 702. Without access to the source code—the raw materials of the software programming—a defendant's right to present a complete defense, by meaningful cross-examination at the appropriate juncture, may be substantially compromised. Relevant to this case, “[a] criminal trial where the defendant does not have ‘access to the raw materials integral to the building of an effective defense’ is fundamentally unfair.” State in the Interest of A.B., 219 N.J. 542, 556, 99 A.3d 782 (2014) (quoting Ake v. Oklahoma, 470 U.S. 68, 77, 105 S.Ct. 1087, 84 L.Ed.2d 53 (1985)). We must keep these principles in mind and front and center during our review of the judge's order denying full access to the discovery sought.

In criminal cases, we ordinarily apply an abuse of discretion standard on discovery motions, State v. Stein, 225 N.J. 582, 593, 139 A.3d 1174 (2016), and on evidentiary determinations, State v. Prall, 231 N.J. 567, 580, 177 A.3d 755 (2018), but here defendant sought access—at a Frye hearing—to proprietary information solely to challenge the reliability of the science underlying novel DNA analysis evidentiary software and expert testimony. An appropriate review therefore requires that we also independently scrutinize the record, including the comprehensive and amplified declarations of the experts, the scientific validation studies and peer-reviewed publications, and judicial opinions. See In re Commitment of R.S., 339 N.J. Super. 507, 531, 773 A.2d 72 (App. Div. 2001) (noting that when matters involve “novel scientific evidence in a criminal proceeding, ‘an appellate court should scrutinize the record and independently review the relevant authorities, including judicial opinions and scientific literature’ ” (quoting Harvey, 151 N.J. at 167, 699 A.2d 596)).

Information pertinent to the Frye inquiry is subject to the same “broad pretrial discovery” otherwise afforded to a criminal defendant under Rule 3:13-3(b). State v. Scoles, 214 N.J. 236, 252, 69 A.3d 559 (2013). Our state's “ ‘open-file approach to pretrial discovery in criminal matters’ is intended ‘[t]o advance the goal of providing fair and just criminal trials.’ ” State v. Hernandez, 225 N.J. 451, 461-62, 139 A.3d 46 (2016) (alteration in original) (quoting Scoles, 214 N.J. at 252, 69 A.3d 559); see State v. Cook, 43 N.J. 560, 564, 206 A.2d 359 (1965) (noting that “discovery has long ... been found to be a sound tool for truth”). Because of the meaningful role that the disclosure of evidence to a defendant has in promoting the search for truth—and reliability at a Frye hearing—pretrial discovery in criminal trials has long received favorable treatment in this state. See State in the Interest of W.C., 85 N.J. 218, 221, 426 A.2d 50 (1981) (noting sharing of pretrial information “encourage[s] the presentation of all relevant material to the jury as an aid in the establishment of truth through the judicial process”). Although that discovery is not so broad, for example, as to indulge an “unfocused, haphazard search for evidence," Hernandez, 225 N.J. at 463, 139 A.3d 46 (quoting State v. D.R.H., 127 N.J. 249, 256, 604 A.2d 89 (1992)), judges are authorized to order discovery even “beyond that mandated by our court rules when doing so will further the truth-seeking function or ensure the fairness of a trial,” ibid. (quoting A.B., 219 N.J. at 560, 99 A.3d 782).

As to the evidence at issue here, a party seeking to shield information from discovery on intellectual property grounds generally bears the burden of showing good cause to demonstrate “that the information sought is a trade secret or is otherwise confidential or proprietary.” Cap. Health Sys., Inc. v. Horizon Healthcare Servs., Inc., 230 N.J. 73, 80, 165 A.3d 729 (2017); see also R. 4:10-3(g) (providing that a protective order may be sought to ensure “[t]hat a trade secret or other confidential research, development, or commercial information not be disclosed or be disclosed only in a designated way”); N.J.S.A. 2A:84A-26 (providing that “[t]he owner of a trade secret has a privilege ... to refuse to disclose the secret and to prevent other persons from disclosing it if the judge finds that the allowance of the privilege will not tend to conceal fraud or otherwise work injustice”). Yet, even once that showing of privilege is made, a criminal defendant should nonetheless be entitled to discovery of the information sought to the extent necessary to ensure a fair trial. Hernandez, 225 N.J. at 463, 139 A.3d 46. But the burden must shift to defendant to demonstrate a sufficient need for the evidence. See Ghigliotty, 463 N.J. Super. at 384-85, 232 A.3d 468 (requiring “definitive” demonstration of need for disclosure of algorithm); cf. Tractenberg v. Twp. of W. Orange, 416 N.J. Super. 354, 367, 4 A.3d 585 (App. Div. 2010) (discussing burden shift in the context of deliberative process privilege).

As we stated earlier, the Court ordered production of the source code in Chun. Although we ordinarily consider published decisions from other jurisdictions as persuasive, they are not binding on us. See Lewis v. Harris, 188 N.J. 415, 436, 908 A.2d 196 (2006) (noting that our courts are “not bound by ... the precedents of other states, although they may provide guideposts and persuasive authority”). The rationale undergirding Chun is binding. More recently, in Ghigliotty, 463 N.J. Super. at 360, 384-85, 232 A.3d 468, we too addressed the disclosure of proprietary information, algorithms underlying the software for BULLETTRAX, a novel device used for three-dimensional ballistics imaging, in contemplation of a Frye hearing. We vacated a motion judge's order requiring the State to produce the algorithms, but only because we viewed the order as prematurely issued. Id. at 384, 232 A.3d 468. We explicitly contemplated—as did the motion judge—that “this information might be needed by defendant's experts to evaluate the reliability of the new technology,” but noted that—unlike here—there was nothing in the record to support that order. Ibid. We explained that a “defendant is required to make a more definitive showing of his need for th[e] material to provide the [judge] with a rational basis to order the State to attempt to produce” the proprietary algorithms. Id. at 384-85, 232 A.3d 468.

Before going any further, we stress one important point. Evaluating the issues on appeal requires a working knowledge of computer software. Without such a foundation, one can miss subtle consequences germane to this Frye hearing. Allowing independent access to the requested information, for the sole purpose of addressing whether the technology underlying the expert testimony is reliable—specifically, whether the source code for that technology is properly implementing the program's design specifications—is obvious. An accused individual's liberty is at stake; DNA evidence is powerful and compelling.11 Practically speaking, if, as Dr. Perlin maintains, the source code he wrote is free of harmful defects, and therefore will not impact the reliability of TrueAllele, then it is to everyone's advantage to learn that at the Frye hearing. If it should turn out there are source code errors that might affect TrueAllele's reliability, the time to discover that information is now, as part of the judge's gatekeeping role. Reliability must be resolved at the Frye hearing rather than in post-conviction relief proceedings.

We are also mindful of the important need to maintain the confidentiality of trade secrets in—as Dr. Perlin emphasized in his own declaration—a “highly competitive commercial environment.” All agree on that. But shrouding the source code and related documents in a curtain of secrecy substantially hinders defendant's opportunity to meaningfully challenge reliability at a Frye hearing. The confluence of these competing and powerful interests compels our holding.

We hold that if the State chooses to utilize an expert who relies on novel probabilistic genotyping software to render DNA testimony, then defendant is entitled to access, under an appropriate protective order, to the software's source code and supporting software development and related documentation—including that pertaining to testing, design, bug reporting, change logs, and program requirements—to challenge the reliability of the software and science underlying that expert's testimony at a Frye hearing, provided defendant first satisfies the burden of demonstrating a particularized need for such discovery. To analyze whether that burden has been met, a trial judge should consider: (1) whether there is a rational basis for ordering a party to attempt to produce the information sought, including the extent to which proffered expert testimony supports the claim for disclosure; (2) the specificity of the information sought; (3) the available means of safeguarding the company's intellectual property, such as issuance of a protective order; and (4) any other relevant factors unique to the facts of the case. Applying this framework to the facts, we conclude defendant satisfied his burden.

(i) Rational basis for accessibility and expert testimony

In addressing this prong—whether there exists a rational basis for accessibility of the proprietary information—we must address the “three ways to establish general acceptance under Frye: expert testimony, authoritative scientific and legal writings, and judicial opinions.” J.L.G., 234 N.J. at 281, 190 A.3d 442. We do this by independently scrutinizing these categories with the benefit of lessons learned by the consequential software errors associated with STRmix and FST. Doing so convinces us that there is a rational basis for full access to TrueAllele's source code and related documentation for purposes of a Frye reliability analysis.

The troubling FST case study demonstrates a rational basis for independent source-code review of probabilistic DNA programs like TrueAllele. After being subjected to an adversarial audit when ProPublica obtained an order for the release of FST's source code, it was revealed that FST had a problem with a certain calculation that was only learned through the adversarial examination of the source code. Steven M. Bellovin et al., Seeking the Source: Criminal Defendants' Constitutional Right to Source Code, 17 Ohio State Tech. L.J. 1, 38 (2021). The audit discovered that certain “loci were removed from the likelihood ratio calculation” without “notice, either intended or actual, provided to the user of FST,” nor any “indication that this behavior [was] intended during [the] examination of FST-related publications and the FST [v]alidation materials.”12 Ibid.; see also Stephanie J. Lacambra et al., Opening the Black Box: Defendants' Rights to Confront Forensic Software, Champion 28, 30 (May 2018) (providing a snippet of the source code and explaining that “if the sum of frequencies is greater than 0.97, a row in the raceTable is removed”).13 As a result, the software was overestimating the likelihood of guilt. Beyond undocumented calculations, it was discovered that FST exhibited code smells,14 which suggested that “the program is below normal professional standards and may have other, not yet detected problems” which are “extremely difficult to detect ... without access to [the] source code.” Bellovin et al., 17 Ohio State Tech. L.J. at 39. At oral argument, the Innocence Project pointed out that, like TrueAllele, FST was subject to multiple validation studies but errors were still found in the source code, proving that validation of this type of evidentiary software is not determinative when evaluating computer science reliability.

Likewise, code errors and miscodes were discovered in TrueAllele's competitor STRmix after it had been developed, validated, and used in criminal prosecutions, further showing that errors in source code are not obvious or always timely found. When the source code was reviewed by independent forensic analysts, it was uncovered that the program produced false results in sixty cases.15 Mr. Adams also reviewed STRmix's code in 2015 and “was able to identify potential issues in STR[m]ix's source code that negatively affected the functioning of the software and could not have been learned from any other source.”

Defendant points out that any program's output could potentially be skewed not only by the inadvertent errors routinely found in lengthy code but by the numerous subtle choices made by programming developers regarding how to interpret input data. Defendant asserts, in part by reference to Mr. Adams' declaration, that many of those biases and errors may be conducive to detection only by a full examination and testing of the code and points to the consequential software errors of STRmix and FST. Indeed, exacerbating the danger of inherent bias or error specifically with respect to probabilistic genotyping software is that the likelihood ratio is not conducive to independent calculation or other precise verification, but highly sensitive to modeling assumptions embodied in the code. Defendant surmises that errors of similar magnitude and consequence to those in STRmix and FST may infect TrueAllele's code, noting that the program is likely to return vastly different likelihood ratios for the same physical sample in successive tests without explanation, and that the code has been edited numerous times without any explanation as to whether errors were remedied or any scrutiny as to whether others were inadvertently introduced. Defendant disputes that the validation studies and judicial reliability determinations of other jurisdictions, both of which the judge here found significant, were viable substitutes for source-code review in this case.

As discussed above, many of the amici amplify defendant's argument that full access to the source code is essential to evaluation. The Innocence Project, LAS, and Upturn all concur on that point, with LAS highlighting the discontinued FST program as a cautionary tale, and Upturn warning that a failure to require production would encourage secrecy and erode criminal defendants' constitutional rights. Drs. Heimdahl and Matthews, meanwhile, confirm and reiterate that errors are ubiquitous in software code and strongly believe that TrueAllele's code likely contains them. Moreover, they discount the validation studies on which the State relies, asserting that none entailed genuinely independent review, and that none involved computer science testing of whether the software operated as intended.

We acknowledge the State, on the other hand, disputes the notion that source-code review is essential to validation, noting that the SWGDAM guidelines require no such review, only testing, such as the sort that uncovered errors in the STRmix software. To be sure, the State is correct that the mechanism for evaluation contemplated in the SWGDAM guidelines is testing rather than source-code review, SWGDAM Guidelines, at 4-11, and that errors found in the STRmix program had been detected first through testing rather than visual examination of the code, Duncan A. Taylor et al., Commentary, A “Source” of Error: Computer Code, Criminal Defendants, and the Constitution, 8 Frontiers in Genetics art. 33, at 1 (2017).16 But production and review of the code for the since-discontinued FST program proved crucial to identification of significant errors, albeit not before compromised test results had already been used in many prosecutions. Lauren Kirchner, Doubts and DNA Evidence, N.Y. Times, Sept. 5, 2017, at A1. We cannot ignore these facts when evaluating whether there exists a rational basis for access to the proprietary information here.

The parties have cited expert testimony, authoritative scientific and legal writings, and judicial opinions that were generated before and after the STRmix and FST's software errors became public. This information bolsters our conclusion that there is a rational basis for the discovery. The State, of course, urged the judge to rely on that body of information to conclude there was no such basis. The testimony predominantly cited was that of Dr. Perlin, the scientific writings were mainly from Dr. Perlin (together with his then current or former employees),17 and the judicial opinions referred to that testimony and those scientific writings. But none of this information explicitly deals with whether TrueAllele's source code itself correctly implements the intended methods, as PCAST emphasized. PCAST Report, at 79.

As to expert testimony, Mr. Adams submitted a twenty-four-page declaration in which he asserted a need for production of the source code and related documentation. In his declaration, he addressed: his qualifications; an overview of his engagement, including whether TrueAllele “has been demonstrated to be in accordance with software engineering standards and principles”; the uncertainty in DNA mixture interpretation; a background on software engineering; details as to V & V, including definitions in the field of software engineering, system integrity, methodologies, code reviews, software testing, documentation, independence, and re-validation and performance checks; software engineering failures; materials relevant for review; requirement specifications, including design descriptions, source code, build instructions and dependencies, executable versions, tests, issue/bug tracking; user manuals, V & V, qualification and user testing, and miscellaneous processes; comment on Cybergenetics' proposed terms of inspection and nondisclosure agreement; and preferred terms for inspection. He also produced a detailed appendix to his declaration including documentation as to the software development process, the IEEE “risk-based, integrity-level scheme.”

Unlike in Ghigliotty, where there was “nothing concrete in the record” to support access to the algorithms the defendant sought for the BULLETTRAX algorithms at issue there, the opposite is true here. 463 N.J. Super. at 384, 232 A.3d 468. In Ghigliotty, we concluded the defendant was “required to make a more definitive showing of his need for th[e] material.” Id. at 384-85, 232 A.3d 468. Here, defendant did that with the proffered expert testimony supporting the claim for disclosure.

As to the validation studies and peer-reviewed articles issued during this timeframe, we have scrutinized them and make the following observations about their application. Since 2009, thirty-six validation studies have been conducted by Cybergenetics, law enforcement crime labs, or both, intending to establish the reliability of TrueAllele. These studies have utilized TrueAllele on both laboratory-generated and casework DNA samples and have tested TrueAllele to determine how it handles mixtures of varying DNA compositions and weights.

Seven of the thirty-six studies have been published in peer-reviewed journals, the first of which was published in 2009. The peer-review process entails a review for accuracy and quality of a scientific paper, in which a scientist describes his or her research and conclusions, and it is either accepted or rejected by two anonymous members of the relevant scientific community. A “laboratory-generated” validation study uses data that has been synthesized in a DNA laboratory and is of a known genotype composition. Four published papers are of this type.18 A “casework” validation study uses DNA data exhibiting real-world issues developed by a crime laboratory in the course of their usual casework activity. Three published papers are of this type.19

Notably here, six of the seven peer-reviewed publications were authored by Dr. Perlin himself. The one study not authored by Dr. Perlin does note that he provided professional guidance.20 PCAST explicitly noted the software developer's participation in such studies as an impediment to reliable validation, noting that, “[w]hile it is completely appropriate for method developers to evaluate their own methods, establishing scientific validity also requires scientific evaluation by other scientific groups that did not develop the method.” PCAST Report, at 80. That was not done here, where Dr. Perlin, a developer with a vested interest in the program's scientific acceptance, was directly involved. Law enforcement agencies, which also sometimes participated, likewise share an interest in the continued viability of the program. In the end, for purposes of reliability in a criminal context, it stands to reason that such an evaluation should be performed by an expert working on behalf of someone in defendant's shoes, with full access to the tools required for evaluation. See United States v. Gissantaner, 417 F. Supp. 3d 857, 880 (W.D. Mich. 2019) (addressing Daubert and the admissibility of STRmix and noting that “studies and articles ... have determined that review of probabilistic genotyping software, independent of that of the developers, is critical for an assessment of its reliability with respect to use in the courts”).

Moreover, despite Dr. Perlin's and the State's insistence that the TrueAllele program affords analysts a tool for objective analysis, it does not inexorably follow that that analysis is reliable. We consider the concept of “programmer blindness,” a common pitfall of non-independent review. “Just as writers are often bad at proofreading their own text, programmers are bad at reading their own code. ... It is often the case that peers are not truly independent reviewers because programmers often have similar training—and thus tend to make the same mistakes.” Bellovin et al., 17 Ohio State Tech. L.J. at 32. Further, even if the program's operation is objective, numerous judgments regarding the appropriate interpretation of data are already baked into the source code, and may not be conducive to detection, comprehension, and analysis except by review of that source code. See Katherine Kwong, The Algorithm Says You Did It: The Use of Black Box Algorithms to Analyze Complex DNA Evidence, 31 Harv. J.L. & Tech. 275, 291 (2017) (noting that “[d]ifferent programs incorporate subtly different choices into their algorithms about how to interpret data, which can yield different results when analyzing the exact same complex mixture,” and that identification of consequent biases requires a “look at the software”). That is particularly so for a probabilistic genotyping program, whose output is not conducive to independent calculation or otherwise verifiable with precision like other analyses. See Christopher D. Steele & David J. Balding, Statistical Evaluation of Forensic DNA Profile Evidence, 1 Ann. Rev. Stat. & Its Application 361, 380 (2014) (explaining that a likelihood ratio “expresses our uncertainty about an unknown event and depends on modeling assumptions that cannot be precisely verified in the context of noisy ... data”).

As to judicial opinions, we note that eighteen courts have rejected calls to allow independent evaluation of TrueAllele's source code, many of which did so after the issues with STRmix and FST came to light. But critically, prior determinations of reliability in other jurisdictions entailed no scrutiny of computer science or source code.21 Instead, the courts depended in large part on Dr. Perlin's own testimony and the existing validation studies which, even if diligently conducted and sound, were not truly independent and did not even evaluate the source code.

The first court to address the question of admissibility was Commonwealth v. Foley, 38 A.3d 882, 889-90 (Pa. Super. Ct. 2012), where the court accepted Dr. Perlin's assertion that validation studies are the best tests of the reliability of source codes. The court reasoned that “scientists can validate the reliability of a computerized process even if the ‘source code’ underlying that process is not available to the public,” emphasizing that making the source code available would have market consequences. Id. at 889. The court reasoned also that TrueAllele “ha[d] been tested and validated in peer-reviewed studies,” citing two studies that had been “published in peer-reviewed journals” and thus “reviewed by other scholars in the field.” Id. at 889-90. At that time, in 2012, TrueAllele had been the subject of two studies, one laboratory-generated validation study, conducted and authored by Dr. Perlin himself,22 and one casework validation study, which was also co-authored by Dr. Perlin.23 The court made no mention of the fact that Dr. Perlin was involved in both the validation studies conducted up to that point. Subsequent courts have placed great emphasis on the observation made in Foley, without further scrutiny, creating an authority “house of cards.” See, e.g., People v. Superior Court (Chubbs), No. B258569, 2015 WL 139069, at *8 (Cal. App. Ct. Jan. 9, 2015); State v. Daniels, No. 2015CF009320AMB (Fla. Cir. Ct. Oct. 31, 2018) (slip op. at 3); State v. Wakefield, 47 Misc.3d 850, 9 N.Y.S.3d 540, 541 (Sup. Ct. 2015); State v. Shaw, No. CR-13-575691 (Ohio C.P. Ct. Cuyahoga Cnty. Oct. 10, 2014) (slip op. at 23); Commonwealth v. Knight, No. 379 WDA 2017, 2017 WL 5951725, at *6 (Pa. Super. Ct. Nov. 29, 2017); State v. Watkins, No. 2017-C-1811 (Tenn. Crim. Ct. Davidson Cnty. Dec. 17, 2018) (slip op. at 13-14).24 Published out-of-state judicial decisions, although persuasive rather than binding, carry great weight, especially after they are cited by other courts. A long line of decisions uniformly in favor of a legal proposition suggests that a legal proposition is generally accepted. We are mindful, however, that in science, the repetition of authority does not automatically establish reliability for purposes of a Frye hearing. The cases identified by the State include a laundry list of admissibility rulings, but to reiterate, none consider whether the TrueAllele source code itself correctly implements its methods, which can only be tested in the manner defendant and amici advocate for here.

We need not risk the same result. Our Supreme Court deemed source-code review of sufficient import to a reliability determination in Chun, 194 N.J. at 68-70, 943 A.2d 114, to order production, and we clearly contemplated the same in Ghigliotty, 463 N.J. Super. at 384-85, 232 A.3d 468, as to the algorithms at issue there. Here, Mr. Adams explained with particularity his need for full access to the code, not simply for visual examination, but for execution and testing, and the terms imposed in Cybergenetics' NDA, such as surveillance, time limits, and restrictions on note-taking and communication, would impede that review. Indeed, Dr. Perlin's own estimate that it would take an individual more than eight years to decipher the code by simple visual inspection belies the State's position that the limited access already offered would be adequate for meaningful review.

In light of the concerns that arise when examining the “black box” validation studies, the out-of-state judicial opinions and orders that have accepted TrueAllele's reliability without source code examination, and errors found in the source codes of the breathalyzer in Chun, FST, and STRmix, judges should examine the reliability of such software with healthy skepticism. Even if the DNA science underpinning probabilistic genotyping analysis has been proven scientifically valid, computer software such as TrueAllele must also properly implement that analysis in its source code; the source code must do as Cybergenetics says it does. We do not suggest that errors found in the source code of other probabilistic genotyping software necessarily means that such errors are present in TrueAllele's source code, but we must ensure that the constitutional rights of criminal defendants are protected by permitting an adversarial review of TrueAllele's source code to ensure that such errors do not also exist there as well. We therefore conclude that there is a rational basis under Frye for production.

(ii) The specificity of the information sought

In his discovery motion, defendant specifically identified the proprietary information sought. In regard to the validation studies, defendant requested discovery of all materials generated, including “[a]ll records and electronic data used as ‘input’ to the TrueAllele system and the software parameters used to analyze this data,” “[a]ll records and electronic data generated by the TrueAllele system and/or laboratory personnel during the course of the study,” “[a]ny analyses ... including bench notes, measurements, statistics, memos, summaries, conclusions, tables, graphics, and any resulting publications, presentations, and reports,” “[a]ll communication relating to the design and results of the study, both within and external to the laboratory,” “[a]ll records of unexpected results, including false positives (false inclusions), false negatives (false exclusions), and the conditions under which the unexpected results were granted,” “[a]ll records of software glitches, crashes, bugs, or errors encountered during the study,” and “[s]oftware version numbers of the components of the TrueAllele[ ] system used for the study.”

Defendant further requested “[s]ource code for the version(s) of the TrueAllele system used in the instant case,” including “all software dependencies such as third-party code libraries, toolboxes, plug-ins, and frameworks,” and “[s]oftware engineering and development materials describing the development, deployment, and maintenance of the version(s) of the TrueAllele software system used in the instant case ..., including the software engineering documents recommended by organizations such as the [IEEE] or the Internal Organization for Standardization (ISO).”

Defendant also specifically requested “[c]ommunication logs and records relating to TrueAllele testing, analysis, and reporting in the instant case, including requests for technical or procedural assistance, bug/crash reports, corrective actions, and software updates” along with “[d]ocumentation of corrective actions for discrepancies and errors.”

Finally defendant requested the forensic casefile generated by the New Jersey State Forensic Laboratory in the matter including: “notes, documents, and data resulting from each phase of testing and analysis,” “documentation related to the evidence collection and examination by the lab, serological testing, DNA extraction, quantitation, amplification, electrophoresis, analysis, and comparison of the samples,” and “all positive and negative controls, allelic ladders, and electronic raw data.”

Defendant provided the necessary information to justify his particularized need for the information requested and limited the scope of his request to that required for an independent analysis and review in this case. The information requested ensures that TrueAllele's source code operates as intended and that any changes to the source code have not negatively affected the intended operation of the program.

(iii) Safeguarding the company's intellectual property—protective order

Entering a protective order for use as part of the Frye hearing will accommodate safeguarding the proprietary information while simultaneously protecting the interests of defendant's liberty and justice. On remand, we direct the judge to issue a protective order that accomplishes these objectives. We leave to the discretion of the judge the details of that task. Two points about that: the judge should retain jurisdiction to enforce the order should that become necessary; and the judge should follow these remarks.

The parties recognize that the entry of a protective order is necessary since they invested “[e]xtensive communications” attempting the negotiate one for the judge to enter. As counsel acknowledged before us, the State made several concessions but “two key areas of disagreement remain.”25 The two areas pertained to liquidated damages for breach of the order, and the terms of the inspection itself. We will generally address both.

First, as to the damages, the State insisted that there be a $1,000,000 automatic civil liability “in the event that the proprietary materials are improperly handled, negligently or otherwise.” Moreover, the State required—on this automatic liability term—that the defense submit to jurisdiction in Pennsylvania and that the defense obtain liability insurance with $3,000,000 in coverage.

But, as the Innocence Project points out, a model protective order from the Northern District of California, whose docket includes among the most complex and financially consequential patent cases in the world, includes no provision for financial liability. U.S. Dist. Ct. for the N.D. of Cal., Model Protective Order for Litigation Involving Patents (Model Protective Order).26 We have not found—and the parties have not provided—any case authorizing disclosure of source code and related proprietary information under a protective order with the restrictions as rigid as Cybergenetics' terms, particularly as to liquidated automatic financial liability for breach of a protective order. Indeed, defendant produced the reasonable protective order issued in Illinois governing access to the source code and related documents by STRmix, and there is no such provision.

Acknowledging that there must be teeth to the protective order, in a proposed order for the judge's consideration, defendant reasonably proposed the following sanctions for breach: “Any person who willfully violates the terms of this Order is subject to civil and criminal sanctions, in addition to any other remedy or proceeding allowed by law.” Defendant did not specifically identify the civil and criminal sanctions, but as counsel for ACDL-NJ pointed out at oral argument, such sanctions could generally include license suspension, disciplinary actions, and civil penalties, just to mention a few. Civil and criminal contempt charges for violating a court order are also a potential consequence for breach. See In re Adoption of N.J.A.C. 5:96 & 5:97, 221 N.J. 1, 17-18, 110 A.3d 31 (2015) (noting that Rule 1:10-3 provides relief to a litigant for another party's failure to abide by a court order); State v. McCray, 458 N.J. Super. 473, 493, 205 A.3d 1178 (App. Div. 2019) (noting that “[t]he goal of the criminal contempt statute[, N.J.S.A. 2C:29-9(a),] is to promote compliance with judicial orders by punishing those who purposely or knowingly fail to comply with those orders”).

Second, as to the terms of the inspection, the State offered to host defense counsel and their experts at the prosecutor's office, which obviates the need for travel, but then prohibited meaningful inspection by permitting only handwritten notes of 170,000 lines of code. According to Dr. Perlin, comprehending the code through such an austere visual inspection would likely take more than eight years. Moreover, the State required the inspection to be supervised and would not allow photographs or copying of any material.

But, as the Innocence Project points out, the model protective order from the Northern District of California includes provisions explicitly permitting certain personnel other than the experts themselves access to the sensitive information, Model Protective Order §§ 7.2, 7.3, and allows the printing of portions of the source code for purposes of analysis, id. § 9(d). Defendant's proposed order, on the other hand, provides reasonable protections, including a prohibition on disclosure to any individual with “any direct or indirect commercial or employment interest in competing software products.” Although a requirement that all notes be handwritten may be included to prevent unauthorized copying and disclosure of source code, such a requirement could be impractical given the form and syntax of source code. Such a requirement may be considered “burdensome in the extreme” because “[m]odern computer source code was never intended to be handwritten even by the original programmer.” Lydia Pallas Loren & Andy Johnson-Laird, Computer Software-Related Litigation: Discovery and the Overly-Protective Order, 6 Fed. Cts. L. Rev. 1, 47 (2012).

As defendant and amici point out, jurisdictions across the country often authorize disclosure of source code in civil litigation to one extent or another on an adequate showing, subject only to a court-issued protective order. See, e.g., WeRide Corp. v. Kun Huang, 379 F. Supp. 3d 834, 854 (N.D. Cal. 2019); Northrop v. Inventive Commc'ns, L.L.C., 199 F.R.D. 334, 335-36 (D. Neb. 2000); Jagex Ltd. v. Impulse Software, 273 F.R.D. 357, 358 (D. Mass. 2011); Dynamic Microprocessor Assocs. v. EKD Comput. Sales, 919 F. Supp. 101, 106 (E.D.N.Y. 1996). The provisions entailed in each order tend to reflect a balance of the rights of the interested parties in light of the circumstances that are attendant to each case.

(iv) Any other relevant factors unique to the facts of the case

Unique to this case is the type of software that is proposed to be used. Probabilistic genotyping differs from traditional methods of DNA analysis in the resulting likelihood ratio that it provides. Rather than providing a result which can be verified against a true value, such as a breathalyzer being compared to a blood draw to ascertain the true blood alcohol content and whether the breathalyzer is within an acceptable margin of error, a likelihood ratio has no precise, independently ascertainable value with which to compare to ensure that the software is providing an acceptable estimation. As Christopher D. Steele and David J. Balding explain,

[l]aboratory procedures to measure a physical quantity such as a concentration can be validated by showing that the measured concentration lies within an acceptable range of error relative to the true concentration. Such validation is infeasible for software aimed at computing a [likelihood ratio] because it has no underlying true value (no equivalent to a true concentration exists). The [likelihood ratio] expresses our uncertainty about an unknown event and depends on modeling assumptions that cannot be precisely verified in the context of noisy [crime scene profile] data.

[Steele & Balding, 1 Ann. Rev. Stat. & Its Application at 380 (fourth alteration in original).]

Additionally, Mr. Adams noted that “[s]ince the likelihood calculations are dependent on the statistical models ... underlying the probabilistic software, any software behaviors affecting the models will necessarily impact the calculated likelihoods and ultimately the reported likelihood ratio.” Because probabilistic genotyping analysis cannot be tested to ensure that is reaching a near-correct result by comparing it to a true value, the closest substitute is to examine the way in which the source code is written to ensure that it functions as the science underpinning probabilistic genotyping necessitates. This is particularly important when even slight changes in the statistical models converted into source code can affect the resulting likelihood ratio. In this way, STRmix and FST serve as important cautionary tales.

Additionally, Drs. Heimdahl and Matthews note that TrueAllele's software integrates multiple scientific disciplines, therefore requiring cross-disciplinary validation to determine reliability. During oral argument, they informed us that each discipline will validate a program under different standards. In particular, V & V in the computer science field cannot be achieved without a thorough examination of the source code which translates validated probabilistic genotyping into executable software. See Natalie Ram, Innovating Criminal Justice, 112 Nw. U. L. Rev. 659, 688 (2018) (noting that “[c]omputer scientists ... have shown that black-box evaluation of systems is the least powerful of a set of available methods for understanding and verifying system behavior. More powerful and effective is white-box testing, in which the person doing a test can see the system's code and uses that knowledge to more effectively search for bugs” (alteration and omission in original) (internal citations and quotation marks omitted)). So, while TrueAllele may be generally accepted in the field of DNA forensics as methodologically sound, such validation may be too narrow, thereby making access to the source code even more important to test whether Dr. Perlin's testimony has gained general acceptance in the computer science community to which it also belongs.

VI.

As technology proliferates, so does its use in criminal prosecutions. Courts must endeavor to understand new technology—here, probabilistic genotyping—and allow the defense a meaningful opportunity to examine it. Without scrutinizing its software's source code—a human-made set of instructions that may contain bugs, glitches, and defects—in the context of an adversarial system, no finding that it properly implements the underlying science could realistically be made. Consequently, affording meaningful examination of the source code, which compels the critical independent analysis necessary for a judge to make a threshold determination as to reliability at a Frye hearing, is imperative.

In summary, defendant articulated a particularized need for the proprietary source code and related information for use at the Frye hearing by (1) demonstrating a rational basis for ordering the State to attempt to produce it, including through expert testimony supporting the claim for disclosure; (2) providing specificity for the information sought; (3) showing through examples from other jurisdictions that the company's intellectual property can be safeguarded by a protective order; and (4) demonstrating that source-code review is particularly crucial to evaluating the unique technology at issue here.
Anything less than full access contravenes fundamental principles of fairness, which indubitably compromises a defendant's right to present a complete defense.

Reversed and remanded for further proceedings. On remand, the judge is directed to compel the discovery of TrueAllele's source code and related materials pursuant to an appropriate protective order, then complete his gatekeeping function at the continued Frye hearing. We do not retain jurisdiction.

38 N.Y.3d 367

Court of Appeals of New York.

The PEOPLE of the State of New York, Respondent,

v.

John WAKEFIELD, Appellant.

No. 3

Decided April 26, 2022

Opinion by Chief Judge DiFore, in which Judges Garcia, Singas and Cannataro concur.

Judge Rivera concurs in result in an opinion, in which Judges Wilson and Troutman concur.

Opinion of the Court

Chief Judge DiFiore

This appeal primarily concerns the admissibility of DNA mixture interpretation evidence generated by the TrueAllele Casework System. We conclude that Supreme Court did not abuse its discretion in finding, following a Frye hearing, that TrueAllele's use of the continuous probabilistic genotyping approach to generate a statistical likelihood ratio—including the use of peak data below the stochastic threshold—of a DNA genotype is generally accepted in the relevant scientific community. We also hold that there was no error in the court's denial of defendant's request for discovery of the TrueAllele software source code in connection with the Frye hearing or for the purpose of his Sixth Amendment right to confront the witness against him at trial.

I.

On April 12, 2010, the victim was found strangled to death in his apartment, with a guitar amplifier cord wrapped around his neck. Several items had been stolen from the victim's home, including a PlayStation 3, a laptop and a distinctive orange duffel bag. Witnesses observed defendant in the company of the victim the weekend of the homicide and defendant admitted to three individuals that he had choked the victim. Defendant did not dispute that he had been present at the victim's home. A separate witness observed defendant with a distinctive orange duffel bag like the one belonging to the victim, attempting to trade a PlayStation and a laptop for drugs. The victim's PlayStation 3 was recovered from the home of a local drug dealer.

At the scene, the police collected DNA samples from several items of evidence and sent them to the New York State Police Forensic Investigation Center (Lab) for PCR DNA typing analysis, using the FBI-selected 15 STR loci standard.1 Defendant's single-source DNA profile was developed from two bottles taken from the victim's home. Relevant to the issues presented here are four DNA profiles developed by the Lab through the samples taken from the front and rear outside collar of the victim's shirt, the victim's dorsal forearm and a section of the amplifier cord used to strangle the victim. The DNA test results developed by the Lab in an electropherogram were then compared to known DNA profiles from defendant and the victim. The Lab concluded that: 1) the two profiles generated from the shirt collar were consistent with at least two donors, one of which was the victim, and defendant could not be excluded as the other contributor; 2) the DNA mixture from the right dorsal forearm was consistent with DNA from the victim, as the major contributor, mixed with at least two additional donors; and 3) the amplifier cord was a mixture of at least two donors, from which the victim could not be excluded as a possible contributor. The results generated from the amplifier cord were not compared to defendant's DNA profile because of the complexity of the mixture.

Since the Lab used an interpretation standard of a stochastic threshold of 50 to 100 relative fluorescence units (RFU), the analyst did not call any alleles based on peaks on the electropherogram below that threshold. As a result, there was insufficient data to allow the Lab to calculate probabilities for the unknown contributors to the DNA mixtures found on the amplifier cord and the front of the shirt collar. The Lab was able to call only 4 out of 15 STR loci and the analyst, using the combined probability of inclusion method,2 generated a statistic that the probability an unrelated individual contributed DNA to the outside rear shirt collar was 1 in 1,088. Using the same number of loci for the profile obtained from the victim's forearm, the analyst generated a statistic that the probability an unrelated individual contributed DNA to the profile was 1 in 422.

The electronic data from the DNA testing of the four samples at issue was then sent to Cybergenetics for additional analysis because its TrueAllele Casework System applies a continuous probabilistic genotyping method of calculating a likelihood ratio—using all of the information generated on the electropherogram, including peaks that fall below a laboratory's stochastic threshold. The likelihood ratio in its modern form was developed by Alan Turing during World War II as a code-breaking method. TrueAllele uses a probability model to assess the values of a genotype objectively. It does not consider a reference sample for any particular DNA profile. Following protocol, once the genotypes were inferred based on mathematical computations from all the data in the electropherograms, the system compared defendant's genotype to all of the statistical genotype possibilities and calculated likelihood ratios as to the presence of defendant's genotype. The ratios were exponentially greater than those generated by the methods employed by the Lab. Specifically, TrueAllele concluded that it was 5.88 billion times more probable that defendant was a contributor to the mixture on the amplifier cord than an unrelated black person, that it was 170 quintillion times more probable that defendant was a contributor to the mixture on the outside rear shirt collar than an unrelated black person, that it was 303 billion times more probable that defendant was a contributor to the mixture on the outside front shirt collar than an unrelated black person, and that it was 56.1 million times more probable that defendant was a contributor to the mixture  on the victim's dorsal forearm than an unrelated black person.

Prior to trial, in March 2014, defendant moved to preclude the introduction of any evidence or testimony derived from the TrueAllele Casework System or, in the alternative, for a Frye hearing to determine the general acceptance of TrueAllele in the relevant scientific community. Defense counsel acknowledged having received from the People approximately 1,500 pages of discovery documents, including reports relating to the DNA analysis, but argued that several additional items “must be disclosed”—specifically, the defense sought the “assumptions” and parameters programmed into the TrueAllele system and the software's source code.

In support of the motion, defendant submitted an affidavit from Ranajit Chakraborty, Ph.D.—a member of the Scientific Working Group on DNA Analysis Methods (SWGDAM)3 and former member of the New York State Commission on Forensic Science DNA Subcommittee. Parenthetically, the DNA subcommittee is charged by Executive Law § 995–b (13) to “assess and evaluate all DNA methodologies proposed to be used for forensic analysis” in the state. Dr. Chakraborty acknowledged that both New York's DNA Subcommittee and the full Commission on Forensic Science (CFS) approved TrueAllele for forensic casework without limitation in 2011 while he was a member. However, since TrueAllele “re-analyzes” the electronic data from another laboratory by including “all allele peak height information including that which extends below” the stochastic threshold, he maintained that it was a novel innovation which had not gained general acceptance in the scientific community. Dr. Chakraborty opined that, when the DNA Subcommittee approved TrueAllele, it had not been presented with evidence of its ability to analyze certain types of complex DNA mixtures and that it “ha[d] not been adequately validated for the type of casework [to which] it is now being applied.” Dr. Chakraborty contended that, in the absence of disclosure of the source code for the software and the underlying assumptions programmed into the system, “TrueAllele cannot be meaningfully validated.”

Supreme Court granted defendant's motion to the extent of ordering a Frye hearing to determine the admissibility of TrueAllele's methodology.

In July 2014, prior to the Frye hearing, defense counsel made a supplemental demand for discovery seeking several items relating to the TrueAllele process including, as relevant here, the TrueAllele source code. The People, who were not in possession of the source code, denied the request as the source code was outside the scope of the former demand discovery provision, CPL 240.20.

The Frye hearing commenced in October 2014 and the People called three witnesses—Dr. Mark Perlin from Cybergenetics, and Dr. Barry Duceman and forensic analyst Jay Caponera from the Lab. Dr. Perlin, who has a medical degree as well as a Ph.D. in both mathematics and computer science, founded Cybergenetics in 1994 and is its Chief Scientist and Executive Officer. He testified that TrueAllele is comprised of two systems—TrueAllele Databank, for interpreting single-source reference samples (used by the New York State Police for the state DNA databank), and TrueAllele Casework, for interpreting more complex DNA mixtures. The True Allele Casework System does not generate a DNA profile, but analyzes the electronic raw data that has already been generated by a forensic crime lab, separates (or deconvolutes) the genotypes and calculates the  likelihood ratio.4 As described at the hearing, the analyst operating the TrueAllele system reviews the raw electronic data and then inputs the data file into the computer, setting parameters such as the number of contributors and the quality of the mixture. The computer then separates the genotypes using the mathematical probability principle of the Markov chain Monte Carlo (MCMC) search to calculate the probability for what the different genotypes could be. The computer conducts a quantitative analysis using all of the electronic data generated in the electropherogram—including patterns and peak heights—as opposed to the qualitative analysis of visual human review, which has a limited data field due to the laboratory's stochastic level.5 Through the mathematical process, the system makes an inference based on the probability of each possibility of the alleles at each locus, then records the genotypes and mixture weights for each contributor. The analyst then looks at the results to determine whether the computer has achieved “concordance” and, if not, the analyst has the option to take additional steps—i.e., ask the computer to model for another donor, allow it to run more cycles or ask it to account for degradation in the sample.

As to general acceptance of the continuous probabilistic genotyping system, the testimony of the People's witnesses established that probabilistic genotyping methods have been recognized by the relevant scientific community such as SWGDAM, the American National Standards Institute and the National Institute of Standards and Technology (NIST) as a valid approach to DNA interpretation—including the fully continuous probabilistic genotyping approach used by TrueAllele. The mathematical and scientific principles underlying the system (MCMC and Bayes’ theorem) are well-established and independent validation of the reliability of the software is available in the form of a free trial that can be used to verify a known sample. Dr. Perlin has presented on the DNA mixture interpretation of TrueAllele at multiple scientific conferences, including internationally, and TrueAllele Casework has been the subject of numerous peer-reviewed published articles in scientific journals. The TrueAllele Casework System had also undergone approximately 25 validation studies—one involving samples created by NIST. Although Dr. Perlin was involved in and coauthored most of the validation studies, his interest in TrueAllele was disclosed as required by the journals who published the studies and the empirical evidence of the reliability of TrueAllele was not disputed. Four validation studies were conducted independently by laboratories.

Two of the independent validation studies were conducted by the New York State Police Lab and were designed to conform to the quality assurance standards of the FBI in order to maintain the Lab's access to the national DNA CODIS database. Jay Caponera testified that he performed the validation studies using complex mixtures of up to four contributors and varying amounts of DNA, including low template samples.6 The system was able to separate the known donor samples from the unrelated profiles provided by staff members—demonstrating that it provides both inculpatory and exculpatory results. In sum, the validation studies found the results generated by TrueAllele were reliable as they were sensitive (identifies the correct person), specific (excludes noncontributors), accurate and reproducible.

The evidence at the hearing further established that NIST used the TrueAllele system to examine the composition and assess the weights of a two-person mixture for its standard reference materials (used by other laboratories to conduct quality assurance).7 Evidence generated by TrueAllele had also been found admissible after hearings conducted in states outside New York, as well as other countries. In addition, the system was used to conduct all forensic casework in the State of Virginia and in Kern County, California.

Defendant did not call an expert witness or introduce any evidence at the hearing. He also did not refute the fundamental mathematical principles of the methodology used by TrueAllele. Rather, in cross-examining the People's witnesses, the defense focused on the fact that the TrueAllele system involves artificial intelligence in its mathematical application by drawing inferences on the data, the “black box” nature of the technology and the small number of independent validation studies conducted without Dr. Perlin's involvement. The defense also emphasized that most forensic laboratories still used the stochastic threshold model. Defendant elicited that some laboratory analysts lack a complete understanding of how the sophisticated mathematical system works and that Dr. Perlin characterized the “main barrier” to implementing the continuous method as the difficulty in educating analysts on the system to the degree they would be able to testify in court.8

Following the hearing, in a lengthy and detailed opinion, Supreme Court found that TrueAllele was generally accepted in the relevant scientific community and denied defendant's motion to preclude testimony and evidence derived therefrom (47 Misc.3d at 859, 9 N.Y.S.3d 540). The court concluded that the quantitative analysis performed by TrueAllele had been empirically tested and found to be reliable and accurate, that it was the subject of publication in scientific journals and favorable peer review, that it was more efficacious than human review using the stochastic method, that it had been validated and the results were reproducible, and that the scientific and mathematical principles upon which it was based had been accepted by the relevant scientific community long ago. The court observed that, notwithstanding that TrueAllele had been in existence since 1999, there was a lack of negative critical work concerning its methods in the scientific community.

II.

Prior to trial, defendant moved for disclosure of the source code in order “to meaningfully exercise his constitutional right to confront his accusers.” He argued that the report generated by TrueAllele was testimonial, that the computer program was the functional equivalent of a laboratory analyst and that the source code was the witness that must be produced to satisfy his right to confrontation. He claimed that Perlin's “surrogate” trial testimony without disclosure of the source code was inadequate—“the TrueAllele Casework System source code itself, and not Dr. Perlin, is the declarant with whom [defendant] has a right to be confronted.” The court denied the motion, finding that the source code was not a witness or testimonial in nature, and that defendant would have the opportunity to confront and cross-examine Dr. Perlin—the analyst and the developer of the software.

Defendant again raised his confrontation argument prior to Dr. Perlin's trial testimony, asserting that the TrueAllele Casework System was the witness and that he needed the source code to effectively cross-examine that witness. When the court questioned how one cross-examines a computer program, defendant represented that, once his experts had the opportunity to review the source code, he would then pose questions to Dr. Perlin based on the experts’ review. The court denied the request, stating that the issue defense counsel raised was a discovery issue and that defendant's ability to cross-examine Dr. Perlin, the developer of the source code, satisfied his right to confrontation.

Both Dr. Perlin and the analyst from the Lab who conducted the electrophoresis testified at trial to the results of the DNA testing and statistical analysis.

Gary Skuse, Ph.D., a professor of biological sciences at the Rochester Institute of Technology, testified at trial as a defense witness. After reviewing 1,278 pages of documents relating to the DNA testing in this case, including the electropherograms, Skuse agreed with the DNA interpretation analysis that defendant's DNA was present in the mixtures found on the shirt collar and amplifier cord and that it was “most likely” present on the victim's forearm. Skuse opined, however, that the DNA may have been present as a result of secondary transfer (not directly from defendant).

The jury convicted defendant of murder in the first degree and robbery in the first degree.

The Appellate Division affirmed (175 A.D.3d 158, 107 N.Y.S.3d 487 [3d Dept. 2019]). The Court held that Supreme Court properly concluded that TrueAllele was generally accepted in the relevant scientific community and rejected defendant's argument that disclosure of the TrueAllele source code was required to properly conduct the Frye hearing. The Court noted that “[t]he record reflects that articles evaluating TrueAllele have been published in six separate forensics journals”; “TrueAllele had undergone approximately 25 validation studies, some of which appeared in peer-reviewed publications”; “[t]he DNA Subcommittee of the New York State Forensic Science Commission offered a binding recommendation that TrueAllele be used by the State Police for its forensic casework” and it was later approved by the full Commission; and “[a]t the time of the Frye hearing, TrueAllele had also been used in various states and had been deemed admissible in Virginia, Pennsylvania and California” (175 A.D.3d at 162–163, 107 N.Y.S.3d 487). As to defendant's claim that the failure to disclose the TrueAllele source code violated his constitutional right to confrontation, the Court concluded that the report generated by TrueAllele was testimonial in nature, but that the source code was not a declarant. The Court cited Dr. Perlin's explanation at the Frye hearing that

“there is human input when utilizing TrueAllele[, including that] a human analyst tells the computer what to download and under what conditions to analyze the data, the analyst tells the computer what questions to ask when interpreting the data and the analyst downloads certain results from the computer, the analyst determines how many ‘runs,’ or cycles, of the data the system will complete and the analyst then makes comparisons to form the likelihood ratios” (175 A.D.3d at 169, 107 N.Y.S.3d 487).

The Court further observed that defendant had the opportunity to confront Dr. Perlin, “his true accuser,” at trial and that defendant did not preserve the alternative argument that the failure to disclose the source code impaired his ability to cross-examine Dr. Perlin as the declarant (175 A.D.3d at 170, 107 N.Y.S.3d 487).

In a concurring opinion, one Justice opined that there was no need to address the relative merit of defendant's arguments as to the violation of his right to confrontation, in light of the absence of “any meaningful attempt by defendant to gain access to, or compel disclosure of, the source code prior to trial” (175 A.D.3d at 173, 107 N.Y.S.3d 487).

A Judge of this Court granted defendant leave to appeal (35 N.Y.3d 1097, 131 N.Y.S.3d 299, 155 N.E.3d 792 [2020]) and we now affirm.

III.

We must address whether the trial court abused its discretion in determining that TrueAllele “is not novel but instead is ‘generally accepted’ under the Frye standard” (47 Misc.3d at 859, 9 N.Y.S.3d 540). Defendant argues that the evidence the People presented at the Frye hearing was insufficient because, absent disclosure of the TrueAllele source code for examination by the scientific community, its “proprietary black box technology” cannot be generally accepted as a matter of law.9 He further asserts that, even if such technology could be generally accepted, the People failed to meet their burden at the hearing, given the dearth of independent validation as a result of Dr. Perlin's involvement in the large majority of studies produced at the hearing.

The well-known Frye test applied to the admissibility of novel scientific evidence (Frye v. United States, 293 F. 1013 [D.C. Cir.1923]) is “whether the accepted techniques, when properly performed, generate results accepted as reliable within the scientific community generally” (People v. Wesley, 83 N.Y.2d 417, 422, 611 N.Y.S.2d 97, 633 N.E.2d 451 [1994]). General acceptance by the relevant scientific community, however, does not require that the procedure be “ ‘unanimously indorsed’ ” (83 N.Y.2d at 423, 611 N.Y.S.2d 97, 633 N.E.2d 451, quoting People v. Middleton, 54 N.Y.2d 42, 49, 444 N.Y.S.2d 581, 429 N.E.2d 100 [1981]).

At issue in Wesley was the general acceptance of DNA evidence—specifically, the restriction fragment length polymorphism (RFLP) methodology, including the assessment of a visual “match” between DNA samples. There, in a concurring opinion, Chief Judge Kaye warned of the “pitfalls of self-validation by a small group” and urged caution in accepting technology that has been validated by individuals with a commercial or professional interest in promoting its use, “developed in commercial laboratories under conditions of secrecy, preventing emergence of independent views” and had not been peer-reviewed (83 N.Y.2d at 439–440, 611 N.Y.S.2d 97, 633 N.E.2d 451 [quotation marks and citations omitted]). Notwithstanding these concerns, Chief Judge Kaye ultimately agreed that, at the time the appeal was decided, “RFLP-based forensic analysis [was] generally accepted as reliable” and those testing procedures were accepted as the standard methodology used in the scientific community until the advent of the PCR STR method used today.

Here, the evidence presented at the Frye hearing established that the relevant scientific community generally accepted TrueAllele's DNA interpretation process and that the continuous probabilistic genotyping approach is more efficacious than human review of the same data using the stochastic threshold. It was undisputed that the foundational mathematical principles (MCMC and Bayes’ theorem) are widely accepted in the scientific community. It was also undisputed that the relevant scientific community was fully represented by those persons and agencies who weighed in on the approach. Although the continuous probabilistic approach was not used in the majority of forensic crime laboratories at the time of the hearing, the methodology has been generally accepted in the relevant scientific community based on the empirical evidence of its validity, as demonstrated by multiple validation studies, including collaborative studies, peer-reviewed publications in scientific journals and its use in other jurisdictions. The empirical studies demonstrated TrueAllele's reliability, by deriving reproducible and accurate results from the interpretation of known DNA samples.

Defendant and the concurrence raise the legitimate concern that the technology at issue is proprietary and the developer of the software is involved in many of the validation studies. This skepticism, however, must be tempered by the import of the empirical evidence of reliability demonstrated here and the acceptance of the methodology by the relevant scientific community. First, Dr. Perlin's hearing testimony established that, in forensic science, “most validation studies are internal and they're not published,” but the FBI's Quality Assurance Standards requires that “a developmental validation for a particular technology” be published (see also NIST, DNA Mixture Interpretation:  A NIST Scientific Foundation Review, at 64 [June 2021 Draft report] https://nvlpubs.nist.gov/nistpubs/ir/2021/NIST.IR.8351–draft.pdf). The interest of the developer was addressed at the Frye hearing in this case and, contrary to defendant's argument,  Dr. Perlin's involvement in many of the validation studies does not preclude a determination of general acceptance as a matter of law.10 The concurrence's claim to the contrary ignores that the performance of validation studies by Dr. Perlin, the State Lab and independent agencies were entirely consistent with the scientific method (see e.g. President's Council of Advisors on Sci. and Tech., Exec. Office of the President, Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature–Comparison Methods, at 46 [2016] https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/PCAST/pcast_forensic_science_report_final.pdf [published after the Frye hearing was held]). Here, unlike in Wesley, there were developer and independent validation studies and laboratory internal validation studies, many published and peer-reviewed. The technology was approved for use by NIST and for forensic casework in New York by both the DNA Subcommittee and the full CFS—entities that were established, in part, in response to the concerns raised in Wesley as to the lack of independent scientific validation of DNA technology (Governor's Approval Mem, Bill Jacket, L 1994, ch 737; Executive Law § 995–b). Importantly, the Lab, in accepting the True Allele Casework System after approval by the CFS, must still conduct additional validation of the program and the hearing testimony indicates that they did so in this case. In addition, NIST's use of the TrueAllele system for its standard reference materials likewise demonstrates confidence within the relevant community that the system generates accurate results.11

Disclosure of the TrueAllele source code was not needed in order to establish at the Frye hearing the acceptance of the methodology by the relevant scientific community.12 First, defendant's initial attempt to obtain the source code was made by a July 2014 supplemental demand under the former demand-discovery provision (former CPL 240.20). Defendant was not entitled to the source code under that provision, as the source code is not a “written report or document” made at the People's request for trial purposes and the proprietary information belonging to Cybergenetics was not in the People's possession or control (see former CPL 240.20[1][c]; former CPL 240.45; People v. Washington, 86 N.Y.2d 189, 630 N.Y.S.2d 693, 654 N.E.2d 967 [1995]; compare People v. DaGata, 86 N.Y.2d 40, 629 N.Y.S.2d 186, 652 N.E.2d 932 [1995] [error to deny defendant access to FBI notes relating to DNA testing not in the People's possession]). As we have previously explained, the former article 240 of the CPL was “a detailed discovery regimen” and “[i]tems not enumerated in article 240 [were] not discoverable as a matter of right unless constitutionally or otherwise specially mandated” (People v. Colavito, 87 N.Y.2d 423, 427, 639 N.Y.S.2d 996, 663 N.E.2d 308 [1996]). Outside of his discovery demand, defendant made no further attempt to demonstrate a particularized need for the source code by motion to the court (see former CPL 240.40[1][c]).

Moreover, defendant's arguments as to why the source code had to be disclosed pay no heed to the empirical evidence in the validation studies of the reliability of the instrument or to the general acceptance of the methodology in the scientific community—the issue for the Frye hearing—and are directed more toward the foundational concern of whether the source code performed accurately and as intended (see Wesley, 83 N.Y.2d at 429, 611 N.Y.S.2d 97, 633 N.E.2d 451). To the extent the testimony at the hearing reflected that the TrueAllele Casework System may generate less reliable results when analyzing more complex mixtures (see also President's Council of Advisors on Sci. and Tech., Exec. Office of the President, Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature–Comparison Methods, at 80 [2016] https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/PCAST/pcast_forensic_science_report_final.pdf [published after the Frye hearing was held]), defendant did not refine his challenge to address the general acceptance of TrueAllele on such complex mixtures or how that hypothesis would have been applicable to the particular facts of this case. As a result, it is unclear that any such objection would have been relevant to defendant's case, where the samples consisted largely of simple (two-contributor) mixtures with the victim as a known contributor (see also NIST, DNA Mixture Interpretation:  A NIST Scientific Foundation Review, at 3 [June 2021 Draft report] https://nvlpubs.nist.gov/nistpubs/ir/2021/NIST.IR.8351–draft.pdf).

IV.

Defendant also argues that the source code for the software is the declarant and that, in the absence of disclosure of the source code, he was deprived of his Sixth Amendment right to confront the witness against him. He maintains that the TrueAllele system involves artificial intelligence and, to some extent, draws its own inferences from the data. He asserts that Dr. Perlin's testimony was therefore that of a surrogate, merely parroting the results of the analyst.

Here, like the Lab reports on the generated DNA profiles, the report created by TrueAllele providing the likelihood ratio that defendant was a contributor to the DNA mixture profile found on the items of evidence is testimonial. The report was prepared by Cybergenetics at the request of the People for purposes of prosecuting  defendant in a pending criminal proceeding. Indeed, the DNA results were sent to TrueAllele precisely because of its “more advanced approach to analyzing the DNA evidence”—i.e., its consideration of patterns and peaks below the stochastic threshold and ability to produce a  higher match statistic. Therefore, the report satisfies our primary purpose test and was testimonial (see People v. John, 27 N.Y.3d at 303–308, 33 N.Y.S.3d 88, 52 N.E.3d 1114).13

However, we reject defendant's novel argument that the source code is the declarant. Even if the TrueAllele system is programmed to have some measure of “artificial intelligence,” the source code is not an entity that can be cross-examined. “[T]he Confrontation Clause provides two types of protections for a criminal defendant: the right physically to face those who testify against him, and the right to conduct cross-examination” (Coy v. Iowa, 487 U.S. 1012, 1017, 108 S.Ct. 2798, 101 L.Ed.2d 857 [1988], quoting Pennsylvania v. Ritchie, 480 U.S. 39, 51, 107 S.Ct. 989, 94 L.Ed.2d 40 [1987] [plurality opinion]). The essential purpose of the provision was to ensure

“ ‘a personal examination and cross-examination of the witness, in which the accused has an opportunity, not only of testing the recollection and sifting the conscience of the witness, but of compelling him to stand face to face with the jury in order that they may look at him, and judge by his demeanor upon the stand and the manner in which he gives his testimony whether he is worthy of belief’ ” (California v. Green, 399 U.S. 149, 157–158, 90 S.Ct. 1930, 26 L.Ed.2d 489 [1970], quoting Mattox v. United States, 156 U.S. 237, 242–243, 15 S.Ct. 337, 39 L.Ed. 409 [1895]).

In Bullcoming v. New Mexico, the United States Supreme Court addressed an argument that a laboratory report could be introduced into evidence through the testimony of an analyst who did not personally perform or observe the test because the gas chromatograph machine, used to analyze the blood alcohol content of the accused's blood sample, was the “true accuser” and the analyst who ran the test was a “mere scrivener” (564 U.S. 647, 659, 131 S.Ct. 2705, 180 L.Ed.2d 610 [2011] [internal quotation marks omitted]). The Court did not expressly address the concept that a machine can be a declarant, but rejected it sub silentio. Instead, the Court focused on the actions taken by the analyst who operated the machine that would be the appropriate subject of cross-examination—e.g., that the blood sample was received in an intact condition, that a particular test was performed on the sample number that corresponded to the case and that the test was performed according to protocol (see 564 U.S. at 660, 131 S.Ct. 2705). In other words, the analyst “certified to more than a machine-generated number” (564 U.S. at 661, 131 S.Ct. 2705; see also John, 27 N.Y.3d at 311, 33 N.Y.S.3d 88, 52 N.E.3d 1114 [declining to “indulge in the science fiction that DNA evidence is merely machine-generated, a concept that reduces DNA testing to an automated exercise requiring no skill set or application of expertise or judgment”]). Similarly, here, the instrument performs its quantitative analysis on electronic data generated by the Lab during the electrophoresis process only after the analyst sets the parameters following a human review of the data. And both the analyst who performed the electrophoresis on the DNA samples and Dr. Perlin, who fully understood the parameters and methodology of the TrueAllele software in its DNA interpretation processes, testified at trial and were subject to cross-examination.

We agree with the Appellate Division that defendant failed to preserve the separate argument that he was entitled to disclosure of the source code in order to fully cross-examine Dr. Perlin as the declarant at trial and, regardless, defendant's argument suffers from the same defect as the request for the source code for purposes of the Frye hearing. Defendant was not entitled to the source code under the former demand discovery statute. After the People refused the demand, defendant failed to make any further attempt to demonstrate a particularized need for the source code by motion to the court (see former CPL 240.40[1][c]; CPL 245.30[3]).

Defendant's remaining arguments, including the arguments raised in his pro se brief, are without merit.

Accordingly, the order of the Appellate Division should be affirmed.

Rivera, J., Concurring in Result

Traditional DNA testing employs human analysis and judgment (see Erin Murphy, The Art in the Science of DNA: A Layperson's Guide to the Subjectivity Inherent in Forensic DNA Typing, 58 Emory LJ 489, 501–508 [2008]). As a consequence, test results may reflect human error and bias, especially in cases involving mixtures of small amounts of DNA from two or more individuals (see e.g. John M. Butler et al., NIST Interlaboratory Studies Involving DNA Mixtures [MIX05 and MIX13]: Variation Observed and Lessons Learned, 37 Forensic Scis Intl: Genetics 81 [2018]; Itiel E. Dror & Greg Hampikian, Subjectivity and Bias in Forensic DNA Mixture Interpretation, 51 Sci & Just 204 [2011]; see also Saul M. Kassin et al., The Forensic Confirmation Bias: Problems, Perspectives, and Proposed Solutions, 2 J Applied Rsch Memory & Cognition 42 [2013]). Enter probabilistic genotyping software—here, a platform branded TrueAllele—a computer-based form of DNA analysis dependent on complex mathematical models and artificial intelligence to derive DNA profiles from complex, multi-person mixtures. While TrueAllele does not completely replace humans, it comes pretty close by eliminating all but the most rudimentary of human participation.

At defendant's trial, the court admitted DNA results developed using the TrueAllele methodology, even though at the time its source code and underlying algorithms were kept from independent evaluators and the defense as trade secrets. The court's decision was an abuse of discretion as a matter of law because it relied on validation studies by interested parties and evaluations founded on incomplete information about TrueAllele's computer-based methodology. Without defense counsel and objective, expert third-party access to and evaluation of the underlying algorithms and source code, the court could not conclude that TrueAllele's brand of probabilistic genotyping was generally accepted within the forensic science community. However, I concur in the result and would affirm defendant's conviction because the evidence against him was overwhelming and the error, even when considered with the other alleged trial errors raised on this appeal, could not have resulted in the guilty verdict.

I.

A.

In 2014, prior to defendant's trial for murder and robbery, the court held a Frye hearing on the admissibility of the prosecutor's DNA evidence developed by use of a proprietary probabilistic genotyping methodology called TrueAllele. The prosecutor sought to establish the likelihood that defendant committed the crimes with results based on TrueAllele's computerized interpretive analysis of mixtures of more than one individual's DNA. In preparation for the hearing, defendant requested but was denied access to TrueAllele's source code, which would have revealed the underlying algorithmic assumptions that replace human judgment.

The majority adequately describes the science-based, analytical challenges presented by mixed DNA samples, like those in issue at defendant's trial, and I do not repeat them here (see majority op. at 374–376, 174 N.Y.S.3d at 316-18, 195 N.E.3d at 23-25). Instead, I focus on the testimonial description of TrueAllele and the studies relied upon by the prosecution and referenced by the majority to support its conclusion that, at the time of the Frye hearing, TrueAllele was generally accepted within the relevant scientific community of forensic scientists.

The most revealing and significant hearing testimony came from Dr. Mark Perlin, M.D., Ph.D., the founder of Cybergenetics and the lead developer of TrueAllele. He testified that TrueAllele Casework was first developed in 1999, after he had already developed and commercialized TrueAllele Databank, a DNA database software.C1 Dr. Perlin stated that TrueAllele is an “expert system,” defined as a “computer system that replicates human expertise,” usually using artificial intelligence, that makes its own inferences. TrueAllele uses a probabilistic genotyping model, which calculates the probabilities that specific allele pairs belong to the genotype of a particular individual in a mixed, low-template DNA sample. Put another way, TrueAllele sifts through scattered genes in a DNA sample and attempts to reconstruct them into a potential genotype, picking from algorithms, equations, and assumptions that can help it to reach a valid conclusion. The core of TrueAllele is a Markov chain Monte Carlo (MCMC) algorithm, an often-used statistical model first developed in the 1950s that determines the probabilities of all the possible probabilities that explain a set of data.

TrueAllele's probabilistic genotyping model, according to Dr. Perlin, was developed in the context of well-known flaws in the field of stochastic, or threshold, modeling in DNA interpretation. Thus, TrueAllele was based on “a need to use normative statistics and mathematical analysis that's used in all other fields where you use all the data, it's examined objectively, you build and validate and test statistical models, use computers and accept that reliable results ... can be learned, taught and used.”

Following the MCMC calculation, TrueAllele “reports out what the genotype is for each contributor at each locus” by “looking at ... the probabilities across all the allele pairs at [a] locus for ... one contributor without particular regard to any particular allele pair.” TrueAllele also accounts for various variables and determines which of those variables to apply in its analysis. Finally—and separately from its inferential processes—TrueAllele compares the “inferred” genotypes against a random population to produce a likelihood ratio based upon a known reference sample, using standard methods of DNA comparison.

Dr. Perlin described an analyst's process of using TrueAllele as follows. Initially, raw data from a .FSA file, produced after an electropherogram is generated in a laboratory from DNA samples, are imported into TrueAllele, at which point TrueAllele analyzes the data “in order to determine the size and heights of the peaks to develop quantitative data.” Then, an analyst asks the computer a question, such as “[a]ssuming that there are two contributors, what are the genotypes of those two contributors.” In general, those questions are preset for ease of use, but trained analysts have the option to ask more complex questions and set additional parameters. After the analyst asks the computer questions, the computer conducts the relevant calculations, drawing on its algorithm and programming to help it determine which equations or models it should use. Generally, as part of the question, the analyst will tell TrueAllele to run a particular number of MCMC calculations. Dr. Perlin testified that the number of calculations can vary depending on the amount of time and computer resources allocated to the case or whether TrueAllele is working with an “easy mixture.” Using TrueAllele, “[t]he computer does the work, we don't, we just set up the variables. And the amount of computation is proportional to the number of variables that you're considering. So ... if it's thinking a lot harder then it's going to take longer.” The analyst also looks for concordance between genotypes across each run of the software and can ask for additional MCMC calculations if necessary. Finally, the software “produces probabilities and then it writes them down.” Thereafter, an analyst can compare the genotype(s) inferred by TrueAllele against known reference samples or DNA databases.

According to Dr. Perlin, unlike threshold DNA models, which exclude data from an analyst's review, TrueAllele is fully continuous, in that it uses “the peak height data and the patterns of those peak rise[s].” In contrast, using a “threshold model,” a human analyst “get[s] a fixed list of possible alleles, and then by looking at all pairwise combinations of those alleles, those will be included genotypes.” Generally, an analyst will input “the alleles or the putative alleles ... into software by hand and then the computer calculates one over the square of the frequencies of those alleles, the sum of it, and then produces a match statistic.”

In comparing TrueAllele to threshold methods of DNA analysis, Dr. Perlin explained that “the computer can do it or the lab can do it.... [H]aving a better way of understanding the data through statistical modeling and computer interpretation is a way of looking at the data you have and extracting more information from it.” Computer analysis can thus go beyond human review of a sample using methods and calculations that a human would be unable to replicate. Essentially, this testimony revealed that TrueAllele extrapolates and processes data to make judgments that supplant human choice, leading to conclusions based on assumptions generated by artificial intelligence.

Dr. Perlin asserted that TrueAllele had gone through 25 unique version numbers for its software since 1999 and had 170,000 lines of source code, inclusive of “the user interface, the way it interacts with databases, [and] the way it solves problems.”C2 TrueAllele had last received a significant update in 2008, when Dr. Perlin and his team “put in more hierarchical modeling to account for even more of the variation” in inferring genotypes. Dr. Perlin testified that the development team “made no effort to preserve” earlier versions of TrueAllele.

TrueAllele underwent an initial validation study in the early 2000s. The software was first used in a criminal trial in Pennsylvania in 2009, and was subsequently found admissible in Virginia, California, Northern Ireland, and Australia. The New York State Police Forensic Investigation Center (FIC) was in the process of onboarding TrueAllele, and a state and international crime lab had both fully adopted TrueAllele for forensic work. Seven other labs had also purchased the TrueAllele software package. The DNA Subcommittee of the New York State Commission on Forensic Sciences and the full Commission had approved TrueAllele for laboratory use. Likewise, the National Institute of Standards and Technology (NIST) had used TrueAllele to verify one of its reference material kits. NIST had also hosted conferences and meetings to encourage laboratories to adopt probabilistic genotyping models like TrueAllele.

Dr. Perlin testified that TrueAllele had undergone approximately 25 validation studies, resulting in several peer-reviewed publications and some prints that were pre-publication at the time of the hearing. Seven of those peer-reviewed studies and two non-peer-reviewed validation studies were entered into evidence.C3 Dr. Perlin either conducted or participated in all of these studies. He also explained that the source code was a “trade secret,” but that the basic mathematical equations underlying TrueAllele were published in the literature. However, “[t]he engineering of the elaboration of [the equations] in more levels of hierarchy” were proprietary. Dr. Perlin argued that source code validation was “not needed” because, in his view, “[t]he question when conducting a validation study had to do with the reliability of the system, the reliability of the software, not the text of the code that underlies that reliable system.”

Barry Duceman, Ph.D., who was then the Director of the Biological Section of FIC, testified that he was in charge of implementing and validating TrueAllele's Databank and Casework systems and obtaining the State Forensic Commission's approval for its use. Dr. Duceman testified to TrueAllele's “increased sensitivity” and “stronger statistical measure,” meaning that TrueAllele can do what a human cannot or will not. He also testified to the threshold approaches currently used in DNA testing, explaining that they resulted in data being excluded from analysis. However, Dr. Duceman testified that his laboratory had yet to use TrueAllele because they were still running further validation studies.

Jay Caponera, a Forensic Scientist 3 with FIC, testified that he had conducted internal validation studies on TrueAllele for FIC. Caponera also discussed his training on the use of TrueAllele and testified to attending conferences hosted by NIST on probabilistic genotyping and the use of TrueAllele in other jurisdictions, as well as his review of the scientific literature. Caponera did not have access to the source code.

The court concluded that TrueAllele was generally accepted in the scientific community, based largely on the validation studies and its use in some laboratories and criminal cases in the United States and abroad (47 Misc.3d 850, 9 N.Y.S.3d 540 [Sup. Ct., Schenectady County 2015]).

B.

The victim was discovered in his home, facedown and propped against the couch. His laptop and videogame console were missing. At trial, the prosecutor argued that defendant strangled the victim and stole his property to sell for drugs. Undisputed evidence established that the victim had died of asphyxia due to ligature strangulation from a guitar cord, and that defendant and victim were known to each other and were together at the victim's home just before the estimated time of the murder.

The prosecution presented testimonial and physical evidence of defendant's guilt. One witness testified that she knew defendant regularly bought drugs with money and by trading property, and that a few months after the victim's death, defendant, while at a “crack house,” told her he had a laptop and videogame system to trade for drugs, and that she saw him with a duffel bag which she identified as the same bag the victim was holding in a photograph taken years prior. The prosecutor also presented evidence that the police recovered the receipt for the videogame console, and a former employee from the maker confirmed that the console went online five days after the victim's death and that the IP address for the console matched the address of a person who another witness identified as a known drug dealer who had supplied one of the drug houses frequented by defendant.

Three witnesses testified that defendant admitted to killing the victim. One witness testified that he knew defendant for decades and that a few days after the murder, defendant showed him a marijuana pipe which he said he took from a man “he had choked and took out” with a guitar cord. The witness came forward to the police months later, after he heard about the murder and a reward for information about the crime. The two other witnesses testified that defendant separately confessed to them when they were each incarcerated with defendant. One of those witnesses testified that defendant admitted to going to the victim's apartment to steal his musical equipment and then killed him when things got out of hand. The other witness testified that he had known defendant for years and that they had smoked crack together. Defendant admitted to killing the victim and stealing his laptop, gaming system, and other items, which he sold for drugs. Defendant also provided the witness with details of his interaction with the victim the evening before the murder, including the type of liquor they were drinking in the apartment and descriptions of the two other people who were present. The court admitted evidence from the victim's garbage of a liquor bottle with defendant's DNA on it which also matched the slang name of the liquor mentioned by defendant. The two individuals who had been with defendant and the victim the night before the victim was killed also testified to seeing defendant drinking alcohol at the victim's apartment.

The prosecution's additional conventional DNA analysis further inculpated defendant. According to the results, neither defendant nor the victim could be excluded from a sample obtained from the rear collar of the victim's shirt containing a mixed profile, with a combined probability of inclusion (CPI) of 1 in 1,088. A DNA sample obtained from a swab taken from the victim's right dorsal forearm was also found to have a CPI of 1 in 422 with regards to defendant and the victim.

As further proof of defendant's guilt, Dr. Perlin testified that, based on his analysis using the TrueAllele methodology, it was 300 million times more probable that defendant contributed to DNA obtained from the guitar cord than an unrelated African American person, one quintillion times more probable that defendant contributed to DNA obtained from the back of decedent's shirt collar than an unrelated African American person, 10 billion times more probable that defendant contributed to DNA obtained from the front of decedent's shirt collar than an unrelated African American person, and 56.1 million times more probable that defendant contributed to DNA obtained from a swab of the cut on decedent's forearm than an unrelated African American person.

Defendant argued that he was entitled to TrueAllele's source code at trial in order to challenge the DNA evidence pursuant to his Sixth Amendment right to confrontation. The court denied the request, concluding that the source code was neither a witness nor testimonial in nature under the Confrontation Clause.

The jury convicted defendant of one count each of first-degree murder and robbery, and the court sentenced him to life imprisonment without parole. The Appellate Division affirmed the judgment of conviction (175 A.D.3d 158, 107 N.Y.S.3d 487 [3d Dept. 2019]). A Judge of this Court granted defendant leave to appeal (35 N.Y.3d 1097, 131 N.Y.S.3d 299, 155 N.E.3d 792 [2020]). I would affirm, but not for the reasons set forth in the majority analysis. Here, the court erred in admitting the TrueAllele results but the error, either alone or considered with defendant's claims of other alleged errors, was harmless.

II.

“The admissibility and scope of expert testimony are subject to the discretion of the trial court, limiting our scope of review to whether the determination to exclude the proffered expert testimony was an abuse of that discretion as a matter of law” (People v. Powell, 37 N.Y.3d 476, 489, 162 N.Y.S.3d 235, 182 N.E.3d 1028 [2021] [citation omitted], citing People v. Lee, 96 N.Y.2d 157, 162, 726 N.Y.S.2d 361, 750 N.E.2d 63 [2001]). New York courts follow the approach set out in Frye v. United States, 293 F. 1013 (D.C. 1923), which asks “whether the accepted techniques, when properly performed, generate results accepted as reliable within the scientific community generally” (People v. Wesley, 83 N.Y.2d 417, 422, 611 N.Y.S.2d 97, 633 N.E.2d 451 [1994]). “In determining whether ...DNA profiling evidence [is] properly admissible, attention must focus on the acceptance of such evidence as reliable by the relevant scientific community” (id.). Under Frye, there must be “consensus[,] [which] ... has been described as ‘a surrogate for determining the reliability of a purported scientific methodology’ ” (People v. Williams, 35 N.Y.3d 24, 37, 124 N.Y.S.3d 593, 147 N.E.3d 1131 [2020], quoting Michael M. Martin et al., New York Evidence Handbook § 7.2.3 at 644 [1997]).

“[T]he particular procedure need not be ‘unanimously indorsed’ by the scientific community but must be ‘generally accept[ed] as reliable’ ” (Wesley, 83 N.Y.2d at 423, 611 N.Y.S.2d 97, 633 N.E.2d 451, quoting People v. Middleton, 54 N.Y.2d 42, 49, 444 N.Y.S.2d 581, 429 N.E.2d 100 [1981]). However, “[a] showing that an expert's opinion has ‘some support’ is not sufficient to establish general acceptance in the relevant scientific community” (Williams, 35 N.Y.3d at 37, 124 N.Y.S.3d 593, 147 N.E.3d 1131, quoting Cornell v. 360 W. 51st St. Realty, LLC, 22 N.Y.3d 762, 783, 986 N.Y.S.2d 389, 9 N.E.3d 884 [2014]). A novel method “should be supported by those with no professional interest in its acceptance” because “Frye demands an objective, unbiased review” (id. at 42, 124 N.Y.S.3d 593, 147 N.E.3d 1131). Indeed, “a proprietary program exclusively developed and controlled by” one individual “is not ‘an appropriate substitute for the thoughtful exchange of ideas ... envisioned by Frye.’ It is an invitation to bias” (id. at 41, 124 N.Y.S.3d 593, 147 N.E.3d 1131, quoting Wesley, 83 N.Y.2d at 441, 611 N.Y.S.2d 97, 633 N.E.2d 451 [Kaye, Ch. J., concurring]).

Defendant is correct that TrueAllele's proprietary algorithm was not generally accepted because its source code had not been tested and assessed as reliable by independent third parties within the relevant forensic scientific community.C4 Although Dr. Perlin has since offered to release the source code in other criminal proceedings, at the time of the Frye hearing here he asserted that TrueAllele's source code was a trade secret and refused to turn it over to defendant (see e.g. State v. Simmer, 304 Neb. 369, 380, 935 N.W.2d 167, 177 [2019] [noting that “Cybergenetics had recently decided to allow defense experts access to the TrueAllele source code, with limitations”]; State v. Baugh, 2019 Ga Super LEXIS 418, *18 [Apr. 29, 2019, No.2017–CR–618, Palmer, J.] [“Dr. Perlin explained that approximately two years ago he agreed to disclose TrueAllele's source code under specific conditions”]; but see United States v. Ellis, 2021 U.S. Dist LEXIS 36176, *1 [WD Pa, Feb. 26, 2021, No. 19–369, Ambrose, J.] [“Cybergenetics was not willing to disclose the source code”], reconsideration denied 2021 WL 1600711, 2021 U.S. Dist LEXIS 78212 [Apr. 23, 2021]). Because there was no opportunity for members of the relevant scientific community to review the source code, and specific software using a complex algorithm cannot be deemed reliable in the scientific community without an independent review of how the software reaches its conclusions—including the inferences made by artificial intelligence—the prosecution failed to satisfy its burden at the Frye hearing. As the President's Council of Advisors on Science and Technology (PCAST) has made clear, while “probabilistic genotyping software programs clearly represent a major improvement over purely subjective interpretation[,] ... they still require careful scrutiny to determine ... whether the software correctly implements the methods” (PCAST, Report to the President, Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature–Comparison Methods 79 [2016], available at https://%1Fobamawhite%1Fhouse.%1Farchives.gov/%1Fsites/%1Fdefault/%1Ffiles/%1Fmicrosites/%1Fostp/%1FPCAST/%1Fpcast_%1Fforensic_%1Fscience_%1Freport_%1Ffinal.pdf  [hereinafter PCAST Report];C5 see also State v. Pickett, 466 N.J. Super. 270, 305–306, 246 A.3d 279, 301 [2021] [“Allowing independent access to the requested information, for the sole purpose of addressing whether the technology underlying the expert testimony is reliable—specifically, whether the source code for that technology is properly implementing the program's design specifications—is obvious”]).C6 Just as a machine may need to be recalibrated and tested regularly for accuracy, the Frye hearing evidence failed to establish that TrueAllele is generally accepted within the forensic scientific community based on its inferential assumptions and its applications of general mathematical principles.

To be sure, Dr. Perlin had published the general mathematical principles behind TrueAllele in the scientific literature at the time of the hearing. But those general principles alone could not serve to validate TrueAllele's unique, proprietary application and inferences that are the basis for its ultimate DNA matching determinations. As Dr. Perlin testified, TrueAllele goes beyond human capabilities in conducting calculations that would be nearly impossible to do by hand; it also substitutes for humans in that it chooses which of its models, algorithms, and equations to run based on the questions posed by human analysts. So while the mathematical principles—such as MCMC—may be commonly accepted and understood, the inferences and interpretive efforts made by TrueAllele that are subject to challenge were unknown and unknowable at the time of the hearing because of Dr. Perlin's refusal to turn over the source code. In effect, disclosure of the formulas, but not the source code, is analogous to a student pointing to the correct equation to use and giving an answer on a mathematics exam, without showing their work. In that case, however, the teacher knows the correct answer to the exam question, notwithstanding whether the student reached the answer incorrectly; here, there is no “generally accepted” correct answer—as a scientific matter—unless the relevant scientific community can determine that TrueAllele was making appropriate choices when doing the math. Put another way, general principles cannot displace the need for an independent source code review of software that makes its own decisions and inferences and where one error in a decimal place or a missing parenthesis, or a choice to assume the existence of an allele based on the computer's judgment of an acceptable probability, can mean the difference between inclusion and exclusion of a person as a DNA donor. Therefore, the court abused its discretion as a matter of law by admitting the results of the TrueAllele's analysis at trial.

The majority's view that the Frye hearing record established acceptance within the scientific community is based on an overly favorable but unsupported view of the testimony and documentary evidence. First, contrary to the majority's suggestion, there is no meaningful involvement by the analyst (see majority op. at 375, 398, 174 N.Y.S.3d at 317, 333, 195 N.E.3d at 24, 40). Dr. Perlin testified that TrueAllele “does the work, we don't” and that it relies on its programming to determine the particular analytical tools that are appropriate based upon the data entered into the system. That means there is all the more reason to provide defendant with access to the source code—the algorithm makes the critical decisions and a human being is an “analyst” in name only, as they merely provide the data that TrueAllele computes and probes. Indeed, this is TrueAllele's defining service (see Kwong at 283 [“Dr. Perlin published an article that called the interpretation method used by most laboratories a ‘random generator’ and argues in interviews that his company's technology produces better probability measurements” (footnote omitted)]).

Second, the majority places undue reliance on evaluations by regulatory entities that had only partial information; the widely accepted nature of TrueAllele's underlying mathematical principles; validation studies in which Dr. Perlin was conflicted due to his financial interest in TrueAllele; and the perceived efficacy of probabilistic genotyping (see majority op. at 381–383, 174 N.Y.S.3d at 321-23, 195 N.E.3d at 28-30). We have previously rejected or been skeptical of each of these types of “supporting evidence.” In Williams, we made clear that “insular endorsement[s]” by executive branch agencies—here, the DNA Subcommittee of the New York State Forensic Science Committee and NIST—cannot “supplant the courts’ obligation to ensure” that a scientific technique has gained general acceptance within the scientific community (35 N.Y.3d at 41, 124 N.Y.S.3d 593, 147 N.E.3d 1131). Notably, the reliance on these agencies by the majority—and a description of them as “the relevant scientific community” for the purposes of the Frye hearing (majority op. at 375–376, ––– N.Y.S.3d at –––– – ––––, ––– N.E.3d at –––– – ––––)—is a jarring turnabout for the Court, given that it is the same view unsuccessfully advocated by a minority in Williams two years ago (see Williams, 35 N.Y.3d at 49–50 and n. 1, 124 N.Y.S.3d 593, 147 N.E.3d 1131 [DiFiore, Ch. J., concurring]). That is particularly true in this appeal because Dr. Perlin criticized the PCAST Report, arguing that “NIST lacks expertise in modern statistical analysis”—which would presumably be a prerequisite to being a part of the relevant scientific community that is qualified to evaluate (Letter from Dr. Mark Perlin, Chief Scientific and Executive Officer, Cybergenetics, to Dr. John Holdren, Co–Chair, PCAST, at 3 [Sept. 16, 2016], available at https://www.cybgen.com/information/newsroom/2016/sep/files/letter.pdf; see Kwong at 289).

The majority—going further—also refers to the American National Standards Institute (ANSI) as being part of the relevant scientific community (see majority op. at 375–376, 174 N.Y.S.3d at 317-18, 195 N.E.3d at 24-25). ANSI, however, is a standards-development organization (SDO), a private industry group that writes “voluntary consensus standards” for use in technological applications (see Am. Socy. for Testing & Materials v. Public.Resource.Org, Inc., 896 F.3d 437, 440–441 [D.C. Cir.2018] [discussing role of SDOs in promulgating standards]; Robert W. Hamilton, The Role of Nongovernmental Standards in the Development of Mandatory Federal Standards Affecting Safety or Health, 56 Tex L Rev 1329, 1341–1368 [1978] [discussing ANSI's history and its procedures for developing industry standards]; see generally Harm Schepel, The Constitution of Private Governance: Product Standards in the Regulation of Integrating Markets 145–52 [2005] [discussing the history of SDOs in the United States]). Including such organizations within the “relevant scientific community,” as the majority does, raises similar concerns about professional interest and profit motive analogous to Dr. Perlin's involvement in the validation studies here (see infra at 400-401, 174 N.Y.S.3d at 334-36, 195 N.E.3d at 41-43). Indeed, Dr. Perlin was on a DNA working group that assisted in the development of a NIST standard approved by ANSI that recognized probabilistic genotyping “as a valid approach to DNA interpretation and reporting” (Perlin et al., New York State TrueAllele Casework Validation Study at 1458, citing NIST Special Publication 500–290, ANSI/NIST–ITL 1–2011, American National Standard for Information Systems: Data Format for the Interchange of Fingerprint, Facial & Other Biometric Information [July 2012], available at https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=910136).

In any case, without the source code, the agencies could not adequately evaluate the use of TrueAllele for this type of DNA mixture analysis, and Dr. Perlin's self-serving statements to the contrary were insufficient to satisfy the prosecution's burden. Williams also rejected the notion that the use of known mathematical principles could satisfy the Frye standard, explaining that such reasoning cannot overcome “the proprietary nature of the [software] or the relatively narrow subsection of the relevant scientific community able to examine and endorse that tool” (Williams, 35 N.Y.3d at 42, 124 N.Y.S.3d 593, 147 N.E.3d 1131).

Third, as for the validation studies, the majority acknowledges that Dr. Perlin conducted or was involved in the majority of them. Given Dr. Perlin's “professional interest in [TrueAllele's] acceptance” (id.), the studies must be viewed with a healthy dose of skepticism and cannot support a conclusion of general acceptance. The majority attempts to minimize this obvious conflict of interest with an unsupported statement that internal validation studies and validation studies conducted by laboratories using the software is “entirely consistent with the scientific method” and because Dr. Perlin's interest was raised at the hearing (majority op. at 382, 174 N.Y.S.3d at 321-22, 195 N.E.3d at 28-29).C7 Contrary to the majority's view, such conflicts do present serious concerns about the integrity of the studies (see Wesley, 83 N.Y.2d at 441, 611 N.Y.S.2d 97, 633 N.E.2d 451 [Kaye, Ch. J., concurring]). As the literature shows, conflicts have long been a source of concern within the scientific community and the standard procedure is to disclose such financial interests, not to assume, as the majority does, that they have little impact on the study itself (see e.g. Singapore Statement on Research Integrity, World Conferences on Research Integrity [2010], https://wcrif.org/statement; Sheldon Krimsky & L.S. Rothenberg, Financial Interest and Its Disclosure in Scientific Publications, 280 JAMA 225 [1998]; Jerome P. Kassirer & Marcia Angell, Financial Conflicts of Interest in Biomedical Research, 329 New Eng J Med 570 [1993]). Put another way, mere disclosure of a conflict of interest and subsequent peer review cannot overcome the potential for bias. Dr. Perlin's professional interest in a favorable assessment of TrueAllele was raised at the hearing, but its significance was discounted, and the validation studies were treated as if conducted by disinterested third parties, confirming, rather than refuting, the court's abuse of its discretion. The remaining two validation studies not involving Dr. Perlin are similarly of little value because they were conducted by the New York State Police FIS, which presented a different form of conflict, as they had already invested significant time and money in adopting TrueAllele for internal use.

Additionally, the majority is incorrect that peer review and publication in a scientific journal implies that “the empirical evidence of the reliability of TrueAllele was not disputed” (majority op. at 376, 174 N.Y.S.3d at 318, 195 N.E.3d at 25). Peer review, while a prerequisite for publication, does not imply that the relevant scientific community has generally accepted a scientific method as reliable. Indeed, as the United States Supreme Court explained in Daubert v. Merrell Dow Pharmaceuticals, Inc.:

“Publication (which is but one element of peer review) is not a sine qua non of admissibility; it does not necessarily correlate with reliability, and in some instances well-grounded but innovative theories will not have been published. Some propositions, moreover, are too particular, too new, or of too limited interest to be published. But submission to the scrutiny of the scientific community is a component of “good science,” in part because it increases the likelihood that substantive flaws in methodology will be detected. The fact of publication (or lack thereof) in a peer reviewed journal thus will be a relevant, though not dispositive, consideration in assessing the scientific validity of a particular technique or methodology on which an opinion is premised” (509 U.S. 579, 593–594 [113 S.Ct. 2786, 125 L.Ed.2d 469] [1993]; see e.g. David H. Kaye et al., The New Wigmore: Expert Evidence § 8.3.2[b][2] [“(P)eer review merely signifies that an article has been subject to some basic scrutiny for methodological and logical flaws. In even the best journals, that scrutiny sometimes is inadequate, and peer review is neither a necessary nor a sufficient condition for demonstrating the validity of a theory or technique” (footnotes omitted)]).

Thus, while peer review might be an appropriate factor for consideration under the federal Daubert standard, it is insufficient under Frye (cf. Powell, 37 N.Y.3d at 514, 162 N.Y.S.3d 235, 182 N.E.3d 1028 [Rivera, J., dissenting]).

In sum, the majority is incorrect that “[i]t was ... undisputed that the relevant scientific community was fully represented by those persons and agencies who weighed in on the [TrueAllele] approach” (majority op. at 381, 174 N.Y.S.3d at 321, 195 N.E.3d at 28). The prosecution's case at the .Frye hearing turned on Dr. Perlin's validation studies, the opinions of executive agencies and private entities, and the mere fact of peer-reviewed publication, none of which are adequate for the reasons I have discussed.

We are left, then, only with the majority's statement that probabilistic genotyping is more efficacious than other DNA typing. That reasoning is circular because the point of accessing the source code is to evaluate the programming and algorithm and to test the very conclusion that TrueAllele is able to provide a reliable assessment of whether and in what ratio an individual's DNA is likely found in DNA mixtures from a crime scene. On this point, the court below and the majority make the same mistake—assuming that if the system appears reliable then we need not know how the system reached its conclusions (see id. at 381, 174 N.Y.S.3d at 321, 195 N.E.3d at 28 [“The empirical studies demonstrated TrueAllele's reliability, by deriving reproducible and accurate results from the interpretation of known DNA samples”]). That is contrary to the basic scientific method, which explores not only the end result but the how and the why of an outcome, or “whether the source code performed accurately and as intended” (id. at 383–384, 174 N.Y.S.3d at 323, 195 N.E.3d at 30). That is no less true in forensic science where, as Dr. Perlin testified, human choice can be based on bias and error. In order to ensure that TrueAllele—created by a human—does not replace one form of bias or flawed assumption with another, third-party independent reviewers must be provided with the source code. As the District Attorney explains, “[t]he development of the computer inferred probability that an allele is present at a specific locus is the major difference between TrueAllele and a human analyst.” That difference is critical to results that either include or exclude an individual as a DNA mixture contributor. Although a human analyst can be asked why they decided a peak was or was not present, or why they believed they could not make a reliable determination of the existence of a peak based on the available data, TrueAllele uses statistical modeling to “infer[ ] the probability that the peak is present.” Without the source code, no independent third party or defendant could challenge TrueAllele's assumptions for what is essentially a mathematical guess—a computer-run, theoretically-based conclusion, but still a guess.

III.

Defendant further argues that the trial court's denial of his request for the source code so that an expert could review it was a violation of his constitutional right to confrontation. The Sixth Amendment Confrontation Clause provides that, “ ‘[i]n all criminal prosecutions, the accused shall enjoy the right ... to be confronted with the witnesses against [them]’ ” (Crawford v. Washington, 541 U.S. 36, 42, 124 S.Ct. 1354, 158 L.Ed.2d 177 [2004]). “[T]he principal evil at which the Confrontation Clause was directed was the civil-law mode of criminal procedure, and particularly its use of ex parte examinations as evidence against the accused” (id. at 50, 124 S.Ct. 1354). Thus, although “the Clause's ultimate goal is to ensure reliability of evidence, ... it is a procedural rather than a substantive guarantee. It commands, not that evidence be reliable, but that reliability be assessed in a particular manner: by testing in the crucible of cross-examination” (id. at 61, 124 S.Ct. 1354). “The upshot is that the role of the trial judge is not, for Confrontation Clause purposes, to weigh the reliability or credibility of testimonial hearsay evidence; it is to ensure that the Constitution's procedures for testing the reliability of that evidence are followed” (Hemphill v. New York, 595 U.S. ––––, ––––, 142 S. Ct. 681, 692, 211 L.Ed.2d 534 [2022]).

“Therefore, ‘[a]s a rule, if an out-of-court statement is testimonial in nature, it may not be introduced against the accused at trial unless the witness who made the statement is unavailable and the accused has had a prior opportunity to confront that witness’ ” (People v. John, 27 N.Y.3d 294, 303, 33 N.Y.S.3d 88, 52 N.E.3d 1114 [2016], quoting Bullcoming v. New Mexico, 564 U.S. 647, 657, 131 S.Ct. 2705, 180 L.Ed.2d 610 [2011]). Under Crawford ’s “primary purpose test for determining whether evidence is testimonial,” this Court has “considered ‘whether the statement was prepared in a manner resembling ex parte examination and ... whether the statement accuses defendant of criminal wrongdoing’ ” (People v. Austin, 30 N.Y.3d 98, 104, 64 N.Y.S.3d 650, 86 N.E.3d 542 [2017], quoting People v. Pealer, 20 N.Y.3d 447, 453, 962 N.Y.S.2d 592, 985 N.E.2d 903 [2013]). “Statements that are considered testimonial include ‘affidavits, ... similar pretrial statements that declarants would reasonably expect to be used prosecutorially ... [and] statements that were made under circumstances which would lead an objective witness reasonably to believe that the statement would be available for use at a later trial’ ” (John, 27 N.Y.3d at 303, 33 N.Y.S.3d 88, 52 N.E.3d 1114, quoting Crawford, 541 U.S. at 51–52, 124 S.Ct. 1354).

The DNA evidence was clearly testimonial and subject to the Confrontation Clause. Forensic reports are testimonial in nature because they are “ ‘functionally identical to live, in-court testimony’ and ... their ‘sole purpose’ [is] evidentiary in nature” (id. at 303–304, 33 N.Y.S.3d 88, 52 N.E.3d 1114, quoting Melendez–Diaz v. Massachusetts, 557 U.S. 305, 310–311, 129 S.Ct. 2527, 174 L.Ed.2d 314 [2009]). Further, it is undisputed that defendant did not have access to the source code, and thus no way to identify and challenge the underlying inferences drawn by TrueAllele and the application of those inferences to the DNA. Defendant could not mount a viable challenge to the conclusions based on the application of TrueAllele's algorithm that linked him to the murder.

Defendant was entitled to cross-examine those who testified about the “sources” of TrueAllele's inferences—i.e., its source code—and “how [it] evaluated those sources in arriving at [its] conclusion” (People v. Stone, 35 N.Y.2d 69, 76, 358 N.Y.S.2d 737, 315 N.E.2d 787 [1974]). Dr. Perlin was a declarant, but defendant could only effectively cross-examine him on his programming assumptions, and any potential biases reflected therein, by reviewing the source code. Although a computer cannot be cross-examined, as Dr. Perlin explained, the computer does the work, not the humans, and TrueAllele's artificial intelligence provided “testimonial” statements against defendant as surely as any human on the stand. Defendant makes a compelling argument that he was entitled to challenge TrueAllele's inferences and choices that led to the DNA interpretations connecting him to the crime in the only way possible: by access to the source code and questioning those who served as the human translators of TrueAllele.

The use of artificial intelligence within our system of justice presents challenging questions and may destabilize our established notions of the dividing line between opinion and uncontestable fact (see e.g. Sonia K. Katyal, Private Accountability in the Age of Artificial Intelligence, 66 UCLA L Rev 54, 62–82 [2019]; Andrea Roth, Machine Testimony, 126 Yale LJ 1972, 2021–2022 [2017]). Courts across the country will decide how our federal and state constitutions may be interpreted in light of continued technological advances and their application in the courtroom. For now, although the questions presented are intellectually challenging and the answers will significantly impact defendants’ rights, I need not resolve thorny esoteric questions in this appeal, nor the more specific question of whether defendant was entitled to the source code under the Confrontation Clause or another constitutional guarantee. As I discuss below, even assuming defendant is correct, any constitutional error or abuse of discretion related to the admission of the TrueAllele results without defendant's access to the source code was harmless.

An error of constitutional dimension is “harmless beyond a reasonable doubt” when “the proof of the defendant's guilt, without reference to the error, is overwhelming,” and there is “no reasonable possibility that the error might have contributed to defendant's conviction” (People v. Crimmins, 36 N.Y.2d 230, 237, 241, 367 N.Y.S.2d 213, 326 N.E.2d 787 [1975]; see People v. Perez, 36 N.Y.3d 1093, 1093–1094, 143 N.Y.S.3d 663, 167 N.E.3d 919 [2021]). Overwhelming proof exists where “the quantum and nature of proof, excising the error, are so logically compelling and therefore forceful in the particular case as to lead the appellate court to the conclusion that ‘a jury composed of honest, well-intentioned, and reasonable [people]’ on consideration of such evidence would almost certainly have convicted the defendant” (Crimmins, 36 N.Y.2d at 241–242, 367 N.Y.S.2d 213, 326 N.E.2d 787).

Here, the totality of the evidence presented at trial, excluding the erroneously admitted TrueAllele DNA evidence, was overwhelming proof of defendant's guilt. There was unchallenged DNA evidence obtained using traditional methods that linked defendant to the crime, with combined probabilities of inclusion—for both decedent and defendant—of 1 in 1,088 for the DNA obtained from decedent's shirt collar, and 1 in 422 for the DNA obtained from decedent's right dorsal forearm. Defendant also did not contest that he had been in decedent's apartment. Moreover, three individuals testified that defendant made inculpatory statements with similar details to each of them about stealing from and killing decedent. Although defendant asserts that those witnesses were unreliable, and that the “jailhouse” witnesses in particular had motive to lie, trial counsel successfully brought those matters to the attention of the jury. Given this evidence, there is “no reasonable possibility” that a jury would have acquitted defendant (id. at 237, 241–242, 367 N.Y.S.2d 213, 326 N.E.2d 787). Thus, because the error was harmless under the more demanding constitutional standard, defendant is not entitled to a new trial (cf. id. at 242, 367 N.Y.S.2d 213, 326 N.E.2d 787 [explaining that an error is harmless under the nonconstitutional standard when “there is a significant probability, rather than only a rational possibility, in the particular case that the jury would have acquitted the defendant had it not been for the error or errors which occurred”]; People v. Easley, 38 N.Y.3d 1010, 1016, 168 N.Y.S.3d 395, 188 N.E.3d 586 [2022, Rivera, J., dissenting] [decided today] [explaining that admission of FST DNA results was not harmless under non-constitutional standard because “(t)here was no eyewitness who saw defendant in possession of the gun, no admission of his guilt, and no video recording depicting him holding the gun at any time”]).C8

V.

The order of the Appellate Division should be affirmed. Although the prosecutor failed to establish that, at the time of the Frye hearing, TrueAllele's methodology was properly validated by disinterested parties with access to the source code, and defendant was denied an opportunity to review the source code because of the developer's proprietary claims, the error, considered alone or with the other alleged constitutional error, was harmless on the facts of this case.

Even though the majority rejects defendant's claim to the source code on the facts of this case, it remains an open question in this Court whether a defendant should be granted access to a proprietary source code under a protective order. This familiar method of ensuring a defendant's right to present a defense would safeguard commercial interests. It provides no help to this defendant, but it is squarely within a court's authority to grant such an order in an appropriate future case.

Houston Federation of Teachers Local 2415, et al., Plaintiffs

v.

Houston Independent School District, Defendant

Order

Stephen Wm. Smith, Magistrate Judge

This case is before the court on plaintiffs' motion (Dkt. 51) for interpretation of the protective order entered September 28, 2015 (Dkt. 47-1). After considering the parties' submissions and argument of counsel at a hearing on March 9, 2016, the motion is granted as set forth below. In accordance with this ruling, plaintiffs' related motions to seal (Dkts. 52, 56) are denied, and SAS's motion for contempt sanctions (Dkt. 54) is denied.

Background

Houston Federation of Teachers (HFT) and individual plaintiffs are suing the Houston Independent School District (HISD) alleging, among other things, that HISD's use of EVAAS to make important employment decisions, including termination and bonus decisions, violates teachers' rights under the Fourteenth Amendment. EVAAS is a proprietary value-added methodology owned by SAS Institute, Inc., a non-party to this suit. Plaintiffs allege that EVAAS uses a "complex and opaque methodology," and that without specific information regarding the rules and methodology used by EVAAS, information not available even to HISD, "it is impossible for a teacher to be able to examine or replicate the analysis, verify the analysis and the resulting score, establish that an alternative value-added model would show that the teacher's students showed academic growth, establish that the assigned score is an unreliable measure, or effectively challenge the analysis performed by SAS® or the resulting EVAAS® score."1

The Protective Order

SAS and HFT negotiated a protective order to facilitate discovery in this case. The protective order permits SAS to designate certain information for "Attorney Eyes Only."

Attorney Eyes Only shall be reserved for SAS information that is deemed to constitute trade secrets and/or proprietary information. For purposes of this order, so-designated information of SAS includes, but is not limited to, source code, models or modeling, analysis of models, programming, design information, methodologies, financial data, operational data, business plans, competitive analyses, personnel files, personal information that is protected by law, and other sensitive information that, if not restricted as set forth in this order, may subject SAS to competitive or financial injury or potential legal liability to third parties.2

The protective order further provides:

Attorney Eyes Only Information shall not be disclosed outside of the context of this litigation, and experts and attorneys may not, outside of the context of this litigation, present observations or conclusions that could not have been made in the absence of their review of Attorney Eyes Only Information.3

The Alleged Violations

After entry of the protective order, plaintiffs' expert, Dr. Jesse Rothstein, was allowed to view SAS source code on a lap top in the office of SAS legal counsel in San Francisco in October 2015. Another plaintiffs' expert, Dr. Audrey Beardsley, did not review the source code. Plaintiffs served the expert reports of Rothstein and Beardsley on HISD in November 2015; the reports have not been filed with the court.

In January 2016, HFT published an "EVAAS Litigation Update."4 This update summarized a portion of Rothstein's expert report in which he concludes that "[a]t most, a teacher could request information about which students were assigned to her, and could read literature -- mostly released by SAS, and not the product of an independent investigation -- regarding the properties of EVAAS estimates. HISD teachers do not have the ability to meaningfully verify their EVAAS scores."5

On January 10, 2016, Beardsley posted an update on her blog "Vamboozled" summarizing what she considered the twelve key highlights of HTF's January 2016 Litigation Update.6 Highlight 12 summarizes Rothstein's conclusion regarding the inability of teachers to verify scores.7

Analysis and Conclusion

SAS concedes that no "Attorney Eyes Only Information" was revealed in the Litigation Update or Beardsley's blog post. However, SAS contends that the publications wrongfully disclose observations or conclusions that could not have been made in the absence of the review of Attorney Eyes Only Information.

SAS interprets the protective order too broadly in this instance. Rothstein's opinion regarding the inability to verify or replicate a teacher's EVAAS score essentially mimics the allegations of HFT's complaint. The Litigation Update made clear that Rothstein confirmed this opinion after review of the source code; but it is not an opinion "that could not have been made in the absence of the review" of the source code. Rothstein has testified by affidavit that his opinion is not based on anything he saw in the source code, but on the extremely restrictive access permitted by SAS.8

In order to prevent further disputes of this type, HFT proposes the addition of the following sentence in the protective order: "Nothing in this Order shall prevent the plaintiffs, their experts and attorneys, from disclosing the conclusions or findings of experts, provided that no Attorney Eyes Only Information is disclosed."

SAS objects that adoption of this language is inappropriate, and would eliminate the negotiated provision of the protective order covering conclusions and observations that could not have been reached without review of Attorney Eyes Only Information. The court disagrees.

A protective order is appropriate to protect proprietary, trade secret information from disclosure. FED. R. CIV. P. 26(c)(1)(G). This particular protective order was drafted to protect conclusions and observations "that could not have been reached without review" of proprietary, trade secret information. The evident purpose of this provision was to guard against the competitive harm that even implicit disclosure of a trade secret might cause. That is a legitimate purpose, but even SAS concedes that neither the Litigation Update nor the related blog post disclosed anything that might legitimately be called trade secret or proprietary, either explicitly or by necessary implication. On the other hand, the overly broad interpretation urged by SAS would inhibit legitimate discussion about the lawsuit, among both the union's membership and the public at large. In order to avoid that unfortunate result, the court agrees that the clarification proposed by HFT, as slightly modified, is appropriate.

The court therefore orders that the following sentence be inserted at the end of section 8 of the protective order:

Nothing in this order shall prevent the plaintiffs, their experts, and attorneys from disclosing the conclusions or findings of experts, provided that no Attorney Eyes Only Information is disclosed, either explicitly or by necessary implication.