Courses

EVIDENCE RULES, CASES, PROBLEMS, AND PUZZLES
Part I ::: Part II ::: Part III ::: Part IV ::: Part V

We will be using the Announcements tab for reading and other assignments, The readings are organized into modules.

The content in this course site will grow as the semester progresses.

Questions are welcome. Use email from here if they are personal or of limited interest; use the Discussions list otherwise.

Stay tuned for more remarks.

General Information

• Office Location: Off campus

• Contact Information: david.kaye@yale.edu

• Course Meeting Times: Tuesdays 12:10-2 EST

• Office Hours: Zoom appointments by request. Tuesday afternoons are normally available.

Course Description

A combination of class discussion, guest lectures, self-study, and participation in the standards development process of the Organization of Scientific Area Committees for Forensic Science (OSAC) and related SDOs will introduce students to the major issues and controversies in forensic science and law. Practicum students will become affiliated members of the OSAC Legal Task Group (LTG). As such, they are expected to attend its monthly virtual meetings (unless these meetings conflict with scheduled classes in other courses).

Recommended Books

There is no required textbook, but the following books are recommended for general reading on the issues we will encounter:

• D.H. Kaye et al., The New Wigmore, A Treatise on Evidence: Expert Evidence, 3d ed. 2021, New York: Aspen Publishers

• McCormick on Evidence, R. Mosteller ed., 8th ed. 2020, ch. 20, Thompson Reuters

• B. Robertson et al., Interpreting Evidence: Evaluating Forensic Science in the Courtroom, 2d ed. 2016, Chichester: John Wiley & Sons

Accommodations

Students with documented disabilities should register with Student Accessibility Services (SAS) to request reasonable disability-related accommodation. Supporting documentation Links to an external site. will be required. If approved, SAS will work with the YLS Registrar’s Office to implement accommodations. More information for G&P students can be found online

Schedule of Assignments

Because the course is new and the practicum involves responses to standards as they are opened for development or released for comment, the schedule of assignments in the syllabus is subject to frequent change. Assignments will be posted as Canvas Announcements and within the Modules folder.

Welcome to the course (and here's the first assignment)

Welcome to the Forensic Science Standards Practicum. I am looking forward to meeting everyone next week (even if it has to be by Zoom). The reading assignments for Tuesday's class are in the first two modules. There are various references listed in the first module, but you do not have to read them for this class. I may point to them during the class, but they give more information about consensus standards setting organizations than you really need to know.

The reading for the second module is required. OSAC's Program Manager will be our first guest speaker during this class. Directions for viewing the OSAC new member training video--an optional task--are in this module.

If you want to get ahead, the third module, on the proposed amendment to Federal Rule of Evidence 702, also is posted. It is unlikely that we will reach this module until the second class session, but you should take a look to see what is there. If you want to volunteer for the task group that will be writing a comment on the amendment, let me know so I can give you some early suggestions. The final day for submitting comments to the Advisory Committee is February 16, and we'll need a tentative, preliminary draft before the second class session.

Writing Competition

A notice of a student writing competition for papers in criminal law or procedure is attached. The focus is on California law, but I am posting it in case anyone might be interested.

Second Assignment

For the second class (on 2/1), Ray Wickenheiser, the chair of the OSAC Forensic Science Standards Board (FSSB), will speak about forensic science standards from the standpoint of the laboratories.

As noted at the end of today's class, the proposed amendment and the accompanying note to Federal Rule of Evidence 702 is nearing the end of the Judicial Conference's  comment period. I said I would require everyone to write a few paragraphs on things you see that might be improved (or a statement that the textual change or the note both seem OK as the Advisory Committee has drafted them). But rather than have them written independently and dropped into Canvas, I think it will be more useful if everyone simply jots down some informed thoughts in the discussion thread on the subject. Please do so BEFORE Monday, 1/31. You may comment on other entries there instead of writing your own analysis from scratch, and you are not limited to a single comment. You don't have to write a lot, but we can start the project with a preliminary discussion already in place.

Third Set of Assignments

As noted in class the other day, individuals have been assigned to review and, if appropriate, to comment on one or more ASB standards. The module named Commenting on Standards contains the instructions. The due date is Monday (by noon). Professor Andrea Roth, the LTG Chair, will drop in for a few minutes.

In light of the reactions to the proposed amendment to FRE 702, We will pursue the challenging issue of "error rates" more systematically. The readings in the module on Error Rates and the one on Interpretation present the broad ideas. The chapter on Forensic Statistics in the Courtroom presupposes some knowledge of statistics, and I shall try to explain it more simply in class. Sections 10.3.3 and 10.3.4 deal with "error rates" and probative value, and those are the pages to read. The related readings in the Interpretation module should be easier reading, but because I have been delayed in getting this assignment posted, I do not expect you to have mastered all of this in time for class.

Recap of Academic Component of Class 3

In class, we asked whether the amendment to Rule 702 should require demonstrably low "error rates" for admissibility of all expert evidence, for just scientific expert evidence, or only for forensic-science evidence. We also asked whether experts should be required to estimate an an error probability for a categorical conclusion. We did not resolve this question, but started to unpack the concept of "error rate." Numerical examples showed that the false-positive error rate (the proportion of declared positive associations out of all the true positive cases in an experiment) does not estimate the probability that a declared positive association is false. I noted that an alternative to "error rates" might be a likelihood ratio or Bayesian presentation, but so far those are just words. The next class will give them some content. (See the Interpretation Module).

The Powerpoint (as a pdf file) with the numerical examples is in the Module on Error Rates. It also contains remarks on the phrases "error rates," "controlling standards," and "validity" as used in Daubert, and on measurement error more generally. It includes definitions of the sensitivity and specificity of a binary classification method (think, "It's a match!" versus "It's an exclusion!") and the complementary conditional error probabilities.

Fourth Assignment

The main topic for the academic component of the class is how forensic analysts can and should convey the significance of the observed similarities and dissimilarities of the specimens being compared (fingerprints, DNA, hair, and so on). Methods for doing so include match probabilities, likelihood ratios, and Bayes factors. Readings are in the Interpretation Module.

For the practical component, please improve or repackage the comments in the comment table on the ASB standard or standards that you were assigned. Individual conferences have been scheduled to help you get the comment table into final shape. The first completed table of comments is for ASB 160 Initial Response to Crime Scene (revision). There was not much to comment on, but it might be helpful to look at the end product. LTG members may add to it or raise concerns before it is filed on 2/21, but given the topic of the standard and the fact that only the revisions were open for comment, I don't expect more activity from other quarters to occur. Your final table is due on Tuesday, 2/15 (unless I give you a different date during the conference).

Finally, please note the new discussion topic on a question from a public defender in Washington state. I would be interested in ideas for a reply.

Fifth Assignment

I. Practical Component 

In preparation for the guest speaker (announced separately), please read the complaint in the Williams case posted in the module entitled "Texas Forensic Science Comm'n Project on Firearms-toolmark Evidence."

UPDATE 2/17: At yesterday's OSAC meeting, thirteen ASB-adopted standards were proposed for the OSAC registry. Two merit our attention. They are ADA 1088-22 (on identification via dental comparisons) and ASB 072-19 (on the validation of bloodstain pattern analysis procedures). Inclusion in the registry is automatic unless at least four FSSB members object. I am inclined to object strongly to ASB 072. The assignment with respect to the dental standard is in the discussion list. It asks for you to be a sounding board for a document that the LTG might send to the board during the period allowed for a petition. As next up in to rotation for individual assignments, Greg has a simple research task relating to the table. (See the Discussion tab.)

The more challenging project is the preparation of a petition to the board to review the bloodstain standard, ASB 072, carefully and vote on it -- rather like a petition for a writ of certiorari to the Supreme Court. A petition for review should establish that the adjudication was inadequate because the comments were not actually addressed, that the standard does not meet the requirements of scientific and technical merit, or both.

Materials on the standard are in the folder on Google Suites. The standard has been criticized in the forensic-science literature as vacuous, prompting a rejoinder from the ASB and a rebuttal from the original authors and other individuals.  Everyone should read these three short articles, the standard, the public comments, and subcommittee's adjudication. To get us moving on a petition, Elaine and Marne will be in charge of grouping the public comments and the subcommittee's responses into a list, or perhaps a table, that avoids redundancy and that keeps track of the distinct comments and how they were answered. Then we will be in a better position to see whether and how well all the comments were addressed. If Elaine and Marne can get some or all of this in place by midnight, Monday 2/21, that would be great.

ASB standards due in March already have been assigned to individuals. Time permitting, we'll talk about the training document assigned to Charlotte, so everyone should do an issue-spotting read of it.

II. Academic Component

The Interpretation Module includes three additional--but short--readings on the likelihood or Bayesian alternatives to the traditional binary classification, two-stage approach to evaluating data from comparisons of objects in forensic science. Please look them over before class on 2/22. The DNA standard I mentioned that calls on analysts to address the support the evidence provides for hypotheses rather than the truth or falsity of the hypotheses is entitled Best Practices Recommendations for Forensic DNA Testimony

Links to an external site.. I do not think we need to go through it in any detail, as the comment period is over. However, do read the opinion in Ivey v. Commonwealth, 486 S.W.3d 846 (Ky. 2016), that also has been added to the Module. We will discuss the evaluation of the DNA  evidence in that case with particular attention to the role of the prior odds and the likelihood ratio. Should the prior odds be the subject of expert testimony? If so, what should the expert say about them? Should the expert be confined to giving the likelihood ratio? How should it be explained?

UPDATE 2/17: In light of developments at the FSSB meeting yesterday, it is possible that we will not have time for this topic on 2/22. The class period could be consumed with the Practical Component. Therefore, if you do not have time to do the new readings listed above and to think through these issues, I will understand.

Update to Error Rate Module

The Error Rate Module has yesterday's PowerPoint and, for optional reading, the statistical appendix to the PCAST report that we went over in class. For still more optional reading, there is an article by Dr. Skene on how courts should handle forensic-science evidence that is less than fully validated.  These items are not assigned readings.

Guest Speaker During 2/22 Class

Our guest speaker for 2/22 will be Lynn Garcia, General Counsel to the Texas Forensic Science Commission. She will describe the work of the commission and a research and writing project that the practicum will undertake. Materials on the project will be collected in the new module entitled "Texas Forensic Science Comm'n Project on Firearms-toolmark Evidence." She asks that you read the complaint filed with the Commission that is in the module. Anything else posted there is not yet assigned.

Sixth Assignment

The main topic for the academic component of the class is how forensic analysts can and should convey the significance of the observed similarities and dissimilarities of the specimens being compared (toolmarks, fingerprints, DNA, hair, and so on). Methods for doing so include source attributions (with or without error rates), likelihood ratios, and Bayes factors. Readings are in the Interpretation Module. We will discuss the evaluation of the DNA  evidence in Ivey v. Commonwealth, 486 S.W.3d 846 (Ky. 2016),  with particular attention to the role of the prior odds and the likelihood ratio. Should the prior odds be the subject of expert testimony? If so, what should the expert say about them? Should the expert be confined to giving the likelihood ratio? How should it be explained?

These questions apply to the Texas project. Please read my 2018 article, the Williams complaint, and the full discussion of firearms-toolmark evidence in the PCAST report. We will use the LTG Google Suites folder, Yale Practicum/Firearms Toolmarks,

Links to an external site. to develop our report for the Texas Commission. The folder contains a document entitled "Tasks" with notes on four areas of inquiry for the report and subfolders for each area. Please sign up for the area you want to assume frontline responsibility for in the document "Responsible Parties."

I will be assigning a few more standards for comment or other tasks to individuals by email, but I will keep these assignments to a minimum.

Seventh Assignment

To prepare for the class on Tuesday, 3/8, please read the first three articles (marked with an asterisk) in the Google Suites folder on error rates

Eight Assignment

Now that we have looked at most of the background material for the forensic science and statistical concepts, remaining class sessions will tilt towards to the practical writing projects -- the Texas report and new standards comments. In class on 3/8, we'll clarify the overall outline of the report, how everyone fits into that picture, and deadlines for the writing. In addition, we may look at the new crop of standards for comment and see how best to approach them.

Substantively, we'll delve into the controversy (or is it controversies) over error rates for firearms-toolmark matching and the like as it pertains to the Texas report. That falls within Rachel's area of writing, so I have asked her to give a short summary of the controversy (or is it controversies) over using error rates in testimony. In later sessions, other students will give similar presentations on the topics that they are addressing for the report to the Commission. After hearing from Rachel, the full group will discuss what the report might say based on what we know so far. Professor Jay Koehler, who has written on the subject of estimating error rates, will attend this part of the class.

Ninth Assignment

I have arranged for Thomas Busey, the lead author of a new study that directly bears on our Texas project and on newly proposed standards for reporting conclusions, to come to our next class (4/5/22). He can only stay until 12:45 because he teaches at 1. Therefore, I am not assigning anyone to describe the study, and everyone must read the article carefully before class. Be prepared to jump into questions for Professor Busey. I am curious to see if you think that the study lives up to its title:  "Validating Strength-of-support Conclusion Scales for Fingerprint, Footwear, and Toolmark Impressions."

We'll also look at the standards that will be announced as available for comment in the OSAC Standards Bulletin that comes out on Tuesday.

Tenth and Eleventh Assignments

No new reading for 4/12! I am still revising the draft of the Texas report and will ask for thoughts. We'll also look at the ASB standards open for comment there and the ones from ASB and ASTM that are up for OSAC registry review this month. If you see any that you think deserve input, insert a note into the tables that are accessible through the preceding links.

Next week, 4/19, in response to popular demand, we'll hear from Cybergenetics (the company that markets TrueAllele). Background readings--one case, and article, and some standards--on DNA mixture probabilistic genotyping software are available in the module entitled Probabilistic Genotyping.

Assignment: Introduction to OSAC 

The backdrop to the creation of OSAC is described in John M. Butler, U.S. Initiatives to Strengthen Forensic Science & International Standards in Forensic DNA, 18 Forensic Sci. Int’l 4 (2015), https://www.fsigenetics.com/action/showPdf?pii=S1872-4973%2815%2930028-4. OSAC’s sister organization, the National Commission on Forensic Science, was dissolved in 2017, leaving OSAC as the only federally funded organization directly concerned with standards for the practice of forensic science in the criminal justice system (other than the Department of Justice itself). There have been modifications to the structure of OSAC since the article appeared.

After reading the article, please peruse OSAC’s public website at https://www.nist.gov/osac. Here you will find the governing documents of the organization and various materials that are especially pertinent to the development and review of standards within OSAC. Please read the following ones:

•    OSAC Organizational Priorities & Minimum Topic Requirements for Standards Development

•    OSAC Proposed Standards Comment Adjudication Steps and Instructions for Subcommittee

•    STRP Instructions for OSAC Proposed Standards

We will not discuss these in detail. They are for general orientation and later reference as you work on comments on standards for SDOs or OSAC units. This work will be done on OSAC's Google Suites. You should have received instructions from Donna Sirk at NIST for accessing this site. If you have trouble accessing OSAC Google Suites (https://sites.google.com/nist.gov/osac/home.). or want to become more of an expert at it, you can consider these tips.

Optional Viewing

A “New Member Orientation Recording” is available for viewing on OSAC’s Google Suites. Once there, click on “OSAC Library” in the navigation bar, then on “Training” in the drop-down menu.

OSAC-JPJ-220125.pptx 

Note on the Law of  Voluntary Standards Setting Organizations 

A process that is generally neglected in legal education is the operation of Standards Setting Organizations (SSOs) and Standards Developing Organizations (SDOs) that produce voluntary consensus standards for use by industries and government agencies. See, e.g., Cynthia L. Dahl, When Standards Collide with Intellectual Property: Teaching About Standard Setting Organizations, Technology, and Microsoft v. Motorola, 9 IP Theory (2020), available at https://scholarship.law.upenn.edu/faculty_scholarship/2195. These organizations fall into three broad categories: 

(1) those that are formally recognized by governments (e.g., the European Telecommunications Standards Institute (ETSI) and the International Organization for Standardization (ISO)); (2) “quasi-formal” groups that are typically large and well organized and share many of the characteristics of formally recognized groups, but lack official governmental recognition (e.g., the Institute of Electrical and Electronics Engineers (IEEE) Standards Association and the Internet Engineering Task Force (IETF)); and (3) smaller, privately organized consortia (also known as special interest groups or fora). In addition, the work of individual SSOs is sometimes coordinated at national and international levels. For example, in the U.S., the American National Standards Institute (ANSI) oversees, accredits and establishes policy for national SSOs that wish to develop American National Standards.

Jorge L. Contreras, Technical Standards, Standards-Setting Organizations and Intellectual Property: A Survey of the Literature (With an Emphasis on Empirical Approaches) (February 26, 2019), in Research Handbooks on the Economics of Intellectual Property Law: Analytical Methods (Peter S. Menell & David Schwartz eds. 2019), available at https://ssrn.com/abstract=290054.

Such organizations have their own rules and norms, but they operate in the shadow of antitrust law, and federal procurement regulations can affect their constitution and operation. See OMB Circular A-119, at 12-43 (rev. 2016), available at https://www.whitehouse.gov/wp-content/uploads/2020/07/revised_circular_a-119_as_of_1_22.pdf, In this course, we will not pursue the external law, but we will learn about the internal rules and norms of several SSOs by participating in the development of standards for forensic science. 

References (not assigned for class discussion)

• SSO&IP-Contreras-19.pdf 

• OMB Circular-119-1.pdf 

• ANSI ExSC_012_2021_balance_outreach.pdf 

• ANSI ExSC_013_2021_lack of dominance.pdf 

• S4568(FSSA).pdf 

Assignment: Proposed Amendment to Rule 702

The period for public comments to a Proposed Amendment to Federal Rule of Evidence 702 closes on February 16, 2022. Following a general discussion of the rule and the proposed amendment in class, we will draft comments on the changes (if there is agreement that a comments are merited). This is not a purely academic exercise. If significant comments emerge, members of the class who agree with them will submit them to the Advisory Committee on Evidence of the Judicial Conference.

For general discussion on the proposed amendment during class on February 1, please read the following materials posted in the Canvas module (and start discussing the amendment and the accompanying note on the discussion list as described in the announcement of the assignment):

• Fed. R. Evid. 702 (as enacted 1975);

• Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993);

• Fed. R. Evid. 702 (as amended 2000, with accompanying note);

• President's Council of Advisors on Sci. & Tech., Exec. Office of the President, Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods 20 (2016) (Recommendation 8(C)); 

• Advisory Committee on the Rules of Evidence, Proposed Amendment to Rule 702, and Advisory Committee’s  Note to Proposed Amendment, Aug. 2021, also available via  https://www.uscourts.gov/rules-policies/proposed-amendments-published-public-comment.

Optional supplemental reading:

• WigSupp-21-702Amend'.pdf 

• Kaye-OSACInBrief-July21-pp3-7.pdf 

• 21-NIST-702A'.pdf 

• 18-K-FordLR(Daubert)'.pdf 

Drafting comments will require further research into the advisory’s committee agendas and notes and perhaps the two symposia on the rule that the committee organized.

Assignment: First Set of Comments on Standards 

Various ASB standards are open for comment, and everyone is assigned to read one or more them as shown in the What's Up table in the ASB Comments Due folder. The table contains two links for each standard. One goes to the ASB website where the standard, and in some case, related materials, can be found. The other link goes to an LTG Google Suites folder that contains the same standard and a table for comments on it.

Please put your draft comments on your assigned standard(s) into the table in the standard's folder. Of course, everyone else is free to comment as well 😀.

If you conclude that no comments are necessary or desirable, you should just indicate that in the master table in the Comments Due folder. If the standard is under revision, comments should be limited to the revised text. The red-lined version shows the revisions. The comment resolution on the ASB website indicates the rationale for the revisions.

We'll run through these tables next class, so please make your entries by 12 AM on Monday.

Comments can be editorial or substantive. Writing problems, missing information or detail, and incorrect statements in draft standards are all fair game. The instructions for OSAC Scientific and Technical Review Panels enumerate various desiderata for good standards (although the comments to SDOs are usually in the form of tables rather than a more elaborate report). The folder LTG Guidance Documents contains draft documents describing what the LTG looks for in standards. (If you have ideas for improving these guidance documents, please place them in the Canvas discussion list.)

As mentioned earlier, there are two principal U.S. standards development organizations for forensic science—ASTM International, and the Academy Standards Board (ASB), of the American Association for Forensic Science (AAFS). In addition, the American Dental Association , the National Fire Protection Association, and a few of the older Scientific Working Groups (SWGs) entertain public comments on standards that apply to forensic-science investigations or proof. 

The nomenclature and content of standards from these groups vary. See ASTM Int’l, Form and Style for ASTM Standards (2020); AAFS Standards Board, Manual and Style Guide for ASB Standards, Guidelines, Best Practice Recommendations, and Technical Reports (2021). Although these two manuals not required reading for class discussion, it may be advisable to consult them in commenting on standards in these organizations. They are available online and in the section of OSAC’s Google Suites (https://sites.google.com/nist.gov/osac/home) used by the OSAC Forensic Science Standards Board’s Legal Task Group. To reach the LTG suite, click on “OSAC Units” in the navigation bar, then on “Legal Task Group.” Click on “Documents” and the folders SDO Terms, Procedures & Access / SDO Style Manuals.

The most effective style for writing comments on SDO and OSAC standards differs from that for commenting on proposed legislation or regulations. For some advice, please read Karen Reczek, Best Practice Recommendations for Submitting Comments on Draft Standards, https://womeninstandards.org/best-practice-recommendations-for-submitting-comments-on-draft-standards/.

Happy commenting -- or not, as the standard may be.

• HBFS-Ch10.pdf 

• 03-Class.pdf 

• 16-PCAST-StatAppendix.pdf 

• 04-Class.pdf 

• 18-Skene-Judic'.pdf 

• Kaye-Report-NIST-Ch2'.pdf

• 17-FSI-Morrison&(PCAST)'.pdf

• 17-Evett&-FSI'.pdf

• 18-Thompson&-Judic(Present)'.pdf

• 20-Kaye-JJ(UltOpin).docx

• 21-TFSC-Complaint-IP(Williams)'.pdf 

• 18-K-CaseWLR(Firearms)'.pdf 

• 19-Coble&-FSIG(Ovrview)'.pdf 

• FLca-21-Daniels(TrueA)'.pdf 

• ASB018-20(ValidPGS)'.pdf 

• SWGDAM-15(ValidPGS)'.pdf 

• ULTR-PGS-18.pdf