Link Search Menu Expand Document
Local Explanations for DL Models

Local Explanations for Deep Learning Models

CS 5966/6966, Fall 2022
TuTh / 09:10AM-10:30AM, 1230 WEB; in-person only

Instructor: Ana Marasović

ana.marasovic@utah.edu

Office hours are at 11:00am Thursdays in Room 2166 in MEB.

Announcements

Please keep track of Teams and Canvas for announcements.

Description

Although neural models have obtained great success across various machine learning (ML) tasks in recent years, they are poorly understood. Consequently, people (including ML experts) have little perception of how to control ML systems they interact with and are typically left wondering:

  • Which part of the input led to assigning this label?
  • How to change the answer to another?
  • In plain English, why is this input assigned this label?
  • Which training examples caused the prediction?

In the first part of this course, we’ll go over methods that aim to address these questions. These methods are developed by ML researchers, usually without involving experts in other fields and human subjects such as doctors or fact checkers.

In the second part of this course, we will revisit evaluation of these methods and focus on how to develop and evaluate methods that best accomplish a specific utility of explanations (e.g., they speed-up information search of a fact checker collaborating with a ML model, reduce cognitive load, etc.). Such methods are developed by considering how people explain between each other (extensively studied by social sciences) and conducting application-grounded human-subject evaluations that are common in HCI.

Methods and examples in this course will heavily focus on Natural Language Processing (NLP) and Computer Vision (specifically images). This course will not cover global methods that analyze models’ average behavior and internals.

You must enroll in the class or audit.

Calendar

Calendar is tentative and subject to change. More details will be added as the semester continues.

Week Date Theme Contents Work due
1 8/23 Introduction Motivation, course overview, and requirements; examples of projects [slides] [readings]
8/25 Background Trasnformer, pretraining-finetuning, data artifacts. [slides] [readings]
2 8/30 Which part of the input led to a prediction? Gradient-based post-hoc explanations [slides] [readings]
9/1 cont. Paper discussion [slides] [readings] Ask discussion questions by Aug 31, 11:59pm
3 9/6 cont. Attention-based post-hoc expanations [slides] [readings]
9/8 cont. Paper discussion [slides] [readings] Project team information due; Ask discussion questions by Sep 7, 11:59pm
4 9/13 cont. Select-then-predict; faithfulness [slides] [readings]
9/15 Which human-interpretable concept led to the prediction? Concept-based explanations; Reviewing [slides] [readings] Papers assigned for review
5 9/20 cont. Paper discussion [slides] [readings] Ask discussion questions by Sep 19, 11:59p
9/22 In plain English, why is this input assigned this label? Free-text explanations [slides] [readings]
6 9/27 cont. Paper discussion [slides] [readings] Ask discussion questions by Sep 26, 11:59pm
9/29 Which training examples caused the prediction? Influence functions [slides] [readings]
7 10/4 cont. Paper discussion [slides] [readings] Review due; Ask discussion questions by Oct 3, 11:59pm
10/6 Which part of the input should be changed to change the prediction to a given label? Contrastive editing; contrastive vector representation [slides] [readings]
8 10/11 Fall Break
10/13 Fall Break
9 10/18 Psychology of human explanations Foundations [slides] [readings]
10/20 Application-grounded, human-subject evaluations of explanations Experimental design, use cases, and challenges [slides] [readings] Project proposals due
10 10/25 cont. Paper discussion [slides] [readings] Ask discussion questions by Oct 24, 11:59pm
10/27 Explainability as a dialog Principles, roadmap, risks, and research opportunities [slides] [readings]
11 11/1 cont. Paper discussion [slides] [readings] Ask discussion questions by Oct 31, 11:59pm
11/3 Trust in AI Prerequisites, causes, and goals of human trust in AI [slides] [readings]
12 11/8 cont. Paper discussion [slides] [readings] Ask discussion questions by Nov 7, 11:59pm
11/10 Advanced topics Few-shot learning [slides] [readings] Project intermediate status report due
13 11/15 cont. Paper discussion Self-attention vs. convolution [slides] [readings] Ask discussion questions by Nov 14, 11:59pm
11/17 Looking back Looking back [slides]
14 11/22 Project Meetings Peer user studies and project feedback Schedule individual meetings on projects
11/24 Thanksgiving Break
15 11/29 Project Presentations Everyone submits their presentations
12/1 Project Presentations
16 12/6 Class cancelled Class cancelled Project report
12/8 Classes End No class

Resources

  • MS Teams. Course communication will be via MS Teams. You should sign into using your uNID and join the “team” for the class by clicking on this link. It’s also the preferred way of communicating with the instructor. Please note that discussion threads and emails are all considered to be equivalent to the classroom, and your behavior in all these venues should conform to the university’s student code.

  • Google Drive. Course materials, including lectures, reading lists, etc., are in a Google Drive folder which has been shared with all students. You can access it (only) with your @gcloud.utah.edu account.

Assignments/Grading

There will be three components to course grades. There are 100 points overall.

  • Project (50 points / 50%).
    • Form a project team and submit project team information (2.5pt).
    • Submit a project proposal (10pt). A document no longer than 2 pages, consisting of the following information (following “Heilmeier Catechism”):
      • What are you trying to do? Articulate your objectives using absolutely no jargon.
      • How is it done today, and what are the limits of current practice?
        • The survey of prior work doesn’t need to be complete, but you should include high-level information about how your problem has been approached in the literature. This will help you think about baselines to compare against in your experiments.
      • What is new in your approach and why do you think it will be successful?
        • A success can also be showing that a published method does not work on a new language, domain, culture, etc.
      • Who cares? If you are successful, what difference will it make?
      • What are the risks?
      • What are the intermediate (Nov 3) and final “exams” (Dec 6) to check for success?
        • You should describe here how you will evaluate your approach.
        • For a data-driven project, you should describe the data that you have at hand. Ideally, you should have all the data that you need – or have a plan that will get you the data within the next couple of weeks.
        • If you are working on a standard data set, there is probably a standard evaluation that has been followed in the literature. You could describe this.
        • For a survey paper, you can commit to saying that success is providing X number of new insights.
        • A successful formalization provides a clearly articulated framework for an old concept.
    • Submit an intermediate status report (10pt). A two page document that says what you have done on the project so far. By this time, you should have interacted with the instructor about your project. The report should describe:
      • The progress you have made towards your goal. (This cannot be just “We collected data”.) (5pt)
      • Details of your plan for rest of the semester (3pt)
      • Pointers to literature (2pt)
    • Presentation (12.5pt): Instead of the final exam, we will have a presentation of the projects in the class. More details as we go along.
    • Submit a project report (15pt): A document no longer than 8 pages, structured like a research paper.
      • What problem did you work on? Why is it interesting?
      • What are the important ideas you explored?
      • What ideas from the class did you use?
      • What did you learn?
      • Results (or for theoretical project, proofs)
      • If you had much more time, how would you continue the project?
      • If your project had two members, your report should explain what each of you did.
      • Each of these components will be equally weighted in the report grade.
  • Paper readings and discussion (30 points / 30%).
    • 20pt: Submit discussion questions before class and participate regularly during class. Note that not every question gives you points (e.g., a question cannot be just “What’s X?” where X is explicitly described in the paper.)
    • 10pt: Lead at least one discussion.
  • Paper review (20 points / 20%) Review ACL/EMNLP reviewing guidelines and write a review for a research paper in explainable AI.

Assignments, including discussion questions, have to be submitted by 11:59pm (Salt Lake City time), unless stated otherwise.

Do not submit assignments via email. Questions for paper discussion have to be submitted to MS Teams. We will announce how to submit other assignments soon.

University of Utah grading scale

Letter Scoring
A 100%-94%
A- 93.9%-90%
B+ 89.9%–87%
B 86.9%–84%
B- 83.9% - 80%
C+ 79.9%–77%
C 76.9%–74%
C- 73.9% - 70%
D+ 69.9%–67%
D 66.9%–64%
D- 63.9% - 60%
E 59.9%–0%

Late Policy

All submissions should be made by the deadline. We will use the timestamp on Canvas/Teams as the submission time. Submissions, except paper discussion, will be accepted up to 24 hours after deadline, but will be assessed a 10% penalty. That is, if your submission is late and scores 90, then your actual grade will be 81=90-9.

Submissions will not be accepted 24 hours after the deadline.

We will be strict about this policy: If the deadline is midnight and you submit the assignment at 12:01 AM, you will face the 10% penalty!

Cheating

Collaboration is encouraged; cheating will not be tolerated.

Honor code for this class

You are encouraged to discuss class materials with your peers. If you want you can form study groups because discussions help understanding. You are also welcome to discuss assignments.

However, you must write your own solutions and code and submit your own solution. Do not copy or ask for assignments from other students or the internet. Do not let someone else copy your submissions either.

If you are caught cheating once, you will receive a failing grade for that submission. If you are caught cheating again, you will fail the class.

For projects, you are free to discuss the project with your classmates, but your work should be your own.

For both assignments and the project, you should cite all sources that you refer to. This includes personal communication, books, papers, websites, etc. Doing so reflects academic integrity.

No double dipping projects across multiple classes

You can not submit the same project to this class and another class that you may be taking at the same time. If you are doing related projects in two different classes, there may be some overlap (e.g. in code libraries, etc.), but they should not be identical. A project that is found to be double-submitted will receive zero credit. If you have questions about this policy, please contact the instructor.

Diversity

It is our intent that students from all diverse backgrounds and perspectives be well-served by this course, that students’ learning needs be addressed both in and out of class, and that the diversity that the students bring to this class be viewed as a resource, strength and benefit. It is our intent to present materials and activities that are respectful of diversity: gender identity sexuality, disability, age, socioeconomic status, ethnicity, race, nationality, religion, and culture. Your suggestions are encouraged and appreciated. Please let us know ways to improve the effectiveness of the course for you personally, or for other students or student groups.

SoC Policies and Guidelines

The class operates under the School of Computing’s policies and guidelines. Among other things, linked page provides information about the academic misconduct policy that we will adhere to.

Also see the College of Engineering guidelines for information about appeals procedures, withdrawal procedures, and adding and repeating courses.

University Drop/Withdrawal Policy

Students may drop a course within the first two weeks of a given semester without any penalties.

Students may officially withdraw (W) from a class or all classes after the drop deadline through the midpoint of a course. A “W” grade is recorded on the transcript and appropriate tuition/fees are assessed. The grade “W” is not used in calculating the student’s GPA.

For deadlines to withdraw from full-term, first, and second session classes, see the U’s Academic Calendar.

University Plagiarism and Cheating Policy

It is assumed that all work submitted to your instructor is your own work. When you have used the ideas of others, you must properly indicate that you have done so.

Plagiarism and cheating are serious offenses and may be punished by failure on an individual assignment, and/or failure in the course. Academic misconduct, according to the University of Utah Student Code,

…Includes, but is not limited to, cheating, misrepresenting one’s work, inappropriately collaborating, plagiarism, and fabrication or falsification of information…It also includes facilitating academic misconduct by intentionally helping or attempting to help another to commit an act of academic misconduct.

For details on plagiarism and other important course conduct issues, see the U’s Code of Student Rights and Responsibilities..

The Content is made available only for your personal, noncommercial educational, and scholarly use. You may not use the Content for any other purpose, or distribute, post or make the Content available to others unless you obtain any required permission from the copyright holder. Some Content may be provided via streaming or other means that restrict copying; you may not circumvent those restrictions. You may not alter or remove any copyright or other proprietary notices included in the Content.

Please see the Code of Student Rights and Responsibilities, Section III.A.5 regarding the use and distribution of class Content and materials. Section III.A.5. prohibits the following: Sale or distribution of information representing the work product of a faculty member to a commercial entity for financial gain without the express written permission of the faculty member responsible for the course. (“Work product” means original works of authorship that have been fixed in a tangible medium and any works based upon and derived from the original work of authorship.)

University Safety Statement

The University of Utah values the safety of all campus community members. You will receive important emergency alerts and safety messages regarding campus safety via text message.

For more safety information and to view available training resources, including helpful videos, visit safeu.utah.edu.

To report suspicious activity or to request a courtesy escort, contact:

Wellness

** Note from the instructor:** Take care of yourself! As a student, you may experience a range of challenges that can interfere with learning, such as strained relationships, increased anxiety, substance use, feeling down, difficulty concentrating and/or lack of motivation. All of us benefit from support during times of struggle. There are many helpful resources available on campus and an important part of having a healthy life is learning how to ask for help. Asking for support sooner rather than later is almost always helpful.

University Statement:

Your personal health and wellness are essential to your success as a student. Personal concerns like stress, anxiety, relationship difficulties, depression, or cross-cultural differences can interfere with a student’s ability to succeed and thrive in this course and at the University of Utah.

Please feel welcome to reach out to your instructor or TA to handle issues regarding your coursework.

For helpful resources to manage your personal wellness and counseling options, contact:

Addressing Sexual Misconduct

Title IX makes it clear that violence and harassment based on sex and gender (which includes sexual orientation and gender identity/expression) is a civil rights offense subject to the same kinds of accountability and the same kinds of support applied to offenses against other protected categories such as race, national origin, color, religion, age, status as a person with a disability, veteran’s status or genetic information.

If you or someone you know has been harassed or assaulted, you are encouraged to report it to university officials:

To file a police report, contact:

If you do not feel comfortable reporting to authorities, the U’s Victim-Survivor Advocates provide free, confidential, and trauma-informed support services to students, faculty, and staff who have experienced interpersonal violence.

To privately explore options and resources available to you with an advocate, contact:

The Americans with Disabilities Act

The University of Utah seeks to provide equal access to its programs, services and activities for people with disabilities.

All written information in this course can be made available in an alternative format with prior notification to the Center for Disability & Access (CDA). CDA will work with you and the instructor to make arrangements for accommodations. Prior notice is appreciated. To read the full accommodations policy for the University of Utah, please see Section Q of the Instruction & Evaluation regulations.

If you will need accommodations in this class, or for more information about what support they provide, contact: