Natural Language Processing
CS 6340/5340, Spring 2024
MoWe / 11:50am - 1:10pm , WEB L102, Recordings
Instructor: Ana Marasović (she/her)
Through this website we will share:
- Schedule & lecture materials (slides, recordings, readings)
- FAQ before enrolling
Syllabus, announcements, communicating grades, policies, etc. are shared on Canvas. The current page you are on is NOT the syllabus. Please review this syllabus carefully – students are responsible for understanding everything there!
Communication will be done through Piazza. You will turn in your assignments to Gradescope.
Many links on this website can be accessed only with your gcloud.utah.edu account.
Calendar
The calendar and readings are tentative and subject to change.
| Date | Theme | Topic | Lecture Materials | Work due |
|---|---|---|---|---|
| 1/8 | Course Logistics; Intro to NLP | [recording] [slides] [readings] | ||
| 1/10 | Elements of a Simple NLP System | Machine learning basics; Perceptron | [recording] [slides] [readings] | |
| 1/15 | MLK Day (no class) | |||
| 1/17 | cont. | Logistic regression; Tokenization; Morphology | [recording] [slides] [readings] | 1/19 drop deadline |
| 1/22 | cont. | Word embeddings | [recording] [slides] [readings] | |
| 1/24 | cont. | Neural networks basics | [recording] [slides] [readings] | HW1 due (by 1/25) |
| 1/29 | Language Modeling (LM) | n-gram LM; RNNs and their shortcomings | [recording] [slides] [readings] | |
| 1/31 | Machine Translation (MT) | MT; Conditional generation; Decoding methods | [recording] [slides] [readings] | |
| 2/5 | Modern (Large) Language Models | Attention; Self-attention (SA); Multi-headed SA; Positional embeddings; Transformers | [recording] [slides] [readings] | |
| 2/7 | cont. | Pretraining; Decoders (GPT); MLM; Encoders (BERT); Encoder-Decoders (T5); Finetuning (FT) | [recording] [slides] [readings] | HW2 due (by 2/8) |
| 2/12 | cont. | In-context learning; Instruction FT (FLAN-T5) | [recording] [slides] [readings] | |
| 2/14 | cont. | Instruction FT (cont.); Learning from human feedback (LLaMA-2) | [recording] [slides] [readings] | |
| 2/19 | Presidents Day (no class) | |||
| 2/21 | Applications | Taskification and Valid Benchmarks | [recording] [slides] [readings] | HW3 due (by 2/22) |
| 2/26 | Midterm course feedback; Midterm review | [recording] [slides] | ||
| 2/28 | Midterm exam | |||
| 3/4 | Spring Break (no class) | |||
| 3/6 | Spring Break (no class) | |||
| 3/11 | Linguistic Structure Prediction | Part-of-speech tagging, Named entity recognition; Hiden Markov Models (Intro) | [recording] [slides] [readings] | |
| 3/13 | cont. | HMM - Parameter Estimation and Inference (Viterbi algorithm) | [recording] [slides] [readings] | |
| 3/18 | cont. | Syntax; Context free-grammars; Constituency parsing | [recording] [slides] [readings] | |
| 3/20 | cont. | Dependency parsing | [recording] [slides] [readings] | |
| 3/25 | cont. | Semantic parsing; Predicate-argument semantics | [recording] [slides] [readings] | |
| 3/27 | cont. | Coreference resolution; Discourse | [recording] [slides] [readings] | HW4 due (by 3/28) |
| 4/1 | cont. | Probing | [recording] [slides] [readings] | |
| 4/3 | No class | |||
| 4/8 | cont. | Question Answering | [recording] [slides] [readings] | |
| 4/10 | cont. | Summarization | [recording] [slides] [readings] | |
| 4/15 | Beyond English, Beyond Text | Multilinguality and Multimodality | [recording] [slides] [readings] | |
| 4/17 | Responsible NLP | Topics in Responsible AI | [recording] [slides] [readings] | FP due (by 4/21) |
| 4/22 | Final overview | [recording] [slides] | ||
| 4/24 | Reading day (no class) | |||
| 4/26 | Final exam (10:30 am – 12:30 pm) |
FAQ Before Enrolling
How will you be evaluated? [Click to expand!]
Your performance in this course will be evaluated by:
- programming homework assignments (40%)
- final project (30%)
- midterm exam (15%)
- final exam (15%)
Formal pre-requisites? Expected background? [Click to expand!]
The formal pre-requisite is C- or better in CS 3505.
This course is designed for computer science majors. We expect that you:
- …are experienced with programming in Python – we won’t be reviewing your your code directly and we expect you to figure out the implementation side on your own; we are here to help you with understanding concepts and translating ideas into pseudocode,
- …are comfortable with basic calculus, probability, and linear algebra,
- …can quickly pick up machine learning and deep learning,
- …can quickly pick up coding in pytorch,
- …are interested in language and don’t aim to just deepen your machine learning knowledge.
If you completed CS 5353/6353 (Deep Learning), CS 5350/6350 (Machine Learning), CS 6957 (NLP with Neural Networks), or CS 6966/5966 (Local Explanations for Deep Learning Models), we expect you will be able to keep up.
Revisiting/polishing your knowledge. You can prepare by:
-
There are a ton of Python resources for people with some programming experience. Check them out here. I highly recommend these slides/colab (caution: there are some typos) and my colleagues suggest these: 1, 2, 3, and 4.
-
Math and machine learning basics are nicely covered in the first part of the Deep Learning book. Obviously, you can use the same book to familiarize yourself with deep learning. If you learn by coding then you will find this resource helpful Practical Deep Learning for Coders by Fast.ai (3: Neural net foundations; 5: From-scratch model, 13: Backpropagation & MLP, 14: Backpropagation)