CS 3033 Special Topics: Emerging Topics in Natural Language Processing

The syllabus is not final and can be changed before the start of the semester in Jan 22 2025.


Course Objective:

We will discuss emerging topics in natural language processing, focusing on understanding the development and usages of large-scale language models. This course is designed for graduate students and highly motivated undergraduates who are interested in NLP research. This course aims to teach and practice: (1) cutting-edge research in natural language processing. (2) how to formulate and evaluate NLP problems and develop solutions for them. (3) how to read and criticize research papers and communicate research both orally and in writing. The course is designed as an active seminar course with a semester-long final project element. Ideally, you should be excited about the intricacies of language understanding and willing to independently explore the literature. The course will cover various aspects of language understanding and generation, such as reasoning, knowledge, and uncertainty.

Course Structure:

The course will have a lecture component and a seminar component. For the lecture component, either the instructor or guest speaker will present a lecture on recent progress in LLM/NLP topics.

For the seminar component, students will prepare and present recent research papers and will lead discussions on it. For this part, we will follow role-playing paper reading seminar. Each week, you are expected to read 2-4 papers carefully.

Everyone are required to read the assigned papers prior to the class. A subset of the students will be assigned “presentation roles” on a rotating basis for which they will need to come prepared. The presentation roles are designed to expose students to the different aspects of being a researcher. By the end of the course, students should be comfortable reading, reviewing, implementing and extending NLP papers.

Throughout this course, you will get to know your classmates, discuss papers and approaches, sometimes struggle to understand challenging papers that is hard to understand. Ideally, you should be excited about the intricacies of language understanding and willing to independently explore the literature.

Prerequisites / Intended audiences:

This is an advanced graduate level course and assumes background in machine learning, programming and basic knowledge of natural language processing. The students are expected to have successfully completed a graduate level course in either (1) natural language processing, (2) deep learning or (3) machine learning. The students should be comfortable with basic fundamentals of machine learning, statistics, and deep learning
programming.

I don’t strictly enforce whether you have taken courses or not, but you should be comfortable with what’s taught in those courses. If you are looking for a lecture-based course with a structured, instructor-driven overview of NLP, this is probably not the right course for you. Consider taking DS-GA.1011 Natural Language Processing with Representation Learning or CSCI-GA.2590 Natural Language Processing. You should be comfortable digesting a research paper from ML/NLP venues.

Note: If you are unclear whether you meet these requirements, please consult the instructor in advance. Auditing is not allowed unless previously discussed with the instructor.

Logistics:

Grading / Workload

% of gradeDue Dates
Class presentation 25%
- Discussion Lead Sign-upJan 24th
- Discussion Lead15%throughout the semester
- Final Project Presentation10%April 30th
Class participation25%
- Provide feedback to classmates intermediate check-ins7%March 24th
- Participation in-class / attendance / role-playing 18%throughout
Mini Writing Assignment10%Feb 26th
Final Project 40%
- One-page Check-in 5%Feb 19th
- Intermediate Check-in (2-3 pages)5%Mar 19th
- Final Write-up (4-6 pages)30%May 3rd

In-class presentation (25% of the total grade)

  • Once or twice during the semester, you will prepare a presentation and lead a discussion of the paper. The instructor will provide the initial list of papers to be presented at each class slot, but if you would like to discuss particular paper, you can discuss it with the instructor to present the paper of your choice.

    You will prepare about 30 minutes material of contents.

  • Discussion Lead (15% of grade) Your presentation will be graded based on:
    • Clarity / Coherence: Each presentation should aim to be relatively self-contained.
    • Comprehensiveness: Covers core contributions of the paper clearly, present work in context of existing work.
    • I highly encourage making slides on Google Slidedeck as it’s easiest to share. But you can use others and send the presentation slides form in PDF at least 24 hour before the class time to the instructor and TA.
  • Final Project Presentation (10% of grade)
    • This will happen in the last week of the class, where you will introduce a short (10 minutes) presentation of one of the writing assignments you have done this semester.

Class participation (25% of the total grade)

  • Provide feedback for classmate’s writing assignments (7% of grade)
    • You will provide a detailed feedback on classmate’s writing assignments.
    • You will be given the draft one week before the final deadline, and should return the feedback in three days, such that they have time to incorporate your feedback.
  • Participation in class discussion (18% of grade)
    • For this seminar class, attendance is mandatory and sessions will not be recorded.
    • You will sign up to help presenter 3-4 times during the semester by selecting one of the “roles” that presenter selects.

Mini Writing Assignment (10% of the total grade)

In this assignment, you study ethical/societal impact of NLP technology. Towards this end, you will do two short writing assignment:

  • Write a version of “AI/NLP Researcher’s Oath”, describing standards and principles for people developing AI/NLP technology. You can think of this something similar to Hippocratic Oath for Doctors. The format is flexible, but you should aim to cover various axis of ethical codes that’s relevant for AI researchers at this day and era. (up to 300 words).
  • You will find one article from the media discussing LLM and its impact on society and write a short commentary about it (up to 300 words). How do you think about the viewpoint presented by the journalist? Do you think they included interviews of adequate parties? What is missing in their articles?

Project (40% of the total grade)

You will complete one out of three track options throughout the semester. For each assignment, you will submit a draft version one week before the deadline, which will be reviewed by your classmates (see class participation section). The final week of the semester, you will present one of your writing assignments to classmates by making a short presentation.

You have to decide between the three options by February 20th. More detailed guideline will be provided later in the course.

  • Track 1: Technical blog post (can be done in 2-3 people group)
    • In this assignment, you are to pick a topic relevant to NLP and write a technical blog article about it. Your post should cover one to three papers in depth and should contain some novel analysis that goes beyond what’s already in those three papers. This can involve reproducing previous results, and running new analysis/codes.
  • Track 2: Final Project (can be done in a group of 2-3 people)
    • If you choose this option, you will design and pursue final project that is relevant to natural language processing. You can choose any topics in NLP (either covered or not covered in the class).
    • You can refer to ACL proceedings or other ML conference proceedings (NeurIPS, ICLR, ICML) for inspiration. Your project can focus on either:
      • A new model architecture for existing problems (a variant of an existing model)
      • A new training, optimization, or evaluation method for existing problems
      • A new application of NLP technology -- here, you will apply an existing model to a new task. Please motivate the task carefully.
      • Experimental and/or theoretical analysis of datasets, approaches, or models.
  • Track 3: Final Project Proposal (individual)
    • This option allows you to write a final project proposal without necessarily carrying out the proposed research yourself. You can think of this as “writing the paper first” practice!
    • As this is only a proposal, you are not limited by computational resources or human resources. Often class project has to be severely limited in scope. Here, you can dream big, assuming you have a large amount of compute resources and even an access to forbidden weights. HOWEVER, you should justify your research idea clearly and carefully. Why, would anyone want to execute this research if they have such resources to spare? What would take to gather such resources? You should have a section that discusses practical limitations of your proposal.

Course Policy / Logistics / Communication

Asking for help

Topics that will be covered in this class

The topic list below is tentative and will be finalized before the week of discussion. Each topic will be discussed for about 4 week, and tentative paper list will be released soon.

Preliminary

Before starting the discussions, we will review recent papers describing architecture and training of base LMs. This will give us background to understand LLMs better for the semester.

Reasoning and cognitive capabilities of LLMs

We will look at recent work that aims to understand and improve the reasoning capabilities of LLMs. This will cover chain-of-thoughts, search methods, alignment methods and using external tools.

Inference time optimization

After the language model is trained, how should one use it? This section will discuss various decoding techniques, aiming to improve computational efficiency or task performance or both. It can cover hardware aware algorithms for optimizations.

Knowledge Augmentation

Language models function as a knowledge base, using knowledge memorized during the pretraining stage. Yet, their memory is limited, and we often augment their memory at inference time through retrieval. We will look into how retrieval models interface with LLMs.

Interaction

In this section, we will look into multi-turn interaction between users and humans. How can we improve LLMs to be better at dialogue tasks that require longer term planning? How does interaction with LLMs impact humans?

Course Schedule

The course schedule can be found here. It will be updated regularly.

Academic Integrity

(Adapted from the website of the College of Arts & Science) Academic honesty means that the work you submit — in whatever form — is original. Obviously, bringing answers into an examination or copying all or part of a paper straight from a book, the Internet, or a fellow student is a violation of this principle. But there are other forms of cheating or plagiarizing which are just as serious — for example, presenting an oral report drawn without attribution from other sources (oral or written), writing a paragraph which, despite being in different words, expresses someone else’s idea without a reference to the source of the idea, or submitting essentially the same paper in two different courses (unless both instructors have given their permission in advance).

Using Artificial Intelligence

The use of artificial intelligence tools (such as ChatGPT) in this class is permitted, but you have to describe (1) how you used them in a few sentences and (2) cite the contents generated by those tools appropriately each time you use them.

If you are considering the use of AI tools but are unsure if you are allowed or the extent to which they may be utilized appropriately, please ask me.

Late day Policy

Each student will have a total of six (6) late days throughout the semester that you can use for final project assignment. You do not need to ask for permissions to use them, but the course staff will keep track of how many days were used. If you exceed the total number of late days, then your assignment will not receive full credit (max 80% of total grade after 1 late day, 60% of total possible grade after 2 late days, 40% of the total possible grade after 3 late days, 0% afterwards).

You cannot use slip day for course presentation or course participation (including peer feedback for final project)).

This budget should cover most emergencies and personal circumstances; I will not approve further extensions, so don’t use up all of your budget on the first assignment deadline.

Attendance and absences

This course relies on in-class discussion and participation, and students who miss a significant amount of class will have difficulty meeting the learning goals of the class. You are expected to attend every single class session.

I will keep track of attendance and this will be part of your participation grade. If you are absent without a valid reason for more than one week, it will impact your overall grade. Religious observance, documented illness (don’t show up sick) or family emergency are grounds for absences to be excused. In case of all absences, you should communicate with me as soon as possible. If you did not show up for classes consistently (or do not participate in class discussions), you will not receive full credit for class participation.

Academic Accommodations

Academic accommodations are available for students with disabilities. The Moses Center website is www.nyu.edu/csd. Please contact the Moses Center for Student Accessibility (212-998-4980 or mosescsd@nyu.edu) for further information. Students who are requesting academic accommodations are advised to reach out to the Moses Center as early as possible in the semester for assistance.

Student Wellness

In a large, complex community like NYU, it's vital to reach out to others, particularly those who are isolated or engaged in self-destructive activities. Student wellness is the responsibility of all of us. The NYU Wellness Exchange is the constellation of NYU’s programs and services designed to address the overall health and mental health needs of its students. Students can access this service 24 hours a day, seven days a week -  wellness.exchange@nyu.edu; (212) 443-9999. Students can call the Wellness Exchange hotline (212-443-9999) or the NYU Counseling Service (212-998-4780) to make an appointment for Single Session, Short-term, or Group counseling sessions.