Overview

Perhaps the most interesting emergent phenomena in nature are the functions of the brain. You start with what is essentially a bunch of neurons and you end up with a thinking thing. We will try to get a feel for how this is possible by examining simple models of learning based on computational analogues of physical neurons.

**Template For Solutions: ** https://docs.google.com/document/d/14-3JKGK7AHgH68CTd0TJkSYQnlGBapsy9xLeYhiU00I/edit?usp=sharing

Computational models of neurons are known as artificial neural networks, or sometimes just neural nets. While neural nets are grossly simplified models of the brain, they have become useful tools in computer science, engineering, and science to solve a variety of interesting and challenging problems. In this unit, we will look at several artificial neural networks in our investigation of learning.

This assignment is graded out of 100. The Hopfield network and page 1 of the diffusion models make up those 100 points. Everything marked extra credit is added on top of that.

• 1. Hopfield Networks (50% of the grade)

  • The Hopfield network is a simple artificial neural network designed by physicist John Hopfield to model how neurons memorize information. We will explore the ability of Hopfield networks to memorize simple images.

• 2. Diffusion Models - page 1 (50% of the grade)

  • 2.2 Diffusion Models - page 2 (extra credit: 25%)

  • Diffusion models learn to generate new samples from a probability distribution by slowly adding noise to data and then learning how to run that process backwards. We will build one for a single point and then use it to generate images.

Extra Credit:

• 3. Masked Diffusion: Generative Models for Text (extra credit: 50%)

  • Do diffusion for text instead of continuous numbers, where the noise is replacing words with a mask. You will write the diffusion that lets a pre-trained language model answer questions and then train your own small text diffusion model. Do this after the diffusion pages.

• 4. Building a Transformer from Scratch (extra credit: 25%)

  • Open up the neural network that the masked diffusion page hands you. You will build a transformer one piece at a time, train it to write stories, and then change one line to turn it into a model that writes left to right. Do this after the masked diffusion page.

• 5. Restricted Boltzmann Machines (extra credit: 50%)

  • The restricted Boltzmann machine (RBM) is a statistical physics-inspired neural net that is able to learn complicated probability distributions. We will demonstrate how to use this physics-inspired tool in engineering contexts by training an RBM to learn hand-written digits.

---