Links to some important research papers or links. I plan to add notes as I go through each topic one by one.

✔ 1. Information theory based (unsupervised) learning

• Invariant Information Clustering
  • Notes
• Mutual Information Neural Estimation
  • Notes
• Deep Infomax
  • Notes
• Learning Representations by Maximizing Mutual Information Across Views
  • Notes
• How Google decoupled MI maximization and representation learning: On Mutual Information Maximization for Representation Learning
  • Notes

✔ 2. Disentangled representations

• Quick overview by Google
  • Notes
• β-VAE, pdf
  • Notes
• Understanding disentangling in β-VAE
  • Notes

Decided not to delve deeper into this topic. It is not mature yet.
* [Disentangling Disentanglement in Variational Autoencoders](https://arxiv.org/abs/1812.02833)
* [Isolating Sources of Disentanglement in Variational Autoencoders](https://arxiv.org/abs/1802.04942)
* [InfoGAN-CR: Disentangling Generative Adversarial Networks with Contrastive Regularizers](https://arxiv.org/abs/1906.06034)
* [Disentangling by Factorising, pdf](https://www.cs.toronto.edu/~amnih/papers/disentangling_nips_ws.pdf)

✔ 3. Self Supervised Learning

• Representation Learning with Contrastive Predictive Coding
• Data-Efficient Image Recognition with Contrastive Predictive Coding
  • Combined notes
• Contrastive Multiview Coding
  • Notes
• Momentum Contrast for Unsupervised Visual Representation Learning
  • Notes
• SimCLR
  • Notes
• MoCo v2: Improved Baselines with Momentum Contrastive Learning
  • The shortest paper I have seen. The authors modified MoCo based on some ideas borrowed from SimCLR.
  • Notes
• Emerging Properties in Self-Supervised Vision Transformers PDF | Official Code

✔ 4. Automatic differentiation

• Automatic differentiation in machine learning: a survey
  • Annotated pdf
• Automatic Reverse-Mode Differentiation: Lecture Notes
  • Annotated pdf
• Reverse mode automatic differentiation

5. NNs and ODEs

• Neural Ordinary Differential Equations
  • Adjoint tutorial
• Augmented Neural ODEs
• Invertible ResNets
• Universal Differential Equations for Scientific Machine Learning

6. Probabilistic Programming

• Probabilistic models of cognition
  • Notes
• The Design and Implementation of Probabilistic Programming Languages
• Composition in Probabilistic Language Understanding

7. Miscellaneous

Memorization in neural networks

• Blog post by BAIR
• Identity Crisis: Paper

Online Learning

• A Modern Introduction to Online Learning

Graph Neural Networks

• A Gentle Introduction to Deep Learning for Graphs, pdf

Normalizing Flows

• Tutorial and implementations for UCB DUL course

Transformers

• Attention is all you need
  • A nice visual introduction
  • Annonated paper
  • Python notebook with illustrations and working code
• Reformer: The Efficient Transformer

Others

• Zero-shot knowledge transfer
• SpecNet
• Deep Learning & Symbolic Mathematics
• Deep Equilibrium Models

8. Theory of neural networks

Lottery tickets

• Lottery ticket hypothesis
• Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
• Rigging the Lottery: Making All Tickets Winners

Others

• What’s Hidden in a Randomly Weighted Neural Network?
• Topological properties of the set of functions generated by neural networks of fixed size
• YOUR CLASSIFIER IS SECRETLY AN ENERGY BASED MODEL AND YOU SHOULD TREAT IT LIKE ONE
• Neural Persistence: A Complexity Measure for Deep Neural Networks Using Algebraic Topology

9. Advanced Variational Inference

• Amortized Population Gibbs Samplers with Neural Sufficient Statistics
• Evaluating Combinatorial Generalization in Variational Autoencoders

10. Causal Inference

• Elements of Causal Inference, pdf
• Causal Inference: What If, pdf
• Theoretical Impediments to Machine Learning With Seven Sparks from the Causal Revolution