lambda.pytorch

[NEW!] Check out our latest work involution in CVPR’21 that bridges convolution and self-attention operators.

PyTorch implementation of LambdaNetworks: Modeling long-range Interactions without Attention.

Lambda Networks apply associative law of matrix multiplication to reverse the computing order of self-attention, achieving the linear computation complexity regarding content interactions.

Similar techniques have been used previously in A²-Net and CGNL. Check out a collection of self-attention modules in another repository dot-product-attention.

Training Configuration

✓ SGD optimizer, initial learning rate 0.1, momentum 0.9, weight decay 0.0001

✓ epoch 130, batch size 256, 8x Tesla V100 GPUs, LR decay strategy cosine

✓ label smoothing 0.1

Pre-trained checkpoints

Citation

If you find this repository useful in your research, please cite

@InProceedings{Li_2021_CVPR,
author = {Li, Duo and Hu, Jie and Wang, Changhu and Li, Xiangtai and She, Qi and Zhu, Lei and Zhang, Tong and Chen, Qifeng},
title = {Involution: Inverting the Inherence of Convolution for Visual Recognition},
booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2021}
}

@inproceedings{
bello2021lambdanetworks,
title={LambdaNetworks: Modeling long-range Interactions without Attention},
author={Irwan Bello},
booktitle={International Conference on Learning Representations},
year={2021},
}