Differential Transformer with low-rank adapters

The differential Transformer computes attention as the difference of two attention maps, which cancels the noise that ordinary attention spreads over irrelevant context. This project asked how to keep that benefit while making the model cheaper to train and better on long inputs.

We measured where each variant puts its attention with a new cumulative relevance score, adapted the architecture with low-rank adapters for training and parameter efficiency, and integrated hyper-attention into the improved model, benchmarking it against FlashAttention-2.

Main results

  • Proposed a cumulative relevance score and used it, with sparsity scores, to compare vanilla, differential and improved differential Transformers across datasets.
  • Examined how low-rank adapters improve the training and parameter efficiency of the differential Transformer, aiming at long-context performance, lower memory use and lower training cost.
  • Integrated hyper-attention into the improved differential Transformer and benchmarked it against FlashAttention-2.