hengtao tantaiScrambled, Skipped, Yet Savvy: Unveiling the Resilient Information Flow in Transformer LayersThis paper presents several innovative experiments and findings regarding the internal workings of transformer layers in large language…2d ago2d ago
hengtao tantaiAdam-mini: A More Efficient Deep Learning Optimizer with Half the Memory UsageAdam(W) is currently the mainstream optimizer for training LLMs, but it requires significant memory due to its storage of the first-order…Aug 1Aug 1
hengtao tantaiDynamic Learning Redefining Sequence Modeling: TTT (Test-Time Training) Unleashing Power for…The paper introduces Test-Time Training (TTT) layers as a novel class of sequence modeling layers, designed to enhance the expressive power…Jul 12Jul 12
hengtao tantaiLongRAG: Mastering Complex Queries and Unlocking High-Efficiency Question Answering with Extended…“LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs” presents the introduction of LongRAG, a new framework designed…Jul 7Jul 7
hengtao tantaiMixture of Attention：Optimizing Large Language Models with Adaptive Attention MechanismsThe paper introduces a novel approach called Mixture of Attention (MoA) to enhance the performance and efficiency of Large Language Models…Jul 6Jul 6
hengtao tantaiMeta Releases LLM Compiler: A Compiler Based on Large Language Models with Both Compilation and…In a significant stride towards enhancing compiler optimization, Meta AI has introduced a groundbreaking suite of tools termed the LLM…Jun 29Jun 29
hengtao tantaiNVIDIA’s LLM: Nemotron-4 340B Trained on 98% Synthetic Data, Surpasses Rivals and Matches GPT-4NVIDIA has launched its groundbreaking open-source model, Nemotron-4 340B, potentially revolutionizing the training of large language…Jun 16Jun 16
hengtao tantaiMamba-2 Innovation: State Space Expanded by 8x and Training Speed Increased by 50%, Structured…The paper titled “Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality” explores the…Jun 10Jun 10
hengtao tantaiMambaOut:Rethinking State-Space Models in the Context of Image ClassificationThe paper presents a theoretical analysis to determine the suitability of Mamba architecture for vision tasks, highlighting two key…May 28May 28
hengtao tantaiDeep Dive into xLSTM: The Evolution of LSTM Architecture and PyTorch Code ImplementationLong Short-Term Memory (LSTM) networks have been a staple in handling sequential data due to their ability to retain information over long…May 201May 201
