Benjamin Marie – Medium

Benjamin Marie

Pinned

Published in

TDS Archive

Run Mixtral-8x7B on Consumer Hardware with Expert Offloading

Finding the right trade-off between memory usage and inference speed

Jan 11, 2024

Run Mixtral-8x7B on Consumer Hardware with Expert Offloading

Jan 11, 2024

How Well Does Qwen3 Handle 4-bit and 2-bit Quantization?

Let’s review Qwen3 and check which one you should use

May 3

How Well Does Qwen3 Handle 4-bit and 2-bit Quantization?

May 3

Published in

TDS Archive

2-Bit VPTQ: 6.5x Smaller LLMs While Preserving 95% Accuracy

Very accurate 2-bit quantization for running 70B LLMs on a 24 GB GPU

Jan 31

2-Bit VPTQ: 6.5x Smaller LLMs While Preserving 95% Accuracy

Jan 31

OLMo 2 and Tulu 3 Quantized

State-of-the-art LLMs on your computer

Dec 3, 2024

OLMo 2 and Tulu 3 Quantized

Dec 3, 2024

Published in

Stackademic

Hymba: Combining Attention Heads and SSM Heads within the Same Layer

Faster and better LLMs

Dec 3, 2024

Hymba: Combining Attention Heads and SSM Heads within the Same Layer

Dec 3, 2024

Is BFloat16’s Precision Not Good Enough for RoPE?

Maybe not, according to a new study

Dec 2, 2024

Is BFloat16’s Precision Not Good Enough for RoPE?

Dec 2, 2024

Vocabulary Parallelism for More Efficient LLMs

Distribute the LLM vocabulary

Nov 27, 2024

Vocabulary Parallelism for More Efficient LLMs

Nov 27, 2024

Judge Arena: A New Leaderboard for LLMs as Evaluators

The battle of the judges!

Nov 25, 2024

Judge Arena: A New Leaderboard for LLMs as Evaluators

Nov 25, 2024

Published in

TDS Archive

DPO Full Training vs. LoRA: How Good is LoRA for DPO Training?

One model, two adapters

Nov 20, 2024

DPO Full Training vs. LoRA: How Good is LoRA for DPO Training?

Nov 20, 2024

More Training Tokens => More Difficult to Quantize!

But only with GPTQ and AWQ?

Nov 18, 2024

More Training Tokens => More Difficult to Quantize!

Nov 18, 2024

Benjamin Marie

Benjamin Marie

Ph.D, research scientist in NLP/AI. Exclusive articles and all my AI notebooks on https://kaitchup.substack.com/

Following

Help

Status

About

Careers

Press

Blog

Privacy

Rules

Terms

Text to speech