Quantize Large Language Models with Just Few Lines of Code — Mengni Wang, Xin He, Yuwen Zhou, Yiyang Cai, Kaokao Lv, Suyue Chen, Wenhua Cheng and Haihao Shen, Intel Corporation As large language models (LLMs) become more prevalent, there is a growing need for quantization methods maintain accuracy while reducing computational costs. …

Creating Your Own Chatbot in Just a Few Minutes — Liang Lv, Xuhui Ren, Xinyu Ye, Kaokao Lv, Qun Gao, Feng Tian, and Haihao Shen, Intel Corporation NeuralChat, a customizable chatbot framework under Intel Extension for Transformers, provides an easy-of-use API to quickly build a chatbot on multiple architectures (e.g., Intel Xeon Scalable Processors and Habana Gaudi Accelerator). NeuralChat is…

Easily Identify the Operators Causing Accuracy Loss — Agata Radys, Zehao Huang, Suyue Chen, and Bartosz Myrcha, Intel Corporation Accuracy loss is a common problem when quantizing deep learning models. There is a tradeoff between reducing the computational and storage requirements of a model while maintaining accuracy. …

Faster, High-Quality Stable Diffusion on Intel Platforms — Xinyu Ye, Haihao Shen, and Hanwen Chang, Intel Corporation We previously demonstrated how to leverage post-training quantization (PTQ) of Intel Neural Compressor to accelerate the inference of Stable Diffusion fine-tuned on the Pokémon dataset: Accelerate Stable Diffusion with Intel Neural Compressor Faster Inference through 8-Bit Post-Training Quantization on Intel Platformsmedium.com

A New Tool for Analyzing Neural Network Quantization — Agata Radys, Suyue Chen, and Bartosz Myrcha, Intel Corporation Model compression One of the biggest challenges with deep learning model optimization is accuracy loss. It is hard to reduce the storage space and computations consumed by a model without losing accuracy. …

Deploy Your Chatbot within Minutes on Intel Platforms — Liang Lv, Wenjiao Yue, Haihao Shen, and Huma Abidi, Intel Corporation In our previous blog, Create Your Own Custom Chatbot, we described how to do parameter-efficient fine-tuning on Intel processors. In this blog, we are happy to introduce Neural Chat, an end-to-end chatbot solution to simplify creation to deployment. …

Introducing Our New Neural Solution Component — Yi Liu, Kaihui Tang, Sihan Chen, Liang Lv, Feng Tian, and Haihao Shen, Intel Corporation In today’s fast-paced world of deep learning (DL), model compression techniques play a critical role in enhancing efficiency and reducing computational requirements. Intel Neural Compressor (INC) is a cutting-edge tool that offers a wide range…

Train Large Language Models Quickly and Easily on Intel Processors — Haihao Shen, Xinyu Ye, Kaokao Lv, Xuhui Ren, and Huma Abidi, Intel Corporation Large language models (LLM) have been attracting a lot of attention lately because of their extraordinary performance on dialog agents such as ChatGPT, GPT-4, and Bard. However, LLM are limited by the significant cost and time required…

Enhancing the SmoothQuant Approach to Quantization — Yintong Lu, Xin He, Heng Guo, Wenhua Cheng, Chang Wang, Mengni Wang, and Haihao Shen, Intel Corporation In this blog, we describe a post-training quantization technique for large language models with enhanced SmoothQuant approach. We also illustrate the usage and demonstrate the accuracy benefits. …