不用find-tune也能保證Llama 2不亂說話！善用“Grammar”客製化輸出內容
本篇接續上篇開源的Llama 2有哪些玩法？本篇文章介紹開發者與Llama 2的互動方式，說明如何在Llama 2運用GBNF。
LLM（Large Language Module）最讓人詬病的問題之一是「亂說話」，得到不是預期的回應。若問題發生在日常對話影響不大，若開發者串接後續商業邏輯，那麼問題就多了，輕則拋出錯誤處理，重則導致一連串錯誤邏輯。
本篇文章介紹如何利用文法（Grammar）規則限縮Llama 2的生成內容，讓輸出維持在可控範圍同時保留LLM發揮創意、生成內容的優勢。
GBNF
llama.cpp用GBNF (GGML BNF)格式來定義形式文法（formal grammars），例如你可以用來限制模型生成JSON格式或只用Emojis做出回應，詳細說明可參考原文GBNF Guide。
以下GBNF文法範例會示範套用過後的效果，讓你能更容易理解它帶來的好處。
GBNF範例
List
List是llama.cpp內的範例grammar，旨在讓Llama 2輸出條列式內容。文法內容如下：
root ::= item+
item ::= "- " [^\r\n\x0b\x0c\x85\u2028\u2029]+ "\n"輸入執行指令：
./main -m ./models/llama-2-13b-chat/ggml-model-Q4_K_M.gguf -n 256 --grammar-file grammars/list.gbnf -ins
Car
List結構非常單純，再來測試複雜一點的例子，文法內容如下：
root ::= CarAndOwner
PerformanceFeature ::= "{" ws "\"engine\":" ws string "," ws "\"horsepower\":" ws number "," ws "\"topSpeed\":" ws number "}"
PerformanceFeaturelist ::= "[]" | "[" ws PerformanceFeature ("," ws PerformanceFeature)* "]"
SafetyFeature ::= "{" ws "\"airbags\":" ws number "," ws "\"parkingSensors\":" ws number "," ws "\"laneAssist\":" ws number "}"
SafetyFeaturelist ::= "[]" | "[" ws SafetyFeature ("," ws SafetyFeature)* "]"
AudioFeature ::= "{" ws "\"brand\":" ws string "," ws "\"speakers\":" ws number "," ws "\"hasBluetooth\":" ws boolean "}"
AudioFeaturelist ::= "[]" | "[" ws AudioFeature ("," ws AudioFeature)* "]"
Features ::= "{" ws "\"audio\":" ws AudioFeature "," ws "\"safety\":" ws SafetyFeature "," ws "\"performance\":" ws PerformanceFeature "}"
Featureslist ::= "[]" | "[" ws Features ("," ws Features)* "]"
Owner ::= "{" ws "\"firstName\":" ws string "," ws "\"lastName\":" ws string "," ws "\"age\":" ws number "}"
Ownerlist ::= "[]" | "[" ws Owner ("," ws Owner)* "]"
Car ::= "{" ws "\"make\":" ws string "," ws "\"model\":" ws string "," ws "\"year\":" ws number "," ws "\"colors\":" ws stringlist "," ws "\"features\":" ws Features "}"
Carlist ::= "[]" | "[" ws Car ("," ws Car)* "]"
CarAndOwner ::= "{" ws "\"car\":" ws Car "," ws "\"owner\":" ws Owner "}"
CarAndOwnerlist ::= "[]" | "[" ws CarAndOwner ("," ws CarAndOwner)* "]"
string ::= "\"" ([^"]*) "\""
boolean ::= "true" | "false"
ws ::= [ \t\n]*
number ::= [0-9]+ "."? [0-9]*
stringlist ::= "[" ws "]" | "[" ws string ("," ws string)* ws "]"
numberlist ::= "[" ws "]" | "[" ws string ("," ws number)* ws "]"這段GBNF旨在結構化呈現車子的訊息，訊息包含車子的「品牌」、「型號」、「年份」、「顏色」、「性能」、「安全性」、「音響」、所屬人等資訊。
輸入執行指令：
./main -m ./models/llama-2-13b-chat/ggml-model-Q4_K_M.gguf -n 256 --grammar-file grammars/car.gbnf -ins --repeat_penalty 1.17647 --ctx_size 2048 --temp 0.2
將這段輸出整理一下：
{
"car": {
"make": "Tesla",
"model": "Model S",
"year": 2015,
"colors": [
"black"
],
"features": {
"audio": {
"brand": "Bowers & Wilkins",
"speakers": 17,
"hasBluetooth": true
},
"safety": {
"airbags": 8,
"parkingSensors": 4,
"laneAssist": 2015
},
"performance": {
"engine": "electric",
"horsepower": 762,
"topSpeed": 155
}
}
},
"owner": {
"firstName": "Elon",
"lastName": "Musk",
"age": 49
}
}測試結果很好，如此多層的巢狀結構也能夠輕易輸出，這讓Llama 2在商業邏輯的應用上變得更容易掌控與串接。
Weather
再來一個範例，來限制Llama 2只能回答特定的字詞，文法內容如下：
root ::= answer
answer ::= (weather | complaint | yesno)
weather ::= ("Sunny." | "Cloudy." | "Rainy.")
complaint ::= "I don't like talking about the weather."
yesno ::= ("Yes." | "No.")輸入執行指令：
./main -m ./models/llama-2-13b-chat/ggml-model-Q4_K_M.gguf -n 256 --grammar-file grammars/weather.gbnf -ins --repeat_penalty 1.17647 --ctx_size 256 --temp 0.2
結果很棒！完全限制了Llama 2不會亂說多餘的話。
Book
再來看最後一個書本資訊的範例，文法內容如下：
root ::= Book
Auther ::= "{" ws "\"firstName\":" ws string "," ws "\"lastName\":" ws string "," ws "\"age\":" ws number "}"
Autherlist ::= "[]" | "[" ws Auther ("," ws Auther)* "]"
Book ::= "{" ws "\"title\":" ws string "," ws "\"subtitle\":" ws string "," ws "\"authers\":" ws Autherlist "," ws "\"genres\":" ws stringlist "," ws "\"pages\":" ws number "," ws "\"description\":" ws string "," ws "\"isbn\":" ws string "}"
Booklist ::= "[]" | "[" ws Book ("," ws Book)* "]"
string ::= "\"" ([^"]*) "\""
boolean ::= "true" | "false"
ws ::= [ \t\n]*
number ::= [0-9]+ "."? [0-9]*
stringlist ::= "[" ws "]" | "[" ws string ("," ws string)* ws "]"
numberlist ::= "[" ws "]" | "[" ws string ("," ws number)* ws "]"輸入執行指令：
./main -m ./models/llama-2-13b-chat/ggml-model-Q4_K_M.gguf -n 256 --grammar-file grammars/book.gbnf -ins
整理一下結構：
{
"title": "The Alchemist",
"subtitle": "A Fable About Following Your Dreams",
"authers": [
{
"firstName": "Paulo",
"lastName": "Coelho",
"age": 71
}
],
"genres": [
"Fantasy",
"Adventure",
"Self-Help"
],
"pages": 224,
"description": "A magical tale about a shepherd boy named Santiago who embarks on a journey to fulfill his personal legend and find his treasure. A story of spiritual and self-discovery that will inspire you to follow your dreams and listen to your heart.",
"isbn": "978-0-14-303533-6"
}
整理一下結構：
{
"title": "The Lean Startup",
"subtitle": "How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses",
"authers": [
{
"firstName": "Eric",
"lastName": "Ries",
"age": 57
}
],
"genres": [
"Business",
"Entrepreneurship",
"Management"
],
"pages": 336,
"description": "A comprehensive guide to launching and growing a successful startup, with practical advice on how to use innovation, experimentation, and customer feedback to build a sustainable business.",
"isbn": "978-0-385-34011-7"
}得到的結果也符合我的預期，唯一美中不足的是部分內容是虛構的，例如ISBN無法查到這本書，得到的內容結構完全符合我要的格式，對於會後商業邏輯的串接完全沒有問題。
GBNF生成工具
GBNF文法規則要怎麼產生呢？從llama.cpp的GBNF Guide，瞭解規則後就可以開始撰寫屬於自己的規則，這是其中一種做法。
對於剛入門的新手推薦使用現成的工具幫助快速上手：
json-schema-to-grammar
如果你熟悉JSON Schema可以用llama.cpp提供的python腳本json_schema_to_grammar.py轉換GBNF。
Grammer Builder
如果你熟悉Typescript可以用別人已經做好的網頁應用Grammar Builder轉換GBNF。
在試用Llama 2的過程中，有發現它的輸出結構很不穩定，有時候長篇大論，有時候以清單方式呈現，想到對話類型的簡單應用之外，中間一度很苦惱Llama 2要如何串接到更複雜的應用，例如用特定格式整理汽車、書本資訊。
仔細瞭解後發現GBNF完美解決了這個問題，並且不需要fine-tune模型就能得到結果，效果非常好，唯一美中不足的部分是無法避免Llama 2的幻想，為了讓Llama 2理解業務內容，最終還是需要fine-tune模型才能做到，下一篇文章來介紹如何fine-tune Llama 2。
本文同步刊登於Softaverse - 不用find-tune也能保證Llama 2不亂說話！善用“Grammar”客製化輸出內容
