Chef 實戰 part1 — 從 0 開始寫 Cookbook 架設 Elasticsearch

Luyo
Luyo
Aug 26, 2017 · 46 min read

在 Learn Chef Rally 學完了 3 個 modules,學了一些基本的用法,先來實際操作一下看會遇到什麼問題。

這次的需求是架設 Elasticsearch + IK analyzer,但這篇文章還不會實作到 IK 的部分。

1. 新增 cookbook

先試著自己生一個 cookbook,參考 Learn Chef Rally 學習筆記 part3 — 基本 cookbook 設定及使用

$ cd ~/learn-chef
$ chef generate cookbook cookbooks/elasticsearch-ik
Hyphens are discouraged in cookbook names as they may cause problems with custom resources. See https://docs.chef.io/ctl_chef.html#chef-generate-cookbook for more information.
$ chef generate cookbook cookbooks/elasticsearch-ik
Generating cookbook elasticsearch-ik
- Ensuring correct cookbook file content
- Committing cookbook files to git
- Ensuring delivery configuration
- Ensuring correct delivery build cookbook content
- Adding delivery configuration to feature branch
- Adding build cookbook to feature branch
- Merging delivery content feature branch to master

Your cookbook is ready. Type `cd cookbooks/elasticsearch-ik` to enter it.
There are several commands you can run to get started locally developing and testing your cookbook.
Type `delivery local --help` to see a full list.
Why not start by writing a test? Tests for the default recipe are stored at:test/smoke/default/default_test.rbIf you'd prefer to dive right in, the default recipe can be found at:recipes/default.rb

他說 “Hyphens are discouraged in cookbook names as they may cause problems with custom resources.”,hyphens 就是 - 符號,可能會造成以後的問題,那就砍掉重來,換成底線好了:

$ rm -rf cookbooks/elasticksearch-ik
$ chef generate cookbook cookbooks/elaticsearch_ik

這次就沒警告了。

2. 更新 recipe/default.rb

先進入新的資料夾並打開 recipe/default.rb

$ cd cookbooks/elasticsearch_ik/
$ vim recipes/default.rb

內容除了註解以外應該會是空的。

首先要安裝 elasticsearch,但要先有 elasticsearch 的 yum repository 才能安裝,所以要先知道怎麼在 Chef 上安裝新的 yum repository。

先 google “elasticsearch yum repo” 找到官網的說明,再 google “chef yum repo” 找到 Chef 的 yum_repository resource 文件,有了這些資訊就可以自己把 resource 兜起來了:

yum_repository 'elasticsearch-5.x' do
description "Elasticsearch repository for 5.x packages"
baseurl "https://artifacts.elastic.co/packages/5.x/yum"
gpgkey 'https://artifacts.elastic.co/GPG-KEY-elasticsearch'
action :create
end

存檔後上傳到 Chef server:

$ knife cookbook upload elasticsearch_ik
Uploading elasticsearch_ik [0.1.0]
Uploaded 1 cookbook.

3. Bootstrap node

接著我想把我的 node 重新取名叫 es-1,就來跑 bootstrap 程序把它重新初始化,參考 Learn Chef Rally 學習筆記 part6 — 初始化及更新 node

$ knife bootstrap 172.31.21.70 --ssh-user centos --sudo --identity-file ~/.ssh/test.pem --node-name es-1 --run-list 'recipe[elasticsearch_ik]'
Creating new client for es-1
Creating new node for es-1
Connecting to 172.31.21.70
172.31.21.70 -----> Existing Chef installation detected
172.31.21.70 Starting the first Chef Client run...
172.31.21.70 Starting Chef Client, version 13.3.42
172.31.21.70 resolving cookbooks for run list: ["elasticsearch_ik"]
172.31.21.70 Synchronizing Cookbooks:
172.31.21.70 - elasticsearch_ik (0.1.0)
172.31.21.70 Installing Cookbook Gems:
172.31.21.70 Compiling Cookbooks...
172.31.21.70 Converging 1 resources
172.31.21.70 Recipe: elasticsearch_ik::default
172.31.21.70 * yum_repository[elasticsearch-5.x] action create
172.31.21.70 * template[/etc/yum.repos.d/elasticsearch-5.x.repo] action create
172.31.21.70 - create new file /etc/yum.repos.d/elasticsearch-5.x.repo
172.31.21.70 - update content in file /etc/yum.repos.d/elasticsearch-5.x.repo from none to 2fa420
172.31.21.70 --- /etc/yum.repos.d/elasticsearch-5.x.repo 2017-08-26 10:41:56.476384138 +0000
172.31.21.70 +++ /etc/yum.repos.d/.chef-elasticsearch-520170826-29977-12oti34.x.repo 2017-08-26 10:41:56.476384138 +0000
172.31.21.70 @@ -1 +1,11 @@
172.31.21.70 +# This file was generated by Chef
172.31.21.70 +# Do NOT modify this file by hand.
172.31.21.70 +
172.31.21.70 +[elasticsearch-5.x]
172.31.21.70 +name=Elasticsearch repository for 5.x packages
172.31.21.70 +baseurl=https://artifacts.elastic.co/packages/5.x/yum
172.31.21.70 +enabled=1
172.31.21.70 +fastestmirror_enabled=0
172.31.21.70 +gpgcheck=1
172.31.21.70 +gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
172.31.21.70 - change mode from '' to '0644'
172.31.21.70 - restore selinux security context
172.31.21.70 * execute[yum clean metadata elasticsearch-5.x] action run
172.31.21.70 - execute yum clean metadata --disablerepo=* --enablerepo=elasticsearch-5.x
172.31.21.70 * execute[yum-makecache-elasticsearch-5.x] action run
172.31.21.70 - execute yum -q -y makecache --disablerepo=* --enablerepo=elasticsearch-5.x
172.31.21.70 * ruby_block[yum-cache-reload-elasticsearch-5.x] action create
172.31.21.70 - execute the ruby block yum-cache-reload-elasticsearch-5.x
172.31.21.70 * execute[yum clean metadata elasticsearch-5.x] action nothing (skipped due to action :nothing)
172.31.21.70 * execute[yum-makecache-elasticsearch-5.x] action nothing (skipped due to action :nothing)
172.31.21.70 * ruby_block[yum-cache-reload-elasticsearch-5.x] action nothing (skipped due to action :nothing)
172.31.21.70
172.31.21.70
172.31.21.70 Running handlers:
172.31.21.70 Running handlers complete
172.31.21.70 Chef Client finished, 5/8 resources updated in 18 seconds

連線進 node 確認 /etc/yum.repos.d/elasticsearch-5.x.repo 檔案已被產生。

yum repo 安裝好後就可以來安裝 package 了。打開 default.rb ,在 yum_repository 之後再加一行:

yum_repository 'elasticsearch-5.x' do
description "Elasticsearch repository for 5.x packages"
baseurl "https://artifacts.elastic.co/packages/5.x/yum"
gpgkey 'https://artifacts.elastic.co/GPG-KEY-elasticsearch'
action :create
end
package elasticsearch

然後上傳並更新:

$ knife cookbook upload elasticsearch_ik
Uploading elasticsearch_ik [0.1.0]
Uploaded 1 cookbook.
$ knife ssh 'name:es-1' 'sudo chef-client' --ssh-user centos --identity-file ~/.ssh/test.pem --attribute ipaddress
172.31.21.70 Starting Chef Client, version 13.3.42
172.31.21.70 resolving cookbooks for run list: ["elasticsearch_ik"]
172.31.21.70 Synchronizing Cookbooks:
172.31.21.70 - elasticsearch_ik (0.1.0)
172.31.21.70 Installing Cookbook Gems:
172.31.21.70 Compiling Cookbooks...
172.31.21.70 Converging 2 resources
172.31.21.70 Recipe: elasticsearch_ik::default
172.31.21.70 * yum_repository[elasticsearch-5.x] action create
172.31.21.70 * template[/etc/yum.repos.d/elasticsearch-5.x.repo] action create (up to date)
172.31.21.70 * execute[yum clean metadata elasticsearch-5.x] action nothing (skipped due to action :nothing)
172.31.21.70 * execute[yum-makecache-elasticsearch-5.x] action nothing (skipped due to action :nothing)
172.31.21.70 * ruby_block[yum-cache-reload-elasticsearch-5.x] action nothing (skipped due to action :nothing)
172.31.21.70 (up to date)
172.31.21.70 * yum_package[elasticsearch] action install
172.31.21.70 - install version 5.5.2-1 of package elasticsearch
172.31.21.70
172.31.21.70 Running handlers:
172.31.21.70 Running handlers complete
172.31.21.70 Chef Client finished, 1/6 resources updated in 27 seconds

這樣就安裝好 elasticsearch 的 5.5.2–1 了。

誒誒等一下,我不要這個最新版本啊,因為目前 IK 的 releases 只支援到 5.5.1,所以我必須給特定的版號才行。

4. 指定 package 版本

先確定一下 5.5.1 在 yum repository 裡完整的 package 版號長什麼樣子,在 node 裡下指令:

$ sudo yum --showduplicates list elasticsearch
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirror.0x.sg
* extras: mirror.0x.sg
* updates: mirror.0x.sg
Installed Packages
elasticsearch.noarch 5.5.2-1 @elasticsearch-5.x
Available Packages
elasticsearch.noarch 5.0.0-1 elasticsearch-5.x
elasticsearch.noarch 5.0.1-1 elasticsearch-5.x
elasticsearch.noarch 5.0.2-1 elasticsearch-5.x
elasticsearch.noarch 5.1.1-1 elasticsearch-5.x
elasticsearch.noarch 5.1.2-1 elasticsearch-5.x
elasticsearch.noarch 5.2.0-1 elasticsearch-5.x
elasticsearch.noarch 5.2.1-1 elasticsearch-5.x
elasticsearch.noarch 5.2.2-1 elasticsearch-5.x
elasticsearch.noarch 5.3.0-1 elasticsearch-5.x
elasticsearch.noarch 5.3.1-1 elasticsearch-5.x
elasticsearch.noarch 5.3.2-1 elasticsearch-5.x
elasticsearch.noarch 5.3.3-1 elasticsearch-5.x
elasticsearch.noarch 5.4.0-1 elasticsearch-5.x
elasticsearch.noarch 5.4.1-1 elasticsearch-5.x
elasticsearch.noarch 5.4.2-1 elasticsearch-5.x
elasticsearch.noarch 5.4.3-1 elasticsearch-5.x
elasticsearch.noarch 5.5.0-1 elasticsearch-5.x
elasticsearch.noarch 5.5.1-1 elasticsearch-5.x
elasticsearch.noarch 5.5.2-1 elasticsearch-5.x

這樣我們就可以找到 5.5.1 在 yum package 裡的版號叫做 5.5.1–1 了。

找一下 package resource 的文件,可以找到指定版本的方法就是加一個 version 的屬性,回到工作站修改 default.rb

yum_repository 'elasticsearch-5.x' do
description "Elasticsearch repository for 5.x packages"
baseurl "https://artifacts.elastic.co/packages/5.x/yum"
gpgkey 'https://artifacts.elastic.co/GPG-KEY-elasticsearch'
action :create
end
package 'elasticsearch' do
version '5.5.1-1'
end

再次上傳及更新 node,這次為了偷懶,我把兩道指令連在一起執行:

$ knife cookbook upload elasticsearch_ik; knife ssh 'name:es-1' 'sudo chef-client' --ssh-user centos --identity-file ~/.ssh/test.pem --attribute ipaddress
(...略)
172.31.21.70 [2017-08-26T11:01:33+00:00] ERROR: yum_package[elasticsearch] (elasticsearch_ik::default line 14) had an error: Chef::Exceptions::Package: Installed package elasticsearch-5.5.2-1 is newer than candidate package elasticsearch-5.5.1-1
172.31.21.70 [2017-08-26T11:01:33+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

這次出現失敗的錯誤訊息了,他說現在的版本比較新,不給降級。回去 package resource 的文件,找到另一個叫 allow_downgrade 的屬性,把它加進 recipe:

yum_repository 'elasticsearch-5.x' do
description "Elasticsearch repository for 5.x packages"
baseurl "https://artifacts.elastic.co/packages/5.x/yum"
gpgkey 'https://artifacts.elastic.co/GPG-KEY-elasticsearch'
action :create
end
package 'elasticsearch' do
allow_downgrade true
version '5.5.1-1'
end

上傳並更新 node:

$ knife cookbook upload elasticsearch_ik; knife ssh 'name:es-1' 'sudo chef-client' --ssh-user centos --identity-file ~/.ssh/test.pem --attribute ipaddress
Uploading elasticsearch_ik [0.1.0]
Uploaded 1 cookbook.
172.31.21.70 Starting Chef Client, version 13.3.42
172.31.21.70 resolving cookbooks for run list: ["elasticsearch_ik"]
172.31.21.70 Synchronizing Cookbooks:
172.31.21.70 - elasticsearch_ik (0.1.0)
172.31.21.70 Installing Cookbook Gems:
172.31.21.70 Compiling Cookbooks...
172.31.21.70 Converging 2 resources
172.31.21.70 Recipe: elasticsearch_ik::default
172.31.21.70 * yum_repository[elasticsearch-5.x] action create
172.31.21.70 * template[/etc/yum.repos.d/elasticsearch-5.x.repo] action create (up to date)
172.31.21.70 * execute[yum clean metadata elasticsearch-5.x] action nothing (skipped due to action :nothing)
172.31.21.70 * execute[yum-makecache-elasticsearch-5.x] action nothing (skipped due to action :nothing)
172.31.21.70 * ruby_block[yum-cache-reload-elasticsearch-5.x] action nothing (skipped due to action :nothing)
172.31.21.70 (up to date)
172.31.21.70 * yum_package[elasticsearch] action install
172.31.21.70 - install version 5.5.1-1 of package elasticsearch
172.31.21.70
172.31.21.70 Running handlers:
172.31.21.70 Running handlers complete
172.31.21.70 Chef Client finished, 1/6 resources updated in 23 seconds

這次就成功了!

5. 啟用 Elasticsearch

安裝好後要來啟動 service,回去參考 Learn Chef Rally 學習筆記 part2 — 安裝及啟動 Apache,在 default.rb 再加入新的 resource:

yum_repository 'elasticsearch-5.x' do
description "Elasticsearch repository for 5.x packages"
baseurl "https://artifacts.elastic.co/packages/5.x/yum"
gpgkey 'https://artifacts.elastic.co/GPG-KEY-elasticsearch'
action :create
end
package 'elasticsearch' do
allow_downgrade true
version '5.5.1-1'
end
service 'elasticsearch' do
action [:enable, :start]
end

上傳並更新 node:

$ knife cookbook upload elasticsearch_ik; knife ssh 'name:es-1' 'sudo chef-client' --ssh-user centos --identity-file ~/.ssh/test.pem --attribute ipaddress
Uploading elasticsearch_ik [0.1.0]
Uploaded 1 cookbook.
172.31.21.70 Starting Chef Client, version 13.3.42
172.31.21.70 resolving cookbooks for run list: ["elasticsearch_ik"]
172.31.21.70 Synchronizing Cookbooks:
172.31.21.70 - elasticsearch_ik (0.1.0)
172.31.21.70 Installing Cookbook Gems:
172.31.21.70 Compiling Cookbooks...
172.31.21.70 Converging 3 resources
172.31.21.70 Recipe: elasticsearch_ik::default
172.31.21.70 * yum_repository[elasticsearch-5.x] action create
172.31.21.70 * template[/etc/yum.repos.d/elasticsearch-5.x.repo] action create (up to date)
172.31.21.70 * execute[yum clean metadata elasticsearch-5.x] action nothing (skipped due to action :nothing)
172.31.21.70 * execute[yum-makecache-elasticsearch-5.x] action nothing (skipped due to action :nothing)
172.31.21.70 * ruby_block[yum-cache-reload-elasticsearch-5.x] action nothing (skipped due to action :nothing)
172.31.21.70 (up to date)
172.31.21.70 * yum_package[elasticsearch] action install (up to date)
172.31.21.70 * service[elasticsearch] action enable
172.31.21.70 - enable service service[elasticsearch]
172.31.21.70 * service[elasticsearch] action start
172.31.21.70 - start service service[elasticsearch]
172.31.21.70
172.31.21.70 Running handlers:
172.31.21.70 Running handlers complete
172.31.21.70 Chef Client finished, 2/8 resources updated in 12 seconds

看起來是沒什麼問題,來 curl 看看能不能抓得到東西:

$ curl 172.31.21.70:9200
curl: (7) Failed connect to 172.31.21.70:9200; 連線被拒絕

抓不到東西。連到 node 裡面看看 elasticsearch 的狀態:

$ sudo service elasticsearch status
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Sat 2017-08-26 11:23:01 UTC; 5s ago
Docs: http://www.elastic.co
Process: 2845 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
Process: 2844 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 2845 (code=exited, status=1/FAILURE)
Aug 26 11:23:01 ip-172-31-21-70 systemd[1]: Starting Elasticsearch...
Aug 26 11:23:01 ip-172-31-21-70 systemd[1]: Started Elasticsearch.
Aug 26 11:23:01 ip-172-31-21-70 elasticsearch[2845]: which: no java in (/usr/local/sbin:/usr/local/bin:/...in)
Aug 26 11:23:01 ip-172-31-21-70 systemd[1]: elasticsearch.service: main process exited, code=exited, st...LURE
Aug 26 11:23:01 ip-172-31-21-70 systemd[1]: Unit elasticsearch.service entered failed state.
Aug 26 11:23:01 ip-172-31-21-70 systemd[1]: elasticsearch.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

elasticsearch 的狀態是啟動失敗,因為沒有安裝 java。我們只好在 resource 裡先增加 java 的安裝設定。

6. 安裝 java

先找到 java 的 yum package 版號,在 node 裡下指令:

$ sudo yum install java

我們就可以知道 package 名稱叫做 java-1.8.0-openjdk ,可以先不要真的安裝,按 N 把它跳掉。

回到工作站加入更新 default.rb

yum_repository 'elasticsearch-5.x' do
description "Elasticsearch repository for 5.x packages"
baseurl "https://artifacts.elastic.co/packages/5.x/yum"
gpgkey 'https://artifacts.elastic.co/GPG-KEY-elasticsearch'
action :create
end
package 'java-1.8.0-openjdk'package 'elasticsearch' do
allow_downgrade true
version '5.5.1-1'
end
service 'elasticsearch' do
action [:enable, :start]
end

上傳並更新 node:

$ knife cookbook upload elasticsearch_ik; knife ssh 'name:es-1' 'sudo chef-client' --ssh-user centos --identity-file ~/.ssh/test.pem --attribute ipaddress
Uploading elasticsearch_ik [0.1.0]
Uploaded 1 cookbook.
172.31.21.70 Starting Chef Client, version 13.3.42
172.31.21.70 resolving cookbooks for run list: ["elasticsearch_ik"]
172.31.21.70 Synchronizing Cookbooks:
172.31.21.70 - elasticsearch_ik (0.1.0)
172.31.21.70 Installing Cookbook Gems:
172.31.21.70 Compiling Cookbooks...
172.31.21.70 Converging 4 resources
172.31.21.70 Recipe: elasticsearch_ik::default
172.31.21.70 * yum_repository[elasticsearch-5.x] action create
172.31.21.70 * template[/etc/yum.repos.d/elasticsearch-5.x.repo] action create (up to date)
172.31.21.70 * execute[yum clean metadata elasticsearch-5.x] action nothing (skipped due to action :nothing)
172.31.21.70 * execute[yum-makecache-elasticsearch-5.x] action nothing (skipped due to action :nothing)
172.31.21.70 * ruby_block[yum-cache-reload-elasticsearch-5.x] action nothing (skipped due to action :nothing)
172.31.21.70 (up to date)
172.31.21.70 * yum_package[java-1.8.0-openjdk] action install
172.31.21.70 - install version 1.8.0.141-1.b16.el7_3 of package java-1.8.0-openjdk
172.31.21.70 * yum_package[elasticsearch] action install (up to date)
172.31.21.70 * service[elasticsearch] action enable (up to date)
172.31.21.70 * service[elasticsearch] action start
172.31.21.70 - start service service[elasticsearch]
172.31.21.70
172.31.21.70 Running handlers:
172.31.21.70 Running handlers complete
172.31.21.70 Chef Client finished, 2/9 resources updated in 13 seconds

看起來沒問題,java 安裝看起來是成功的,再回 node 確認 elasticsearch service 的狀態:

$ sudo service elasticsearch status
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Sat 2017-08-26 11:39:41 UTC; 3s ago
Docs: http://www.elastic.co
Process: 4583 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
Process: 4582 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 4583 (code=exited, status=1/FAILURE)
Aug 26 11:39:41 ip-172-31-21-70 elasticsearch[4583]: OpenJDK 64-Bit Server VM warning: If the number of ...s=N
Aug 26 11:39:41 ip-172-31-21-70 elasticsearch[4583]: OpenJDK 64-Bit Server VM warning: INFO: os::commit_...12)
Aug 26 11:39:41 ip-172-31-21-70 elasticsearch[4583]: #
Aug 26 11:39:41 ip-172-31-21-70 elasticsearch[4583]: # There is insufficient memory for the Java Runtime...ue.
Aug 26 11:39:41 ip-172-31-21-70 elasticsearch[4583]: # Native memory allocation (mmap) failed to map 206...ry.
Aug 26 11:39:41 ip-172-31-21-70 elasticsearch[4583]: # An error report file with more information is saved as:
Aug 26 11:39:41 ip-172-31-21-70 elasticsearch[4583]: # /tmp/hs_err_pid4583.log
Aug 26 11:39:41 ip-172-31-21-70 systemd[1]: elasticsearch.service: main process exited, code=exited, st...LURE
Aug 26 11:39:41 ip-172-31-21-70 systemd[1]: Unit elasticsearch.service entered failed state.
Aug 26 11:39:41 ip-172-31-21-70 systemd[1]: elasticsearch.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

還是失敗,這次是記憶不夠的問題,因為我開的機器是 t2.micro ,記憶只有 1G,必須告訴 elasticsearch 不要用這麼多記憶體去做初始化。

7. 新增 jvm.options template

這個參數可以在 /etc/elasticsearch/jvm.options 這個設定檔裡做設定,我們可以用 file 這個 resource 將檔案覆寫掉,參考 Learn Chef Rally 學習筆記 part3 — 基本 cookbook 設定及使用,用指令新增一個 template 檔案:

$ chef generate template jvm.options
Recipe: code_generator::template
* directory[/home/yo/learn-chef/cookbooks/elasticsearch_ik/templates] action create (up to date)
* template[/home/yo/learn-chef/cookbooks/elasticsearch_ik/templates/jvm.options.erb] action create
- create new file /home/yo/learn-chef/cookbooks/elasticsearch_ik/templates/jvm.options.erb
- update content in file /home/yo/learn-chef/cookbooks/elasticsearch_ik/templates/jvm.options.erb from none to e3b0c4
(diff output suppressed by config)
- restore selinux security context

然後把 node 裡的 jvm.options 的內容複製貼到工作站這邊的 templates/jvm.options 裡面,再找到裡面的這兩行:

-Xms2g
-Xmx2g

改成:

-Xms256m
-Xmx256m

接著回來修改 default.rb

yum_repository 'elasticsearch-5.x' do
description "Elasticsearch repository for 5.x packages"
baseurl "https://artifacts.elastic.co/packages/5.x/yum"
gpgkey 'https://artifacts.elastic.co/GPG-KEY-elasticsearch'
action :create
end
package 'java-1.8.0-openjdk'package 'elasticsearch' do
allow_downgrade true
version '5.5.1-1'
end
template '/etc/elasticsearch/jvm.options' do
source 'jvm.options.erb'
end
service 'elasticsearch' do
action [:enable, :start]
end

上傳及更新 node:

$ knife cookbook upload elasticsearch_ik; knife ssh 'name:es-1' 'sudo chef-client' --ssh-user centos --identity-file ~/.ssh/test.pem --attribute ipaddress
Uploading elasticsearch_ik [0.1.0]
Uploaded 1 cookbook.
(...略)
172.31.21.70 - update content in file /etc/elasticsearch/jvm.options from 1b092a to caf01f
172.31.21.70 --- /etc/elasticsearch/jvm.options 2017-07-18 20:47:01.000000000 +0000
172.31.21.70 +++ /etc/elasticsearch/.chef-jvm20170826-12413-188gy9.options 2017-08-26 13:53:40.297855964 +0000
172.31.21.70 @@ -19,8 +19,8 @@
172.31.21.70 # Xms represents the initial size of total heap space
172.31.21.70 # Xmx represents the maximum size of total heap space
172.31.21.70
172.31.21.70 --Xms2g
172.31.21.70 --Xmx2g
172.31.21.70 +-Xms256m
172.31.21.70 +-Xmx256m
(...略)

到 node 裡面確認 elasticsearch 狀態:

$ sudo service elasticsearch status
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2017-08-26 13:54:17 UTC; 1min 46s ago
Docs: http://www.elastic.co
Process: 12835 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 12836 (java)
CGroup: /system.slice/elasticsearch.service
└─12836 /bin/java -Xms256m -Xmx256m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=7...
Aug 26 13:54:17 ip-172-31-21-70 systemd[1]: Starting Elasticsearch...
Aug 26 13:54:17 ip-172-31-21-70 systemd[1]: Started Elasticsearch.
Aug 26 13:54:17 ip-172-31-21-70 elasticsearch[12836]: OpenJDK 64-Bit Server VM warning: If the number of...s=N
Hint: Some lines were ellipsized, use -l to show in full.

Elasticsearch 終於成功跑起來了!

8. 新增 elasticsearch.yml template

現在 node 的 elasticsearch service 已經跑起來了,照理說用 curl 應該要能抓到東西了:

$ curl 172.31.21.70:9200
curl: (7) Failed connect to 172.31.21.70:9200; 連線被拒絕

可惡怎麼還是不行?到 node 裡面試試看:

$ curl localhost:9200
{
"name" : "jDu6QyJ",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "gBbJ_Y_3SJW3n2wzu905cg",
"version" : {
"number" : "5.5.1",
"build_hash" : "19c13d0",
"build_date" : "2017-07-18T20:44:24.823Z",
"build_snapshot" : false,
"lucene_version" : "6.6.0"
},
"tagline" : "You Know, for Search"
}

明明就可以啊,看來是網路設定的問題。因為 elasticsearch 預設的 host 是 localhost 所以用 IP 會解析不到,必須把 /etc/elasticsearch/elasticsearch.yml 設定檔裡的 network.host 參數改掉。

方式一樣是用 template 來做,先下指令:

$ chef generate template elasticsearch.yml
Recipe: code_generator::template
* directory[/home/yo/learn-chef/cookbooks/elasticsearch_ik/templates] action create (up to date)
* template[/home/yo/learn-chef/cookbooks/elasticsearch_ik/templates/elasticsearch.yml.erb] action create
- create new file /home/yo/learn-chef/cookbooks/elasticsearch_ik/templates/elasticsearch.yml.erb
- update content in file /home/yo/learn-chef/cookbooks/elasticsearch_ik/templates/elasticsearch.yml.erb from none to e3b0c4
(diff output suppressed by config)
- restore selinux security context

然後修改 templates/elasticsearch.yml.erb

network.host: 172.31.21.70

再修改 default.rb

yum_repository 'elasticsearch-5.x' do
description "Elasticsearch repository for 5.x packages"
baseurl "https://artifacts.elastic.co/packages/5.x/yum"
gpgkey 'https://artifacts.elastic.co/GPG-KEY-elasticsearch'
action :create
end
package 'java-1.8.0-openjdk'package 'elasticsearch' do
allow_downgrade true
version '5.5.1-1'
end
template '/etc/elasticsearch/jvm.options' do
source 'jvm.options.erb'
end

template '/etc/elasticsearch/elasticsearch.yml' do
source 'elasticsearch.yml.erb'
end
service 'elasticsearch' do
action [:enable, :start]
end

上傳及更新 node:

$ knife cookbook upload elasticsearch_ik; knife ssh 'name:es-1' 'sudo chef-client' --ssh-user centos --identity-file ~/.ssh/test.pem --attribute ipaddress
(...略)
172.31.21.70 * template[/etc/elasticsearch/elasticsearch.yml] action create
172.31.21.70 - update content in file /etc/elasticsearch/elasticsearch.yml from e3b0c4 to 251503
(...略)

curl 一次:

$ curl 172.31.21.70:9200
curl: (7) Failed connect to 172.31.21.70:9200; 連線被拒絕

可惡,為什麼還是不行?

原因其實很簡單,因為前一次的 chef-client 啟動 elasticsearch 已經成功了,而我的 recipe 並沒有要求重新啟動 service,所以必需手動重啟 elasticsearch。

9. 重啟 Elasticsearch

要 restart 很簡單,連到機器裡下指令就好,但問題是以後如果有 50 台 nodes 的話,總不可能一台一台連進去 restart 吧?

雖然 service resource 有提供 :restart 的功能,但我不希望每次 chef-client 都去重啟一次 elasticsearch。

這時候我想到 knife ssh 裡面有一段是 sudo chef-client ,表示我其實可以把這段換成任何我想下的指令,那就來試試看把 sudo chef-client 換成 sudo service elasticsearch restart

$ knife ssh 'name:es-1' 'sudo service elasticsearch restart' --ssh-user centos --identity-file ~/.ssh/test.pem --attribute ipaddress
172.31.21.70 Restarting elasticsearch (via systemctl): [ OK ]

看起來沒問題,再 curl 一次試試看:

$ curl 172.31.21.70:9200
{
"name" : "jDu6QyJ",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "gBbJ_Y_3SJW3n2wzu905cg",
"version" : {
"number" : "5.5.1",
"build_hash" : "19c13d0",
"build_date" : "2017-07-18T20:44:24.823Z",
"build_snapshot" : false,
"lucene_version" : "6.6.0"
},
"tagline" : "You Know, for Search"
}

哈哈總算抓到啦!我的第一個 Chef 實戰第一步到這邊算是大功告成了!


心得

  1. 其實整個設定 Chef recipe 的過程會遇到的問題,基本上跟自己重灌一台機器會遇到的問題是差不多的,所以把整個過程能夠用格式化的方式寫下來真的是很有幫助。
  2. 每次一遇到問題做了修改,就要重新上傳一次 cookbook 到 Chef server 上,然後再遠端更新 node,其實有點不 make sense,主要是因為根本不知道這次的修改是不是有效的,要先跑完整個程序再去做測試才能知道結果, 有時候只是不小心打錯字或是語法錯誤,就又要重來一次,有點沒效率。看來還是要找時間再回去 Learn Chef Rally 進修。

verybuy-dev

VeryBuy 研發手札

)
Luyo

Written by

Luyo

Founder, Developer of VeryBuy — https://www.verybuy.cc

verybuy-dev

VeryBuy 研發手札

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade