Week 3 and 4, GSoC 2017 — dozens of cloud vm’s, ansibling, finding bugs, testing

28 Jun 2017

At last I have got a hold of IRC’s and I declare my love for irssi. The combination of tmux and irssi is a boon for me. I tried different clients like weechat, Epic but I found irssi to be more appealing, quite frankly use any of these two if you are looking for something using which you can chat on irc’s on the terminal.

My current installation of irssi is there on my VPC hosted on linode. For logging my chat’s and conversations, I have setup logrotate daemon to log the channels and private chats in their respective directories.

So what does it look like? Definitely a huge shift from the web chat interface and more customisable.

I love the current setup. Just one thing which I think is missing out from here is that I would like to get a notification when I get a message or someone mentions my name in a channel. Will have to look that up.

You can try Limechat if you are on a MAC.

Between, I will be there with the handle tasdikrahman hanging around the channels #ovirt on OFTC as well as on freenode, mostly #ansible and #gsoc.

Anyways, that’s for my recent addition of irc love.

Talking about work, I have been working on my existing PR looking into the feedback which my mentor and other members of the infra team have given.

And right now, I am working on adding playbooks for configuring a remote DWH on an ovirt-engine setup.

Prelude

Reading up the docs, the DWH is a historic database that allows users to create reports over a static API using business intelligence suites that enable you to monitor the system. It contains the ETL (Extract Transform Load) process created using Talend Open Studio and DB scripts to create a working history DB.

This history database(ovirt_engine_history to be precise) can be utilized by any application to extract a range of information at the data center, cluster, and host levels.

Simply put, it’s a BI tool for your ovirt installation.

Ovirt-Architecture

oVirt Engine uses PostgreSQL 8.4.x as the database platform to store information about the state of the virtualization environment, its configuration and performance. At install time, ovirt engine creates a PostgreSQL database called engine. You have the option to either install this on the same host or a different host(remote)

The ovirt-engine-dwh package creates a second database called ovirt_engine_history, which contains historical configuration information and statistical metrics collected every minute over time from the engine operational database. Tracking the changes to the database provides information on the objects in the database, enabling the user to analyze activity, enhance performance, and resolve difficulties.

You can track configuration data, statistical data into your ovirt_engine_history database based on your needs.

But why should someone run the DWH service on a different host?

Quite obviously running a whole lot of services on one host would be very memory heavy. And what happens when someone tries to squeeze out a lot of things from a single instance?

There is a decrease in performance output due to fight amongst the processes for resources. So it’s very logical to distribute your services to different hosts if possible.

two host setup where the DWH is separated from the main host engine installation

The above architecture basically shows the simple setup of the ovirt-engine and dwh all being on the same host.

Now even when you are trying to seperate out the load by distributing the services running in a host, you further have a flexibily offered by oVirt here.

Some being.

  1. engine db itself being on a differnt host rather than on the host where ovirt-engine is running.
  2. DWH being installed on a different host than the one where ovirt-engine is installed.(feature being available only after oVirt Engine 3.5)
  3. DWH being installed on a different host and the DWH db being on a 3rd host.

The first one is covered by the ansible role ovirt-engine-remote-db which lets you do so

I tried tackling the 2nd one on the list in my 3rd week.

A simple setup where both the DWH and the ovirt-engine are on the same host

Had the initial discussions about issue #9 with Lukas and we narrowed down to the end goals and tasks for that particular issue on Github.

The docs are pretty clear on the process itself and a bit of poring over them made the things which were to be done.

As I had mentioned in my earlier post about not automating something which you haven’t achieved manually yet.

That applied here too.

For starters, I rebuilt 2x2gig Centos7 boxes on Linode which were hardened using my own custom ansible playbook called ansible-bootstrap-server. I had a good lesson about security when one of my servers got compromised so this one was a must.

Some assumptions for the two systems

Some assumptions of the systems that I was going to work with and which should be followed if the path followed is the same as mine

Expanding on the first one, it should be able to do so by using the FQDN’s of the respective VM’s in question.

To add to the above,

  • Ports 80, 443, 5432(postgres) should be open on both these hosts.

You can open the above ports using firewalld using (assuming you are root)

$ firewall-cmd --zone=public --add-port=80/tcp --permanent
$ # further adding any rules to appended to iptables
$ firewalld-cmd --reload # for the rules to take effect
$ iptables -S # to check whether the changes have taken effect

iptables would spit every rule defined out there for your system. That would get ugly real quick. A cleaner way would be to do a

[root@dwhtest-3-engine ~]# iptables-save | grep 443
-A IN_public_allow -p tcp -m tcp --dport 443 -m conntrack --ctstate NEW -j ACCEPT
[root@dwhtest-3-engine ~]#

After these, the machine is just set for configuration.

Manual installation

Install and setup ovirt-engine on machine A, ovirt-engine-dwh on machine B, see that dwhd service on B collects data from the engine on A.

On A:

Installing DWH on a remote host from the one having ovirt-engine

So ovirt-engine is set on machine A.

The only difference here being when running engine-setup on the host, answering No to configuring Data Warehouse:

...
Configure Data Warehouse on this host (Yes, No) [Yes]: No
...

The other thing to note in the above installation is the answerfile which was generated out of the installation process.

Answerfile?

Tools like engine-setup, engine-cleanup and ovirt-engine-rename which are based out on OTOPI (key value pair store of data and configuration), all generate something called as an answerfile upon completion of their (be it successful or erroneous).

This file is placed under /var/lib/ovirt-engine/setup/answers/ ending with a *.conf.

Going from a particular state S0 to S1 using the answerfile

So have system A which is present on some state0, “S0” in this sense includes basically everything relevant — versions of relevant packages, enabled repos, history (such as previous runs of these tools, other data accumulated over time, etc), other manual configuration, etc.

The expected way to use these answer files is:

  1. Have a system A in some state S0
  2. Run one of the tools interactively, answer its questions as needed, let it create an answer file Ans1. Basically answering all the places where the program was waiting for STDIN.
  3. System A is now in state S1.
  4. Have some other system B in state S0, that you want to bring to state S1.
  5. Run there the same tool with –config-append=Ans1
When used this way, the tools should run unattended. If they still ask questions, it’s generally considered a bug.

Which allows and makes way for automation.

Manually editing such an answer file is generally not supported/expected and should not be needed. You might do that to achieve special non-standard goals. If you do that, you should thoroughly verify that it works for you, and use in a controlled environment — same known initial state, same versions of relevant stuff, etc.

Leveraging the things seen in this answerfile

We already have in place an ansible role for installing ovirt-engine along with DWH on the same host here named ovirt-engine-setup.

Having a look at an existing answerfile, take ovirt-engine-setup/templates/answerfile_4.0_basic.txt.j2 for example.

As it’s based out on OTOPI, and I had for reference the newly generated anwerfile from the succesfull install on machine A.

What I really wanted was a diff between these two answerfiles, one which was generated when DWH was installed along with the engine on the same host and the other being when the DWH would be configured on a remote host

This gave me a clear idea of what I needed to do with the existing answer file to make it suitable for configuring a remote DWH.

Changes made to one of the answerfile templates engine-setup to accommodate the option of not installing the DWH on the host

the variable ovirt_engine_dwh_db_configure would be a bool defined inside our play yaml file telling ovirt-engine-setup to not configure DWH on that host.

This modification sets us up for the first part.

Configuring the remote DWH manually first

If you are trying to access the web adming on machine A, it would show you an error like this.

On machine B

Restarting the ovirt-engine service on machine A starts the whole thing.

Debugging tips

If for any reason, the admin panel is still giving you an error.

  • Try checking the output of iptables -S, to see whether you are allowing incoming connections on port 5432 on both machines.
  • Are both hosts able to resolve to their public IP’s using the provided FQDN
  • Try entering those passwords slowly.
  • If all fails, the logs of each engine-setup command would be there for your rescue.

Automating it

flow which would be followed

For testing my newly created ovirt-engine-install-remote-dwh, provisioned 2x2gig CentOS 7 vms on linode.

With these done, the servers are ready for my new ansible roles.

First would be the installation of the ovirt-engine host

---
- hosts: engine
vars:
ovirt_engine_type: 'ovirt-engine'
ovirt_engine_version: '4.1'
ovirt_engine_organization: 'dwhmanualenginetest.ovirt.org'
ovirt_engine_admin_password: 'secret'
ovirt_rpm_repo: 'http://resources.ovirt.org/pub/yum-repo/ovirt-release41.rpm'
ovirt_engine_organization: 'ovirt.org'
ovirt_engine_dwh_db_host: 'remotedwh.ovirt.org'
ovirt_engine_dwh_db_configure: false
roles:
- role: ovirt-common
- role: ovirt-engine-install-packages
- role: ovirt-engine-setup

Running this play file would be done like

$ ansible-playbook site.yml -i inventory --skip-tags skip_yum_install_ovirt_engine_dwh,skip_yum_install_ovirt_engine_dwh_setup

After which you have to configure your remote DWH installation to the previous host which has the ovirt-engine installation which again has two possibilities

On your machine B which would be dwhmanualdwhtest.ovirt.org in this case

---
- hosts: dwh-remote
vars:
ovirt_engine_type: 'ovirt-engine'
ovirt_engine_version: '4.1'
ovirt_rpm_repo: 'http://resources.ovirt.org/pub/yum-repo/ovirt-release41.rpm'
ovirt_engine_host_root_passwd: 'admin123' # the root password of the host where ovirt-engine is installed
ovirt_engine_firewall_manager: 'firewalld'
ovirt_engine_db_host: 'dwhmanualenginetest.ovirt.org' # FQDN of the ovirt-engine installation host, should be resolvable from the new DWH host
ovirt_engine_host_fqdn: 'dwhmanualenginetest.ovirt.org'
ovirt_engine_dwh_db_host: 'localhost'
ovirt_dwh_on_dwh: True
roles:
- role: ovirt-common
- role: ovirt-engine-install-packages
- role: ovirt-engine-remote-dwh-setup

Running this play file would be done like

$ ansible-playbook site.yml -i inventory --skip-tags skip_yum_install_ovirt_engine,skip_yum_install_ovirt_engine_dwh

After this you have to restart your ovirt-engine on the host installed by doing a service ovirt-engine restart

And you have the whole setup now.

Possibilty of the database of dwh on 3rd machine

I have not taken this case as of now in the new playbook which I wrote. I was able to figure out how to do it manually but the specific changes for getting this option to work using our roles is still pending.

Next goals

  • The very first thing to do now would be to modify the existing ovirt-engine-setup-remote-dwh that I wrote to include the case of putthing the DB of the DWH on a 3rd machine.
  • Also, Migration dwh from local machine of engine to remote machine in a new playbook which counts with already installed engine with local dwh and moved dwh to this remote server is also on the task list
  • Writing tests for the new additions

Pitfalls

The most arcane error that I faced during this was when I was trying to automate the configuration of the remote DWH with the engine host.

arcane error which was filed as a bug on bugzilla later

From a first glance at the error traceback, it was clear that there was some error reading the engine hostname from the OTOPI based answerfile that I had generated.

I cross checked again and again from the answerfile which was generated from the manual installation of remote DWH as described in the process above, but to no avail.

Both of them looked just the same. I was not missing anything out from the original answerfile and they were basically the same thing minus the hostnames and passwords.

To be frank, I really didn’t know what was missing out here.

On a closer look at the manual input being given while installation and the answerfile being generated from it, I noticed that the engine HOST FQDN that we provide is not being logged in the OTOPI answerfile.

Cross checking with the answerfile being generated on a new host, and sure enough. There was a prompt for the host engine FQDN which confirmed my hunch.

hunch which turned out to be true

This had to be the case! I mean there was no other explanation. I asked Lukas and he said this possibly might be a bug and suggested that I file a bug on bugzilla.

Which I did here at https://bugzilla.redhat.com/show_bug.cgi?id=1465859 which lead me to my first bug report :)

Cross checking with the answerfile being generated on a new host, and sure enough. There was a prompt for the host engine FQDN which confirmed my hunch.

Anyway, now I just had to figure out a way of what was the OTOPI config being logged in the log files when I did enter this FQDN. Checking the logs for that particular engine-run, the logs showed me that the variable being logged.

It was OVESETUP_ENGINE_CONFIG/fqdn. I placed this on my template answerfile and added the remote engine host value to it.

And sure enough, it ran successfully :)

Crazy day! But that feeling when you successfully debug something. It’s always priceless no matter if it isn’t the first time you are doing so.

Until next time!

Links


Originally published at tasdikrahman.me on June 28, 2017.

If you liked what you read, you can share the love by clicking the heart button. My twitter handle is @tasdikrahman

Totally inspired by FreeCodeCamp.com