Published in
15 min readDec 30, 2017
Decentralized resource management for Openstack cloud(IAAS)
In terms of cloud, resource management/metering is one of the key task of the operational team. When it comes to small and medium size businesses this is far more important due to the limited resources available.
How do we start IAAS @WSO2
Few years back we started to use Openstack as IAAS for internal usage and we have nearly 5 tenants dedicated for different purposes which is backed by OpenLDAP as a keystone. Mainly more than 400 engineers start to use the cloud for various purposes such as
- To simulate customer scenarios
- For various dev activities
- For performance testing and many more…
For different tenants we have dedicated admins assigned who are responsible for managing instances for others and when work is done they are terminating instances upon acknowledgement by the relevant engineer.
Typical human mistakes we faced.
One of the main issue we faced with this approach was we were reaching the resource limits in the cloud, after analyzing all instances we found that relevant engineers who used the instances have forgot to inform tenant admins even after they completed their tasks. This keeps happening no matter we educate the folks so we have to find alternative way to overcome this issue.
Openstack resource monitoring with Telemetry
Mainly we decided to go with this approach to identify instances which are not used based on resource usage pattern so that we can terminate instances automatically. We used Telemetry to collect resource usage on each instances consisting of Network and VCPU data. Using Telemetry API we fetched data within a given time period and stored the average value (VCPU and Network usage) in to Mysql database per each instance. We used bash scripts to calculate the average resource usage per instance for past 5 days and if the value is less than a given threshold we trigger email alerts and if the average resource usage is lesser than the threshold for past 10 days instance will be terminated.
We tried this approach for nearly 1 year but again we faced the resource limitation. This time also it was a human error and we identified when engineers use instances to setup/test certain deployments they left the deployment running on the instances. Downside is even though people do not use instances anymore Telemetry consider those instances are being used due to the resource consumption by the running deployments.
What next?
Finally we have decided to terminate instances after given time period and the same time provide the ability to extend the instance retention period based on tenant admin’s approval. So wrote simple PHP web application which is mainly responsible for
- Creating/Deleting instances , by default instances are having limited life time and the creator has the ability to select the owner for an instance.
- Provide the ability to extend the default life time upon tenant admin’s approval
- Provide the ability to mark the instance as permanent upon tenant admin’s approval
- Based on remaining life time alerts will be triggered to the instance owner and if not extended instance will be terminated.
Since we have decided to use the PHP app, we have disabled the instance creation/deletion for all other users and restricted only to the PHP application user using nova policy file role definition. So far this approach seems working fine for us in terms of efficient usage of compute resources.
Below are the sample PHP code we used for each functions.We tested this with Openstack Rocky. For some API calls it uses tenant name and for some tenat IDs.We map tenant name -> tenant ID mapping in the config file as below.
tenant_id_infra="7c27vb4589lk89e2366383b090029";and from the code when we require tenant ID we refer as below when the tenant name is provided ($tenant).
${"tenant_id_" . $tenant}Getting the auth token as PHP app user for a given tenant and store in the DB for other API calls.
function authenticateWithIaas($tenant)
{
/*
* Logic:
*
* $tenant : tenant name
*
* Usually auth token is issued with 3600 validity.
* Here we are using a user, who has access to all tenants for instance management,
*
* 1. Check valid token in DB for a given user+tenant, if found return it.
* 2. If valid token not (expired) found ,create new one , store and return it.
* 3. If no token found for given user+tenant create new one , store and return it.
*/
include('config/vars.php');
$log = new log();
$date = date_create();
$current_timestamp = date_timestamp_get($date);
//Mysql conn
$conn = mysql_connect($mysql_server_ip, $mysql_uname, $mysql_passwd);
//Iaas credentials From Config File
$user = $iaas_user;
$password = $iaas_user_password;
if (!$conn) {
die('Could not connect: ' . mysql_error());
}
mysql_select_db($mysql_db);
$sql = "SELECT auth_token_created_at from infra_portal_diy_stg.iaas_auth_sessions where user_id= '{$user}' AND tenant='{$tenant}' ";
$retval = mysql_query($sql, $conn);
$created_timestamp = mysql_fetch_assoc($retval);
if ($retval) {
//Token exists for given tenant. Now check the validity of the token.
//Should be less than 3600s old
$log->user("Token Found for tenant - " . $tenant . ", checking the validity", $_SESSION['s_username']);
//Get the Current Time stamp and compare with token created time stamp
$age_of_the_token = ($current_timestamp - $created_timestamp['auth_token_created_at']);
if ($age_of_the_token < 3600) {
//Token is valid
$log->user("Token Found for tenant- " . $tenant . " in DB, its valid , Age of the Token: " . $age_of_the_token, $_SESSION['s_username']);
$sql3 = "SELECT auth_token from infra_portal_diy_stg.iaas_auth_sessions where user_id= '{$user}' AND tenant='{$tenant}'";
$retval3 = mysql_query($sql3, $conn);
$auth_token_from_db = mysql_fetch_assoc($retval3);
return $auth_token_from_db['auth_token'];
} else {
//Token is not valid so create new and store it
$log->user("Token Found for tenant- " . $tenant . " in DB , its Not valid (Expired) , Age of the Token: " . $age_of_the_token, $_SESSION['s_username']);
$data = "{\"auth\":{\"identity\":{\"methods\":[\"password\"],\"password\":{\"user\":{\"domain\":{\"id\":\"default\"},\"name\":\"$user\",\"password\":\"$password\"}}},\"scope\":{\"project\":{\"domain\":{\"id\":\"default\"},\"name\": \"$tenant\"}}}}";
$curl0 = curl_init($iaas_keystone_token_url);
curl_setopt($curl0, CURLOPT_HTTPHEADER, array('Content-Type: application/json',));
curl_setopt($curl0, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl0, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl0, CURLOPT_HEADER, 1);
curl_setopt($curl0, CURLOPT_NOBODY, 1);
curl_setopt($curl0, CURLOPT_POST, true);
curl_setopt($curl0, CURLOPT_POSTFIELDS, $data);
$curl_response0 = curl_exec($curl0);
$header_size = curl_getinfo($curl0, CURLINFO_HEADER_SIZE);
$headers = substr($curl_response0, 0, $header_size);
$headers = explode("\r\n", $headers);
$auth_token_pices = explode(" ", $headers[3]);
$auth_token = trim($auth_token_pices[1]);
//Get the Token and store it on the DB
$sql2 = "INSERT INTO infra_portal_diy_stg.iaas_auth_sessions (user_id,tenant,auth_token,auth_token_created_at) VALUES ('{$user}','{$tenant}' ,'{$auth_token}','{$created_timestamp}') ON DUPLICATE KEY UPDATE auth_token='{$auth_token}' ,auth_token_created_at='{$current_timestamp}' ";
$retval2 = mysql_query($sql2, $conn);
if ($retval2) {
$log->user("Iaas Token successfully saved", $_SESSION['s_username']);
} else {
$log->user("Iaas Token saving Failed", $_SESSION['s_username']);
}
return $auth_token;
}
} else {
//Token not yet stored for given Tenant+user
$log->user("Token Not Found for tenant -" . $tenant . ", Creating new Token", $_SESSION['s_username']);
$data = "{\"auth\":{\"identity\":{\"methods\":[\"password\"],\"password\":{\"user\":{\"domain\":{\"id\":\"default\"},\"name\":\"$user\",\"password\":\"$password\"}}},\"scope\":{\"project\":{\"domain\":{\"id\":\"default\"},\"name\": \"$tenant\"}}}}";
$curl0 = curl_init($iaas_keystone_token_url);
curl_setopt($curl0, CURLOPT_HTTPHEADER, array('Content-Type: application/json',));
curl_setopt($curl0, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl0, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl0, CURLOPT_HEADER, 1);
curl_setopt($curl0, CURLOPT_NOBODY, 1);
curl_setopt($curl0, CURLOPT_POST, true);
curl_setopt($curl0, CURLOPT_POSTFIELDS, $data);
$curl_response0 = curl_exec($curl0);
$header_size = curl_getinfo($curl0, CURLINFO_HEADER_SIZE);
$headers = substr($curl_response0, 0, $header_size);
$headers = explode("\r\n", $headers);
$auth_token_pices = explode(" ", $headers[3]);
$auth_token = trim($auth_token_pices[1]);
//Get the Token and store it on the DB
$sql2 = "INSERT INTO infra_portal_diy_stg.iaas_auth_sessions (user_id,tenant,auth_token,auth_token_created_at) VALUES ('{$user}','{$tenant}' ,'{$auth_token}','{$created_timestamp}') ON DUPLICATE KEY UPDATE auth_token='{$auth_token}' ,auth_token_created_at='{$current_timestamp}' ";
$retval2 = mysql_query($sql2, $conn);
if ($retval2) {
$log->user("Iaas Token successfully saved", $_SESSION['s_username']);
} else {
$log->user("Iaas Token saving Failed", $_SESSION['s_username']);
}
return $auth_token;
}
}Since we are using PHP app user, for instance creation ,this user is having access to all tenants. So from the PHP app we have to restrict tenant access. We refer LDAP structure we are using to authenticate users to find tenants a given user is belong.
sample LDAP Structure we used.
dn: ou=openstack-rocky,dc=wso2,dc=com
objectClass: organizationalUnit
objectClass: top
ou: openstack-rockydn: ou=groups,ou=openstack-rocky,dc=wso2,dc=com
objectClass: organizationalUnit
objectClass: top
ou: groupsdn: ou=services,ou=groups,ou=openstack-rocky,dc=wso2,dc=com
objectClass: organizationalUnit
objectClass: top
ou: servicesdn: cn=services,ou=services,ou=groups,ou=openstack-rocky,dc=wso2,dc=com
objectClass: groupOfNames
objectClass: top
cn: services
member: uid=cinder,ou=staff,dc=wso2,dc=com
member: uid=glance,ou=staff,dc=wso2,dc=com
member: uid=nova,ou=staff,dc=wso2,dc=com
member: uid=neutron,ou=staff,dc=wso2,dc=com
member: uid=ceilometer,ou=staff,dc=wso2,dc=com
member: uid=heat,ou=staff,dc=wso2,dc=com
member: uid=placement,ou=staff,dc=wso2,dc=com
member: uid=heat_cfn,ou=staff,dc=wso2,dc=comdn: ou=admin,ou=groups,ou=openstack-rocky,dc=wso2,dc=com
objectClass: organizationalUnit
objectClass: top
ou: admindn: cn=admin,ou=admin,ou=groups,ou=openstack-rocky,dc=wso2,dc=com
objectClass: groupOfNames
objectClass: top
cn: admin
member: uid=admin,ou=staff,dc=wso2,dc=comdn: ou=infra,ou=groups,ou=openstack-rocky,dc=wso2,dc=com
objectClass: organizationalUnit
objectClass: top
ou: infradn: cn=infra_admin,ou=infra,ou=groups,ou=openstack-rocky,dc=wso2,dc=com
objectClass: groupOfNames
objectClass: top
cn: infra_admin
member: uid=admin,ou=staff,dc=wso2,dc=comdn: cn=infra_member,ou=infra,ou=groups,ou=openstack-rocky,dc=wso2,dc=com
objectClass: groupOfNames
objectClass: top
cn: infra_member
member: uid=infra_portal_diy,ou=staff,dc=wso2,dc=comdn: cn=infra_reader,ou=infra,ou=groups,ou=openstack-rocky,dc=wso2,dc=com
objectClass: groupOfNames
objectClass: top
cn: infra_reader
member: uid=user1,ou=staff,dc=wso2,dc=com
member: uid=user2,ou=staff,dc=wso2,dc=com
member: uid=user3,ou=staff,dc=wso2,dc=com
member: uid=user4,ou=staff,dc=wso2,dc=com
member: uid=user5,ou=staff,dc=wso2,dc=com
member: uid=user6,ou=staff,dc=wso2,dc=com
We retrieve list of tenant from above LDAP structure using below function.
function get_iaas_tenant_for_user($user_id)
{
/*
* Logic:
*
* $user_id : User name using for IAAS auth
*
* For instance creation we use common user who has access to all tenants, so we need to restrict access
* to each tenant based on user.
*
* IAAS auth is LDAP integrated, we have LDAP structure for that containing groups to map with tenant.
*
* 1. By getting user id we query the LDAP to get the groups the user id contains, from that we get the list of
* tenants the user is authorized to access.
*/
include('config/vars.php');
$log = new log();
$ldapconn = ldap_connect($private_ldap_host) or die("Could not connect to LDAP server.");
ldap_set_option($ldapconn, LDAP_OPT_PROTOCOL_VERSION, 3);
$ldapbind = ldap_bind($ldapconn, $private_ldap_admin_rdn, $private_ldap_admin_passwd);
$ldap_base_dn = "ou=groups,ou=openstack-rocky,dc=wso2,dc=com";
$private_ldap_search_result = ldap_search($ldapconn, $ldap_base_dn, "(&(member=uid=" . $user_id . ",ou=staff,dc=wso2,dc=com)(objectClass=groupOfNames))");
$entry = ldap_first_entry($ldapconn, $private_ldap_search_result);
do {
$dn = ldap_get_dn($ldapconn, $entry);
$pieces = explode("=", explode(",", $dn)[1]);
//Create Array of Groups which user belongs
$iaas_user_tenants[] = $pieces[1];
} while ($entry = ldap_next_entry($ldapconn, $entry));
return $iaas_user_tenants;
ldap_close($ldapconn);
}We refer list of tenants returned by above code to determine list of authorized tenants for a given user.
Getting list of flavors for instance creation is as follows.
function get_iaas_flavors($token, $tenant)
{
/*
*
* Get the Iaas flavors for instances creation.
*
* In this API we should provide tenant ID instead of tenant name. We refer tenant ID from config file.
*
*/
include('config/vars.php');
$log = new log();
$service_url0 = $iaas_compute_api_url . "/" . ${"tenant_id_" . $tenant} . '/flavors';
$curl0 = curl_init($service_url0);
curl_setopt($curl0, CURLOPT_HTTPHEADER, array("X-Auth-Token:" . $token, 'Content-Type: application/json'));
curl_setopt($curl0, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl0, CURLOPT_SSL_VERIFYPEER, false);
$curl_response0 = curl_exec($curl0);
#Decode the Json Object in to Associative Array
$json_user_info0 = json_decode($curl_response0, true);
foreach ($json_user_info0 as $flavors) {
foreach ($flavors as $flavor) {
$flavor_id_name[$flavor['id']] = $flavor['name'];
}
}
return $flavor_id_name;
}Using below function we can get details of a given flavor , such as RAM , Disk size.
function get_flavor_details($flavor_id, $token, $tenant)
{
/*
*Get the flavor details , such as RAM, Disk size , VCPUs
*
* In this API we should provide tenant ID instead of tenant name. We refer tenant ID from config file.
*
*
*/
include('config/vars.php');
$log = new log();
$service_url0 = $iaas_compute_api_url . "/" . ${"tenant_id_" . $tenant} . '/flavors/' . $flavor_id;
//echo $service_url0;
$curl0 = curl_init($service_url0);
curl_setopt($curl0, CURLOPT_HTTPHEADER, array("X-Auth-Token:" . $token, 'Content-Type: application/json'));
curl_setopt($curl0, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl0, CURLOPT_SSL_VERIFYPEER, false);
$curl_response0 = curl_exec($curl0);
#Decode the Json Object in to Associative Array
$json_user_info0 = json_decode($curl_response0, true);
return $json_user_info0;
}Getting a list of all publicly shared images.
function get_iaas_images($token)
{
/*
*Get the all publicly shared image list to be use for instance creation.
*
*/
include('config/vars.php');
$log = new log();
$service_url0 = $iaas_glance_api_url . "/" . 'images?sort_key=name&sort_dir=asc&limit=20';
$curl0 = curl_init($service_url0);
curl_setopt($curl0, CURLOPT_HTTPHEADER, array("X-Auth-Token:" . $token, 'Content-Type: application/json'));
curl_setopt($curl0, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl0, CURLOPT_SSL_VERIFYPEER, false);
$curl_response0 = curl_exec($curl0);
#Decode the Json Object in to Associative Array
$json_user_info0 = json_decode($curl_response0, true);
$image_id_name_array = array();
while (!empty($json_user_info0['images'])) {
foreach ($json_user_info0 as $images) {
foreach ($images as $image) {
$image_id_name[$image['id']] = $image['name'];
}
}
$image_id_name_array = array_merge($image_id_name_array, $image_id_name);
$last_image_name = end($image_id_name);
$last_image_id = array_keys($image_id_name, $last_image_name)[0];
$service_url0 = $iaas_glance_api_url . "/" . 'images?sort_key=name&sort_dir=asc&limit=20&marker=' . $last_image_id;
$curl0 = curl_init($service_url0);
curl_setopt($curl0, CURLOPT_HTTPHEADER, array("X-Auth-Token:" . $token, 'Content-Type: application/json'));
curl_setopt($curl0, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl0, CURLOPT_SSL_VERIFYPEER, false);
$curl_response0 = curl_exec($curl0);
#Decode the Json Object in to Associative Array
$json_user_info0 = json_decode($curl_response0, true);
}
//Paginate to get all images
return $image_id_name_array;
}Getting list of networks and return based on selected tenant.
function get_networks($token, $tenant)
{
/*
* Get the all networks the common user has access to, and based on the selected tenant match and return the
* network list along with network name+id
*
* Here tenant ID will be used instead of tenant name.
*
*/
include('config/vars.php');
$log = new log();
$tenant = ${"tenant_id_" . $tenant};
$service_url0 = $iaas_neutron_api_url . "/" . 'networks';
//echo $service_url0;
$curl0 = curl_init($service_url0);
curl_setopt($curl0, CURLOPT_HTTPHEADER, array("X-Auth-Token:" . $token, 'Content-Type: application/json'));
curl_setopt($curl0, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl0, CURLOPT_SSL_VERIFYPEER, false);
$curl_response0 = curl_exec($curl0);
#Decode the Json Object in to Associative Array
$json_user_info0 = json_decode($curl_response0, true);
//Filter the network list by tenant to exclude admin network(external)
foreach ($json_user_info0['networks'] as $networks) {
if ($networks['tenant_id'] == $tenant) {
$tenant_networks[$networks['id']] = $networks['name'];
}
}
return $tenant_networks;
}Get the SSH public keys stored under PHP app user.
function get_keys($token, $tenant)
{
/*
* Since we are using common user all user's public keys will be maintained under common user's account.
*
* This will return all available keys for instance creation.
*
* Here tenant ID will be used instead of tenant name.
*
*/
include('config/vars.php');
$log = new log();
$service_url0 = $iaas_compute_api_url . "/" . ${"tenant_id_" . $tenant} . '/os-keypairs';
//echo $service_url0;
$curl0 = curl_init($service_url0);
curl_setopt($curl0, CURLOPT_HTTPHEADER, array("X-Auth-Token:" . $token, 'Content-Type: application/json', "X-Auth-Project-Id:" . ${"tenant_id_" . $tenant}));
curl_setopt($curl0, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl0, CURLOPT_SSL_VERIFYPEER, false);
$curl_response0 = curl_exec($curl0);
#Decode the Json Object in to Associative Array
$json_user_info0 = json_decode($curl_response0, true);
foreach ($json_user_info0 as $keypairs) {
foreach ($keypairs as $keypair) {
$key_pair_names_list[] = $keypair['keypair']['name'];
}
}
return $key_pair_names_list;
}Also we use below function to show the tenant quota limits.
function get_tenant_limit($token, $tenant)
{
/*
* This will return tenant quota limits for display purposes.
*
*
*
* Here tenant ID will be used instead of tenant name.
*
*/
include('config/vars.php');
$log = new log();
$service_url0 = $iaas_compute_api_url . "/" . ${"tenant_id_" . $tenant} . '/limits';
//echo $service_url0;
$curl0 = curl_init($service_url0);
curl_setopt($curl0, CURLOPT_HTTPHEADER, array("X-Auth-Token:" . $token, 'Content-Type: application/json', "X-Auth-Project-Id:" . ${"tenant_id_" . $tenant}));
curl_setopt($curl0, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl0, CURLOPT_SSL_VERIFYPEER, false);
$curl_response0 = curl_exec($curl0);
#Decode the Json Object in to Associative Array
$json_user_info0 = json_decode($curl_response0, true);
return $json_user_info0['limits']['absolute'];
}Finally create the instances as below and store the details in DB, in the creation we can select a owner of the instance (Unless creator will be the owner), where he will be the point of contact for the given instance and he will get all alerts with regards to instance retention and termination.
function create_iaas_instance($token, $tenant, $instance_name, $image_id, $key_name, $falvor_id, $network_id, $instance_owner, $instance_creator)
{
/*
* This will create a instances once all other required params are provided.
*
*
*
* Here tenant ID will be used instead of tenant name.
*
* 1. Once instances creation API invoked and it was successful, those information will be saved in DB.
* 2. We have 2 policies in terms of retention time. For DEV tenant we use default 2 day and for other tenants it is 14 days. ]
* Retention information also get saved for a given instances in the DB.
* 3. We use separate scripts to check the DB and generate alerts when retention time reach and terminate if not extended from the DB.
*
*/
$log = new log();
include('config/vars.php');
//Mysql conn
$conn = mysql_connect($mysql_server_ip, $mysql_uname, $mysql_passwd);
if (!$conn) {
die('Could not connect: ' . mysql_error());
}
$service_url0 = $iaas_compute_api_url . "/" . ${"tenant_id_" . $tenant} . '/servers';
$curl0 = curl_init($service_url0);
curl_setopt($curl0, CURLOPT_HTTPHEADER, array("X-Auth-Token:" . $token, 'Content-Type: application/json', "X-Auth-Project-Id:" . ${"tenant_id_" . $tenant}));
$data = "{\"server\": {\"name\": \"$instance_name\", \"imageRef\": \"$image_id\", \"key_name\": \"$key_name\", \"flavorRef\": \"$falvor_id\", \"max_count\": \"1\", \"min_count\": \"1\", \"networks\": [{\"uuid\": \"$network_id\"}], \"security_groups\": [{\"name\": \"default\"}]}}";
curl_setopt($curl0, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl0, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl0, CURLOPT_POST, true);
curl_setopt($curl0, CURLOPT_POSTFIELDS, $data);
$curl_response0 = curl_exec($curl0);
$httpcode = curl_getinfo($curl0, CURLINFO_HTTP_CODE);
#Decode the Json Object in to Associative Array
$json_user_info0 = json_decode($curl_response0, true);
$instance_uuid = $json_user_info0['server']['id'];
$date = date_create();
$current_timestamp = date_timestamp_get($date);
/* default instance_life_time_in_days
*
* for DEV tenant - 2
* other tenant - 14
* */
if ($tenant == "dev") {
$default_instance_life_time = 2;
} else {
$default_instance_life_time = 14;
}
//On instance creation success, record it.
if ($httpcode == "202") {
echo "<font color='green' size='4'> Requested to launch the instance.</font> </br>";
//Store the Instance details in the DB up on Instance creating WS returned Suvvess
$sql2 = "INSERT INTO infra_portal_diy_stg.iaas_instance_list (instance_uuid,instance_name,tenant,instance_owner,instance_creator,instance_keypair_name,instance_flavor_id,
instance_network_id,instance_image_id,instance_created_at,instance_life_time_in_days,is_instance_active,is_instance_permanant)
VALUES ('{$instance_uuid}','{$instance_name}','{$tenant}' ,'{$instance_owner}','{$instance_creator}','{$key_name}','{$falvor_id}','{$network_id}'
,'{$image_id}','{$current_timestamp}','{$default_instance_life_time}','1','0') ";
$retval2 = mysql_query($sql2, $conn);
if ($retval2) {
$log->user("Iaas Instance Created and Recorded to the DB Successfully.", $_SESSION['s_username']);
} else {
$log->user("Iaas Instance Created But Recording Failed to the DB. " . $sql2, $_SESSION['s_username']);
}
} else {
echo "<font color='red' size='4'>Failed to launch the instance.</font>";
}
}Getting the instance state(ACTIVE, SHUTT OFF..etc) using below code.
function getInstanceDetails($authToken, $tenant, $uuid)
{
/*
* This will get instance status for display purposes.
*
* Here tenant ID will be used instead of tenant name.
*
*/
include('config/vars.php');
$log = new log();
$service_url0 = $iaas_compute_api_url . "/" . ${"tenant_id_" . $tenant} . '/servers/' . $uuid;
$curl0 = curl_init($service_url0);
curl_setopt($curl0, CURLOPT_HTTPHEADER, array("X-Auth-Token:" . $authToken));
curl_setopt($curl0, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl0, CURLOPT_SSL_VERIFYPEER, false);
$curl_response0 = curl_exec($curl0);
#Decode the Json Object in to Associative Array
$json_user_info0 = json_decode($curl_response0, true);
return $json_user_info0;
For start/stop instances we use below functions.
function stopIaasInstance($tenant, $uuid)
{
/*
* This will stop instance
*
* Here tenant ID will be used instead of tenant name.
*
*/
include('config/vars.php');
$log = new log();
$iaas_auth_token = authenticateWithIaas($tenant);
$service_url0 = $iaas_compute_api_url . "/" . ${"tenant_id_" . $tenant} . '/servers/' . $uuid . '/action';
$curl0 = curl_init($service_url0);
curl_setopt($curl0, CURLOPT_HTTPHEADER, array("X-Auth-Token:" . trim($iaas_auth_token), 'Content-Type: application/json', "X-Auth-Project-Id:" . ${"tenant_id_" . $tenant}));
$data = "{\"os-stop\": \"null\"}";
curl_setopt($curl0, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl0, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl0, CURLOPT_POST, true);
curl_setopt($curl0, CURLOPT_POSTFIELDS, $data);
$curl_response0 = curl_exec($curl0);
$httpcode = curl_getinfo($curl0, CURLINFO_HTTP_CODE);
#Decode the Json Object in to Associative Array
$json_user_info0 = json_decode($curl_response0, true);
if ($httpcode == "202") {
echo "<font color='green' size='4'> Requested to stop the instance.</font>";
} else {
echo "<font color='red' size='4'> Failed to stop the instance.</font>";
}
}
function startIaasInstance($tenant, $uuid)
{
/*
* This will start instance
*
* Here tenant ID will be used instead of tenant name.
*
*/
include('config/vars.php');
$log = new log();
$iaas_auth_token = authenticateWithIaas($tenant);
$service_url0 = $iaas_compute_api_url . "/" . ${"tenant_id_" . $tenant} . '/servers/' . $uuid . '/action';
$curl0 = curl_init($service_url0);
curl_setopt($curl0, CURLOPT_HTTPHEADER, array("X-Auth-Token:" . trim($iaas_auth_token), 'Content-Type: application/json', "X-Auth-Project-Id:" . ${"tenant_id_" . $tenant}));
$data = "{\"os-start\": \"null\"}";
curl_setopt($curl0, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl0, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl0, CURLOPT_POST, true);
curl_setopt($curl0, CURLOPT_POSTFIELDS, $data);
$curl_response0 = curl_exec($curl0);
$httpcode = curl_getinfo($curl0, CURLINFO_HTTP_CODE);
#Decode the Json Object in to Associative Array
$json_user_info0 = json_decode($curl_response0, true);
if ($httpcode == "202") {
echo "<font color='green' size='4'> Requested to start the instance. </font>";
} else {
echo "<font color='red' size='4'> Failed to start the instance.</font>";
}
}Finally when we terminate instances we update the DB as well as it is terminated.
function destroyIaasInstance($tenant, $uuid)
{
/*
* This will terminate instance
*
* Here tenant ID will be used instead of tenant name.
*
* Once terminated this will be marked as terminated in the DB as well.
*
*/
include('config/vars.php');
$log = new log();
$iaas_auth_token = authenticateWithIaas($tenant);
//Mysql conn
$conn = mysql_connect($mysql_server_ip, $mysql_uname, $mysql_passwd);
if (!$conn) {
die('Could not connect: ' . mysql_error());
}
$service_url0 = $iaas_compute_api_url . "/" . ${"tenant_id_" . $tenant} . '/servers/' . $uuid;
$curl0 = curl_init($service_url0);
curl_setopt($curl0, CURLOPT_HTTPHEADER, array("X-Auth-Token:" . trim($iaas_auth_token), 'Content-Type: application/json', "X-Auth-Project-Id:" . ${"tenant_id_" . $tenant}));
curl_setopt($curl0, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl0, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl0, CURLOPT_CUSTOMREQUEST, "DELETE");
$curl_response0 = curl_exec($curl0);
#Decode the Json Object in to Associative Array
$json_user_info0 = json_decode($curl_response0, true);
$httpcode = curl_getinfo($curl0, CURLINFO_HTTP_CODE);
if ($httpcode == "204") {
echo "<font color='green' size='4'> Requested to terminate the instance.</font>";
//Update the DB to set the is_instance_active=0
$sql2 = "UPDATE infra_portal_diy_stg.iaas_instance_list SET is_instance_active='0' WHERE instance_uuid='{$uuid}'";
$update_inactive_instance_result_set = mysql_query($sql2, $conn);
if ($update_inactive_instance_result_set) {
$log->user("Mysql DB update Success For recording Inactive Instance state", $_SESSION['s_username']);
} else {
$log->user("Mysql DB update Failed For recording Inactive Instance state", $_SESSION['s_username']);
}
} else {
echo "<font color='red' size='4'> Failed to terminate the instance.</font>";
}
}For Instance retention we use separate script which is executed via Cron. This will check the DB for instances details and generate alerts and terminate instances when retention period expired. Below is the table structure we use for mainly for storing auth tokens and instance details.
Table: iaas_auth_sessions
Create Table: CREATE TABLE `iaas_auth_sessions` (
`id` int(20) unsigned NOT NULL AUTO_INCREMENT,
`user_id` varchar(500) NOT NULL,
`tenant` varchar(500) NOT NULL,
`auth_token` varchar(10000) NOT NULL,
`auth_token_created_at` varchar(100) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `unique_index` (`user_id`,`tenant`)
) ENGINE=InnoDB AUTO_INCREMENT=14 DEFAULT CHARSET=latin1Table: iaas_instance_list
Create Table: CREATE TABLE `iaas_instance_list` (
`id` int(20) unsigned NOT NULL AUTO_INCREMENT,
`instance_uuid` varchar(500) NOT NULL,
`instance_name` varchar(500) NOT NULL,
`tenant` varchar(100) NOT NULL,
`instance_owner` varchar(200) NOT NULL,
`instance_creator` varchar(200) NOT NULL,
`instance_keypair_name` varchar(200) NOT NULL,
`instance_flavor_id` varchar(500) NOT NULL,
`instance_network_id` varchar(500) NOT NULL,
`instance_image_id` varchar(500) NOT NULL,
`instance_created_at` varchar(100) NOT NULL,
`instance_life_time_in_days` int(5) NOT NULL,
`is_instance_active` int(2) NOT NULL,
`is_instance_permanant` int(2) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1799 DEFAULT CHARSET=latin1
Other than the above details we are separately maintaining tenant admins/approvals, who can approve instance retention extend requests. We use below table structures for that.
Table: iaas_tenant_admins
Create Table: CREATE TABLE `iaas_tenant_admins` (
`id` int(20) unsigned NOT NULL AUTO_INCREMENT,
`tenant` varchar(500) NOT NULL,
`user_id` varchar(500) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `unique_index` (`user_id`,`tenant`)
) ENGINE=InnoDB AUTO_INCREMENT=69 DEFAULT CHARSET=latin1Table: iaas_extend_approval_request
Create Table: CREATE TABLE `iaas_extend_approval_request` (
`id` int(20) unsigned NOT NULL AUTO_INCREMENT,
`instance_name` varchar(500) NOT NULL,
`instance_uuid` varchar(500) NOT NULL,
`instance_tenant` varchar(500) NOT NULL,
`instance_life_time_in_days` int(5) NOT NULL,
`remaining_lifetime_in_days` int(5) NOT NULL,
`extra_lifetime_needed_in_days` int(5) NOT NULL,
`is_requesting_for_permanant_instance` int(2) NOT NULL,
`instance_owner` varchar(500) NOT NULL,
`instance_creator` varchar(500) NOT NULL,
`approved_by` varchar(500) NOT NULL,
`approved_at` varchar(100) NOT NULL,
`is_approved` int(2) NOT NULL,
`is_rejected` int(2) NOT NULL,
`rejected_by` varchar(500) NOT NULL,
`rejected_at` varchar(100) NOT NULL,
`reason_for_requsting` varchar(5000) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=865 DEFAULT CHARSET=latin1
Thanks.
