Empowering Incident Response with Automated Remediation: A Journey of Hope and Transformation
Published in
14 min readSep 5, 2023
Humans as a civilization are often seen as problem solvers and providers. This fundamental aspect of our nature has been a driving force throughout our history, and it continues to shape our present and future, even in the face of extraordinary challenges.
With ever-increasing IT incidents, harnessing our problem-solving capabilities and adapting them to the evolving landscape becomes even more critical. It is the need of the hour to have systems in place for automated remediation and self-healing systems, which can save and protect themselves from any kind of breach/intrusion.
Hence we embarked on a mission to automate complex problem-solving techniques, making them accessible and reliable for individuals with varying levels of experience. It’s an exciting challenge, to witness the gears grinding against each other to solve a production outage.
Once you identify the usual suspects, it’s just a game of combination and permutations, employing the power of elimination. Over years of working in DevOps, this becomes kind of second nature to you, imprinted in your muscle memory. The trick is to leverage this crystallized wisdom and put/package into a strong and resilient framework so that you can fix incidents quickly and reliably with a few keystrokes because time as we know is of utmost importance when it comes to incidents and rescue missions like these.
We are pooling our knowledge and resources to dissect and comprehend the intricacies of the challenges we are facing. Our system is designed to leverage Slack, a familiar communication platform in the world of chaos, as its entry point. We seamlessly initiate/execute remediation playbooks, each tailored to address specific issues (High Database Connections, Increased Response Time, performance bottlenecks caused by database spikes, elusive locks, and more). Our automated playbooks will be the knight in shining armor, and first level of defense for battling such wars efficiently.
It feels like modern-day wizards, wielding the power of automation to conjure solutions with a few keystrokes. The world is transforming before our eyes. Yet, our journey is far from over. The threats and avenues for failure are ever-evolving. Thus, we will remain committed to refining our framework, adapting, and expanding its capabilities, and training new generations to wield tools like this to make them better equipped to deal with chaotic incidents and help anchor them on the shore of clarity.
In a world where IT incidents, and anomalies, are a norm, let’s take the first step toward restoring order and promise. Embracing automation and leveraging complex problem-solving techniques, we can become architects of our destiny, forging a path to a brighter future — one automated playbook at a time.
OK, that’s enough theory (disguised as creative story-telling), let’s deep-dive into the implementation specifics of our Automated Remediation Template (ART from now on).
Components of ART are:
- Slack
- Slash Commands (Interactive automated commands that execute cool playbooks)
- Remediation Scripts/Recipes (Executed in response to the above Slash Commands)
- Jenkins (to orchestrate and view Remediation job status/debug output for visibility and audit purposes)
Check this cool/official blog post from Slack on how to integrate Slash commands for executing any API/Scripts remotely from the comfort of your Slack workspace. ChatOps ftw!
Now, let’s look at the most interesting part, i.e. Our Remediation scripts:
#!/bin/sh
# Check if an argument was provided
#if [ $# -eq 0 ]; then
# echo "Usage: $0 <option>"
# exit 1
#fi
SSM_DETAILS=$(aws secretsmanager get-secret-value \
--secret-id arn:aws:secretsmanager:ap-southeast-1:<aws_acct_id>:secret:<super_secret_db_pass \
--query SecretString \
--output text \
--region ap-southeast-1)
USERNAME=`echo ${SSM_DETAILS} | jq -r '.username'`
PGSQL_PASSWORD=`echo ${SSM_DETAILS} | jq -r '.password'`
DATABASE=`echo ${SSM_DETAILS} | jq -r '.dbname'`
# DATABASE=<db_name>
# USERNAME=<username>
# HOSTNAME=<db_hostname_identifier>
export PGPASSWORD=$PGSQL_PASSWORD
#Bash Script directory path
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
DATE=`date +'%d%m%G'`
LOG="/tmp/KILL_ALL_IDLE_${BUILD_NUMBER}.log"
touch $LOG
################
OPTION="${OPTION}"
HOSTNAME="${HOSTNAME}"
INTERVAL="${INTERVAL}"
# Switch-case construct
case "$OPTION" in
"LWLock")
echo "Option LWLock selected"
psql -h $HOSTNAME -U $USERNAME $DATABASE -c "SELECT pg_terminate_backend (pg_stat_activity.pid)
from pg_stat_activity
WHERE pg_stat_activity.datname = 'crm'
and wait_event_type = 'LWLock';"
;;
"IdleQueries")
echo "Option IdleQueries selected"
psql -h $HOSTNAME -U $USERNAME $DATABASE -c " select datname, pid,now() - query_start as "runtime",state,state_change,query from pg_stat_activity WHERE pid <> pg_backend_pid()
AND state in ('idle', 'idle in transaction', 'idle in transaction (aborted)', 'disabled')"
;;
"All")
echo "Option All selected"
IDLECONNTIME=`psql -h $HOSTNAME -U $USERNAME $DATABASE -c "select now();"`
MAXCONN=`psql -h $HOSTNAME -U $USERNAME $DATABASE -c "show max_connections;"`
CONN=`psql -h $HOSTNAME -U $USERNAME $DATABASE -c "select count(*) from pg_stat_activity;"`
echo "$HOSTNAME: ALL IDLE Connetions Start Time (script start Time before killing session) : $IDLECONNTIME"
echo "Maximum Connections Defined : $MAXCONN"
echo "Current Connection Count : $CONN"
IDLECONNCOUNT=`psql -h $HOSTNAME -U $USERNAME $DATABASE -c "select count(*) from pg_stat_activity WHERE pid <> pg_backend_pid()
AND state in ('idle', 'idle in transaction', 'idle in transaction (aborted)', 'disabled')
AND state_change < current_timestamp - INTERVAL '$INTERVAL' MINUTE;"`
echo "Count of ALL IDLE connections more than $INTERVAL min: $IDLECONNCOUNT"
echo "$HOSTNAME: Connections ALL IDLE for more than $INTERVAL min"
psql -h $HOSTNAME -U $USERNAME $DATABASE -c " select datname, pid,now() - query_start as "runtime",state,state_change,query from pg_stat_activity WHERE pid <> pg_backend_pid()
AND state in ('idle', 'idle in transaction', 'idle in transaction (aborted)', 'disabled')
AND state_change < current_timestamp - INTERVAL '$INTERVAL' MINUTE;"
psql -h $HOSTNAME -U $USERNAME $DATABASE -c " SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE pid <> pg_backend_pid()
AND state in ('idle', 'idle in transaction', 'idle in transaction (aborted)', 'disabled')
AND state_change < current_timestamp - INTERVAL '$INTERVAL' MINUTE;"
psql -h $HOSTNAME -U $USERNAME $DATABASE -c " SELECT pg_terminate_backend (pg_stat_activity.pid)
from pg_stat_activity
WHERE pg_stat_activity.datname = 'crm'
AND state_change < current_timestamp - INTERVAL '120 minutes';"
;;
*)
echo "Invalid option: $option"
exit 1
;;
esac
IDLECONNTIME2=`psql -h $HOSTNAME -U $USERNAME $DATABASE -c "select now();"`
echo "ALL IDLE Connetions End Time (script End Time After session killed) : $IDLECONNTIME2"
exit 0;
Notice, that the above job is parameterized, and has choices to terminate queries which are Idle, Queries causing locks, and a generic case where we terminate queries held up for more than the selected duration/Interval.
Check the script in action below: (Some details are masked and redacted for security purposes)
Started by user Anon
Running as SYSTEM
[ATLASSIAN CLOUD PLUGIN] [WARN] Not a WorkflowRun, automatic builds and deployments won't work.
Building in workspace /var/lib/jenkins/jobs/pg-terminate-idle-queries/workspace
[workspace] $ /bin/sh /tmp/jenkins8237791744091738544.sh
/tmp/jenkins8237791744091738544.sh: 27: /tmp/jenkins8237791744091738544.sh: Bad substitution
Option IdleQueries selected
datname | pid | runtime | state | state_change | query
----------+-------+------------------+---------------------+----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
crm | 2377 | 00:00:20.692958 | idle | 2023-09-05 17:10:30.550753+05:30 | DEALLOCATE pdo_stmt_0000028b
crm | 3059 | 00:00:01.176911 | idle | 2023-09-05 17:10:50.066829+05:30 | COMMIT
crm | 3070 | 00:00:01.158376 | idle | 2023-09-05 17:10:50.085353+05:30 | COMMIT
crm | 2149 | 00:00:27.768672 | idle | 2023-09-05 17:10:23.475036+05:30 | DEALLOCATE pdo_stmt_00000067
crm | 3034 | 00:00:02.160069 | idle | 2023-09-05 17:10:49.083652+05:30 | COMMIT
crm | 3659 | 00:00:02.576169 | idle | 2023-09-05 17:10:48.667547+05:30 | COMMIT
crm | 3096 | 00:00:02.030139 | idle | 2023-09-05 17:10:49.213587+05:30 | COMMIT
crm | 13864 | 00:00:37.196732 | idle | 2023-09-05 17:10:14.049263+05:30 | COMMIT
crm | 11778 | -00:00:00.00046 | idle | 2023-09-05 17:10:51.244169+05:30 | DEALLOCATE pdo_stmt_000007a3
crm | 2406 | 00:00:11.315875 | idle | 2023-09-05 17:10:39.927847+05:30 | DEALLOCATE pdo_stmt_0000095e
crm | 2376 | 00:00:09.274044 | idle | 2023-09-05 17:10:41.969663+05:30 | DEALLOCATE pdo_stmt_00003155
crm | 2729 | 00:00:32.900445 | idle | 2023-09-05 17:10:18.343263+05:30 | DEALLOCATE pdo_stmt_0000002a
crm | 3080 | 00:00:01.139687 | idle | 2023-09-05 17:10:50.104034+05:30 | COMMIT
crm | 4296 | 00:00:01.239719 | idle | 2023-09-05 17:10:50.00401+05:30 | COMMIT
crm | 4397 | 00:00:02.159981 | idle | 2023-09-05 17:10:49.083742+05:30 | COMMIT
crm | 3068 | 00:00:00.674608 | idle | 2023-09-05 17:10:50.569111+05:30 | COMMIT
crm | 1385 | 00:00:34.73883 | idle | 2023-09-05 17:10:16.506341+05:30 | COMMIT
crm | 3056 | 00:00:01.406787 | idle | 2023-09-05 17:10:49.836939+05:30 | COMMIT
crm | 3232 | 00:00:00.179626 | idle | 2023-09-05 17:10:51.064096+05:30 | COMMIT
crm | 3815 | 00:00:02.718547 | idle | 2023-09-05 17:10:48.525174+05:30 | COMMIT
crm | 3097 | 00:00:00.161215 | idle | 2023-09-05 17:10:51.082485+05:30 | DEALLOCATE pdo_stmt_00000c62
crm | 3425 | 00:00:02.015659 | idle | 2023-09-05 17:10:49.22806+05:30 | COMMIT
crm | 3112 | 00:00:02.859097 | idle | 2023-09-05 17:10:48.384628+05:30 | COMMIT
crm | 4272 | 00:00:01.528979 | idle | 2023-09-05 17:10:49.714748+05:30 | COMMIT
crm | 4603 | 00:00:01.071812 | idle | 2023-09-05 17:10:50.171907+05:30 | COMMIT
crm | 3021 | 00:00:01.49782 | idle | 2023-09-05 17:10:49.745899+05:30 | COMMIT
crm | 3081 | 00:00:00.950438 | idle | 2023-09-05 17:10:50.293286+05:30 | COMMIT
crm | 3029 | 00:00:02.013247 | idle | 2023-09-05 17:10:49.23047+05:30 | COMMIT
rdsadmin | 11795 | 00:00:09.880609 | idle | 2023-09-05 17:10:41.363105+05:30 | SELECT value FROM rds_heartbeat2
crm | 3250 | 00:00:02.780012 | idle | 2023-09-05 17:10:48.463708+05:30 | COMMIT
crm | 3183 | 00:00:01.092083 | idle | 2023-09-05 17:10:50.151635+05:30 | COMMIT
crm | 4672 | 00:00:00.990416 | idle | 2023-09-05 17:10:50.253301+05:30 | COMMIT
crm | 3798 | 00:00:00.249838 | idle | 2023-09-05 17:10:50.993884+05:30 | COMMIT
crm | 1347 | 00:00:41.072389 | idle | 2023-09-05 17:10:10.171315+05:30 | DEALLOCATE pdo_stmt_0000009e
crm | 3131 | 00:00:01.571463 | idle | 2023-09-05 17:10:49.67226+05:30 | COMMIT
crm | 4369 | 00:00:01.09513 | idle | 2023-09-05 17:10:50.148589+05:30 | COMMIT
crm | 3111 | 00:00:02.909698 | idle | 2023-09-05 17:10:48.334021+05:30 | COMMIT
crm | 3123 | 00:00:01.718886 | idle | 2023-09-05 17:10:49.524846+05:30 | COMMIT
crm | 3121 | 00:00:02.43473 | idle | 2023-09-05 17:10:48.808987+05:30 | COMMIT
crm | 3215 | 00:00:00.55731 | idle | 2023-09-05 17:10:50.686412+05:30 | COMMIT
crm | 3152 | 00:00:01.04833 | idle | 2023-09-05 17:10:50.195393+05:30 | COMMIT
crm | 3130 | 00:00:01.905754 | idle | 2023-09-05 17:10:49.337971+05:30 | COMMIT
crm | 3151 | 00:00:02.317058 | idle | 2023-09-05 17:10:48.926668+05:30 | COMMIT
crm | 3590 | 00:00:01.133051 | idle | 2023-09-05 17:10:50.110673+05:30 | COMMIT
crm | 4119 | 00:00:02.479124 | idle | 2023-09-05 17:10:48.764598+05:30 | COMMIT
crm | 3856 | 00:00:01.183103 | idle | 2023-09-05 17:10:50.060625+05:30 | COMMIT
crm | 3181 | 00:00:01.176917 | idle | 2023-09-05 17:10:50.066836+05:30 | COMMIT
crm | 3329 | 00:00:01.406762 | idle | 2023-09-05 17:10:49.836952+05:30 | COMMIT
crm | 3277 | 00:00:02.255391 | idle | 2023-09-05 17:10:48.988328+05:30 | COMMIT
crm | 3310 | 00:00:01.929529 | idle | 2023-09-05 17:10:49.314193+05:30 | COMMIT
crm | 3879 | 00:00:01.613869 | idle | 2023-09-05 17:10:49.629852+05:30 | COMMIT
crm | 3529 | 00:00:00.56273 | idle | 2023-09-05 17:10:50.680991+05:30 | COMMIT
crm | 3335 | 00:00:00.404638 | idle | 2023-09-05 17:10:50.839096+05:30 | COMMIT
crm | 3994 | 00:00:01.554959 | idle | 2023-09-05 17:10:49.688765+05:30 | COMMIT
crm | 3975 | 00:00:02.015667 | idle | 2023-09-05 17:10:49.228056+05:30 | COMMIT
crm | 3465 | 00:00:00.11256 | idle | 2023-09-05 17:10:51.131146+05:30 | COMMIT
crm | 3361 | 00:00:02.829491 | idle | 2023-09-05 17:10:48.414238+05:30 | COMMIT
crm | 3612 | 00:00:01.718839 | idle | 2023-09-05 17:10:49.52488+05:30 | COMMIT
rdsadmin | 12203 | 00:00:00.83821 | idle | 2023-09-05 17:10:50.405499+05:30 | COMMIT
crm | 3545 | 00:00:01.898858 | idle | 2023-09-05 17:10:49.34486+05:30 | COMMIT
crm | 3385 | 00:00:00.148037 | idle | 2023-09-05 17:10:51.095685+05:30 | COMMIT
crm | 3408 | 00:00:02.546519 | idle | 2023-09-05 17:10:48.697208+05:30 | COMMIT
crm | 3561 | 00:00:01.618848 | idle | 2023-09-05 17:10:49.624875+05:30 | COMMIT
crm | 3448 | 00:00:01.534644 | idle | 2023-09-05 17:10:49.709088+05:30 | COMMIT
crm | 3607 | 00:00:01.048382 | idle | 2023-09-05 17:10:50.19534+05:30 | COMMIT
crm | 8958 | 00:02:38.927255 | idle | 2023-09-05 17:08:12.316461+05:30 | DEALLOCATE pdo_stmt_00000020
crm | 3745 | 00:00:00.950385 | idle | 2023-09-05 17:10:50.293333+05:30 | COMMIT
crm | 4626 | 00:00:00.556476 | idle | 2023-09-05 17:10:50.687244+05:30 | COMMIT
crm | 4442 | 00:00:00.227095 | idle | 2023-09-05 17:10:51.016631+05:30 | COMMIT
crm | 4561 | 00:00:01.618849 | idle | 2023-09-05 17:10:49.624876+05:30 | COMMIT
crm | 3778 | 00:00:01.58853 | idle | 2023-09-05 17:10:49.655194+05:30 | COMMIT
crm | 3631 | 00:00:01.718853 | idle | 2023-09-05 17:10:49.524866+05:30 | COMMIT
crm | 4416 | 00:00:01.613908 | idle | 2023-09-05 17:10:49.629811+05:30 | COMMIT
crm | 3842 | 00:00:01.816497 | idle | 2023-09-05 17:10:49.427229+05:30 | COMMIT
crm | 4463 | 00:00:00.950422 | idle | 2023-09-05 17:10:50.29329+05:30 | COMMIT
crm | 3728 | 00:00:00.901827 | idle | 2023-09-05 17:10:50.341896+05:30 | COMMIT
crm | 4726 | 00:00:00.461477 | idle | 2023-09-05 17:10:50.782248+05:30 | COMMIT
crm | 3485 | 00:00:02.263646 | idle | 2023-09-05 17:10:48.980077+05:30 | COMMIT
crm | 4692 | 00:00:01.703795 | idle | 2023-09-05 17:10:49.539927+05:30 | COMMIT
crm | 4107 | 00:00:00.03446 | idle | 2023-09-05 17:10:51.20927+05:30 | COMMIT
crm | 4144 | 00:00:00.324452 | idle | 2023-09-05 17:10:50.919271+05:30 | COMMIT
crm | 13026 | 00:00:01.087389 | idle | 2023-09-05 17:10:50.156319+05:30 | DEALLOCATE pdo_stmt_000003de
crm | 3914 | 00:00:00.482727 | idle | 2023-09-05 17:10:50.76099+05:30 | COMMIT
crm | 4029 | 00:00:01.844414 | idle | 2023-09-05 17:10:49.399309+05:30 | COMMIT
crm | 3832 | 00:00:02.434732 | idle | 2023-09-05 17:10:48.808986+05:30 | COMMIT
crm | 4170 | 00:00:00.804648 | idle | 2023-09-05 17:10:50.439089+05:30 | COMMIT
crm | 4357 | 00:00:00.126632 | idle | 2023-09-05 17:10:51.117099+05:30 | COMMIT
crm | 13164 | 00:06:05.399506 | idle | 2023-09-05 17:04:45.844205+05:30 | DEALLOCATE pdo_stmt_0000002d
crm | 3690 | 00:00:02.136155 | idle | 2023-09-05 17:10:49.107567+05:30 | COMMIT
crm | 4662 | 00:00:02.664714 | idle | 2023-09-05 17:10:48.579007+05:30 | COMMIT
crm | 4198 | 00:00:01.37436 | idle | 2023-09-05 17:10:49.869363+05:30 | COMMIT
crm | 4214 | 00:00:02.848585 | idle | 2023-09-05 17:10:48.395135+05:30 | COMMIT
crm | 17450 | 00:00:02.456532 | idle | 2023-09-05 17:10:48.787178+05:30 | DEALLOCATE pdo_stmt_00000051
crm | 17621 | 00:00:00.024769 | idle | 2023-09-05 17:10:51.218944+05:30 | DEALLOCATE pdo_stmt_0000001f
crm | 4338 | 00:00:00.674629 | idle | 2023-09-05 17:10:50.569091+05:30 | COMMIT
crm | 4486 | 00:00:00.829738 | idle | 2023-09-05 17:10:50.413984+05:30 | COMMIT
crm | 4055 | 00:00:00.91556 | idle | 2023-09-05 17:10:50.328161+05:30 | COMMIT
crm | 4248 | 00:00:02.57618 | idle | 2023-09-05 17:10:48.667543+05:30 | COMMIT
crm | 17633 | 00:00:00.394684 | idle | 2023-09-05 17:10:50.849019+05:30 | DEALLOCATE pdo_stmt_00000003
crm | 17406 | -00:00:00.000665 | idle | 2023-09-05 17:10:51.244377+05:30 | DEALLOCATE pdo_stmt_0000007f
crm | 4718 | 00:00:01.518468 | idle | 2023-09-05 17:10:49.725253+05:30 | COMMIT
crm | 4228 | 00:00:00.136993 | idle | 2023-09-05 17:10:51.106734+05:30 | COMMIT
crm | 29267 | 00:06:33.328056 | idle | 2023-09-05 17:04:17.915693+05:30 | DEALLOCATE pdo_stmt_00000004
crm | 4258 | 00:00:01.61393 | idle | 2023-09-05 17:10:49.629792+05:30 | COMMIT
crm | 17654 | 00:00:00.02866 | idle | 2023-09-05 17:10:51.215049+05:30 | DEALLOCATE pdo_stmt_00000008
crm | 4535 | 00:00:00.562666 | idle | 2023-09-05 17:10:50.681052+05:30 | COMMIT
crm | 14679 | 00:00:53.661515 | idle | 2023-09-05 17:09:57.582186+05:30 | DEALLOCATE pdo_stmt_00000020
crm | 14781 | 00:00:04.872948 | idle | 2023-09-05 17:10:46.370758+05:30 | DEALLOCATE pdo_stmt_00000ece
crm | 4507 | 00:00:02.002412 | idle | 2023-09-05 17:10:49.241312+05:30 | COMMIT
crm | 17109 | 00:00:03.352965 | idle | 2023-09-05 17:10:47.890744+05:30 | DEALLOCATE pdo_stmt_000000c8
crm | 4579 | 00:00:01.460914 | idle | 2023-09-05 17:10:49.782802+05:30 | COMMIT
crm | 9687 | 00:00:12.232708 | idle | 2023-09-05 17:10:39.011001+05:30 | DEALLOCATE pdo_stmt_0000009f
crm | 17123 | 00:00:00.010001 | idle | 2023-09-05 17:10:51.233708+05:30 | DEALLOCATE pdo_stmt_00000154
crm | 4706 | 00:00:01.460954 | idle | 2023-09-05 17:10:49.78277+05:30 | COMMIT
crm | 15452 | 00:00:00.022851 | idle | 2023-09-05 17:10:51.220865+05:30 | DEALLOCATE pdo_stmt_00000410
crm | 11001 | 00:00:06.785846 | idle | 2023-09-05 17:10:44.45787+05:30 | DEALLOCATE pdo_stmt_000005dc
crm | 17591 | 00:00:01.193285 | idle | 2023-09-05 17:10:50.050416+05:30 | DEALLOCATE pdo_stmt_00000006
crm | 4647 | 00:00:00.333599 | idle | 2023-09-05 17:10:50.910123+05:30 | COMMIT
crm | 17533 | 00:00:00.028481 | idle | 2023-09-05 17:10:51.215224+05:30 | DEALLOCATE pdo_stmt_000000b6
crm | 17625 | -00:00:00.001597 | idle | 2023-09-05 17:10:51.245298+05:30 | DEALLOCATE pdo_stmt_0000002e
crm | 17615 | 00:00:00.865785 | idle | 2023-09-05 17:10:50.377917+05:30 | DEALLOCATE pdo_stmt_0000000a
crm | 7779 | 00:00:00.002148 | idle | 2023-09-05 17:10:51.241564+05:30 | DEALLOCATE pdo_stmt_00000783
crm | 5975 | 00:02:49.574064 | idle | 2023-09-05 17:08:01.669643+05:30 | DEALLOCATE pdo_stmt_000003d2
crm | 17601 | 00:00:00.795085 | idle | 2023-09-05 17:10:50.448632+05:30 | DEALLOCATE pdo_stmt_00000010
crm | 16879 | 00:00:06.245147 | idle | 2023-09-05 17:10:44.998561+05:30 | DEALLOCATE pdo_stmt_00000228
crm | 17559 | -00:00:00.000948 | idle | 2023-09-05 17:10:51.244658+05:30 | DEALLOCATE pdo_stmt_000000a8
crm | 16627 | 00:00:17.022466 | idle | 2023-09-05 17:10:34.221242+05:30 | DEALLOCATE pdo_stmt_00000008
crm | 14985 | 00:00:47.773563 | idle | 2023-09-05 17:10:03.470145+05:30 | DEALLOCATE pdo_stmt_00000020
crm | 17641 | 00:00:00.274188 | idle in transaction | 2023-09-05 17:10:50.969524+05:30 | DEALLOCATE pdo_stmt_00000013
crm | 11866 | 00:00:00.018373 | idle | 2023-09-05 17:10:51.225335+05:30 | DEALLOCATE pdo_stmt_00002716
crm | 17628 | 00:00:00.616821 | idle | 2023-09-05 17:10:50.626884+05:30 | DEALLOCATE pdo_stmt_00000003
crm | 17468 | 00:00:38.361239 | idle | 2023-09-05 17:10:12.882477+05:30 | DEALLOCATE pdo_stmt_00000012
crm | 14053 | 00:00:01.61204 | idle | 2023-09-05 17:10:49.631665+05:30 | DEALLOCATE pdo_stmt_00000331
crm | 30435 | 00:06:09.285878 | idle | 2023-09-05 17:04:41.957829+05:30 | DEALLOCATE pdo_stmt_0000047f
crm | 17642 | 00:00:00.077259 | idle | 2023-09-05 17:10:51.166449+05:30 | DEALLOCATE pdo_stmt_00000014
crm | 17655 | 00:00:00.013945 | idle | 2023-09-05 17:10:51.229779+05:30 | DEALLOCATE pdo_stmt_00000008
crm | 17658 | -00:00:00.001627 | idle | 2023-09-05 17:10:51.245335+05:30 | DEALLOCATE pdo_stmt_0000000d
crm | 12575 | 00:00:00.060098 | idle | 2023-09-05 17:10:51.183613+05:30 | DEALLOCATE pdo_stmt_00001cc2
crm | 16849 | 00:00:00.028432 | idle | 2023-09-05 17:10:51.215296+05:30 | COMMIT
crm | 2353 | 00:00:01.446553 | idle | 2023-09-05 17:10:49.797171+05:30 | COMMIT
(142 rows)
ALL IDLE Connetions End Time (script End Time After session killed) : now
----------------------------------
2023-09-05 17:10:51.301721+05:30
(1 row)
[ATLASSIAN CLOUD PLUGIN] [WARN] Not a WorkflowRun, onCompleted() won't be propagated to listeners
Finished: SUCCESSAnd, that’s how you invoke/execute your Runbooks, which perform Database operations in a remote Postgres Cluster in an automated manner using the power and flexibility of Slack, and the comfort and visibility of Jenkins all with just the simplicity of a Slack Slash command.
The level of abstraction and complexity streamlined into this golden path is beautiful and elegant.
Do let me know if you can think of any unusual, crazy playbooks/automated recipes to leverage ART into your slack workspaces to tackle some of your interesting/day-to-day DevOps activities.
