A Security Log Generator for SOCs

A shameless plug of my Security Log Generator Tool

Security Log Generator — https://github.com/cruikshank25/Security-Log-Generator

From a high level, how it works

The Problem

Problem: “I need logs to verify my alerts/capacity/performance/<insert_use_case_here>.”

Problem: “My logs are only in production or I don’t have the tools to generate the logs I need.”

I’m sure many of you that work in cyber security or data analytics space have had this problem before. You need some logs of a specific type, whether its IDS logs, Windows Event logs, Linux logs, Firewall logs etc. But the problem is, these logs are either in production or the solution has not been built yet. But, you, unfortunately, have been tasked with developing alerts or ensuring capacity and performance or writing field extractions for the logs etc.

Solution: I’ll generate my own logs of the same format as the tool I need.

The Security Log Generator aims to resolve these issues, by giving you an easy and convienient way to generate logs of various types in formats that you might frequently find in a SOC (Security Operations Centre). The Security Log Generator is made to be a generalised solution that could be used for many purposes such as:

• Capacity testing a logging solution
• Performance testing a logging solution
• Testing your alerting capabilities
• Testing the on-boarding of logs

Just to name a few.

The Solution

As of its initial release (11th January 2023) it supports IDS (Intrusion Detection System) and Web Access style logs, however, the tool has been designed in an extensible way to quickly allow adding of new formats. Currently in development are event generators for the following type of logs:

• Endpoint Anti-Virus Logs
• Windows Security Event Logs
• Linux Event Logs
• Perimeter Device Logs (firewalls, vpns, proxies etc.)
• User Defined Custom Logs

Here’s a couple of examples of the synthetic data that the log generator produces:

ids.log:

2023-01-20 11:17:51,652 - ids_logger_1 - low_severity - HTTP - 10.24.67.168:26663 --> 127.204.71.181:80 - ACK - Denial of service (DoS)
2023-01-20 11:17:51,759 - ids_logger_1 - low_severity - UDP - 152.64.62.199:64803 --> 253.179.35.209:13725 - SYN - Phishing
2023-01-20 11:17:51,869 - ids_logger_1 - low_severity - TCP - 226.245.99.42:6083 --> 136.148.117.169:55445 - SYN - Phishing
2023-01-20 11:17:51,978 - ids_logger_1 - low_severity - TCP - 232.29.138.136:18105 --> 48.123.189.215:18327 - ACK - Denial of service (DoS)
2023-01-20 11:17:52,087 - ids_logger_1 - low_severity - TCP - 24.170.49.85:3043 --> 231.1.204.207:13262 - SYN - Worm Propagation Attempt
2023-01-20 11:17:52,195 - ids_logger_1 - low_severity - TCP - 66.206.63.196:17338 --> 111.19.221.137:57282 - ACK - Malicious traffic
2023-01-20 11:17:52,305 - ids_logger_1 - low_severity - TCP - 166.25.11.47:1922 --> 104.228.176.6:29319 - RST - SQL Injection
2023-01-20 11:17:52,414 - ids_logger_1 - low_severity - UDP - 149.133.130.220:46881 --> 2.45.9.148:20796 - ACK - Port scanning
2023-01-20 11:17:52,522 - ids_logger_1 - low_severity - TCP - 189.240.175.88:8622 --> 254.7.200.44:43849 - SYN - Denial of service (DoS)
2023-01-20 11:17:52,632 - ids_logger_1 - critical_severity - HTTPS - 220.208.137.231:34142 --> 17.182.213.158:443 - RST - Port scanning
2023-01-20 11:17:52,741 - ids_logger_1 - medium_severity - UDP - 158.49.136.210:19038 --> 224.87.200.218:47727 - RST - Port scanning
2023-01-20 11:17:52,848 - ids_logger_1 - low_severity - SMTP - 95.160.237.87:21970 --> 253.66.114.189:25 - SYN - Denial of service (DoS)
2023-01-20 11:17:52,958 - ids_logger_1 - low_severity - HTTPS - 62.124.57.198:34735 --> 255.21.109.208:443 - ACK - PING NMAP
2023-01-20 11:17:53,068 - ids_logger_1 - low_severity - SMTP - 253.232.56.102:15687 --> 64.252.244.233:25 - ACK - Malicious traffic
2023-01-20 11:17:53,177 - ids_logger_1 - high_severity - UDP - 49.158.10.176:59336 --> 7.179.61.75:60876 - SYN - SQL Injection
2023-01-20 11:17:53,285 - ids_logger_1 - medium_severity - TCP - 219.245.223.99:10584 --> 29.179.211.39:39872 - SYN - Denial of service (DoS)
2023-01-20 11:17:53,393 - ids_logger_1 - low_severity - TCP - 131.91.1.121:5616 --> 25.90.188.19:29387 - PSH - Port scanning
2023-01-20 11:17:53,502 - ids_logger_1 - high_severity - TCP - 90.87.72.116:38608 --> 56.117.103.103:7705 - FIN - Denial of service (DoS)
2023-01-20 11:17:53,610 - ids_logger_1 - low_severity - HTTPS - 82.22.164.181:54425 --> 142.38.162.244:443 - RST - Malware
2023-01-20 11:17:53,720 - ids_logger_1 - low_severity - HTTP - 86.49.190.115:17509 --> 16.142.111.140:80 - SYN - Malware
2023-01-20 11:17:53,829 - ids_logger_1 - low_severity - TCP - 23.93.197.72:57641 --> 105.47.26.38:19226 - SYN - Denial of service (DoS)
2023-01-20 11:17:53,938 - ids_logger_1 - low_severity - HTTP - 59.108.199.139:28881 --> 242.22.196.156:80 - SYN - Port scanning
2023-01-20 11:17:54,046 - ids_logger_1 - low_severity - DHCP - 22.135.177.210:20222 --> 77.115.90.177:67 - SYN - Worm Propagation Attempt
2023-01-20 11:17:54,154 - ids_logger_1 - low_severity - ICMP - 185.20.77.228:17502 --> 61.122.211.225:1 - ACK - Port scanning
2023-01-20 11:17:54,264 - ids_logger_1 - low_severity - HTTPS - 252.193.255.57:3833 --> 114.33.87.46:443 - RST - Port scanning
2023-01-20 11:17:54,372 - ids_logger_1 - low_severity - HTTP - 105.219.2.220:43579 --> 42.182.88.21:80 - SYN - Denial of service (DoS)
2023-01-20 11:17:54,480 - ids_logger_1 - high_severity - ICMP - 8.193.175.155:18063 --> 114.87.210.42:1 - ACK - Denial of service (DoS)
2023-01-20 11:17:54,589 - ids_logger_1 - low_severity - DNS - 2.210.87.136:64993 --> 102.184.110.199:53 - SYN - SQL Injection

access.log:

[2023-01-20 11:19:13,352] - access_logger_1 - 181.225.54.204 - robert "GET posts HTTP/1.1 202 76038 http://morales.composts.htm?userparam=Kristi Rogers&id2=2192&datetime=2021-08-21 02:50:55" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
[2023-01-20 11:19:13,510] - access_logger_1 - 147.183.202.179 - corey "GET tags HTTPS/1.1 400 61211 http://cherry-michael.comtags.html" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36"
[2023-01-20 11:19:13,678] - access_logger_1 - 239.105.167.91 - mary "GET app/tags HTTP/1.1 500 6884 http://jones.comapp/tags.jsp" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36"
[2023-01-20 11:19:13,851] - access_logger_1 - 111.250.204.197 - jessica "DELETE list/tags/wp-content HTTPS/1.1 409 4287 http://harris.comlist/tags/wp-content.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15"
[2023-01-20 11:19:14,015] - access_logger_1 - 225.236.17.178 - william "GET explore/search/wp-content HTTPS/1.1 206 2117 http://short-ortega.orgexplore/search/wp-content.html" ""Googlebot/2.1 (+http://www.googlebot.com/bot.html)""
[2023-01-20 11:19:14,191] - access_logger_1 - 25.205.223.42 - lisa "GET tags/app HTTP/1.1 200 93644 http://davis.comtags/app.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36"
[2023-01-20 11:19:14,357] - access_logger_1 - 143.112.48.47 - charles "PUT main/app HTTPS/2.0 500 16626 http://bailey-rodriguez.commain/app.htm" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36"
[2023-01-20 11:19:14,535] - access_logger_1 - 193.24.70.120 - patrick "GET main/explore HTTPS/2.0 403 65427 http://white.commain/explore.html" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36"
[2023-01-20 11:19:14,703] - access_logger_1 - 163.225.105.171 - brittney "GET categories HTTPS/1.1 200 58857 http://fischer.comcategories.php?userparam=Anita Henderson&id2=4070&datetime=2020-08-10 11:32:27" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36"
[2023-01-20 11:19:14,877] - access_logger_1 - 29.6.240.249 - larry "POST tag HTTP/1.1 202 17988 http://stanley-english.orgtag.html?userparam=Gregory Robinson&id2=9855&datetime=2022-11-11 19:00:08" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
[2023-01-20 11:19:15,050] - access_logger_1 - 18.26.231.230 - nathaniel "POST explore HTTP/1.1 407 32346 http://gonzalez-wilkins.comexplore.php" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
[2023-01-20 11:19:15,222] - access_logger_1 - 104.8.252.245 - sarah "DELETE main HTTPS/2.0 200 8721 http://wright.commain.htm" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15"
[2023-01-20 11:19:15,397] - access_logger_1 - 248.111.184.191 - jeffrey "GET main/posts HTTP/1.1 205 53210 http://bullock.commain/posts.asp" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
[2023-01-20 11:19:15,573] - access_logger_1 - 130.161.189.172 - matthew "PUT blog HTTP/1.1 206 36942 http://hudson.infoblog.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15"
[2023-01-20 11:19:15,749] - access_logger_1 - 123.217.97.199 - aaron "GET tag/posts HTTPS/1.1 407 17618 http://martin.biztag/posts.jsp" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36"
[2023-01-20 11:19:15,930] - access_logger_1 - 58.43.174.52 - jill "POST categories HTTP/1.1 415 82818 http://hogan.bizcategories.htm" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15"
[2023-01-20 11:19:16,093] - access_logger_1 - 228.64.252.173 - jake "POST blog/categories/tag HTTP/1.1 200 58552 http://brown.comblog/categories/tag.htm?userparam=Tony Underwood&id2=174192330&datetime=2020-09-25 07:51:09" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15"
[2023-01-20 11:19:16,274] - access_logger_1 - 227.200.120.30 - martin "GET blog/explore HTTP/1.1 400 16597 http://mitchell.comblog/explore.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15"
[2023-01-20 11:19:16,444] - access_logger_1 - 107.140.154.219 - juan "GET search/tags/category HTTP/1.1 500 97959 http://deleon.comsearch/tags/category.html?userparam=Vickie Howard&id2=76177&datetime=2020-03-17 19:46:31" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36"
[2023-01-20 11:19:16,613] - access_logger_1 - 177.154.6.13 - morgan "GET blog/wp-content/search HTTP/1.1 415 48388 http://sullivan-davis.comblog/wp-content/search.htm" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15"
[2023-01-20 11:19:16,791] - access_logger_1 - 12.237.69.39 - philip "GET tags/posts/posts HTTP/1.1 203 16603 http://brown.nettags/posts/posts.php" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
[2023-01-20 11:19:16,961] - access_logger_1 - 93.136.212.168 - richard "DELETE main/wp-content HTTPS/1.1 204 50096 http://frazier.commain/wp-content.htm" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15"
[2023-01-20 11:19:17,142] - access_logger_1 - 192.161.203.104 - vanessa "GET tags/category/list HTTP/1.1 200 11766 http://sanders-ashley.comtags/category/list.html?userparam=Eric Thompson&id2=936395&datetime=2020-09-15 01:18:31" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
[2023-01-20 11:19:17,308] - access_logger_1 - 102.196.23.105 - robert "GET wp-content HTTP/1.1 202 51370 http://miller.comwp-content.htm" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36"
[2023-01-20 11:19:17,483] - access_logger_1 - 171.69.83.250 - cynthia "POST wp-content/main HTTP/1.1 406 54609 http://bullock.comwp-content/main.htm" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
[2023-01-20 11:19:17,641] - access_logger_1 - 8.5.30.34 - tara "CONNECT app/blog/blog HTTPS/1.1 301 47540 http://graham-smith.comapp/blog/blog.php?userparam=Joshua Scott&id2=652340656&datetime=2021-12-29 09:06:26" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36"
[2023-01-20 11:19:17,810] - access_logger_1 - 245.207.136.247 - kevin "PUT categories/search/tag HTTP/1.1 414 35777 http://rogers.bizcategories/search/tag.php" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
[2023-01-20 11:19:17,973] - access_logger_1 - 224.129.111.12 - howard "GET tags/list HTTP/1.1 401 74985 http://fritz.orgtags/list.html" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36"

Note: All of the above data is purely synthetic and generated by the log generator tool and/or the underlying Faker Python Library, the tools only dependency (https://pypi.org/project/Faker/).

The above events are generated with a combination of randomness, biased randomness, or through the Faker library (for realistic domain names and usernames). Some of this can be influenced through the configuration to get it to produce the data that is required for your use case (such as the number of events and the time it takes to write them).

The most recent feature is the capability to influence the generation of data over time using a sine wave. This is to add ‘bumpiness’ to the data which might better represent real-life logging that has ‘peaks’ and ‘troughs’ over time (think 9 am when everyone is logging on for work vs. 3 am when everyone is asleep).

By modifying the amplitude, frequency, sample rate, duration, and stretch configuration values, we can generate events over time with varied distributions like the below examples (showing the count of events over time):

A high amplitude and sample rate, low duration and frequency

a high frequency and duration, a lower amplitude and sample rate

Summary

The initial problem that sparked this project was not having several log formats in the development environment I was working in — however, the log formats existed in production and I needed to develop alerts for those log formats. This meant I needed to create synthetic data to prove my alerts could work in the development environment before the push to production — no one wants to be this guy:

I wouldn’t trust Austin Powers with my production environment

Building this tool has allowed me to solve part of that problem and I was able to generate synthetic logs that represented what we had in production and develop valid alert logic.

Let me know in the comments how this tool could be useful to you or what functionality you would like to see in future releases.

Thanks for Reading.