Splunk Your Health

Export, Input Analyse and Visualise your Apple Health Data with Splunk

Introduction

The amount of data an individual or company generates everyday is growing exponentially, and with it, the complexity of analysing this data. This article aims to demonstrate one of many use-cases of Splunk as a data analysis tool. I will explain the process of exporting health data from from Apple’s Health App and, then analysing and visualising this on Splunk

What is Splunk?

Splunk is a software platform used to search, analyse and visualise the machine-generated data gathered from websites, applications, sensors and devices.

What is the Apple Health App?

Apple Health App

Health App is the iOS health informatics mobile app used to track the user’s health and physical metrics such as steps taken, body weight, heart weight and mood. The app works by collecting data from smart wearbles, phones and other third-party data sources authorised by the user. For the purposes of this article I will be using my iPhone which collects data from itself and my Apple Watch.

Although the Health App already displays some visualisations about the data collected; it is very generic and may not answer specific questions such as:

1. How many steps in total did I take?
2. What is my most active period during the week?
3. What is my overall trend?
4. What was my most active day?
5. When was my first step with my iPhone? and possibly where?

These are all interesting questions which I hope to answer using my data and Splunk so let’s get started.

Overview of process

1. Export Apple Health Data
2. Convert data to CSV
3. Input data into Splunk
4. Analyse data
5. Visualise data

Exporting Health Data

To export the Health data open the Health App on your phone

1. Select profile on the top right corner
2. Scroll down to Export All Health Data and click “Export” when prompted
3. Wait for exporting to complete
4. Transfer data to your laptop. Email, air-drop , add to iCloud drive etc
5. You should see an attachement called export.zip.
6. Unzip this and locate the export.xml file
7. Export complete. We will now convert this file to CSV

Converting .XML file to .CSV

To convert this file we will use a Python script called applehealthdata.py which can be found on Github and was created by Test-Driven Data Analysis.

1. Copy the code for applehealthdata.py
2. Paste onto any code editor and save in the same location as the .XML file and name it applehealthdata.py
3. In the terminal cd to the directory of the above files
4. Run the following code:

applehealthdata.py export.xml

and you should see the following output

Reading data from export.xml ... done

The script has now completed the conversion and you should see many .CSV files in the original directory. You can have a look at any of these but we will be focusing on StepCount.csv

Select the StepCount.csv to analyse

Input Data Into Splunk

We can now input our converted data into splunk for analysis. I will not be going over the installation of Splunk on this article but you can find that here.

New Index:Select Settings > indexes > New index

• Index name (apple)
• Save

New App:Select Apps > Manage Apps > Create App

• Name (Apple Health)
• Folder Name (apple_health)
• Save

Input data

Splunk works on time based and needs a timestamp of the data when it is indexed. The “endDate” is one of the field headers of StepCount.csv. We need to setup a new sourcetype “csv:apple:health” with the “endDate” as TIMESTAMP_FIELDS.

Select Settings > Add Data (icon) > Upload > Select File

Next > Change timestamp > Advanced >

• Timestamp field (endDate)
• Save as (csv:apple:health)
• Next
• Host field value (iPhone or Applewatch etc)
• Index (apple)
• Review
• Submit

The process is now complete. We can now use Splunk’s Search Processing Language (SPL) and query some searches

Analysing The Data

First Step taken with iPhone

index="apple" sourcetype="csv:apple:health" source="stepcount.csv" host="Alqanit’s Apple Watch" 
| stats earliest(_time) AS _time

SPL used for Splunk Search.

My first step with the iPhone was 13:23 on 10th August 2017. Historic moment

Total number of steps

index="apple" sourcetype="csv:apple:health" source="stepcount.csv" host="Alqanit’s Apple Watch" 
| stats sum(value) AS TotalSteps

Total number of steps taken with iPhone… 🚶‍♂️

Highest steps in one day

Most active hour/days in the week

Change visualisation to Punchard (Click on find more visualizations)

Most active days are Monday-Friday, 7am-6pm

Conclusion

There are many ways to improve this project below are some of the improvements that can be made

1. Have a clear goal for why you want to analyse your data. For example, find correlation between you weight gain/loss and your daily step count
2. Automate the exporting and converting of data by using methods such as “HTTP Get input”, iPhone shortcut automation or iPhone triggers to send the data after a certain time or any other metric
3. Using the iOS App: “Health Auto Export” to automatically collect Apple Health data and send to Splunk
4. Using a combination of third-party apps such as “Myfitnesspal” and “Strava” to get a wider picture of your physical and health habits
5. Use Splunk to find your location hot map. Find the areas where you take the most steps
6. Find accompanying photos of specific days such a picture from the day you took your first step with your iPhone
7. Incoorprate medical health data from your medical provider to see correlation between illness and daily activity
8. Find the times where your heart rate was abnormally high and find pictures of these moments to figure out what caused such spikes

When it comes to analysing your data; the sky is the limit. You can answer specific questions about your daily habits and make improvements in your life and ,along with this, you can track your goals and keep yourself accountable.

Thank you for reading and now Go Splunk Yourself!