Using data visualization techniques and Gephi tool to visually analyze Game of Thrones characters and their interactions

Introduction

The Idea

It is a well-known fact that Game of Thrones, a series of novels written by George R.R. Martin and its television counterpart, have deeply enticed several fans all over the world. With a plethora of characters and plots, the story is highly intricate and incredibly alluring. As a huge fan myself, having participated in many discussions with friends predicting what’s going to happen next, I thought it was an exciting opportunity for me to perform a Game of Thrones character network analysis using Gephi to figure out who are the main protagonists that emerge and how their interactions have been with others throughout the series of books.

The Inspiration

I began my research by searching for articles on network data visualizations, and one of the best sources I could find was Martin Grandjean’s blog. Martin Grandjean is a Researcher in History and Digital Humanities at the University of Lausanne and his blog consists of numerous articles on data analysis and visualization. In one of his posts, ‘Complex network visualization for the history of interdisciplinarity: Mapping research funding in Switzerland’, Martin analyzed various interdisciplinary Swiss National Science Foundation projects and their funding in Swiss Francs. The result was spectacular and really helped shape my own visualization. I found Martin’s use of a dark background to create excellent visual contrast for a boosted readability of the network highly interesting. The legend added to the visualization also made everything easy to understand. Furthermore, the fact that only significant nodes were labeled created a clutter-free visualization making it more usable.

While performing my research on Game of Thrones network data analysis, I came across Network of Thrones — A Song of Math and Westeros, a blog by Andrew Beveridge (Associate Professor at Macalester College). His blog post has the most detailed network analysis of Game of Thrones characters, covering a season-wise analysis of the television series as well as a novel-wise analysis of each of George R. R. Martin’s books. This was a very interesting way to highlight the differences between the novels and the television series. The use of different colors for nodes and edges was also a great way to signify clusters.

Yet another amazing article I came across was Connected World: Untangling the Air Traffic Network. This was again written by Martin Grandjean where he maps out the connections between all airports around the world by creating a spectacular network visualization. What I found most interesting was the fact that his visualization was very similar to a geographically accurate representation of the dataset. This indicated how clustering in network graphs can be very similar to geographical locations pertaining to the nodes, and I quote Martin Grandjean — “A network, in its very essence, is already a map”.

Materials

Tools -

Gephi (Windows version 0.9.2) — An open-source software for network visualization and analysis. I used this software to create and analyze my network data visualization.

Microsoft Excel — A spreadsheet software by Microsoft, part of the MS Office suite. I used it for viewing the dataset CSV files and understanding them before plugging them into Gephi.

Dataset -

The dataset I used was taken from Andrew Beveridge’s GitHub — A Song of Ice and Fire Page

The dataset was created by connecting two characters whenever their names (or nicknames) appeared within 15 words of one another in the series of “A Song of Ice and Fire” novels (volumes 1 through 5). The edge weights correspond to the number of interactions between these characters. The details of the dataset are -

• Nodes — 796 (Characters in “A Song of Ice and Fire”)
• Edges — 2823 (Undirected interactions between these characters)

Process

Feeding the dataset into Gephi and running some statistics

The first step in my process was understanding the two CSV files consisting of the node table and edges table, which I downloaded from Andrew Beveridge’s GitHub page. Once the dataset made sense to me, I proceeded with importing the nodes and edges tables into Gephi. By default, the tool generated a visualization, but it did not seem to make sense as all the nodes were in one cluttered mess. Therefore, in order to improve the usability, I needed to perform some layout improvements as well as aesthetic improvements.

Improving the layout of the visualization

After this, in order to improve the visualization, I ran the Force Atlas 2 algorithm multiple times, testing out various permutations and combinations of its parameters. I found that checking the ‘Dissuade Hubs’ setting allowed the node clusters to spread out thereby making the whole network appear cleaner. I also tuned the settings to have a scaling factor of 100.0 and ‘prevent overlap’ setting checked, in order to ensure that everything was nicely spread out and there was enough room to accommodate labels and there were no overlaps of the nodes. Furthermore, I also set the ‘gravity’ to 100.0 to bring nodes that were very far apart, closer to the spacialization space. Once the Force Atlas 2 algorithm made the layout a lot better, there was still a huge scope for improvement as the visualization was still quite difficult to understand. To remedy this, I performed the Expansion algorithm several times by a scale factor of 2x each, to reach the right point where the nodes were nicely spread out without making the whole visualization too large. Finally, I also ran the Label Adjust algorithm, to make sure there was enough room for the labels of each of the nodes, without any overlap.

Playing around with colors and weights

Next, I started thinking about how the use of colors and weights could improve the visualization. I applied a color partition on the nodes by ‘modularity class’ and selected an appropriate palette for them. This gave the same color to the nodes that were more densely connected together as compared to the others. I also set the node size ranking on the ‘degree’ to make those nodes larger that had more connections (or a higher degree). I set the labels to grow in proportion to the node size as well so that more prominent nodes were easier to read. As I was inspired by Connected World: Untangling the Air Traffic Network by Martin Grandjean, I made use of a black background for my visualization to create a good contrast. This way, the nodes, and the edges were clearer and were more visually appealing. I gave the labels a standard color of white, which would be easy to read above the node and edge colors as well as the black background. Finally, I also set the color of the edges to be a mix of the colors of each node pairs. This way it was easy to tell just by looking at an edge, whether the node was connected to a node belonging to the same cluster (Based on modularity class) or a different one. The edge weights were also set to vary based on the strength of the connection, which was retrieved from the dataset, which corresponds to the number of interactions between the characters in the book.

Results & Analysis

The final visualization rendered on Gephi after the above-mentioned series of steps and tweaks is shown below. You may also download the full resolution visualization to zoom and inspect the intricacies (Note — Please allow it to render as it is a large file).

Some important statistics for my visualization were -

• Average Degree: 7.093
• Average Weighted Degree: 81.982
• Graph Density: 0.009
• Network Diameter: 9
• Modularity: 0.566

By looking at the visualization, I was amazed to find that Tyrion has been the most prominent protagonist throughout the series, closely followed by Jon Snow. Although Jon Snow was quite predictable, I always had a strong feeling that Tyrion would be one of the victors (or should I say survivors) at the end of the story. After seeing these results, I am quite certain that Tyrion is going to do great things in the upcoming season.

Although I had also felt that Daenerys, the Khaleesi herself, might end up on the Iron Throne, her slightly less significant presence in the visualization seems to contradict my prediction. However, it is also true that Khaleesi had been on a different side of the map altogether for a quite significant period of time, reducing her interactions with the other characters deeply involved in the Kings Landing and Winterfell. Her absence from Westeros, therefore, could be one of the reasons behind the slightly smaller node representation of Daenerys.

Another interesting conclusion that can be drawn from the visualization is the fact that the location of the nodes and how they cluster together has a strong connection with where the characters were geographically placed in the land of Westeros and its surrounding regions. For instance, Jon Snow, Samwell Tarly and others are all clustered up, representing the brothers of the Night’s Watch, whereas, Khaleesi, Drogo, and Jorah Mormont are all clustered representing the Essos area, where their story runs in parallel. Also, the Lannisters are clustered up representing their presence in the Kings Landing region and most of the Starks, representing Winterfell.

Finally, it is also amazing to see how Arya Stark is also quite significant in the visualization and her cluster is comprised of lesser-known characters from Braavos, but she still has a strong weighted connection with Sansa Stark, with whom, as per the story, she maintains touch.

Reflection & Future Direction

Gephi is an open source software and I really appreciate its ability to render network visualizations so quickly and effectively, free of cost! However, I still believe that there’s a huge scope for improvement for the overall interface, which is extremely dated and not easy to use. Furthermore, a lack of the well known undo feature was a huge inconvenience. For instance, if I performed an action that was undesirable, I needed to reload the last save point and redo all the steps. This slightly suppressed my desire to explore the tool in depth, as experimentation with various actions was a cumbersome effort.

For this project, I covered a combined dataset of volumes 1 through 5 of the series “A Song of Ice and Fire”. On his GitHub page, Andrew Beveridge had also uploaded datasets for each of the volumes individually. I believe It would be a great future project for me to perform a book-wise analysis of the character network and highlight the changes as the story progresses.

I would also like to find a dataset that covers the television series — Game of Thrones so that I can map the network of characters on a per season basis and analyze it. Furthermore, it would also be very interesting to compare and contrast the novels versus the television series and find out how the two differ.

Finally, a whole new dataset awaits! Game of Thrones season 8 is due to be released in 2019 and I can’t wait to grab the dataset once the entire season is aired. It would be fun to analyze the final season, hopefully not too many characters would die due to the white walkers and I would still have a significant number of nodes to work with!

Using Timeline JS to take a look at noteworthy infographic inventions from the 19th century that we heavily use today

Introduction

Infographics, a term created by clipping and compounding the words ‘Information’ and ‘Graphics’, are visual or graphic representations of information which allow clear and swift communication. They leverage human beings’ visual system’s ability to identify patterns and trends.

Creating infographics is not a new technique in any way, as the earliest storytellers were the cavemen that lived on the earth forty thousand years ago. These cave dwellers took up their brushes in the form of cave paintings, which told stories about various events that they thought were noteworthy. This was hence, an act of using graphics to present information so as to communicate with others, which is what we define as ‘Infographics’.

There have been many iconic infographic inventions in the past that were so robust, that they are still widely used today with minimum, or in some cases, no modifications at all. It is very intriguing to learn about the inventors’ creative genius that led to these incredible information visualizations. I, therefore, decided to create a timeline view of some of these spectacular inventions belonging to the 19th century, which I found to be the most relatable due to my background in Engineering and Technology, having used many of these myself in the recent past. My goal is to showcase these fantastic time-tested inventions in a chronological order, using a popular information visualization tool.

Materials

Tools -

• TimelineJS — An open source tool developed by Northwestern University’s Knight Lab that allows users to create interactive timelines by simply filling up a spreadsheet.
• Google Sheets — A web-based application that allows users to create, update and modify spreadsheets and share the data online. This tool also allows collaboration and is a part of Google’s G-Suite of web applications.
• OneNote — An application that allows users to create notebooks in which they can store, organize, and search resources and notes. It is a part of Microsoft’s Office 365 suite.

Dataset –

• RJ Andrews, “History of Infographics”

Methodology

I began my study by analyzing RJ Andrew’s “History of Infographics”, which was shared by my information visualization professor. It is a wonderful data source to find out about iconic inventions in the world of infographics in an interactive fashion. I started reading blurbs about each of the inventions on the website, while simultaneously making a list of the ones that really piqued my interest. Some of these gave me a huge nostalgia trip as well, as I had used them several years back when I was pursuing my Bachelor’s in Engineering — for instance, Mendeleev’s Periodic Table, which I used to use quite a lot to fetch elements and figure out their atomic number.

With a lot of Google searching, I was able to find good articles about the visualizations that I had selected to showcase, so as to fetch the clearest definitions I could find to put onto the timeline. This helped me learn more about these visualizations and their variations, for instance, the pie chart has a variant that helps visualize hierarchical data, by using concentric circles. This variant is known as the ‘multilevel pie chart’.

My next step was to choose appropriate sample images for these visualizations in order to effectively communicate their appearance and use. Yet again, I made use of the articles I had previously visited to find good sample images. Google image search also helped me discover good images that I could use.

As I was collecting all the aforementioned data points, I was simultaneously entering them into my OneNote app in an organized manner. After this was complete, I transferred the data over to the Timeline JS spreadsheet [Fig 1] and followed the steps on their website to generate the timeline. I made several changes thereafter to modify the look and feel by changing the images used to improve readability and effectiveness.

Results

Please access –

• TimelineJS Project — Noteworthy Infographic Inventions from the 19th Century
• Google Sheets Template

From this timeline [Fig 2], I hope to make viewers aware of how far back in time these amazing inventions were made in the field of Infographics. I want them to see how robust these were and how little or nothing has changed about them since their invention. I hope my use of imagery helps them understand the visualizations at first glance and learn how the 19th century contributed to the world of infographics.

Reflection & Future Directions

I always enjoy the feeling of learning a new tool or technique and TimlineJs was no exception. I felt that it is truly a spectacular tool which someone having no technical background can also use. However, my technology background really made me gauge the shortcomings of this tool based on my own experience of HTML and JavaScript. I constantly felt the lack of user interface customizations, which makes the bottom section of the timelines look almost the same for many users. This way anyone who is familiar with the tool can tell at first glance that TimelineJS was used. This brings down the uniqueness factor significantly.

Yet another pain point of mine was that although most of my events were coincidentally evenly spaced to about a decade in between each, one event was almost two decades apart, making it appear as an awkward long jump on my timeline. I believe that a way to show an axis break (or kink) between larger gaps and shortening them a bit could be a good idea, making it look more evenly spaced.

Finally, I invested a lot of time in selecting background images for my timeline. However, I later realized that the images, even though subtle, were huge sources of distraction and also made the text harder to read. I, therefore, decided to remove the background images and use a subtle background color instead. I will definitely avoid this in the future.

References

• RJ Andrews, History of Infographics
  https://infowetrust.com/scroll/
• University of Leicester
  https://www2.le.ac.uk/offices/ld/resources/numerical-data/pie-charts
• Study.com
  https://study.com/academy/lesson/what-is-a-stacked-bar-chart.html
• Wyzant.com
  https://www.wyzant.com/resources/lessons/math/statistics_and_probability/averages/cumulative_frequency_percentiles_and_quartiles
• Berkeley
  https://evolution.berkeley.edu/evolibrary/article/0_0_0/evotrees_primer_02
• Chegg
  https://www.chegg.com/homework-help/definitions/normal-curve-31
• Lucid Chart
  https://www.lucidchart.com/pages/organizational-charts
  https://www.lucidchart.com/pages/venn-diagram
• Live Science
  https://www.livescience.com/25300-periodic-table.html
• Population Education
  https://populationeducation.org/what-population-pyramid/
• Wikipedia — Poverty Map
  https://en.wikipedia.org/wiki/Poverty_map

Remember the television show of the 80s called Knight Rider? If yes, I’m sure you are picturing the uber-cool black colored Pontiac Firebird with the blinking red lights. The show was a hit, mostly thanks to the futuristic feature-packed beast of a car that assisted its undercover-cop owner in solving crimes. The car could self-drive, follow voice commands, make decisions and so much more. In today’s times, big players like Tesla Motors, Mercedes-Benz and Audi are working strongly to make cars smarter by the day, and we can see the Knight Rider car become a reality. Equipped with various sensors and complex electronic systems, cars can assist drivers with near self-driving capability. The responsibility of making decisions for safety and navigation is shifting from the driver to the artificially intelligent systems. Easy integration of software such as Android Auto and Apple CarPlay with the in-car systems allows adding the ‘Smart’ factor to almost any car. With this rapidly changing landscape where consumers are being exposed to smarter vehicles, several challenges arise, many of which can be tackled by User Experience techniques.

UX Challenges in Smart Cars

Driver Distraction

Infotainment systems in cars are extremely useful for viewing and controlling various settings and features as well as managing the entertainment systems. However, driver interaction with these systems can be a source of distraction. Wired connectivity of the infotainment unit with personal devices can also cause physical obstruction in some cases, not to mention the painful process of managing cables and finding difficult to access connection ports. These issues can not only be frustrating for the driver but can also cause obstruction or distractions that could lead to accidents (Reuters, Car Dashboards that act like smartphones raise safety issues, July 2015).

Trust & Confidence

With advanced systems on board, cars these days can predict collision scenarios and detect pedestrians among various other intelligent actions. When the car makes several of the decisions for you, it is quite possible that a lack of trust or confidence could creep in (Mashable, Self-Driving Cars Still Need to Earn the Public’s Trust, Oct 2017). Questions like “What if the system doesn’t work?” and “What if there is a malfunction?” are quite common when users first experience smart cars.

Perceived Safety

Drivers or passengers can develop high levels of perceived safety due to the powerful and promising capabilities of smart cars. While it is a good idea to make people feel safe, it can also lead to negative results, such as drivers relying completely on the onboard systems and not paying sufficient attention during dangerous situations (Auto-ui.org, User Experience Design for Vehicles, 2012)

Driver Emotion

The driver’s emotional state is an important issue for automotive safety. Emotions affect perception and action, and hence, they are extremely relevant. Several driving behaviors are negatively affected by emotions, linking stress, anger or aggression to accidents. Frustrations arising due to poor usability of in-car interfaces and controls or difficulty navigating can also contribute heavily to a negative emotional state (Auto-ui.org, User Experience Design for Vehicles, 2012).

How UX Design is Making Smart Cars Better

User (Driver) Research

There are many contexts possible for a driver, be it a slippery drive on the rainy roads of Portland or a foggy encounter on the crests and troughs of San Francisco. If these contexts are not kept in mind while designing for smart cars, there is a high chance for the design to fail, and failure in a driving scenario could have terrible consequences. Research is hence essential, where UX Practitioners observe and interview people driving in real-world scenarios, while unveiling their pain points and understanding their behavior (Key Lime Interactive, User Research Methods for Automotive User Experience, Oct 2015).

Google, for example, conducts its research by simulating real-world scenarios and combining it with special eye-tracking sensors to understand how drivers interact with Android Auto (Android Central, This is how Google Tests the Safety of Android Auto, 2016).

Augmented Reality (AR)

Many companies these days are working on technologies and techniques to augment reality for the drivers to prevent distractions and instill a sense of confidence. This is often achieved by creating overlaying interfaces that either appear directly on the windscreen of the vehicle or appear on top of a real-time view of the environment around the car displayed on the infotainment screens. The interface elements are intentionally made translucent, allowing good visibility of the environment with minimal obstruction. UX designers make use of typography, design principles and user research to create and refine the design of the overlaying interfaces for clarity and understandability (CNET — Roadshow, Augmented Reality in the Car Steps Towards Production at CES 2017, Jan 2017).

Visualizations and graphical representations are heavily used to allow quicker recognition through peripheral vision, eliminating the need to fixate on the interface rather than on the road. These techniques also allow giving users timely feedback for any decisions made by the car’s artificial intelligence and displaying information about pre-determined decisions beforehand. This allows building trust with the user, as it eliminates the element of surprise when the car makes a decision.

An example of AR in vehicles is Harman’s LIVS concept showcased at CES 2017, which is capable of placing street signs over intersections, greatly helping drivers (Harman, HARMAN Unveils the Ultimate In-Car Experience for Intelligent, Intuitive Autonomous Driving at CES 2017, 2017).

Voice Commands & Gestures

Gestures and voice commands are powerful ways to interact with smart cars. They reduce hand-eye coordination efforts and the need for haptic feedback while using touchscreen interfaces or other controls of the car. With devices such as Google Home and Amazon Alexa in the market and assistants like Siri and Google Assistant on almost every smartphone out there, consumers are getting more comfortable with using voice commands as time passes. Gestures are like second nature to humans and it is a great idea to leverage this human trait for interaction with smart cars (Medium, Reimagining In-Car UX, Sep 2014).

For instance, BMW allows drivers to use gestures to accept in-car phone calls and control the music volume. Even several voice commands exist to perform vital tasks that drivers are highly likely to perform, such as navigation (Medium, 5 Car UX Design Trends, Aug 2015) (Carnation Software, BMW Owner’s Manual).

UX Designers incorporate these interaction methods to reduce cognitive load on the drivers and allowing a more natural and usable way to perform actions and increase driver confidence. Maybe one day we can finally say goodbye to blindly and clumsily hunting for knobs in some corner of the car.

Conclusion

Many significant advances in the merged automotive and digital fields have been made and will continue well into the foreseeable future. Consumers have also become more immersed in the digital landscape, and their needs and demands are advancing exponentially. With increasing competition and easier access to advanced technology, it is essential for manufacturers to ensure top-notch in-car UX to maintain their position in the market. We might still be a long way from a time when every vehicle would be self-driven with the most advanced and intelligent systems on board, but it is quite intriguing to imagine new ways humans would interact with these marvels and how UX professionals would carve the in-car experience.

How usable is IMDb’s Android app?

Introduction

IMDb or Internet Movie Database is a popular online database of information related to movies, TV shows, video games, awards, and events. The database comprises of information on production crew and cast, summaries, synopses, reviews and ratings, which millions of registered users consume daily. As per Google Play Store, there are over 100 million users of the app on the Android platform!

Android App by IMDb: Well-designed?

The IMDb Android app’s design language has evolved slowly and has recently adopted Material Design. While some design decisions made across the app are backed by strong logic and can be applauded, few flaws can also be uncovered in the design. Here are some of the key findings -

Failed to scroll on the home screen: Encountered a gulf of execution

Upon launching the app, a scrollable newsfeed-like interface is presented. With a goal — ‘find a good movie’ in mind, there is an immediate intent to explore the landing screen. The mind creates a plan to scroll for exploration. However, due to an obtrusive advert that blocks the lower ~10% of the screen (which happens to be the most usable area of the display), scrolling fails, as the swipe is registered on the advert’s overlay element, rather than on the app’s interface (Fig. A). This creates a gulf of execution as the intended action could not be executed.

Recommendation: Embedding the advert in the app’s main interface, rather than having it as a static overlay, will eliminate the obstruction and bridge the gulf of execution, thereby improving the design (Fig. B). An alternate solution would be to substantially reduce the height of the advert to decrease the probability of the user’s thumb landing on it.

Swiping between tabs is disabled: A logical constraint

With increasing importance being given to design, several Android apps have greatly improved their overall quality and conformance to the Material Design guidelines. The ‘easy to understand’ navigation of the IMDb app makes it an excellent example of this. Labels such as ‘Home’, ‘Movies’, ‘TV’ etc. are straightforward and map very well with users’ potential goals.

Navigation tabs are seen in several popular apps such as Messenger, Twitter etc. and users are well versed with the interaction patterns associated with them (For instance, switching between tabs can be achieved by swiping left or right in the tab container) However, when it comes to the IMDb app, tabs cannot be swiped across (Fig. C). But why though?

It is important to note that there are several horizontal-scroll lists in the app (Eg. ‘Most popular movies’ in Fig. C) that encourage exploration by swiping left or right to reveal more list items. Having the same gesture for swiping between tabs can cause a conflict and make the interface error prone (users might want to scroll through the list, but might end up switching tabs or vice-versa). The designers of the app have, therefore, intelligently made use of Donald Norman’s principle of constraints by disabling ‘swipe to switch tabs’, so that users do not fall prey to making these errors.

Unable to find articles about awards or events: Poor discoverability

As stated by Donald Norman — “Two of the most important characteristics of good design are discoverability and understanding”, it is very important to ensure that features or functions that the user desires are discoverable. The IMDb app, while mostly working well in this regard, does seem to fail in a particular scenario. With the goal of ‘reading articles about awards or events’ in mind, tapping on the ‘Awards & Events’ tab seems logical, but the landing screen only displays videos at the top with no indication of any other information. On revisit, however, articles suddenly appear below the video-container (Fig. D). So, what went wrong?

There was lack of feedback to indicate that the app is loading up list items below the video. There were also no signifiers present, to indicate that there is a list of articles present (maybe in the form of a heading label or a container). These factors lead to poor discoverability of the feature, resulting in failure to achieve the goal of finding articles about events or awards.

Recommendation: Improving discoverability can be achieved by fixing the root causes of the problem: i.e. improving feedback, and providing strong signifiers. A loading / in-progress spinner can be added for feedback, and a container along with a heading label (such as ‘Articles’) can be included below the video-container to indicate the upcoming content. (Fig. E)

Conclusion

In light of the above, it is clear that even though IMDb is the world’s most popular database related to movies, TV shows etc., and their Android app is quite well-designed, robust and effective, a few design flaws still plague the interface. Making improvements to these potential flaws is sure to greatly enhance the experience for users.

Using Tableau, a popular Data Visualization tool, to analyze defacement crime in NYC through information visualization techniques

Introduction

The Idea

When I first came to New York from India in 2017, I noticed several graffitied buildings throughout the city. I had never seen anything quite like that before, and I had several questions racing through my mind — “Who would do this?”, “Why would they want to deface someone’s property?”, “How does the police deal with this situation?”.

A year later, while working on searching for datasets, I came across ‘Encroachments and Defacements’ dataset hosted on ‘NYC OpenData’. I was immediately interested in visualizing various aspects of defacement crimes in New York City over time and hence selected this dataset.

The Inspiration

Since my dataset was that of crime reports, I began searching for crime data visualization examples. I found a great New York Times’ article (works only on Internet Explorer), which displayed visualizations of homicides in New York City between the years 2003 and 2011. A great feature for users was that the visualization displayed the crime data in the form of a timeline, where the year could be selected to see the crimes committed in that timeframe. Users could also see the data for all years at once, which was great. Another thing I really found interesting about this visualization was the fact that I could see the crime for each of New York City’s boroughs. They had used different shades of blue to represent the various boroughs, but one thing I didn’t like was the fact that the shades of blue were too similar to easily differentiate between them. I had to look very carefully to make out the difference.

Another example that drove my design decisions was an NYC.gov website that displayed various crime maps. It had many filters based on ‘Crime Type’ and ‘Date Range’ which was great for users that might want to isolate the data. The markers on the map varied in size based on the number of crimes in a neighborhood, and they helped effectively describe the count visually for users.

Finally, I also found a series of visualizations very interesting from Statista.com, which had made use of bar and line charts to depict the timeline of various kinds of crimes in the US. I thought that it was an effective approach for highlighting the rise or fall of crimes from year to year for users. This would also enable users to compare data side by side quickly and easily.

Materials

Tools –

Microsoft Excel — A spreadsheet software by Microsoft, part of the MS Office suite.

OpenRefine — A tool for working with messy data, which helps in cleaning it up quickly and effectively.

Tableau Public — A free to use data visualization software, which has many visualization techniques built into it.

Dataset –

The dataset I used for my lab is ‘Encroachments and Defacements’, which I found while browsing through datasets on NYC OpenData.

Process

Cleaning up the data

After I had selected the dataset, I started off with cleaning up the data to make it usable and ready for analysis. I used the OpenRefine tool to upload the CSV dataset (obtained from NYC OpenData) and made improvements to its raw form. Many of the dates were in different formats and hence I had to bring uniformity to it. Even the datatype needed to be transformed from ‘text’ to ‘date’. I also had to add a facet for getting the data only for ‘Defacement’ crimes, as the set also included data for ‘Abandonment’, ‘Encroachment’ and ‘ATM‘, which I didn’t include in the scope of this project. I also transformed many of the number cells’ datatype from ‘text’ to ‘number’.

Transferring the data into Tableau

After the data was clean and ready for analysis, I exported the data from OpenRefine as an Excel file for importing into Tableau. However, I encountered an error in the OpenRefine console and, therefore, had to export as a CSV instead of Excel. The workaround was to open the refined CSV in Microsoft Excel and then use its ‘export’ feature to convert it to Excel format. I was, hence, finally able to import the data into Tableau.

Analyzing the data + Creating Visualizations

After loading the data into Tableau, I first started off by trying to create a visualization for the number of reported defacement crimes per year, for each borough. I chose the line chart visualization since it is great for showcasing time series data and immediately, the difference in the number of reported crimes between boroughs was evident. However, I noticed that the line chart had values for the year 2018 as well (since the data for 2018 was present in the dataset). But since 2018 is the current year and still in progress, the data for this year wasn’t complete. It, hence, did not make sense to include this year, and I added an exclusion for it.

It was also important to understand how much time the police took to resolve these reported issues. To understand this for each borough and each year, I had to create a calculated field that would be the difference between the initial crime report date and the closing date columns. This difference would give me the resolution time as it would give the number of days between when the crime was reported and when the case was closed. After creating this calculated field, I added the average function on these values for each year per borough and put it against year on the x-axis to form a line chart. Yet again, I selected the line chart as it is very effective for visualizing and comparing time series data. As opposed to the previous visualization, I did not exclude the year 2018 in this case, as an average value will not vary significantly for the rest of the year.

Finally, I was most excited about the geo-data that was available in the dataset, and I proceeded with creating a bubble map visualization of the defacement crimes. I used the latitudes and longitude coordinates to plot the points on the map for each crime report. I color-coded the points using an accessible color palette on the map for each borough, to make it easy to understand the difference in their borders for all users. Another great aspect was that the size of the points was bigger for more crime reports in the same neighborhood over the years, which could help users understand the magnitude of crimes in a region. I added filters for the year so that users could isolate the data for each year as well.

Results

Please access my tableau project — Defacement Crime in NYC (Year 2014 to 2018)

From this visualization, I hope to make viewers aware of how defacement crime has changed over the years, and how efficiently the police is taking care of these problems. I also hope to make users understand the difference in the crime rate and resolution time upon moving from borough to borough.

Visualization 1: Number of Defacement Crimes Reported in NYC

It is interesting to see how the Bronx and Brooklyn have steadily increasing defacement crime rates since 2014. Best effort observed is in the Bronx, the average resolution time of which, dropped to less than half of what was observed in 2014.

The highest rate is observed in Queens in 2016, which is significantly higher than all other boroughs in that year, but it steeply drops below that of Brooklyn in 2017.

Manhattan also peaks above the Bronx in 2016 but is still on the lower side overall. Manhattan also sees a trend similar to Queens, with the peak in 2016 and a steep drop in 2017.

Brooklyn emerges as the top defacement target in 2017, closely followed by the Queens. Next in line in 2017 are Bronx, Manhattan and Staten Island. Staten Island has the lowest defacement rate and has maintained it quite steadily between 2013 and 2017.

Visualization 2: Average Resolution Time for Defacement Crimes in NYC

The police have been making a solid effort in lowering the resolution time for defacement crimes in New York City.

The Bronx and Brooklyn saw their best year in terms of resolution time in 2016, while Manhattan and Staten Island are currently (2018) in their best year.

Brooklyn has been seeing regular ups and downs in its average resolution time between 2013 and 2018, however, currently, it is extremely close to its best year in 2016.

In 2018, Brooklyn, Queens and the Bronx even out on their resolution times, with Manhattan better off by around 10 days, and Staten Island, emerging the best by around 10 days lower than Manhattan.

Visualizations 3: Defacement Crime Map of NYC

It is very interesting to see how the crimes are spread across the city and its boroughs. Certain neighborhoods that are worst affected over the years can easily be identified by looking at the size of the marker. For instance, West, 15th Street in Brooklyn is the worst affected overall with 3360 defacement reports over the years. 91 Avenue in Queens is next in the worst affected neighborhoods with 1350 crime reports, which is still significantly lower than Brooklyn.

Close to JFK Airport, streets 113 through 130 in Queens seem quite badly affected with several markers spread across.

Manhattan’s Upper East Side seems to be its worst affected part with a dense scatter of defacement crime markers.

The Bronx and Brooklyn are overall quite evenly affected throughout. Staten Island emerges as the cleanest borough with scarcely scattered markers.

Reflection & Future Directions

Learning Tableau has been a great journey so far, and I was amazed at the ability of the tool to create beautiful visualizations out of raw data. I also loved the fact that there was a color-blind friendly palette to make visualizations more accessible.

I also learned that data can be messy, and the cleanup process is not always a one-time activity. A need to further clean up the data can crop up any time during the analysis and visualization phases as well. For instance, I noticed bad data for the year 2013 during the visualization phase. While it was easy to exclude it from the visualization in the line charts, it could not be done in case of the map visualization. I had opted to show the year filter for users and Tableau did not allow me to hide the year ‘2013’ from the list, even though it let me exclude the data from the visualization. This was making it confusing, as there was no data for 2013 to be visualized, but the filter still showed the year 2013. I, therefore, had to clean up the excel sheet (remove bad data for 2013) and refresh the dataset into Tableau.

While searching for crime data on the NYC OpenData website, I found several datasets for various other crimes, and I feel it would be great to explore those for any future projects. I would also like to try and look for other sources that help me find data for the entire United States of America, and it would be extremely interesting to see how the other states are doing compared to New York.

References

• Few, “Effectively Communicating Numbers: Selecting the Best Means and Manner of Display”
• From Data to Viz — Line Charts
• From Data to Viz — Bubble Map