CSV Data Importer Neo4j-Bigbang

What is Neo4j-Bigbang?

Neo4j-Bigbang is a python package to allow you to import CSV data into Neo4j database in the easiest way. This provides two commands bigbang-node and bigbang-rel and these will make all the magics.

Neo4j-Bigbang on PyPI

Pains you may have when dealing with CSV

Have you tried importing CSV data into Neo4j Database? If yes, you would probably ended up to this Neo4j Official Import CSV tutorial. You should realize that this requires you

• to take a lot of steps
• to learn Neo4j special terms
• to write Cypher Scripts
• to prepare RDB like well formatted CSV data

In this case, you need to manually manipulate how data should be treated. You may have to make decisions on each data with great care.

What Neo4j-Bigbang does for you

• Import Nodes, labels and properties with automated type detection.
• Create Relationships and Nodes at once with single CSV file.
• Avoid inserting the same node twice.
• Avoid creating the same relationship twice.
• Update labels and properties on existing Nodes and Relationships.
• You can force inserting the same node multiple times if you want to.

You don’t have to define schemas in advance, but just set properties in your CSV. You don’t need to write Cypher script at all.

All you need to do is, prepare your CSV file and run bigbang!

Setup Neo4j-Bigbang

Before explaining anything, let’s try it on and see how it’s like.

Prepare environment

$ mkdir your-dir
$ cd your-dir
$ python -m venv venv
$ source venv/bin/activate
$ export NEO4J_URI=bolt://{hostname}:7687 NEO4J_USER={username} NEO4J_PASSWORD={password}

Install neo4j-bigbang

$ pip install neo4j-bigbang

Run command, create relationships

In this example, I chose the easiest way to show how to create your Graph, so this time I would use bigbang-rel command.

Prepare CSV data under importing directory

$ mkdir importing
$ vim importing/relationships.csv

Sample CSV relationships

type,target_fields_in,target_values_in,target_labels_in,target_fields_out,target_values_out,target_labels_out,position
APPEAR_IN,name,Mouse,Character|Person,name,The Matrix,Movie,ally
APPEAR_IN,name,Neo,Character|Person,name,The Matrix,Movie,protagonist
APPEAR_IN,name,Morpheus,Character|Person,name,The Matrix,Movie,ally
APPEAR_IN,name,Dozer,Character|Person,name,The Matrix,Movie,ally
APPEAR_IN,name,Trinity,Character|Person,name,The Matrix,Movie,ally
APPEAR_IN,name,Oracle,Character|ComputerProgram,name,The Matrix,Movie,prophet
APPEAR_IN,name,Architect,Character|ComputerProgram,name,The Matrix,Movie,enemy
APPEAR_IN,name,Deus Ex Machina,Character|HardwareInterface,name,The Matrix,Movie,enemy
APPEAR_IN,name,Agent Smith,Character|ComputerProgram,name,The Matrix,Movie,enemy
APPEAR_IN,name,Machine City,City,name,The Matrix,Movie,place
APPEAR_IN,name,The Matrix,VirtualRealityWorld,name,The Matrix,Movie,virtual world
IS_CREW_OF,name,Mouse,Character|Person,name,Nebuchadnezzar,Vehicle|Battleship,
IS_CREW_OF,name,Neo,Character|Person,name,Nebuchadnezzar,Vehicle|Battleship,
IS_CREW_OF,name,Morpheus,Character|Person,name,Nebuchadnezzar,Vehicle|Battleship,captain
IS_CREW_OF,name,Dozer,Character|Person,name,Nebuchadnezzar,Vehicle|Battleship,pilot
IS_CREW_OF,name,Trinity,Character|Person,name,Nebuchadnezzar,Vehicle|Battleship,
HELP,name,Mouse,Character|Person,name,Neo,Character|Person,
HELP,name,Morpheus,Character|Person,name,Neo,Character|Person,believes Neo is the One
HELP,name,Trinity,Character|Person,name,Neo,Character|Person,girl friend
HELP,name,Oracle,Character|ComputerProgram,name,Neo,Character|Person,adviser
GAVE_CHOISE,name,Architect,Character|ComputerProgram,name,Neo,Character|Person,
CHALLENGED,name,Agent Smith,Character|ComputerProgram,name,Neo,Character|Person,
ASKED_HELP,name,Neo,Character|Person,name,Deus Ex Machina,Character|HardwareInterface,
KILLED,name,Agent Smith,Character|ComputerProgram,name,Oracle,Character|ComputerProgram,
WORK_TOGETHER_WITH,name,Morpheus,Character|Person,name,Trinity,Character|Person,
DESIGNED,name,Architect,Character|ComputerProgram,name,The Matrix,VirtualRealityWorld,
WAS_BORN_IN,name,Neo,Character|Person,name,The Matrix,VirtualRealityWorld,
IS_INTERFACE_OF,name,Deus Ex Machina,Character|HardwareInterface,name,Machine City,City,
TRY_TO_DESTROY,name,Agent Smith,Character|ComputerProgram,name,Machine City,City,
TRY_TO_DESTROY,name,Agent Smith,Character|ComputerProgram,name,The Matrix,VirtualRealityWorld,
DELETED,name,Deus Ex Machina,Character|HardwareInterface,name,Agent Smith,Character|ComputerProgram,

This CSV file is specified for bigbang-rel command.

Run to Generate Graph

$ bigbang-rel -c

That’s it! This will create a graph for you. Now you know how easy Neo4j-Bigbang is.

Relationship CSV Requirement

When you create relationships.csv, It has to have at least the following fields required by the bigbang-relcommand.

• type: Type of the relationship
• target_fields_in: Target property on the first Node
• target_values_in: Target property’s value on the first Node
• target_fields_out: Target property on the second Node
• target_values_out: Target property’s value on the second Node

Optionally you can also define

• target_labels_in
• target_labels_out

Everything else you define will become properties on the relationship.

Example

If you want to make a relationships between…

Node: Person {
   name: 'Mike'
}

and

Node: Person {
    name: 'Anna'
}

Then, you can set your CSV values like

target_fields_in => name
target_values_in => Mike
target_labels_in => Person
target_fields_out => name
target_values_out => Anna
target_labels_out => Person

Run command, import Nodes

You can also import only nodes without relationships by using bigbang-node . This will concentrate on manipulating node. You can roughly create relationships and nodes with bigbang-rel command, and later you can set detailed properties on those nodes, or vice versa.

Prepare CSV data under importing directory

$ mkdir importing
$ vim importing/nodes.csv

Sample CSV nodes

name,description,labels
Nebuchadnezzar,Nebuchadnezzar is a fictional hovership captained by Morpheus in the The Matrix franchise.,Vehicle|Battleship
Oracle,"She is a prophet in the Matrix. She looks like a person, but is a computer program created for some perposes.",Character|ComputerProgram
Agent Smith,Agent Smith is a fictional character and the main antagonist of The Matrix trilogy.,Character|ComputerProgram
Switch,Switch was a member of the crew of the Nebuchadnezzar.,Character|Person

This CSV file is specified for bigbang-node command.

Run to Generate Nodes

$ bigbang-node

This will create only one Node, if you noticed. This is because, other nodes are already there in the graph. But if you want to update those nodes with the new properties, in this case description is added, then you should put -u option there.

$ bigbang-node -u

Then all the nodes will be updated.

Neo4j-Bigbang has a lot more options and flexibility.

• You can create relationships between existing nodes or creating at once.
• You can update properties of existing relationships
• You can choose node picking property and labels when you create relationship
• You can choose importing CSV filename.
• If you want to set all the same TYPE in relationship CSV, you could set your CSV file name as relationship TYPE
• If you want to set all the same Label in node CSV, you could set your CSV file name as node Labels
• You can choose creating new nodes or updating ones or skipping.

Hope this will help you to begin your Neo4j journey without any hassles.