How to Upload/Download Files to/from Notebook in my Local machine

Problem:

I saved my Pandas or Spark dataframe to a file in a notebook. Where did it go? How do I read the file I just saved?

Pandas and most other libraries have APIs to read or write from or to local file system.

Where is the local file system for a notebook?

Let me explain you with the help of Data Science Experience architecture.

Every notebook is tightly coupled with a Spark service on Bluemix. You can also couple it with Amazon EMR. But A notebook must have a platform to run its notebook instance.

Since a notebook runs on an associated service, it uses that service’s local disk; in Apache spark service’s case, that is GPFS. If you run !pwd in a Python notebook, you will see the GPFS tenant directory for your spark service.

/gpfs/fs01/user/sc1f-6086fed227dde8-2c631c8ff999/notebook/work

Now you know that sc1f-6086fed227dde8-2c631c8ff999 is the tenant ID for your spark service.

Let’s say you have:

import base64  
import pandas as pd
df = pd.DataFrame(data = [[1,2],[3,4]], columns=['Col 1', 'Col 2'])

Save the file to local GPFS:

#write dataframe to file
df.to_csv("test1.csv")
#read it back
pd.read_csv("test1.csv").head()

Now, you might want to download the above dataframe to your local machine.

Solutions

Option 1

Download the file through the notebook — but only if the file is in CSV format.

The following function was provided by Polong Lin:

from IPython.display import HTML

def create_download_link( df, title = "Download CSV file", filename = "data.csv"):
csv = df.to_csv()
b64 = base64.b64encode(csv.encode())
payload = b64.decode()
html = '<a download="{filename}" href="data:text/csv;base64,{payload}" target="_blank">{title}</a>'
html = html.format(payload=payload,title=title,filename=filename)
return HTML(html)

create_download_link(df)

Running this function will give you a link to download the file into the notebook output cell.

Option 2

You can download or upload any type of file. Since the files are on Spark service’s local disk,
the Spark service exposes an API to allow you to download or upload files.

Use the REST API described in Using Spark Interactive API to upload or download the file from your Spark service.

For example, this REST tenant API only exposes one directory under your tenant directory https://spark.bluemix.net/tenant/data/\:

!ls -l /gpfs/fs01/user/sc1f-6086fed227dde8-2c631c8ff999/data/total 3552  
-rw-r----- 1 sc1f-6086fed227dde8-2c631c8ff999 users 1207391 May 17 19:36 96ae523867d444f6284ceed939588a57.CSV
drwx------ 2 sc1f-6086fed227dde8-2c631c8ff999 users 4096 May 3 15:55 b5219fcc83e93b4eae8d36990ead96682bea61af
drwx------ 2 sc1f-6086fed227dde8-2c631c8ff999 users 4096 May 3 15:11 libs
drwx------ 3 sc1f-6086fed227dde8-2c631c8ff999 users 4096 May 3 15:11 spark131batch
drwx------ 3 sc1f-6086fed227dde8-2c631c8ff999 users 4096 May 3 15:11 spark141batch
drwx------ 2 sc1f-6086fed227dde8-2c631c8ff999 users 4096 May 3 15:11 workdir

You can either move the file we saved earlier to a data directory

/gpfs/fs01/user/sc1f-6086fed227dde8-2c631c8ff999/data/

by running

!cp test1.csv /gpfs/fs01/user/sc1f-6086fed227dde8-2c631c8ff999/data/

or you can save the file directly to that directory:

df.to_csv("/gpfs/fs01/user/sc1f-6086fed227dde8-2c631c8ff999/data/test1.csv")

For downloading, you can use the curl command at your local machine's command prompt or terminal.

For Download

curl -O -X GET -u tenant_id:tenant_secret -H 'X-Spark-service-instance-id:instance_id' https://spark.bluemix.net/tenant/data/<filename.ext>

Values for the variables can be found in Bluemix.net, just open your Spark service and look for service credentials.

Here is the command that I ran on my local machine:-

charless-MacBook-Pro-3:~ charles$ curl -O -X GET -u sc1f-6086fed227dde8-2c631c8ff999:22b2bc31-f19e-4fc8-8b60-47eebd950786 -H 'X-Spark-service-instance-id:17e8347c-571d-4128-9c1f-6086fed227dd' https://spark.bluemix.net/tenant/data/test1.csv% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload Upload Total Spent Left Speed
100 25 0 25 0 0 97 0 --:--:-- --:--:-- --:--:-- 99
charless-MacBook-Pro-3:~ charles$ cat test1.csv
,Col 1,Col 2
0,1,2
1,3,4

Now you can upload the files to the Spark service

For example:

curl -X PUT -k -u tenant_id:tenant_secret -H 'X-Spark-service-instance-id:instance_id' --data-binary "@nameofthefiletouploa_withextension"  https://spark.bluemix.net/tenant/data/nameofthefiletouploa_withextension
charless-MacBook-Pro-3:~ charles$ cat > mylocaldataset.txt
this is my file from local machine.
^Z
[1]+ Stopped cat > mylocaldataset.txt

charless-MacBook-Pro-3:~ charles$ cat mylocaldataset.txt
this is my file from local machine.
charless-MacBook-Pro-3:~ charles$ curl -X PUT -k -u sc1f-6086fed227dde8-2c631c8ff999:22b2bc31-f19e-4fc8-8b60-47eebd950786 -H 'X-Spark-service-instance-id:17e8347c-571d-4128-9c1f-6086fed227dd' --data-binary "@mylocaldataset.txt" https://spark.bluemix.net/tenant/data/mylocaldataset.txt {"url":"https://spark.bluemix.net/tenant/data/mylocaldataset.txt"}charless-MacBook-Pro-3:~ charles

Note the @ sign it is important.

If the URL is returned the file was successfully uploaded.

Originally published at datascience.ibm.com on August 14, 2017.

Written by

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store