Box Developer Blog
Published in

Box Developer Blog

Sharing 100k folders with the Box Platform

Photo by Tima Miroshnichenko

The Box CLI allows developers to manipulate the content objects in bulk. In this article we're exploring how to share folders in an automated way.

Inspiration for this article came from a developer query in our forum, “Is there an automated way to create shared links to folders?”

The immediate answer is “yes”, but let’s take a look at the details of this use case.

UPDATE: Box CLI v3.6.0 see below…

Use Case

For a sense of scale there are 100,000 folders containing 2.5 million images, so doing this manually is impractical.

The objective is to create a table with information about each set of images (folder), and include the shared link.

This table will power a searchable app, and allow users to open and view the contents of each folder.

We’ll be using the Box CLI, and all the scripts are available in this Github repo.

Let’s get started.

Step 0: Create some sample folders

For this demo to work we need to create some sample folders, and there is a script for that…

This creates 110 folders on a topic and sub-topic hierarchy. A drop in the ocean compared to the use case, but it will give us something to work with.

Step 1: List all the folders

We could try to search for folder using the API, but this implies the folders have some naming convention we can search on, or some other clever search possibility, like metadata.

For example our structure is /100k/Topic_X/Topic_X_SubTopic_Y, so we could do something like:

box search topic --fields type,id,name,parent --type folder --content-types name --all --csv

With the following result:

type,id,name,parent.type,parent.id,...
folder,178236571387,Topic_6,folder,178236212260,0,0,100k,,,,,
folder,178235748420,Topic_5,folder,178236212260,0,0,100k,,,,,
folder,178234289941,Topic_3,folder,178236212260,0,0,100k,,,,,
folder,178234049722,Topic_0_SubTopic_4,folder,178237404679,0,0,Topic_0,,,,,
folder,178236564501,Topic_0_SubTopic_3,folder,178237404679,0,0,Topic_0,,,,,
folder,178236588187,Topic_0_SubTopic_7,folder,178237404679,0,0,Topic_0,,,,,

Note: if you are following along on your terminal and just created the folders, search uses an index and it may take some time for new objects to be indexed.

Since we don’t know if that is the case, we’ll go the hard core way and create a script that lists all folders, recursively.

It will be slow though.

However if we include the recursion level we can do some filtering.

In our assumption of the tree structure there is no point in checking the last level of folders, since these will only have files inside. In that case we can stop the recursion at the last level.

Most of our Box CLI sample scripts are done in powershell, this time we’ll use some good old bash. It runs in linux, mac-os and windows subsystem for linux (wsl).

So this line box folders:items 123 --csv-fields type,id,name,description lists the content of folder_id=123 , and outputs a csv formatted list.

Then we filter only the folders (if [[ “$type_name” == “folder” ]]), and include in the output the $parent_id and $level in the output.

Finally we check the recursion level if [ $level -lt 1 ], and stop it if it is the last level (in our case the last level is 1).

Step 2: Filter the list

From the complete list of folders, we are now filtering to get just the last level of folders, these are the ones for which we want to create the shared links.

This is the result:

parent_id,level,itemType,itemID,name
178237404679,1,folder,178235112928,Topic_0_SubTopic_0
178237404679,1,folder,178236444531,Topic_0_SubTopic_1
178237404679,1,folder,178234402718,Topic_0_SubTopic_2
178237404679,1,folder,178236564501,Topic_0_SubTopic_3
178237404679,1,folder,178234049722,Topic_0_SubTopic_4

Step 3: Creating the shared links in bulk

Now that we have the list with the folders for which we want to create the shared links we can just pipe it to the Box CLI.

Resulting in:

url,effective_access,effective_permission
https://app.box.com/s/srmp646j6zxgzrqbomj2z3496t5qqn6u,open,can_prev
https://app.box.com/s/5pndfkdk2n58v5hsy1b3h67aw13obmx6,open,can_prev
https://app.box.com/s/6crq7lmwswt1q47ws14ocdutd8r9m7sj,open,can_prev
https://app.box.com/s/v1ozyflwycsct007swqp1fzxigl0fo2i,open,can_prev
https://app.box.com/s/fz37gkcfjyoxy175pjli46db9ibzr95h,open,can_prev

So now we have 2 files we need to join.

The command paste -d, is necessary because the shared link creation does not output the folder_id, so we are joining the list of folders with the result of the shared link creation. This will only work if there were no errors in the creation before. If there are errors we will get a mismatched list between folder id and url.

A safer option is to take the shared link creation output and get the details of the folder from the url:

box shared-links:get --bulk-file-path tree_100k_shared_links_tmp.csv --csv --fields type,id,name,shared_link,parent

Resulting in:

type,id,name,shared_link.url,shared_link.effective_access,shared_link.effective_permission,shared_link.is_passwoe
folder,178235112928,Topic_0_SubTopic_0,https://app.box.com/s/srmp646j6zxgzrqbomj2z3496t5qqn6u,open,can_preview,,0
folder,178236444531,Topic_0_SubTopic_1,https://app.box.com/s/5pndfkdk2n58v5hsy1b3h67aw13obmx6,open,can_preview,,0
folder,178234402718,Topic_0_SubTopic_2,https://app.box.com/s/6crq7lmwswt1q47ws14ocdutd8r9m7sj,open,can_preview,,0

Or get the details of the shared link from the original list of folders:

box folders:get --bulk-file-path tree_100k.csv --csv --fields type,id,name,shared_link,parent

Resulting in:

type,id,name,parent.type,parent.id,parent.sequence_id,parent.etag,parent.name,shared_link.url,shared_link.effectt
folder,178237404679,Topic_0,folder,178236212260,0,0,100k,,,,,,,,,,
folder,178235112928,Topic_0_SubTopic_0,folder,178237404679,0,0,Topic_0,https://app.box.com/s/srmp646j6zxgzrqbomj,
folder,178236444531,Topic_0_SubTopic_1,folder,178237404679,0,0,Topic_0,https://app.box.com/s/5pndfkdk2n58v5hsy1b,

Another approach

A completely different approach would be to create the shared links one by one from the folder list, and get all the information we need with one single call to the API:

Although apparently slower, it has the advantage of creating the final output file without the need of running secondary commands, and properly handling errors.

parent_id,level,itemType,itemID,name,url,effective_access,effective_permission
178237404679,1,folder,178235112928,Topic_0_SubTopic_0,https://app.box.com/s/srmp646j6zxgzrqbomj2z3496t5qqn6u,open,can_preview
178237404679,1,folder,178236444531,Topic_0_SubTopic_1,https://app.box.com/s/5pndfkdk2n58v5hsy1b3h67aw13obmx6,open,can_preview
178237404679,1,folder,178234402718,Topic_0_SubTopic_2,https://app.box.com/s/6crq7lmwswt1q47ws14ocdutd8r9m7sj,open,can_preview

Updated Approach

With the release of Box CLI v3.6.0, the engineering team included the id and the type of the object in the output of the box shared-links:create command.

This means you no longer need to merge files or make secondary calls to know the relation between the shared link URL and the file or folder object it belongs to.

The output now looks something like:

box shared-links:create -y --bulk-file-path $csv_file_bulk --access open --no-can-download --csv --save-to-file-path $csv_file_out

type,id,etag,shared_link.url,shared_link.effective_access,shared_link.effective_permission,shared_link.is_password_enabled,shared_link.download_count,shared_link.preview_count,shared_link.access,shared_link.permissions.can_preview,shared_link.permissions.can_download,shared_link.permissions.can_edit
folder,178235112928,0,https://app.box.com/s/1j7n1mx6zr5rl9wv7zam7mnfivkkdtgt,open,can_preview,,0,0,open,1,,
folder,178236444531,0,https://app.box.com/s/xgfx4vhq19ele0tbm7qfjt2iotvcbbvb,open,can_preview,,0,0,open,1,,
folder,178234402718,0,https://app.box.com/s/cefiib29qt03ypqrf7cbwclogxuxdy4m,open,can_preview,,0,0,open,1,,
folder,178236564501,0,https://app.box.com/s/wcb1qhp2z1abm3n9fvw7eh5b8ed4segv,open,can_preview,,0,0,open,1,,
folder,178234049722,0,https://app.box.com/s/u83avb70c7qyjxv4jd3kk6iyddez0dkw,open,can_preview,,0,0,open,1,,

The scripts on GitHub have been updated.

So big kudos to the team, Artur Jankowski, Lukasz Socha and Minh Cong for, in record time, turning feedback from the community into reality.

Clean up

Removing the shared links is just a matter of piping the output of the folder list to the delete command.

box shared-links:delete --bulk-file-path tree_100k_shared_link_tmp.csv

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store