Google Earth Engine Takeout: Tools and Guide for Code and Asset Transfer
Over the span of last two weeks or so, my university advised me that they were moving drive access from umail which was the university hosted gmail into google apps. Now though this didn’t mean much to most people for a select few of us who use the connected account for Google Earth Engine, it meant losing access to the account credentials and essentially access to the account. This also meant any analysis that stayed in our drives needed to be transferred and most important the codes and assets needed a new home as well. It is with the same imperative when I started to write the blog post that I wanted to know how to create a tool similar to the google takeout function or to be put simply how could I replicate a Google Earth Engine account and transfer everything over to a new account. Even if you are not replicating your account, think of this as an add on which allows you to download your entire code repositories, create an asset report and best of all iteratively change permissions to all image-collection and images whether or not within a folder.
Note: This is something that I have tested and have designed only for a windows machine with python 2.7.14 but can be easily ported into an different operating system. Use these tool and steps at your own risk and backup your scripts always just in case.
If you still want to proceed which I assume you do in case you are still reading, I am including descriptions links to the tool I made and the steps I used to achieve the same. The tool is a single command line interface with three sections.
Before you do this make sure of a few things
- Both your google accounts have an external password, since it requires that to download and perform a lot of the operations. Also enable Let Less Secure App use your account on both these accounts.
- Your system has native python available in terminal or command prompt depending on what kind of system you are using. You can check this by typing
- Git is installed on your system. For windows you can find installation here
- Earth Engine Command Line(earthengine cli) interface is installed, instructions are in the developer page
- You have authenticated earthengine cli using
- Make sure you visit the git source for your account within earth engine and allow access.
- Check git is accessible via your system path type
git helpand check if the system can reach installed git command line tools.
Now we setup and install the tool by running the following set of steps. The Github repository containing this tool and codes can be found here
For windows download the zip here and after extraction go to the folder containing “setup.py” and open command prompt at that location and type
python setup.py install
pip install -r requirements.txt
Now call the tool for the first time, by typing in
geetakeout -h. This will only work if you have python in the system path which you can test for opening up terminal or command prompt and typing
python. If you want, you don’t have to install the tool to use it you can browse to the folder geetakeout inside the main zipped folder. You can then call the tool by simple
python geetakeout.py -h
The housekeeping and credential setup is optional since most of you have probably installed the earthengine cli and authenticated it using
Anatomy of the Process: How to transfer step by step
Getting first things out of the way is to understand the three sections of this tool. To make life and this process simpler I designed the tool to have a flow so you can run these tools one after the other. The EE Setup and Housekeeping sections are optional , since I will generally updated the selenium driver for mozilla and it assumes you have authenticated your earthengine CLI . The tool might show an error if you have not authenticated using
If you have installed the tool run
geetakeout -hIf you have migrated into the folder
python geetakeout.py -h
Setting up the case study
For this blog I decided to make the transfer simple I have a university account but since my university if shifting umail services to a google app service it means my domain would change from @umail.iu.edu to @iu.edu which are separate accounts. I created the iu.edu GEE account recently.
This also means that the root path for my home folder and repository are different. The idea is simple to be able to replicate the codes and assets from one account to the other. This includes every type if assets including collections but also making sure that the structure of the folders are same. That being said you will still have to change the home path in the codes but if the structure is same then only a single root-path changes.
So now that we have a setup, I am going to approach it step by step and have a walk through to explain the process better.
Step 1: Getting your Repository Lists(gee_repo)
This assumes that you have visited the Git Source for your codes in Google Earth Engine and authorized it. If not allow it here and then you are set to download your repo contents and perform git operations. The tool is setup for accessing all repositories that are shared with you. This downloads the list into an html file which can then be parsed for your repositories.
Step 2: Setting up your Git with Earth Engine Credentials(git_auth)
You can do this using two methods
- The first simple includes you visiting your gitsource account page that we accessed earlier and click
Generate Passwordand follow instructions.
I am going to talk more about the second method because this eliminates the need to get the password again and again since it is save as passkey. This will authenticate your git client with your git password using a browser less login and also store your gitkey
We will use this again to setup our second account post authorization. This will print our gitkey location and make sure you copy that so you can swap in out as needed. Note the name of the key is in the format
git-”username” in this case it is
Step 3: Authorize your Git Client with Git Key(git_swap)
The next step is to use the saved gitkey to authorize the git client. We are setting everything up so that we can clone the repositories to which we have access.
Step 4: Clone your repositories(git_clone)
This tool makes use of your earlier created GIT list, now that your git client has been authorized in step 2, you should be able to download your repos. This tool uses the account already linked to your terminal account. If you are not sure try
earthengine ls to see your username. The export path is noted for the collection of repositories.
Step 5: Working with Assets: Generating Asset Report(ee_report)
This includes all your assets , including tables, images, image collections and folders. We need to make sure we have this list to work on copying over your assets to the secondary account. Running this is simple and just requires a location for the csv file (the full path).
The output is a csv file consiting of the type of asset and the asset path to be replicated in the new account. And now that we have the list time to get permissions to copy these assets.
Step 6: Setting Permissions to Assets(ee_permissions)
We now use the report file generated to grant read access to all assets in your account. Once this is completed you will be able to copy your assets apart from being able to copy your codes.
Let us Begin to Copy : We change gears and switch over to the destination account
Step 7: Setting up the Destination Account(ee_user and git_swap)
Now we have to do two steps one after the other, do a quick earthengine authenticate and authenticate to your new account and perform Step 2 and Step 3 this time using your new account. The tool
ee_user will also allow you to change your accounts. I already created Step 2 for my secondary account and now I will use that to authorize my git client with the new account.
Step 8: Replicate Repositories (git_create)
To setup our new account we need to build the outline of the earlier account, the repolists and folders inside these repos and then similary the folders and empty collections in the secondary account.
Note: Git cannot push an empty repository so if you have an empty repository delete it before downloading and pushing to new account
The repo lists now look similar
Step 9: Push to New Account(git_replicate)
Now we push all codes from our earlier account to our new account, this way our repositories will now be populated with the most recent codes.
“Do not push to any repository that already has code because this will overwrite it”
Step 10: Replicate Asset Structure(asset_create) and Assets (asset_replicate)
This is similar to git_create here we replicate the collection and folder structure so we can push our assets to them. You pass it the original report you created from your primary account and it sets up as needed.
This has replicated the collection and Image folder structures.
However this is still empty and the last step makes sure that your assets are actually copied over to your new asset home. I have included a counter to measure transfers left incase it is a large collection.
The final results is your assets and codes all copied, you will still have to edit codes to change your path as needed but for now we have replicated an Earth Engine account into a new location.
There you go, over the last 10 steps we have managed to replicate and move an earth engine account from one place to another. Though I found this useful to move accounts within a university setting, I see some value in moving accounts and replicating when a project member leaves a project or for simply migrating at large. For now if an owner of an account deletes his/her account or looses access to his/her account and even if you are a writer to the repository and the collection, you will loose access to these codes and assets. So this can aid in maintaining continuity by moving codes to more persistent account.
Though I have not tested this tools in a linux setting, these setup tools can be adapted and used easily in that framework, since I have tested the individual components in such setups.
If you find this tool or guide useful please click the clap 👏 button below to show your support! and if you have used it to recreate something or to create better workflow, let me know.