Firebase Data Manager — Part 2

In this part, we will explain how we integrate Firebase Data Manager with gitlab, you can find the first part of the article here.

Gitlab CI/CD Integration

We built a CI/CD integration to control any modifications made to the Firestore. First of all, we wanted to construct a validation system, so we wrote validation.sh, which has three basic operations: delete, update, and rename.

Let’s go over the three processes one by one.

We get a list of files that have been modified, removed, or renamed using git commands.

git fetch
FILE_LIST=($(git diff --name-only origin/${CI_COMMIT_REF_NAME}  origin/master))
CHANGED_FILE_LIST=($(git diff --no-commit-id --name-only --diff-filter=dr HEAD^ HEAD))
DELETED_FILE_LIST=($(git diff --no-commit-id --name-only --diff-filter=D HEAD^ HEAD))
RENAMED_FILE_LIST=($(git diff --no-commit-id --name-status --diff-filter=R HEAD^ HEAD))

for updated files:

First of all, we divide the whole path of the changed file into the document directory, document name and collection directory that we will need.

We create the temp file variable; our purpose is to keep the changed file in the temp folder initially. We check to make sure we have made any changes to the Firestore backup files or not. If a change is made to the Firestore backup folder, we want to write it to the temp file first, therefore we use the scp command to copy our changed file to the temp file folder.

To distinguish any changes we made to the document or collection, we made a number of arguments, which we used to trigger the Python file;

parser = argparse.ArgumentParser()
 parser.add_argument('-f', '--file_path', default=None, help='File Path ')
 parser.add_argument('-c', '--collection', default=None, help='Collection ')
 parser.add_argument('-d', '--document', default=None, nargs='?', help='Document Name Optional')
 parser.add_argument('-o', '--operation', default=None, help='Operation')
 args = parser.parse_args()

• File Path : The system takes file path as -f input. The corresponded file includes the data we want to update or create
• Collection: The system takes collection as -c input. It describes path of collection
• Document: We are taking document name by using -d parameter
• Operation: With operation parameter, the we are defining the current operation like update or delete. As an additional note, since firebase is using same function for creating and updating documents and collections, we considered update and create operation as same manner in python code.

Finally, we can execute run.sh, which triggers Python code by giving the collection and document directories with the -o update operation argument.

CURRENT_TIMESTAMP="$(date +%s)"
FILE_NAME="$(basename -- $file)"
PARENT_DIR="${file%/*}"
DOCUMENT_NAME="${PARENT_DIR##*/}"
DOCUMENT_DIR="${PARENT_DIR##*firestore_backup_data/}"
COLLECTION_DIR="${DOCUMENT_DIR%/*}" 
PARSED_TEMP_FILE_NAME=${file////}
TEMP_FILE_NAME=$CURRENT_TIMESTAMP-$PARSED_TEMP_FILE_NAME
TEMP_FILE="$PATH/temp/$TEMP_FILE_NAME"

if [[ "$file" != *"firestore_backup_data/"* ]]; then
    scp -o StrictHostKeyChecking=no -r $file @ip:./$PATH/$file
    continue
  fi

scp -o StrictHostKeyChecking=no -r $file @ip:/$TEMP_FILE

ssh -o StrictHostKeyChecking=no @ip 
sh $PATH/run.sh -f $TEMP_FILE -c $COLLECTION_DIR -d $DOCUMENT_NAME -o update

for deleted files:

As in the update section, we check whether the change is in the backup folder; if it is not in the backup folder, we delete it directly from the machine’s folder path.

If a document or collection is deleted from the backup folders, the python code with the given parameters is executed, so the collection and document changes in the Firestore parts are completed in this stage.

if [[ "$file" != *"firestore_backup_data/"* ]]; then
    ssh -o StrictHostKeyChecking=no @ip "rm -rf $PATH/$file"
    continue
  fi

ssh -o StrictHostKeyChecking=no @ip 
  sh $PATH/run.sh -f $PATH/temp -c $COLLECTION_DIR -d $DOCUMENT_NAME -o delete

for rename files:

A document or collection can be renamed as follows;

Finally, we send the temp file with the new collection directory and the new document name with the update parameter, as well as the old collection directory and the document name with the delete parameter, triggering the python code.

We remove our temp file that we placed on the machine after passing through other control steps. Eventually, we finish our CI/CD step in this way.

if [[ "${#RENAMED_FILE_LIST[@]}" -gt 0 && "${RENAMED_FILE_LIST[@]}" == *firestore_backup_data/* ]]; then
        replaced_files=$(echo ${RENAMED_FILE_LIST[@]} | sed -e "s/R[0-9]* /R /g")
        split_files=( $(echo ${replaced_files[@]} | awk 'BEGIN {FS="R "} {for(i=1;i<=NF;i++)print $i}') )
        if [[ $((${#split_files[@]} % 2)) -eq 0 ]]; then
          for i in "${!split_files[@]}"
          do
            if [[ $(($i % 2)) -eq 0 && "${split_files[i]}" == *firestore_backup_data/* ]]; then
              old_file="${split_files[i]}"
              new_file="${split_files[i+1]}"
              
              FILE_NAME_OLD="$(basename -- $old_file)"
              PARENT_DIR_OLD="${old_file%/*}"
              DOCUMENT_NAME_OLD="${PARENT_DIR_OLD##*/}"
              DOCUMENT_DIR_OLD="${PARENT_DIR_OLD##*firestore_backup_data/}"
              COLLECTION_DIR_OLD="${DOCUMENT_DIR_OLD%/*}" 

              FILE_NAME_NEW="$(basename -- $new_file)"
              PARENT_DIR_NEW="${new_file%/*}"
              DOCUMENT_NAME_NEW="${PARENT_DIR_NEW##*/}"
              DOCUMENT_DIR_NEW="${PARENT_DIR_NEW##*firestore_backup_data/}"
              COLLECTION_DIR_NEW="${DOCUMENT_DIR_NEW%/*}" 

              PARSED_TEMP_FILE_NAME=${new_file////}
              TEMP_FILE_NAME=$CURRENT_TIMESTAMP-$PARSED_TEMP_FILE_NAME
              TEMP_FILE="$PATH/temp/$TEMP_FILE_NAME"
              scp -o StrictHostKeyChecking=no -r $new_file @ip:$TEMP_FILE

              ssh -o StrictHostKeyChecking=no @ip 
              "sh $PATH/run.sh -f $TEMP_FILE -c $COLLECTION_DIR_NEW -d $DOCUMENT_NAME_NEW  -o update && \
              sh $PATH/run.sh -f $TEMP_FILE -c $COLLECTION_DIR_OLD -d $DOCUMENT_NAME_OLD -o delete"

              RESULT=$?
              if [ "$RESULT" -ne "0" ]; then
                FAILED=1
                echo $file has ERROR.
              else
                echo Passed.
              fi
            fi
          done
        else
          echo "ERROR:Renamed File list length must be a multiple of two, because it contains old version and the new version name of the renamed file."
          exit 1
      fi
    else
      echo "There is no any renamed document file."
    fi

  ssh -o StrictHostKeyChecking=no @ip  "rm -rf $PATH/temp/$CURRENT_TIMESTAMP-*.json"

By using this structure, we can easily update our documents and collections via Gitlab. On Firestore, it takes a lot of time to change multiple names, remove documents, or edit fields. However, using this approach you can change more than 1000 file without any struggle in 5 minutes!