Migrating Media from Cloudinary to AWS S3 Using Strapi

Introduction

This blog post outlines a comprehensive process for migrating media assets from Cloudinary to AWS S3 using Strapi. This migration can offer several benefits, including potential cost reduction and increased control over your media assets.

Prerequisites

Before starting the migration, ensure you have the following:

• An AWS account with a created bucket.
• Access to your Strapi administration panel (either production or local) to utilize the Strapi media API for downloading assets.

Migration Strategy

We’ll employ a safe and efficient migration strategy involving these key steps:

1. Download Media Assets:

• Download all media assets (images, files, videos, etc.) from your current Strapi instance to your local machine via the strapi media API.
• Simultaneously create JSON metadata files for each asset, storing essential information.

2. Set Up Local Strapi Environment:

• Establish a local Strapi environment configured to use AWS S3 for media storage.
• Configure the AWS S3 provider with your AWS credentials and bucket information.

3. Upload Assets to AWS S3:

• Upload the downloaded media assets to your local Strapi instance.
• Strapi will automatically transfer these assets to your designated AWS S3 bucket.

Detailed Steps

Step 1: Download Media Assets

• We need to have a Node.js script to fetch media files from the Strapi API in batches.
• Create local directories to store downloaded assets and corresponding JSON metadata.
• For each asset, save the downloaded file and its metadata as JSON.

Step 1.1: Download Media Assets

Here is the Node.js script that handles the downloading process. This script fetches media files in batches and saves them locally.

downloadFiles.js

const axios = require("axios")
const fs = require("fs")
const path = require('node:path');
const stream = require("stream")
const util = require("util")
const pipeline = util.promisify(stream.pipeline);


const FILE_JSON_FOLDER = "./tmp_media/json"
const MEDIA_FOLDER = "./tmp_media/media"
function createIfNotExists(folderName) {
    if (!fs.existsSync(folderName)) {
        console.log(`${folderName} does not exist.`)
        fs.mkdirSync(folderName)
        if (!fs.existsSync(folderName)) {
            throw new Error("Could not create folder.")
        }
    }
}
createIfNotExists("./tmp_media")
createIfNotExists(FILE_JSON_FOLDER)
createIfNotExists(MEDIA_FOLDER)
async function getFiles(start, limit) {
    try {
        const res = await axios.get(`https://admin.your-domain.com/upload/files?_start=${start}&_limit=${limit}`) // ADD this in your .env file.
        const { data } = res
        return data
    } catch (e) {
        console.error(`Failed to fetch files in range ${start} - ${limit}`)
        console.error(e.response)
        return e
    }
}
async function* loopFiles() {
    let start = 0
    const limit = 100
    const errorCount = 0
    const errorLimit = 10
    while (true) {
        const files = await getFiles(start, limit)
        if (files.length === 0) {
            break
        }
        for (const file of files) {
            yield file
        }
        if (errorCount > errorLimit) {
            break
        }
        start += limit
    }
}
// Download file as stream
async function downloadFileAsStream(fileUrl) {
    try {
        const imgStream = await axios.get(
            fileUrl,
            { responseType: 'stream' }
        )
        const headers = imgStream['headers']
        const disposition = headers['content-disposition']
        const fileName = !!disposition ? disposition.replace(/.+filename=(.+)/, "$1") : fileUrl.replace(/.+\/(.+)/, "$1")
        await pipeline(imgStream.data, fs.createWriteStream(`${MEDIA_FOLDER}/${fileName.replace("\"", "")}`));
        return fileName
    } catch (e) {
        console.error(`Could not download file ${fileUrl}`, e)
        return null
    }
}
// Process all files
async function processAllFiles() {
    for await (const file of loopFiles()) {
        const fileStr = JSON.stringify(file, null, 2)
        const filePath = path.join(FILE_JSON_FOLDER, '/', `${file.id}.json`)
        fs.writeFileSync(filePath, fileStr)
        const downloadedFile = await downloadFileAsStream(file.url)
        console.info(`Processing file id: ${file.id} - ${downloadedFile}`)
    }
}
processAllFiles().then(() => console.log("Finished!"))

Now run the Script:

Execute the script to download all your media files to your local computer:

node downloadFiles.js

It should look something like this after you run the script:

Step 2: Configure Local Strapi

• Set up a new Strapi project or configure an existing one to use AWS S3.
• Install the required AWS S3 upload provider for Strapi.
• Create environment variables in your .env file to store AWS credentials and bucket information.
• Configure Strapi’s middleware and plugins to use the AWS S3 provider.
• Test the configuration by uploading a sample image to ensure it’s stored in the AWS S3 bucket.

Step 2:1 Configure Local Strapi and AWS S3

This step involves setting up Strapi to use your AWS S3 bucket for media storage.

1.1: Environment Variables

1. Create a file named .env in the root directory of your Strapi project (if it doesn't exist already).
2. Inside the .env file, add the following environment variables, replacing the placeholders with your actual AWS credentials and bucket information:

AWS_ACCESS_KEY_ID=your_aws_access_key_id
AWS_ACCESS_SECRET=your_aws_secret_access_key
AWS_REGION=your_aws_region
AWS_BUCKET=your_aws_bucket_name

Important Security Note: Storing your AWS credentials directly in a .env file is not ideal for production environments. Consider using a more secure method like environment variables managed by your hosting provider or a secrets management service.

2.2: Configure AWS S3 Provider

• Install the AWS S3 upload provider for Strapi:

npm install @strapi/provider-upload-aws-s3 --save
# or
yarn add @strapi/provider-upload-aws-s3

Update your config/middlewares.js :

module.exports = [
  'strapi::errors',
  {
    name: 'strapi::security',
    config: {
      contentSecurityPolicy: {
        useDefaults: true,
        directives: {
          'connect-src': ["'self'", 'https:'],
          'img-src': [
            "'self'",
            'data:',
            'blob:',
            'bucket-name.s3.region.amazonaws.com',
          ],
          'media-src': [
            "'self'",
            'data:',
            'blob:',
            'bucket-name.s3.region.amazonaws.com',
          ],
          upgradeInsecureRequests: null,
        },
      },
    },
  },
  'strapi::cors',
  'strapi::poweredBy',
  'strapi::logger',
  'strapi::query',
  'strapi::body',
  'strapi::session',
  'strapi::favicon',
  'strapi::public',
];

Update your config/plugins.js file to configure the AWS S3 provider:

module.exports = ({ env }) => ({
  upload: {
    provider: 'aws-s3',
    providerOptions: {
      accessKeyId: env('AWS_ACCESS_KEY_ID'),
      secretAccessKey: env('AWS_ACCESS_SECRET'),
      region: env('AWS_REGION'),
      params: {
        Bucket: env('AWS_BUCKET')
      }
    }
  }
});

This configuration tells Strapi to use the aws-s3 provider and provides the necessary credentials and bucket information retrieved from the environment variables.

2.3: Test the Configuration

• Run your Strapi project:

npm run develop # or yarn develop

• Upload an image via Strapi’s admin panel.
• Verify that the image is saved in your AWS S3 bucket and accessible through the provided URL.

By following these steps, you should have successfully configured Strapi to use your AWS S3 bucket for media storage.

Step 3: Upload Assets to AWS S3

• Create a script to process the downloaded JSON metadata files.
• For each asset:
• Find the corresponding downloaded file.
• Upload the file to the local Strapi instance using the Strapi upload API.
• Update the JSON metadata with the new asset URL and any additional information.

Step 3.1: Upload Assets to AWS S3

Create a new script file named uploadFiles.js to upload the previously downloaded media to Strapi:

const axios = require('axios');
const fs = require('fs');
const path = require('path');
const FormData = require('form-data');


const FILE_FOLDER = './tmp_images/images_1';
const JSON_FILE_FOLDER = './tmp_images/json_1';
const PROCESSED_JSON_FOLDER = './tmp_images/processed_json';
const STRAPI_UPLOAD_URL = 'http://localhost:1337/upload'; //API endpoint change it accordingly.

// Ensuring the processed JSON folder exists
if (!fs.existsSync(PROCESSED_JSON_FOLDER)) {
    fs.mkdirSync(PROCESSED_JSON_FOLDER, { recursive: true });
}

// Function to upload a file to Strapi
async function uploadFile(filePath, fileName, retries = 3) {
    const formData = new FormData();
    formData.append('files', fs.createReadStream(filePath));
    try {
        const response = await axios.post(STRAPI_UPLOAD_URL, formData, {
            headers: formData.getHeaders(),
            maxContentLength: Infinity,
            maxBodyLength: Infinity,
        });
        console.log(`Uploaded: ${fileName}`);
        return response.data;
    } catch (e) {
        console.error(`Error uploading file ${fileName}:`, e.message);
        if (retries > 0) {
            console.log(`Retrying upload (${retries} attempts left)...`);
            return await uploadFile(filePath, fileName, retries - 1);
        }
        console.error(`Could not upload file ${fileName} after multiple attempts`);
        return null;
    }
}

// Function to process all JSON files
async function processFiles() {
    try {
        const jsonFiles = fs.readdirSync(JSON_FILE_FOLDER);
        for (const jsonFile of jsonFiles) {
            const jsonFilePath = path.join(JSON_FILE_FOLDER, jsonFile);
            let json;
            try {
                const jsonFileContent = fs.readFileSync(jsonFilePath, 'utf8');
                json = JSON.parse(jsonFileContent);
            } catch (e) {
                console.error(`Error reading or parsing JSON file ${jsonFile}:`, e.message);
                continue;
            }
            if (!json || typeof json !== 'object') {
                console.error(`Invalid JSON content in file ${jsonFile}`);
                continue;
            }
            const { id, url } = json;
            if (!id || !url) {
                console.error(`Missing 'id' or 'url' in JSON file ${jsonFile}`);
                continue;
            }
            const fileName = path.basename(url, path.extname(url));
            const newJsonFilePath = path.join(PROCESSED_JSON_FOLDER, `${id}.json`);
            if (fs.existsSync(newJsonFilePath)) {
                console.log(`File already processed: ${fileName}`);
                continue;
            }
            let imageFilePath;
            const files = fs.readdirSync(FILE_FOLDER);
            for (const file of files) {
                if (file.startsWith(fileName)) {
                    imageFilePath = path.join(FILE_FOLDER, file);
                    break;
                }
            }
            if (!imageFilePath || !fs.existsSync(imageFilePath)) {
                console.error(`Image file not found for extracted file name: ${fileName}`);
                continue;
            }
            const uploadedFile = await uploadFile(imageFilePath, fileName);
            if (!uploadedFile) {
                console.error(`Could not upload file ${fileName}`);
                continue;
            }
            const newUrl = uploadedFile[0].url;
            const getFormats = uploadedFile[0].formats;
            const newJson = { id, url: newUrl, formats: getFormats };
            fs.writeFileSync(newJsonFilePath, JSON.stringify(newJson, null, 2));
            console.log(`Processed file: ${fileName} - ${newUrl}`);
        }
    } catch (e) {
        console.error('Error reading JSON file folder:', e.message);
    }
}
processFiles().then(() => console.log('Finished!')).catch(err => console.error('An error occurred during processing:', err));

Step 3.2: Run the Script

Execute the script to upload all your media files to the new Strapi setup:

node uploadFiles.js

Additional Considerations

• The provided scripts have been tested with a dataset of 5000 assets and are suitable for both large and small datasets.
• While this guide uses Strapi version 3 and Node.js version 20, the process should be adaptable to newer versions.
• Prioritize robust error handling and testing throughout the migration process.
• Consider implementing additional security measures to protect your AWS credentials.

Conclusion

By following these steps and leveraging the provided scripts, you can successfully migrate your Strapi media assets from Cloudinary to AWS S3. This migration can offer cost savings and improved control over your media.

Disclaimer: This blog post is a general guide and may require adjustments based on specific project requirements and technical environments.

Written by: Murtaza Jafari, in collaboration with Gil Fernandes

Reflections on AI

.

Happy Coding!