Files Uploader CLI with Nodejs, Typescript, and AWS S3 [Part 2/2]
An interactive CLI application to help uploading file or folder to AWS S3
Introduction
Hi folks, in the first part (https://medium.com/@techwithharry/files-uploader-cli-with-nodejs-typescript-and-aws-s3-part-1-2-86fc29e919ad), we already finished more than 70% of the application where we can start uploading a file to a specific folder. However, there are still some challenges that we need to solve in this part 2 such as:
- Upload a folder instead of just a file
- Create a new folder while uploading
- Creating a new bucket and uploading a file or folder to that newly created folder as well.
So let’s get started!
Getting Started
1. Upload a folder instead of just a file
We need to add one more helper function in the file aws.ts that handles the general logic to upload a folder of files. In addition, sometimes if we also want to upload a folder to S3, we want to be able to choose if we want to keep the root/parent folder or not. Thus, it’s better to add another prompt question to ask if we want to keep the parent folder or not.
Add to then end of file aws.ts with a new helper function uploadToS3 as below
// aws.ts
// Upload a folder to S3
export const uploadToS3 = (
bucketName: string,
uploadingPath: string,
s3UploadingPath: string,
keepParentFolder: boolean = false
) => {
return new Promise(async (resolve) => {
const stats = fs.statSync(uploadingPath);
// Check if this is a directory or not
if (stats.isDirectory()) {
if (keepParentFolder) {
const currentFolder = path.basename(uploadingPath);
}
const files = fs.readdirSync(uploadingPath);
const result: Record<string, {}> = {};
for (const file of files) {
const filePath = path.join(uploadingPath, file);
const uploadingResult = await putObject(
bucketName,
filePath,
keepParentFolder
? path.join(s3UploadingPath, path.basename(uploadingPath))
: s3UploadingPath
);
result[filePath] = uploadingResult;
}
resolve(result);
} else {
resolve(putObject(bucketName, uploadingPath, s3UploadingPath));
}
});
};In inquirer.ts add a basic prompt function to ask for a confirm question about keeping the parent folder and export it.
// inquirer.ts
// Prompt asking for flatten folder choice
export const promptFlattenFolderChoiceQuestion = (): Promise<{
keepParentFolder: boolean;
}> => {
return new Promise<{ keepParentFolder: boolean }>((resolve) => {
inquirer
.prompt([
{
type: "confirm",
name: "keepParentFolder",
message: chalk.green("Do you want to keep parent folder?"),
},
])
.then(resolve);
});
};In index.ts add this below block of code at the end of the main function, outside the while loop to ask user to confirm keeping parent folder.
// index.ts
...
while (true) {
....
}
// NOTE: Add these two lines below
const { keepParentFolder } = await promptFlattenFolderChoiceQuestion();
await uploadToS3(selectedBucket, filepath, selectedFolder, keepParentFolder);
}
main();Try to run npm start <path-to-a-folder> and here is an example of the result.
As you can see from the example above, the files are uploaded correctly to our selecting folder on S3 and we are able to select whether to keep the parent folder or not. Sweet!
2. Add the ability to create a new folder on the fly
The next target is to give the user the ability to create a new folder while uploading the file/folder. A folder in AWS S3 is just a prefix, so this can be done easily by prepending the folder name to the filename, this logic is the same as what we already did in the above section to keep the parent folder.
Add a new prompt helper function to ask for folder name under file inquirer.ts . We also prepare ahead for the type argument with both folder and bucket type so that we can reuse this for the next section.
// inquirer.ts
// Prompt asking for a name
export const promptNameQuestion = (
type: "folder" | "bucket"
): Promise<{
name: string;
}> => {
return new Promise<{ name: string }>((resolve) => {
inquirer
.prompt([
{
type: "text",
name: "name",
message: chalk.green(`What is the new ${type} name?`),
},
])
.then(resolve);
});
};Add this block of code in file index.ts under the if statement for NEW_FOLDER_CHOICE . This block of code asks for the user prompt to enter a new folder name (no blank folder allowed) and configure the folder path on AWS S3.
// index.ts
...
// NOTE: Add the below code
} else if (folder === NEW_FOLDER_CHOICE) {
let folderName: string = "";
// Keep asking till user input the folder name
while (!folderName) {
folderName = (await promptNameQuestion("folder")).name;
}
selectedFolder = prefix + folderName + "/";
break;
} else {
...Here is an example of creating a new folder on the fly
As we already know folder on AWS S3 is just a prefix, so we can create subfolders by having the separator / . For example, the below screenshot displays the newly created folder name study/coding/nodejs . AWS S3 will understand to create 3 subfolders study , coding , and nodejs respectively.
3. Create a new bucket
Firstly, let’s add the helper function on aws.ts to help create a new bucket.
// aws.ts
// List all buckets in S3
export const createBucket = (bucketName: string) => {
return new Promise<{ error: number | string }>((resolve) => {
s3.createBucket({ Bucket: bucketName }, (err: AWSError) => {
if (err && err.message) {
console.log(chalk.red(err.message));
return resolve({ error: err.message });
} else {
console.log(
chalk.green(`Successfully created new bucket s3://${bucketName}`)
);
return resolve({ error: 0 });
}
});
});
};Then in the index.ts file, add the code inside the if statement that selects NEW_BUCKET_CHOICE . This code just basically keeps asking for a new bucket name and try to create that bucket until successful. Because we may enter an existing name that is not globally unique or not follows S3 bucket name policies such as the name length, including / or containing uppercase characters. You can read more here (https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html).
// index.ts
...
// NOTE: Add the below code
if (bucket === NEW_BUCKET_CHOICE) {
while (!selectedBucket) {
let newBucketName = "";
while (!newBucketName) {
newBucketName = (await promptNameQuestion("bucket")).name;
}
// Create new bucket
const { error } = await createBucket(newBucketName);
if (!error) {
selectedBucket = newBucketName;
}
}
} else {
...Try to run the application and here is the example result. As you can see from the screenshot below, I intentionally try to create a bucket name abc123 and the program outputs this bucket already existed, and therefore an error is logged to the console. Therefore, it asks again for the bucket name you want to create again until it can successfully create it.
If you follow up till now, congratulation, you did a great job. There are still many rooms to extend this CLI application to serve more purposes such as:
- Ability to multi-upload, if the file size is large (this is recommended by AWS).
- Ability to display files as well in case the user wants to override
- Ability to download files/folders, we sometimes just do not want to upload only.
- Ability to go back on selection because we may select the wrong folder.
- Ability to ignore a set of files that could be predefined in
.gitignore
That’s it, hope you can have a good time building, learning, and having fun with this simple project. You can find the GitHub repository of this project here https://github.com/hnngo/upload-tool-s3.
If you find my blog interesting, please consider following it for more updates and insights at Tech with Harry.
