Build a blog application on Google App Engine: Image module (part 6)
This is the sixth part of a multipart tutorial on how to build a small blog application in Node.js using the Google Datastore and deploy it to Google App Engine. If you missed the beginning, jump to the first part where I explain how to set up the project and where you’ll find the links to the other parts.
In this post, we are going to create the Images module that will allow us to upload and delete a featured image in Google Cloud Storage.
But first, let’s do a quick refactor. Let’s move the creation of the file upload middleware that we created in the previous post, to the Images module. By doing this, the middleware will be available to other modules from a single place. Open the “admin.routes.ts” file and delete the line that imports multer at the top as well as the block starting with const uploadInMemory = multer(...)
Create a new file in the Images module and name it “middlewares.ts”. Add the following into it:
// modules/images/middlewares.tsimport multer from "multer";export interface ImagesMiddleware {
uploadInMemory: multer.Instance;
}export default (): ImagesMiddleware => ({
/**
* Multer handles parsing multipart/form-data requests.
* This instance is configured to store images in memory
*/
uploadInMemory: multer({
storage: multer.memoryStorage(),
limits: {
fileSize: 5 * 1024 * 1024 // no larger than 5mb
},
fileFilter: (req, file, cb) => {
// Validate image type
if (["image/jpeg", "image/png"].indexOf(file.mimetype) < 0) {
const err = new Error(
`File type not allowed: ${req.file.mimetype}`
);
return cb(err, false);
}
return cb(null, true);
}
})
});
Great, now that we’ve moved the uploadInMemory() middleware to its own layer, open the “index.ts” entry file of our Images module and replace its content with:
// modules/images/index.tsimport { Context } from "./models";import initImagesMiddleware, {
ImagesMiddleware
} from "./middlewares";export interface ImagesModule {
middlewares: ImagesMiddleware;
}export default (context: Context): ImagesModule => {
const middlewares = initImagesMiddleware(); return {
middlewares
};
};
Now let’s go back to the “admin.routes.ts” file and use our newuploadInMemory() middleware from the Images module.
// modules/admin/admin.routes.ts...
router.get("/edit-post/:id", routesHandlers.editPost);
router.post(
"/create-post",
[images.middlewares.uploadInMemory.single("image")],
routesHandlers.createPost
);
router.post(
"/edit-post/:id",
[images.middlewares.uploadInMemory.single("image")],
routesHandlers.editPost
);
...
Great! We have a middleware that will parse any image file coming into the request. We now need a middleware that will upload this file to Google Cloud Storage. For that, we will create an abstraction layer to upload and delete a file from GCS. Let’s get started!
In the Images module, create a new file “google-cloud-storage.ts” and add the following:
// modules/images/google-cloud-storage.tsimport { Context } from "./models";export interface GoogleCloudStorage {
uploadFile(
fileName: string,
mimetype: string,
buffer: Buffer
): Promise<any>;
}export default ({
config,
logger,
storage
}: Context): GoogleCloudStorage => {
const bucketId = config.gcloud.storage.bucket;
const bucket = storage.bucket(bucketId); /**
* Returns the public, anonymously accessible URL to a given Cloud
* Storage object. The object's ACL has to be set to public read.
*
* @param {string} objectName -- Storage object to retrieve
*/
const getPublicUrl = (objectName: string) =>
`https://storage.googleapis.com/${bucketId}/${objectName}`; const uploadFile = (
fileName: string,
mimetype: string,
buffer: Buffer
) => {
return new Promise((resolve, reject) => {
const gcsname =
Date.now() + fileName.replace(/\W+/g, "-").toLowerCase();
const googleStorageFile = bucket.file(gcsname); const stream = googleStorageFile.createWriteStream({
metadata: {
contentType: mimetype,
cacheControl: "public, max-age=31536000" // cahe 1 year
},
validation: "crc32c",
predefinedAcl: "publicRead"
}); stream.on("error", reject); stream.on("finish", () => {
resolve({
cloudStorageObject: gcsname,
cloudStoragePublicUrl: getPublicUrl(gcsname)
});
}); stream.end(buffer);
});
}; return {
uploadFile
};
};
Let’s go quickly through the code. First,we import the Context type that we will add in a minute. We then declare our GoogleCloudStorage interface that for now has only one method: uploadFile().
Here again, the Context is a required input to the layer and we’ve destructured the config, logger and storage object out of it. We then read the bucketId from our configuration object and create a Google Cloud Storage bucket instance (bucket) where we will upload the file.
We then generate a unique name (by taking the current date and removing any non-word character by a dash) for the file and finally we create a Storage File object (googleStorageFile) where we will be able to write (stream) data to.
Let’s now create the Express middleware that will make use of our new GoogleCloudStorage layer and save the file in Google Storage. Open the “middlewares.ts” file and add the following:
// modules/images/middlewares.tsimport { Request, Response, NextFunction } from "express"; // Add
import multer from "multer";
import { GoogleCloudStorage } from "./google-cloud-storage"; // Addexport interface ImagesMiddleware {
uploadInMemory: multer.Instance;
uploadToGCS(req: Request, _: Response, next: NextFunction): void;
}export default (
googleCloudStorage: GoogleCloudStorage // Add the dependency
): ImagesMiddleware => ({
/**
* Multer handles parsing multipart/form-data requests.
* This instance is configured to store images in memory
*/
uploadInMemory: multer({ ... }),
/**
* Middleware to upload a file in memory (buffer) to Google Cloud
* Storage Once the file is processed we add a
* "cloudStorageObject" and "cloudStoragePublicUrl" property
*/
uploadToGCS(req, _, next) {
if (!req.file) {
return next();
} const { originalname, mimetype, buffer } = req.file;
googleCloudStorage
.uploadFile(originalname, mimetype, buffer)
.then(
({ cloudStorageObject, cloudStoragePublicUrl }: any) => {
(<any>req.file).cloudStorageObject = cloudStorageObject;
(<any>(
req.file
)).cloudStoragePublicUrl = cloudStoragePublicUrl; next();
}
)
.catch((err: any) => {
next(err);
});
}
});
We first import a few typings from Express. Then, in our middleware, we first check if there is a file on the request object, if there are none we exit the middleware by calling next(). The rest of the code is quite straight-forward, the important part is when the file finishes uploading we add two properties to the file object: cloudStorageObject and cloudStoragePublicUrl. Those are the properties that we will save in the Datastore in order to display the image or delete it when needed.
Let’s quickly add now our missing types. Create a “models.ts” file in the same folder and add the following:
// modules/images/models.tsimport Storage from "@google-cloud/storage";
import { Logger } from "winston";export type Config = {
gcloud: {
projectId: string;
storage: {
bucket: string;
};
};
};export type Context = {
logger: Logger;
config: Config;
storage: Storage;
};
Let’s initialize our “google-cloud-storage” layer and provide it to our middlewares layer. Open the “index.ts” file from our Images modules and add the following:
// modules/images/index.tsimport { Context } from "./models";import initImagesMiddleware, {
ImagesMiddleware
} from "./middlewares";import initGCS, { // Add this import
GoogleCloudStorage
} from "./google-cloud-storage";export interface ImagesModule {
middlewares: ImagesMiddleware;
GCS: GoogleCloudStorage; // Add this line
}export default (context: Context): ImagesModule => {
const GCS = initGCS(context); // Add this
const middlewares = initImagesMiddleware(GCS); // Edit return {
middlewares,
GCS
};
};
We now need to provide the Context to our Images module. Open the main “modules.ts” file:
// modules.ts...export default (context: Context): AppModules => {
const utils = initUtilsModule();
const images = initImagesModule(context); // Edit this line
...
And the last thing to do is, of course, to add our uploadToGCS() Express middleware to our admin routes:
// modules/admin/admin.routes.ts...router.post(
"/create-post",
[
images.middlewares.uploadInMemory.single("image"),
images.middlewares.uploadToGCS // Add this line
],
routesHandlers.createPost
);
router.post(
"/edit-post/:id",
[
images.middlewares.uploadInMemory.single("image"),
images.middlewares.uploadToGCS // Add this line
],
routesHandlers.editPost
);...
Great! with this, we should be able to upload an image file to our Google Storage bucket. But we still need to save in our BlogPost entities the two properties we’ve added on the file object (cloudStorageObject an cloudStoragePublicUrl). For that, we are going to use a gstore middleware that will execute right before a BlogPost entity is saved in the Datastore.
gstore middleware
A middleware (or a hook) is a function that is executed before or after a target function is called (read my other post about hooks). We can add as many middlewares as we need and they will all execute in sequence. In our current use case, each time a blogPost.save() method is called. In gstore-node , hooks are declared on the schema, using the pre() and post() methods.
// Middleware examplefunction doSomething() {
... logic that returns a Promise
}function doSomethingElse() {
... logic that returns a Promise
}/*
* We add the 2 middleware to our schema.
* They will be executed before the entity is saved in the
* Datastore. If they throw an error, the entity won't be saved.
*/
myGstoreSchema.pre('save', [doSomething, doSomethingElse]);
Let’s create our first middleware to extract the two file properties we’ve added and put them on the entity data that will be saved in the Datastore.
Create a file “blog-post.db.hooks.ts” in the blog-post module folder and add the following:
// modules/blog/blog-post/blog-post.db.hooks.tsimport { Context, Modules } from "../models";export default (
{ gstore }: Context,
{ images, utils }: Modules
) => {
/**
* Middleware to initialize the entityData
* before saving it in the Datastore
*/
function initEntityData(): Promise<any> {
this.entityData = addCloudStorageData(this.entityData); // A gstore middleware must always return a Promise
return Promise.resolve();
} /**
* If the entity has a "file" property attached to it
* we retrieve its publicUrl and cloudStorageObject
*/
function addCloudStorageData(entityData: any) {
if (entityData.file) {
return {
...entityData,
posterUri: entityData.file.cloudStoragePublicUrl || null,
cloudStorageObject:
entityData.file.cloudStorageObject || null
};
} else if (entityData.posterUri === null) {
/**
* Make sure that if the posterUri is null
* the cloud storage object is also null
*/
return { ...entityData, cloudStorageObject: null };
}
return entityData;
} return {
initEntityData
};
};
The first thing we see is that, here again, our database hook layer will need the Context and Modules to be provided. We then have an initEntityData() function which will be the middleware that will execute before a BlogPost entity is saved in the Datastore. gstore-node sets the middleware scope (this) on the entity being saved. We then call the addCloudStorageData() method to either add or remove the “posterUri” and “cloudStorageObject” properties from the entity.
Let’s now attach this middleware to the BlogPost schema. We could open our “blog-post.db.ts” file and add it right there, but if we did that we would add business (domain) logic to our database layer. Instead, we are going to use some helpers from our Utils module.
Open the “blog-post.db.ts” file and add the following:
// modules/blog/blog-post/blog-post.db.ts...type FunctionReturnPromise = (...args: any[]) => Promise<any>;export interface BlogPostDB {
...
deletePost(id: number): Promise<DeleteResult>;
addPreSaveHook(
handler: FunctionReturnPromise | FunctionReturnPromise[]
): void;
addPreDeleteHook(
handler: FunctionReturnPromise | FunctionReturnPromise[]
): void;
addPostDeleteHook(
handler: FunctionReturnPromise | FunctionReturnPromise[]
): void;
}export default (context: Context, modules: Modules): BlogPostDB => {
... const ancestor = ["Blog", "default"]; const {
addPreSaveHook,
addPreDeleteHook,
addPostDeleteHook
} = utils.gstore.initDynamicHooks(schema, context.logger); ... /**
* DB API
*/
return {
... deletePost(id) {
return BlogPost.delete(id, ancestor);
},
addPreSaveHook, // Add this line
addPreDeleteHook, // Add this line
addPostDeleteHook // Add this line
};
};
Great, we now have 3 methods to attach middlewares to our gstore schema and the database layer does not have to know about them. Now open the “blog-post.domain.ts” file and make the following modifications to add the initEntityData middleware to our schema:
// modules/blog/blog-post/blog-post.domain.tsimport marked from 'marked';
import Boom from 'boom';
import initDBhooks from './blog-post.db.hooks'; // Add this line
...export default (
context: Context,
{ blogPostDB, images, utils }: Modules
): BlogPostDomain => {
/**
* Add "pre" and "post" hooks to our Schema
*/
const { initEntityData } = initDBhooks(context, {
images,
utils
});
blogPostDB.addPreSaveHook([initEntityData]); ...
};
We are almost there! When we initiated our BlogPost module, we didn’t provide the required Modules dependency. Let’s correct that:
// modules/blog/index.ts...const blogPost = initBlogPost(context, modules); // Edit this line
And now let’s forward the dependency to our blogPostDomain initialization:
// modules/blog/blog-post/index.ts...const blogPostDomain = initDomain(context, {
blogPostDB,
...modules
});
Great! You should now be able to create a post and upload a featured image with it. If you navigate to your Google Cloud Console, open the Storage > Browser and go inside the bucket you have defined for this project, you should see the image uploaded there.
Awesome! But what happens if you now edit the post and upload a new image? Try it. You should now have 2 images in the Google Storage Bucket. The old one and the new one. This is obviously not what we want.
In order to fix this, we will need:
- a method to delete a file from Google Storage.
- a middleware that will call this method to delete any featured image associated with a blog post.
Let’s start by creating a deleteFile() method in our GoogleCloudStorage interface.
// modules/images/google-cloud-storage.tsimport async from "async"; // Add this line
import arrify from "arrify"; // Add this line
import { Context } from "./models";export interface GoogleCloudStorage {
uploadFile(
fileName: string,
mimetype: string,
buffer: Buffer
): Promise<any>;
deleteFile(objects: string | Array<string>): Promise<any>; // Add
}export default ({
config,
logger,
storage
}: Context): GoogleCloudStorage => {
... const uploadFile = () => { ... }; /**
* Delete one or many objects from the Google Storage Bucket
* @param {string | array} objects -- Storage objects to delete
*/
const deleteFile = (objects: string | Array<string>) => {
return new Promise((resolve, reject) => {
const storageObjects = arrify(objects);
const fns = storageObjects.map(o => processDelete(o)); async.parallel(fns, err => {
if (err) {
return reject(err);
} logger.info(
"All object deleted successfully from Google Storage"
); return resolve();
});
}); // ---------- function processDelete(fileName: string) {
return (cb: async.AsyncFunction<null, Error>) => {
logger.info(`Deleting GCS file ${fileName}`); const file = bucket.file(fileName);
file.delete().then(
() => cb(null),
err => {
if (err && err.code !== 404) {
return cb(err);
}
cb(null);
}
);
};
}
}; return {
uploadFile,
deleteFile // Add this line
};
};
Perfect. We now need to create a middleware that will call the deleteFile() method and make sure that any previously saved image is deleted before a new one is saved to Google Storage.
Open out “blog-post.db.hooks.ts” file and add the following:
// modules/blog/blog-post/blog-post.db.hooks.tsimport { Entity } from 'gstore-node'; // Add this line
import { Context, Modules } from '../models';
import { BlogPostType } from './models'; // Add this lineexport default (
{ gstore }: Context,
{ images, utils }: Modules
) => {
/**
* If the entity exists (it has an id) and we pass "null" as
* posterUri or the entityData contains a "file", we fetch the
* entity to check if it already has an feature image.
* We use the Dataloader instance to fetch the entity.
*/
function deletePreviousImage(): Promise<any> {
if (!this.entityKey.id) {
return Promise.resolve();
} if (
this.posterUri === null ||
typeof this.entityData.file !== "undefined"
) {
const dataloader = this.dataloader
? this.dataloader
: this.context && this.context.dataloader; return dataloader
.load(this.entityKey)
.then((entity: Entity<BlogPostType>) => {
/**
* If there is no entity or cloudStorageObject
* there is nothing to do here...
*/
if (!entity || !entity.cloudStorageObject) {
return;
} /**
* Delete the object from Google Storage
*/
return images.GCS.deleteFile(entity.cloudStorageObject);
});
} return Promise.resolve();
} ... return {
initEntityData,
deletePreviousImage,
};
};
Let’s now attach this middleware to our BlogPost schema.
// modules/blog/blog-post/blog-post.domain.ts...export default (
context: Context,
{ blogPostDB, images, utils }: Modules
): BlogPostDomain => {
/**
* Add "pre" and "post" hooks to our Schema
*/
const { initEntityData, deletePreviousImage } = initDBhooks(
context,
{ images, utils }
);
blogPostDB.addPreSaveHook([deletePreviousImage, initEntityData]); ...};
Fantastic! You can now edit a post, update the featured image and the old image (if there is any) will be deleted from Google Storage when updating the entity.
We need to do something very similar when deleting a post. For that, we are going to add another middleware that will execute right before a BlogPost entity is deleted.
Open the “blog-post.db.hook.ts” file and add the following:
// modules/blog/blog-post/blog-post.db.hooks.ts...export default (
{ gstore }: Context,
{ images, utils }: Modules
) => {
... /**
* Delete image from GCS before deleting a BlogPost
*/
function deleteFeatureImage() {
// We fetch the entityData to see if there is a
// cloud storageobject
return this.datastoreEntity().then(
(entity: Entity<BlogPostType>) => {
if (!entity || !entity.cloudStorageObject) {
return;
}
return images.GCS.deleteFile(entity.cloudStorageObject);
}
);
} return {
initEntityData,
deletePreviousImage,
deleteFeatureImage
};
};
We have used the gstore-node helper functiondatastoreEntity() that will go and fetch the entity data from the Datastore for us (remember the scope “this” is set on the entity being deleted). We then delete the cloud storage object if there is one. Let’s now attach this middleware to our schema:
// modules/blog/blog-post/blog-post.domain.ts.../**
* Add "pre" and "post" hooks to our Schema
*/
const {
initEntityData,
deletePreviousImage,
deleteFeatureImage // Add this
} = initDBhooks(context, { images, utils });blogPostDB.addPreSaveHook([deletePreviousImage, initEntityData]);
blogPostDB.addPreDeleteHook(deleteFeatureImage); // Add this...
Great! You can now delete a post and with it, any featured image uploaded to Google Storage.
It has been a long post, so if you’re reading this it means that I managed to not get you lost along the way :)
Like always, please reach out in the comments if you have any question or if you see any error.
With the Images module under our belt, our application starts looking really really good! One important part is missing though for a blog: allow the users to comment on a post.
This is what we will be seeing in the next part of this tutorial. Let’s go for it!
