The proper way to handle external data sources (pt1)

Photo by Stacey Gabrielle Koenitz Rozells on Unsplash

The problem

Often after fetching data from external data sources, we’d need to parse them to our own data structure. A few reasons for that are:

• Sanity check: we would want to assure what the data is, which structure it has, before passing it forward in our internal logics.
• Easier maintenance: imagining if we do not have a gateway to deserialise 3rd party data and use the library’s external types across your code base. Upgrading to a new version, having breaking changes, is going to be a nightmare. As there will be dozens of places need to be changed. On the other hand, having a centralised schema where the rest of the code base follow. Combining with proper test factories. Our migration will involve much less hassles.

The use case

Let’s have a look at this simple use case, where we would look at several approaches for:

1. Make an API request to fetch an user object
2. Parse the data to our own Userdata structure

2 solutions that we can try in this series are:

1. Pure JS: manually parse the data and throwing custom errors
2. fp-ts & io-ts to achieve the same thing, but in a more “pure”, functional way

Pure TS solution of parsing data

The high-level flow would look like:

type SuccessResponse = {
  success: true
  user: User
}

type ErrorResponse = {
  success: false
  errorCode: string
}

const executeWholeFlow = async (userId: string): Promise<SuccessResponse | ErrorResponse> => {
  try {
  // Step 1: fetch the user from API
    const maybeUser = await fetchUserFromApi(userId)
  // Step 2: parse the API response
    const parsedUserDbOutput = parseDbUserOutput(maybeUser)
 } catch(error) {
  // TODO: handle the error
 }
}

where we’d return either SuccessResponseor ErrorResponse. Now let’s dive into each step!

Step 1: fetch the user from API

To make it simple, we are not going to call any real API. We’d instead just have a simple fetchById function to either resolve a Promise of type unknown(happy case) or throws a custom UserApiError(failure case) based on the given id, for the sake of simplicity.

The error class looks like:

class UserApiError extends Error {
  constructor(message: string) {
    super(message)
  }
}

The fetching looks like:

const userApi = {
  fetchById: (id: string): Promise<unknown> => {
    const onlyAvailableId = '1'
    if(id === onlyAvailableId) return Promise.resolve({id: onlyAvailableId, name: 'Foo'})
    
    throw new UserApiError('not found')
  }
}

const fetchUserFromApi = (id: string): Promise<unknown> => {
  return userApi.fetchById(id)
}

We’d treat the return type in happy case to be Promise<unknown> because we are going to parse it to our own data structure in the next step.

💡If you are curious about why unknown is useful in this case, checkout another article of mine:

The unknown type in TypeScript

Step 2: parse the API response

Now that we got the API response, let’s try to parse it:

type User = {
  id: string
  name: string
}

class ParseUserDataError extends Error {
  constructor(message: string) {
    super(message)
  }
}

const parseDbUserOutput = (val: unknown): User => {
  if (typeof val !== 'object' || val === null || !('id' in val) || !('name' in val)) {
    throw new ParseUserDataError('parsing_api_response_error')
  }

  return val as User
}

Awesome! Our parser parseDbUserOutput now either returns a User if we assure that the API response is an object having id and name. Otherwise it would throw a custom error ParseUserDataError.

💡 You can always use external schema validation libraries e.g. Zod or RunType to parse the data

Why bother having a custom error, you might ask? We might want to handle it differently compared to previous UserApiError. E.g. showing different error messages in the front-end or having different logging and monitoring for these.

Putting things together

Now that we went through internal logics, let’s revisit the main function executeWholeFlow and especially see how can we handle the error there:

const executeWholeFlow = async (userId: string): Promise<SuccessResponse | ErrorResponse> => {
  try {
    const maybeUser = await fetchUserFromApi(userId)
    const parsedUserDbOutput = parseDbUserOutput(maybeUser)
    return {
      success: true,
      user: parsedUserDbOutput
    }
  } catch(error) {
    if(error instanceof UserApiError) {
      return {
        success: false,
        errorCode: 'api_error'
      }
    }
    if(error instanceof ParseUserDataError) {
      return {
        success: false,
        errorCode: 'parsing_api_response_error'
      }
    }

    return {
      success: false,
      errorCode: 'unknown_error'
    }
  }
}

Pretty nice! Now our parsing functionality is in a great shape. In the current implementation, we achieved:

1. Parse the external API response to our own User data structure.
2. Differentiate errors thrown from different logics. This way, we have way better visibility on what the final error actually is, be it API error where 3rd party API returns a failure response, or parsing error when their response does not match our schema.

Room for improvements

Our flow works now. However, there are definitely rooms for improvements:

Tedious custom errors

It is quite tedious to have 1 custom error class for each logic. Let’s say we want to introduce a logic to persist the user object to our database → we’d need a new error PersistToDbError. And the more complex our flow is, the more custom errors we need to introduce.

“Loose” error handling

You might have noticed that our flow currently does not enforce handling error.

Let’s say when fetching the user from API, fetchUserFromApi might throw UserApiError. However, you are not required to explicitly handle that error in catch block of executeWholeFlow.

So, the more steps we expand the flow with, the more likely it is to overlook error handling. Because we lack something to enforce us doing so.

io-ts & fp-ts comes in to save

We can improve our flow and tackle potential issues above by making parsing and handling errors a must. In practice, this means our code would not compile if we lack either one of those.

This way we are getting the best out of TypeScript’s amazing type system. This results in codes less prone to bugs and explicit error handling.

Observability also comes at ease. Because once we have a generic error type, we only need one abstraction to observe and monitor different error types.

Then it will be much easier making sense out of issues occurred in production, and more importantly, where to find them. That could save hours of debugging, rather than bagging our heads against the wall having TypeError: cannot read property ‘foo’ of undefined.

Since this article is already quite long, let’s wrap it up here. Stay tuned for the follow-up one where we would look at the flow in a different angle — pure functions and explicit error handling.

That’s the end of this article. It means the world for me if you like it and more importantly, get something out of it 🤗

Your ideas and thoughts are always welcomed 🤗 Please jot them down in the comment section.

✍️ Written by

Vinh Le @vinhle95

👨🏻‍💻🤓🏋️‍🏸🎾🚀

Say Hello 👋 on:

🔗 Twitter | 🔗 LinkedIn | 🔗 Github