An Insider’s View: My experience developing certification questions with AWS
Examine many modern IT job postings and you are likely to find prior experience with a public cloud provider listed among the requirements. With the future of many IT capabilities lying in cloud technology, employers want to ensure they are hiring the type of talent that will enable them to be successful. As the undoubted leader in the infrastructure-as-a-service (IaaS) and public cloud market, the demand for personnel with experience using Amazon Web Services (AWS) is at an all-time high.
As the popularity and adoption rate of its technology grew, AWS launched their Global Certification Program in April 2013. The goal was twofold:
- Allow an individual to gain visibility for their expertise and knowledge of designing, deploying and operating solutions on AWS
- Help employers identify skilled professionals to lead IT initiatives using AWS technologies.
Originally launched as a single exam, the ‘AWS Solutions Architect — Associate’ exam, the certification program has grown to include 6 role-based and 2 speciality exams across multiple domains. The following graphic (see Image 1), taken from AWS’s certification website, illustrates the breadth of certification paths available:
As a daily user of AWS and someone who holds the five available Associate and Professional level certifications, this is a topic which I have a vested interested in. As I have worked towards and completed these examinations over the past 16 months, I’ve often wondered who exactly was responsible for the development of testing materials. At times, this wonder has been borne out of frustration as a result of what I felt were vaguely worded or outdated questions. On other occasions, it’s been out of genuine curiosity: How could AWS possibly expect to keep certification materials up-to-date given the mind boggling release rate of new features and services?
To my surprise, I was recently given an opportunity to try and answer some of my questions. As an AWS Partner Network (APN) Ambassador, I was invited by AWS to participate in a three day certification question development workshop for one of their professional level certifications. While there are certain details I can’t divulge, below I will highlight my experience working with AWS at this workshop in addition to my takeaways from this invaluable opportunity.
To help break the ice, the workshop began with introductions around the room. Other than a fellow Slalom employee from our Seattle office, the remainder of the nine participants were Solutions Architects and subject matter experts from a wide variety of AWS teams around the world. Our two workshop facilitators discussed why we were there and set expectations for the coming days. The overarching message was clear; AWS is committed to improving and modernizing their certification exams to accurately represent their platform and qualify those who use it. It was evident that AWS has taken feedback on their certifications seriously, prompting a focused effort to improve the overall candidate experience. Following our introduction and level-setting discussion, the remainder of the three day workshop was spent iterating between sessions of individual item writing and group item review.
Guidelines were provided around high-level content areas to focus on, but we were left to our own devices when writing test questions. An interesting challenge, I soon discovered that generating clear, concise test questions for a professional AWS certification is not an easy task. You are forced to think both creatively and logically to construct fictional test scenarios with a highly technical focus. Perhaps the most difficult part is formulating multiple choice answers, making them equally attractive while ensuring that only one is definitively correct. Proof of AWS’s overall commitment to certification improvement and consistency, questions were expected to be accompanied with a clear rationale on their focus and evidence to support all answers provided. It wasn’t enough to simply identify an answer as correct; clear, factual evidence was required to detail why an answer was either correct or incorrect.
Unlike item creation, item review was a collaborative process conducted in small groups. Although all items were reviewed with the author remaining anonymous, the workshop facilitators had warned us not to take any criticism of questions we had authored personally. This warning soon made sense as each question was brought under intense scrutiny from all group members, including myself. However, I view this is a testament to the workshop participants and their willingness to help AWS provide clear, concise and technically accurate questions to exam candidates. Throughout the reviewal process, written feedback was provided to the author to help guide continued improvement when necessary. Treating it as a form of product development, utilizing a constant feedback loop and adopting a ‘fail fast’ mentality allowed for the rapid improvement of question materials during the workshop. On a personal level, I found that I was able to utilize feedback to rapidly improve the quality of the items I had developed. While this review process may sound exhaustive, this internal group review is only the first of many steps that a proposed question will be subjected to if it’s deemed a good exam candidate. This level of scrutiny and detailed review again illustrates AWS’s commitment to driving improvements throughout their certification program.
Attending the certification item development workshop highlighted the commitment AWS has made to improving and modernizing their entire certification examination process. Developing materials for a professional level examination is a challenging task, but the depth of review and quality of feedback received made it a particularly rewarding experience. Looking forward, I know that a continued focus on clarity, consistency, and technical accuracy will enable AWS to provide a certification program that their industry-leading platform deserves.
