Writing The Unity Expert Programmer Certification Exam

THE CHALLENGE

• Two complete new exams needed to be created in only a few weeks
• Exam writing was divided into three phases: planning, creation, and review
• The user experience (tester experience) was one of the highest priorities. The questions needed to be objective, yet engage critical thinking. Previous exams suffered from subjective responses (correct answers) and negative feedback from the developer community.

The challenge at hand was to create two new exams in a very short period of time. The contracted test writing team would integrate approximately 60 questions each, meaning a total of 120 questions needed to be written. Additionally, 10 reserve questions were set aside as a good measure, for a grand total of 130 new questions to be created.The exam needed to be written in 1 week, with a review period of 2 weeks in order to ensure the questions were sufficient. Additional review and user testing would then take place over a longer period of time after the final review by the Unity team. As the goal of the exam is to replace a technical interview for individuals seeking lead programming positions, the test needed to meet a certain quantitative standard, which meant expectations were high considering the relatively short timeframe for development.

The exam writing was broken into 3 categories: planning, creation, and review. The planning phase determined the topics that needed to be covered on the exam. The creation phase consisted of the question writing and lasted 1 week. It was scheduled to take place in the Unity office in downtown San Francisco. Given the impending COVID-19 restrictions, the session was conducted remotely at the last minute, which threw an additional complication into the process. Finally, the review phase, which would last for 2 weeks, was used to determine which questions would make the final exam and which questions needed to be modified or removed altogether.

‍

THE PROCESS

• Interview questions were used as an initial reference
• Working with time limits and the multiple-choice format was crucial
• Questions regarding programming did not require programming, given testers would not have access to an IDE (integrated development environment).
• We focused on the user experience by designing test questions around real world scenarios
• The most notable challenge was creating questions that held the caliber of a technical interview, while also adhering to the multiple-choice format.

KXR’s initial strategy was to use interview questions as a reference. These interview questions were culled from our own hiring process, as well as our individual experiences in the past being hired for previous positions and projects. Many of the questions that KXR created for this exam stemmed from our real world experience and the challenges we faced working on Unity development projects.

Another strategy KXR used was to reference a hypothetical candidate who would be just on the cusp of passing this exam, someone with about 2-4 years of Unity experience. Keeping this candidate in mind was helpful while writing questions, because the ideal question for the exam is one that challenges the expertise of this type of candidate, but is solvable by a programmer with a bit more experience. The content also needed to cover a range of topics, as well as the breadth of a specific topic, meaning no two questions should be too similar.

How did we focus on the user experience? For one, test questions need to be challenging, but not too lengthy. This balance was necessary so that each question could be answered in a period of 1-3 minutes by the programmer. The incorrect answers (called distractors) also needed to be feasible answers. Without proper care, multiple choice tests can be written in a way that a good test taker can pass the exam without any knowledge of the content. The length, structure, and phrasing need to be taken into account for each distractor in order to create an effective multiple choice question. Care was also taken to ensure questions were phrased as objectively as possible. The test will be given in multiple languages, so localization was a very important consideration as well. A few of the content creators were multilingual, which allowed the team to ensure translations would be accurate.

All test questions were designed around real world scenarios, and require the test taker to think critically about all of the provided information. Writing the test in this way removes problematic recall-based questions, which are better tests of memorization than problem solving. Creating test questions that were articulated clearly was of the utmost importance, to ensure test takers could accurately answer the questions in a timely manner.

The most notable challenge was creating questions that held the caliber of a technical interview, while also adhering to the multiple-choice format. The limitations of a multiple-choice test as a format for evaluating programming skill definitely posed some potential issues we had to address. Typically, technical interview questions rely on a dialogue between the interviewer and interviewee, as well as open-ended questions to assess a candidate’s thought process. Because the questions needed to fit a multiple-choice format and be answered quickly, the programmer being evaluated can’t actually do any real programming during the test. Thus, the questions needed to evaluate programming knowledge through testing a higher-level understanding of programming and development structures and concepts.

‍

‍

RESULTS

• Successful creation of two new certification exams
• KXR represented the training and educational sectors of Unity programming
• Unity will release the certificate in late 2020 or early 2021

A successful delivery.  The Unity team now has two new versions of their Expert Certification Exam.

KXR was one part of the team of a dozen other professional developers recruited to create this exam. Many of the other developers recruited solely focused on gaming, so it was rewarding for us to represent the training and education aspects of Unity programming.

Unity will be releasing this certificate at the end of 2020 or beginning of 2021. Thank you Unity for the opportunity to work on this project, and it was a pleasure working with the other Unity programmers who helped create the test!