Unity Technologies, the creator of the wildly popular Unity 3D engine, contracted KairosXR (KXR) to re-write their Unity certification exam for aspiring developers. Unity 3D is the leading tool for developing AR and VR experiences, as well as most 2D and 3D mobile games. Unity Technologies is also a longtime client of KXR, having become familiar with KXR’s development work (educational content, workshops and training, and developed Unity applications) through our collaboration with their enterprise training teams. Previously, KXR conducted an onsite training workshop for one of Unity’s enterprise clients, which was designed to prepare the team for the expert certification exam. Impressed by KXR’s improvements to the existing curricula and demo training application, Unity contracted KXR to write their new Expert Programmer Certification Exam.
The Expert Programmer Certification Exam is the highest Unity programming certification that exists, and it was originally designed for gameplay programming. As Unity’s user base and market grew well beyond traditional game design, Unity decided to recreate the exam to reflect this evolution. The goal of the exam is to replace a technical interview for a lead Unity developer. In other words, if a developer has passed this exam, a potential employer should feel confident that the developer is technically qualified for the position.
The project consisted of writing technical questions for the expert-level certification exam, and Unity needed enough questions to create 2 complete versions of the exam. We were excited and honored to be selected to participate, as the Unity team recruited top developers from around the world.
Unity is working to build out their certification tiers for various levels of users. You can see their certifications here: https://certification.unity.com/
The challenge at hand was to create two new exams in a very short period of time. The contracted test writing team would integrate approximately 60 questions each, meaning a total of 120 questions needed to be written. Additionally, 10 reserve questions were set aside as a good measure, for a grand total of 130 new questions to be created.The exam needed to be written in 1 week, with a review period of 2 weeks in order to ensure the questions were sufficient. Additional review and user testing would then take place over a longer period of time after the final review by the Unity team. As the goal of the exam is to replace a technical interview for individuals seeking lead programming positions, the test needed to meet a certain quantitative standard, which meant expectations were high considering the relatively short timeframe for development.
The exam writing was broken into 3 categories: planning, creation, and review. The planning phase determined the topics that needed to be covered on the exam. The creation phase consisted of the question writing and lasted 1 week. It was scheduled to take place in the Unity office in downtown San Francisco. Given the impending COVID-19 restrictions, the session was conducted remotely at the last minute, which threw an additional complication into the process. Finally, the review phase, which would last for 2 weeks, was used to determine which questions would make the final exam and which questions needed to be modified or removed altogether.
KXR’s initial strategy was to use interview questions as a reference. These interview questions were culled from our own hiring process, as well as our individual experiences in the past being hired for previous positions and projects. Many of the questions that KXR created for this exam stemmed from our real world experience and the challenges we faced working on Unity development projects.
Another strategy KXR used was to reference a hypothetical candidate who would be just on the cusp of passing this exam, someone with about 2-4 years of Unity experience. Keeping this candidate in mind was helpful while writing questions, because the ideal question for the exam is one that challenges the expertise of this type of candidate, but is solvable by a programmer with a bit more experience. The content also needed to cover a range of topics, as well as the breadth of a specific topic, meaning no two questions should be too similar.
How did we focus on the user experience? For one, test questions need to be challenging, but not too lengthy. This balance was necessary so that each question could be answered in a period of 1-3 minutes by the programmer. The incorrect answers (called distractors) also needed to be feasible answers. Without proper care, multiple choice tests can be written in a way that a good test taker can pass the exam without any knowledge of the content. The length, structure, and phrasing need to be taken into account for each distractor in order to create an effective multiple choice question. Care was also taken to ensure questions were phrased as objectively as possible. The test will be given in multiple languages, so localization was a very important consideration as well. A few of the content creators were multilingual, which allowed the team to ensure translations would be accurate.
All test questions were designed around real world scenarios, and require the test taker to think critically about all of the provided information. Writing the test in this way removes problematic recall-based questions, which are better tests of memorization than problem solving. Creating test questions that were articulated clearly was of the utmost importance, to ensure test takers could accurately answer the questions in a timely manner.
The most notable challenge was creating questions that held the caliber of a technical interview, while also adhering to the multiple-choice format. The limitations of a multiple-choice test as a format for evaluating programming skill definitely posed some potential issues we had to address. Typically, technical interview questions rely on a dialogue between the interviewer and interviewee, as well as open-ended questions to assess a candidate’s thought process. Because the questions needed to fit a multiple-choice format and be answered quickly, the programmer being evaluated can’t actually do any real programming during the test. Thus, the questions needed to evaluate programming knowledge through testing a higher-level understanding of programming and development structures and concepts.
A successful delivery. The Unity team now has two new versions of their Expert Certification Exam.
KXR was one part of the team of a dozen other professional developers recruited to create this exam. Many of the other developers recruited solely focused on gaming, so it was rewarding for us to represent the training and education aspects of Unity programming.
Unity will be releasing this certificate at the end of 2020 or beginning of 2021. Thank you Unity for the opportunity to work on this project, and it was a pleasure working with the other Unity programmers who helped create the test!