• Case Study
NCARB Successfully Runs a Critical, 500-User Pilot Testing Program with Lineup
Founded in 1919, the National Council of Architectural Registration Boards (NCARB) is a nonprofit organization comprising the architectural licensing boards of all 50 U.S. states and five territories. The organization is responsible for developing and administering national programs for candidates pursuing architectural licensure, while also helping licensed architects expand their professional reach and advocating for the protection of the public's health, safety, and welfare within the architecture industry.
In 2022, NCARB decided to migrate the delivery of its professional licensure exam to a new test vendor that was charged with delivering the exam in two ways: via online proctoring and in person at numerous secure testing sites. To ensure a successful migration, NCARB chose to run a large-scale operational pilot program several months ahead of the new test vendor launch, a critical part of which included recruiting and managing a large group of volunteer testers who met specific criteria. NCARB needed a tool that could assemble, communicate with, and collect feedback from the pilot test group — and found the solution in Lineup.
“There was no precedent for the use of Lineup to accomplish what we were trying to do. But we found that the tools and capabilities of the platform fit our needs and goals.”
The Challenges
NCARB's migration to the new test vendor was a highly complex process involving the integration of multiple systems across different testing vendors — and the pilot program before the launch was equally complex.
NCARB needed to ensure the pilot program could properly load-test the new exam delivery system and that the pilot testers accurately represented its general candidate population. The organization's predefined selection criteria for the testers included candidate demographics, testing jurisdiction, urban/rural delivery, modality of desired testing (online proctored vs. a test center), preferred exam division, and the need for Americans with Disabilities Act (ADA) testing accommodations.
With a diverse group of pilot testers in place, NCARB then needed to rely on various communications and surveys to ensure all aspects of exam scheduling and delivery were functioning as expected. They also needed to be able to identify failures, pain points, opportunities, and other concerns that needed resolved before the system went live to all architectural licensure candidates across the country. Given these unique needs, the right tools were necessary for the success of the migration.
The Solution
NCARB already used Lineup to manage its volunteer committees and was familiar with the platform's features. The pilot program required a heavier lift from Lineup than NCARB was used to using, but they decided to try it with a platform they knew and trusted instead of implementing another software.
With Lineup in place, NCARB established the pilot program's workflow in the platform based on two different user groups: members of NCARB's exam team and the pilot testers.
The exam team began by importing into Lineup a list of more than 1,000 individuals who had previously expressed interest in the pilot testing program, and then used the available data in Lineup's database to filter and review the initial list. Then, they were able to cull the list down to meet the desired group number and makeup based on NCARB's predefined criteria.
Through Lineup, each selected individual was invited to the pilot test group and was asked to accept the terms and conditions of the program. Once the volunteer testers accepted their invitations and joined the program, which was tracked through Lineup, the exam team outlined the first steps of the pilot testing process to the testers — including providing directions on how the testers could migrate their testing records to the new system and schedule an exam with the new vendor. All communications to the pilot testers were drafted and sent through Lineup's communications tools.
After these steps were completed, the exam team sent a multi-question survey, developed and delivered in Lineup, to the testers group asking for feedback on the process. They analyzed and used the results to address any areas of concern with the exam migration and scheduling software.
Once all the pilot testers had taken their exam with the new vendor, the exam team sent a second Lineup survey that focused on the testing experience itself. Again, they used the results to address any concerns with both online and in-person exam delivery and modifications of the test check-in policies.
Finally, once the pilot testing program concluded, the exam team used Lineup to send a final "thank you" communication to all the testers involved in the program.
The Results
By utilizing Lineup for the pilot program, NCARB was able to:
Recruit a representative set of volunteer testers
An initial goal of the pilot program was to make sure the group of pilot testers was representative of the overall tester population to cover different human variables and circumstances that could arise with exam scheduling and delivery.
That meant going through an extensive pre-selection process that Lineup's database, customizable profiles, and filters made much easier. With Lineup, NCARB could pinpoint the exact individuals they were looking for who met the specified criteria.
As a result, NCARB was able to easily filter nearly 1,500 potential pilot testers down to its working group of 500. The exam team could feel confident that the cross section of individuals they selected gave them the breadth and depth of user insight they needed to ensure the migration to the new vendor was as watertight and worry-free as it needed to be ahead of such a major launch.Accomplish multiple tasks in a single platform
NCARB did not have to consider or implement any other software solutions to complete any of the multiple tasks associated with managing the pilot test group.
Instead, NCARB efficiently used the same Lineup platform to organize, filter, review, select, and assemble all the volunteer testers. Then — once those volunteers had signed onto the program — the organization continued using the same system to communicate with and survey the group, as well as analyze and evaluate the provided user feedback. All relevant information was collected and accessible in one place, bringing efficiency and convenience to the exam team in charge of overseeing the pilot testers.
“One of the biggest advantages of using Lineup, compared to other software, was that it eliminated the need for us to use multiple platforms to accomplish our goals... We could do everything we needed in a single, one-stop-shop software.”
Communicate and Engage Effectively with Testers
Managing such a large-scale program with hundreds of testers would present communication challenges in even the best of circumstances. But NCARB's exam team did not have to worry about how they would communicate with the testers since Lineup made it simple to reach and engage with each individual — all from the same system.
During the one-month pilot program, NCARB sent over 3,000 Lineup communications to the pilot testers, including invitations, instructions, general communications, and feedback surveys. The Lineup-generated surveys achieved a high response rate of 82%.
With such effective engagement and the ability to tap into valuable tester insight, NCARB achieved what it set out to with the pilot program — paving the way for a successful migration and launch of the new test vendor to better serve its architectural licensure candidates.
•Disclaimer: Lineup was initially developed for and is currently funded by NCARB.
Related Content
AILA Case Study
AILA Increases Volunteer Interest by 30% and Improves Volunteer Diversity with Lineup
BPS Case Study
Board of Pharmacy Specialties Efficiently Manages 2,800+ Volunteers with Lineup