Passing the test
If you’ve ever taken a test to get a professional license or to demonstrate mastery to move up a level in your career, it is likely that the test has been validated. Castle Worldwide, which creates and delivers reliable tests, depends on SAS® Office Analytics for Midsize Business to collect and analyze test information for its products so it can be sure all tests are meeting exacting standards. SAS streamlined the company's analytics process, shaving off two to three weeks in test development time, helping Castle expand into industries with rapid rates of change, where both accuracy and speed are required to win contracts.
"Before SAS, our analytics process was in bits and pieces," said Andrew C. Dwyer, PhD, Senior Psychometrician at Castle Worldwide. "It took a few days for IT support. There was a lot of importing and exporting of data between multiple software programs. To repeat a result required enormous amounts of time and money to manually document our process. Knowing who did what is a legal imperative. Now, SAS documents our workflow so we can quickly replicate the process. That documentation boosts client confidence, helping us maintain our reputation for excellence in psychometrics, test delivery and scoring."
SAS documents our workflow so we can quickly replicate the process. That documentation boosts client confidence, helping us maintain our reputation for excellence in psychometrics, test delivery and scoring.
Getting the test right
Castle provides examination delivery, design and development services to certification and licensure organizations, government agencies, corporations, educational institutions, and trade and professional associations around the world. The stakes are high since these tests determine an individual's ability to earn a professional license, be promoted or retain a job. "Public safety is at stake, so our tests are built with a degree of rigor that is state of the art," explains Robert Pedigo, Vice President of Client Services at Castle Worldwide.
Castle had used a variety of statistical tools to analyze such issues as whether test items needed to be removed or to flag suspicious behavior of examinees. But the process, including the requisite documentation, was labor-intensive. Dwyer says his team needed to make a request to IT for data, restructure it to a usable format (often using Excel), run the analytics using a variety of analytic tools, then create professional-looking results tables in Excel, and copy and paste it into a Word document to send to the client. Castle considered other solutions that made that process more seamless but they often required purchasing too many separate modules for different types of analysis. The company also looked at using open-source software.
"R was out of the question – we couldn't guarantee a repeatable outcome given the changing nature of the open source program," Pedigo says. "SAS includes everything from analytical capabilities for innovating to highly visual reports that executives expect. SAS helps us speed the best possible workforce assessment and development programs to our clients."
Castle users can work within the familiar Microsoft Office environment (Word, Excel or PowerPoint) as they produce management reports and manipulate analytical results. Castle also uses SAS Analytics Pro for Midsize Business to access data from any source, manipulate and analyze the data, and present information in business maps, charts, plots and 3-D relationship graphs.
Ease of use and speed matter
One of the factors that Castle must consider in supporting its exams extends beyond the tests' validation; Castle must also be able to run analysis if there is a possibility of a security breach. "We have to work fast to determine if there are indicators that confirm or refute (security breach) questions. We become emergency workers. SAS helps us do it fast," Pedigo says.
The Castle psychometricians say they like working with SAS. SAS® Enterprise Guide® (part of the SAS package) allows new users to come up to speed on SAS quickly. The stored processes can be run by test developers and others quickly and easily. Dwyer says the solution gives Castle firepower in its competition with other test validation services vendors. "We're competing with two industry giants. SAS benefits our goals," Dwyer says.