“The Examination is outdated!”
Have you heard that? Thought it? Said it? All of the above? We have too. “We” being the Universal Accreditation Board (UAB); and we have begun the very lengthy process to significantly update the Examination to ensure that all questions are current, relevant to the profession, accurate and valid. I’ll walk you through the process of how we are doing that, but first let me dispel the misconception that the Examination hasn’t been updated in years. We actually update the Examination in small ways throughout the year. More on that in a minute.
Let me start with the framework for the questions. We have a document called the blueprint. The blueprint lists each of the KSAs — the areas of knowledge, skills and abilities — that a candidate needs to successfully pass the Examination. And each of the KSAs has specific learning objectives — specific statements about what we are going to ask a candidate in this area.
For example, under the current KSA of Researching, Planning, Implementing and Evaluating Programs (RPIE), one of the learning objectives is Audience Identification and Communication. What we are trying to determine here is if the candidate can identify appropriate audiences and the opinions, beliefs, attitudes, cultures and values of each, and if the candidate can prioritize and properly sequence communications to different audiences.
Under the current KSA of Ethics and Law, one of the learning objectives is Integrity. Here the questions are going to be written to determine if the candidate recognizes and deals professionally with ethical and legal issues.
The current blueprint has 10 KSAs and a total of 43 objectives. Each objective has approximately three to five questions on the Examination. The proposed changes reduce the number of KSAs to six, and once approved, the number of objectives also could be reduced; however, the number of questions on the Examination will fluctuate very little, if at all.
Once we know what areas are going to be tested, we begin the process of item-writing, literally soliciting potential Examination questions. We seek input from subject matter experts and invite them to submit potential questions. In the past, questions were submitted by “the usual suspects” — current and former UAB members, those who had participated in the past, and Accredited friends and colleagues that we persuaded into volunteering a few hours of their time. In an effort to expand this pool of usual suspects, we’ve put a link on the Mentors/Teach section of the UAB website (PRAccreditation.org) that allows all practitioners to submit potential questions.
Item-writing sounds easier than it is, which perhaps is why the group willing to participate has remained small over the years. Each question needs to be supported through a reference in one of the books on our bookshelf. That not only validates the questions, but ensures that each question can be legally defended if we ever were challenged by someone who didn’t pass the Examination. To defend a question and its answer by saying, “Well, everyone knows that,” doesn’t quite cut it.
Once questions are written, they go through a technical review process. We assemble a panel of six to eight APRs to review each question and the answer options. We want to ensure that each question meets its objective; that it is clear and unambiguous, that it is challenging enough so that a person who is qualified to pass the Examination will answer it correctly, but a person who is not qualified to pass the Examination usually will not answer it right. We want to ensure that the correct answer is clearly right and the wrong answers are clearly wrong but not implausible. If a particular answer is too ridiculously incorrect, then no one will choose it. Likewise, providing choices with two answers that both could be right would not make a good question.
Each technical review session is booked for two hours, and we typically complete four to six questions per hour. (If this interests you, the volunteer line forms to the right.)
Once the questions are written and reviewed, they go into the testing cycle as beta questions. About a quarter of the Examination’s questions in any administration are in the beta phase. The candidate answers the question, but the answer — right or wrong — does not factor into his or her final score, because we don’t know yet if it is a statistically valid question. Of the 194 questions on the current Examination, there are 54 beta questions.
After at least 100 administrations of the Examination, our psychometrician conducts an in-service analysis of the entire Examination to provide statistics on how each question scored. The technical review panels are convened again to review the beta questions, but this time we have statistics behind them. For example, if few of the people who passed the Examination got a particular question correct, then that question is probably poorly structured or simply too hard; or if many of the candidates who failed the Examination got a particular question right, it’s probably too easy; or if most of the candidates who took the Examination did not select answer option D for a given question, that could be a bad distractor or a bad wrong answer. The technical review team might tweak the question a little bit — make the wrong answers more wrong, the right answers more clearly right, reword it entirely or throw out the question.
If a question is edited enough to influence how the next candidate would answer it (more than just making AP Style or similar corrections), it goes back into cycle again as a beta question. If it tests well, it can become a scored question.
This process of statistical validation occurs after at least 100 individual administrations of the Examination for all questions, not just the betas. A question that could have tested well for years, and many 100-application cycles, could begin to show poor results, perhaps because it’s no longer relevant or the aspect of the profession that it addresses has changed to the point that the question no longer is accurate. That question would be flagged for the technical review and be subject to revision. This is what I referenced when I mentioned early on that we do review and update the Examination in small ways on an ongoing basis. The Examination is always in a state of review and revision and being updated in small ways.
So if the Examination is always in a state of review and revision, what’s the excitement about this year?
The excitement is that we are proposing fundamental changes to the KSAs for the first time. When the KSAs change, the objectives must be revised. When the objectives change, all the existing questions must be realigned with new objectives so that their statistics match. Some questions might not align with a new objective, so they would need to be eliminated. And there could be new objectives that don’t have enough questions in the bank, so they would need new questions.
In addition to this realignment, the UAB’s Examination work group conducted an exhaustive review last summer of all of the scored questions to determine if they are current, relevant and accurate. Many that have been testing well were determined to need updating or serious rewriting; and when we make changes to a question, we’re back to beta (see above).
We’re talking about a significant number of new questions being offered this year, and with considerably more than one-fourth of the Examination having new, untested questions, we’re no longer looking at the existing Examination. We’re looking at an entirely new Examination, which translates to a full-on beta.
So once we get to this point of having a full beta Examination, we submit that new Examination to the testing center, which needs 90 days to prepare it for testing. This is called the publishing phase. Once it’s been published, we need another cycle of 100 candidates to take the beta Examination. Unlike the “regular” Examination in which none of the candidates have earned their APR, we need about 50 already-Accredited candidates to take the beta Examination to help validate the questions. Once we’ve completed the 100-application cycle, our psychometrician conducts another in-service analysis, we convene new technical review panels and tweak the questions, and we republish. Again. In between these phases, we also set the cut score, which is the threshold at which we determine what will be a passing score.
During the beta-test phase, the current Examination still will be available for candidates to take. Once the beta Examination is ready to be republished as the new APR Examination, we will have a temporary blackout window in which the Examination will not be available. This likely will be in the fourth quarter of 2015.
There are considerable steps in the process, some of which are controlled by the UAB (item-writing, technical reviews) and some which are not (in-service analysis, publishing). We’ve set an ambitious goal of having all of this completed in 2015 so that a new Examination can be ready by January 2016. And this just covers the Examination itself. In order for candidates sitting for the new Examination to prepare adequately, we need to update our study guide, the online course, Accreditation chair resources, and so on …
Hopefully I’ve demystified at least a few aspects of the Examination for you and shed some light on why we can’t just put out a new Examination sooner than 2016. In the event I’ve gone further and inspired the writer/editor in you to want to be part of the item-writing or technical review processes, please let me know. The invitation is open, and the UAB welcomes you to participate in this process.
Kathleen M. Giery, APR, CPRC is the chair of the Universal Accreditation Board Examination Work Group. She can be contacted at gieryk [at] lifequest.ufl.edu.
Leave a Comment