End of 2023 Testing Policy Update
December 16, 2023
As we go into a new year of Flesh & Blood Judge testing we’re making some updates to our test policies to help make sure candidates aren’t getting stuck without a development path to grow and to make sure our exams are serving the needs of our community.
We’re doing three main things to make sure that happens:
- Implementing a new, more hands-on, retake policy starting after 3 failed attempts;
- More broadly offering printed exams as an option to judges for whom online testing isn’t the right fit;
- Creating better study tools to help guide candidates to success on the exams;
We also are going to implement an idea that we’ve talked about since certifications began — release exams for each set. This will allow us to make sure that judges are up to date on rules and policy changes along with making clear who is still an active judge.
Under current policy candidates can retake an exam as many times as needed to pass. Under the new policy after a 3rd failure of an exam candidates will need to engage with a mentor to go through a process to unlock additional attempts. This process will come with additional conditions to help guide candidates to success and ensure that all candidates meet the level requirements.
This policy will apply to the Level 1 and Level 2 exams but not the Level 0 exam.
We intend to roll out the exact details of the policy for each attempt as needed so we can adjust based on the results that we’re seeing. Today we’re launching the requirements for a 4th attempt and the additional process.
Attempts taken pre-JudgeHub won’t be counted and candidates who’ve already made 4 or more attempts will be counted as having made 3 as of this update.
Fourth Attempt Procedure
The procedure for unlocking the 4th attempt is designed to make sure that candidates have a mentor they’re working with and are doing things that will actively help them succeed on this attempt.
To start the process, a candidate will reach out to their Judge Community Representative Lead (JCRL). If the candidate is already working with a mentor, they should mention that in their message to the JCRL.
If the candidate is not already working with a mentor, the JCRL will help them identify potential mentors in their area, it is ultimately the responsibility of the candidate to make sure they have a mentor.
The JCRL may choose not to approve a specific person as a mentor if they do not think it is in the best interest of the candidate or the program.
Once the mentor has been identified the candidate and mentor will have to have a conversation that covers the following:
- The candidate’s results from previous exams. This is an exception to our normal exam confidentiality policy — the entire exam should be discussed to identify strengths and weaknesses.
- An assessment of how the candidate has grown and changed over the previous attempts.
- A set of specific, verifiable, actions the candidate will take to develop their understanding of their areas of weakness.
This will be recorded as a review on JudgeHub using a new review type that will be available to approved mentors — the Candidate Advancement Plan. These reviews (unlike normal reviews) will be visible to the JCRLs in addition to the subject and author.
After the review is submitted the candidate should begin to complete the tasks required and continue to develop their skills.
When the completion of the requirements has been verified by the mentor the mentor will reach out to the JCRL to say that they have a candidate ready to take the exam.
- If there are enough questions in the pool the candidate has not seen the exam can be scheduled immediately
- If there are not enough question in the pool the candidate has not seen the candidate will be added to a list to be informed when a new exam is ready (expected to be 4 months after the previous exam is released)
Once the exam is available the candidate and mentor should schedule when they wish to do the exam. For fourth attempts the exam must be done in person as a printed exam. Candidates still have their normal level of access to the internet during this exam. We recommend you schedule at least 5 minutes per question to make sure there’s time to complete the exam.
After the exam, the mentor should go over the entire exam (both questions answered correctly and incorrectly) with the candidate. The mentor will then enter the answers given into JudgeHub and once that’s complete the score will be official and the candidate can review the official explanations at their convenience.
If a 5th attempt is needed the steps will grow out of the results of the 4th exam. We expect most of those procedures to be more tailored to a given candidate.
A number of candidates have approached the exam team to ask about taking a printed exam as they do not do well with the digital interface or wish they could have paper to mark with their thoughts. We’ve had printed exams available for a while now upon request but since we didn’t advertise them we didn’t get many requests.
They function just like a normal exam and follow the normal exam rules just instead of using the website you’re marking your answers on a paper sheet. These must be given in person with a proctor assigned to collect the materials and enforce testing policies (i.e. no asking others for help).
Until we understand how many people wish to use this method we’re going to do it on an individual basis, to begin the process email me at exams@fabjud.ge and we’ll start setting it up for you. The default availability for paper exams will be at Tier 3 (Calling), Tier 4 (Pro Tours/Worlds), and Conference events but exceptions can be made. We need your request at least 7 days in advance of the start of the event you wish to test at to ensure we can get it there in time (we may be able to accommodate closer in requests but it will be subject to capacity).
Printed exams use the normal cooldowns and restrictions so if you’re waiting for a new set of questions to be released the paper exam process will not get you around the wait.
To date there’s been a very nebulous “study everything” mentality around the L1 and L2 exams, that’s not what the exam team intends because it makes the exams much more daunting than desired. To that end over the next few weeks we’re going to release a study guide for each of the exams that makes clear the content that can appear on them. These guides are designed to get you ready for the exam and to make clearer our expectations of knowledge at each step of the process.
Starting in early 2024 as we launch the next round of exam updates we’re also going to be retiring questions out of the pool to maintain 3 active exams at any given time. Those retired questions will be formed into practice exams that you can take. Questions will be retired out of that pool when they stop being relevant (instead of being updated as they are in live exams) so there may not be a full exam of questions in the practice pool.
Starting with Heavy Hitters we will be rolling out a release exam for each set going forward. This exam will consist of between 5 and 10 questions with a passing score of 80%. These exams will cover rules and policy changes from recent sets (not necessarily just from the newest set) and common interactions we’re seeing within the meta of both Classic Constructed and Blitz to ensure all judges are up to date on rulings.
The exam will be released no later than 6 weeks after the set releases (though the exam team is aiming for closer to 4 weeks).
How this Impacts Level 2 Judges
For Level 2 Judges, they will need to complete this exam before the next release exam is published or else they will be down leveled to Level 1.
- A former Level 2 Judge who is a single exam behind may return to Level 2 by completing the past due exam and the current exam then emailing the exam team (exams@fabjud.ge) to request reinstatement (you may only use this process once every two years, if you miss another exam during that time you will be treated as though you’ve missed multiple exams).
- A former Level 2 Judge who has missed 2 or more exams but has been de-leveled for less than a year may return to Level 2 by re-completing the Level 2 Checklist with the exception of the Level 2 Recommendation, and completing the most recent release exam. Once this is complete please email the exam team (exams@fabjud.ge) to request reinstatement.
- A former Level 2 Judge who has been de-leveled for more than a year must re-complete the entire Level 2 Checklist to return to Level 2. Judges in this situation may just use the standard checklist tool.
A judge who de-levels to Level 1 during a given year will not be de-leveled to 0 during that year’s end of year update unless they have missed all four of the year’s exams.
How this Impacts Level 1 Judges
For Level 1 Judges, they will need to complete at least half of these exams each year (we currently expect to have four exams next year so two must be completed, if there are an odd number of exams in a year this is half rounded up) to maintain their level. This will be checked shortly before the World Championship each year. We will issue a warning 30 days in advance of the deadline. Once the World Championship has passed we will reset any judge not completing this requirement to Level 0.
If a judge certifies for Level 1 after the second release exam is published in a given year, they will not be subject to deleveling that year (but we encourage them to complete the release exams anyway).
This is a big round of updates and hopefully improves the situation that a number of members of the program find themselves stuck between exams. We’re also finally bringing the last major component of testing that was promised shortly after the first Level 1 exams rolled out over two years ago. Everyone of us knows that this system is not perfect (nor can it ever be) so your feedback is welcomed and encouraged – you can either bring it up on the JudgeHub forums, the JudgeHub Discord or email me directly with feedback.
4 thoughts on “End of 2023 Testing Policy Update”