A/B testing is a method to determine the best-performing variant by simultaneously running two versions of the same thing, in this case, your hiring workflow. You randomly assign applicants to A or B version of the workflow and at the end, see which one performed better. We recommend establishing a baseline with one workflow that has live applicants and performance measures first, before A/B testing.
To help you build your first workflow check out some of the recommendations in this article on building the Primary Workflow.
Why A/B Test?
You have built a hiring process, but is it performing the best it can? Many factors can contribute to the effectiveness of the workflow such as images, stage order, use of SMS, etc. The effectiveness of the workflow is also subjective. To some, the most important metric is the conversion rate while others may want to achieve faster completion time. The advantage of simultaneously running two versions of the same workflow is that you can isolate which factor contributes to what outcome.
Designing A/B Testing
In order for A/B testing to work, you can only test one factor at a time. That means you can only change one variable in version B, or you will not be able to tell which variable contributed to the performance change. Determine what you want to test. Here are some examples to help you get started.
Variable | Hypothesis |
SMS Messages | Increase applicant engagement and improve conversion rate and completion time. |
4 Follow-Up Messages | Decrease applicant drop-off. |
Different message verbiage | Increase applicant engagement and improve conversion rate. |
Testing a new integration | Decrease user supervision and inbound questions from applicants. |
Adjust the order of your stages | Increase applicant completion rate. |
Add a welcome message explaining the process on the Applicant Portal | Decrease support tickets and increase conversion rate. |
Setup
Create Version B Workflow
First, you'll create the version B workflow:
Clone the original workflow and make the updates to the variable you're testing.
Make sure to rename the Opening so you know which is version B.
Randomly Assign Applicants to Workflow
Next, you'll randomly assign applicants to the new workflow:
Go to the Company Settings page and under General Company settings, check the box for Generate a random integer between 0 and 100 upon creation under the Applicants section. Click Update Applicant Setting.
Applicants will now have a Data Key titled "Rand" added to their Applicant Profile.
In order to use this new Data Key in a Rule stage, you must create a new Data Collection stage with this Data Key (we will be deleting the stage shortly after its creation). Once your new Data Collection stage is added, add a custom question (any question type) to it.
Assign "rand" as the Data Key.
Create a test applicant and place them in this new Data Collection stage.
Complete the rand Data Key-related question as the applicant. This Data Key will now be available for use in a Rule Stage.
Delete the Data Collection stage you just created.
Add a Rule stage to Version A's workflow to the top of the workflow to route applicants to the Version B Opening based on the random number assigned to them. We recommend doing a rule that filters those who are greater than 50, that way it's split down the middle.
Make sure to make note of your changes and when you made the change.
Use Fountain's Analytics page to view conversions between the funnels.