A/B-testing job ads: using data to find out what really brings applications

Job ads are almost everywhere written by gut feeling and never touched again. Yet the ad is the first and often only point of contact with potential applicants - and small differences in title, intro or tone can double or halve the number of applications. Without a test, you guess which version works. With a simple A/B test, you know.

A/B testing sounds like a marketing department and a statistics degree, but at its core it's simple: you run two versions of the same role in parallel that differ in exactly one thing, and after a fair runtime you look at which brought more qualified applications. Any small team can do that.

The most common mistake is changing too much at once. If you simultaneously change title, intro and salary signal and variant B wins, you don't know what worked - maybe it was the title and the salary signal even hurt. A clean A/B test changes exactly one variable. Everything else stays identical. Only then is the result interpretable.

If you have several ideas, test them one after another, not at once. First the title, then - with the winning title - the intro, then the salary signal. That takes longer, but each test gives you a clear, transferable insight you can apply to the next ad.

Not every variable is equally effective. The title is almost always the biggest lever, because it decides whether someone opens the ad at all - 'Senior Backend Engineer (f/m/d)' versus 'Backend pro for our product team' can be the difference between found and overlooked. The second influential variable is the salary signal: a concrete range often pulls more and more suitable applications than 'attractive salary'.

Other worthwhile tests: the first two sentences of the intro (task first or company first?), the length of the requirements list (a shorter must-have list often lowers the drop-off rate), and the call-to-action at the end. What's rarely worth testing are cosmetic things like font size or image colours - they barely move the application count.

An A/B test needs enough data to be meaningful. If each variant gets only five views, every result is chance. As a rule of thumb for small teams: run the test until each variant has gathered a two-digit number of applications or at least a solid three-digit number of views. For very rare roles with little traffic, A/B testing is hard - then it's more worthwhile to learn cleanly from each individual ad.

And measure the right metric. Clicks and views are tempting because they're big and early - but an ad with many clicks and many unsuitable applications has lost, not won. The metric that counts is the number of qualified applications that make it to interview. A good A/B test optimises for quality, not noise.

The practical problem with A/B testing isn't the concept but clean tracking: which application came via which variant? Whoever maintains that by hand in a spreadsheet loses the overview after two weeks. You need a pipeline that records the source and ad variant per application so you can compare honestly at the end.

In KI BMS you can publish two parallel roles with different titles or texts, each writes into the pipeline via its own public form, and the source is recorded per application. The funnel analysis then shows which variant brought not just more applications but more applications with real conversion to interview. That way you test with data rather than gut feeling - and every insight makes the next ad better.