'Not stated' is a valid answer
Most proficiency tools always return a level. This one treats missing evidence as an outcome, so downstream analysis can tell 'the JD asks for expert Python' apart from 'the JD mentions Python'.
Skill Proficiency Engine
Tags every skill in a job description with one of five proficiency levels, backed by a quote from the JD, and answers 'not stated' instead of guessing.
- Role
- Built
- Where
- Zinnov (Draup)
- When
- 2026
- Access
- Work · private repo
Problem
Job descriptions list skills but rarely say how good someone needs to be at each one. An LLM asked to fill that gap will produce a confident level for every skill, including the ones the JD never mentions. The engine has to tag each mapped skill with one of five levels, Beginner to Master, and every level has to be backed by text in the JD.
01
Assemble, then check the cache
Inputs are rebuilt from confirmed fields only. A cache keyed on the JD, the skill, the model, and the prompt and rubric version skips anything already resolved.
02
Explicit evidence before any model
The engine finds the sentence that mentions the skill, then tries a years-of-experience pattern and a level-keyed phrase and verb lexicon. Evidence found this way skips the LLM entirely and is never overridden later.
03
Batched proposals, with a quote
Only unresolved skills reach the model, ten at a time. Every proposed level must come with a quote from the JD or it is rejected, and a failing batch is retried and then split in half down to single skills.
04
One place to standardize
Every raw signal, whether years, a phrase, a verb, or a model label, maps to one of five canonical levels in a single function.
05
Verify the quote twice
Every entry gets a mechanical check that its quote really appears in the JD. Entries that came from the model also get a narrow second call that asks only whether the quote supports the level.
06
Exactly one entry per skill
Every input skill ends with one entry recording how it was resolved: years, phrase, verb, or LLM-verified. A failed check becomes NOT_STATED rather than a guessed level.
Measured
- Canonical proficiency levels
- 5
- Anti-hallucination layers: quote check + verifier
- 2
- Skills per LLM batch, bisected on failure
- 10
- Tests
- 37
How it's verified
- Harness
- Deterministic extraction first, then batched model calls of ten with retry and bisect-on-failure, cached by JD, skill, model, and prompt version.
- Guardrails
- Every model-proposed level needs a quote from the JD, the quote must appear verbatim, and a second verifier call must agree; anything else becomes NOT_STATED.
- Testing
- 37 tests.
In depth
Two layers against hallucination
A quote is mandatory, the quote must be a substring of the JD, and a second model call must agree that the quote supports the level. Any failure ends in NOT_STATED.
Scope kept small on purpose
The first version made one call per skill. This version resolves what it can without a model, batches the rest, and defers cross-JD comparison to a separate later stage instead of building it in.
Decisions
- 01Deterministic evidence first, and the model can never override it.
- 02Bisect-on-failure batching, so one bad skill does not sink the other nine.
- 03A single function maps every signal to the five levels, so the rubric lives in one place.
Stack
Python · Gemini · Pydantic v2 structured output · pandas · pytest