If caster level is less than max caster level, then the penalty = 1.0.
If caster level is at or greater than max caster level, then the penalty = (22 + max level - caster level) / 20.
The penalty is capped at 0. Max caster level per spell is also pretty inconsistent, but it's around 4 levels higher than the level at which you get the spell.
Basically, once the caster is 3 levels higher than the max cast level of the spell, the standard coefficient of the spell is multiplied by that penalty (losing 5% of the coefficient per level), with the coefficient reaching zero when the player is 22 levels higher than the max cast level.

Basically, starting around 7 levels after you learn a spell of a certain rank...

--CORE: Factor for downranking, factor for spells under level 20
calculation.penaltyMod = 1
if spell.spellLevel then
if (spell.spellLevel < 20) and not baseSpell.NoLowLevelPenalty then
calculation.penaltyMod = 1 - ((20 - spell.spellLevel) * 0.0375)
end
if not (spell.NoDownrank or baseSpell.NoDownrank) then
local mod = math_min(1,math_max(0,(22 + spell.spellLevel + (spell.Downrank or baseSpell.Downrank or 5) - playerLevel) * 0.05))
if mod < 1 then
calculation.penaltyMod = calculation.penaltyMod * mod
calculation.downrank = true
end
end

Я пока вижу только слова: "раньше было так, сейчас эдак."
Ни одного расчета как должно быть и как есть.