Pillar III: Probability & Info Theory // Zoth Math Academy

Pillar III: Probability & Info Theory Formal Systems

Logit distributions, Shannon entropy, and sampling

• Softmax temperature τ: 0.70
• Nucleus cutoff p: 0.90 (top-p)
• Shannon entropy H: 1.482 bits

Closed-Form Mathematical Formulation

P(wᵢ) = exp(zᵢ / τ) / Σ exp(zⱼ / τ)

Advanced Tensor Equation

H(X) = −Σ p(x) log₂ p(x)