To Explore The Strange New World Beyond Data Distribution: System Behavior, Causality Tax, and Non-causal Base Model

Abstract: We show that the causality of language models may not be necessary nor optimal when system behavior is incorporated as a first-principle Bayesian feature. We propose the SBD framework, which incorporates system behavior as an irreducible component of the evidence lower bound and reveals a Causality Tax: causality can emerge as a suboptimal approximation with an additional structural error. We validate the predicted impact of system behavior through implicit measurements, theoretical-bound-guided controls, and Neural Tangent Kernel evaluations. We also construct Green Shell, a non-causal variational family that replaces the sequential dependency chain of system-behavior components with a divide-and-conquer partition. Its NTK spectra achieve tighter error bounds and stronger generalization in the lazy-training regime.