I provide an interpretation of Wittgenstein's much criticised remarks on Gödel's First Incompleteness Theorem in a paraconsistent framework: in taking Gödel's proof as a paradoxical derivation, Wittgenstein was consequent upon his deliberate rejection of the standard distinction between theory and metatheory. The reasoning behind the proof of the truth of the Gödel sentence is then performed within the formal system itself, which turns out to be inconsistent. I show that the model-theoretic features of paraconsistent arithmetics match with many intuitions underlying Wittgenstein's philosophy of mathematics, such as its strict finitism and the insistence on the decidability of any mathematical question.