Karpathy defines Software 2.0 as a paradigm where the core of programming shifts from writing code to organizing datasets, designing objective functions, and training neural networks.