Notes on The Bitter Lesson by Dr. Sutton
Dr. Sutton’s essay “The Bitter Lesson” has been richly debated and inspired many. In hindsight, it is scarily prophetic. Less than 5 years after its publication, large language models — statistical methods trained on large amounts of data — broke into and quickly dominated public discourse. Language itself is a generalization of its underlying representation. To put thoughts into words, as I do now, is to ground ideas into the frame of vocabulary. Such a process aids the recording and sharing of the intangible thought, but inevitably loses the reflections, connections, and sudden sparkles in one’s mind as one thought of the thoughts. ...