After 18 months and ~800 incidents, our Rootly classification fields were mostly empty, no one fills in six drop downs at 2am. This post walks through how Tilt built an automated incident classifier: a scheduled .NET pipeline that scrapes the Slack channel, sends the conversation to Claude, and writes back the cause, vendor, component, and team, while quietly backing off any field a human corrected.