The Horoscope Test

The pipeline was straightforward on paper: take tagged ideas from the vault, score them, pick the best one, generate a draft, check the quality, publish. Fully automated.

What actually happened was a debugging session.

First problem: macOS’s built-in sed behaves differently than the GNU version. Commands that worked in tests failed silently in the script. Took two debugging passes to notice the difference.

Second problem: set -e exits the script if any command returns a non-zero exit code. grep returns 1 when it finds no matches. So the script was dying silently every time a search came up empty, which looked like success — nothing ran, nothing failed visibly.

Third problem: Claude meta-commentary was leaking into the drafts. The AI assistant narrating its own process mid-draft. “I’ll write this in the style of…” followed by the actual content. Fixed by adding a constraint that strips everything before the first heading.

Then there’s the quality gate I hadn’t planned for.

A lot of AI-generated content passes surface-level checks — no slop phrases, correct reading level, reasonable structure — but still says nothing specific. You could swap the topic and it would read fine. This is the horoscope problem: content that sounds plausible and applies to everyone, which means it’s useful to no one.

The fix was a scoring step that asks: could this sentence have been written about a different subject? A score of 10 means every sentence is specific to this exact situation. The first auto-generated post scored 8 out of 10 and went live.

The architecture ended up with a clear division: the local model does mechanical work (scoring, classification, formatting). Claude makes judgment calls (selection, voice matching, quality assessment). The local model is fast and cheap at pattern-matching; it’s not useful for knowing when something is specific enough to be worth reading.

The pipeline runs on demand now. Whether it runs well is a question the horoscope test answers each time.