AI Product Management
Not a fourth stage of the loop — a set of problems that cuts across all three. Whether to use a model at all, how to tell whether it works, and what it costs to run.
Start by asking if you should
Most "add AI to this" requests arrive as a board slide rather than a customer problem. A decision table is often faster, cheaper and testable.
Design for being wrong
The teams shipping well-received AI features are not the ones with better models — everyone has the same models. They designed for the 15% where it fails.
Evals are a product job
Deciding what good output looks like encodes tone, risk appetite and user need. Delegate the rubric and you have delegated the product.
Guides in this track
Common questions
Do I need to understand the technology?
Enough to reason about trade-offs, not enough to build it. Know that cost scales with context length, that output is non-deterministic, that retrieval and fine-tuning solve different problems, and that latency is measured in seconds. You do not need to know how attention works.
How do I know whether our AI feature is any good?
Not from adoption or engagement, which measure usage rather than quality. You need an eval suite: a fixed set of representative inputs, a rubric scoring several dimensions independently, and a weekly human-reviewed sample. Edit rate is the best single production proxy.
Everyone is shipping AI features. Are we behind?
Possibly, but shipping a bad one is worse than shipping none. A confidently wrong answer in front of your largest customer costs more trust than a missing feature does. The teams that look fast usually spent several unglamorous weeks on evaluation first.