Everyone was refining the premise. I tested the premise.
- The ask
- Improve the automated messages within the existing design.
- What I tested
- Whether the design's central assumption held at all.
A major online real estate marketplace sent automated text messaging to consumers under a human identity. Every variant on the table treated that identity as a given and PMs asked how to refine it. Discovery pointed to a more valuable question underneath: whether the premise was right in the first place.
So I kept the incremental variants and added one that tested the assumption directly — a mixed-effects online experiment with thousands of homebuyers, powered to separate the effect of the premise from the effect of the wording.
The study redirected the product to abandon the human identity rather than optimizing within it. The resulting approach shipped across both primary calls to action and remains in production. Effect sizes and business impact I'm glad to walk through in conversation.