post

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Communication

What burnout taught me: lessons from product managers

Product Management

Anthropic CEO makes bold AI claim while Workforce new product chief wants you to love software

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

Deep dive: Creating effective retention strategies

5 difficult conversations and how to have them

Microsoft lays off 3% of workforce while renegotiating partnership with OpenAI

Navigating the workplace as a young product manager

How Stack Overflow is competing with AI – Jody Bailey (CPTO, Stack Overflow)

Louron serves as the Editor at Mind the Product, bringing nearly a decade of experience in editorial positions across business and technology publications. For any editorial inquiries, you can connect with him on LinkedIn or Twitter.

Louron Pratt

<p>In this <span style="text-decoration: underline;"><a href="https://www.mindtheproduct.com/mtpcon/digital/americas/">#mtpcon Digital Americas</a></span> session, <span style="text-decoration: underline;"><a href="https://www.linkedin.com/in/arindone">Anthony Rindone</a></span>, product manager for experimentation at Split discusses how to make better product decisions through experimenting.</p>
<h2>Breaking metrics down</h2>
<p>Anthony says that experimenting with products is a humbling experience, if you succeed in 30% of the experiments you make, then you are in a world-class position, meaning that the number of times experiments you invest in turn out to fail. At Split, he explains that every change is a feature, and every feature is an experiment. He goes on to discuss three case studies from the experimentation trenches.</p>
<p>In his first example, he explains that he has encountered using the wrong overall evaluation criterion (OEC) to initiate experiments. A better approach to experimentation is to leverage a broader field of possibilities. By opening your range, you’re able to break things down and create sub-metrics and track what is specifically changing through that.</p>
<h2>Integrating multiple variations over time</h2>
<p>Throughout his second case study he talks about when introducing a new feature, it experiences a novelty effect. This can be misleading to how well received a new feature will be. If it’s a brand new addition it’s bound to receive more traction, but this reception may not necessarily last for long. By running multiple variations over four weeks, you can see what actual effect these experiments have. While running several variations, it’s important to sweat out every piece of detail.</p>
<p>“When you think about your whole application from end to end, there’s going to be some overlapping through new campaigns or feature roll-outs,” Anthony says, “improving your segmentation and being more precise with testing is key to tracking new variations effectively over time.”</p>
<h2>Have a specific hypothesis</h2>
<p>Finally, Anthony talks about his third example of having a vague hypothesis. The wider you cast your net, the more difficult it is to have a clear decision process or answer, he explains.</p>
<p>“If you don’t have a clear decision-making process, it creates loads of answers of a wide variety,” he says. Consequently, you end up setting yourself up for failure where you have two opposite conclusions. Both can’t be right. “The conclusion is normally decided by the highest-paid person in the room, which may not always be the right one.”</p>
<p>A hypothesis is more than a guess, it’s a foundational element on how to make decisions, he says. Build a falsifiable hypothesis upfront, identify the target segment at the start of your experiment. Doing so will provide an action plan for each outcome which everyone agrees on.</p>
<h2>To wrap up</h2>
<p>Anthony closes by providing some best practices with experimentation:</p>
<ol>
<li>Focus on the bigger picture</li>
<li>&#8230;But still, sweat the details</li>
<li>Having precise, falsifiable hypothesis’ are foundational needs to effective decisions-making</li>
</ol>
<h2>Explore more content like this</h2>
<p>Ordinarily, premium content such as this can only be accessed by Mind the Product members. Today, we’re offering up this talk to everyone in the community. If you like the content and want more like it, you can check out our <a href="https://www.mindtheproduct.com/membership/compare/" target="_blank" rel="noopener">Mind the Product membership plans</a>.</p>
<p>Still in need of inspiration? Browse even more <a href="https://www.mindtheproduct.com/category/user-testing/" target="_blank" rel="noopener">user testing</a> content or use our <a href="https://www.mindtheproduct.com/content-a-z/" target="_blank" rel="noopener">Content A-Z</a> to find specific topics of interest.</p>
<p>If you’re a Mind the Product member, you can also check out more <a href="https://www.mindtheproduct.com/tag/mtpcon/" target="_blank" rel="noopener">#mtpcon video content</a>.</p>


core/freeform

Hypothesis and Testing

The journey to product experimentation by Split

Progressive product development: release products that deliver by Split

Surveillance art, dying phones, and fake likes – Dries Depoorter at MTP Engage Hamburg

Understanding and executing effective research

What’s stopping teams from running experiments by Cameron Savage

How to be a better data-driven product manager

Product’s role in experimentation culture by Terry Lee

Stay Humble or Be Humbled — Case Studies from the Experimentation Trenches by Anthony Rindone