Not all experiments succeed

Building on my gambler or scientist post…

One of the most rewarding parts of leadership is carrying the accountability for your team’s work. Accountability isn’t good or bad, it’s simply the responsibility for outcomes; the wins and the losses. On balance, it’s hugely rewarding. But it also means you’re the one delivering bad news to the business. Delays and experiments that don’t work out sit with you just as much as the successes.

If you’re experimenting and pushing boundaries, some things will get delayed or fail. Frankly, shit happens.

Ideally, you’re reporting on how something came in on budget with a clear ROI. But when the numbers start stacking against you, those conversations with the business get trickier. Starting with why and communicating early always helps. The temptation under pressure is to over-explain, but clarity and confidence matter more.

Sometimes, things still won’t work. In those moments you just need to “pay the cost to be the boss.” Take the learnings, shield the team, and do what’s right for the business. That’s the less glamorous side of management: the lonely moments where you dig deep, stay stoic, and keep moving forward.

As long as the team is learning and developing, experiments have value. What diminishes ROI and credibility is repeating the same mistakes.

Here are two examples from my own experience.

Delaying the merge

Towards the later stages of a major project we were ahead of schedule, working with the business to release a user merge with minimal customer impact. Everything was on track.

At that point I made the call to include a dependency that would significantly improve the UX. It wasn’t essential, but it felt like the right long-term choice for users and the product.

Unfortunately, as release approached, the dependency caused issues. Our deadlines slipped, and despite the team delivering their part, we now faced a delay.

We went all in to fix it, but the damage was done. The goodwill was gone, the business saw a failure, and pressure ramped up. My priority became setting a revised date we could guarantee and shielding the team from the additional pressure.

Part of that also meant making a difficult decision. It became clear that one person wasn’t a fit for the way we needed to work. Letting them go was hard, but it was the right call for the team, the business, and ultimately for the individual.

This felt like a personal failure to me. My management style is built on creating an environment where everyone can excel, and in this case, I couldn’t make that happen. Alongside the delay, it was another reminder that leadership comes with moments where your ideals collide with reality.

Protecting the team sometimes means removing friction, not just defending it from external pressure. I took the learning, stayed stoic, owned the bad news, and focused on doing what’s best for the business. We did eventually ship the user merge, but for me it was under a cloud. Not a happy time, but it made me more aware of the gap between my ideals and the realities of leadership.

Self-hosting images, replacing imgix

As our imgix renewal came up, costs had increased significantly. We ran a spike to explore alternatives. The team built a PoC combining open source tools, Cloudflare, and AWS. Early results suggested we could save 50% compared to renewal.

We tested it with a small % of users using Unleash. It worked technically, but real-world usage meant more AWS scaling and higher costs. The savings dropped to 15–20%. Disappointing!

However, this gave us leverage. Armed with a working alternative, I went back to imgix and negotiated a better deal, closing the gap to 10%. We also had a fallback option in case imgix went down. The PoC wasn’t adopted, but it paid off in reduced costs and stronger vendor resilience. It also made us appreciate imgix more.

Lessons learned (and still learning)

Not all experiments succeed. Sometimes they’re costly. Sometimes they hurt. But they always teach.

My philosophy as a leader is to create an environment where everyone can excel. In reality, not every experiment works, not every dependency pays off, and not every person thrives in the environment you build. That tension between ideals and reality is part of the job.

Leadership isn’t about avoiding those moments, it’s about how you handle them. Taking accountability. Protecting the team where you can. Making hard calls when you must. And always learning so you don’t fail the same way twice.

And, let’s never mention my failed live AI All Hands demo experiment…