When outages are good

A colleague in the Data Science team did a talk today about how an outage in a feature from a couple of weeks ago was a natural experiment. His started by saying that even though outages made most people sad, for him it was a great moment because it helped him validate the theory that ‘notifications led an increase in engagement across other parts of the product’ and use the outage as an experiment. In this particular case the experiment would typically be hard to run primarily because it would have material impact on product metrics (if the theory was true). This was a theory that was not often tested, in other words it wasn’t clear if there was a causal relationship between notifications and engagement and if so how much.

It got me thinking about how outages might lead to similar learnings –
1) If a feature has been broken for a month and no one noticed, is it really important?
2) If you had a site speed regression try to see if there was is a meaningful drop in any product metrics you care about, or in other words does site speed matter

I am sure there are other lessons to be learnt when outages occur and this presentation will remind me to keep my eyes open for them.