Notes from Industry

You may miss useful knowledge if you are only doing A/B testing, or following the Change One Single Thing (COST) approach to experimentation. Alternatives that provide more insight into your process are shown, and the costs of obtaining that insight are discussed.


The COST (change one single thing) approach to experimentation has be shown, at least 100 years ago, to be an ineffective approach to experimentation. Yet it persists. Why? The A/B testing approach is a variant of COST, but with fewer drawbacks, though still inefficient in capturing all there is to learn. Let’s dive in by introducing common terminology to see why. If design of experiments is a familiar topic for you, please skip on to section 3.

Experiments can be applied wherever you might imagine, so the examples used below come from baking, learning a new language, the sciences…

Relying on R² to judge a regression model’s performance is misguided and misleading


R² can be calculated before even fitting a regression model, which doesn’t make sense then to use it for judging prediction ability. Also you get the same R² value if you flip the input and output around. Again, this is nonsensical for a prediction metric.

The intention of your regression model is the important factor for choosing an appropriate metric, and a suitable metric is probably not R².

This article explains which better alternatives exist: the standard error, confidence intervals and prediction intervals.

1. Why do we build linear regression models?

Historical data is collected and a relationship between the input 𝑥 and the output 𝑦 is…

You wouldn’t wake up on Wednesday and decide to start training 3 days in a row to run a 42km marathon on Saturday. Why do we treat our brains that way when learning new skills? Costly corporate training is invariably given minimal attention, leading to huge waste.


To eliminate this waste, consider 2 points.

Trust educational science, which has shown spread-out learning to be more effective than cramming in new concepts.

Secondly, business decisions are often based on return on investment (ROI). Training choices based on ROI will make you more accountable and force you to consider more effective approaches.

The usual state of affairs

Training your colleagues, or employees seems to be a checkbox item — something part of their development plan or performance review.

What is the manager to do when faced with a group of staff that need skills developed in a rapidly changing and digitizing corporate world?

Kevin Dunn

Kevin is an engineer & data scientist with 20+ years experience across a variety of industries. Still continually learning new ways to extract value from data.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store