Why product development is difficult

10 min readDec 13, 2021

Product development is frequently done based on intuition and trial-and-error. Using optimization and mathematical models can assist the subject-matter expert to speed this up process, and even mostly automate it.

But, there are number of fascinating challenges to solve before getting there.

What is (optimization-based) product development?

A specified set of properties (targets, or outputs) is first identified. For example: your product must work at a temperature of at most 120°C; and it must have a weight of 7 grams for a fixed volume. We have 2 targets in this example: temperature and weight.

The aim is to make a product with these 2 properties by specifying 3 major groups of parameters:

Raw materials to use (selection step)
The amounts of these materials (relative ratios)
The steps used to combine the materials (process conditions)

You will not get it right the first time. So product development is iterative. For example, you might meet the weight constraint, but not the temperature constraint.

In each iteration you will update the (1) materials selected, and/or (2) the ratios and/or (3) the process conditions used.

To continue the above example, in each iteration you try using other materials, in differing ratios, and/or with different conditions (e.g. you change how you blend the materials, mixing speed, duration of blend, etc).

Using an optimization-based approach means you are guided by two types of models of the system which try to capture reality. A forward prediction model (inputs → outputs), to predict the targets for a given set of values in groups 1, 2 and 3 above. The inverted, or reverse model (outputs → inputs), also called a prescriptive model, may be built to try and predict the input conditions (experiments that we can run later), that give the desired target values. The inverse model doesn’t always exist — more on that later.

Each iteration can be viewed as a DBTL cycle, which is popular in the biotechnology area for developing products. It has seen immense use recently, especially in the area of autonomous process optimization.

Design: specify values of the 3 parameter groups above.
Build. You build your product by running one, or more experiments, based on the design(s) you came up with.
Test. After the experiment(s), you measure, or test, the product. (Note that in some interpretations, the build phase is getting your experiments ready to go, and the test phase is where you run the experiments and measure results).
Learn. How close are you to achieving the desired specifications? You will not get it right first time. So what have you learned? How accurate were your model’s predictions? What will you do differently in the next DBTL cycle?

Note: you might not explicitly use models. Even when develop based on intuition, process knowledge, or “by-eye” based on plots of results, you are still using some implicit model of the system, even if you don’t represent that model mathematically.

So why is this a tough problem?

There are 6 major areas where issues come up: the target specifications themselves, each of the D, B, T and L steps, and also managing the iterative cycle itself. Let’s look at each of these 6 in turn.

1. Problems related to the specifications

In practice, the targets or outputs are often correlated. For example, as one of them increases, so does the other. But the specifications might assume they are independent, or have constraints on one target, while the other is free to float. The correlation structure must therefore be taken into account.

Product design can also be used for updating a product. Your company sells product A, but new government legislation requires you to meet a new constraint (e.g. on pollution). How do add that to the mix, while keeping the other specifications as they were?

Another issue: how do you handle asymmetric specifications? What if the weight should ideally be above 4 grams (a soft specification)? But it definitely can not exceed 7 grams (a hard spec). How to incorporate that asymmetry mathematically?

Each DBTL cycle can be solved using optimization. These problems have an objective function. So how do you mathematically capture the product specifications (targets) in the optimization?

Imagine you have 5 properties specified. One way is to use the deviation of your current best experiment from the specification. But do you use the sum of squares of the deviations? The absolute value sum? Do all 5 outputs weigh equally, or is there a ranking; and if so, how do you represent that? How do you use scaling to avoid targets with large numeric values from swamping small ones?
How do you incorporate costs or profits into the objective? It is often recommended to cast these problem into one where you maximize profit. Easier said than done — it can be phenomenally difficult to estimate the income, and all costs. Do you know the production costs? Emissions costs? Labour requirements? Cost of deviations from specification? What is the market is willing to pay? How to capture the complexity of all manufacturing steps? How to account for the number of raw materials and how to source them all at large scale?
It is one thing to produce a product, but will it be robust? How will it work in the hands of your customers and not in the hands of your quality control lab? If we optimize only for “closeness to product specification" we can likely find a solution, but one which isn’t optimal, nor robust at large scale.

Let’s now look at the 4 elements of the DBTL cycle.

2. Problems related to the Design step

For each iteration we can specify either n=1 or n>1 new experiments to augment our knowledge. In modern robotic systems the number of n parallel experiments to run can be quite high. How to select these points, and their replicates?
The first DBTL iteration has little knowledge/data to work with. How to get effective search directions right at the start, on which materials and conditions to select?
Step size: imagine an input x=0.5; it doesn’t make sense to use x = 0.499 at the next iteration, but x=0.2 may be too large a step. So where is the balance? And how to incorporate this knowledge so the entire DBTL cycle can be automated, or even have a complete autonomous system that self-directs sequential DBTL iterations?
Location of points to benefit the predictive model. You can guess that placing an input at x=0.2, 0.3 and 0.4 would provide a good starting model that indicates which direction to move in to achieve a target. But do other placements benefit the model fitting and subsequent predictions? Now what about the multivariate case (combinatorially large!) with multiple inputs?
When should you bring in new materials to test in the next iteration, and how do you select materials to select? This can grow into a large combinatorial problem, as we potentially have a large database of ingredients to select from, sometimes well over 100 variants from the same family. Rather, can we cast the selection into a numeric continuum, and if so, how? In other words, don’t look at 100 discrete plastics, but rank them on a scale of thermal stability, and that scale then becomes a new, single continuous variable.
The presence of a substance, even by a small amount may have an on-off switch effect. Leaving that substance out versus including it leads to strong nonlinearities. How to model and account for this?
How do you account for (potentially nonlinear) interactions between raw materials? For example: two additives are possible. The one interacts with another raw material in an undesirable direction, and the other non-interacting additive, however, is expensive.
Some of the process conditions (group 3 parameters) are discrete. For example: stirrer speed may be off, low, and high. This makes mathematical modelling more complex.
In manufacturing processes the order matters. How to represent and model these manufacturing steps?
Multiplicity of solutions: in the (forward) predictive model there are often multiple ways to the achieve the same output. Reversing that prediction model around to a prescriptive model is an underdetermined problem, therefore a unique prescriptive model doesn’t exist. For example, when solving x + y = 4, there are infinite combinations of the x and y input values that equal the output value of 4. We have this situation whenever there are more manipulated variables than target variables, which is almost always the case in any product development.

3. Problems related to the Build step

How consistent can you run your experiments? Using the exact same inputs, how repeatable are you? Large errors, or lack of control at this step is disastrous, since we cannot be sure if an improvement shows up above the noise. With a noisy system: when you go to new operating region, is the improvement noise, or is it just the same as before?
Even if you are as consistent as you possibly can be, there is inevitably experimental variation within and especially between cycles (blocking). How do you add references and controls in each iteration? How do you correct for these blocked effects?
Is your experimental system representative of the customer’s usage of your product? Can you mimic your customers’ diverse ways of using the product?

4. Problems related to the Test step

When you measure the target outputs, how reproducible are you? Sometimes we see large shifts from iteration to iteration, caused by other (uncontrollable) parameters. Can you correct for (or better, eliminate) that?
In some complex systems we see interactions in the outputs: for example, the viscosity measurement is influenced by the sample’s pH. Accounting for these interactions and correlations is essential.
To reduce cost of development, sometimes a substitute measurement is required, to allow more frequent experiments. For example, you cannot bring in your taste-testing panel to assess every experiment. In other systems the lab measurements take days to complete. How can substitute measurements and/or soft-sensors be used to accelerate your development cycles?

5. Problems related to the Learn step

How do you learn from the experimental test results? Which direction do you go for the next iteration? Which conditions are promising and worth exploring? Or do you stay more or less in the neighbourhood when you are close to the target and exploit the existing area?
About this explore-exploit strategy: how does it change as you iterate through the DBTL cycles several times? Does your strategy adapt (first more exploration, later more exploitation) as you go along, and if so, how do you automate it?
How can you, from the new test results, eliminate raw materials, and thereby reduce your search space for subsequent iterations?
Which model types do you use for the forward (input → output) predictive models? Gaussian processes, simpler linear/quadratic curves, splines, and so on. Can you either invert these models, or otherwise build a prescriptive model (outputs → inputs) from the data?
For these models: when do you update the model and/or throw away old data, or data far away from the targets? When is a data point an outlier, and not a promising lead? (How many times do we read stories of that one data point which turned out to be the insightful ah-ha moment).
Since your model doesn’t give exact predictions, how can you use the model’s uncertainty to help guide you for the next iteration?

6. Problems with the DBTL cycle itself

The iterations must stop at some point. The obvious case is when the goal is reached.

But what about cases when each iteration gives diminishing returns? Can you stop early and avoid costly extra experiments?
Is it identifiable when the goal is simply unattainable? When no feasible solution exists.
By its very definition, an optimal solution is always in a nonlinear region. A linear solution can be infinitely maximized or minimized, and no practical process allows this. So with a nonlinear optimum, how do we identify that have we reached that peak (or valley)? Might there be an even higher peak elsewhere?
Storage and representation of knowledge. How do you logically name and identify each experiment? Store the ratios and recipes, and in which units? Are you consistent across all cycles? How are missing data within experiments handled and modelled? Do you also store covariates? (e.g. the ambient humidity and temperature, the names of the operators, the amounts and lot numbers of materials which are constant in all experiments). These “constants" and/or uncontrolled parameters are often not stored, till one day it is suddenly realized they have impact; but by then the original values are forgotten. How do you represent all series and parallel steps in the manufacturing of the product, and record the process conditions?
The model maintenance step. Model versioning, knowing which input data were used (and their provenance), how these data were preprocessed, model fitting done; i.e. a full audit trail. And model updates: when does one have a signal that a predictive or prescriptive model is not performing well anymore (e.g. drift or shift of inputs) and should therefore be updated.

Phew!

The purpose of this post was not to identify probable solutions. It was to simply capture many of the issues. Not all issues appear in every product development problem.

In future posts I plan to outline how tools such as design of experiments, response surface methods (RSM), multivariate models, Bayesian optimization and “regular” optimization are used to help answer these issues in the product design problem.

Have I missed any issues that you have encountered?

I’d love to hear from you on how you go about addressing any of the points above. I’ll keep this article up to date and give credit to others’ inputs.

Thanks to Willi Gottstein for comments on a draft version of this article.