Martin C. Martin

Models break down when you exploit them. Examples:

• “Number of lines of debugged code” is a good measure of programmer productivity, until you tie programmer reward to it. The programmers write very verbose code.
• Stock price is a good measure of the performace of a company, unless you tie management’s reward to it by giving them stock options. Then they manipulate the stock price, as happened in the ’90s at places like Enron and Worldcom.
• Macroeconomic models, for example that high unemployment coincided with low wage inflation and vice versa, are valid until people try to exploit them.
  

  Such trade-offs, [Robert Lucas] argued, existed only if no one expected policymakers to exploit them. Unanticipated inflation would erode the real value of wages, making workers cheaper to hire. But if central bankers tried to engineer such a result, by systematically loosening monetary policy, then forward-looking workers would pre-empt them, raising their wage claims in anticipation of higher inflation to come. Cheap money would result in higher prices, leaving unemployment unchanged.

  – The Economist July 13th 2006, Big questions and big numbers

What does this imply for adaptive behavior? When a creature discovers a relationship, that may hold as long as it doesn’t try to exploit it.

Eric Bonabeau pointed me to an interesting article on calibrating dynamic models: “Model calibration as a testing strategy for system dynamics models” by Rogelio Oliva in the European Journal of Operational Research (v. 151, 2003, pp. 552-568). In it, he points out some problems with the global “estimate parameters by minimizing an overall score:”

• It often makes sense to estimate parameters of a subsystem based on that subsystem’s performance. “Graham (1980) provides two strong reasons for estimating parameters with data below the level of aggregation. First, most factual knowledge about the system falls into the category of unaggregate data, and parameters can be estimated independently of model equations. Second, parameters that are directly observable, or that can be estimated from unaggregate data – records, interviews, etc. – increase the models ability to anticipate events outside of the historical experience, and are intrinsically more robust than parameters that are inferred from observed behavior.”[Thought: what about minimizing the weighted sum of the overall error, and the subsystem error? That might have the right properties: the subsystem parameters are “mostly” controlled by the subsystem error, but they can be “tweaked” a bit to reduce the overall error.]
• “Since not all model parameters affect all output variables in the model, as the number of data series in the error function increases, individual parameters become less significant; variations in individual parameter values have a small impact in a complex error function, thus resulting in wider confidence intervals for the estimated parameters. Similarly, increasing the number of parameters to be estimated through an error function reduces the degrees of freedom in the estimation problem, thus resulting in wider confidence intervals, i.e., less efficient estimators.”

Maybe it’s time to ditch the idea of a single task with a single error measure? It doesn’t really fit into the idea of a living being that performs many tasks, reusing parts of one task in another, and often performing several tasks at once.

It fits well with the cybernetics style “many interacting control loops,” especially if you consider a control loop as trying to optimize some cost function, rather than maintaining a set point.  It also fits well with my video game ideas, of having a model, goals and strategies at many levels.  Where does this leave the MDL/Reinforcement Learning approach?

Graham, A.K., 1980. Parameter estimation in system dynamics modeling. In: Randers, J. (Ed.), Elements of the System Dynamics Method. Productivity Press, Cambridge, MA, pp. 143–161.