Noise: A Flaw in Human Judgement by Daniel Kahneman, Oliver Sibony, & Cass Sunstein

A number of studies have shown that statistical models that judges put together are more accurate than judges themselves. The judge would come up with a number of criteria and decide how much weight to give each one. Single judges are noisy because they will vary the weight given the criteria from one judgment to the next. Using a mechanical model will reduce noise. (Doug: Consider putting a formula together for evaluating teacher candidates. I would include appearance, personality, GPA, course difficulty, recommendations, and the quality of a writing sample.)

An algorithm is a process or set of rules to be followed. Using them is considered a mechanical judgment as opposed to clinical judgments made by humans. They can be simple or complex and they are noise-free. This explains why they tend to outperform human judgment, but they are not perfect. When you have multiple predictors you can weigh each one, but doing so is not likely to improve the outcome much. The trick is to decide which variables to look at. Rules using fewer variables are likely to be almost as good as rules using many predictors. You should override your model when you have decisive information that your model can’t handle. When using AI realize that it just finds patterns rather than using magic. AI also requires rich data. Mechanical judgment can reduce bias, but is not likely to eliminate it. For example, racial bias baked into previous convictions won’t be eliminated by the application of an algorithm.

Objctive Ignorance combines what you don’t know that you could know and what you don’t know because it hasn’t happened yet. Studies of political pundits show that they are hardly better than chance with long-range predictions with the most confident people tending to be the least accurate. People who engage in predictive tasks underestimate objective ignorance as faith can masquerade for rational confidence.

As life unfolds we rely on hindsight to explain what happened. When hindsight fails we are genuinely surprised. While correlation does not imply causation, causation does imply correlation. Causal thinking helps us make sense of a world that is far less predictable than we think. It also explains why we view the world as far more predictable than it really is. The degree of objective ignorance sets a ceiling on our predictions and our understanding. Unfortunately, we tend to underestimate our objective ignorance.