Noise: A Flaw in Human Judgement by Daniel Kahneman, Oliver Sibony, & Cass Sunstein

A judgment is a decision where the instrument used is the human mind. The most common measure of variability is the standard deviation, which the authors use here. Matters of judgment occupy a place between known facts and taste. There are two types of reliability. They are within-person where the same person judges a situation more than once over time and between-person where multiple people judge the same thing at the same time. The authors provide a scenario involving the hiring of a CEO for you to experience the process of judging.

The main idea here is that being very high one time and very low another time does not mean that on average you are right. The key equation is Bias + Noise = Error. Bias is simply the average of errors. If your average error is zero your high judgments are equal to your low judgments. The less noise, the smaller the standard deviation. That means your bell-shaped curve has less spread. Here we are introduced to the concept of mean squared error. It is equal to bias squared plus noise squared. This means that bias and noise are independent of each other and equally weighted when it comes to determining error.

This chapter deals with a noise audit from 1981 of a federal judicial system where 208 judges each judged sixteen cases. The measure of noise within each case is the standard deviation of the terms assigned in that case. The mean sentence was 7.0 years and the standard deviation around that mean was 3.4 years. This is considered the system noise. When we compare judges to each other we find some more lenient than others. These deviations are called level errors. In this sample, the level errors were 2.4. The deviation within the sentences for a single judge is called pattern error. The pattern errors for each judge add up to zero.
The three types of noise are compared to each other with this equation. System Noise² = Level Noise² + Pattern Noise².

Occasion noise is the noise we see when looking at the same person making repeated judgments about the same thing. We do not always produce identical judgments when faced with the same facts on two occasions. Measuring occasion noise is not easy. If you present someone with the same problem a second time they may recognize it and simply reproduce their earlier answer.
The wisdom-of-crowds effect comes into play here. If the same person makes an estimate on two occasions, the average is more likely to be closer to the correct answer. The average of two judges, however, is likely to be even more accurate. This gives a rationale for the notion that you should sleep on it and think again. Some other sources of occasional noise are mood, stress, fatigue, weather, time of day, and the order in which a specific judgment appears. Occasion noise is usually less than the difference among individuals. In other words, you are less different from the you of last week than you are from someone else today.

The first few people in a group who express an opinion can sway the group. The best way to make your group “wise” is to have people express opinions independently without knowing what others think. Aggregating independent judgments is a good way to reduce noise. The opinion of an initial speaker can cause an information cascade. When it comes to politics when members of one party hear the opinion of the other party they are likely to reflexively disagree rather than do their own thinking. This was especially apparent when Trump was president as Democrats were quick to disagree with anything he did or said.