Trapped Priors As A Basic Problem Of Rationality
# | #bayes, #rationality, #acx, #psychology, #politics, #mental-model
...the brain combines raw experience (eg sensations, memories) with context (eg priors, expectations, other related sensations and memories) to produce perceptions. You donāt notice this process; you are only able to consciously register the final perception, which feels exactly like raw experience.
... more scientifically literate people are more likely to have partisan positions on science (eg agree with their own party's position on scientifically contentious issues, even when outsiders view it as science-denialist). If they were merely biased, they should start out wrong, but each new fact they learn about science should make them update a little toward the correct position. That's not what we see. Rather, they start out wrong, and each new fact they learn, each unit of effort they put into becoming more scientifically educated, just makes them wronger. That's not what you see in normal Bayesian updating. It's a sign of a trapped prior.
Scott presents an elegant model for when evidence updates people in the opposite direction. The core idea is that if the priors are strong enough, the perception of evidence (sensation) itself will be so tainted by the priors that we fail to make the right update. A must read.
...maybe you feel like you are using a particular context independent channel (eg hearing). Unbeknownst to you, the information in that channel is being context-modulated by the inputs of a different channel (eg vision). You donāt feel like āthis is what Iām hearing, but my vision tells me differently, so Iāll compromiseā. You feel like āthis is exactly what I heard, with my ears, in a way vision didnāt affect at allā.