This is an interesting angle on something I've noticed, and perhaps a better explanation of it, but I'll put my observation in the ring in case it offers any other insights: Smart people can end up developing faulty world views because they are good at arguing, and every time they win an argument, they convince themselves more and more of the infallibility of their world view, when in fact all that happened was they encountered yet another person who couldn't spot the flaw in it.
An example of this is extreme libertarianism. For a long time, I had convinced myself that it was totally sound because if I argued politics with most people I knew, I could shut them down with logic and "prove" my case. The logic and robot-like consistency of it just reassured me that I was unassailably correct. It took me getting to the point where I realized that I didn't truly believe some of my own arguments, despite the fact that I had successfully made them countless times, to snap out of it and realize that my views, as rigidly consistent as they were, were probably disastrous from a public policy standpoint. I rationalized a position to others, which ultimately rationalized it to me as well. Don't do that.
> "I didn't truly believe some of my own arguments"
One of my core philosophies of argument is that I always argue honestly. That is, I give my real reasons for believing things (as well as I understand them) even if I know of other arguments that sound persuasive but that I don't believe as strongly.
I find this means I lose arguments slightly more often (which is a good thing; I end up changing my mind), but I also win handily quite a bit more often. I see far fewer stalemated arguments, and far fewer arguments that devolve into quoting my side's famous thinkers at people who quote their side's famous thinkers back.
It was difficult at first, but with practice it's become fairly easy.
I think the problem is that you never encountered anyone who actually understood the best reasons for the policies you opposed, or even had a model (however ad hoc) in which the relevant benefits could be expressed.
I see that kind of thing all the time: I heard bad arguments against the minimum wage for years before I was even aware of a purported disemployment effect. Ditto for the relationship between traditional morality and STD/population growth concerns.
I don't think you can blame yourself for that. To the extent that an idea makes sense, there should be someone that appreciates it, and can convey its logic. To the extent that your acquaintances were not those people (and did not subject their beliefs to the same checks you did), neither did they have any business being more confident than you in their views.
Sure, your ideas may have been disastrous as policy, but how could you have known that? You can't update on evidence you haven't gotten -- let alone evidence which resists revelation by interested adversaries!
The issue that hamburglar seemed to have is that he was relying too heavily on social proof. Something like "My ideas are the best out of the people I know so I should bet heavily on them" does not work well when your friends to do not cover most if not all of the relevent solution space with well thought out arguments.
That's why I said that hamburglar might not have a basis for being confident in an absolute sense, but definitely had grounds for being more confident than people who can't even spot a flaw in the argument.
(And FWIW, "trusting social proof", as the term is used, would mean deferring to the group beliefs; I think you mean to say s/he was overestimating the peer group's representativeness. But even then, I think the group was representative of the population at large: very few people possess the kind of understanding to do that, nor hold themselves to the standard that would make them seek it.)
> And FWIW, "trusting social proof", as the term is used, would mean deferring to the group beliefs;
I did not know the phrase had commonly held definition. I meant more along the lines hamburglar's heuristic for deciding when they were more likely to be correct was overly dependent on not just the people who he happened to be around but other people in general.
> But even then, I think the group was representative of the population at large: very few people possess the kind of understanding to do that, nor hold themselves to the standard that would make them seek it.
My comment was not about a group representing the population, but rather representative of the possible solutions weighted by relevance or similar metric.
> I think the problem is that you never encountered anyone who actually understood the best reasons for the policies you opposed,
hamburglar should have/be use heuristics that do not depend on encountering the right people to gather and judge evidence when possible.
>I did not know the phrase had commonly held definition.
"Social proof" refers to the general phenomenon of people forming opinions in alignment with what they observe others believing. I understand what you meant, but it's confusing to frame that as a social proof issue, given its common usage.
>My comment was not about a group representing the population, but rather representative of the possible solutions weighted by relevance or similar metric.
I know, but that's it's wrong: the group was representative of the typical arguments and reasoning offered regarding how to model government policy. Most people lack the understanding to articulate what's wrong with unfamiliar or "absurd" proposals; casting his net wider wouldn't have helped much, even in the internet age.
>hamburglar should have/be use heuristics that do not depend on encountering the right people to gather and judge evidence when possible.
Then what should they depend on? Again, you can't update on evidence that even interested adversaries can't find! At most, it means you should lower your confidences all around, but even then, the people he argued with should have lowered them even further, for the very same reasons.
An example of this is extreme libertarianism. For a long time, I had convinced myself that it was totally sound because if I argued politics with most people I knew, I could shut them down with logic and "prove" my case. The logic and robot-like consistency of it just reassured me that I was unassailably correct. It took me getting to the point where I realized that I didn't truly believe some of my own arguments, despite the fact that I had successfully made them countless times, to snap out of it and realize that my views, as rigidly consistent as they were, were probably disastrous from a public policy standpoint. I rationalized a position to others, which ultimately rationalized it to me as well. Don't do that.