Listening to the other side is essential for communication and conflict resolution. However, even when a listener listens well, the speaker may still exclaim, “You are not listening to me!” We reason this occurs because speakers think someone who disagrees with them simply has not listened. Across seven studies with 2,090 participants, we find that speakers think a listener listened better when the listener agrees with them than when they do not. This effect emerged despite the conversation being the same and the listener signaling they understood the speaker. The effect held across communication contexts, discussions of moral or non-moral issues, and controlling for other positive impressions of the listener. We find that naïve realism explains this effect: speakers believe their views are correct, so they think listeners who agree with them can better process objective information. We discuss the implications of these findings for managing conflict and group decision-making.