The old man in me (I'm 67) just shakes my head when I read comments like this.I did not. Every friend I mentioned this to said despite the ride, the 22's look great and/or borderline too small already, and that I should keep them no matter the ride quality. So I was persuaded to just keep them and get used to how they ride.
Unless your friends are riding in your vehicle with you, their opinions don't matter. You can't see your wheels while you are bumping along in a vehicle with poor ride quality.
My Limited has the factory 22s and I think the ride quality sucks over rough pavement while in Normal mode. It gets better in Sport mode, but still not great. My plan is to get a set of 20s or maybe even 18s with Michelin Defender tires before this winter. I don't care what my friends might think.