"You have given me too much money"

So last night I was in New York City celebrating my brother-in-law's cousin's 21st birthday (you follow that?) I'm taking a cab back to my brother's apartment, and the driver starts making conversation with me. He turns out to be a pretty cool guy, and he even took a route that saved me a buck or so. And despite growing up in the city area, my sense of direction sucks, so he easily could have screwed me over in that regard. So when it came time to pay up, I left him a pretty nice tip. I should also acknowledge that I NEVER know what to tip taxi driver's, so I wasn't even aware if it was exorbitant or not.

Well, apparently it was, because he tells me, "You have given me too much money." What do you do in this situation? I ended up telling him it was just fine, because honestly I didn't spend quite as much as I expected that night, and I took the subway to the bar (which is about 5 times less expensive). I also figured that the extra money helped him more than it hurt me.

But my question is this: When somebody offers the information that you've given too much money, is there ever a circumstance where it's acceptable to take some back?