I've forgotten how many times I've had to fix my USB because of this...

Zero?

Have you? Cause I live life on the edge and never eject my harddrive before removing it. And I’ve never had an issue.

But please don't do it with SD cards. That actually can damage them.

Ive had to reformat drives because the file system got corrupted after I unplugged it without unmounting it first. Other times, especially with FAT32 drives, the chances you write don't actually happen and you find yourself going back to your computer to do them again