Broadly there are two things which concerns me:

1: Clients' businesses fail miserably or change their direction.

2: Instead of focusing and improving the quality of their work/product, they prioritise remaining things instead.

BONUS: Don't forget those individuals who dream really big but fails to take any action towards it, just talking. I stay away from these people personally.

Because of these reasons design gets vanished or no longer valid for their new business venture, and don't forget the time and dedication it took to create, as a freelancer it hurts a bit.

I like working for non-profit organisations, most of them look for volunteers. Your work and efforts are alive, and you have to be jack of all trades IMO when dealing with them. Additionally in the process you will meet some extraordinary individuals.