If you are working for a IT company and if you start taking direct work from a client, would it be unprofessional or unethical or is it fine..

Probably depends on the situation. If the work you're getting from the client has to do with what you're doing at the company you currently work at, I'd say unproffesional.

If its something on the side in your free time which has nothing to do with the company, for me that's fine. Even done it a couple of times where I work and they found it ok too.

Depends on your company policy and you, if you or your employer/contract have a problem with it, I think is fine

I have a clause in my contract that if a company wants to deal with me directly they would have to buy me out.

Check the contract. There’s probably a restriction.

Check the contract. There’s probably a restriction.

Personal opinion, don't do it. Let the client go through the proper channels. Let your project manager deal with them (but be involved or your PM might take shitty requirements) so that you're not directly answerable to them. 'Cause clients tend to fuck you over.