Its very simple to get PMs to leave you alone; have a good long talk with them about their goals, motivations, and methods. Try to understand their thinking, then preempt the visit to your desk with something to make them not come to your desk.

Or get a WFH deal and dont answer any comm requests.