Is there an AI that generates malware already?
No sarcasm, I really wonder it there is already a model that finds exploits and makes rootkits or something.
It could make red teams obsolete.

I saw one of those way back when the AI hype was going. I didn't save the link

there's jailbroken darker AIs though, yes. or were at the tail-end of the AI hype

I just saw the screenshots and claims and it didn't seem fake to me

they hosted their own models

Yes. We already have server side polymorphs for a long time now.
The solution was dynamic behavior analysis - which will solve the "whatever AI can spew".
There is also a cool approch to defend against malware I encountered - based on "unique code" vs "old code pattern". Malware tend to the first, while non malware tend to the second. The reason is simple: what self respecting malware auther use glibc? answer: noobs.
Also - Checkout the "grand cyber challlenge" from 2016.

Github Copilot

People are using AI to create fake mods on Minecraft mod websites. There has been attempts to put malware into people's machines this way.

I can imagine known exploits, fuzzing, generating fake content for payload delivery, etc...
Can't currently imagine it would "discover" new malware/approach, which wouldn't be based on existence within its training data