A while ago, i wanted to automate literally everything with AI. So, taking screenshots, clicking, mousing, typing automation it is! Of course, you can imagine how the process looks like. But the project failed because when I asked for x/y coordinates of elements in a nice json output from the AI (of the screenshot) it all gave wrong values. I found out that those values are batshit wrong AFTER building the whole application that only worked in expection of that part (kinda most important).

Fuck.

But it really understood "Go to the address bar of the browser" (and it would do on the backgran a query to dimensions of the address bar. that's where the neat AI comes in). It could locate elements with poor descriptions. It understood very well. Only coordinates went wrong.