One of the challenges I face when developing games with AI is having to describe coordinates using text, especially during the design or level design stages. It would be much more practical if we could interact directly with the map or game screen—selecting areas visually and then instructing the AI to perform actions on those selected regions.
For example, I might want to place an object at a specific coordinate. But describing its exact pixel location in text is both tedious and inefficient. In such cases, having visual tools like selection boxes or cropping tools would greatly improve the usability and effectiveness of working with AI.