Artworks 1.7c is now 15 years old. So if it's an app you can't live without it would probably make sense to upgrade to Artworks2. Development of Artworks is now being handled by Martin Wuertner. You can get all the details from
http://www.mw-software.com/If you don't want to upgrade then the only suggestion I can make is to use Artworks when VirtualRPC is running in full screen mode, rather than in the desktop. If you want to use the RISC OS menus whilst VirtualRPC is running in a window then you will need to hold down CTRL before clicking on MENU (see the Mouse section of the VirtualAcorn User Guide).
If you 'play' with the option, shift and control keys it's quite easy to accidentally issue a keyboard shortcut to MacOS X, which could explain the majority of the problems.