That's a real drag, all the more so since that doesn't happen when running Windows XP in VMWare (the mouse 'just works': you can just roll the cursor in and out of the VMWare window and it works as it should).
The trick is to install the 'vmmouse' driver (which comes with VMWare-tools) in X.org in the Linux guest. (This solution comes mostly from a post by 'zaroff' on the Ubuntu Forums.)
After installing Linux in VMWare server, click VM->Install VMWare Tools... on the VMWare menu. This makes Linux think you've just mounted a CD with a couple of files on it (the 'CD' will probably show up on the desktop). Unpack the .tar.gz file, cd into the vmware-tools-distrib directory, and run the vmware-install.pl installer.
When I did this, I found that when the installer re-wrote /etc/X11/xorg.conf, it didn't put in a DefaultDepth directive in the "Screen" section, and I had to add a DefaultDepth 24 line to that section.
Next you need to install the vmmouse driver. A good start is to run the following inside the vmware-tools-distrib directory:
find . -type f -name 'vmmouse*'
You need to copy the correct vmmouse driver (depending on what version of X.org your Linux guest is running) into the X.org input modules directory. For an Ubuntu 6.10 Linux guest, I needed to copy the XOrg/7.0/vmmouse_drv.so to /usr/lib/xorg/modules/input. (A friend was having the same trouble running a CentOS 4.4 guest: he needed to copy XOrg/6.8.x/vmmouse_drv.o to /usr/X11R6/lib/modules/input/.)
Next you need to make 2 changes to /etc/X11/xorg.conf:
- add Load "vmmouse" to the "Module" section
- change Driver "mouse" to Driver "vmmouse" in the "InputDevice" section