I will have to admit that Docky is one of my guilty pleasures running under Linux. I noticed I was having artifacts on either side of my dock any time I enabled 3D mode. After a lot of Googling the issue, I saw that I was not alone. Most of the articles suggested a bug in the driver and suggested updating to the new driver (in my case, this was the proprietary ATI driver). Updating it did not seem to make a difference. As a fluke, I quit Docky and restarted it and the artifacts were gone and all was right with the world (until I made another configuration change). Quitting and restarting solved the issue again.
So if you are having issues with your Docky install, quit Docky and start it again. It might not work in all cases, but it worked a treat in my case.