Question about ssh and rsync?

I want to rsync some files from my mailserver to a remotehost for back up. I want to allow rsync to use ssh to securely transfer the files. I have generated a rsa key pair(on the localhost) and exported the public key to the remotehost. When I issue the rsync command at the command line (on the localhost) it works fine. However, if I put the commands in the crontab (directly) or have crontab call a bash script... it doesn't work. I have tested the use of crontab (both ways) without trying to transfer to a remotehost and they work fine. The permissions look ok. Why won't this work?