Can only get RMagick to work writing with 777

Hi -

I am at a loss with chmod. The only way I can get RMagick to work
properly writing to a file is if I set the dir to 777 permissions, but
I feel like this is a huge security risk. Can someone give me a push
in the right direction with chown/chmod, etc. for RMagick?

All I want is for everyone to be able to upload images, and then my
app processes and saves those images in a directory.

Thanks for any help,