A place where the Joyent community can gather, help each other out, and stay informed.
You are not logged in.
Update 2: I forgot to add the --delete option to the synchronize command. This means that whatever you delete on TextDrive, won't be deleted on StrongSpace. Please see the article to find the new command, and update your scripts!
Update: the article is now live at http://textfilter.org/article/backing-up-to-strongspace
Below is the original draft:
---
I sat down to fix backing up to Strongspace today. Here's how it works. This is a draft cause I'd like to get some verification of all the commands, I'll put it up at TextFilter once it's vetted.
First step in the process is setting up the SSH connection from TextDrive to Strongspace. The problem here is that the login needs to be automatic: when the backup is run as a cron job there's nobody around to enter the password! Unfortunately doing this securely is nigh impossible. There is ssh-agent, which will remember your password, but it is session based
Last edited by Mark Wubben (2006-05-19 11:09:41)
Offline
I had a quick skim through your article, Mark, and that's the method I use (blank SSH key and rsync'ed cron job). There was one stumbling block that I came across, though. Sub-users's Maildirs need to be chmod-ed so that only they can read them, otherwise mail doesn't get delivered. However, this means that the main user can't back up their mail, which was something that I wanted to do.
So, I set up a cron job for the sub-user that rsyncs their Maildir into a folder they can write to (that's actually in the main user's home dir but chgrp-ed and chmod-ed for them), and chmods it so that the main user can read. This runs just before the main backup. I also excluded the original Maildir from the rsync otherwise it'll spit (non-fatal) errors and I like a nice clean sync :).
Offline
Thanks Mark!
It takes time and effort to these things. I appreciate and see that it has higher priority than TextPanel, although TextPanel will finally come, right? :)
Offline
I don't work for TextDrive so I wouldn't know, ngungo...
Offline
Great walkthrough, Mark. Thanks much.
Andy, could you elaborate a bit on how you backup your maildirs? I don't quite get how mail users can run a cron job.
Offline
Mark Wubben wrote:
I don't work for TextDrive so I wouldn't know, ngungo...
Oh! I am sorry. My fault, I apologize.
Offline
The article is now live at http://textfilter.org/article/backing-up-to-strongspace
Offline
Ooh.. how silly. I forgot to remove the BBCode. And I threw in some Markdown for good measure as well!
Offline
schussat wrote:
Andy, could you elaborate a bit on how you backup your maildirs? I don't quite get how mail users can run a cron job.
Via usermin: I have them run something along the lines of
/usr/local/bin/rsync -rltvz --delete /user/home/myusername/domains/mydomain/homes/theirusername/Maildir /somewhere/they/can/write/to
Offline
usermin! Hah. I didn't even know that was there. Thanks.
So I just replicate the cron job for each user, rsyncing their Maildir to a tree that the main user can read, and chmod it to g+r so that the permissions are correct? Then that tree gets wrapped up in the rsync-to-strongspace script.
Have I got it?
Offline
Hi Mark,
I will ask silly questions here, but if you prefer me to put my questions at your textfilter I'll post them there.
1. When I SFTP to my strongspace account I did not see any .ssh directory so I create one on my root /home/ngungoatstrongspace/ then upload the authorized_keys file over. Is that OK?
2. The same for TextDrive. I create one on my root /users/home/ngungoattextdrive/ then upload the ss file over. Is that OK?
3. I use FileZilla to set the file ss to 400 since I don't know how to navigate by command line.
4. I login to webmin > Running Processes and type
ssh -i ss ngungoatstrongspace@accountname.strongspace.com
I got an error:
Warning: Identity file ss does not exist. Pseudo-terminal will not be allocated because stdin is not a terminal. Host key verification failed.
I guess I (being unix ignorant) don't know how to navigate to the file ss, it could not find the file.
Thanks,
ngungo
Offline
Ngungo, correct on all counts (and asking here is fine).
You need to provide the absolute path to the "ss" file, something like "/users/home/ngungo/ss".
Offline
All right, I'm going to throw in one more question for Andy. I think I've got things all figured out and working well, with the exception of an error from rsync about not being able to set the times on my Maildirs. I bet this is the same harmless error you mentioned in your first post. I tinkered with the exclude option in rsync, but that didn't do anything for me. What's the way to exclude the Maildir to suppress that error?
Offline
Man! Feel awfully stupid! I haven't made it work yet!
Hi Mark,
I've been trying many times every steps over, what was that possible wrong thing I making this time.Output from ssh -i /users/home/ngungo/.ssh/ss ngungo@ngungo.strongspace.com .. Pseudo-terminal will not be allocated because stdin is not a terminal. Host key verification failed.
Offline
Might just not work through Webmin.
Offline
Should I turn in a ticket, provided I have ssh files on both sides? All I want is ability to backup to StrongSpace.
Offline
ngungo, have you tried testing this at a shell? It does work fine there. Mark's suggestion that it may not work through webmin is a good one.
Offline
schussat wrote:
ngungo, have you tried testing this at a shell? It does work fine there. Mark's suggestion that it may not work through webmin is a good one.
I use Windows XP, how do I get into a shell?
Offline
ngungo wrote:
I use Windows XP, how do I get into a shell?
Offline
I've set everything up, the test command runs fine but when the cron job runs I get the following message:
: Permission denied
invalid characters in scp command!
here:
try using a wildcard to match this file/directory
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(443)
Any ideas as to what's going wrong?
Bonus question: if the authorized_keys file has already got a key on it, how do we add the content of another key? Blank line at the end of the file and paste the other key there?
Thanks for your help in advance.
Last edited by konstantinos (2006-05-07 20:32:14)
Offline
End of the file, yeah. Not sure what's up with the invalid character error, what are the file/dir names you're syncing?
Offline
Mark Wubben wrote:
End of the file, yeah. Not sure what's up with the invalid character error, what are the file/dir names you're syncing?
Here you go:
#!/bin/sh
/usr/local/bin/rsync -azq -e "ssh -i /home/XYZ/.ssh/ss" /home/XYZ/domains/ XYZ@XYZ.strongspace.com:/home/XYZ/txd-backup/weekly/domains/
/usr/local/bin/rsync -azq -e "ssh -i /home/XYZ/.ssh/ss" /home/XYZ/backups/databases/ XYZ@XYZ.strongspace.com:/home/XYZ/txd-backup/weekly/databases/
Offline
Hmm, perhaps some oddly named files inside those directories?
Offline
Mark Wubben wrote:
Hmm, perhaps some oddly named files inside those directories?
Nah, nothing out of the ordinary, AFAIK. Do you think this is an issue I should address to the TxD folks?
Offline
A bump, in case someone out there knows what's causing this issue.
If not, I'll have to file a ticket with TxD I think.
Offline
I forgot to add the --delete option to the synchronize command. This means that whatever you delete on TextDrive, won't be deleted on StrongSpace. Please see the article to find the new command, and update your scripts!
Offline
OK, I have spent all day trying to get this sorted out, but without (much) success. Maybe someone here might know what I am doing wrong?
The story so far:
I have generated the keyfiles as above, renamed the ss.pub to authorized_keys via the command line, and uploaded them as follows:
Textdrive:
/users/home/USERNAME/.ssh/ss <- chmod'd to 400
Strongspace:
/home/matthall/.ssh/authorized_keys <- 644
When I try to connect over ssh to the server via my textdrive shell, I am still prompted for my strongspace password.
Any help at all would be greatly appreciated!
Offline
make sure you do not have a passphrase on the keys.
See step 3 here: http://www.akadia.com/services/ssh_test … icate.html
Offline
Yeah, the keys have no passphrase - I am being asked for my StrongSpace password.
Thanks, though!
Offline
OK, My problem has been resolved:
Make sure that the .ssh directory on StrongSpace is chmod 700, and the authorized_keys file is chmod 600.
Tally-ho!
Oh, I posted every single step I took to set the keys up on my blog here:
SSH Access to StrongSpace from Textdrive
Last edited by MattH (2006-12-11 23:24:50)
Offline
I'm trying to backup stuff up from TextDrive to Strongspace following these instructions and am part way there.
However, if I try to SSH into Strongspace from TD I'm always prompted for my passphrase. I can't work out what step I'm missing in order to be able to SFTP to Strongspace without a password or passphrase. Any ideas?
Offline
I created a simple backup script:
http://textsnippets.com/posts/show/587
Offline
roeland, where do you configure the password?
Offline
Mark Wubben wrote:
roeland, where do you configure the password?
The password for mysql is in the the PWD_MYSQL variable.
For Strongspace I just use the ssh keys like you explained.
Or do you mean something else?
Offline
Ah, I missed the SSH keys bit :)
Offline
>Mark Wubben:
Off topic I know, but could the keys part of this be used so that you can set up an automator applet via FUSE to automount a textdrive account without having to enter a password?
Offline
The SSH keys part works with SSH in general so, yes, FUSE should login fine without asking for a password if you set them up.
At least mine does ;)
Last edited by juxta (2007-02-02 21:15:25)
Offline
ok I have gone through this procedure a few times and for some odd reason cannot get it to work.
I have followed the steps and the files are in proper position and permissions are set correctly I think.
I have no trouble accessing StrongSpace using the public key without a password in the shell.
This is the (simple) error message I am getting; rsync stars to work and then after beginning I get:
USERNAME@SS_USERNAME.strongspace.com:/home/SS_USERNAME/joyentbackups/allsitesweekly/: not found
..when I have created the '/joyentbackups/allsitesweekly/' folders in StrongSpace at the root.
there must be something very simple I am overlooking as I cannot understand why I am getting a not found message for a directory that does indeed exist.
thanks in advance for any suggestions!
~J.D.
Offline
ok i am still struggling with this but now I am getting a different error message!! Can anyone give me some assistance with this if you have the time?
rsync error: some files could not be transferred (code 23) at main.c(977) [sender=2.6.9]
/users/home/USERNAME/scripts/joyent_entire_structure_backup_wk.sh: SS_USERNAME@USERNAME.strongspace.com:/home/USERNAME/joyentbackups/allsitesweekly/: cannot execute
I'm pretty sure everything was done according to this thread and the previous posters blog posting (which was very helpful).
Thanks in advance!!
~J.D.
Offline
