think tank forum » technology » incremental svn repository backups to amazon s3

can't log in?

· register an account
· recover your account
· search the forums

17 years ago
link

lucas
i ❤ demo

the only data that my server holds that i care about are my seven svn repositories. i want these backed up incrementally at least daily. here's my script (in progress):

#!/bin/sh

export AWS_ACCESS_KEY_ID=s3_access_key_id
export AWS_SECRET_ACCESS_KEY=s3_secret_access_key

mkdir /home/user/.backup-tmp
mkdir /home/user/.backup-tmp/svn
chmod 700 -R /home/user/.backup-tmp
svnadmin hotcopy /home/user/svn/repo-0 /home/user/.backup-tmp/svn/repo-0
svnadmin hotcopy /home/user/svn/repo-1 /home/user/.backup-tmp/svn/repo-1
svnadmin hotcopy /home/user/svn/repo-2 /home/user/.backup-tmp/svn/repo-2
svnadmin hotcopy /home/user/svn/repo-3 /home/user/.backup-tmp/svn/repo-3
svnadmin hotcopy /home/user/svn/repo-4 /home/user/.backup-tmp/svn/repo-4
svnadmin hotcopy /home/user/svn/repo-5 /home/user/.backup-tmp/svn/repo-5
svnadmin hotcopy /home/user/svn/repo-6 /home/user/.backup-tmp/svn/repo-6

duplicity --no-encryption /home/user/.backup-tmp s3+http://bucket

export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=

rm -rf /home/user/.backup-tmp

i'd love to hear feedback on integrity/security.

17 years ago
link

lucas
i ❤ demo

references:
http://forum.slicehost.com/comments.php?DiscussionID=498
http://svnbook.red-bean.com/en/1.4/svn.reposa … t.html#svn .reposadmin.maint.backup

17 years ago
link

lucas
i ❤ demo

and i think i'm going to back up four times daily. these are in MST, and are based off of when I commit most things.

4pm
8pm
12am
4am

:D

virtually all commits are made between the hours of 2pm and 3am.

17 years ago
link

lucas
i ❤ demo

another ref:
http://www.randys.org/2007/11/16/how-to-autom … -duplicity

16 years ago
link

DaGr8Gatzby
Drunk by Myself

Lucas ... is SVN really all that jazz?

16 years ago
link

lucas
i ❤ demo

for me, yeah

16 years ago
link

dannyp
dʎuuɐp

is there an online svn client or some sort of (java?) platform that would allow you to check out or commit stuff when i'm not on my svn-client-enabled machine?

16 years ago
link

lucas
i ❤ demo

there is a php api and a php web client, but i don't know its status

16 years ago
link

lucas
i ❤ demo

damn this shit is fast.

backup #1:

No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1199443344.34 (Fri Jan  4 10:42:24 2008)
EndTime 1199443345.64 (Fri Jan  4 10:42:25 2008)
ElapsedTime 1.31 (1.31 seconds)
SourceFiles 165
SourceFileSize 7809948 (7.45 MB)
NewFiles 165
NewFileSize 7809948 (7.45 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 165
RawDeltaSize 7768988 (7.41 MB)
TotalDestinationSizeChange 2843240 (2.71 MB)
Errors 0
-------------------------------------------------

backup #2:

--------------[ Backup Statistics ]--------------
StartTime 1199443489.33 (Fri Jan  4 10:44:49 2008)
EndTime 1199443490.09 (Fri Jan  4 10:44:50 2008)
ElapsedTime 0.75 (0.75 seconds)
SourceFiles 488
SourceFileSize 11750518 (11.2 MB)
NewFiles 324
NewFileSize 3944666 (3.76 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 324
RawDeltaSize 3829978 (3.65 MB)
TotalDestinationSizeChange 3022848 (2.88 MB)
Errors 0
-------------------------------------------------

16 years ago
link

lucas
i ❤ demo

this one took a few minutes..

--------------[ Backup Statistics ]--------------
StartTime 1199443709.09 (Fri Jan  4 10:48:29 2008)
EndTime 1199443978.15 (Fri Jan  4 10:52:58 2008)
ElapsedTime 269.06 (4 minutes 29.06 seconds)
SourceFiles 720
SourceFileSize 196620885 (188 MB)
NewFiles 269
NewFileSize 185021919 (176 MB)
DeletedFiles 0
ChangedFiles 451
ChangedFileSize 11598966 (11.1 MB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 720
RawDeltaSize 13903312 (13.3 MB)
TotalDestinationSizeChange 183549125 (175 MB)
Errors 0
-------------------------------------------------

16 years ago
link

lucas
i ❤ demo

here's what my keys look like for my bucket:

1. duplicity-full-signatures.2008-01-04T10:42:23Z.sigtar.gz
2. duplicity-full.2008-01-04T10:42:23Z.manifest
3. duplicity-full.2008-01-04T10:42:23Z.vol1.difftar.gz
4. duplicity-inc.2008-01-04T10:42:23Z.to.2008-01-04T10:44:48Z.manifest
5. duplicity-inc.2008-01-04T10:42:23Z.to.2008-01-04T10:44:48Z.vol1.difftar.gz
6. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.manifest
7. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol1.difftar.gz
8. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol10.difftar.gz
9. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol11.difftar.gz
10. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol12.difftar.gz
11. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol13.difftar.gz
12. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol14.difftar.gz
13. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol15.difftar.gz
14. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol16.difftar.gz
15. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol17.difftar.gz
16. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol18.difftar.gz
17. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol19.difftar.gz
18. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol2.difftar.gz
19. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol20.difftar.gz
20. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol21.difftar.gz
21. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol22.difftar.gz
22. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol23.difftar.gz
23. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol24.difftar.gz
24. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol25.difftar.gz
25. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol26.difftar.gz
26. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol27.difftar.gz
27. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol28.difftar.gz
28. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol29.difftar.gz
29. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol3.difftar.gz
30. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol30.difftar.gz
31. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol31.difftar.gz
32. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol32.difftar.gz
33. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol33.difftar.gz
34. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol34.difftar.gz
35. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol35.difftar.gz
36. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol36.difftar.gz
37. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol4.difftar.gz
38. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol5.difftar.gz
39. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol6.difftar.gz
40. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol7.difftar.gz
41. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol8.difftar.gz
42. duplicity-inc.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.vol9.difftar.gz
43. duplicity-inc.2008-01-04T10:48:27Z.to.2008-01-04T10:55:33Z.manifest
44. duplicity-inc.2008-01-04T10:48:27Z.to.2008-01-04T10:55:33Z.vol1.difftar.gz
45. duplicity-inc.2008-01-04T10:55:33Z.to.2008-01-04T10:57:13Z.manifest
46. duplicity-inc.2008-01-04T10:55:33Z.to.2008-01-04T10:57:13Z.vol1.difftar.gz
47. duplicity-new-signatures.2008-01-04T10:42:23Z.to.2008-01-04T10:44:48Z.sigtar.gz
48. duplicity-new-signatures.2008-01-04T10:44:48Z.to.2008-01-04T10:48:27Z.sigtar.gz
49. duplicity-new-signatures.2008-01-04T10:48:27Z.to.2008-01-04T10:55:33Z.sigtar.gz
50. duplicity-new-signatures.2008-01-04T10:55:33Z.to.2008-01-04T10:57:13Z.sigtar.gz

16 years ago
link

lucas
i ❤ demo

damn.. this is the real deal. here's the best part: all of the files (in rev HEAD) stored in all of my svn repositories total 400MB. but,

lucas@wlw:~$ du -sh svn
195M    svn

so there's some compression going on for sure. plus that 195MB includes all of the logs and the full history, not just rev HEAD. so that's awesome.

oh yeah, and duplicity does backups incrementally. so it is the real deal!

16 years ago
link

lucas
i ❤ demo

ok, i verified the integrity of my svn backups. it looks great!

i'm going to use encryption, though, just to keep my s3 secret access key out of the clear.

amazon aws s3 pricing:

United States

Storage
$0.15 per GB-Month of storage used

Data Transfer
$0.10 per GB - all data transfer in
$0.18 per GB - first 10 TB / month data transfer out
$0.16 per GB - next 40 TB / month data transfer out
$0.13 per GB - data transfer out / month over 50 TB

Requests
$0.01 per 1,000 PUT or LIST requests
$0.01 per 10,000 GET and all other requests*
* No charge for delete requests

so i can run an incremental backup up to 1000 times per month, and this whole backup service will cost me $0.05 per month (assuming ~200MB stored with minimal compressed commit sizes).

\o/

16 years ago
link

asemisldkfj
the law is no protection

haha, this rules.

16 years ago
link

sriehl
surreal

wouldn't it be .15 (up to a gig of space) + .10 (for the transfer) +.01*number of times backed up (+ .18 if you need to get your data back from them)

so between 31cents and 49cents?

16 years ago
link

sriehl
surreal

^assuming you do 6,000 backups and no lists.

16 years ago
link

lucas
i ❤ demo

i don't know if they round up the the next whole unit or not. i assumed not. i guess either way, it's less than $1/month. silly cheap.