charles skipped what i see as a highly critical question, personal backup. my life is on a 13" macbook air, all data, mail back decades (i do not save all mail), etc. the whole drive is encrypted, my main reason for moving to lion. i have two time machine drives, one at home and one i carry on the road. both are encrypted. belt and braces, i also use unison to sync my laptop's home data to a server in colo. it goes to a freebsd geli, i.e. encrypted, partition. all keys and other critical private data are in a text file on my laptop encrypted with gpg, which i use emacs crypt++ to access. that key file, a bunch of x.509 certs, ... are copied to an ironkey usb. randy
On Sat, 2011-08-13 at 14:12 +0900, Randy Bush wrote:
charles skipped what i see as a highly critical question, personal backup.
Very good point. For my laptops, nearline field storage includes my laptop's drive and a portable external drive. Online and nearline home storage is a network attached storage array running proprietary X-RAID (like RAID-5) with a hot-spare drive. All my machines (desktops, servers and laptops) are set to perform regular backups to the NAS. Offline backups are done to a series of external USB HDs that are rotated into place for nightly incremental and weekly full backups. Current retention schema is 4 weeks of backups with a one week offsite physical rotation (performed monthly to a safety deposit box). I'm at the moment trying to figure out a good way for doing streaming backups to an offsite DC. -- /*=================[ Jake Khuon <khuon@NEEBU.Net> ]=================+ | Packet Plumber, Network Engineers /| / [~ [~ |) | | -------- | | for Effective Bandwidth Utilisation / |/ [_ [_ |) |_| NETWORKS | +==================================================================*/
On Sat, 2011-08-13 at 14:12 +0900, Randy Bush wrote:
charles skipped what i see as a highly critical question, personal backup.
Very good point.
For my laptops, nearline field storage includes my laptop's drive and a portable external drive. Online and nearline home storage is a network attached storage array running proprietary X-RAID (like RAID-5) with a hot-spare drive. All my machines (desktops, servers and laptops) are set to perform regular backups to the NAS. Offline backups are done to a series of external USB HDs that are rotated into place for nightly incremental and weekly full backups. Current retention schema is 4 weeks of backups with a one week offsite physical rotation (performed monthly to a safety deposit box). I'm at the moment trying to figure out a good way for doing streaming backups to an offsite DC.
We used to use DVD's for off-site backup, but that's not been the best of solutions. I've been experimenting with external hard drives but I am less comfortable with them; I've seen too many drives fail. The idea of letting them sit for awhile and praying they spin up later bothers me. :-) On the other hand, running Unison to a server someplace else has obvious benefits and some downsides too. Backups remain a tricky problem to get right. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
We used to use DVD's for off-site backup, but that's not been the best of solutions. I've been experimenting with external hard drives but I am less comfortable with them; I've seen too many drives fail. The idea of letting them sit for awhile and praying they spin up later bothers me. :-) On the other hand, running Unison to a server someplace else has obvious benefits and some downsides too. Backups remain a tricky problem to get right.
... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
For backups I use a combination of: 1) Sync server to a single purpose windows XP box in a different building (why windoze will become apparent shortly) 2) Continuous dribble of remote backup from XP backup to backblaze (doesn't support server OS for obvious reasons) 3) Monthly or so backup of really critical files to microsd cards in safety deposit box. They are inexpensive and very small, so I just buy a new one for each backup and let them pile up in the box. It's not perfect but works for me, and I can use backblaze to remotely "recover" a file if access to my server is down :) -Dorn
When my parents finally got broadband years ago and I wired their house, I loaded Linux on an old x86 PC and stashed it in a corner of their garage. rsync over an ssh tunnel runs nightly from a Linux server in my house (about 90 miles away) to theirs. I don't sync everything, but it gives me comfort knowing that my family photos and such are secure somewhere else. It gives the added advantage of letting them browse our family photo archive locally. David Ramsey Charlotte, NC On Sat, Aug 13, 2011 at 10:13 AM, Joe Greco <jgreco@ns.sol.net> wrote:
On Sat, 2011-08-13 at 14:12 +0900, Randy Bush wrote:
charles skipped what i see as a highly critical question, personal backup.
Very good point.
For my laptops, nearline field storage includes my laptop's drive and a portable external drive. Online and nearline home storage is a network attached storage array running proprietary X-RAID (like RAID-5) with a hot-spare drive. All my machines (desktops, servers and laptops) are set to perform regular backups to the NAS. Offline backups are done to a series of external USB HDs that are rotated into place for nightly incremental and weekly full backups. Current retention schema is 4 weeks of backups with a one week offsite physical rotation (performed monthly to a safety deposit box). I'm at the moment trying to figure out a good way for doing streaming backups to an offsite DC.
We used to use DVD's for off-site backup, but that's not been the best of solutions. I've been experimenting with external hard drives but I am less comfortable with them; I've seen too many drives fail. The idea of letting them sit for awhile and praying they spin up later bothers me. :-) On the other hand, running Unison to a server someplace else has obvious benefits and some downsides too. Backups remain a tricky problem to get right.
... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Backups remain a tricky problem to get right.
Yeah. I've been using external USB terabyte disks, which work OK but are irritatingly flaky. I keep thinking that this is what tape is for, but every time I look at AIT or LTO tapes and jukeboxes, they seem to be about a generation behind the disks in capacity or so expensive that I can buy ten TB disks for the cost of a five tape jukebox. R's, John
On 2011-08-13 16:53 , John Levine wrote:
Backups remain a tricky problem to get right.
Yeah. I've been using external USB terabyte disks, which work OK but are irritatingly flaky.
I keep thinking that this is what tape is for, but every time I look at AIT or LTO tapes and jukeboxes, they seem to be about a generation behind the disks in capacity or so expensive that I can buy ten TB disks for the cost of a five tape jukebox.
Simple and cheap solution: - NAS box with a proper RAID-5 disk-set (or just one of your servers that has enough disk space) use this to do your backups to with your favorite tool (duplicity, rsync, etc) at the times that you think is useful. (And one can create a TimeMachine box using the netatalk package) This is your 'always online' backup, containing the last X revs - Disk-dock attached to the NAS (eg http://www.disk2go.com/d2go/shop/downloads/produkt/index.html?t_ParentID=241) Just plug in a disk there once in a while and rsync to it. One could keep multiple disks with the same data for multiple different revisions or when using rsync hardlinks even all revisions. As one can get SATA disks at 2TB or 3TB one will still have to chunk up the NAS box into pieces most of the time but that is all scriptable (filesystem labels are useful to detect which disk is there to automate that selection even further or heck put a file on it describing the contents). Presto, off-line backups, just like a tape solution but much much faster and much much cheaper and easily expandable/upgradeable to new bigger harddisk models to grow along with your RAID box and other needs. Don't forget to store those offline backups far away from the NAS box and to of course properly secure all of it so that if somebody physically steals it or hack the thing remotely they don't have a copy of your whole live, google/facebook and other such companies would hate to sink to the second spot of who owns your data... Greets, Jeroen
On Sat, 2011-08-13 at 09:13 -0500, Joe Greco wrote:
We used to use DVD's for off-site backup, but that's not been the best of solutions. I've been experimenting with external hard drives but I am less comfortable with them; I've seen too many drives fail. The idea of letting them sit for awhile and praying they spin up later bothers me. :-)
I can understand that. That's why I use a rotation and retention system as well as two NASes syncing to one-another with the backups being performed from the second NAS. I treat them like I used to treat tape. The onsites contain nightly incrementals as well as weekly full dumps. These drives are rotated every week. Every month a copy of the latest full dump is copied to another set of drives which are sent to offsite storage. I keep two copies offsite. To implement this, I need a minimum of nine USB drives. I actually have 10 so I can have an extra one onsite as a hardware replacement-spare. Online Nearline Onsite Offsite Storage Backup Backup Backup ---------------------------------------------- [USB7]<--{[USB5],[USB6]} | ^ V | [USB8] | | | V | [NAS1]>>>>>[NAS2]>>>>>[USB0]>>>{[USB3],[USB4]} | V [USB1] | V [USB2] Nothing's perfect. There's still risk but there's some amount of assurance. -- /*=================[ Jake Khuon <khuon@NEEBU.Net> ]=================+ | Packet Plumber, Network Engineers /| / [~ [~ |) | | -------- | | for Effective Bandwidth Utilisation / |/ [_ [_ |) |_| NETWORKS | +==================================================================*/
On 13/08/2011, at 3:12 PM, Randy Bush <randy@psg.com> wrote:
charles skipped what i see as a highly critical question, personal backup.
my life is on a 13" macbook air, all data, mail back decades (i do not save all mail), etc. the whole drive is encrypted, my main reason for moving to lion.
i have two time machine drives, one at home and one i carry on the road. both are encrypted.
belt and braces, i also use unison to sync my laptop's home data to a server in colo. it goes to a freebsd geli, i.e. encrypted, partition.
all keys and other critical private data are in a text file on my laptop encrypted with gpg, which i use emacs crypt++ to access. that key file, a bunch of x.509 certs, ... are copied to an ironkey usb.
randy
I found the "mobile account/portable home directory" feature in os x server to be very useful for wife and kids (powerbooks). They get backed up and don't realize. If they crash a drive or if i upgrade a machine I just log them back in and resynch the machine. no more lost homework. jy
On Aug 13, 2011, at 1:12 AM, Randy Bush wrote:
charles skipped what i see as a highly critical question, personal backup.
I've been wondering this as well. My home backups are somewhat large and not yet offsite due to their size. (~4.7TB). This is due to both purchased digital media storage and photography. These are sufficiently large that the problem is harder to resolve, as nobody "makes" a 5TB drive I can ship to a colo. Drive failures in the "backups" host also become painful to work with as I'm cheap so the ZFS pool isn't 100% mirrored. Some machines use netatalk plus the "defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1" hack. I've considered just subbing to backblaze as it's "cheap" on a single-host basis, but need something closer to ~5-7TB plus some room for growth (maybe 250G-500G/year). I have considered just trenching copper/fiber (see other thread) to a neighbor and placing the host there, but it's not much geographical diversity, plus if the neighbor moves it becomes fun to re-explain what you are doing. (worse if they're non-techie). The biggest problem I've seen is with a "cloud"/"tubes" provider, my upload speed puts the 4.7TB initial sync somewhere around 145 days (assuming 3Mb/s upload). Is anyone aware of a solution for this that is sensible $$$ without rolling my own (i'm estimating about 2-3k to do this...)? And preferably costs maybe $ or $$? - Jared
On Aug 13, 2011, at 10:44 PM, Jared Mauch wrote:
On Aug 13, 2011, at 1:12 AM, Randy Bush wrote:
charles skipped what i see as a highly critical question, personal backup.
I've been wondering this as well.
My home backups are somewhat large and not yet offsite due to their size. (~4.7TB).
This is due to both purchased digital media storage and photography. These are sufficiently large that the problem is harder to resolve, as nobody "makes" a 5TB drive I can ship to a colo. Drive failures in the "backups" host also become painful to work with as I'm cheap so the ZFS pool isn't 100% mirrored.
Some machines use netatalk plus the "defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1" hack.
I've considered just subbing to backblaze as it's "cheap" on a single-host basis, but need something closer to ~5-7TB plus some room for growth (maybe 250G-500G/year). I have considered just trenching copper/fiber (see other thread) to a neighbor and placing the host there, but it's not much geographical diversity, plus if the neighbor moves it becomes fun to re-explain what you are doing. (worse if they're non-techie).
The biggest problem I've seen is with a "cloud"/"tubes" provider, my upload speed puts the 4.7TB initial sync somewhere around 145 days (assuming 3Mb/s upload).
Is anyone aware of a solution for this that is sensible $$$ without rolling my own (i'm estimating about 2-3k to do this...)? And preferably costs maybe $ or $$?
Stretching the definition of "cloud" (why not, everyone else has), you can include SmugMug, Flixter, Picassa, etc. For $20-$40/year (yes, _year_), some of these allow infinite uploads. Moreover, while you only get a few MB/s per session, you can open multiple simultaneous uploads. I've filled my upstream pipe uploading things - which still takes days, but not 100s of days. Anyway, this is one way to get an off-site backup, lower your b/w requirements on your personal server, and even get someone else to do things like make thumbnails, multiple sizes / magnifications, allow (paid) prints to be made, etc., etc. -- TTFN, patrick
On Sat, Aug 13, 2011 at 10:44 PM, Jared Mauch <jared@puck.nether.net> wrote:
My home backups are somewhat large and not yet offsite due to their size. (~4.7TB).
I've considered just subbing to backblaze as it's "cheap" on a single-host basis, but need something closer to ~5-7TB plus some room for growth (maybe 250G-500G/year).
I wrestled with this same problem when I started using Backblaze, and yes, my initial full backup took about six months, but it did get done. Backblaze doesn't have a way to prioritize, but the way I dealt with it was to exclude the media directories from the backup for the first month or so while the more important stuff got backed up, then drop their exclusion and let them start trickling up. The 500G/year of new data shouldn't be a problem, since that's under 2G/day, which should not be a problem to push up over a decent connection. I didn't worry during the initial six months since I also have some backups made to bare 2tb drives in my safety deposit box (I use one of those desktop adapters you can drop bare drives in). All backups pass through the windows XP box since that's the gateway to backblaze. It has 4x2tb drives on it. No raid or anything since it is a copy itself, just 4 separate volumes. All of the "most important stuff" lives on C so if I had to switch backup services I can exclude D/E/F from backup until C is done. To keep power usage reasonable drives are slower RPM and also set to spin down when idle 15 mins.
My home backups are somewhat large and not yet offsite due to their size. (~4.7TB).
We (NAC) run a rather large ZFS thing to sell cheap 'scratch space.' When I say large, I think it surpasses well over 100 TB at this point. So for me, it was easy. At home, my stuff spins on disk (nice big green drives, so relatively low power too) on a Win2008 Server box. So, I bought a very cool product, "Super Flexible File Synchronizer" which is unbelievably cool for what it costs. It can copy from anything, to anything, and has a gazillion options. I have what is called "synthetic backups" enabled, which is a real time (it watches at the OS to disk level), as when it sees a file change, it gets copied over the other side in a few moments. It does zip-before-copy (with encryption), it does revisioning, it does delayed deletes, etc.. Perfect for home.
--As of August 13, 2011 2:12:24 PM +0900, Randy Bush is alleged to have said:
charles skipped what i see as a highly critical question, personal backup.
--As for the rest, it is mine. Personal system: Important files are on the fileserver, on a RAID-Z volume. It's backed up nightly using Tarsnap; the keyfiles to that are on two machines, and two USB drives. (Including an Ironkey in the firesafe.) Tarsnap is worth looking at if you are looking at an offsite backup for small-scale use: On a per-GB basis, it's not especially cheap, but it bills on a per-byte basis, so if you aren't storing hundreds of GB, it's cheap. Daniel T. Staal --------------------------------------------------------------- This email copyright the author. Unless otherwise noted, you are expressly allowed to retransmit, quote, or otherwise use the contents for non-commercial purposes. This copyright will expire 5 years after the author's death, or in 30 years, whichever is longer, unless such a period is in excess of local copyright law. ---------------------------------------------------------------
participants (12)
-
Alex Rubenstein -
Daniel Staal -
David Ramsey -
Dorn Hetzel -
Jake Khuon -
Jared Mauch -
Jeffrey S. Young -
Jeroen Massar -
Joe Greco -
John Levine -
Patrick W. Gilmore -
Randy Bush