I’m thinking of backing all of my family’s digital assets up. It includes less than 4 TB of information. Most are redundant video files that are in old encodings or not encoded at all and there are a lot of duplicate images and old documents. I’m gonna clean this stuff up with a bash script and some good old manual review, but first I need to do some pre-planning.

  • What’s the cheapest and most flexible NAS I can make from eBay or local? What kind of processors and what motherboard features?
  • What separate guides should I follow to source the drives? What RAID?
  • What backup style should I follow? How many cold copies? How do I even handle the event of a fire?

I intend to do some of this research on my own since no one answer is fully representative but am appreciative of any leads.

  • @Majestic@lemmy.ml
    link
    fedilink
    4
    edit-2
    1 day ago

    If you’re just backing up and not serving this data just get 2-3 4TB drives (new, recertified, or used) and an external dock and test the drive then back it up then test again and check SMART both times. Place one drive with a relative or trusted friend. Connect and power up each of the drives at least once annually, refresh the data with anything new at that time and check the smart stats, consider running at least a quick SMART test to ensure none are mechanically failing then back to being unplugged. Really every 3-6 months would be ideal to power on and check SMART but I wouldn’t pester a relative that often for the external one, 1-2 times a year should be fine for that.

    This strategy protects you from cryptolocker malware by not leaving any of them live and accessible.

    • What’s the cheapest and most flexible NAS I can make from eBay or local? What kind of processors and what motherboard features?

    Cheapest or most flexible, choose one. If you want absolute cheapest but not that flexible you can buy a used office PC, a Thinkcenter or Dell optiplex are the most reliable ones though depending on the model they may accommodate anywhere from 1 to if you’re lucky 4 (though commonly only 2) drives via that many SATA ports (often half the SATA ports are 1.0/2.0 for DVD drives so you may not get full speed). Finding space inside them for more than 1 drive could also be a problem depending on form factor but mid-tower models often have room for 2 with space for a third lying on the case itself if you really want to push it.

    Most flexible I suppose someone else’s old NAS build, a used case with room for at least 4 3.5" drives gives you a little room to expand.

    • What separate guides should I follow to source the drives? What RAID?

    You don’t need RAID, it’s not a back-up solution, RAID is for high data availability and integrity. If you really want to you can set-up a RAID 1 I suppose though know this means you’d require at minimum 4 disks for your data and one copy and 6 disks for two copies.

    As to sourcing the drives, there are various companies, server parts deals is one that’s well known and decent though their presently available sizes may be larger than what you’re after. No matter whether the drive is brand new, recertified or bought used on ebay the recommendation is test, test, test. Even new drives can be bad. Run a full SMART test at least once, check the SMART data and make sure there are no failure indicators. If you want to be really thorough I’d suggest checking the SMART data when you get it, noting anything concerning, running an extended/full SMART test then after that finishes formatting the drive but unchecking quick format and doing a slower format option that writes zeros across the drive, then filling with your data, then doing another full/extended SMART test and again checking the SMART values before putting it away. Re-test and check SMART at least annually if you’re keeping the drives cold.

    • What backup style should I follow? How many cold copies? How do I even handle the event of a fire?

    At least two copies, ideally three, at least one copy off-site for things such as fire. If you don’t have a relative, friend, or workplace where you can stash an off-site copy your option would be basically cloud storage back-up which for 4TB wouldn’t be too costly (backblaze personal would allow this much IF you keep one copy connected to a computer that has their app and is turned on at least monthly and they’re $100 a year though note they will delete your data if you go more than 30 rolling days without syncing so if there is a disaster you have a limited time to either get another drive and download it again or contact them and pay to have a copy shipped to you before it’s deleted).

    You could also I suppose invest in a fireproof safe though that doesn’t protect against burglary where they steal your safe thinking it has valuables in it. You really need a copy off-site. Other options would be a bank safe deposit box though probably more costly.

    One way to get friends to help is to buy more storage space than you need, say two 8TB drives and you offer to back-up a copy of their stuff at your house so you have a copy of their stuff+yours at your house and they have the same copy at theirs. Though you could also use separate drives.

    Most are redundant video files that are in old encodings or not encoded at all

    All re-encoding unless it’s from lossless to lossless induces degradation. For archival purposes I’d suggest against re-encoding unless it’s to another lossless format or unless they’re in a lossless format or very high bitrate (>20MBps video for SD or 1080p HD) and you’re keeping a high bitrate in the new encoding. Also avoid hardware encoding, it’s faster but introduces more degradation and is less precise than software encoding. Removing duplicates is another matter.

  • @ragebutt@lemmy.dbzer0.com
    link
    fedilink
    English
    72 days ago

    4tb is nothing in the days of 20tb hdds

    If you want a NAS you don’t need a beefy pc. I started mine with an ewaste office pc (like literally a $40 2013 pc from a doctors office). I now have a much beefier setup that is still all recycled hardware, 10th gen intel build, but mainly because I wanted to add wayyyyyy more drives and do stuff with vms and local llms and such. otherwise you really don’t need “power” and having it is actually detrimental as it will cost you more money in electricity to run (also the environment)

    Keep in mind a NAS/raid is NOT BACKUP. It is far more resilient to keep your data on a raid array of 3-4 4tb drives than simply keeping it on a second external drive that might just die at any moment. But all it takes is one day where a drive in the array dies and then a parity drive dies during rebuild and then poof, your data is gone.

    You could do raid 1 with like 3 mirror disks but this is excessive and you could still get got by various things: bitrot, house burning down, power surge, controller failure, you fuck things up, etc

    A proper backup solution is necessary if the data is critical.

    • @IronKrill@lemmy.ca
      link
      fedilink
      2
      edit-2
      1 day ago

      a NAS/raid is NOT BACKUP

      To emphasise your point, I recently had a scare with corruption in my ZFS pool causing me to have to transfer all my data off and start the pool fresh. I’ve since begun viewing any RAID pool as a single drive, which helps see the situation more accurately. Instead of thinking “I have 6 drives with 2 parity, I can withstand 2 failures!” think “I have one pool, I can withstand 1 failure.” Because the moment anything about that pool breaks you are shit outta luck. Prevention against single drive failures is only one part of the puzzle.

      • @ragebutt@lemmy.dbzer0.com
        link
        fedilink
        English
        3
        edit-2
        1 day ago

        I had a time where the dreaded situation people warn about happen: a parity drive failed during rebuild to replace a drive that had failed. Rebuild operations are stressful on drives. And if you’re like me in a home setup you probably aren’t swapping out drives preemptively, you’re waiting until smart errors occur because you’re not made of money and the data isn’t truly critical

        There are many ways to pursue proper backups (to op), backblaze works for me (though there are some that are more privacy focused, not that backblaze is terrible on that front). I also have tape backup but this is overkill and $$$, really only worthwhile to pursue if you have a gigantic nas and also are good at tinkering (can refurbish cheap drives sold as broken). There are a number of other options too

  • @Pfeffy@lemmy.world
    link
    fedilink
    11 day ago

    Check how much of that stuff is actually needed to physically back up. Odds are it’s on the cloud already. I got a house calls for this kind of thing a few times a day and almost always the answer is "your Apple/Google/Microsoft storage has all your pictures and videos already.

    I get people that ask me about setting up network storage arrays pretty often, but almost none of them want to go through with it.

    I don’t know what your time is worth, and I would still set something up just because I felt like it or was curious, but almost nobody needs a home network storage system. Just figure out what people need to sync, set it up, and plug a hard drive into your router. Odds are once you try to figure out what they are trying to sink, you’ll realize that it already has automatically synced to their cloud storage.

    If you really want to do it as a project and are near Maryland, I will give you some closed box Network storage arrays from 2012 with hundreds of terabytes between them in unused fiber optic hard drives.

  • @rutrum@programming.dev
    link
    fedilink
    English
    42 days ago

    The two top of the line tools for making backups are restic and borgbackup (in my opinion). They would allow you to easily compress and encrypt some local directories to another computer or cloud service. I personally use borgbackup with external backups at borgbase.com

  • @Dudewitbow@lemmy.zip
    link
    fedilink
    42 days ago
    • What’s the cheapest and most flexible NAS I can make from eBay or local? What kind of processors and what motherboard features?

    cheapest is some decade old oem desktop ideally with as many sata ports as possible. the most flexible is whatever gives the most pci-e lane bandwith as they can be converted to most things. processor features most dont matter unless the NAS is also a media server, which you want an igpu that can do hardware encoding to whatever usecase you have.

    • What separate guides should I follow to source the drives? What RAID?

    i dont have a reccomendation on drive, but if you value drive redundancy, raid 1 (mirror) or raid 6(if youre using zfs, that would be zfs z2, this layout is basically requires 3 drives to die in order to lose data)

    • What backup style should I follow? How many cold copies? How do I even handle the event of a fire?

    its on you on how you want to handle offsite backups be it cloud, or you having a clone that you manually backup offsite. pick whatever suits your needs

    • @IronKrill@lemmy.ca
      link
      fedilink
      11 day ago

      zfs z2, this layout is basically requires 3 drives to die in order to lose data)

      RAIDZ2 has 2 parity disks and thus can only withstand 2 drive failures, no?

  • @arcosenautic@lemmy.world
    link
    fedilink
    English
    22 days ago

    Whatever you decide to do, please implement 3-2-1 backups (3 backups: 2 backups offline, 1 offsite). This will cost more obviously, but family data is important and its safety shouldn’t be skimped on.

    • @catloaf@lemm.ee
      link
      fedilink
      English
      42 days ago

      The 2 isn’t two offline, it’s two forms of media. Hard drive, optical, tape, etc., in case there’s some defect or disaster that renders one unusable. The offsite one should probably be offline, but unless you’re really paranoid, a cloud backup counts as one for each category. (But it needs to be a managed cloud backup, not just a copy in Google drive, because you want to be protected against accidental deletions.)

      • @arcosenautic@lemmy.world
        link
        fedilink
        English
        115 hours ago

        I see, thanks for clearing that up. In my situation, keeping a few terabytes on 3 hard drives and backing up to a reliable cloud copy (not sync, as you mentioned) is good enough for me. Optical media is impractical with the amount of data, and tape drives far too expensive for just a few TBs.

  • @liliumstar@lemmy.dbzer0.com
    link
    fedilink
    English
    52 days ago

    I would just backup to an external drive or whatever locally. If you want a NAS, a simple SBC will do. If you really want RAID, get an old optiplex and hook up a couple drives in RAID-1.

    For drives, get some CMR WD drives under warranty.

    Keep one copy encrypted in the cloud, one offline local, another offsite not overly near you.

    • @andioop@programming.dev
      link
      fedilink
      English
      117 hours ago

      Also doing something pretty similar to OP and do not anticipate needing any more than 4TB for awhile, first I have seen of external drives being approved. (Only own laptops, very intimidated by all this SATA stuff right now—am new and every time I try to learn more on r/datahoarders I feel slammed by information overload.)

  • @catloaf@lemm.ee
    link
    fedilink
    English
    42 days ago

    4 TB isn’t that much data. Do you even need a NAS, if you’re not planning on serving the media? If you have a desktop PC, just stick a 4 TB drive or two in there, and keep an offsite copy either by cloud backup or external drive at someone else’s house (or bank safe deposit box if you want to be really sure).

    • I prefer keeping drives like this external so I can turn them off when I’m not using them. No need having the disks spin up just because my machine woke from sleep or something decided to access it. Plus should you get some malware on your computer if the disk wasn’t turned on it shouldn’t be affected.