I was probably unnecessarily conservative. I could switch to raid 5 and gain some space if necessary in the future. Thanks
Need help with deciding on a new external desktop drive
- Thread starter Viathelens
- Start date
If you would like to post, you'll need to register. Note that if you have a BCG store account, you'll need a new, separate account here (we keep the two sites separate for security purposes).
Thank you Patrick!
Wow I just reread all that and my head is spinning.
the moral to me is the following:
1. Stick with my 8 tb high speed SSD drive for processing and current year images. Move older stuff onto slower large capacity drives.
2. Watch closely my storage space. Long before I run out take action to find a solution to go to the next level. Don’t procrastinate.
3. i am going to stick with DAS. I don’t need any more
than one work station and I don’t need the extra expense and complexity. I recognize that Synology may be a solution that might be more of an appliance so less strain on the brain.
4. If I expand my storage I am also going to have to expand my backup capacity as well. If I need a 40 tb net storage I am also going to need 40 tb in a backup drive.
5. Never rely on a raid to also be its own backup. The backup needs to be completely a separate drive or separate raid.
6. OWC is a good source for decent quality RAID storage for DAS.
7. At some point I need to weed the garden. I need to cut out the crap images that will never go anywhere. I need to be ruthless. There is only so much one human being can review in any sane amount of time.
8. Multiple drives for old stuff is ok keeps it organized. Get one drive to store and a second for backing it up.
9. Everything backed up to the cloud.
NOW GO OUT AND SHOOT
the moral to me is the following:
1. Stick with my 8 tb high speed SSD drive for processing and current year images. Move older stuff onto slower large capacity drives.
2. Watch closely my storage space. Long before I run out take action to find a solution to go to the next level. Don’t procrastinate.
3. i am going to stick with DAS. I don’t need any more
than one work station and I don’t need the extra expense and complexity. I recognize that Synology may be a solution that might be more of an appliance so less strain on the brain.
4. If I expand my storage I am also going to have to expand my backup capacity as well. If I need a 40 tb net storage I am also going to need 40 tb in a backup drive.
5. Never rely on a raid to also be its own backup. The backup needs to be completely a separate drive or separate raid.
6. OWC is a good source for decent quality RAID storage for DAS.
7. At some point I need to weed the garden. I need to cut out the crap images that will never go anywhere. I need to be ruthless. There is only so much one human being can review in any sane amount of time.
8. Multiple drives for old stuff is ok keeps it organized. Get one drive to store and a second for backing it up.
9. Everything backed up to the cloud.
NOW GO OUT AND SHOOT
I have one remaining question I need answered.
Lets say I buy say an 8 bay OWC RAId enclosure and put in a RAID 5. I put four drives in initially. Can I later add drives to expand storage one by one or does the whole thing have to be redone from scratch?
Or does that only work with Synology?
I will feel well-rounded and fully informed when I know the answer to that question.
Lets say I buy say an 8 bay OWC RAId enclosure and put in a RAID 5. I put four drives in initially. Can I later add drives to expand storage one by one or does the whole thing have to be redone from scratch?
Or does that only work with Synology?
I will feel well-rounded and fully informed when I know the answer to that question.
I was able to find the answer to my own question.
WIth OWC if you add a drive to an existing RAID box you are going to have to completely erase and restore the whole drive. This means you will have to have some place to put everything.
I would think the way to do this is to first back up the whole RAID data. Then switch the drives, reformat everything and restore the drive from the backup. You better know what you are doing when you do this.
Another way would be to get another RAID enclosure and populate it with larger drives, then you can copy all of your data from the earlier drive onto your larger RAID.
then you can go back to your first RAID and put bigger drives in it. It can then serve as the new backup,
If I have this wrong someone let me know.
WIth OWC if you add a drive to an existing RAID box you are going to have to completely erase and restore the whole drive. This means you will have to have some place to put everything.
I would think the way to do this is to first back up the whole RAID data. Then switch the drives, reformat everything and restore the drive from the backup. You better know what you are doing when you do this.
Another way would be to get another RAID enclosure and populate it with larger drives, then you can copy all of your data from the earlier drive onto your larger RAID.
then you can go back to your first RAID and put bigger drives in it. It can then serve as the new backup,
If I have this wrong someone let me know.
Does anyone have thoughts on "bit rot" in the present context?
Thanks for this info. I am of limited experience, but my understanding about Synology is that I can add drive(s) without erasing my data on the existing volume. But certain criteria must be met as outlined in this article: https://kb.synology.com/en-global/DSM/help/DSM/StorageManager/storage_pool_expand_add_disk?version=7
I think I added a 5th drive to my 4-drive RAID 6 Synology NAS (Diskstation ds1522+) before I populated it with my data. But I'm pretty sure I could have added a drive without erasing anything, according to the article. I know that I can easily replace a failing drive. The NAS runs a while to redistribute the data onto the new drive added to the array. The added drive is re-formatted in the operation.
I read in the same article under another section that the entire Synology NAS can be upgraded to a larger size by adding larger drives one by one. The NAS needs to be in "Healthy" status and it will take a while, but is fairly automatic. And backing everything up beforehand is always recommended.
You’re right…which for us on this forum…as I’ve said all along…there’s nothing wrong with a NAS, but DAS provides the same capabilities for lower cost along with an SSD for the catalog…and while the NAS and 10GB Ethernet might be slightly faster (and I didn’t run the numbers, but Thunderbolt is pretty fast)…the difference in LR performance for the vast majority of people on this forum just isn’t worth the extra cost…unless one values the appliance approach more and isn’t interested in managing the same capabilities on a DAS.
I’ve asked OWC this question…waiting on tech support to get back to me.
Edited several days later…got a response from them that expanding the RAID with SoftRAID requires backup, redo RAID and restore. That will actually be faster (probably) than replacing the drives one at a time with the requisite RAID repair the NAS software does…and it requires backing up the contents of the RAID beforehand but you’re doing that anyway I hope. So…it is easier bit slower to use the NAS and upgrade a drive at a time…but given I’m going to backup either the RAiD or NAS separately anyway…my personal recommendation is still that for home use the NAS is a more expensive way to provide the varied services it provides than direct attached storage…which one should choose depends on whether you want an appliance or are computer savvy enough to enable those services on your own which isn’t really difficult at all. Where NAS or it’s big brother the SAN make sense is for large numbers of clients (10s to 1000s) and on a network that supports 10GB Ethernet.
Last edited:
John Navitsky
Well-known member
Note this is the case for MOST RAID solutions, especially direct attach based ones. So this is typical.
Yes, but the problem becomes what do you back it up to and how long will that take.
This is probably one of the more straightforward strategies. And I'll also mention, once you start getting to large file sizes, you don't have to have a SINGLE place for backups. Often you can back up some stuff to the first RAID, and some stuff to the second RAID. Same with your source stuff. I have THREE raw image drives, and i simply break things up across them by date.
---
Do note NAS systems often can solve this problem because they are using more sophisticated systems inside. For example ZFS based systems have "sets" of RAID or mirrors, so you can simply add another "set". So if you have an existing 3 or 4 drive configuration, you can add another 3/4 drives and add a second set and double your space.
Systems based on UNraid have a way to add space, although I'm not super clear on how it works.
DAS with Windows storage spaces will let you add space as well. Although note I don't have direct experience with it and there was a comment about it, above, that suggested it may have some teething issues.
John Navitsky
Well-known member
Interesting sidebar
There seems to be a variety of different LR things that have an impact on performance, and I haven't completely figured out the exact nature of each.
1) the LR cache. you specify location and an upper size limit (max 200GB). i'm not 100% sure what *exactly* goes there, there was some talk of image previews, but it's clear not ALL image previews go here (and previews is a whole topic onto itself!)
2) the catalog. as @Anjin San suggests, you probably want this on a SSD and probably a FAST ssd (nvme). one gotcha here is the catalog directory also includes some previews (?!?!), so it can get a lot larger than it seems would be necissary. there also seems to be some upper limits where a performance tends to get slow even on a fast drive
John Navitsky
Well-known member
the following video covers bit rot. the tl;dr is that most current RAID systems don't have protections against bit rot. some approaches like ZFS can protect you from bit rot in most situations.
The cache and previews are in the same location as the catalog unless one specifically aims them elsewhere. I know that Hudson Henry puts the cache by itself on a separate SSD than his catalog…but then he has the catalog on a different external SSD and putting the cache elsewhere forces him to take both externals as well as backup drives when he is away from home…and I’ve never seen any testing that compares catalog and cache on the same fast SSD to then being on separate fast SSDs…my guess is that the performance will be essentially identical as far as practical differences are concerned,
I keep catalog and current year originals on a 4TB SSD and create both Smart and 1:1 previews on import with the latter being deleted after 30 days because I’m usually done PP on an import by then. With about 40K Z8/9 images, catalog, and previews on the 4TB drive take up 1.6TB. I would have bought an 8TB one but OWCs portable TB SSDs don’t come that big and I take the master catalog drive with me on travel along with 3x 4TB Samsug T7s for backup. If I take t(ex laptop I can but usually do not import, just copy to the master drive and clone that to the backup drives…and I have a powered hub that allows me to do the same thing if I only have the iPad although it is a little slower.
Anjin, (original thread starter here). I'm still working on what to buy and now thinking, like you said above, of getting an SSD drive, 4TB OWC desktop thunderbolt, and a 32TB ThunderBay4 HDD thunder bay 3/4. As you note, I would download current import to SSD and then move it at some point to HDD. I keep my LrC catalog on my Mac Studio that has a 2TB internal SSD. I don't travel with my catalog, I use the export/import as catalog function when I return home.
I'm also curious if anyone knows anything about the OWC miniStack STX external drives on the OWC site? They are thunderbolt 4, I was looking at these if I did not go with a RAID system. They have a 28TB with 20TBHDD and 8TB SDD and this seemed like it might work, too, although then I'd need similar sized backup HDD drives.
Still not decided at this point but great comments on all this so far from everyone. Very helpful. Comments?
John Navitsky
Well-known member
i think the primary benefit of the ministack is the physical form factor. so if you're pairing with the mac mini, they can fit in nicely.
but if you're just looking for an external drive, i might look at the owc 1m2 (for a single) or the 4m2.
but it should be fine also
I thought about the travel catalog idea as well…but keeping keywords in sync means more work…so I ended up with the external for the catalog and bought a pair of them, only one goes on travel and the daily clone one stays home. I actually mostly don’t actually import on travel anyway…but like being able to do so if there is enough downtime (there isn’t in Africa).
The mini stack is primarily aimed at Mini users as it is the same size for sitting underneath. The problem with it for me is that being a non RAID there is no redundancy…with ThunderBay provides a higher reliability for the main storage drive…no reason you couldn’t just do a pair of 20 TB or larger enterprise drives and clone for redundancy…except they are mostly USB and thus slower…but with import and catalog on an SSD the speed reduction isn’t much of a deal, I notice that looking at or editing files from older years out on the spinning drive is slower but it’s incrementally slower, not drastically. One additional idea for the catalog on TB external for me is that if I decided to just use a laptop as my main computer and plug it into a big monitor and TB dock at home if needed it’s just a matter of moving the drive. If I ever get off my lazy butt and run an Ethernet cable from the cable connection to the office…I could retire the 2013 mini out here in the entertainment center, move the Studio and its associated externals and RAID out here, and leave the Studio Display in the office and carry the laptop back when PP is happening. When we lived in the RV for 8 years…the catalog was on my laptop and all the originals were on a drive in the overhead with wifi connection…but I was dealing with smaller and fewer originals back then.
There really are a lot of right answers to this question…the issue with the SSD/spinner combo in the ministack is figuring out how you want to configure things…but with one of those and a separate USB 20TB drive to clone o from the spinner for redundancy…thats one of the many right solutions. But think medium to long term…when I got the ThunderBay mini (less noise)…I went for the largest drives I could even though it gave me 12TB and I didn’t need that much I wanted space to grow…and as it turned out 45MP bodies and 20FPS came along…but I still only have about 6TB filled after 4+ years and only about 2.5 of that is images. The rest is network backups of the laptops and the several hundred CDs we ripped before moving into the RV where space was limited. You can’t get bigger drives in the 2.5 inch size…so if I need more space I will have to get the 3.5 drive version and suffer with the slight extra noise…and as the drives in mine age and start to fail I might upgrade anyway…one really needs to replace spinning drives after 5-6 years anyway as a prophylactic measure and mine will be 5 years old in Feb. but OTOH…I’ve got several 10 year old Seagate 2.5 inch USB Backup+ drives that have been running continuously that long and are still going…but the old sysadmin in me starts to get hinky at the 5 year point or so…and my main local backup drive is only a year or so old 20TB one from OWC.
John Navitsky
Well-known member
i think the optimal solution for this problem is a 1:1 between drives and catalogs, with the catalog ON the drive
Thanks. I don't keyword when I travel, just view and sometimes process and it's very easy to what is required to export/import metadata when I return home so I could do keywords and they would simply import into the desktop over the image file. I'll have to continue to ponder this, a lot of info. I'm pretty sure it's time to change drives, in addition to the fact that I have no space.
Well…thats true…but really big SSDs are still pretty expensive so we have to make do
.groob
Active member
I’ve never understood the need for local redundancy when you use an online backup system like Backblaze. What’s the point of local redundancy? Seems like added cost and complexity for no benefit.
John Navitsky
Well-known member
you still need more than two copies and remote storage is expensive
groob
Active member
Why do I need more than 2 copies? Because my hard drives and my cloud server might fail simultaneously? That possibility seems wildly remote. Also, Amazon offers unlimited photo storage with a Prime account. I am likely going to switch to it.
John Navitsky
Well-known member
best practices suggest:
at least three copies, at least one of which is off-site
to answer your question, it's surprisingly easy to loose two copies. also remember many failures are not because of hardware failure, but often due to other things like human error. for example, you accidentally over-write something and then it gets backed up, now both your copies are bad.
Remote storage…either BackBlaze or DropBox or Sync or whatever is not expensive. If you've got a lot of images then there may be a lengthy time period to upload them all…and there may be monthly bandwidth caps that trip you up and cause some extra expense…but any of those solutions is around $100 a year and you're already paying the Internet bill anyway so that doesn't really count.
What John said re backups. And if your single backup consists of a daily or weekly or whatever clone job…and you don't discover the issue with the file you have to have until after the corrupted one is synced to the backup…you're screwed. BackBlaze or DropBox or something with versioned backup history saves your bacon in this case…and while one could do it with DB, BackBlaze makes it easier.