Need help with deciding on a new external desktop drive

[Anjin San]

Someone can correct me if i am wrong but I am assuming Synology devices are set up at NAS only and they are not the best choice if you are going DAS.

That begs the question, what IS the best quality equipmentN for a DAS RAID?

Yep…you can’t hook up a NAS directly…it has to be network connected. For DAS…as long as you have Thunderbolt I recommend OWC…been buying from them for decades and they have excellent support. Their ThunderBay comes in 4 and 8 drive versions using 3.5 inch drives…the mini uses 2.5 inch, isn’t as fast but is fast enough the way I use mine, and is quiet…but only goes up to 4x 4TB drives for 12 TB once RAIDed. I’m sure there are other brands of RAIDs…but no experience as I’ve stuck with OWC products for the same reasons I use Apple ones…value and price/performance/reliability.

 

[Anjin San]

Dropbox!? I hate Dropbox. It’s a mess and I’m sure not using it as my image repository.

Don’t know why you say it’s a mess…but Sync then is another alternative and there are others, as well as direct sharing of the local drive via VPN services. I was just answering the question. Of one is using a NAS and connectin* 9r sharing across the internet…the vendor provides the VPN and DNS related services to connect the various users as your home IP changes.

 

[Anjin San]

it’s a bit philosophical and probably not for people not super into computers, but i think this is a good discussion about problems with current hw raid

one thing you might glean from this is hw raid isn’t the panacea people may assume

the other thing you might take-away is there is no free lunch

I agree…at this point in time any reasonably current computer can do software RAID just fine…and if you really need the hundreds to thousands of simultaneous connections then you’re going to use a big file server or NAS or SAN anyway…most likely the latter for enterprise solutions. B7t for us mere mortals…software RAID is the most cost effective solution.

 

[wotan1]

So I am not a tech expert I am just a guy who has to use computer equipment to store an expanding volume of images and I also need a reasonably efficient photo processing system with higher resolution monitors.

I recently switched from PC to MAC. I made the decision because at the time MAC with the new silicone chips could deliver better performance with higher resolution monitors for less cost. It was the right decision for me at the time and I am not dumping on PC in the photo processing world.

Now that I am in MAC I find I need faster and larger external drives to manage my processing. Higher speed SSD drives work well for this purpose, but individual SSD drives seem to be limited in size. The latest OWC drives apparently max out at 8 TB. I have a fast 8 tb ssd drive and it is working well for me so far but with these latest camera systems I am rapidly accumulating images way beyond what I ever did before getting into the Z8/9 world. God forbid I add a 150 mp medium format camera.

I am told not to bother with RAID just get drives that attach directly and use thunderbolt 4 or better yet USB 4 drives.

So what happens if eventually 8 tb is not enough? Do I then divide up my photo world among multiple individual drives? As a non-computer expert I do better if everything is in one place where I can easily find it. Do I now have to learn to manage the multiple drive storage solution? Or do I break down and do a RAID with multiple SSD's when things get out of control? Is there a software way to combine individual drives into one for people like me to manage?

I am OK for now but those are my dilemmas looking forward.

 

[Anjin San]

So I am not a tech expert I am just a guy who has to use computer equipment to store an expanding volume of images and I also need a reasonably efficient photo processing system with higher resolution monitors.

I recently switched from PC to MAC. I made the decision because at the time MAC with the new silicone chips could deliver better performance with higher resolution monitors for less cost. It was the right decision for me at the time and I am not dumping on PC in the photo processing world.

Now that I am in MAC I find I need faster and larger external drives to manage my processing. Higher speed SSD drives work well for this purpose, but individual SSD drives seem to be limited in size. The latest OWC drives apparently max out at 8 TB. I have a fast 8 tb ssd drive and it is working well for me so far but with these latest camera systems I am rapidly accumulating images way beyond what I ever did before getting into the Z8/9 world. God forbid I add a 150 mp medium format camera.

I am told not to bother with RAID just get drives that attach directly and use thunderbolt 4 or better yet USB 4 drives.

So what happens if eventually 8 tb is not enough? Do I then divide up my photo world among multiple individual drives? As a non-computer expert I do better if everything is in one place where I can easily find it. Do I now have to learn to manage the multiple drive storage solution? Or do I break down and do a RAID with multiple SSD's when things get out of control? Is there a software way to combine individual drives into one for people like me to manage?

I am OK for now but those are my dilemmas looking forward.

Yeah…when you get past 8TB today you can’t get a bigger SSD. So that means multiple drives hooked up and separated in LR in some fashion…or a RAID…or a NAS and 10GB Ethernet…or a combination which is what I do. 4TB TB SSD has catalog, previews, and current year images. Older year images are on a TB ThunderBay mini which i chose since 2.5 inch drives are quiet and 3.5s aren’t…but that limits the RAID to 4x4TB currently or 12 TB in RAID 5. If I run out of space there…then if larger 2.5 inch drives are not viable I will get the larger ThunderBay and a longer TB cable and stick it in the closet near the computer desk. They.l still be noisier than the 2.5s are…but the closet door will help.

 

[John Navitsky]

So what happens if eventually 8 tb is not enough? Do I then divide up my photo world among multiple individual drives?

yep, that's what i do. you'll see here where i have "raw1", "raw2", "raw3". each of those is an internal 8tb ssd, backed up to a second 8tb ssd, then in turn backed up to a tertiary storage as well as cloud (no raid involved until you get to tertiary storage)

You can only see EXIF info for this image if you are logged in.

As a non-computer expert I do better if everything is in one place where I can easily find it. Do I now have to learn to manage the multiple drive storage solution?

i do it all in LR. admittedly, there's enough images in the system to start to tax lightroom's ability to manage this in one catalog...

Or do I break down and do a RAID with multiple SSD's when things get out of control?

the problem is, if you can't fit everything in three 8tb drives, putting that together in a 24tb clump just creates other problems for you.

Is there a software way to combine individual drives into one for people like me to manage?

under pc, you can mount whole drives as directories. i'm guessing you can do this on mac as well. but... as i said, i don't think having them 'visible' as different drives really creates that much of an issue.

 

[John Navitsky]

Yeah…when you get past 8TB today you can’t get a bigger SSD.

technically you can. you can buy u.2 ssd drives up to about 30TB, although obviously they're pretty expensive.

u.2 basically takes the same electronic interface as m.2 nvme drives, and you can even get adapter cables to go from m.2 headers to u.2. alternately, you can put a u.2 (or two u.2, or even four, but it's a long card and you need x16 slot) drives on a pci card and just plug them in. basically it works the same way as a m.2 to pci adapter, basically the interface is just 4x pci lanes.

the pci adapter is probably the easiest way for most of us, but OWC does have external cases that can take u.2 drives via Thunderbolt

my u.2 boot drive on an inexpensive pci adapter card:

You can only see EXIF info for this image if you are logged in.

 

[Viathelens]

John and Angin, do you think you might want to start your own thread for all of this technical detail and maybe someone will answer my simple question for me? Thanks.

 

[John Navitsky]

apologies.

i think the thunder bay would be a reasonable choice given your statements

if they say that’s max speed, they probably measure that striped across four disks

so just based on that, i’d expect about half that if you run four disks, with two mirrored sets (a stripe of mirrors)

about half that again if you only run two disks in a mirror

 

[Viathelens]

apologies.

i think the thunder bay would be a reasonable choice given your statements

if they say that’s max speed, they probably measure that striped across four disks

so just based on that, i’d expect about half that if you run four disks, with two mirrored sets (a stripe of mirrors)

about half that again if you only run two disks in a mirror

Two disks with data, two disks as duplicate. Would the speed still be considered fast?

 

[John Navitsky]

typically mirrors can be reasonably fast. best case is writes will be the same as one drive IF the link is fast enough push both dives at the same time (and thunderbolt should be). reads theoretically can actually be faster because data can be fetched from either drive but i would expect it to simply be similar to having a single drive

you will basically double the speed of a mirror when you stripe across two mirrors. this configuration is called raid 10, or a stripe of mirrors

 

[Anjin San]

technically you can. you can buy u.2 ssd drives up to about 30TB, although obviously they're pretty expensive.

u.2 basically takes the same electronic interface as m.2 nvme drives, and you can even get adapter cables to go from m.2 headers to u.2. alternately, you can put a u.2 (or two u.2, or even four, but it's a long card and you need x16 slot) drives on a pci card and just plug them in. basically it works the same way as a m.2 to pci adapter, basically the interface is just 4x pci lanes.

the pci adapter is probably the easiest way for most of us, but OWC does have external cases that can take u.2 drives via Thunderbolt

my u.2 boot drive on an inexpensive pci adapter card:

I thought we were talking externals…but yeah, bigger ones are still godawful expensive.

 

[Anjin San]

John and Angin, do you think you might want to start your own thread for all of this technical detail and maybe someone will answer my simple question for me? Thanks.

Apologies as well…thought we had.

For storage space…get either a ThunderBay mini with 2.5 inch drives or the regular ThunderBay with 3.5 In. The mini will currently only get you to 12TB at RAID5 but is quieter and cheaper. The non mini is louder and can provide more space. Either will be adequately fast for LR with the 3.5 model being faster…but neither will be as fast as the internal SSD or an external TB SSD. This is why I have 2 drives for L….catalog and current year originals live on a 4TB Thunderbolt OWC drive and older year images get moved every January to a ThunderBay mini which I chose for quiet over capacity…if it ever runs out then I will either archive old originals on another catalog, get bigger 2.5 inch drives for it or even SSDs if they’re available and iTB each, or get the full size ThunderBay and stick it in the closet on a longer cord or just live with the noise.

OWC ships SoftRAID with the drives…and I think it will do striping which would increase speed as well as the more typical RAID5…but then the capacity becomes an issue…which isn’t a problem with the more drives you can get in the full size ThunderBay…but then it is more cost…which was my reasoning behind my LR mixed SSD and spinning drives system. Steve also has a ThunderBay IIRC…and I believe it’s got spinning drives but he keeps the catalog on the internal SSD, we were chatting about setups in Serengeti in April.

I would put SSDs into the ThunderBay mini and get speed…but cost prohibitive for space I don’t need now amd AFAIK,you can’t get anything bigger than 4TB in the 2.5 form factor…so I’m kicking the more capacity can down the road for now.

 

[John Woodworth]

I have had a G Raid DAS going for the past three years. Raid 1 to two 6TB platters so 6 TB of redundant capacity. Connected via TBolt.

Backed up into the cloud with BackBlaze.

I keep my LR catalog and the current project(s) on SSDs (2 TB) in the computer so I don't deal with latency of the platters spinning up in the DAS during routine processing. These folders are relatively small and are backed up in the short term on OneDrive, but the folder settings are such that a copy of the files remains on my local SSD. When they get big--say 200GB or so--I use LR to move them to the DAS where they are accommodated by Backblaze and automatically removed from OneDrive (which is expensive). LR indexes files on both drives.

All files on both drives are in the same catalog and remain "instantly" available, although there is some drag associated with the drives spinning up if you are accessing a file located on the DAS. Processing files from the DAS (as opposed to transfering them back to the internal SSDs) is workable if you are just touching up a few images from previous projects.

Hope this helps.

 

[Ralph]

Two disks with data, two disks as duplicate. Would the speed still be considered fast?

I’ve been using this setup from OWC. https://eshop.macsales.com/shop/owc-mercury-elite-pro-dual. You can purchase the enclosure with drives included or you can purchase just the enclosure and then search for the drives you want. That’s what I did. I purchased two 16gig 3 1/2 “ Toshiba drive. You can set it up as individual drives or as a RAID. I set it up as 2 individual backup drives. What’s nice about this enclosure is that it also has 3 extra usb ports for other peripherals.

 

[Viathelens]

I went around and around about NAS and other options when I was setting up my system. What I am reporting is a summary of what I learned:

1. A NAS drive is not necessary unless you are editing from multiple locations or need remote access. NAS is also more expensive to build and maintain. DAS or direct attached storage is more efficient and simpler when you do your editing from a single computer.
2. NAS is normally slower than DAS because access is governed by network speed. What Hudson Henry did was speed up network access by adding higher speed network equipment. That adds complexity and expense to the network system. None of that complexity is necessary with a DAS connection.
3. Thunderbolt connections found in later model MAC computers are plenty fast enough when used with the proper cables for a DAS connection. This becomes a much simpler and less expensive process than modifying network speed. As indicated elsewhere, a high speed SSD drive directly attached to the computer by Thunderbolt is for practical purposes as effective as an internal drive.
4. All drives can eventually fail. You need an effective backup system which includes both on site and remote backup. That way if you have a catastrophic failure you can replace the defective drive and restore the data from a backup. Backup drives don't have to be as fast as drives used for active photo importing and processing so they can be less expensive.

NOW ABOUT RAID DRIVES:

RAID means redundant array of independent drives. A RAID drive groups individual drives together and has a system for managing access to the drives. RAID drives handle the management either through software or hardware. Generally hardware RAID systems are more expensive.

How you configure a RAID drive depends on the need you are trying to serve. You can speed up access by using multiple slower drives to serve as a single combined drive (striping). Or you can use drives to mirror content to adjacent drives so if one drive fails it can be replaced without loss of data. Or you can increase storage capacity by simply adding individual drives to the RAID.

With a proper backup system a RAID may not be necessary. A simple backup management system is effective to protect your images and will work with individual drives. Here is how it works:

a. You import your images to the main processing drive as usual. Do not reformat your cards until your newly imported images have been backed up.
b. Set your backup systems up so the backup is done at least daily. Back up both to a local backup drive and to cloud storage.
c. Verify the backup has been completed. Backup systems verify the backups have been made.
d. Now you can safely reformat your data card.

With this system you know your images have been captured in three places, one of which is off site. Should a drive fail you can replace it and restore your images to that drive simply and easily.

A question on your "Now about RAID drives" and backup. Why would it be advantageous for me to physically do the backup myself versus letting the RAID system do the backup? Isn't that just adding a step that I would not have to do if I used the RAID system, which I have been using for several years now. Thanks.

 

[Viathelens]

I’ve been using this setup from OWC. https://eshop.macsales.com/shop/owc-mercury-elite-pro-dual. You can purchase the enclosure with drives included or you can purchase just the enclosure and then search for the drives you want. That’s what I did. I purchased two 16gig 3 1/2 “ Toshiba drive. You can set it up as individual drives or as a RAID. I set it up as 2 individual backup drives. What’s nice about this enclosure is that it also has 3 extra usb ports for other peripherals.

Thanks for this. I think using Thunderbolt is faster but I may be wrong, a lot of things to review on this subject! I do like the amount of storage available at a decent price. I really just need to throw a lot more images away!! I'm at over 125,000 images (correction, at 212,477!!) in my LrC catalog now and have filled 10TB of storage space.

 

[Anjin San]

A question on your "Now about RAID drives" and backup. Why would it be advantageous for me to physically do the backup myself versus letting the RAID system do the backup? Isn't that just adding a step that I would not have to do if I used the RAID system, which I have been using for several years now. Thanks.

Unless you’ve set the RAID up for mirroring…they are primarily used for larger drive sizes…and in any event they’re are NOT backups. Ypu still need to have adequate backup strategies…the 3, 2, 1 system (3 copies, 2 media, and 1 offsite) is a good place to start. The RAID also provides reliability as in the typical 5 config any drive can fail and the array is still fully functional. Striping and mirroring provide other advantages at the cost of capacity…but most 2 drive arrays are probably used as a mirror and multiple (4 or more) for capacity/failure redundancy…and as I said any of them still require backup (As does a NAS as was discussed in another thread).

 

[Viathelens]

Unless you’ve set the RAID up for mirroring…they are primarily used for larger drive sizes…and in any event they’re are NOT backups. Ypu still need to have adequate backup strategies…the 3, 2, 1 system (3 copies, 2 media, and 1 offsite) is a good place to start. The RAID also provides reliability as in the typical 5 config any drive can fail and the array is still fully functional. Striping and mirroring provide other advantages at the cost of capacity…but most 2 drive arrays are probably used as a mirror and multiple (4 or more) for capacity/failure redundancy…and as I said any of them still require backup (As does a NAS as was discussed in another thread).

I set up my current RAID system at RAID 1, so a mirror system which does seem like a backup approach to me even if it's not called that. I also use offsite storage.

 

[Anjin San]

deleted

 

[Anjin San]

I set up my current RAID system at RAID 1, so a mirror system which does seem like a backup approach to me even if it's not called that. I also use offsite storage.

The problem is that they are mirrored…so if an image is corrupted somehow it gets corrupted on both drives. RAID 1 provides redundancy so you don’t lose anything if a drive fails…but it is still not a backup…ypu can’t go back and get the changes you made yesterday (well, you can sorta get there in LR or PS by undo…but any other file you can’t go back to a previous version or restore if your house burns down or something happens that takes out both drives in the array. You’re doing offsite which is also good…but a direct connected backup drive with a cloning app that does versions like CarbonCopyCloner on macOS or whatever its equivalent on Windows is gives you a local backup as well.

 

[wotan1]

The RAID system is nothing but a way to combine multiple drives into a single drive. With a RAID as I understand it you either combine drives to provide mirroring (which is redundancy) or you do it to gain speed (Striping).

I already had a RAID drive left over from the PC so I merely took the RAID and made it the backup drive.

I would not trust a RAID by itself to be my sole on site backup. Should something go wrong with the RAID controller you could lose everything. Better to have a completely separate drive(s) for local backup. Having worked with the SoftRaid software that is provided by OWC I would not trust that it is infallible. I think it is essential to have a completely separate drive for backup that is not dependent on the RAID controller whether it is hardware or software controlled.

I also needed a super fast SSD drive for the main processing work. It would be super expensive as I understood it to get a RAID that both has enough speed and also serves as backup.

At the same time I use Backblaze for a cloud backup.

 

[Viathelens]

Serious question as those numbers seem off…are you converting everything to TIFF or many layer PSD files? I’ve got more images than that and they only take up 4 TB or so…granted a lot of them are D7500 jpegs from before I switched to RAW but at least 60K of them are Z7II/8/9 uncompressed RAW.

No conversions other than images I work on and convert, some TIFF, some PSD. Only RAW images. I have two disks set as RAID 1. They both show almost full. One 8TB, so 4TB actual storage and one 12TB so 6TB actual storage. While I do have some space left on the second one I don't want to take it to the end and create a problem for myself so I'm leaving some space. LrC does not show up red in its check system but I don't want to get there so trying to do the replacement before I hit the actual end. I have 246GB on one remaining and 933GB remaining on the most recent one I have been using. I shoot a lot and I shoot a lot of images and I don't want to be downloading, walk away, and then come back to find out the system has sent my images to some wrong place that I have to spend hours trying to figure out.

 

[Viathelens]

The problem is that they are mirrored…so if an image is corrupted somehow it gets corrupted on both drives. RAID 1 provides redundancy so you don’t lose anything if a drive fails…but it is still not a backup…ypu can’t go back and get the changes you made yesterday (well, you can sorta get there in LR or PS by undo…but any other file you can’t go back to a previous version or restore if your house burns down or something happens that takes out both drives in the array. You’re doing offsite which is also good…but a direct connected backup drive with a cloning app that does versions like CarbonCopyCloner on macOS or whatever its equivalent on Windows is gives you a local backup as well.

The RAID system is nothing but a way to combine multiple drives into a single drive. With a RAID as I understand it you either combine drives to provide mirroring (which is redundancy) or you do it to gain speed (Striping).

I already had a RAID drive left over from the PC so I merely took the RAID and made it the backup drive.

I would not trust a RAID by itself to be my sole on site backup. Should something go wrong with the RAID controller you could lose everything. Better to have a completely separate drive(s) for local backup. Having worked with the SoftRaid software that is provided by OWC I would not trust that it is infallible. I think it is essential to have a completely separate drive for backup that is not dependent on the RAID controller whether it is hardware or software controlled.

I also needed a super fast SSD drive for the main processing work. It would be super expensive as I understood it to get a RAID that both has enough speed and also serves as backup.

At the same time I use Backblaze for a cloud backup.

More to think about!

 

[Anjin San]

No conversions other than images I work on and convert, some TIFF, some PSD. Only RAW images. I have two disks set as RAID 1. They both show almost full. One 8TB, so 4TB actual storage and one 12TB so 6TB actual storage. While I do have some space left on the second one I don't want to take it to the end and create a problem for myself so I'm leaving some space. LrC does not show up red in its check system but I don't want to get there so trying to do the replacement before I hit the actual end. I have 246GB on one remaining and 933GB remaining on the most recent one I have been using. I shoot a lot and I shoot a lot of images and I don't want to be downloading, walk away, and then come back to find out the system has sent my images to some wrong place that I have to spend hours trying to figure out.

Thanks…as a long time sysadmin I don’t think I’ve ever actually setup a RAID as mirrors because other than redundancy against a drive failure it provides no advantages…almost every one I’ve ever done was RAID 5 with 4 or more drives to provide space and redundancy…and if there were more than about 6 or 7 drives in the array I usually set one as a hot spare and RAIDed N-1 as RAID 5. Striping for write speed wasn’t something I ever needed to do. If you need more space I would go with a 4 or 8 drive array myself and separate backup single drives locally as well as offsite. My local backups ar a daily clone and a weekly clone with the latter setup to keep versions via CarbonCopyCloner.