Ok, I feel with you. Haven't had such a fast RAID yet. Copying one evening worth of Arriraw (only OpenGate 75fps and 120fps 2.8K 16:9, no normal speed, all maximum) skater footage with 1,5 TB in a few hours was kind of hard to a 4 bay RAID5 TB-Array (was a fun project and no second target (expect of slow USB3 single drives) was available.

In T449#7481, @RainerFritz wrote:

Hi !

irieger:
For example you have a shooting day where you have to deal with let's say 1,5TB of data reading the source twice costs a lot of time.
As you need to do an "optical" check to the material (hashes can not see problems with the picture) and process it then for example to an offline material as well.
So you end up by reading the source files on set at least three times with no parallel processing in the copy process that would be
a 4th time.

Nope, my script reads two times. One for copies (coping in a buffer and writing to all destinations in parallel). And one time for checksum.

Hey Rainer,

Well, Apple can be a bit of a pain sometimes but they are opening up ProRes more and more so maybe it is an option. A few years back there was no official solution for ProRes writing on Windows and now different applications are getting the go for example.

In T449#7312, @BAndiT1983 wrote:

I know that everyone demands rsync or similar thing, but e.g. librsync depends on Cygwin and I try to avoid many dependencies. Still searching for alternative libs before rolling some sort of own implementation. OC doesn't need the network part of rsync, as Qt would be used to copy (or plain C++11), which should be portable. Also constrain OC to local drives (built-in oder removable ones) for now. Later OC would be sending data over local network (not internet, not for now at least), but this should be similar to the "local" workflow inside the application.

I don't see rsync as a main need. Mostly it is a fast local transfer you need onset or so not the kind of network transfer or you use a network file system and can use the tool as if you are working local.

Cool, I will keep an eye on your project an maybe find some time to help. Would be cool to finally have a nice data management tool. Have tried a few of the commercial ones and none of them really satisfied me so there is some space to fill.

A few points that I see as important parts:

• 1:1 copy as mentioned earlier. (Maybe with an option to exclude .Spotlight, .Trash etc. when on a mac that sadly creates them directly when connecting the first time ...)
• Just have a list of targets where you can add from one to multiple (not only two) targets
• Parallel Copy: Read from source to RAM and copy to all target drives in parallel to speed up progress
• Parallel verification: Some simple tools do a checksum file(or to ram) from the source and check the targets then. Skip this and do checksums of source and all targets at once and compare then.
• add a project mode where you can set the target drives and directory scheme for this project that can be loaded as a starting point
• variable based directory scheme so that you can have something like "/media/irieger/MY_EXTERNAL_DRIVE/OperationApertus/Footage/%Y-%m%-%d/%camNo"
• an option to add another transfer to a queue that fill be processed when the current is finished
• an option to start a parallel file transfer (what I think about here is if you need to copy a small card from the sound guy who wants to finish in the evening and not have to let him wait until all the RAW footage is backed up or something similiar)

Would be nice if the user can select a checksum method and the checksums are then written to a file for manual checking (for example a md5sum-file that can be checked with md5sum -c $file. Checksum algorithms I'd really like to see are md5sum an xxHash (https://code.google.com/p/xxhash/), which seems to be much fast so could be helpful when transfering huge amounts of 4K raw sequences.