Hello all!

to start this discussion, my Name is Hermann Voßeler (aka "Ichthyostega"), I am core dev of the Lumiera project. The goal is to create a modern OpenSource editing software suitable for professional film making -- we think there is or should be a natural connection between free software, building and owning your tools, and free and independent movie making, »Open Video«.But we're not there yet. This is also true for Lumiera, which is a highly ambitious project. We have quite a pile of code, we did a lot of analysis, design and architecture, but we're still far from even the first working integrated version of the system as envisioned.

The idea for this outreach, exchange and possible collaboration is to cover new ground and while doing so, find out how to build a link and integration between such applications in general. If there is such a thing like Apertus, it is just natural that we want to get its power and performance well integrated into the editing process. But we also see this effort as blueprint for integration with other software. Speaking for the Lumiera project, we can contribute insight into film production and editing workflows, and of course into some of the technical challenges involved.

Workflow types

to summarise some of the previous discussion: we propose the following ideal-typical workflows as point of reference for detailed analysis and planing.

Workflow-A ("point and shot")
After loading the material into OpenCine, the user carries out some basic and generic settings and adjustments for the footage at hand, just to develop the entire footage in one batch run and export the result into a widely accepted (high- or even low-definition media format). From this point on, all of the further processing and editing is done with other pre-existing software. In case the further workflow uncovers some deficiencies in the initially chosen raw development settings, just the affected parts are developed again with better settings, and the resulting material is integrated into the edit living within the external software.

Workflow-B ("proxy and finish")
The user does the initial generic setup in OpenCine, but performs a raw development just for the purpose of exporting proxy media, again in a widely supported export format. This proxy footage is subsequently loaded into external well established editing software, where all of the further editing decisions are done. At some point, third party services for SFX might be involved, and for this purpose, partial export from OpenCine of the relevant footage is done in high quality. When it comes to finishing, the user provides an EDL (edit decision list) and some partial an specific adjustments to the raw development settings for some scenes. From these ingredients, the user expects to build a complete version of the finished movie's video track in final quality, which will be loaded into some authoring software, combined with the final sound mix and encoded into delivery format.

Workflow-C ("fully integrated")
Again the user starts in OpenCine with developing the footage into a working copy (rsp. proxy media), to be loaded into the editing software. But, during the following editing and post procession, the full capabilites of this editing software are exploited to build up the final version of the film, way beyond what reasonably can be represented by a mere EDL. Consequently the user needs to stay within the realm of the editing software, and expect the final render to be performed from there. Thus, the editing software has to coordinate and execute the final rendering process, relying on the integrated colaboration with OpenCine to create the final product in a single shot, maybe to be fed in a final step through authoring software to create the intended distribution and delivery format(s).

some notes at what IMHO would be the next steps...

We should consider which of these workflows we feel we must, or want to support, and why and how. This is prerequisite for deciding on the Architecture for possible integration. This is a generic consideration; Lumiera acts as a pivot or concrete example case here.

Small typo, it's T609, as T604 is about sociocracy.

Discussion

Workflow-A is a must, but is trivially suported: all it needs on the side of OpenCine is to provide some kind of "Project" or persistent "Session", i.e. something to store

• all the shots belonging together
• the settings for these shots, so they can be reproduced / tweaked later.

Workflow-B corresponds to the standard way films where made for the 1st century, just translated into the digital era. In the way described above, OpenCine would take on the role of the Lab or printing service from analogue film production: for this to work, OpenCine needs to...

• define an "Edit Decision List" feature
• and a way to refer to footage and settings.

When it comes to finishing, the user somehow provides the concrete EDL (cut list), and OpenCine is the host (the controlling entity) to carry out the finishing process. But note, from a higher level, this workflow can be automated yet more. The editing application could generate the EDL, then fire up OpenCine and hand over control for the finishing process.

Workflow-C to the contrary is optional. From a merely pragmatic POV, Workflow-B is all you need to finish even the most complicated movies, under the condition that there is a way to "inject" and integrate rendered VFX, SFX and partial integrations finished with other software. But we still do want Workflow-C, for various reasons

• it is innovative. Only with digital video we can now do such a thing
• more efficient usage of resources, since settings are only applied "virtually". Excessive storage requirements are postponed until the very end, the finishing phase.
• potential for "high-end" imaging, since one intermediary copy/rendering can be omitted on round-trips.
• people have come to expect it. Starting from "TV/video", applications like Premiere and later Final Cut have done much to popularise this working style: everything is placed "under a single roof" and the technical entrance barrier is significantly lowered -- allowing small teams and more artistically oriented people to conduct a full-blown production
• digital processing allows more than just editing, but Workflow-B has a natural tendency to penalise anything beyond a "simple standard edit". Unfortunately there is no hope of ever standardising all the possibilities of compositing and montage; so the very thing that creates this additional artistic liberty beyond "just cuts", is locking the user into the specific editing system to provide these features: we're more or less forced to do the finishing under the governance of the editing software.
• for an application like Lumiera, Workflow-C is the raison d'etre

some points to note

on a technical level, there is much overlap in the features OpenCine needs to provide for all of these Workflows.

• we need a way to refer to some takes
• we need a way to represent "settings done on a take" (even if this representation is opaque and governed by OpenCine).
• the most significant difference with Workflow-C is that the editing application is host (conducting the finishing process), and OpenCine has to work as slave and carry out the operations encoded in the aforementioned "Settings"

I would suggest that we start from a very simple base to evaluate which direction to take. Lumiera should be the master and responsible for the session, at least for now.

Here is a possible workflow between Lumiera and OC:

• Lumiera loads a frame into some buffer
• Buffer is handled over to OC plugin
• OC processes the image data, it could be as simple as subtracting/removing some color channel
• Lumiera shows the frame in the preview

Now, here are some related things which we have to discuss and evaluate:

• How would you set the format of data, so OC knows how to process it? Shoud OC provide an enum of common formats, e.g. 16bit float, 32 bit float, 12bit integer etc.?
• What about some sort of event system to signalize that processing is finished or would you just wait till the processing thread is done? There are some interesting signal-slot libs out there which rely purely on C++11, e.g. https://github.com/NoAvailableAlias/nano-signal-slot.
• What over settings are necessary? Afterwards we could analyze what's the best way to pass them over to OC or back.

In T599#8979, @BAndiT1983 wrote:

• What about some sort of event system to signalize that processing is finished or would you just wait till the processing thread is done?

As far as Lumiera is concerned: always the latter! No signalling, no interrupts, no blocking please.
This is really crucial for the performance. We have a thread pool, where the number of worker threads corresponds to the number of CPUs (or allocated CPUs or whatever). And we assign a single job to a thread.This job is assumed to max out a single core and terminate within a finite time interval and we expect (or hope) it will not do anything which causes locks, memory barriers or anything else that might interfere with the work done on other cores.

If we get the GPU into that mix, the same principle applies. But in detail that would mean to clarify a lot of fine points regarding the management of tasks on the GPU.

We will never attempt to abort a job, rather we'll drop its results to the floor and forget about it. When a job (and thus a thread) goes amok, the scheduler will of course detect that after some timeout, but the only thing we'll do in this case is just to commit suicide, i.e. blast away our own process, without much ado or shutdown clean-up. We build a system with unlimited undo/redo and a database like task transaction log, so the best idea is to crash as quick as possible.

In T599#8979, @BAndiT1983 wrote:

I would suggest that we start from a very simple base to evaluate which direction to take. Lumiera should be the master and responsible for the session, at least for now.

Agreed. So this point is settled.

Here is a possible workflow between Lumiera and OC:

• Lumiera loads a frame into some buffer
• Buffer is handled over to OC plugin
• OC processes the image data, it could be as simple as subtracting/removing some color channel
• Lumiera shows the frame in the preview

Agreed. This is a very sensible starting point. Consequently this means the API is cast in terms of buffers-with-data.
The API functions will all be (with the exception of meta-operations like init) of the kind: "here is a pointer to a buffer, type is xyz, do this-and-that". Basically these are pure functions (stateless). We should try to get as close as possible to that ideal, to reduce complexity. That is, if OpenCine does require some stateful context, then we should abstract that out as a context handle. An example would be some GPU context, that needs to be initialised beforehand. Lumiera would then first call an API function to build that context, and then pass the context handle back into the calc function. Lumiera will also guarantee that a clean-up call will happen (unless we commit suicide, of course, in which case the kernel has to wipe off the bloody slyme, hehe)

Now, here are some related things which we have to discuss and evaluate:

• How would you set the format of data, so OC knows how to process it? Shoud OC provide an enum of common formats, e.g. 16bit float, 32 bit float, 12bit integer etc.?

This.
Lumiera is the host, so it is responsible for the execution of the process. But OpenCine(library) is the service provider. So OpenCine defines the possible types and formats it expects and can handle. An Enum is a good idea. Alongside with the library, there is an interface header, which Lumiera will #include

• What other settings are necessary?

now we get to the most challenging point. We need somehow to encode the kind of operation, that has to be done, and the operation parameters. There are two possible approaches

• we use binary format. Then we need to define some kind of self-describing header (typically based on ENUMs, i.e. just integer IDs which encode a type or function number)
• or we use a textual format

In the light of what we've learned in the last decade of enterprise software, I would strongly recommend to consider the latter approach. But, please, really feel free to question and challenge my approach!

So what I'd propose is to describe the actual operations and the arguments in JSON, and hand them over in a std::string (we're C++ on both sides. If we want to stick to plain C on the interface level, which might be arguable, then the caller has to allocate a char buffer with the command string and keep it alive until after returning from the call)

No, C please! ;) Don't want to wake that dinosaur, Linux is still full of that stuff.

Will comment on the other stuff soon. Still reflecting and testing plus it's Monday which was pretty busy.