Cutting a video results in too many subtitles

[LukasKalbertodt]

When a video is cut in the editor, and then republished, it ends up with too many subtitle tracks. Example: https://explore.opencast.org/webinars/v/F0ZbZ3UZVm5

We talked about it and apparently it's just bad workflow config. Specifically: the speechtotext step is in the partial-publish and archives the subtitles. Katrin said something about moving the step somewhere else.

[KatrinIhler]

Katrin would run that step during import once, archive the result, and then only ever cut the captions when the event is republished :D

[KatrinIhler]

Maybe do github.com/opencast/opencast/issues/6719 upstream as well? There was some new discussion in the community that we could do this with the base model by default.

[KatrinIhler]

@LukasKalbertodt Can you check why I can't assign Caro here, even though she's in the developer team of the namespace? 😅 Username is ciegler

[LukasKalbertodt]

@gregorydlogan I don't see why ciegler cannot be assigned and I don't have a lot of access to this repository's settings, maybe you can have a look?

[ciegler]

I agree with Katrin in putting generate-subtitles-with-whisper into a different workflow.
We should also check for existing subtitles and delete them or skip the generation before creating new ones.

I have some questions regarding the use case of the subtitles generation:

• if subtitles have been uploaded should they be overwritten by the generated subtitles from whisper
• should subtitle creation via whisper be available independently from publishing the event
• should there be an option to generate an english translation of the original audio language (if it is not english)
• should subtitles via whisper always be generated when running the workflow (+ deleting the old generated subtitles) or should subtitles only be generated once and the generation skipped if there already exist generated subtitles

[LukasKalbertodt]

if subtitles have been uploaded should they be overwritten by the generated subtitles from whisper

No not overwritten. They should certainly be kept. The workflow has a parameter that can be used to control whether whisper should run. At least that was my idea.

should subtitle creation via whisper be available independently from publishing the event

As a separate workflow that can be triggered by the admin UI? That would be nice! Altho it's not high priority

should there be an option to generate an english translation of the original audio language (if it is not english)

Maybe? #4

should subtitles via whisper always be generated when running the workflow (+ deleting the old generated subtitles) or should subtitles only be generated once and the generation skipped if there already exist generated subtitles

It was explained to me that cut operations cut subtitles. While that is probably not perfect (if you cut into one VTT section, that cannot be split), it should probably be sufficient for everything and it would be great if we could skip the costly whisper step.

(But that's all just my interpretation, feel free to also wait for Katrins reply to these :D)

[KatrinIhler]

No not overwritten. They should certainly be kept. The workflow has a parameter that can be used to control whether whisper should run. At least that was my idea.

Since we do whisper in the import step then, old videos should not be affected. For new videos, I guess we could run whisper if no subtitles were uploaded during import. Or we remove the upload functionality completely if we're satisfied by whisper - is that even used right now?

I'd also consider removing the config parameter turning whisper on since we should always want subtitles, right?

should subtitle creation via whisper be available independently from publishing the event

Could be interesting for older videos, but probably not a priority.

should there be an option to generate an english translation of the original audio language (if it is not english)

Great idea. I wonder if we can automate this - check if the first generated captions are english (not sure if that's possible right now), and if not, also generate english captions? this could even be a config for the whisper WOH (also translate if first captions aren't in english).

should subtitles via whisper always be generated when running the workflow (+ deleting the old generated subtitles) or should subtitles only be generated once and the generation skipped if there already exist generated subtitles

Since whisper should run on import, which only happens once, there should be no prior captions available (unless we uploaded some, see above). The only other case would be the separate workflow mentioned above, which should probably overwrite, if we do that at all.

In general, I'd try to keep this as simple as possible while still being useful, to keep the difference to the community workflows as small as possible.

[ciegler]

Ok, to summarize:

• we want subtitles created on import (original language and english translation for non english events)
• we still need to decide whether to remove the option to upload subtitles
• subtitle creation triggered after a published event -> nice to have, but no priority

[ciegler]

It was explained to me that cut operations cut subtitles. While that is probably not perfect (if you cut into one VTT section, that cannot be split), it should probably be sufficient for everything and it would be great if we could skip the costly whisper step.

The subtitles are cut together with the video, before publishing. As far as I understood it: when the video is cut a smil file is attached to the event. This is then used for cutting and is applied to the subtitles and the event.

[gregorydlogan]

I've given @ciegler triage access. If she needs write let me know, wasn't sure what we needed here.

[KatrinIhler]

The subtitles are cut together with the video, before publishing. As far as I understood it: when the video is cut a smil file is attached to the event. This is then used for cutting and is applied to the subtitles and the event.

Exactly!