Subtitling

Introduction

This guide explains the overall process for handling subtitles with Limecraft Flow.

Limecraft Flow is able to generate subtitles either automatically or manually. The subtitles can be viewed in the subtitle editor of Flow-UI. They can also be exported in a variety of subtitle formats.

This section is structured as follows:

  1. Technical Structure: Provides information on the internal structure of subtitles.

  2. Language and Subtitle Presets: Covers the utilization of language and preset options.

  3. Automatic Subtitle Generation: Details the process of automatically generating subtitles.

  4. Subtitle Retrieval: Explains how to retrieve subtitles.

  5. Subtitle Management: Covers tasks related to subtitle management, including updating and deleting.

  6. Subtitle Export: Provides instructions for exporting subtitles.

  7. Subtitling Statuses: Explains the various subtitling statuses and their meanings.

  8. Subtitle Searching: Describes how to search for specific subtitles.

  9. Customize Subtitling Process: Provides guidance on customizing the subtitling workflow with post-processing, automatic transcription, and import functionalities.

Structure

Internally, the Flow-API stores subtitles as an Annotation with objectType:StructuredAnnotation, funnel:Subtitle and a specific structure.

Learn more about the Subtitle Annotation here.

Language and Subtitle Presets

A Subtitle annotation on a clip will have a language, to indicate which language the subtitles are in which is not necessarily the same as the clip audio language.

It will also have a subtitlePresetId, which points to the subtitle preset used to create these subtitles.

Subtitle Presets combine styling and spotting settings into a preset which can be used to generate subtitles.

More information on Languages and Presets can be found here

Automatic generation

Subtitles can be generated automatically by the Advanced Subtitle Workflow which is started at:

POST /production/{prId}/mo/{moId}/service/advanced_subtitle

Start the advanced subtitle workflow.

Details
Description
Parameters
Path Parameters
Name Description Required Type

prId

ID of the production.

Long

moId

ID of the media object.

Long

Body Parameters
Name Description Required Type

AdvancedSubtitleRequest

AdvancedSubtitleRequest

AdvancedSubtitleRequest

Field Name Required Type Description Format

applyFrameRateConversion

Boolean

Do subtitle post-processing to convert the timings of the subtitle from a defined frame rate to the media frame rate

applyShotAlignment

Boolean

Do subtitle post-processing to snap the subtitle timings to shot boundaries.

applySpeakerIds

Boolean

Do subtitle post-processing to apply subtitle coloring based on speaker ids.

frameRateConversionOptions

FrameRateConversionOptions

generateSubtitleFromASR

Boolean

Instruct the workflow to use the transcription for the subtitle creation, the workflow will fail when no transcription is available.

generateTranscriptFromASA

Boolean

Instruct the workflow to generate a transcript using alignment from where to start the subtitle creation.

generateTranscriptFromASR

Boolean

Instruct the workflow to generate a transcription, the workflow will fail when a transcription is already available.

importFileResource

InputFileResource

language

String

The language of the subtitles.

overwriteSubtitleAllowed

Boolean

When generating the subtitles, allow replacing the existing subtitles. If false, the task will fail if subtitles are present.

overwriteTranscriptAllowed

Boolean

When generating the transcript, allow replacing the existing transcription. If false, the task will fail if a transcription is present.

removeBlankSubtitles

Boolean

Instruct the import task to remove any imported blank subtitle lines

speechAlignmentFileResource

InputFileResource

subtitlePresetId

String

Use the settings in this Subtitle Preset to generate the Subtitle annotation.

subtitleTimingCorrections

List of SubtitleTimingCorrection

Do subtitle post-processing to adjust the timings of the subtitle

subtitlingConfiguration

subtitlingConfiguration

transcriptionDictionaryId

Long

Id of the dictionary to use when generating the transcript

int64

transcriptionEngineConfiguration

Object

transcriptionLanguage

String

When transcription is needed, this language will be used.

transcriptionSpeechEngine

String

The transcription engine to use when generating the transcript

Return Type

WorkflowMinWrapper

Field Name Required Type Description Format

accountId

Long

int64

completed

Long

int64

created

Long

int64

errorReports

List of TaskReport

href

String

hrefs

Map of Href

id

String

label

String

mediaObjectId

Long

int64

objectType

String

productionId

Long

int64

publishedFiles

List of object

requiredRights

List of ProductionPermission

serverTime

Date

date-time

status

String

Enum: Inited, Started, Completed, Error, Cancelled, Paused, CompletedPending, ErrorPending, WaitForCallback, Scheduled,

taskReports

List of TaskReport

tasks

List of TaskAssignmentMinWrapper

userId

Long

int64

workflowCompleted

Date

date-time

workflowFailed

Date

date-time

workflowId

String

workflowStarted

Date

date-time

Content Type
  • application/json

Responses
Table 1. http response codes
Code Description Datatype

201

Returns the started workflow

WorkflowMinWrapper

400

Bad request.

``

404

The production or any of the referenced objects were not found.

``

The body of this request is a AdvancedSubtitleRequest JSON object.

Using this workflow, the user can:

  • generate subtitles based on a transcript

  • import subtitles in any subtitle format

Another way to create subtitles automatically is through Translation, which is discussed on its own page.

To generate subtitles, the workflow should be started using the following request:

{
    "language": "en",
    "subtitlePresetId": "default"
}

The workflow will look for an existing transcript and divide into subtitles, with maximum compliance with the spotting rules defined in the production settings.

However, if no transcript is present, the workflow will simply proceed without creating any subtitles.

Further customization and info on using ASR and imports can be found in [customize-the-advanced-subtitling-workflow].

Supported languages

So what values can you use for the language parameter? Refer to Speech Engines and supported features to learn which language codes are supported by the automatic transcription and subtitler.

Wait for the workflow to finish

The calls mentioned above will return a MediaObjectWorkflowReport.

Its workflowId field gives you a reference to the workflow that was started. Once this workflow completes, the Subtitle annotation will have been created. To learn how to wait for a workflow to complete, see this page.

Retrieve a subtitle

To retrieve a subtitle for a particular language and preset you need to issue the following call:

GET /production/{prId}/mo/{moId}/subtitle/{language}/{presetId}

Getting or initialising the subtitle with given language and subtitlePresetId

Details
Description
Parameters
Path Parameters
Name Description Required Type

prId

The id of the Production.

Long

moId

ID of the media object.

Long

language

The language code of the subtitle.

String

presetId

The subtitlePresetId of the subtitle.

String

Query Parameters
Name Description Required Type

includeTranslatedToIds

Include the ids of the subtitles to which this subtitle has been translated.

Boolean

noCreate

Boolean

Return Type

Content Type
  • /

Responses
Table 2. http response codes
Code Description Datatype

200

The exported subtitle object, depending on the the Accept-Header. Simple method for exporting using .srt, .vtt, etc..

``

400

Bad request.

``

404

The production or any of the referenced objects were not found.

``

422

Validation exception such as non-existing subtitlePresetId.

``

If no Subtitle exists for that combination of language and subtitlePresetId, it will be created automatically. This behaviour can be suppressed by passing noCreate=true to the call.

If there are no presets used in the production (which we discourage), you can also use:

GET /production/{prId}/mo/{moId}/subtitle/{language}

Getting or initialising the subtitle with given language and no subtitlePresetId

Details
Description
Parameters
Path Parameters
Name Description Required Type

prId

ID of the production.

Long

moId

ID of the media object.

Long

language

The language code of the subtitle.

String

Query Parameters
Name Description Required Type

includeTranslatedToIds

Include the ids of the subtitles to which this subtitle has been translated.

Boolean

noCreate

Boolean

Return Type

Content Type
  • /

Responses
Table 3. http response codes
Code Description Datatype

200

The exported subtitle object, depending on the the Accept-Header. Simple method for exporting using .srt, .vtt, etc..

``

400

Bad request.

``

404

The production or any of the referenced objects were not found.

``

422

Validation exception such as non-existing subtitlePresetId.

``

which returns the result in the following sequence:

  • select the subtitle without subtitlePresetId;

  • select the subtitle with the defaultSubtitlePresetId;

  • select the only subtitle with that language regardless of the subtitlePresetId;

  • When there are multiple subtitles for a language, with different subtitlePresetId (none of them without subtitlePresetId or equal to the default subtitlePresetId), the call will fail and return an error;

  • nothing

Manage Subtitles

Patch

To change particular properties of a Subtitle, use JSON Patch.

PATCH /production/{prId}/mo/{moId}/subtitle/{language}/{presetId}

Patches the subtitle with given language and subtitlePresetId

Details
Description
Parameters
Path Parameters
Name Description Required Type

prId

ID of the production.

Long

moId

ID of the media object.

Long

language

The language code of the subtitle.

String

presetId

ID of the subtitle preset.

String

Body Parameters
Name Description Required Type

body

Object

body

Field Name

Required

Type

Description

Format

Return Type

SubtitleAnnotation

Field Name Required Type Description Format

annotationProductionId

Long

int64

clipMetadata

ClipMetadata

created

Date

The time when this resource was created

date-time

createdBy

String

The request or process that created this resource

createdByShareId

Long

int64

createdBySharedUserId

Long

int64

creatorId

Long

The id of the user who created this resource

int64

crossProduction

Boolean

customFields

CustomFields

deleted

Date

date-time

description

String

Textual contents of the Annotation

end

Long

The frame range described by the annotation runs up to end, but not including it. Should be less than or equal to the amount of frames the MediaObject has.

int64

funnel

String

Describes how the Annotation should be interpreted by the client application. Can be thought of as a subtype.

genericType

String

id

Long

The id of this resource

int64

includeTranslatedTo

Boolean

includesFrom

Set of string

keyframeFrames

Long

int64

label

String

language

String

lastUpdated

Date

The time when this resource was last updated

date-time

mediaObject

MediaObject

mediaObjectId

Long

int64

modifiedBy

String

The request or process responsible for the last update of this resource

objectType

String

The data model type or class name of this resource

origin

String

productionId

Long

int64

rating

Double

double

relatedToId

Long

int64

securityClasses

Set of string

Enum:

source

String

spatial

String

Link the Annotation to a specific part of the video or image frame. A Media Fragments Spatial Dimension description string is expected.

start

Long

First frame of the Annotation. 0 is the first frame of the clip. The start frame is included in the frame range the annotation describes.

int64

structuredAnnotation

SubtitleObject

structuredDescription

Object

systemFields

CustomFields

tags

Set of string

translatedFromId

Long

int64

translatedToIds

Set of long

int64

type

String

Enum: MINIME, EBU_CORE, TRANSCRIBER, TECHNICAL_DETAILS, GENERIC, SUBTITLE, SHOT,

version

Long

The version of this resource, used for Optimistic Locking

int64

Content Type
  • application/json

Responses
Table 4. http response codes
Code Description Datatype

200

The subtitle object

SubtitleAnnotation

400

Bad request.

``

404

The production or any of the referenced objects were not found.

``

409

The version in the update object does not match with the version stored causing an optimistic lock.

``

422

Validation exception such as non-existing subtitlePresetId.

``

For example, the following PATCH will change the second subtitle (/subtitles/1):

change the second subtitle using PATCH
[
    {
        "op": "replace",
        "path": "/structuredDescription/subtitles/1",
        "value": {
            "start": 92,
            "end": 238,
            "confidence": 1,
            "region": "bottomRegion",
            "textParts": [
                {
                    "content": "this ",
                    "type": "span",
                    "style": "spanStyle"
                },
                {
                    "content": "is in cyan",
                    "type": "span",
                    "style": "cyanStyle"
                }
            ],
            "warnings": []
        }
    }
]

Update

Use a PUT to update the entire subtitle annotation.For most use cases, using Patch is preferred.

PUT /production/{prId}/mo/{moId}/subtitle/{language}/{presetId}

Updates the subtitle with given language and subtitlePresetId

Details
Description
Parameters
Path Parameters
Name Description Required Type

prId

ID of the production.

Long

moId

ID of the media object.

Long

language

The language code of the subtitle.

String

presetId

ID of the subtitle preset.

String

Body Parameters
Name Description Required Type

SubtitleAnnotation

SubtitleAnnotation

SubtitleAnnotation

Field Name Required Type Description Format

annotationProductionId

Long

int64

clipMetadata

ClipMetadata

created

Date

The time when this resource was created

date-time

createdBy

String

The request or process that created this resource

createdByShareId

Long

int64

createdBySharedUserId

Long

int64

creatorId

Long

The id of the user who created this resource

int64

crossProduction

Boolean

customFields

CustomFields

deleted

Date

date-time

description

String

Textual contents of the Annotation

end

Long

The frame range described by the annotation runs up to end, but not including it. Should be less than or equal to the amount of frames the MediaObject has.

int64

funnel

String

Describes how the Annotation should be interpreted by the client application. Can be thought of as a subtype.

genericType

String

id

Long

The id of this resource

int64

includeTranslatedTo

Boolean

includesFrom

Set of string

keyframeFrames

Long

int64

label

String

language

String

lastUpdated

Date

The time when this resource was last updated

date-time

mediaObject

MediaObject

mediaObjectId

Long

int64

modifiedBy

String

The request or process responsible for the last update of this resource

objectType

String

The data model type or class name of this resource

origin

String

productionId

Long

int64

rating

Double

double

relatedToId

Long

int64

securityClasses

Set of string

Enum:

source

String

spatial

String

Link the Annotation to a specific part of the video or image frame. A Media Fragments Spatial Dimension description string is expected.

start

Long

First frame of the Annotation. 0 is the first frame of the clip. The start frame is included in the frame range the annotation describes.

int64

structuredAnnotation

SubtitleObject

structuredDescription

Object

systemFields

CustomFields

tags

Set of string

translatedFromId

Long

int64

translatedToIds

Set of long

int64

type

String

Enum: MINIME, EBU_CORE, TRANSCRIBER, TECHNICAL_DETAILS, GENERIC, SUBTITLE, SHOT,

version

Long

The version of this resource, used for Optimistic Locking

int64

Return Type

SubtitleAnnotation

Field Name Required Type Description Format

annotationProductionId

Long

int64

clipMetadata

ClipMetadata

created

Date

The time when this resource was created

date-time

createdBy

String

The request or process that created this resource

createdByShareId

Long

int64

createdBySharedUserId

Long

int64

creatorId

Long

The id of the user who created this resource

int64

crossProduction

Boolean

customFields

CustomFields

deleted

Date

date-time

description

String

Textual contents of the Annotation

end

Long

The frame range described by the annotation runs up to end, but not including it. Should be less than or equal to the amount of frames the MediaObject has.

int64

funnel

String

Describes how the Annotation should be interpreted by the client application. Can be thought of as a subtype.

genericType

String

id

Long

The id of this resource

int64

includeTranslatedTo

Boolean

includesFrom

Set of string

keyframeFrames

Long

int64

label

String

language

String

lastUpdated

Date

The time when this resource was last updated

date-time

mediaObject

MediaObject

mediaObjectId

Long

int64

modifiedBy

String

The request or process responsible for the last update of this resource

objectType

String

The data model type or class name of this resource

origin

String

productionId

Long

int64

rating

Double

double

relatedToId

Long

int64

securityClasses

Set of string

Enum:

source

String

spatial

String

Link the Annotation to a specific part of the video or image frame. A Media Fragments Spatial Dimension description string is expected.

start

Long

First frame of the Annotation. 0 is the first frame of the clip. The start frame is included in the frame range the annotation describes.

int64

structuredAnnotation

SubtitleObject

structuredDescription

Object

systemFields

CustomFields

tags

Set of string

translatedFromId

Long

int64

translatedToIds

Set of long

int64

type

String

Enum: MINIME, EBU_CORE, TRANSCRIBER, TECHNICAL_DETAILS, GENERIC, SUBTITLE, SHOT,

version

Long

The version of this resource, used for Optimistic Locking

int64

Content Type
  • application/json

Responses
Table 5. http response codes
Code Description Datatype

200

The subtitle object

SubtitleAnnotation

400

Bad request.

``

404

The production or any of the referenced objects were not found.

``

409

The version in the update object does not match with the version stored causing an optimistic lock.

``

422

Validation exception such as non-existing subtitlePresetId.

``

Delete

DELETE /production/{prId}/mo/{moId}/subtitle/{language}/{presetId}

Deletes the subtitle with given language and subtitlePresetId

Details
Description
Parameters
Path Parameters
Name Description Required Type

prId

ID of the production.

Long

moId

ID of the media object.

Long

language

The language code of the subtitle.

String

presetId

ID of the subtitle preset.

String

Query Parameters
Name Description Required Type

deleteAssemble

Boolean

detachAssemble

Boolean

Return Type

SubtitleAnnotation

Field Name Required Type Description Format

annotationProductionId

Long

int64

clipMetadata

ClipMetadata

created

Date

The time when this resource was created

date-time

createdBy

String

The request or process that created this resource

createdByShareId

Long

int64

createdBySharedUserId

Long

int64

creatorId

Long

The id of the user who created this resource

int64

crossProduction

Boolean

customFields

CustomFields

deleted

Date

date-time

description

String

Textual contents of the Annotation

end

Long

The frame range described by the annotation runs up to end, but not including it. Should be less than or equal to the amount of frames the MediaObject has.

int64

funnel

String

Describes how the Annotation should be interpreted by the client application. Can be thought of as a subtype.

genericType

String

id

Long

The id of this resource

int64

includeTranslatedTo

Boolean

includesFrom

Set of string

keyframeFrames

Long

int64

label

String

language

String

lastUpdated

Date

The time when this resource was last updated

date-time

mediaObject

MediaObject

mediaObjectId

Long

int64

modifiedBy

String

The request or process responsible for the last update of this resource

objectType

String

The data model type or class name of this resource

origin

String

productionId

Long

int64

rating

Double

double

relatedToId

Long

int64

securityClasses

Set of string

Enum:

source

String

spatial

String

Link the Annotation to a specific part of the video or image frame. A Media Fragments Spatial Dimension description string is expected.

start

Long

First frame of the Annotation. 0 is the first frame of the clip. The start frame is included in the frame range the annotation describes.

int64

structuredAnnotation

SubtitleObject

structuredDescription

Object

systemFields

CustomFields

tags

Set of string

translatedFromId

Long

int64

translatedToIds

Set of long

int64

type

String

Enum: MINIME, EBU_CORE, TRANSCRIBER, TECHNICAL_DETAILS, GENERIC, SUBTITLE, SHOT,

version

Long

The version of this resource, used for Optimistic Locking

int64

Content Type
  • application/json

Responses
Table 6. http response codes
Code Description Datatype

200

The subtitle object

SubtitleAnnotation

400

Bad request.

``

404

The production or any of the referenced objects were not found.

``

409

The version in the update object does not match with the version stored causing an optimistic lock.

``

422

Validation exception such as non-existing subtitlePresetId.

``

Export

POST /production/{prId}/mo/{moId}/subtitle/{language}/{presetId}

Creates an export workflow that will generate the subtitle file which can be downloaded as part of the workflow resources.

Details
Description
Parameters
Path Parameters
Name Description Required Type

prId

ID of the production.

Long

moId

ID of the media object.

Long

language

The language code of the subtitle.

String

presetId

ID of the subtitle preset.

String

Body Parameters
Name Description Required Type

SubtitleExportRequest

Form body containing parameters for the subtitling request.

SubtitleExportRequest

SubtitleExportRequest

Field Name Required Type Description Format

disclaimers

List of DisclaimerConfiguration

Disclaimers can be used to add additional subtitles containing the disclaimers.

downloadName

String

The filename to use. It is best to omit the extension, it will be added automatically.

ebuDisplayStandardCode

Integer

For subtitle_ebustl exportType.

int32

ebuStlLanguageCode

String

For subtitle_ebustl exportType. Setting the language code (iso 639-1 standard).

ebuStlMarginBottom

Integer

For subtitle_ebustl exportType. Bottom margin.

int32

ebuStlMarginTop

Integer

For subtitle_ebustl exportType. Top margin.

int32

ebuStlTeletextDoubleHeight

Boolean

For subtitle_ebustl exportType. Use double height for text (teletext export only).

ebuStlTeletextUseBox

Boolean

For subtitle_ebustl exportType. Introduce a box around the text (teletext export only).

ebuStlTextJustification

String

For subtitle_ebustl exportType. The text justification (`unchanged`, `left`, `centered`, `right`).

format

String

The export format of the subtitle file. Should be one of ebustl, ebuttd, webvtt or srt.

includeColorCodes

Boolean

Export with color tags.

timeBase

String

Set to 'smpte' for SMPTE time and to 'media' for time relative to start of video.

Query Parameters
Name Description Required Type

includeTranslatedToIds

Include the ids of the subtitles to which this subtitle has been translated.

Boolean

Return Type

AsyncResponse

Field Name Required Type Description Format

expectedResult

Date

date-time

href

String

hrefs

Map of Href

location

URI

uri

objectType

String

productionId

Long

int64

workflowId

String

Content Type
  • /

Responses
Table 7. http response codes
Code Description Datatype

202

The request was successful and the generation has started.

AsyncResponse

400

Bad request.

``

404

The production or any of the referenced objects were not found.

``

The export call will return a JSON object with a workflowId field. The workflowId gives you a reference to the workflow that was started.

To learn how to wait for this workflow to complete, see this page.

Once the workflow has completed, the workflow report will have a location field containing the url where the subtitle export file can be downloaded. The export will result in a zip of the files. If the result of the export is a single file, no zip will be created but that file can be downloaded directly.

More information on exporting subtitles can be found at Export

SubtitlingStatuses

The MediaObjectAnnotation of the clip contains a subtitlingStatuses field which contains the status for each existing subtitle of the clip.

Note that subtitlingStatuses is used to populate the subtitle selector in the subtitle editor of Flow-UI. If the status isn’t set, it won’t be shown in Flow-UI!

Example:

{
  "subtitlingStatuses": {
    "en;13": "AUTOMATIC_COMPLETED",
    "en": "EDITING", // no preset
    "fr;15": "EDITING"
  }
}

The values of the subtitlingStatuses map are any of the following:

status Description

NOT_STARTED

The subtitling for this version has not started. Same as if the key wouldn’t exist.

EDITING

Editing has started.

AUTOMATIC_STARTED

The automatic subtitling process is busy

AUTOMATIC_FAILED

The automatic subtitling was started but has failed

AUTOMATIC_COMPLETED

The automatic subtitling has completed succesfully

COMPLETED

The subtitling for this version has completed. Editing won’t be possible in Flow-UI unless the status is changed to EDITING.

The keys used in the subtitlingStatuses map are a combination of the language and the subtitlePresetId, separated by a ";" character.

Search Subtitles themselves

The Subtitle Annotations themselves are available in the index. They have a language and a subtitlePresetId field.

Find clips with a given status, language or preset

The MediaObjectAnnotations have a subtitlingStatusLanguagePresetTags field.It contains ;- separated entries consisting of the status, language and subtitlePresetId.

To look for all clips having subtitles in Dutch, you could use a filter like this:

subtitlingStatusLanguagePresetTags:(*;nl;*)

To look for all clips having Dutch subtitles for a particular preset, you could use a filter like this:

subtitlingStatusLanguagePresetTags:(*;nl;sp_1643191773874_1191)

To look for all completed subtitles with that preset, for any language, use:

subtitlingStatusLanguagePresetTags:(COMPLETED;*;sp_1643191773874_1191)

In Flow-UI, the Subtitle Status filter is based on this field.

Customize the Advanced Subtitle Workflow

The Advanced Subtitle Workflow is capable of handling all kinds of inputs and post-processing.

More information on controlling the workflow can be found here