Editing view, AI agent connections and automatic subtitles (0.4.0)

THIRD_PARTY_NOTICES.md

@@ -7,15 +7,23 @@ FreeVideo code uses [Apache-2.0](LICENSE). Its dependencies and models retain th
 | Qt, PySide6 and Shiboken6 | LGPL-3.0; [component notices](freevideo_engine/launcher/licenses/NOTICE.txt) |
 | PyInstaller bootloader | GPL-2.0 with the bootloader exception |
 | CPython / psutil | PSF / BSD-3-Clause |
+| [QuickJS Python binding](https://github.com/PetterS/quickjs) 1.19.4 (author: Petter Strandmark) and QuickJS (© 2016–2021 Fabrice Bellard, © 2017–2021 Charlie Gordon) | [MIT notices](freevideo_engine/licenses/quickjs-MIT.txt) |
 | [ComfyUI](https://github.com/Comfy-Org/ComfyUI), including the H3 text encoder code | GPL-3.0 |
 | [VDN-H3](https://github.com/OpenVDN/vdn-minimax-h3), [Diffusers](https://github.com/huggingface/diffusers), [SageAttention](https://github.com/thu-ml/SageAttention) | Apache-2.0 |
 | [VDN-H3 and H3 model weights](https://huggingface.co/OpenVDN/vdn-minimax-h3-edge/blob/main/LICENSE) | MiniMax H3 Community License |
 | [MiniMax H3 latent upscaler](https://huggingface.co/LBH-123-AI/Minimax_h3_latent_Upscaler) | [MIT](freevideo_engine/licenses/latent-upscaler-MIT.txt) |
 | [FreeToken](https://github.com/FlashML-org/FreeToken) prompt VLM operators, [Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) optional weights | [Apache-2.0](freevideo_engine/licenses/FreeToken-Apache-2.0.txt) |
+| [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx) 1.13.8 speech runtime, built by FreeVideo without text-to-speech, ALSA and PortAudio (modified build; `__init__.py` edited; source snapshot 11afbd00 kept by FreeVideo; optional download for automatic subtitles, [build note](freevideo_engine/licenses/sherpa-onnx-build-NOTE.txt)). Includes statically linked kaldi-native-fbank, kaldi-decoder, kaldifst, OpenFst, simple-sentencepiece (Apache-2.0), KissFFT (BSD-3-Clause), pybind11 (BSD-3-Clause), nlohmann/json (MIT), hclust-cpp and Darts-clone (BSD-2-Clause), ThreadPool (Zlib). The Windows build also contains msvcp140.dll and msvcp140_1.dll (Microsoft Visual C++ 2022 runtime 14.44.35211, © Microsoft Corporation), unmodified and redistributed as permitted by Microsoft for its redistributable files. | Apache-2.0 |
+| [ONNX Runtime](https://github.com/microsoft/onnxruntime) 1.28.2 (CPU), © Microsoft Corporation; its third-party notices are shipped inside the sherpa-onnx wheel | MIT |
+| [Eigen](https://gitlab.com/libeigen/eigen) 5.0.1, compiled into sherpa-onnx and ONNX Runtime; source: https://gitlab.com/libeigen/eigen/-/tree/5.0.1 | MPL-2.0 |
+| [Silero VAD](https://github.com/snakers4/silero-vad) v5.0 voice-activity model, © 2020-present Silero Team (optional download; license file downloaded with the model) | [MIT](freevideo_engine/licenses/Silero-VAD-MIT.txt) |
+| [NVIDIA NeMo stt_en_conformer_ctc_small](https://huggingface.co/nvidia/stt_en_conformer_ctc_small), © NVIDIA, exported to ONNX and quantized to int8 by the sherpa-onnx project, changed from the original; not endorsed by NVIDIA (optional download for English subtitles; [notice](freevideo_engine/licenses/NeMo-stt_en_conformer_ctc_small-NOTICE.txt)) | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) |
+| [OpenAI Whisper tiny](https://github.com/openai/whisper), © 2022 OpenAI, exported to ONNX and quantized to int8 by the sherpa-onnx project (optional download for spoken-language detection) | [MIT](freevideo_engine/licenses/Whisper-MIT.txt) |
+| [SenseVoiceSmall](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) by FunASR / FunAudioLLM (Alibaba Group), converted to ONNX and quantized to int8 by the sherpa-onnx project, used unchanged under its original name (optional download for multilingual subtitles; [notice](freevideo_engine/licenses/SenseVoice-NOTICE.txt)) | [FunASR Model Open Source License Agreement v1.1](freevideo_engine/licenses/FunASR-MODEL_LICENSE-v1.1.txt) |
 | [Lucide](https://lucide.dev) icons (image editor: eyedropper, hand, crop) | [ISC](freevideo_engine/licenses/Lucide-ISC.txt) |
-| [Noto Sans CJK SC and Noto Sans Mono CJK SC](https://github.com/notofonts/noto-cjk) 2.004, © 2014-2021 Adobe: subsets renamed FreeVideo Sans SC and FreeVideo Mono SC (image editor text, share card, Linux launcher) | [SIL OFL 1.1](freevideo_engine/licenses/Noto-CJK-Sans-OFL.txt) |
-| [Noto Serif CJK SC](https://github.com/notofonts/noto-cjk) 2.003, © 2017-2024 Adobe: subset renamed FreeVideo Serif SC (image editor text) | [SIL OFL 1.1](freevideo_engine/licenses/Noto-CJK-Serif-OFL.txt) |
-| [LXGW WenKai](https://github.com/lxgw/LxgwWenKai) v1.522, © 2021-2026 LXGW, © 2020 The Klee Project Authors: subset renamed FreeVideo Kai SC (image editor text) | [SIL OFL 1.1](freevideo_engine/licenses/LXGW-WenKai-OFL.txt) |
+| [Noto Sans CJK SC and Noto Sans Mono CJK SC](https://github.com/notofonts/noto-cjk) 2.004, © 2014-2021 Adobe: subsets renamed FreeVideo Sans SC and FreeVideo Mono SC (image editor text, editor text and subtitles / 剪辑视图的文字和字幕, share card, Linux launcher) | [SIL OFL 1.1](freevideo_engine/licenses/Noto-CJK-Sans-OFL.txt) |
+| [Noto Serif CJK SC](https://github.com/notofonts/noto-cjk) 2.003, © 2017-2024 Adobe: subset renamed FreeVideo Serif SC (image editor text, editor text and subtitles / 剪辑视图的文字和字幕) | [SIL OFL 1.1](freevideo_engine/licenses/Noto-CJK-Serif-OFL.txt) |
+| [LXGW WenKai](https://github.com/lxgw/LxgwWenKai) v1.522, © 2021-2026 LXGW, © 2020 The Klee Project Authors: subset renamed FreeVideo Kai SC (image editor text, editor text and subtitles / 剪辑视图的文字和字幕) | [SIL OFL 1.1](freevideo_engine/licenses/LXGW-WenKai-OFL.txt) |
 
 Original launcher license texts are included in Windows packages. Other runtime dependencies are listed in [constraints](constraints), with their license information in the installed packages.
 

__init__.py

@@ -5,6 +5,12 @@
 async def comfy_entrypoint():
     from .freevideo_engine.comfy_progress import register as progress_routes
     progress_routes()
+    try:
+        from .freevideo_engine.comfy_chain import register as chain_routes
+        chain_routes()
+    except Exception as error:
+        import logging
+        logging.warning('[FreeVideo] Chain routes not registered: %s', error)
     from .freevideo_engine.comfy_upload import register
     register()
     from .freevideo_engine.comfy_setup import register as setup_routes
@@ -17,11 +23,25 @@ async def comfy_entrypoint():
     update_routes()
     from .freevideo_engine.comfy_library import register as library_routes
     library_routes()
+    from .freevideo_engine.comfy_editor import register as editor_routes
+    editor_routes()
+    from .freevideo_engine.editor_export import register as editor_export_routes
+    editor_export_routes()
     from .freevideo_engine.comfy_capabilities import register as capabilities_routes
     capabilities_routes()
+    from .freevideo_engine.subtitle_ears import register as subtitle_ears_routes
+    subtitle_ears_routes()
+    from .freevideo_engine.subtitle_files import register as subtitle_file_routes
+    subtitle_file_routes()
     from .freevideo_engine.comfy_share import register as share_routes
     share_routes()
     from .freevideo_engine.comfy_metadata import register as workflow_backfill
     workflow_backfill()
+    try:
+        from .freevideo_engine.agent.routes import register as agent_routes
+        agent_routes()
+    except Exception as error:
+        import logging
+        logging.warning('[FreeVideo] Agent routes not registered: %s', error)
     from .freevideo_engine.comfy_nodes import FreeVideoExtension
     return FreeVideoExtension()

freevideo_engine/access_proxy.py

@@ -9,12 +9,14 @@
 import argparse
 import asyncio
 import hmac
+import os
 from pathlib import Path
 import ssl
 from urllib.parse import urlencode
 
 from aiohttp import ClientSession, ClientTimeout, DummyCookieJar, TCPConnector, WSMsgType, client_exceptions, web
 from yarl import URL
+from . import proxy_identity
 
 COOKIE = 'freevideo_access'
 DAYS = 30
@@ -54,7 +56,8 @@ def forwarded(request):
     headers = []
     for name, value in request.headers.items():
         lower = name.lower()
-        if lower in HOP_BY_HOP or lower == 'authorization':
+        if lower in HOP_BY_HOP or lower in ('authorization', proxy_identity.HEADER.lower(),
+                                           proxy_identity.PEER.lower(), proxy_identity.ORIGIN.lower()):
             continue
         if lower == 'cookie':
             value = '; '.join(part for part in value.split(';') if part.strip() and
@@ -69,6 +72,10 @@ def create(upstream, token):
     upstream = upstream.rstrip('/')
     sockets = 'ws' + upstream[len('http'):]
     client = []
+    hop_secret = os.environ.pop(proxy_identity.ENV, '')
+
+    def forwarding_headers(request):
+        return forwarded(request) + proxy_identity.headers(request, hop_secret)
 
     async def startup(app):
         # No total timeout: downloads and the progress socket last as long as they need.
@@ -82,7 +89,7 @@ async def forward(request):
         body = request.content if request.body_exists else None
         try:
             async with client[0].request(request.method, URL(upstream + request.raw_path, encoded=True),
-                                                     headers=forwarded(request), data=body, allow_redirects=False) as reply:
+                                                     headers=forwarding_headers(request), data=body, allow_redirects=False) as reply:
                 response = web.StreamResponse(status=reply.status, reason=reply.reason)
                 for name, value in reply.headers.items():
                     if name.lower() not in HOP_BY_HOP:
@@ -100,7 +107,7 @@ async def forward(request):
     async def websocket(request):
         browser = web.WebSocketResponse(max_msg_size=0)
         await browser.prepare(request)
-        headers = [(k, v) for k, v in forwarded(request) if k.lower() not in (
+        headers = [(k, v) for k, v in forwarding_headers(request) if k.lower() not in (
             'sec-websocket-key', 'sec-websocket-version', 'sec-websocket-extensions', 'sec-websocket-protocol')]
         try:
             async with client[0].ws_connect(URL(sockets + request.raw_path, encoded=True), headers=headers,

freevideo_engine/adaptive.py

@@ -659,13 +659,21 @@ def complete_observation(metrics, canvas, output, artifacts, telemetry):
     from .system import memory_complete, memory_peak
     from .validation import _retained_array_header, finite_number
     expected = canvas['frames']
+    continuation = metrics.get('continuation')
+    output_frames = expected
+    if continuation is not None:
+        from .continuation import OVERLAP_FRAMES
+        if (not isinstance(continuation, dict) or continuation.get('overlap_frames') != OVERLAP_FRAMES
+                or continuation.get('new_frames') != expected - OVERLAP_FRAMES):
+            raise ValueError('Invalid continuation output geometry')
+        output_frames = continuation['new_frames']
     preview = metrics.get('preview') is True
     if preview:
         # A preview decodes its first pass at the planned smaller canvas.
         first = (metrics.get('sampling_plan') or {}).get('first') or {}
         canvas = dict(canvas, width=first.get('width'), height=first.get('height'))
     if (metrics.get('success') is not True or metrics.get('phase') != 'complete'
-            or metrics.get('finite_latents') is not True or metrics.get('frames') != expected
+            or metrics.get('finite_latents') is not True or metrics.get('frames') != output_frames
             or any(metrics.get('geometry', {}).get(k) != canvas[k] for k in ('width','height','frames','fps'))):
         raise ValueError('A successful complete request with matching geometry is required.')
     steps = metrics.get('config', {}).get('steps')
@@ -678,26 +686,32 @@ def complete_observation(metrics, canvas, output, artifacts, telemetry):
             or not same_strategy(metrics, canvas) or not all(finite_number(t) and t > 0 for t in times)):
         raise ValueError('A probe or incomplete sample cannot become a resource observation.')
     rgb_shape, dtype = _retained_array_header(Path(artifacts)/'rgb.npy')
-    if str(dtype) != 'uint8' or rgb_shape != (expected, canvas['height'], canvas['width'], 3):
+    if str(dtype) != 'uint8' or rgb_shape != (output_frames, canvas['height'], canvas['width'], 3):
         raise ValueError('Retained video frames do not match the complete request.')
     audio_shape, audio_dtype = _retained_array_header(Path(artifacts)/'audio.npy')
     if (str(audio_dtype) != 'float32' or len(audio_shape) != 2 or audio_shape[0] != 2
-            or abs(audio_shape[1]/32000. - expected/canvas['fps']) > 1/canvas['fps']):
+            or abs(audio_shape[1]/32000. - output_frames/canvas['fps']) > 1/canvas['fps']):
         raise ValueError('Retained stereo audio does not match the request duration.')
     import av
     with av.open(str(output)) as container:
         video, audio = container.streams.video, container.streams.audio
         if len(video) != 1 or len(audio) != 1:
             raise ValueError('The completed MP4 must contain video and audio.')
         stream = video[0]
-        if ((stream.width, stream.height, stream.frames) != (canvas['width'],canvas['height'],expected)
+        # A request rendered at a larger size is saved at the size asked for.
+        shown = metrics.get('output_size') or canvas
+        if ((stream.width, stream.height, stream.frames) != (shown['width'],shown['height'],output_frames)
                 or float(stream.average_rate) != canvas['fps'] or audio[0].codec_context.channels != 2):
             raise ValueError('MP4 metadata does not match the requested output.')
     if metrics.get('sampling_reused') is True:
         # The output is validated, but these observations cover only a decoder
         # retry. Do not teach placement/ETA that a full request cost this little
         # time or occupied only the decoder's memory.
         return None
+    if continuation is not None:
+        # Clean prefix rows and the shorter decode have different timing from
+        # ordinary text generation. Do not seed that request's history with it.
+        return None
     if preview or metrics.get('first_pass_source') == 'preview':
         # A preview stops after its first pass, and its upscale reuses that pass
         # from an earlier request. Neither costs what a whole request costs.

freevideo_engine/agent/__init__.py

@@ -0,0 +1,15 @@
+"""Agent-native control of FreeVideo: job specs, media tools, headless edits and MCP.
+
+Agent features are enabled by default. Set FREEVIDEO_AGENT=0 (or false, no, off)
+where ComfyUI runs and for the managed commands to disable the agent API, token,
+commands and controls."""
+import os
+
+API_VERSION = 1
+SWITCH = 'FREEVIDEO_AGENT'
+
+
+def enabled(environ=None):
+    """On by default; only FREEVIDEO_AGENT=0/false/no/off explicitly disables it."""
+    value = (os.environ if environ is None else environ).get(SWITCH, '')
+    return value.strip().lower() not in ('0', 'false', 'no', 'off')

freevideo_engine/agent/access.py

@@ -0,0 +1,63 @@
+"""Peer and browser-origin policy, independent of bearer authentication."""
+import ipaddress
+from urllib.parse import urlsplit
+
+from .jobs import JobError
+from .. import proxy_identity
+
+
+def peer_address(request):
+    peer = request.transport.get_extra_info('peername') if request.transport else None
+    try:
+        address = ipaddress.ip_address(peer[0].split('%')[0])
+        address = address.ipv4_mapped if isinstance(address, ipaddress.IPv6Address) and address.ipv4_mapped else address
+        if address.is_loopback and proxy_identity.trusted(request):
+            address = ipaddress.ip_address(request.headers.get(proxy_identity.PEER, '').split('%')[0])
+        return address.ipv4_mapped if isinstance(address, ipaddress.IPv6Address) and address.ipv4_mapped else address
+    except (TypeError, ValueError, IndexError):
+        return None
+
+
+def lan_enabled():
+    from ..comfy_bridge import installation_root
+    from ..launcher_settings import Store
+    try:
+        return Store(installation_root()).currently_allows('lan_access')
+    except (OSError, ValueError):
+        return False
+
+
+def check(request, *, local=False):
+    address = peer_address(request)
+    proxied = proxy_identity.HEADER in request.headers
+    # Even a proxy client on loopback is never a local control-plane client.
+    if (proxied and (not proxy_identity.trusted(request) or local or not lan_enabled())
+            or address is None or not address.is_loopback and (local or not address.is_private or not lan_enabled())):
+        raise JobError('local_only', 'Peer is not permitted')
+    socket = request.transport.get_extra_info('sockname') if request.transport else None
+    try:
+        host = urlsplit(request.scheme + '://' + request.host)
+        allowed = {socket[0].split('%')[0]} if socket else set()
+        if address.is_loopback:
+            allowed.update(('localhost', '127.0.0.1', '::1'))
+        scheme = request.scheme
+        if proxied:
+            proxy_origin = urlsplit(request.headers.get(proxy_identity.ORIGIN, ''))
+            if (host.hostname, host.port) == (proxy_origin.hostname, proxy_origin.port):
+                allowed.add(proxy_origin.hostname)
+                socket = (proxy_origin.hostname, proxy_origin.port)
+                scheme = proxy_origin.scheme
+        if host.hostname not in allowed or host.port not in (None, socket[1]):
+            raise ValueError
+        if len(request.headers.getall('Origin', [])) > 1:
+            raise ValueError
+        origin = request.headers.get('Origin')
+        if origin is not None:
+            parsed = urlsplit(origin)
+            port = parsed.port or (443 if parsed.scheme == 'https' else 80)
+            own_port = host.port or (443 if scheme == 'https' else 80)
+            if (parsed.scheme != scheme or parsed.hostname != host.hostname or port != own_port
+                    or parsed.path or parsed.query or parsed.fragment or parsed.username or parsed.password):
+                raise ValueError
+    except (ValueError, TypeError, IndexError):
+        raise JobError('origin_forbidden', 'Origin or Host does not identify this instance') from None

freevideo_engine/agent/assets.py

@@ -0,0 +1,37 @@
+"""Public media identities; filesystem resolution stays behind the authenticated API."""
+from urllib.parse import quote, parse_qs, urlsplit
+from pathlib import PurePosixPath
+import re
+
+PREFIX = '/freevideo/agent/v1'
+
+
+def view(relative, kind='input'):
+    return dict(file=str(relative), url=PREFIX + '/media/' + kind + '/' + quote(str(relative), safe='/') + '/file')
+
+
+def public(value, _key=None):
+    """Migrate legacy media views while discarding server-local filesystem fields."""
+    if isinstance(value, list):
+        return [public(item, _key) for item in value]
+    if isinstance(value, str):
+        if _key in ('url', 'thumbnail') and value.startswith(PREFIX + '/'):
+            return value
+        if PurePosixPath(value).is_absolute() or re.match(r'^(?:[A-Za-z]:[\\/]|\\\\)', value):
+            return '[local path]'
+        return value
+    if not isinstance(value, dict):
+        return value
+    # Migrate the URL before recursing; /view is an obsolete media URL, not
+    # an absolute filesystem path to preserve.
+    result = {k: v for k, v in value.items() if k != 'path'}
+    url = result.get('url')
+    if isinstance(url, str) and url.startswith('/view?'):
+        query = parse_qs(urlsplit(url).query)
+        relative = result.get('file') or '/'.join((query.get('subfolder', [''])[0], query.get('filename', [''])[0])).lstrip('/')
+        kind = query.get('type', ['output'])[0]
+        if relative and kind in ('input', 'output') and not PurePosixPath(relative).is_absolute() and '..' not in PurePosixPath(relative).parts:
+            result.update(view(relative, kind))
+        else:
+            result.pop('url', None)
+    return {k: public(v, k) for k, v in result.items()}