Add official Kling API providers
This commit is contained in:
xucailiang
parent
0c202b507a
commit
7c5dfdd31a
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
name: ai-video-gen
|
||||
description: |
|
||||
Generate AI videos from text prompts using multiple provider gateways. Use when: (1) Generating videos from text descriptions, (2) Creating AI-generated video clips for content production, (3) Image-to-video generation with a reference image, (4) Choosing between video generation providers (VEO, Kling, Sora, Runway, Seedance, MiniMax). Supports two gateways: HeyGen API and fal.ai API.
|
||||
Generate AI videos from text prompts using multiple provider gateways. Use when: (1) Generating videos from text descriptions, (2) Creating AI-generated video clips for content production, (3) Image-to-video generation with a reference image, (4) Choosing between video generation providers (VEO, Kling, Sora, Runway, Seedance, MiniMax). Supports HeyGen API, fal.ai API, and Kling official direct API.
|
||||
allowed-tools: mcp__heygen__*
|
||||
metadata:
|
||||
openclaw:
|
||||
|
|
@ -9,16 +9,18 @@ metadata:
|
|||
env_any:
|
||||
- HEYGEN_API_KEY
|
||||
- FAL_KEY
|
||||
- KLING_API_KEY
|
||||
---
|
||||
|
||||
# Video Generation (Multi-Gateway)
|
||||
|
||||
Generate AI videos from text prompts. Supports multiple providers via two API gateways:
|
||||
Generate AI videos from text prompts. Supports multiple providers via three API paths:
|
||||
|
||||
| Gateway | Env Variable | Providers | Tool |
|
||||
|---------|-------------|-----------|------|
|
||||
| **fal.ai** | `FAL_KEY` | **Seedance 2.0** (standard + fast), Kling v3/v2.1, MiniMax, VEO | `seedance_video`, `kling_video`, `minimax_video`, `veo_video` |
|
||||
| **HeyGen** | `HEYGEN_API_KEY` | VEO 3.1, Kling Pro, Sora v2, Runway Gen-4, Seedance Pro / Lite (1.x) | `heygen_video` |
|
||||
| **Kling Official** | `KLING_API_KEY` | Kling official Classic, Turbo, and basic Omni video | `kling_official_video` |
|
||||
|
||||
**Preferred premium default — Seedance 2.0.** When any premium gateway is configured (`FAL_KEY` → `seedance_video`, or HeyGen's Video Agent / Avatar Shots path), Seedance 2.0 is the preferred default for cinematic, trailer, and high-fidelity clip work. It is the only model in the fleet with **single-pass native synchronized audio, multi-shot generation, director-level camera control, and lip-sync from quoted dialogue**, and it ranks #1 on Artificial Analysis Elo as of early 2026. Switch off it only when the user has a specific reason (budget, provider preference, stylistic fit like VEO for photoreal landscape or Kling for specific anime look). See Layer 3 `seedance-2-0` for the authoritative prompting and parameter guide.
|
||||
|
||||
|
|
@ -30,9 +32,12 @@ Use whichever configured gateway best matches the user's available providers and
|
|||
|
||||
- **HeyGen:** Set `HEYGEN_API_KEY` to access the multi-model gateway.
|
||||
- **fal.ai:** Set `FAL_KEY` to access Kling, MiniMax, and Veo through fal.ai.
|
||||
- **Kling Official:** Set `KLING_API_KEY` to access Kling's official direct API via `provider="kling_official"`.
|
||||
|
||||
Do not describe either gateway as the default or top choice without checking the registry and current task fit first.
|
||||
|
||||
fal.ai Kling (`kling_video`, `provider="kling"`) and Kling Official (`kling_official_video`, `provider="kling_official"`) are different paths. Do not reuse fal.ai queue URLs, `FAL_KEY`, or image upload behavior when the official provider is selected.
|
||||
|
||||
```bash
|
||||
curl -X POST "https://api.heygen.com/v1/workflows/executions" \
|
||||
-H "X-Api-Key: $HEYGEN_API_KEY" \
|
||||
|
|
|
|||
|
|
@ -0,0 +1,200 @@
|
|||
---
|
||||
name: kling-official
|
||||
description: Official Kling direct API guidance for OpenMontage providers. Use before calling `kling_official_video`, `kling_official_image`, `kling_tts`, `kling_avatar`, or `kling_lip_sync`.
|
||||
metadata:
|
||||
openclaw:
|
||||
requires:
|
||||
env_any:
|
||||
- KLING_API_KEY
|
||||
---
|
||||
|
||||
# Kling Official Direct API
|
||||
|
||||
Use this skill for OpenMontage tools with `provider="kling_official"`. This is not the fal.ai Kling gateway. Official Kling uses `KLING_API_KEY`, optional `KLING_API_BASE_URL`, and `Authorization: Bearer <KLING_API_KEY>`.
|
||||
|
||||
## Provider Split
|
||||
|
||||
- `kling_video` uses fal.ai, `FAL_KEY`, fal.ai queue URLs, and `provider="kling"`.
|
||||
- `kling_official_video` uses Kling official API, `KLING_API_KEY`, official task protocols, and `provider="kling_official"`.
|
||||
- `kling_official_image` uses the same official auth and task protocol for image generation.
|
||||
- `kling_tts` uses the official audio TTS endpoint and stays in the existing `tts` capability.
|
||||
- `kling_avatar` and `kling_lip_sync` use official avatar/lip-sync endpoints and stay in the existing `avatar` capability. They do not replace local `talking_head` or `lip_sync`.
|
||||
|
||||
Never silently switch between these paths. If the selected provider is unavailable, surface the blocker and ask before substituting.
|
||||
|
||||
## Auth And Endpoint
|
||||
|
||||
Default base URL:
|
||||
|
||||
```text
|
||||
https://api-singapore.klingai.com
|
||||
```
|
||||
|
||||
Users may override it with `KLING_API_BASE_URL`, for example for a regional endpoint. All requests send JSON and:
|
||||
|
||||
```text
|
||||
Authorization: Bearer <KLING_API_KEY>
|
||||
```
|
||||
|
||||
## Task Protocols
|
||||
|
||||
Classic APIs:
|
||||
|
||||
- Create ID path: `data.task_id`
|
||||
- Statuses: `submitted`, `processing`, `succeed`, `failed`
|
||||
- Result paths: `data.task_result.videos[]`, `data.task_result.images[]`, `data.task_result.audios[]`
|
||||
|
||||
Turbo APIs:
|
||||
|
||||
- Create ID path: `data.id`
|
||||
- Poll path: `GET /tasks?task_ids=<id>`
|
||||
- Statuses: `submitted`, `processing`, `succeeded`, `failed`
|
||||
- Result path: `data[0].outputs[]`
|
||||
|
||||
Keep the parsers separate. Do not write a fuzzy parser that guesses between `task_id` and `id` or between `succeed` and `succeeded`.
|
||||
|
||||
## Omni References
|
||||
|
||||
Video Omni and Image Omni stay inside the existing provider tools through `api_family="omni"`.
|
||||
Do not create selector-level Omni operations.
|
||||
|
||||
Video Omni accepts official reference structures:
|
||||
|
||||
- `image_list[]` with `image_url` and optional `type` such as `first_frame` or `end_frame`.
|
||||
- `video_list[]` with `video_url`, official `refer_type` values such as `feature` or `base`, and optional `keep_original_sound`.
|
||||
- `element_list[]` with official `element_id` values.
|
||||
- Structured `multi_prompt[]`; do not split natural language into shots automatically.
|
||||
|
||||
Local image references may be normalized through `tools/_kling/media.py`. Local video paths must not be silently uploaded through fal.ai; ask for or require a reachable URL.
|
||||
|
||||
Image Omni accepts `image_list[]` with official `image` values. Prompt placeholders such as `<<<image_1>>>` must map stably to the provided image order. If the prompt already contains placeholders, validate that the referenced images exist and do not insert duplicates.
|
||||
|
||||
## Phase 3 Capability Boundaries
|
||||
|
||||
TTS, avatar, and lip sync are provider additions to existing OpenMontage capabilities. Audio effects and video effects are official Kling endpoints, but they are not registered as default OpenMontage tools until a pipeline has a stable capability slot for them.
|
||||
|
||||
- Do not add `sound_effects` or `video_effects` capabilities from inside a provider implementation.
|
||||
- Do not let video effects enter the ordinary `video_generation` selector path.
|
||||
- Do not disguise short sound effects as long background music unless a pipeline explicitly consumes that shape and the tool's `best_for` / `not_good_for` says so.
|
||||
|
||||
## Video Parameters
|
||||
|
||||
Use `operation` for OpenMontage semantics:
|
||||
|
||||
- `text_to_video`
|
||||
- `image_to_video`
|
||||
- `reference_to_video`
|
||||
|
||||
Use `api_family` for official protocol choice:
|
||||
|
||||
- `classic`
|
||||
- `turbo`
|
||||
- `omni`
|
||||
|
||||
Important constraints:
|
||||
|
||||
- Official video provider input schema must not expose top-level `image_url`; use `reference_image_url` or `reference_image_path`.
|
||||
- Classic image-to-video accepts `reference_image_url` or a local path converted to raw base64 in official field `image`.
|
||||
- Turbo image-to-video requires a URL first frame. Do not upload local files through fal.ai as a fallback.
|
||||
- Send `aspect_ratio` only where the current schema supports it: Classic text-to-video, Turbo text-to-video, and Video Omni.
|
||||
- Default paid path should avoid `4k`, native sound, or batch behavior unless explicitly selected.
|
||||
|
||||
## Image Parameters
|
||||
|
||||
Use `api_family="generation"` for `/v1/images/generations` and `api_family="omni"` for `/v1/images/omni-image`.
|
||||
|
||||
Generation/edit path:
|
||||
|
||||
- `prompt` is required and should stay under the official 2500 character limit.
|
||||
- `image_url` passes through as official `image`.
|
||||
- `image_path` is converted to raw base64 and sent as official `image`.
|
||||
- `image_reference` can be `subject` or `face`.
|
||||
|
||||
Omni path:
|
||||
|
||||
- Put references in `image_list[]` using official `image` values.
|
||||
- Use prompt placeholders such as `<<<image_1>>>` only when the prompt needs to bind a specific reference image.
|
||||
|
||||
## TTS Parameters
|
||||
|
||||
`kling_tts` uses:
|
||||
|
||||
- `text`
|
||||
- `voice_id`
|
||||
- `voice_language`, currently `zh` or `en`
|
||||
- `voice_speed`
|
||||
|
||||
Require an explicit `voice_id` unless an official account-specific default has been verified. Do not hard-code a made-up voice. Download every returned audio item, set `data.output_path` to the first local file, and include `voice_id`, `voice_language`, `voice_speed`, `task_id`, and non-zero `cost_usd`.
|
||||
|
||||
## Avatar Parameters
|
||||
|
||||
`kling_avatar` uses `/v1/videos/avatar/image2video` and accepts:
|
||||
|
||||
- avatar image via URL or local path converted to raw base64
|
||||
- `audio_id` or `sound_file`
|
||||
- optional `prompt`
|
||||
- `mode`, such as `std` or `pro`
|
||||
|
||||
Keep it separate from local `talking_head`. Pipelines that want Kling avatar output must list and choose it explicitly.
|
||||
|
||||
## Lip Sync Parameters
|
||||
|
||||
`kling_lip_sync` has two steps:
|
||||
|
||||
1. `POST /v1/videos/identify-face` with `video_id` or `video_url`
|
||||
2. `POST /v1/videos/advanced-lip-sync` with `session_id`, `face_choose[]`, and `audio_id` or `sound_file`
|
||||
|
||||
Local video paths must not be silently uploaded through fal.ai or any other provider. If multiple faces are returned and the user did not pass `face_id` or `face_choose`, stop and return the face list for confirmation unless `auto_select_face=True` was explicitly set. If auto-selecting, record the selection reason and selected face in the result/artifact.
|
||||
|
||||
## Audio Effects And Video Effects
|
||||
|
||||
Official Kling audio effects (`/v1/audio/text-to-audio`, `/v1/audio/video-to-audio`) and video effects (`/v1/videos/effects`) are intentionally not default OpenMontage selector tools in Phase 3. Record the non-mapping reason in docs/tests instead of registering tools that current pipelines might misuse.
|
||||
|
||||
## Elements Helper
|
||||
|
||||
Elements are an internal Kling Official helper, not a new OpenMontage tool capability.
|
||||
Use `tools/_kling/elements.py` to normalize `element_list[].element_id`, optionally query read-only element endpoints, and record element metadata when queried. Do not create or delete elements from the default provider path.
|
||||
|
||||
## Account Usage Helper
|
||||
|
||||
Account Usage is diagnostic only. Use `tools/_kling/account.py` for low-frequency `/account/costs` checks, with local cache and throttle protection. Do not call it before every generation and do not put it in selectors or production pipeline stages.
|
||||
|
||||
For `1101` or `1102`, surface that the account or resource pack is exhausted and include an account-usage diagnostic hint.
|
||||
|
||||
## Callback Notes
|
||||
|
||||
Providers may accept `callback_url`, but polling remains the default execution mode.
|
||||
|
||||
- Classic and Omni paths pass `callback_url` at the top level.
|
||||
- Turbo paths pass it as `options.callback_url`.
|
||||
- Successful results should record `callback_requested=true`, `polling_used=true`, the `callback_url`, and `task_id`.
|
||||
- Validate callback URLs before sending; only absolute `http` or `https` URLs should pass.
|
||||
|
||||
## Error Handling
|
||||
|
||||
Surface official `code`, `message`, and `request_id` whenever available.
|
||||
|
||||
Do not retry:
|
||||
|
||||
- Auth failures: `1000`-`1004`
|
||||
- Balance/resource-pack exhaustion: `1101`, `1102`
|
||||
- Permission/model access: `1103`
|
||||
- Parameter errors: `1200`, `1201`
|
||||
- Safety policy: `1301`
|
||||
|
||||
Limited retry is acceptable for:
|
||||
|
||||
- `1302` request too fast
|
||||
- `1303` concurrency/resource-pack slot limit
|
||||
- `5000`, `5001`, `5002` server/maintenance/backlog errors
|
||||
|
||||
For `1303`, explain that the account hit a concurrency or resource-pack slot limit.
|
||||
|
||||
## Cost Governance
|
||||
|
||||
Official Kling is a paid remote API. Provider tools must return non-zero conservative estimates from `estimate_cost()` and include `cost_usd` on successful paid results. Treat estimates as low-confidence until account usage reconciliation is implemented.
|
||||
High-cost Omni inputs such as multiple references, element IDs, `result_type="series"`, `mode="4k"`, and `sound="on"` must increase or flag the cost estimate.
|
||||
|
||||
## Prompt Notes
|
||||
|
||||
For video, start from the universal OpenMontage video prompt skeleton: subject, subject motion, scene, spatial framing, and camera. Kling tends to respond well to clear temporal action order, camera movement verbs, and concise negative prompts. For reference workflows, state what should stay consistent from the reference and what should change.
|
||||
|
|
@ -0,0 +1,456 @@
|
|||
---
|
||||
name: video-toolkit
|
||||
description: Create professional videos autonomously using Codex-video-toolkit — AI voiceovers, image generation, music, talking heads, and Remotion rendering.
|
||||
metadata:
|
||||
openclaw:
|
||||
emoji: "🎬"
|
||||
skillKey: "video-toolkit"
|
||||
os: ["darwin", "linux"]
|
||||
requires:
|
||||
bins: ["node", "python3", "ffmpeg", "npm"]
|
||||
---
|
||||
|
||||
# Video Toolkit
|
||||
|
||||
Create professional explainer videos from a text brief. The toolkit uses open-source AI models on cloud GPUs (Modal or RunPod) for voiceover, image generation, music, and talking head animation. Remotion (React) handles composition and rendering.
|
||||
|
||||
## CRITICAL: Toolkit Path
|
||||
|
||||
The toolkit lives at a fixed path. **ALWAYS `cd` here before running any tool command.**
|
||||
|
||||
```bash
|
||||
TOOLKIT=~/.openclaw/workspace/Codex-video-toolkit
|
||||
cd $TOOLKIT
|
||||
```
|
||||
|
||||
**NEVER run tool commands from inside a project directory.** Tools resolve paths relative to the toolkit root.
|
||||
|
||||
## Setup
|
||||
|
||||
### Step 1: Check Current State
|
||||
|
||||
```bash
|
||||
cd ~/.openclaw/workspace/Codex-video-toolkit
|
||||
python3 tools/verify_setup.py
|
||||
```
|
||||
|
||||
If everything shows `[x]`, skip to "Quick Test" below. Otherwise continue setup.
|
||||
|
||||
### Step 2: Install Python Dependencies
|
||||
|
||||
```bash
|
||||
cd ~/.openclaw/workspace/Codex-video-toolkit
|
||||
pip3 install --break-system-packages -r tools/requirements.txt
|
||||
```
|
||||
|
||||
Note: `--break-system-packages` is needed on Debian/Ubuntu with managed Python (PEP 668). Safe inside containers.
|
||||
|
||||
### Step 3: Configure Cloud GPU Endpoints
|
||||
|
||||
The toolkit needs cloud GPU endpoint URLs in `.env`. Check if `.env` exists and has Modal endpoints:
|
||||
|
||||
```bash
|
||||
cat ~/.openclaw/workspace/Codex-video-toolkit/.env | grep MODAL
|
||||
```
|
||||
|
||||
If Modal endpoints are configured, you're ready. If not, **ask the user to provide Modal endpoint URLs** or set up Modal:
|
||||
|
||||
```bash
|
||||
pip3 install --break-system-packages modal
|
||||
python3 -m modal setup # Opens browser for authentication
|
||||
|
||||
# Deploy each tool — capture the endpoint URL from output
|
||||
cd ~/.openclaw/workspace/Codex-video-toolkit
|
||||
modal deploy docker/modal-qwen3-tts/app.py
|
||||
modal deploy docker/modal-flux2/app.py
|
||||
modal deploy docker/modal-music-gen/app.py
|
||||
modal deploy docker/modal-sadtalker/app.py
|
||||
modal deploy docker/modal-image-edit/app.py
|
||||
modal deploy docker/modal-upscale/app.py
|
||||
modal deploy docker/modal-propainter/app.py
|
||||
modal deploy docker/modal-ltx2/app.py # Requires: modal secret create huggingface-token HF_TOKEN=hf_...
|
||||
```
|
||||
|
||||
**LTX-2 prerequisite:** Before deploying LTX-2, create a HuggingFace secret and accept the [Gemma 3 license](https://huggingface.co/google/gemma-3-12b-it-qat-q4_0-unquantized):
|
||||
```bash
|
||||
modal secret create huggingface-token HF_TOKEN=hf_your_read_access_token
|
||||
```
|
||||
|
||||
Add each URL to `.env`:
|
||||
```
|
||||
MODAL_QWEN3_TTS_ENDPOINT_URL=https://...modal.run
|
||||
MODAL_FLUX2_ENDPOINT_URL=https://...modal.run
|
||||
MODAL_MUSIC_GEN_ENDPOINT_URL=https://...modal.run
|
||||
MODAL_SADTALKER_ENDPOINT_URL=https://...modal.run
|
||||
MODAL_IMAGE_EDIT_ENDPOINT_URL=https://...modal.run
|
||||
MODAL_UPSCALE_ENDPOINT_URL=https://...modal.run
|
||||
MODAL_DEWATERMARK_ENDPOINT_URL=https://...modal.run
|
||||
MODAL_LTX2_ENDPOINT_URL=https://...modal.run
|
||||
```
|
||||
|
||||
Optional but recommended — Cloudflare R2 for reliable file transfer:
|
||||
```
|
||||
R2_ACCOUNT_ID=...
|
||||
R2_ACCESS_KEY_ID=...
|
||||
R2_SECRET_ACCESS_KEY=...
|
||||
R2_BUCKET_NAME=video-toolkit
|
||||
```
|
||||
|
||||
### Step 4: Verify and Quick Test
|
||||
|
||||
```bash
|
||||
cd ~/.openclaw/workspace/Codex-video-toolkit
|
||||
python3 tools/verify_setup.py
|
||||
```
|
||||
|
||||
All tools should show `[x]`. Then run a quick test to confirm the GPU pipeline works:
|
||||
|
||||
```bash
|
||||
cd ~/.openclaw/workspace/Codex-video-toolkit
|
||||
python3 tools/qwen3_tts.py --text "Hello, this is a test." --speaker Ryan --tone warm --output /tmp/video-toolkit-test.mp3 --cloud modal
|
||||
```
|
||||
|
||||
If you get a valid .mp3 file, setup is complete. If it fails, check:
|
||||
- `.env` has the correct `MODAL_QWEN3_TTS_ENDPOINT_URL`
|
||||
- Run `python3 tools/verify_setup.py --json` and check `modal_tools` for which endpoints are missing
|
||||
|
||||
**Cost:** Modal includes $30/month free compute. A typical 60s video costs $1-3.
|
||||
|
||||
---
|
||||
|
||||
## Creating a Video
|
||||
|
||||
### Step 1: Create Project
|
||||
|
||||
```bash
|
||||
cd ~/.openclaw/workspace/Codex-video-toolkit
|
||||
cp -r templates/product-demo projects/PROJECT_NAME
|
||||
cd projects/PROJECT_NAME
|
||||
npm install
|
||||
```
|
||||
|
||||
Templates: `product-demo` (marketing/explainer), `sprint-review`, `sprint-review-v2` (composable scenes).
|
||||
|
||||
### Step 2: Write Config
|
||||
|
||||
Edit `projects/PROJECT_NAME/src/config/demo-config.ts`:
|
||||
|
||||
```typescript
|
||||
export const demoConfig: ProductDemoConfig = {
|
||||
product: {
|
||||
name: 'My Product',
|
||||
tagline: 'What it does in one line',
|
||||
website: 'example.com',
|
||||
},
|
||||
scenes: [
|
||||
{ type: 'title', durationSeconds: 9, content: { headline: '...', subheadline: '...' } },
|
||||
{ type: 'problem', durationSeconds: 14, content: { headline: '...', problems: ['...', '...'] } },
|
||||
{ type: 'solution', durationSeconds: 13, content: { headline: '...', highlights: ['...', '...'] } },
|
||||
{ type: 'stats', durationSeconds: 12, content: { stats: [{value: '99%', label: '...'}, ...] } },
|
||||
{ type: 'cta', durationSeconds: 10, content: { headline: '...', links: ['...'] } },
|
||||
],
|
||||
audio: {
|
||||
backgroundMusicFile: 'audio/bg-music.mp3',
|
||||
backgroundMusicVolume: 0.12,
|
||||
},
|
||||
};
|
||||
```
|
||||
|
||||
Scene types: `title`, `problem`, `solution`, `demo`, `feature`, `stats`, `cta`.
|
||||
|
||||
**Duration rule:** Estimate `durationSeconds` as `ceil(word_count / 2.5) + 2`. You will adjust this after generating audio in Step 4.
|
||||
|
||||
### Step 3: Write Voiceover Script
|
||||
|
||||
Create `projects/PROJECT_NAME/VOICEOVER-SCRIPT.md`:
|
||||
|
||||
```markdown
|
||||
## Scene 1: Title (9s, ~17 words)
|
||||
Build videos with AI. The product name toolkit makes it easy.
|
||||
|
||||
## Scene 2: Problem (14s, ~30 words)
|
||||
The problem statement goes here. Keep it punchy and relatable.
|
||||
```
|
||||
|
||||
**Word budget per scene:** `(durationSeconds - 2) * 2.5` words. The -2 accounts for 1s audio delay + 1s padding.
|
||||
|
||||
### Step 4: Generate Assets
|
||||
|
||||
**CRITICAL: All commands below MUST be run from the toolkit root, not the project directory.**
|
||||
|
||||
```bash
|
||||
cd ~/.openclaw/workspace/Codex-video-toolkit
|
||||
```
|
||||
|
||||
#### 4a. Background Music
|
||||
|
||||
```bash
|
||||
cd ~/.openclaw/workspace/Codex-video-toolkit
|
||||
python3 tools/music_gen.py \
|
||||
--preset corporate-bg \
|
||||
--duration 90 \
|
||||
--output projects/PROJECT_NAME/public/audio/bg-music.mp3 \
|
||||
--cloud modal
|
||||
```
|
||||
|
||||
Presets: `corporate-bg`, `upbeat-tech`, `ambient`, `dramatic`, `tension`, `hopeful`, `cta`, `lofi`.
|
||||
|
||||
#### 4b. Voiceover (per-scene)
|
||||
|
||||
Generate ONE .mp3 file PER SCENE. Do NOT generate a single voiceover file.
|
||||
|
||||
```bash
|
||||
cd ~/.openclaw/workspace/Codex-video-toolkit
|
||||
|
||||
# Scene 01
|
||||
python3 tools/qwen3_tts.py \
|
||||
--text "The voiceover text for scene one." \
|
||||
--speaker Ryan --tone warm \
|
||||
--output projects/PROJECT_NAME/public/audio/scenes/01.mp3 \
|
||||
--cloud modal
|
||||
|
||||
# Scene 02
|
||||
python3 tools/qwen3_tts.py \
|
||||
--text "The voiceover text for scene two." \
|
||||
--speaker Ryan --tone warm \
|
||||
--output projects/PROJECT_NAME/public/audio/scenes/02.mp3 \
|
||||
--cloud modal
|
||||
|
||||
# ... repeat for each scene
|
||||
```
|
||||
|
||||
**Speakers:** `Ryan`, `Aiden`, `Vivian`, `Serena`, `Uncle_Fu`, `Dylan`, `Eric`, `Ono_Anna`, `Sohee`
|
||||
**Tones:** `neutral`, `warm`, `professional`, `excited`, `calm`, `serious`, `storyteller`, `tutorial`
|
||||
|
||||
For voice cloning (needs a reference recording):
|
||||
```bash
|
||||
cd ~/.openclaw/workspace/Codex-video-toolkit
|
||||
python3 tools/qwen3_tts.py \
|
||||
--text "Text to speak" \
|
||||
--ref-audio assets/voices/reference.m4a \
|
||||
--ref-text "Exact transcript of the reference audio" \
|
||||
--output projects/PROJECT_NAME/public/audio/scenes/01.mp3 \
|
||||
--cloud modal
|
||||
```
|
||||
|
||||
#### 4c. Scene Images
|
||||
|
||||
```bash
|
||||
cd ~/.openclaw/workspace/Codex-video-toolkit
|
||||
python3 tools/flux2.py \
|
||||
--prompt "Dark tech background with blue geometric grid, cinematic lighting" \
|
||||
--width 1920 --height 1080 \
|
||||
--output projects/PROJECT_NAME/public/images/title-bg.png \
|
||||
--cloud modal
|
||||
```
|
||||
|
||||
Image presets (use `--preset` instead of `--prompt --width --height`):
|
||||
`title-bg`, `problem`, `solution`, `demo-bg`, `stats-bg`, `cta`, `thumbnail`, `portrait-bg`
|
||||
|
||||
```bash
|
||||
cd ~/.openclaw/workspace/Codex-video-toolkit
|
||||
python3 tools/flux2.py \
|
||||
--preset title-bg \
|
||||
--output projects/PROJECT_NAME/public/images/title-bg.png \
|
||||
--cloud modal
|
||||
```
|
||||
|
||||
#### 4d. Video Clips — B-Roll & Animated Backgrounds (optional)
|
||||
|
||||
Generate AI video clips for b-roll cutaways, animated slide backgrounds, or intro/outro sequences:
|
||||
|
||||
```bash
|
||||
cd ~/.openclaw/workspace/Codex-video-toolkit
|
||||
|
||||
# B-roll clip from text
|
||||
python3 tools/ltx2.py \
|
||||
--prompt "Aerial drone shot over a European city at golden hour, cinematic wide angle" \
|
||||
--output projects/PROJECT_NAME/public/videos/broll-europe.mp4 \
|
||||
--cloud modal
|
||||
|
||||
# Animate a slide/screenshot (image-to-video)
|
||||
python3 tools/ltx2.py \
|
||||
--prompt "Gentle particle effects, soft ambient light shifts, very slight camera drift" \
|
||||
--input projects/PROJECT_NAME/public/images/title-bg.png \
|
||||
--output projects/PROJECT_NAME/public/videos/animated-title.mp4 \
|
||||
--cloud modal
|
||||
|
||||
# Abstract intro/outro background
|
||||
python3 tools/ltx2.py \
|
||||
--prompt "Dark moody abstract background with flowing blue light streaks, bokeh particles, cinematic" \
|
||||
--output projects/PROJECT_NAME/public/videos/intro-bg.mp4 \
|
||||
--cloud modal
|
||||
```
|
||||
|
||||
Use in Remotion compositions with `<OffthreadVideo>`:
|
||||
```tsx
|
||||
<OffthreadVideo src={staticFile('videos/broll-europe.mp4')} />
|
||||
```
|
||||
|
||||
**LTX-2 rules:**
|
||||
- Max ~8 seconds per clip (193 frames at 24fps). Default is ~5s (121 frames).
|
||||
- Width/height must be divisible by 64. Default: 768x512.
|
||||
- ~$0.20-0.25 per clip, ~2.5 min generation time.
|
||||
- Cold start ~60-90s. Subsequent clips on warm GPU are faster.
|
||||
- Generated audio is ambient only — use voiceover/music tools for speech and music.
|
||||
- ~30% of generations may have training data artifacts (logos/text). Re-run with `--seed` to vary.
|
||||
|
||||
#### 4e. Talking Head Narrator (optional)
|
||||
|
||||
Generate a presenter portrait, then animate per-scene clips:
|
||||
|
||||
```bash
|
||||
cd ~/.openclaw/workspace/Codex-video-toolkit
|
||||
|
||||
# 1. Generate portrait
|
||||
python3 tools/flux2.py \
|
||||
--prompt "Professional presenter portrait, clean style, dark background, facing camera, upper body" \
|
||||
--width 1024 --height 576 \
|
||||
--output projects/PROJECT_NAME/public/images/presenter.png \
|
||||
--cloud modal
|
||||
|
||||
# 2. Generate per-scene narrator clips (one per scene, NOT one long video)
|
||||
python3 tools/sadtalker.py \
|
||||
--image projects/PROJECT_NAME/public/images/presenter.png \
|
||||
--audio projects/PROJECT_NAME/public/audio/scenes/01.mp3 \
|
||||
--preprocess full --still --expression-scale 0.8 \
|
||||
--output projects/PROJECT_NAME/public/narrator-01.mp4 \
|
||||
--cloud modal
|
||||
|
||||
# Repeat for each scene that needs a narrator
|
||||
```
|
||||
|
||||
**SadTalker rules — follow these exactly:**
|
||||
- **ALWAYS** use `--preprocess full` (default `crop` outputs a square, wrong aspect ratio)
|
||||
- **ALWAYS** use `--still` (reduces head movement, looks professional)
|
||||
- **ALWAYS** generate per-scene clips (6-15s each), NEVER one long video
|
||||
- Processing: ~3-4 min per 10s of audio on Modal A10G
|
||||
- `--expression-scale 0.8` keeps expressions subtle (range 0.0-1.5)
|
||||
|
||||
#### 4e. Image Editing (optional)
|
||||
|
||||
Create scene variants from existing images:
|
||||
|
||||
```bash
|
||||
cd ~/.openclaw/workspace/Codex-video-toolkit
|
||||
python3 tools/image_edit.py \
|
||||
--input projects/PROJECT_NAME/public/images/title-bg.png \
|
||||
--prompt "Make it darker with red tones, more ominous" \
|
||||
--output projects/PROJECT_NAME/public/images/problem-bg.png \
|
||||
--cloud modal
|
||||
```
|
||||
|
||||
#### 4f. Upscaling (optional)
|
||||
|
||||
```bash
|
||||
cd ~/.openclaw/workspace/Codex-video-toolkit
|
||||
python3 tools/upscale.py \
|
||||
--input projects/PROJECT_NAME/public/images/some-image.png \
|
||||
--output projects/PROJECT_NAME/public/images/some-image-4x.png \
|
||||
--scale 4 --cloud modal
|
||||
```
|
||||
|
||||
### Step 5: Sync Timing
|
||||
|
||||
**ALWAYS do this after generating voiceover.** Audio duration differs from estimates.
|
||||
|
||||
```bash
|
||||
cd ~/.openclaw/workspace/Codex-video-toolkit
|
||||
for f in projects/PROJECT_NAME/public/audio/scenes/*.mp3; do
|
||||
echo "$(basename $f): $(ffprobe -v error -show_entries format=duration -of csv=p=0 "$f")s"
|
||||
done
|
||||
```
|
||||
|
||||
Update each scene's `durationSeconds` in `demo-config.ts` to: `ceil(actual_audio_duration + 2)`.
|
||||
|
||||
Example: if `01.mp3` is 6.8s, set scene 1 `durationSeconds` to `9` (ceil(6.8 + 2) = 9).
|
||||
|
||||
### Step 6: Review Still Frames
|
||||
|
||||
```bash
|
||||
cd ~/.openclaw/workspace/Codex-video-toolkit/projects/PROJECT_NAME
|
||||
npx remotion still src/index.ts ProductDemo --frame=100 --output=/tmp/review-scene1.png
|
||||
npx remotion still src/index.ts ProductDemo --frame=400 --output=/tmp/review-scene2.png
|
||||
```
|
||||
|
||||
Check: text truncation, animation timing, narrator PiP positioning, background contrast.
|
||||
|
||||
### Step 7: Render
|
||||
|
||||
```bash
|
||||
cd ~/.openclaw/workspace/Codex-video-toolkit/projects/PROJECT_NAME
|
||||
npm run render
|
||||
```
|
||||
|
||||
**Output:** `out/ProductDemo.mp4`
|
||||
|
||||
---
|
||||
|
||||
## Composition Patterns
|
||||
|
||||
### Per-Scene Audio
|
||||
|
||||
Use per-scene audio with a 1-second delay (`from={30}` = 30 frames = 1s at 30fps):
|
||||
|
||||
```tsx
|
||||
<Sequence from={30}>
|
||||
<Audio src={staticFile('audio/scenes/01.mp3')} volume={1} />
|
||||
</Sequence>
|
||||
```
|
||||
|
||||
### Per-Scene Narrator PiP
|
||||
|
||||
```tsx
|
||||
<Sequence from={30}>
|
||||
<OffthreadVideo
|
||||
src={staticFile('narrator-01.mp4')}
|
||||
style={{ width: 320, height: 180, objectFit: 'cover' }}
|
||||
muted
|
||||
/>
|
||||
</Sequence>
|
||||
```
|
||||
|
||||
**ALWAYS use `<OffthreadVideo>`, NEVER `<video>`.** Remotion requires its own component for frame-accurate rendering.
|
||||
|
||||
### Transitions
|
||||
|
||||
```tsx
|
||||
import { TransitionSeries, linearTiming } from '@remotion/transitions';
|
||||
import { fade } from '@remotion/transitions/fade';
|
||||
import { glitch } from '../../../lib/transitions/presentations/glitch';
|
||||
import { lightLeak } from '../../../lib/transitions/presentations/light-leak';
|
||||
```
|
||||
|
||||
**NEVER import from `lib/transitions` barrel** — import custom transitions from `lib/transitions/presentations/` directly.
|
||||
|
||||
---
|
||||
|
||||
## Error Recovery
|
||||
|
||||
| Problem | Solution |
|
||||
|---------|----------|
|
||||
| Tool command fails with "No module named..." | Run `pip3 install --break-system-packages -r tools/requirements.txt` from toolkit root |
|
||||
| "MODAL_*_ENDPOINT_URL not configured" | Check `.env` has the endpoint URL. Run `python3 tools/verify_setup.py` |
|
||||
| SadTalker output is square/cropped | You forgot `--preprocess full`. Re-run with that flag |
|
||||
| Audio too short/long for scene | Re-run Step 5 (sync timing) and update config |
|
||||
| `npm run render` fails | Make sure you're in the project dir, not toolkit root. Run `npm install` first |
|
||||
| "Cannot find module" in Remotion | Check import paths. Custom components use `../../../lib/` relative paths |
|
||||
| Cold start timeout on Modal | First call after idle takes 30-120s. Retry once — second call uses warm GPU |
|
||||
|
||||
---
|
||||
|
||||
## Cost Estimates (Modal)
|
||||
|
||||
| Tool | Typical Cost | Notes |
|
||||
|------|-------------|-------|
|
||||
| Qwen3-TTS | ~$0.01/scene | ~20s per scene on warm GPU |
|
||||
| FLUX.2 | ~$0.01/image | ~3s warm, ~30s cold |
|
||||
| ACE-Step | ~$0.02-0.05 | Depends on duration |
|
||||
| SadTalker | ~$0.05-0.20/scene | ~3-4 min per 10s audio |
|
||||
| Qwen-Edit | ~$0.03-0.15 | ~8 min cold start (25GB model) |
|
||||
| RealESRGAN | ~$0.005/image | Very fast |
|
||||
| LTX-2.3 | ~$0.20-0.25/clip | ~2.5 min per 5s clip, A100-80GB |
|
||||
|
||||
**Total for a 60s video:** ~$1-3 depending on scenes and narrator clips.
|
||||
|
||||
Modal Starter plan: $30/month free compute. Apps scale to zero when idle.
|
||||
|
|
@ -5,6 +5,11 @@
|
|||
FAL_KEY= # FLUX images, Google Veo video, Kling video, MiniMax video, Recraft images
|
||||
# Get one at https://fal.ai/dashboard/keys
|
||||
|
||||
# --- Kling official direct API ---
|
||||
KLING_API_KEY= # Official Kling API key; enables video, image, TTS, avatar, lip sync
|
||||
KLING_API_BASE_URL= # Optional endpoint override; leave blank for default https://api-singapore.klingai.com
|
||||
# Mainland China accounts can use https://api-beijing.klingai.com
|
||||
|
||||
# --- Google (one key unlocks image gen + TTS) ---
|
||||
GOOGLE_API_KEY= # Google Imagen images, Google Cloud TTS (700+ voices, 50+ languages)
|
||||
# Get one at https://aistudio.google.com/apikey
|
||||
|
|
|
|||
17
README.md
17
README.md
|
|
@ -173,6 +173,10 @@ This repo is built for agentic operation. If you're an OpenClaw-style agent, her
|
|||
# Image + video gateway:
|
||||
FAL_KEY=your-key # FLUX images + Google Veo, Kling, MiniMax video + Recraft images
|
||||
|
||||
# Kling official direct API:
|
||||
KLING_API_KEY=your-key # Official Kling video, image, TTS, avatar, lip sync
|
||||
KLING_API_BASE_URL= # Optional; default Singapore API endpoint
|
||||
|
||||
# Free stock media:
|
||||
PEXELS_API_KEY=your-key # Free stock footage and images
|
||||
PIXABAY_API_KEY=your-key # Free stock footage and images
|
||||
|
|
@ -418,11 +422,12 @@ Each tool declares which Layer 3 skills it relies on. The agent reads Layer 1 to
|
|||
> **Full setup guide with pricing and free tiers:** [`docs/PROVIDERS.md`](docs/PROVIDERS.md)
|
||||
|
||||
<details>
|
||||
<summary><strong>Video Generation — 14 providers</strong></summary>
|
||||
<summary><strong>Video Generation — 15 providers</strong></summary>
|
||||
|
||||
| Provider | Type | Notes |
|
||||
|----------|------|-------|
|
||||
| **Kling** | Cloud API | High quality, fast |
|
||||
| **Kling (fal.ai)** | Cloud API | High quality, fast via fal.ai gateway |
|
||||
| **Kling Official** | Cloud API | Official direct API with separate `kling_official` provider |
|
||||
| **Runway Gen-4** | Cloud API | Cinematic quality, Gen-3 Alpha Turbo / Gen-4 Turbo / Gen-4 Aleph |
|
||||
| **Google Veo 3** | Cloud API | Long-form, cinematic. Via fal.ai or HeyGen. |
|
||||
| **Grok Imagine Video** | Cloud API | Strong reference-image video and xAI-native short-form generation |
|
||||
|
|
@ -440,7 +445,7 @@ Each tool declares which Layer 3 skills it relies on. The agent reads Layer 1 to
|
|||
</details>
|
||||
|
||||
<details>
|
||||
<summary><strong>Image Generation — 10 tools/providers</strong></summary>
|
||||
<summary><strong>Image Generation — 11 tools/providers</strong></summary>
|
||||
|
||||
| Provider | Type | Notes |
|
||||
|----------|------|-------|
|
||||
|
|
@ -449,6 +454,7 @@ Each tool declares which Layer 3 skills it relies on. The agent reads Layer 1 to
|
|||
| **Grok Imagine Image** | Cloud API | Strong image edits, style transfer, and multi-image compositing |
|
||||
| **GPT Image 2** | Cloud API | OpenAI's image model |
|
||||
| **Recraft** | Cloud API | Design-focused generation |
|
||||
| **Kling Official** | Cloud API | Official direct API for Kling image generation and reference workflows |
|
||||
| **Local Diffusion** | Local GPU | Stable Diffusion, free |
|
||||
| **Pexels** | Stock | Free stock images |
|
||||
| **Pixabay** | Stock | Free stock images |
|
||||
|
|
@ -458,12 +464,13 @@ Each tool declares which Layer 3 skills it relies on. The agent reads Layer 1 to
|
|||
</details>
|
||||
|
||||
<details>
|
||||
<summary><strong>Text-to-Speech — 4 providers</strong></summary>
|
||||
<summary><strong>Text-to-Speech — 5 providers</strong></summary>
|
||||
|
||||
| Provider | Type | Notes |
|
||||
|----------|------|-------|
|
||||
| **ElevenLabs** | Cloud API | Premium voice quality |
|
||||
| **Google TTS** | Cloud API | 700+ voices, 50+ languages — best for localization |
|
||||
| **Kling Official TTS** | Cloud API | Official Kling narration when a `voice_id` is known |
|
||||
| **OpenAI TTS** | Cloud API | Fast, affordable |
|
||||
| **Piper** | Local | Completely free, offline |
|
||||
|
||||
|
|
@ -516,6 +523,8 @@ Each tool declares which Layer 3 skills it relies on. The agent reads Layer 1 to
|
|||
|------|-------------|
|
||||
| **Talking Head** | SadTalker / MuseTalk avatar animation |
|
||||
| **Lip Sync** | Wav2Lip audio-driven lip synchronization |
|
||||
| **Kling Avatar** | Official Kling cloud avatar presenter generation |
|
||||
| **Kling Lip Sync** | Official Kling cloud lip-sync with explicit face selection |
|
||||
|
||||
**Composition & Rendering:**
|
||||
|
||||
|
|
|
|||
BIN
diagram.png
BIN
diagram.png
Binary file not shown.
|
Before Width: | Height: | Size: 5.8 KiB After Width: | Height: | Size: 5.2 KiB |
|
|
@ -386,6 +386,8 @@ All config is validated via Pydantic models in `lib/config_model.py`.
|
|||
| `OPENAI_API_KEY` | openai_tts, openai_image | TTS fallback, GPT Image 2 |
|
||||
| `XAI_API_KEY` | grok_image, grok_video | Grok image editing/generation, Grok video generation |
|
||||
| `FAL_KEY` | flux_image, kling_video, veo_video, minimax_video, recraft_image | fal.ai hosted models (FLUX, Veo, Kling, MiniMax, Recraft) |
|
||||
| `KLING_API_KEY` | kling_official_video, kling_official_image, kling_tts, kling_avatar, kling_lip_sync | Official Kling direct API for video, image, TTS, avatar, and lip sync |
|
||||
| `KLING_API_BASE_URL` | kling_official_video, kling_official_image, kling_tts, kling_avatar, kling_lip_sync | Optional official Kling API endpoint override |
|
||||
| `HEYGEN_API_KEY` | heygen_video | Multi-provider video generation |
|
||||
| `PEXELS_API_KEY` | pexels_image, pexels_video | Stock media |
|
||||
| `PIXABAY_API_KEY` | pixabay_image, pixabay_video | Stock media |
|
||||
|
|
@ -396,6 +398,18 @@ All config is validated via Pydantic models in `lib/config_model.py`.
|
|||
| `VIDEO_GEN_LOCAL_ENABLED` | local video tools | Enable local GPU generation |
|
||||
| `VIDEO_GEN_LOCAL_MODEL` | wan, hunyuan, ltx, cogvideo | Select local model |
|
||||
|
||||
Kling Official Phase 2 adds deeper Omni reference support inside the existing
|
||||
`kling_official_video` and `kling_official_image` providers. Elements and Account
|
||||
Usage live under `tools/_kling/` as internal helpers for element ID references and
|
||||
low-frequency account diagnostics; they are not separate pipeline stages,
|
||||
selectors, or generated-asset capabilities.
|
||||
|
||||
Kling Official Phase 3 adds provider tools only where OpenMontage already has a
|
||||
matching capability slot: `kling_tts` for `tts`, plus `kling_avatar` and
|
||||
`kling_lip_sync` for `avatar`. Official Kling audio effects and video effects are
|
||||
not registered as tools yet because current pipelines do not define stable
|
||||
`sound_effects` or `video_effects` capability routing.
|
||||
|
||||
---
|
||||
|
||||
## Visual Style System
|
||||
|
|
|
|||
|
|
@ -17,11 +17,12 @@ Everything you need to know about every provider in OpenMontage — setup instru
|
|||
| 5 | **~$0.03/image** | fal.ai | FLUX images + Kling/Veo/MiniMax video + Recraft — broad single-key image + video coverage |
|
||||
| 6 | **~$0.05/image** | OpenAI | GPT Image 2 images + OpenAI TTS |
|
||||
| 7 | **~$0.04/image** | Google Imagen | Imagen 4 images (shares the Google API key) |
|
||||
| 8 | **$12/month** | Runway | Gen-4 video — highest quality AI video |
|
||||
| 9 | **pay-as-you-go** | HeyGen | Avatar videos, multi-model video gateway |
|
||||
| 10 | **pay-as-you-go** | Suno | Full song generation with vocals and lyrics |
|
||||
| 11 | **$0 + GPU** | Local video gen | WAN 2.1, Hunyuan, CogVideo, LTX — free, offline |
|
||||
| 12 | **$0 + GPU** | Local Diffusion | Stable Diffusion images — free, offline |
|
||||
| 8 | **pay-as-you-go** | Kling Official | Official direct Kling video, image, TTS, avatar, and lip-sync API, separate from fal.ai Kling |
|
||||
| 9 | **$12/month** | Runway | Gen-4 video — highest quality AI video |
|
||||
| 10 | **pay-as-you-go** | HeyGen | Avatar videos, multi-model video gateway |
|
||||
| 11 | **pay-as-you-go** | Suno | Full song generation with vocals and lyrics |
|
||||
| 12 | **$0 + GPU** | Local video gen | WAN 2.1, Hunyuan, CogVideo, LTX — free, offline |
|
||||
| 13 | **$0 + GPU** | Local Diffusion | Stable Diffusion images — free, offline |
|
||||
|
||||
### Environment Variable Summary
|
||||
|
||||
|
|
@ -45,6 +46,10 @@ DOUBAO_SPEECH_VOICE_TYPE= # Default Doubao speaker/voice type
|
|||
# MULTI-MODEL GATEWAY (one key, 6+ tools)
|
||||
FAL_KEY= # FLUX, Recraft, Kling, Veo, MiniMax video
|
||||
|
||||
# KLING OFFICIAL DIRECT API
|
||||
KLING_API_KEY= # Official Kling video, image, TTS, avatar, lip sync
|
||||
KLING_API_BASE_URL= # Optional; default https://api-singapore.klingai.com
|
||||
|
||||
# VIDEO
|
||||
HEYGEN_API_KEY= # HeyGen avatar video gateway
|
||||
RUNWAY_API_KEY= # Runway Gen-4 video (direct)
|
||||
|
|
@ -133,6 +138,48 @@ No subscription — pure pay-as-you-go, no minimum spend.
|
|||
|
||||
---
|
||||
|
||||
### Kling Official — Direct API
|
||||
|
||||
> **Official Kling path.** This is separate from `kling_video` via fal.ai: it uses Kling's official `Authorization: Bearer <KLING_API_KEY>` API, provider name `kling_official`, and direct Classic/Turbo/Omni task protocols.
|
||||
|
||||
**Tools unlocked:** `kling_official_video`, `kling_official_image`, `kling_tts`, `kling_avatar`, `kling_lip_sync`
|
||||
**Env vars:** `KLING_API_KEY`, optional `KLING_API_BASE_URL`
|
||||
|
||||
#### Setup
|
||||
|
||||
1. Create or open a Kling AI Open Platform account.
|
||||
2. Generate an official API key in the Kling API console.
|
||||
3. Add to `.env`:
|
||||
```bash
|
||||
KLING_API_KEY=your-key-here
|
||||
# Optional, defaults to Singapore:
|
||||
KLING_API_BASE_URL=https://api-singapore.klingai.com
|
||||
```
|
||||
|
||||
#### What It Is Best For
|
||||
|
||||
- Direct official Kling API provenance rather than fal.ai gateway routing
|
||||
- Text-to-video, image-to-video, and deep Video Omni reference workflows via `kling_official_video`
|
||||
- Text-to-image, image edit/reference, and Image Omni multi-reference or series workflows via `kling_official_image`
|
||||
- Text-to-speech via `kling_tts` when you already know the official Kling `voice_id`
|
||||
- Cloud avatar presenter clips via `kling_avatar`, without replacing local `talking_head`
|
||||
- Cloud lip-sync via `kling_lip_sync`, with explicit face selection for multi-person videos
|
||||
- Accounts that need to use official Kling model permissions, resource packs, or regional endpoints
|
||||
|
||||
#### Notes
|
||||
|
||||
- `provider="kling_official"` is intentionally different from fal.ai's `provider="kling"`.
|
||||
- Official Kling is a paid remote API. OpenMontage uses conservative cost estimates and includes high-cost factors such as Omni references, series output, 4k mode, and native sound.
|
||||
- Local image paths are sent as raw base64 for supported Classic/image-generation fields. Turbo image-to-video requires a URL and will not silently upload through fal.ai.
|
||||
- Video Omni and Image Omni can pass official `element_id` references through `element_list`; Elements remain an internal Kling Official helper, not a standalone OpenMontage capability.
|
||||
- Account Usage is available as a low-frequency diagnostic helper under `tools/_kling/account.py`; it is not a selector or pipeline tool.
|
||||
- `callback_url` is passed through and recorded when supplied, but OpenMontage still polls tasks by default.
|
||||
- `kling_tts` requires an explicit `voice_id`; OpenMontage does not guess a default official voice.
|
||||
- `kling_avatar` and `kling_lip_sync` register under the existing `avatar` capability and coexist with local SadTalker/Wav2Lip tools. Current avatar pipelines must opt into them explicitly; registry discovery alone does not replace local tools.
|
||||
- Official Kling audio effects and video effects are documented but intentionally not registered as OpenMontage tools yet, because current pipelines do not have a stable sound-effects or video-effects capability slot for them.
|
||||
|
||||
---
|
||||
|
||||
### ElevenLabs — Voice, Music, Sound Effects
|
||||
|
||||
> **Premium voice quality.** Best TTS for narration-heavy videos. Also generates music and sound effects.
|
||||
|
|
@ -729,6 +776,7 @@ These tools require only FFmpeg or Python packages — no GPU, no API key.
|
|||
| **Google** | `GOOGLE_API_KEY` | `google_tts`, `google_imagen` | Free tier + paid |
|
||||
| **ElevenLabs** | `ELEVENLABS_API_KEY` | `elevenlabs_tts`, `music_gen` | Free tier + paid |
|
||||
| **fal.ai** | `FAL_KEY` | `flux_image`, `recraft_image`, `kling_video`, `veo_video`, `minimax_video` | Pay-as-you-go |
|
||||
| **Kling Official** | `KLING_API_KEY` | `kling_official_video`, `kling_official_image`, `kling_tts`, `kling_avatar`, `kling_lip_sync` | Pay-as-you-go |
|
||||
| **OpenAI** | `OPENAI_API_KEY` | `openai_tts`, `openai_image` | Paid only |
|
||||
| **xAI** | `XAI_API_KEY` | `grok_image`, `grok_video` | Paid only |
|
||||
| **Runway** | `RUNWAY_API_KEY` | `runway_video` | Free trial + paid |
|
||||
|
|
@ -747,14 +795,14 @@ How many providers cover each capability:
|
|||
|
||||
| Capability | Cloud Providers | Local Providers | Free Options |
|
||||
|-----------|----------------|-----------------|--------------|
|
||||
| **Image Generation** | FLUX, Grok, Google Imagen, GPT Image 2, Recraft | Local Diffusion | Pexels, Pixabay (stock) |
|
||||
| **Video Generation** | Grok, Kling, Runway, Veo, Higgsfield, MiniMax, HeyGen | WAN, Hunyuan, CogVideo, LTX | Pexels, Pixabay (stock) |
|
||||
| **Text-to-Speech** | ElevenLabs, Google TTS, OpenAI | Piper | Piper, Google free tier, ElevenLabs free tier |
|
||||
| **Image Generation** | FLUX, Kling Official, Grok, Google Imagen, GPT Image 2, Recraft | Local Diffusion | Pexels, Pixabay (stock) |
|
||||
| **Video Generation** | Grok, Kling Official, Kling via fal.ai, Runway, Veo, Higgsfield, MiniMax, HeyGen | WAN, Hunyuan, CogVideo, LTX | Pexels, Pixabay (stock) |
|
||||
| **Text-to-Speech** | ElevenLabs, Google TTS, Kling Official, OpenAI | Piper | Piper, Google free tier, ElevenLabs free tier |
|
||||
| **Music Generation** | ElevenLabs, Suno | — | ElevenLabs free tier |
|
||||
| **Post-Production** | — | FFmpeg (compose, stitch, trim, mix, enhance, grade) | All free |
|
||||
| **Analysis** | — | WhisperX, Scene Detect, Frame Sampler, CLIP/BLIP-2 | All free |
|
||||
| **Enhancement** | — | Upscale, BG Remove, Face Enhance, Face Restore | All free |
|
||||
| **Avatar** | — | SadTalker, Wav2Lip | All free |
|
||||
| **Avatar** | Kling Official | SadTalker, Wav2Lip | Local tools are free |
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,784 @@
|
|||
# 可灵官方 API 集成阶段 1：核心视频与图像 Provider
|
||||
|
||||
状态：实施指导文档。
|
||||
|
||||
来源：从 `docs/kling-official-integration-plan.md` 拆分而来。本阶段合并原计划中的 P0「准备与保护」和 P1「核心视频与图像」。
|
||||
|
||||
执行顺序：必须先完成本文件，再进入 `docs/kling-official-phase-2-omni-operations.md`。
|
||||
|
||||
## 1. 阶段目标
|
||||
|
||||
本阶段的目标是让可灵官方 API 以正式 provider 形式接入 OpenMontage 的工具系统，先交付两个可用能力：
|
||||
|
||||
| 能力 | 文件 | tool name | provider | capability |
|
||||
|------|------|-----------|----------|------------|
|
||||
| 官方视频生成 | `tools/video/kling_official_video.py` | `kling_official_video` | `kling_official` | `video_generation` |
|
||||
| 官方图像生成 | `tools/graphics/kling_official_image.py` | `kling_official_image` | `kling_official` | `image_generation` |
|
||||
|
||||
本阶段完成后，OpenMontage 应能通过 registry 自动发现这两个工具，并通过 `video_selector` / `image_selector` 在用户指定 `preferred_provider="kling_official"` 时选中官方直连路径。
|
||||
|
||||
本阶段不要求完整接入音频、TTS、数字人、口型、视频特效、元素管理和账户用量。这些能力只在后续阶段评估；只有能自然落入现有 OpenMontage capability 或作为 provider 内部 helper 的部分才接入。
|
||||
|
||||
本阶段是 provider 接入，不是新增 OpenMontage 功能面。现有 pipeline、stage director、selector、artifact schema 和 checkpoint 规则保持不变。
|
||||
|
||||
## 2. 不可变规则
|
||||
|
||||
实施时必须遵守以下规则：
|
||||
|
||||
- 本阶段只覆盖 Kling official API。Volcengine Jimeng/即梦不属于本计划；Jimeng 的鉴权、provider 命名、环境变量、签名 client 和模型枚举都必须单独设计，不能混进这些可灵官方工具。
|
||||
- 不改 pipeline 体系。官方可灵第一阶段只新增 BaseTool provider，不新增 pipeline，不重写已有 pipeline manifest。
|
||||
- 不改 pipeline stage 顺序，不新增 canonical artifact，不新增 orchestrator 状态。
|
||||
- 不覆盖 `tools/video/kling_video.py`。该文件是现有 fal.ai Kling provider，行为必须保持不变。
|
||||
- 官方 provider 必须统一使用 `provider="kling_official"`，不能复用 `provider="kling"`，否则 selector 无法稳定区分 fal.ai 网关和官方直连。
|
||||
- 第一阶段只实现 API Key 鉴权：`Authorization: Bearer <KLING_API_KEY>`。AK/SK JWT 不进入本阶段。
|
||||
- 所有官方可灵工具都必须声明 `dependencies = ["env:KLING_API_KEY"]`，使 `provider_menu_summary()` 能自动生成 setup offer。
|
||||
- `KLING_API_BASE_URL` 是可选覆盖项，默认值应为 `https://api-singapore.klingai.com`。
|
||||
- 所有付费官方可灵 provider 必须重写 `estimate_cost()`，不能继承 `BaseTool.estimate_cost()` 的 `0.0` 默认值。
|
||||
- 所有官方可灵 provider 的 `agent_skills` 必须包含新建的 `kling-official` skill。视频工具还必须保留通用视频提示 skill，例如 `ai-video-gen`。
|
||||
- 实施前必须重新抽取当前官方文档 schema chunk，并固化为测试 fixture。不能直接把原总计划中的 chunk 文件名当作当前事实。
|
||||
- 官方视频工具的 input schema 不得暴露顶层 `image_url` 字段，避免 `video_selector` 误触发 fal.ai 的图片上传逻辑。
|
||||
- 所有远端生成结果必须下载到 OpenMontage 的输出路径或项目 artifacts 中，不能只返回官方临时 URL。
|
||||
- CI 默认不能打真实可灵付费 API。真实调用必须通过显式环境变量开启。
|
||||
|
||||
## 3. Pipeline 调用链
|
||||
|
||||
本阶段必须接入现有调用链，而不是创造新的编排路径：
|
||||
|
||||
```text
|
||||
pipeline stage director
|
||||
-> selector tool（video_selector / image_selector）
|
||||
-> registry.get_by_capability(...)
|
||||
-> kling_official_* provider
|
||||
-> tools/_kling client/parser/media helper
|
||||
-> Kling official API
|
||||
-> ToolResult + artifacts
|
||||
-> stage canonical artifact
|
||||
-> checkpoint
|
||||
```
|
||||
|
||||
实施含义：
|
||||
|
||||
- pipeline 只看到 `video_generation` / `image_generation` capability，不感知官方可灵协议细节。
|
||||
- stage director 仍按现有方式调用 selector 或具体 tool，不写可灵专用编排。
|
||||
- selector 只做 provider selection 和通用参数转发，不承担可灵 payload 构造。
|
||||
- 官方 API 协议差异只存在于 `kling_official_video`、`kling_official_image` 和 `tools/_kling/` helper。
|
||||
- 所有生成文件仍写入 stage 传入的 `output_path` 或项目目录，不写 repo root。
|
||||
- 如果用户选择官方可灵，proposal/preflight 只说明 provider/model/cost，不改变 pipeline。
|
||||
|
||||
## 4. 准备与保护
|
||||
|
||||
开始写代码前先完成这些检查：
|
||||
|
||||
1. 创建实现分支，建议使用 `codex/` 前缀，例如 `codex/kling-official-phase-1`。
|
||||
2. 查看工作区状态，确认不会覆盖用户已有改动。
|
||||
3. 确认 Python 依赖可用，尤其是 HTTP 客户端依赖。优先复用仓库现有依赖；若新增依赖，必须同步依赖文件和文档。
|
||||
4. 阅读这些本地文件以确认当前实现契约：
|
||||
- `tools/base_tool.py`
|
||||
- `tools/tool_registry.py`
|
||||
- `tools/video/video_selector.py`
|
||||
- `tools/graphics/image_selector.py`
|
||||
- `tools/video/kling_video.py`
|
||||
- `tools/video/_shared.py`
|
||||
5. 不开始实现 provider，直到 schema fixture 刷新完成。
|
||||
|
||||
## 5. 官方 Schema Fixture
|
||||
|
||||
官方文档是 SPA，正文和 OpenAPI schema 会被拆进懒加载 chunk。同一个 build id 下资源文件名也可能变化，因此必须在实施时重新定位当前文档资源。
|
||||
|
||||
建议新增 fixture 路径：
|
||||
|
||||
```text
|
||||
tests/fixtures/kling_official/schema_snapshot.json
|
||||
```
|
||||
|
||||
fixture 至少包含：
|
||||
|
||||
```json
|
||||
{
|
||||
"build_id": "...",
|
||||
"source_urls": ["..."],
|
||||
"chunk_names": ["..."],
|
||||
"extracted_at": "YYYY-MM-DDTHH:MM:SSZ",
|
||||
"endpoints": {},
|
||||
"models": {},
|
||||
"task_statuses": {},
|
||||
"result_paths": {},
|
||||
"core_field_enums": {}
|
||||
}
|
||||
```
|
||||
|
||||
必须固化的核心信息：
|
||||
|
||||
- API base URL 默认值和可覆盖环境变量。
|
||||
- Classic 任务状态：`submitted`、`processing`、`succeed`、`failed`。
|
||||
- Turbo 任务状态：`submitted`、`processing`、`succeeded`、`failed`。
|
||||
- Classic 创建 ID 路径：`data.task_id`。
|
||||
- Turbo 创建 ID 路径：`data.id`。
|
||||
- Classic 结果路径：`data.task_result.videos[]`、`data.task_result.images[]`。
|
||||
- Turbo 结果路径：`data[0].outputs[]`。
|
||||
- 第一阶段要支持的视频模型枚举。
|
||||
- 第一阶段要支持的图像模型枚举。
|
||||
- `aspect_ratio`、`duration`、`resolution`、`mode`、`sound` 等核心字段枚举。
|
||||
|
||||
测试要求：
|
||||
|
||||
- 如果当前官方 HTML 的 build id、入口 chunk 或核心 schema 与 fixture 不一致，测试应提示先刷新 fixture。
|
||||
- 测试不应依赖原计划中的旧 chunk 文件名。
|
||||
- fixture 是实现依据之一，不是替代错误处理和 runtime 验证的借口。
|
||||
|
||||
## 6. 共享可灵 Client
|
||||
|
||||
新增目录：
|
||||
|
||||
```text
|
||||
tools/_kling/
|
||||
├── __init__.py
|
||||
├── client.py
|
||||
├── errors.py
|
||||
├── media.py
|
||||
└── schemas.py
|
||||
```
|
||||
|
||||
### 6.1 `client.py`
|
||||
|
||||
最低接口：
|
||||
|
||||
```python
|
||||
class KlingClient:
|
||||
def __init__(self, api_key=None, base_url=None, session=None): ...
|
||||
def post(self, path: str, payload: dict[str, Any]) -> dict[str, Any]: ...
|
||||
def get(self, path: str, params: dict[str, Any] | None = None) -> dict[str, Any]: ...
|
||||
def download(self, url: str, output_path: Path, timeout: int = 180) -> Path: ...
|
||||
```
|
||||
|
||||
任务接口：
|
||||
|
||||
```python
|
||||
def create_classic_task(path: str, payload: dict[str, Any]) -> str: ...
|
||||
def poll_classic(path: str, task_id: str, result_key: str, timeout_seconds: int, poll_interval: float) -> list[dict]: ...
|
||||
def create_turbo(path: str, payload: dict[str, Any]) -> str: ...
|
||||
def poll_turbo(task_id: str, timeout_seconds: int, poll_interval: float) -> list[dict]: ...
|
||||
```
|
||||
|
||||
实现规则：
|
||||
|
||||
- 从 `KLING_API_KEY` 读取默认 API Key。
|
||||
- 从 `KLING_API_BASE_URL` 读取可选 base URL；未设置时使用 `https://api-singapore.klingai.com`。
|
||||
- 所有请求都发送 `Authorization: Bearer <key>`。
|
||||
- 所有 JSON 请求都发送明确的 JSON headers。
|
||||
- HTTP 非 2xx 时尝试解析 JSON 中的 `code`、`message`、`request_id`；如果不是 JSON，保留响应文本片段。
|
||||
- 业务 `code != 0` 时抛出 `KlingAPIError`。
|
||||
- 下载方法负责创建父目录，返回最终 `Path`。
|
||||
|
||||
### 6.2 `errors.py`
|
||||
|
||||
新增：
|
||||
|
||||
```python
|
||||
class KlingAPIError(Exception):
|
||||
code: str | int | None
|
||||
message: str
|
||||
request_id: str | None
|
||||
http_status: int | None
|
||||
```
|
||||
|
||||
新增：
|
||||
|
||||
```python
|
||||
def is_retryable_kling_error(error: KlingAPIError) -> bool: ...
|
||||
```
|
||||
|
||||
错误处理规则：
|
||||
|
||||
| HTTP | 业务码 | 含义 | 行为 |
|
||||
|------|--------|------|------|
|
||||
| 401 | 1000-1004 | 鉴权失败或 token 无效 | 不重试，提示 `KLING_API_KEY` / Authorization |
|
||||
| 429 | 1101/1102 | 欠费、资源包耗尽或过期 | 不重试，提示账户或资源包 |
|
||||
| 403 | 1103 | 接口或模型无权限 | 不重试，提示模型权限 |
|
||||
| 400 | 1200/1201 | 参数非法 | 不重试，暴露官方 message |
|
||||
| 404 | 1202/1203 | method/resource/model 无效 | 不重试，标记实现或模型配置问题 |
|
||||
| 429 | 1302 | 请求过快 | 可有限退避重试 |
|
||||
| 429 | 1303 | 并发或 QPS 超资源包限制 | 可有限退避重试，错误文案必须说明并发槽 |
|
||||
| 400 | 1301 | 内容安全策略 | 不重试，提示修改输入 |
|
||||
| 500/503/504 | 5000-5002 | 服务端错误、维护、积压超时 | 可有限退避重试 |
|
||||
|
||||
退避规则：
|
||||
|
||||
- 只对 `1302`、`1303`、`5000`、`5001`、`5002` 做有限重试。
|
||||
- 不对鉴权、余额、权限、参数、安全策略错误重试。
|
||||
- 重试耗尽后保留最后一次官方错误信息。
|
||||
|
||||
### 6.3 `schemas.py`
|
||||
|
||||
放置轻量常量和 dataclass：
|
||||
|
||||
- Classic/Turbo 协议枚举。
|
||||
- Classic/Turbo 状态常量。
|
||||
- 第一阶段模型枚举。
|
||||
- `ClassicTaskResult`、`TurboTaskResult` 等轻量解析结果。
|
||||
|
||||
不要写“猜字段”的通用任务解析器。Classic 和 Turbo 的字段名、状态值、结果路径不同，必须分开解析。
|
||||
|
||||
### 6.4 `media.py`
|
||||
|
||||
实现：
|
||||
|
||||
- `strip_data_uri_prefix(value)`：去掉 `data:image/...;base64,` 等前缀。
|
||||
- `image_file_to_raw_base64(path)`：本地图片转 raw base64。
|
||||
- `normalize_image_input(url=None, path=None)`：URL 直接返回 URL，本地路径转 raw base64。
|
||||
- 下载图片/音频/视频到 output path 的共用 helper。
|
||||
|
||||
第一阶段可以先把这些 helper 放在 `tools/_kling/media.py`。不要为了抽象过早修改现有 fal.ai 工具。
|
||||
|
||||
## 7. 官方视频 Provider
|
||||
|
||||
新增：
|
||||
|
||||
```text
|
||||
tools/video/kling_official_video.py
|
||||
```
|
||||
|
||||
基础契约：
|
||||
|
||||
```python
|
||||
class KlingOfficialVideo(BaseTool):
|
||||
name = "kling_official_video"
|
||||
version = "0.1.0"
|
||||
tier = ToolTier.GENERATE
|
||||
capability = "video_generation"
|
||||
provider = "kling_official"
|
||||
stability = ToolStability.EXPERIMENTAL
|
||||
execution_mode = ExecutionMode.SYNC
|
||||
determinism = Determinism.STOCHASTIC
|
||||
runtime = ToolRuntime.API
|
||||
dependencies = ["env:KLING_API_KEY"]
|
||||
agent_skills = ["ai-video-gen", "kling-official"]
|
||||
```
|
||||
|
||||
### 7.1 支持范围
|
||||
|
||||
| OpenMontage operation | `api_family` | 官方协议 | 端点 |
|
||||
|-----------------------|--------------|----------|------|
|
||||
| `text_to_video` | `classic` | Classic | `/v1/videos/text2video` |
|
||||
| `image_to_video` | `classic` | Classic | `/v1/videos/image2video` |
|
||||
| `text_to_video` | `turbo` | Turbo | `/text-to-video/kling-3.0-turbo` |
|
||||
| `image_to_video` | `turbo` | Turbo | `/image-to-video/kling-3.0-turbo` |
|
||||
| `text_to_video` | `omni` | Classic Omni | `/v1/videos/omni-video` |
|
||||
| `image_to_video` | `omni` | Classic Omni | `/v1/videos/omni-video` |
|
||||
| `reference_to_video` | `omni` | Classic Omni | `/v1/videos/omni-video` |
|
||||
|
||||
`video_selector` 的标准 operation 仍是 `text_to_video`、`image_to_video`、`reference_to_video`、`rank`。Turbo 和 Omni 不应变成 selector 层的新 operation，而应通过 `api_family` 选择。
|
||||
|
||||
直接调用 provider 时可以兼容 `operation="omni_video"` 作为别名，但 selector 路径不要依赖这个别名。
|
||||
|
||||
### 7.2 Input Schema 规则
|
||||
|
||||
建议字段：
|
||||
|
||||
```python
|
||||
{
|
||||
"required": ["prompt"],
|
||||
"properties": {
|
||||
"prompt": {"type": "string"},
|
||||
"operation": {"enum": ["text_to_video", "image_to_video", "reference_to_video"], "default": "text_to_video"},
|
||||
"api_family": {"enum": ["classic", "turbo", "omni"], "default": "classic"},
|
||||
"model_name": {
|
||||
"enum": [
|
||||
"kling-v1",
|
||||
"kling-v1-5",
|
||||
"kling-v1-6",
|
||||
"kling-v2-master",
|
||||
"kling-v2-1",
|
||||
"kling-v2-1-master",
|
||||
"kling-v2-5-turbo",
|
||||
"kling-v2-6",
|
||||
"kling-v3",
|
||||
"kling-video-o1",
|
||||
"kling-v3-omni"
|
||||
],
|
||||
"default": "kling-v3"
|
||||
},
|
||||
"duration": {"enum": ["3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15"], "default": "5"},
|
||||
"aspect_ratio": {"enum": ["16:9", "9:16", "1:1"], "default": "16:9"},
|
||||
"resolution": {"enum": ["720p", "1080p"], "default": "720p"},
|
||||
"mode": {"enum": ["std", "pro", "4k"], "default": "std"},
|
||||
"sound": {"enum": ["on", "off"], "default": "off"},
|
||||
"negative_prompt": {"type": "string"},
|
||||
"reference_image_url": {"type": "string"},
|
||||
"reference_image_path": {"type": "string"},
|
||||
"reference_tail_image_url": {"type": "string"},
|
||||
"reference_tail_image_path": {"type": "string"},
|
||||
"image_list": {"type": "array"},
|
||||
"video_list": {"type": "array"},
|
||||
"element_list": {"type": "array"},
|
||||
"camera_control": {"type": "object"},
|
||||
"watermark": {"type": "boolean", "default": False},
|
||||
"callback_url": {"type": "string"},
|
||||
"external_task_id": {"type": "string"},
|
||||
"output_path": {"type": "string"}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
强制规则：
|
||||
|
||||
- 不暴露顶层 `image_url`。
|
||||
- 使用 `reference_image_url` / `reference_image_path` 表达首帧。
|
||||
- 使用 `reference_tail_image_url` / `reference_tail_image_path` 表达尾帧。
|
||||
- `reference_image_path` 在工具内部转 raw base64，不能走 fal.ai 上传。
|
||||
- `model_variant` 可以兼容老参数，但内部主字段应是 `model_name`。
|
||||
- `model_name` enum 必须来自当前 schema fixture。上面的枚举是阶段 1 初始范围；如果官方 schema 更新，先更新 fixture 和 contract 测试，再更新 input schema。
|
||||
- `supports` 至少声明：
|
||||
|
||||
```python
|
||||
{
|
||||
"text_to_video": True,
|
||||
"image_to_video": True,
|
||||
"reference_to_video": True,
|
||||
"reference_image": True,
|
||||
"negative_prompt": True,
|
||||
"aspect_ratio": True
|
||||
}
|
||||
```
|
||||
|
||||
### 7.3 Payload Builder 规则
|
||||
|
||||
Classic 文生视频：
|
||||
|
||||
- 端点：`POST /v1/videos/text2video`
|
||||
- 发送 `model_name`、`prompt`、`negative_prompt`、`sound`、`cfg_scale`、`mode`、`camera_control`、`aspect_ratio`、`duration`、`watermark_info`、`callback_url`、`external_task_id` 等官方支持字段。
|
||||
- 默认 `model_name` 建议为 `kling-v3`。
|
||||
|
||||
Classic 图生视频：
|
||||
|
||||
- 端点：`POST /v1/videos/image2video`
|
||||
- 必须有 `reference_image_url` 或 `reference_image_path`。
|
||||
- 将输入转换成官方字段 `image`。
|
||||
- 支持尾帧时使用官方字段 `image_tail`。
|
||||
- 不要盲目发送 `aspect_ratio`，除非当前 schema 明确支持。
|
||||
|
||||
Turbo 文生视频：
|
||||
|
||||
- 端点：`POST /text-to-video/kling-3.0-turbo`
|
||||
- payload 结构为 `prompt`、`settings`、`options`。
|
||||
- `duration` 必须从字符串转成 int。
|
||||
- `settings.resolution` 支持 `720p` / `1080p`。
|
||||
- `settings.aspect_ratio` 支持 `16:9` / `9:16` / `1:1`。
|
||||
|
||||
Turbo 图生视频：
|
||||
|
||||
- 端点：`POST /image-to-video/kling-3.0-turbo`
|
||||
- payload 使用 `contents[]`。
|
||||
- prompt 用 `{ "type": "prompt", "text": "..." }`。
|
||||
- 首帧 URL 用 `{ "type": "first_frame", "url": "..." }`。
|
||||
- 如果本地图片转成 raw base64 后当前官方 schema 不支持，必须给出清晰错误或先上传到可访问 URL；不要静默走 fal.ai。
|
||||
- 不要盲目发送 `aspect_ratio`，除非当前 schema 明确支持。
|
||||
|
||||
Video Omni：
|
||||
|
||||
- 端点：`POST /v1/videos/omni-video`
|
||||
- 第一阶段只要求基础 `prompt`、`image_list`、`video_list`、`element_list`、`sound`、`mode`、`aspect_ratio`、`duration` 可用。
|
||||
- 深度多参考、多镜头 helper 放到第二阶段。
|
||||
|
||||
### 7.4 输出规则
|
||||
|
||||
成功后：
|
||||
|
||||
- 下载第一个无水印 `url` 到 `output_path`。
|
||||
- 返回 `provider="kling_official"`。
|
||||
- 返回 `model`，Classic/Omni 用 `model_name`，Turbo 可用 `kling-3.0-turbo`。
|
||||
- 返回 `task_id`、`operation`、`api_family`、`output_path`。
|
||||
- 将远端 URL、下载路径、任务 ID 放入 `artifacts` 或 `data`，便于复现。
|
||||
- 对视频调用 `tools/video/_shared.py::probe_output(output_path)`。
|
||||
|
||||
失败时：
|
||||
|
||||
- 参数错误返回可理解的 ToolResult error，不要让 KeyError、IndexError 泄漏。
|
||||
- 官方错误要保留 `code`、`message`、`request_id`。
|
||||
- `1303` 并发错误文案必须包含“并发/资源包限制”。
|
||||
|
||||
### 7.5 成本估算
|
||||
|
||||
必须实现：
|
||||
|
||||
```python
|
||||
def estimate_cost(self, params: dict[str, Any]) -> float: ...
|
||||
```
|
||||
|
||||
要求：
|
||||
|
||||
- 默认 paid 输入不能返回静默 `0.0`。
|
||||
- 如果官方价格无法稳定映射美元，返回保守估算，并在 dry-run 或结果 metadata 中写入 `cost_estimate_confidence="low"`。
|
||||
- 成功的 paid ToolResult 必须写入 `cost_usd`。`cost_usd` 应来自同一个 `estimate_cost()` 逻辑；如果后续 Account Usage 能提供实际用量，可在阶段 2 以后用实际用量校正。
|
||||
- 默认不启用 `4k`、`sound="on"`、批量、多结果等高成本能力。
|
||||
- proposal/preflight 展示成本时必须说明官方可灵是 paid API。
|
||||
|
||||
### 7.6 Registry Metadata
|
||||
|
||||
视频 provider 必须补齐 registry/provider menu 可见的元数据：
|
||||
|
||||
- `best_for`：说明官方可灵直连适合哪些视频生成场景。
|
||||
- `not_good_for`：说明不适合的场景，例如本地离线、免费生成、非可灵模型能力。
|
||||
- `install_instructions`：说明配置 `KLING_API_KEY`，不要硬编码过期 URL。
|
||||
- `fallback_tools`：列出可替代的视频 provider，例如现有 fal.ai Kling 或其它视频生成工具；只作为候选，不允许静默切换。
|
||||
- `supports`：至少包含文生视频、图生视频、参考输入、负向提示、宽高比能力。
|
||||
- `resource_profile` / `retry_policy`：如果 BaseTool 契约已有对应字段，按 API 远端生成和长轮询任务填写。
|
||||
- `idempotency_key_fields`：至少考虑 `prompt`、`operation`、`api_family`、`model_name`、`reference_image_url/path`、`duration`、`aspect_ratio`。
|
||||
- `side_effects`：标记为 paid remote generation，避免 proposal/preflight 把调用当作免费本地操作。
|
||||
|
||||
## 8. 官方图像 Provider
|
||||
|
||||
新增：
|
||||
|
||||
```text
|
||||
tools/graphics/kling_official_image.py
|
||||
```
|
||||
|
||||
基础契约：
|
||||
|
||||
```python
|
||||
class KlingOfficialImage(BaseTool):
|
||||
name = "kling_official_image"
|
||||
version = "0.1.0"
|
||||
tier = ToolTier.GENERATE
|
||||
capability = "image_generation"
|
||||
provider = "kling_official"
|
||||
runtime = ToolRuntime.API
|
||||
dependencies = ["env:KLING_API_KEY"]
|
||||
agent_skills = ["kling-official"]
|
||||
```
|
||||
|
||||
### 8.1 支持范围
|
||||
|
||||
| image_selector 语义 | `api_family` | 官方端点 |
|
||||
|--------------------|--------------|----------|
|
||||
| `generation_mode=generate` | `generation` | `/v1/images/generations` |
|
||||
| `generation_mode=edit` 或有图片输入 | `generation` | `/v1/images/generations`，填 `image` 和 `image_reference` |
|
||||
| `generation_mode=generate/edit` | `omni` | `/v1/images/omni-image` |
|
||||
|
||||
`image_selector` 的标准 operation 仍是 `generate` 和 `rank`。Omni 不应变成 selector 层的新 operation，应通过 `api_family=omni` 表达。
|
||||
|
||||
### 8.2 Input Schema 规则
|
||||
|
||||
建议字段：
|
||||
|
||||
```python
|
||||
{
|
||||
"required": ["prompt"],
|
||||
"properties": {
|
||||
"prompt": {"type": "string"},
|
||||
"negative_prompt": {"type": "string"},
|
||||
"operation": {"enum": ["generate"], "default": "generate"},
|
||||
"generation_mode": {"enum": ["generate", "edit"], "default": "generate"},
|
||||
"api_family": {"enum": ["generation", "omni"], "default": "generation"},
|
||||
"model_name": {
|
||||
"enum": [
|
||||
"kling-v1",
|
||||
"kling-v1-5",
|
||||
"kling-v2",
|
||||
"kling-v2-new",
|
||||
"kling-v2-1",
|
||||
"kling-v3",
|
||||
"kling-image-o1",
|
||||
"kling-v3-omni"
|
||||
],
|
||||
"default": "kling-v3"
|
||||
},
|
||||
"image_url": {"type": "string"},
|
||||
"image_path": {"type": "string"},
|
||||
"image_urls": {"type": "array", "items": {"type": "string"}},
|
||||
"image_paths": {"type": "array", "items": {"type": "string"}},
|
||||
"image_reference": {"enum": ["subject", "face"]},
|
||||
"image_fidelity": {"type": "number", "default": 0.5},
|
||||
"human_fidelity": {"type": "number", "default": 0.45},
|
||||
"resolution": {"enum": ["1k", "2k", "4k"], "default": "1k"},
|
||||
"aspect_ratio": {"enum": ["16:9", "9:16", "1:1", "4:3", "3:4", "3:2", "2:3", "21:9", "auto"], "default": "16:9"},
|
||||
"n": {"type": "integer", "default": 1},
|
||||
"result_type": {"enum": ["single", "series"], "default": "single"},
|
||||
"series_amount": {"type": "string"},
|
||||
"element_list": {"type": "array"},
|
||||
"watermark": {"type": "boolean", "default": False},
|
||||
"callback_url": {"type": "string"},
|
||||
"external_task_id": {"type": "string"},
|
||||
"output_path": {"type": "string"}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
图像工具可以暴露 `image_url` / `image_path`，因为 `image_selector` 没有 fal.ai 自动上传逻辑。
|
||||
|
||||
`model_name` enum 必须来自当前 schema fixture。上面的枚举覆盖普通图像生成和 Image Omni 的第一阶段范围；如果官方 schema 更新，先更新 fixture 和 contract 测试，再更新 input schema。
|
||||
|
||||
`supports` 至少声明：
|
||||
|
||||
```python
|
||||
{
|
||||
"text_to_image": True,
|
||||
"image_edit": True,
|
||||
"negative_prompt": True,
|
||||
"aspect_ratio": True
|
||||
}
|
||||
```
|
||||
|
||||
### 8.3 Payload Builder 规则
|
||||
|
||||
图像生成：
|
||||
|
||||
- 端点：`POST /v1/images/generations`
|
||||
- 必填 `prompt`。
|
||||
- 支持 `negative_prompt`、`image`、`image_reference`、`image_fidelity`、`human_fidelity`、`element_list`、`resolution`、`n`、`aspect_ratio`。
|
||||
- `prompt` 长度要遵守官方限制；超限应在调用前报参数错误。
|
||||
|
||||
图像编辑：
|
||||
|
||||
- 当 `generation_mode=edit` 或存在 `image_url` / `image_path` 时走同一 generation 端点。
|
||||
- `image_url` 直接填官方 `image`。
|
||||
- `image_path` 转 raw base64 后填官方 `image`。
|
||||
- `image_reference` 只允许官方枚举，例如 `subject`、`face`。
|
||||
|
||||
Image Omni：
|
||||
|
||||
- 端点：`POST /v1/images/omni-image`
|
||||
- 支持 `image_list`、`element_list`、`resolution`、`result_type`、`n`、`series_amount`、`aspect_ratio`。
|
||||
- 第一阶段只要求基础可用；复杂多图引用 helper 放到第二阶段。
|
||||
|
||||
### 8.4 输出规则
|
||||
|
||||
成功后：
|
||||
|
||||
- 下载 `data.task_result.images[]` 中的图片。
|
||||
- `data.output_path` 指向第一张图片。
|
||||
- 如果 `n > 1` 或 `result_type="series"`，`artifacts` 必须返回全部图片路径。
|
||||
- 根据响应 Content-Type 或 URL 推断扩展名；无法判断时默认 `.png`。
|
||||
- 返回 `provider`、`model`、`task_id`、`api_family`、`output_path`。
|
||||
|
||||
失败时：
|
||||
|
||||
- 保留官方 `code`、`message`、`request_id`。
|
||||
- 参数错误应清楚说明是 prompt、参考图、分辨率、数量还是权限问题。
|
||||
|
||||
### 8.5 成本估算
|
||||
|
||||
同视频 provider：
|
||||
|
||||
- 必须重写 `estimate_cost()`。
|
||||
- 默认 paid 输入不能静默返回 `0.0`。
|
||||
- `n > 1`、`2k/4k`、`series` 应提高估算。
|
||||
- 估算不确定时记录 `cost_estimate_confidence="low"`。
|
||||
- 成功的 paid ToolResult 必须写入 `cost_usd`。`cost_usd` 应来自同一个 `estimate_cost()` 逻辑；如果后续 Account Usage 能提供实际用量，可在阶段 2 以后用实际用量校正。
|
||||
|
||||
### 8.6 Registry Metadata
|
||||
|
||||
图像 provider 必须补齐 registry/provider menu 可见的元数据：
|
||||
|
||||
- `best_for`：说明官方可灵适合主体一致性、角色参考、Omni 多参考图等场景。
|
||||
- `not_good_for`：说明不适合的场景，例如本地离线、免费生成、非可灵模型能力。
|
||||
- `install_instructions`：说明配置 `KLING_API_KEY`，不要硬编码过期 URL。
|
||||
- `fallback_tools`：列出可替代图像 provider；只作为候选，不允许静默切换。
|
||||
- `supports`：至少包含文生图、图像编辑、负向提示、宽高比能力。
|
||||
- `resource_profile` / `retry_policy`：如果 BaseTool 契约已有对应字段，按 API 远端生成和长轮询任务填写。
|
||||
- `idempotency_key_fields`：至少考虑 `prompt`、`api_family`、`model_name`、`image_url/path`、`aspect_ratio`、`resolution`、`n`。
|
||||
- `side_effects`：标记为 paid remote generation，避免 proposal/preflight 把调用当作免费本地操作。
|
||||
|
||||
## 9. Selector 衔接
|
||||
|
||||
原则：新增官方可灵 provider 不应要求重写 selector。`video_selector` 和 `image_selector` 已经通过 registry 自动发现 provider，本阶段优先只新增 provider 和 contract 测试。
|
||||
|
||||
不要为了可灵专用字段改 selector 的核心选择逻辑。只有当字段属于跨 provider 的通用参数，且现有 selector 会丢弃该字段时，才允许补充 selector schema 或透传列表。
|
||||
|
||||
`video_selector` 当前应继续使用这些通用字段：
|
||||
|
||||
- `prompt`
|
||||
- `operation`
|
||||
- `preferred_provider`
|
||||
- `allowed_providers`
|
||||
- `aspect_ratio`
|
||||
- `duration`
|
||||
- `reference_image_path`
|
||||
- `reference_image_url`
|
||||
- `output_path`
|
||||
|
||||
`image_selector` 当前应继续使用这些通用字段：
|
||||
|
||||
- `prompt`
|
||||
- `negative_prompt`
|
||||
- `generation_mode`
|
||||
- `image_url`
|
||||
- `image_path`
|
||||
- `image_urls`
|
||||
- `image_paths`
|
||||
- `preferred_provider`
|
||||
- `allowed_providers`
|
||||
- `aspect_ratio`
|
||||
- `resolution`
|
||||
- `n`
|
||||
- `output_path`
|
||||
|
||||
要求：
|
||||
|
||||
- `api_family`、`model_name`、`sound`、`watermark`、`image_reference` 等可灵专用参数优先放在 provider input_schema 中，通过直接调用 provider 或 selector 的普通透传进入 provider。
|
||||
- 如果 selector 当前已经透传未知字段，不要仅为“可发现性”改 selector。
|
||||
- 如果必须改 selector，只能做 provider-neutral 的最小透传；不得加入可灵专用分支。
|
||||
- 不能破坏其他 provider 的选择和调用。
|
||||
- `video_selector` 用 `preferred_provider="kling_official"` 时必须选中官方视频工具。
|
||||
- `image_selector` 用 `preferred_provider="kling_official"` 时必须选中官方图像工具。
|
||||
- `reference_image_path` 不能触发 `upload_image_fal()`。
|
||||
|
||||
## 10. 文档和 Skill 更新
|
||||
|
||||
本阶段必须同步更新：
|
||||
|
||||
```text
|
||||
.env.example
|
||||
README.md
|
||||
docs/PROVIDERS.md
|
||||
docs/ARCHITECTURE.md
|
||||
.agents/skills/ai-video-gen/SKILL.md
|
||||
skills/creative/video-gen-prompting.md
|
||||
skills/INDEX.md
|
||||
.agents/skills/kling-official/SKILL.md
|
||||
```
|
||||
|
||||
具体要求：
|
||||
|
||||
- `.env.example` 增加 `KLING_API_KEY=` 和 `KLING_API_BASE_URL=`。
|
||||
- `README.md` provider key 列表加入官方可灵。
|
||||
- `docs/PROVIDERS.md` 新增 “Kling Official” 小节。
|
||||
- `docs/PROVIDERS.md` 明确 fal.ai Kling 和 official Kling 是两个路径。
|
||||
- `docs/ARCHITECTURE.md` API key 映射表加入 `KLING_API_KEY`。
|
||||
- `.agents/skills/ai-video-gen/SKILL.md` metadata `env_any` 加入 `KLING_API_KEY`。
|
||||
- `skills/creative/video-gen-prompting.md` 增加官方可灵适用场景和参数注意事项。
|
||||
- `skills/INDEX.md` 让后续 agent 能发现 `kling-official`。
|
||||
- 新增 `.agents/skills/kling-official/SKILL.md`，覆盖鉴权、任务协议、错误处理、参数、成本治理和提示注意事项。
|
||||
|
||||
## 11. 测试要求
|
||||
|
||||
新增或更新以下测试。
|
||||
|
||||
### 11.1 Client 测试
|
||||
|
||||
建议文件：
|
||||
|
||||
```text
|
||||
tests/contracts/test_kling_official_client.py
|
||||
```
|
||||
|
||||
覆盖：
|
||||
|
||||
- 未设置 `KLING_API_KEY` 时工具不可用。
|
||||
- 设置 `KLING_API_KEY` 后 headers 是 `Authorization: Bearer ...`。
|
||||
- base URL 默认 `https://api-singapore.klingai.com`。
|
||||
- `KLING_API_BASE_URL` 可以覆盖。
|
||||
- `code != 0` 抛 `KlingAPIError`，保留 `code`、`message`、`request_id`。
|
||||
- `code=1303` 被识别为可重试并发错误。
|
||||
- Classic create 解析 `data.task_id`。
|
||||
- Classic poll 成功解析 `data.task_result.videos/images/audios[]`。
|
||||
- Turbo create 解析 `data.id`。
|
||||
- Turbo poll 成功解析 `data[0].outputs[]`。
|
||||
- schema fixture 包含必需字段。
|
||||
|
||||
### 11.2 视频 Provider 测试
|
||||
|
||||
建议文件：
|
||||
|
||||
```text
|
||||
tests/contracts/test_kling_official_video.py
|
||||
```
|
||||
|
||||
覆盖：
|
||||
|
||||
- registry 能发现 `kling_official_video`。
|
||||
- `capability="video_generation"`。
|
||||
- `provider="kling_official"`。
|
||||
- input schema 不包含顶层 `image_url`。
|
||||
- `operation=text_to_video, api_family=classic` 构造 `/v1/videos/text2video` payload。
|
||||
- `operation=image_to_video, api_family=classic` 使用 `reference_image_url/path` 构造官方 `image` 字段。
|
||||
- `operation=text_to_video, api_family=turbo` 构造 `prompt/settings/options`。
|
||||
- `operation=image_to_video, api_family=turbo` 构造 `contents[]`。
|
||||
- `operation=reference_to_video, api_family=omni` 构造 `video_list[]` 或基础参考输入。
|
||||
- 成功后下载视频、返回 artifact、调用 `probe_output`。
|
||||
- `video_selector` 用 `preferred_provider="kling_official"` 能选中官方工具。
|
||||
- `reference_image_path` 不触发 `upload_image_fal()`。
|
||||
- `agent_skills` 包含 `kling-official`。
|
||||
- 默认 paid 输入的 `estimate_cost()` 不返回静默 `0.0`。
|
||||
|
||||
### 11.3 图像 Provider 测试
|
||||
|
||||
建议文件：
|
||||
|
||||
```text
|
||||
tests/contracts/test_kling_official_image.py
|
||||
```
|
||||
|
||||
覆盖：
|
||||
|
||||
- registry 能发现 `kling_official_image`。
|
||||
- `capability="image_generation"`。
|
||||
- `provider="kling_official"`。
|
||||
- generate payload 使用 `/v1/images/generations`。
|
||||
- edit payload 将 `image_path` 转 raw base64。
|
||||
- `api_family=omni` payload 使用 `/v1/images/omni-image` 和 `image_list[]`。
|
||||
- 多图片结果全部写入 artifacts。
|
||||
- `image_selector` 用 `preferred_provider="kling_official"` 能选中官方工具。
|
||||
- `agent_skills` 包含 `kling-official`。
|
||||
- 默认 paid 输入的 `estimate_cost()` 不返回静默 `0.0`。
|
||||
|
||||
### 11.4 文档和 Skill 测试
|
||||
|
||||
若仓库已有相关 contract 测试，补充：
|
||||
|
||||
- provider catalog 包含 `kling_official`。
|
||||
- docs provider table 包含 `KLING_API_KEY`。
|
||||
- `.agents/skills/ai-video-gen/SKILL.md` metadata `env_any` 包含 `KLING_API_KEY`。
|
||||
- `.agents/skills/kling-official/SKILL.md` 存在。
|
||||
- 官方可灵 provider 的 `agent_skills` 引用 `kling-official`。
|
||||
|
||||
### 11.5 Live QA
|
||||
|
||||
真实调用只允许显式开启：
|
||||
|
||||
```bash
|
||||
RUN_KLING_LIVE_TESTS=1 KLING_API_KEY=... pytest tests/qa/test_kling_official_live.py
|
||||
```
|
||||
|
||||
live smoke 限制：
|
||||
|
||||
- 文生图 1 张。
|
||||
- 文生视频最短时长 3s 或 5s。
|
||||
- 不跑批量。
|
||||
- 不跑 4k。
|
||||
- 不默认开声音。
|
||||
|
||||
## 12. 阶段验收清单
|
||||
|
||||
阶段 1 完成前逐项确认：
|
||||
|
||||
- `registry.support_envelope()` 能看到 `kling_official_video`。
|
||||
- `registry.support_envelope()` 能看到 `kling_official_image`。
|
||||
- 未设置 `KLING_API_KEY` 时两个工具状态为 `UNAVAILABLE`。
|
||||
- setup offer 指向 `KLING_API_KEY`。
|
||||
- 设置 `KLING_API_KEY` 时两个工具状态为 `AVAILABLE`。
|
||||
- `video_selector` 可通过 `preferred_provider="kling_official"` 选中官方视频工具。
|
||||
- `image_selector` 可通过 `preferred_provider="kling_official"` 选中官方图像工具。
|
||||
- selector 没有新增可灵专用选择分支；如有 selector 改动，必须是 provider-neutral 的最小透传。
|
||||
- `tools/video/kling_video.py` fal.ai 版本行为不变。
|
||||
- 官方视频工具没有顶层 `image_url` schema。
|
||||
- Classic 和 Turbo 两套 parser 均有 fixture 覆盖。
|
||||
- schema fixture 已按当前官方 HTML/chunk 重新抽取。
|
||||
- 两个官方 provider 的 `agent_skills` 都包含 `kling-official`。
|
||||
- 两个官方 provider 都实现非默认 `estimate_cost()`。
|
||||
- 两个官方 provider 的 paid 成功结果都写入 `ToolResult.cost_usd`。
|
||||
- 两个官方 provider 都补齐 `best_for`、`not_good_for`、`install_instructions`、`fallback_tools`、`supports` 等 registry metadata。
|
||||
- 图像多结果 artifacts 有测试覆盖。
|
||||
- README、docs、skill 明确 fal.ai Kling 与 official Kling 的差异。
|
||||
- 相关 contract 测试通过。
|
||||
|
||||
## 13. 完成后进入下一阶段
|
||||
|
||||
只有当本阶段验收清单全部完成后，才能进入第二阶段：
|
||||
|
||||
```text
|
||||
docs/kling-official-phase-2-omni-operations.md
|
||||
```
|
||||
|
||||
第二阶段会在本阶段 client、parser、provider 基础上增强 Omni、Elements、Account Usage 和 Callback。
|
||||
|
|
@ -0,0 +1,464 @@
|
|||
# 可灵官方 API 集成阶段 2：Omni、Elements、账户用量与 Callback
|
||||
|
||||
状态：实施指导文档。
|
||||
|
||||
来源：从 `docs/kling-official-integration-plan.md` 拆分而来。本阶段对应原计划中的 P2「Omni、元素、账户、callback」。
|
||||
|
||||
执行顺序：必须在 `docs/kling-official-phase-1-core.md` 完成并验收后执行。本阶段完成后再进入 `docs/kling-official-phase-3-media-avatar-effects.md`。
|
||||
|
||||
## 1. 阶段目标
|
||||
|
||||
本阶段不再解决“官方可灵能否被 OpenMontage 调用”的基础问题，而是在阶段 1 的视频、图像 provider 和共享 client 基础上增强同一个可灵官方 provider：
|
||||
|
||||
- 深化 Video Omni / Image Omni 支持。
|
||||
- 增加 Elements 引用能力，但默认作为 `kling_official_video` / `kling_official_image` 的内部 helper，不新增 OpenMontage 管理功能。
|
||||
- 增加 Account Usage 账户用量读取能力，但默认作为 provider preflight/错误诊断 helper，不新增常规 pipeline 工具。
|
||||
- 规范 callback 透传和 artifacts 记录。
|
||||
|
||||
这些能力提高的是官方可灵 provider 的参数覆盖和诊断质量，不应该改变 OpenMontage 现有 pipeline、selector、stage artifact 或 checkpoint 流程。
|
||||
|
||||
## 2. 进入条件
|
||||
|
||||
开始本阶段前必须确认：
|
||||
|
||||
- 阶段 1 验收清单已完成。
|
||||
- `tools/_kling/` client、errors、media、schemas 已存在并有测试覆盖。
|
||||
- `kling_official_video` 和 `kling_official_image` 已能被 registry 发现。
|
||||
- `preferred_provider="kling_official"` 对视频和图像 selector 均可用。
|
||||
- schema fixture 已按当前官方文档刷新。
|
||||
- 付费成本估算不再静默返回 `0.0`。
|
||||
- fal.ai 版 `kling_video` 行为未被改变。
|
||||
|
||||
如果上述任一条件不满足，先回到阶段 1 修复。
|
||||
|
||||
## 3. 不可变规则
|
||||
|
||||
本阶段必须遵守：
|
||||
|
||||
- 不新增 pipeline。仍通过现有 provider/selector/capability 体系接入。
|
||||
- 不新增 OpenMontage capability。阶段 2 的能力都挂在阶段 1 已有的 `video_generation` / `image_generation` provider 内部。
|
||||
- 不更改 `provider="kling_official"` 命名。
|
||||
- 不把 Elements 做成普通生成 provider，也不新增 `asset_management` capability。
|
||||
- 不把 callback 作为默认执行路径。当前仍以 polling 为主，callback 是高级透传能力。
|
||||
- Account Usage 不进入生产 pipeline stage，不进 selector；它只是 `tools/_kling` 下的可选诊断 helper。官方 QPS 限制为低频接口，必须做本地节流或缓存。
|
||||
- Omni 深度能力必须建立在阶段 1 的 `api_family=omni` 上，不新增 selector 层 operation。
|
||||
- 高成本 Omni、多参考、多元素调用必须进入成本估算。
|
||||
- Omni 付费调用的成功 ToolResult 必须继续写入 `cost_usd`。如果 Account Usage 能返回可核对的实际用量，应把估算成本和实际用量的校正结果记录到 ToolResult data 或项目 artifacts。
|
||||
- 所有远端结果、元素 ID、任务 ID、引用关系必须写入 artifacts 或 ToolResult data，便于复现。
|
||||
- 如果官方 schema 有变化，先更新 fixture 和测试，再改实现。
|
||||
|
||||
## 4. 工作流 A：Omni 深度接入
|
||||
|
||||
阶段 1 只要求基础 Omni 可用。本阶段要让 Omni 能真正承担复杂参考输入。
|
||||
|
||||
### 4.1 Video Omni 增强范围
|
||||
|
||||
目标端点：
|
||||
|
||||
```text
|
||||
POST /v1/videos/omni-video
|
||||
GET /v1/videos/omni-video/{id}
|
||||
GET /v1/videos/omni-video?pageNum=1&pageSize=30
|
||||
```
|
||||
|
||||
增强字段：
|
||||
|
||||
- `image_list[].image_url`
|
||||
- `image_list[].type`，例如 `first_frame`、`end_frame`
|
||||
- `video_list[].video_url`
|
||||
- `video_list[].refer_type`，例如 `feature`、`base`
|
||||
- `video_list[].keep_original_sound`
|
||||
- `element_list[].element_id`
|
||||
- `multi_shot`
|
||||
- `shot_type`
|
||||
- `multi_prompt`
|
||||
- `sound`
|
||||
- `mode`
|
||||
- `aspect_ratio`
|
||||
- `duration`
|
||||
|
||||
实现要求：
|
||||
|
||||
- 在 `kling_official_video` 中把 Omni payload builder 拆成独立 helper，避免塞进 `execute()`。
|
||||
- 支持 URL 和本地文件输入的标准化。本地图片仍由 `tools/_kling/media.py` 转换；本地视频如果官方只接受 URL，必须明确报错或要求用户提供可访问 URL，不能静默上传到 fal.ai。
|
||||
- `operation=reference_to_video` 时，必须明确要求至少一种参考输入：图片、视频或 element。
|
||||
- `video_list` 中的 `refer_type` 必须保留官方枚举，不要随意翻译成内部枚举后丢失原值。
|
||||
- `keep_original_sound` 默认应保守设置为 `no` 或不发送，避免无意保留参考视频声音。
|
||||
- `sound="on"` 属于高成本/高差异输出能力，默认不启用。
|
||||
- `mode="4k"` 默认不启用。
|
||||
|
||||
### 4.2 Image Omni 增强范围
|
||||
|
||||
目标端点：
|
||||
|
||||
```text
|
||||
POST /v1/images/omni-image
|
||||
GET /v1/images/omni-image/{id}
|
||||
GET /v1/images/omni-image?pageNum=1&pageSize=30
|
||||
```
|
||||
|
||||
增强字段：
|
||||
|
||||
- `image_list[].image`
|
||||
- `element_list[].element_id`
|
||||
- `resolution`
|
||||
- `result_type`
|
||||
- `n`
|
||||
- `series_amount`
|
||||
- `aspect_ratio`
|
||||
- prompt 中的 `<<<image_1>>>` 引用语法
|
||||
|
||||
实现要求：
|
||||
|
||||
- 提供 prompt reference helper，把用户输入的多图参考稳定映射为 `<<<image_1>>>`、`<<<image_2>>>` 等。
|
||||
- helper 必须返回映射 metadata，例如第几个引用对应哪个 URL/path。
|
||||
- 如果用户 prompt 已经包含 `<<<image_1>>>`，不要重复插入；应校验引用数量和 `image_list` 是否一致。
|
||||
- `result_type="series"` 和 `series_amount` 必须进入成本估算。
|
||||
- `resolution="4k"` 默认不启用。
|
||||
- `aspect_ratio="auto"` 只在官方当前 schema 支持时发送。
|
||||
|
||||
### 4.3 多镜头 `multi_prompt`
|
||||
|
||||
多镜头能力要做成明确结构，不要把用户自然语言拆分后随意发送。
|
||||
|
||||
建议 schema：
|
||||
|
||||
```python
|
||||
"multi_prompt": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"prompt": {"type": "string"},
|
||||
"duration": {"type": "string"},
|
||||
"camera_control": {"type": "object"},
|
||||
"image_refs": {"type": "array"},
|
||||
"element_refs": {"type": "array"}
|
||||
},
|
||||
"required": ["prompt"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
规则：
|
||||
|
||||
- `multi_shot=true` 时，普通 `prompt` 是否生效以官方 schema 为准；如果官方标注无效，不要同时依赖它。
|
||||
- `shot_type` 必须为官方枚举，例如 `customize` 或 `intelligence`。
|
||||
- 多镜头 payload builder 要有单元测试覆盖。
|
||||
- 多镜头默认不在 selector 自动路径启用，除非用户明确传入。
|
||||
|
||||
### 4.4 Omni 输出处理
|
||||
|
||||
输出规则：
|
||||
|
||||
- 仍下载最终媒体到本地 output path。
|
||||
- 如果官方返回多个媒体，全部写入 artifacts。
|
||||
- ToolResult data 至少包含：
|
||||
- `task_id`
|
||||
- `api_family="omni"`
|
||||
- `operation`
|
||||
- `model`
|
||||
- `output_path`
|
||||
- `remote_outputs`
|
||||
- `references_used`
|
||||
- `element_ids`
|
||||
- 对视频继续调用 `probe_output()`。
|
||||
- 对图像继续推断文件扩展名，默认 `.png`。
|
||||
|
||||
## 5. 工作流 B：Elements 引用 Helper
|
||||
|
||||
Elements 是 Video Omni 和 Image Omni 的官方参数能力。本阶段只把它作为官方可灵 provider 的内部引用 helper，目的是让 `element_list[].element_id` 可以被视频/图像 provider 正确传入和记录。
|
||||
|
||||
不要在本阶段把 Elements 产品化成独立 OpenMontage 管理功能。
|
||||
|
||||
### 5.1 建议文件
|
||||
|
||||
建议只新增底层 helper：
|
||||
|
||||
```text
|
||||
tools/_kling/elements.py
|
||||
```
|
||||
|
||||
不建议新增 `tools/kling_elements.py`。只有当已有 pipeline 或用户工作流明确需要独立元素管理入口时，才另开设计文档讨论。
|
||||
|
||||
### 5.2 支持范围
|
||||
|
||||
本阶段的 Elements 范围只服务 `element_list[].element_id` 引用、校验和 metadata 记录。允许封装的端点应保持只读或引用校验：
|
||||
|
||||
```text
|
||||
GET /v1/general/advanced-custom-elements/{id}
|
||||
GET /v1/general/advanced-custom-elements
|
||||
GET /v1/general/advanced-presets-elements
|
||||
```
|
||||
|
||||
明确不在本阶段实现：
|
||||
|
||||
```text
|
||||
POST /v1/general/advanced-custom-elements
|
||||
POST /v1/general/delete-elements
|
||||
```
|
||||
|
||||
原因：创建/删除 element 是素材管理功能，不是“新增可灵官方供应商”的必要路径。若后续确实需要元素生命周期管理，必须单独设计用户入口、权限、artifact 生命周期和 pipeline 使用方式，不能混在本阶段 provider 增强里。
|
||||
|
||||
### 5.3 Helper 契约
|
||||
|
||||
helper 不继承 `BaseTool`，不进入 registry，不进入 selector。
|
||||
|
||||
建议函数：
|
||||
|
||||
| helper | 用途 |
|
||||
|--------|------|
|
||||
| `normalize_element_list(...)` | 校验并标准化传给视频/图像 provider 的 `element_list` |
|
||||
| `get_custom_element(...)` | 可选查询单个自定义元素，用于校验用户传入的 element id |
|
||||
| `list_preset_elements(...)` | 可选列出官方预设元素，供诊断或文档使用 |
|
||||
|
||||
输入规则：
|
||||
|
||||
- provider 接收 `element_list` 时，只负责校验结构、透传给官方 API、记录 metadata。
|
||||
- 不在默认路径创建或删除 element。
|
||||
- 如果后续确实需要创建/删除 element，必须单独设计，不混在本阶段 provider 接入里。
|
||||
- 对 preset elements 只读。
|
||||
|
||||
输出规则：
|
||||
|
||||
- 在 ToolResult data 中记录本次使用的 `element_ids`。
|
||||
- 如查询过 element 详情，将 element metadata 写入项目 artifacts，便于复现。
|
||||
- 如果官方查询响应是异步任务，使用共享 Classic parser 或新增专用 parser，不要猜字段。
|
||||
|
||||
### 5.4 Artifacts 约定
|
||||
|
||||
建议在项目中记录：
|
||||
|
||||
```text
|
||||
projects/<project-name>/artifacts/kling_elements.json
|
||||
```
|
||||
|
||||
结构建议：
|
||||
|
||||
```json
|
||||
{
|
||||
"provider": "kling_official",
|
||||
"elements": [
|
||||
{
|
||||
"element_id": 123,
|
||||
"kind": "character",
|
||||
"name": "main-presenter",
|
||||
"source": "...",
|
||||
"created_at": "...",
|
||||
"task_id": "...",
|
||||
"reusable": true
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## 6. 工作流 C：Account Usage 诊断
|
||||
|
||||
Account Usage 用于官方可灵 provider 的 setup/preflight 和错误诊断，不是生产生成能力，不进入 pipeline stage。
|
||||
|
||||
### 6.1 建议文件
|
||||
|
||||
建议只新增底层 helper：
|
||||
|
||||
```text
|
||||
tools/_kling/account.py
|
||||
```
|
||||
|
||||
不建议新增 `tools/kling_account_usage.py`。如果后续需要用户显式运行账户诊断，再单独设计工具入口。
|
||||
|
||||
### 6.2 支持端点
|
||||
|
||||
目标端点：
|
||||
|
||||
```text
|
||||
GET /account/costs
|
||||
```
|
||||
|
||||
字段：
|
||||
|
||||
- `start_time`
|
||||
- `end_time`
|
||||
- `resource_pack_name`
|
||||
|
||||
### 6.3 使用场景
|
||||
|
||||
必须支持：
|
||||
|
||||
- provider setup/preflight 中检查资源包或余额可见性。
|
||||
- 捕获 `1101` / `1102` 后提供更清楚的账户诊断。
|
||||
- 在用户要求排查“为什么可灵不能生成”时提供低成本诊断。
|
||||
|
||||
不要求：
|
||||
|
||||
- 每次生成前都调用账户用量接口。
|
||||
- 在 CI 中调用真实账户接口。
|
||||
|
||||
### 6.4 节流和缓存
|
||||
|
||||
规则：
|
||||
|
||||
- 官方 Account Usage QPS 低，必须本地节流。
|
||||
- 同一进程内相同参数短时间重复查询应使用缓存。
|
||||
- 如果被节流，返回“最近一次缓存结果”或清楚说明需要稍后重试。
|
||||
- 不允许为了诊断在短时间内循环打账户接口。
|
||||
|
||||
### 6.5 输出
|
||||
|
||||
如果 helper 被 provider 调用，ToolResult data 建议包含：
|
||||
|
||||
- `resource_pack_subscribe_infos`
|
||||
- `queried_range`
|
||||
- `cached`
|
||||
- `throttle_status`
|
||||
- `provider="kling_official"`
|
||||
- 如果用于校正生成成本，记录 `reconciled_cost_usd`、`cost_source` 和关联的 `task_id` 或时间窗口。
|
||||
|
||||
如果官方返回余额结构随资源包变化，保留原始字段，并给出轻量归一化摘要。
|
||||
|
||||
## 7. 工作流 D：Callback 支持
|
||||
|
||||
阶段 2 只要求 callback 透传和记录，不要求实现完整 callback receiver。
|
||||
|
||||
### 7.1 Provider 参数
|
||||
|
||||
视频、图像、Omni、后续音频工具都应接受：
|
||||
|
||||
```python
|
||||
"callback_url": {"type": "string"}
|
||||
```
|
||||
|
||||
并按官方 schema 放入对应字段：
|
||||
|
||||
- Classic：顶层 `callback_url`。
|
||||
- Turbo：`options.callback_url`。
|
||||
- 其他端点按当前 fixture 确认。
|
||||
|
||||
### 7.2 默认执行模式
|
||||
|
||||
默认仍然是 polling：
|
||||
|
||||
- 工具创建任务。
|
||||
- 工具轮询任务到终态。
|
||||
- 工具下载结果。
|
||||
- 工具返回 ToolResult。
|
||||
|
||||
即使传入 `callback_url`，当前工具也不应立即假设 callback receiver 会写入 artifacts，除非 receiver 已明确存在并通过测试。
|
||||
|
||||
### 7.3 Artifacts 记录
|
||||
|
||||
如果传入 callback，ToolResult data 应记录：
|
||||
|
||||
- `callback_url`
|
||||
- `callback_requested=true`
|
||||
- `polling_used=true`
|
||||
- `task_id`
|
||||
|
||||
如果后续实现 receiver，可追加：
|
||||
|
||||
- `callback_received_at`
|
||||
- `callback_payload_path`
|
||||
- `callback_status`
|
||||
|
||||
### 7.4 失败处理
|
||||
|
||||
- callback URL 无效时，优先在调用前做基本 URL 校验。
|
||||
- 官方 callback 投递失败不应影响 polling 结果，只要 polling 成功。
|
||||
- 如果 polling 失败但 callback 成功，必须能从 artifacts 找到 callback payload。
|
||||
|
||||
## 8. 文档和 Skill 更新
|
||||
|
||||
本阶段更新：
|
||||
|
||||
```text
|
||||
docs/PROVIDERS.md
|
||||
docs/ARCHITECTURE.md
|
||||
.agents/skills/kling-official/SKILL.md
|
||||
skills/INDEX.md
|
||||
docs/kling-official-integration-plan.md
|
||||
```
|
||||
|
||||
要求：
|
||||
|
||||
- `docs/PROVIDERS.md` 增加 Omni 深度能力、Elements、Account Usage 的说明。
|
||||
- `docs/ARCHITECTURE.md` 标明 Elements 和 Account Usage 只是可灵官方 provider 的内部引用/helper 与诊断能力，不是生成 pipeline 或独立产品功能。
|
||||
- `.agents/skills/kling-official/SKILL.md` 增加 Omni 引用语法、多参考输入、元素 ID 引用和 callback 注意事项。
|
||||
- `skills/INDEX.md` 仅标出 Elements/Account Usage 作为 `kling_official` helper 的用途，不新增 capability 分类。
|
||||
- 原总计划如果继续保留，应标注阶段 2 已拆分到本文件。
|
||||
|
||||
## 9. 测试要求
|
||||
|
||||
### 9.1 Omni 测试
|
||||
|
||||
覆盖：
|
||||
|
||||
- Video Omni 多 `image_list` payload。
|
||||
- Video Omni `video_list` payload。
|
||||
- Video Omni `element_list` payload。
|
||||
- `reference_to_video` 无参考输入时报参数错误。
|
||||
- `multi_prompt` payload。
|
||||
- `sound="on"`、`mode="4k"` 成本估算提高或标记高成本。
|
||||
- Image Omni 多图引用 prompt helper。
|
||||
- prompt 中已有 `<<<image_1>>>` 时不重复插入。
|
||||
- `result_type="series"` 多结果 artifacts。
|
||||
|
||||
### 9.2 Elements 测试
|
||||
|
||||
覆盖：
|
||||
|
||||
- Elements helper 不进入 registry。
|
||||
- Elements helper 不挂到普通 `video_generation` / `image_generation` selector 路径。
|
||||
- `element_list` 标准化和校验。
|
||||
- 查询元素 payload。
|
||||
- preset list payload。
|
||||
- 默认路径不创建/删除 element。
|
||||
- element metadata 写入 artifacts。
|
||||
|
||||
### 9.3 Account Usage 测试
|
||||
|
||||
覆盖：
|
||||
|
||||
- endpoint、参数和 Authorization header。
|
||||
- QPS 节流。
|
||||
- 相同查询缓存。
|
||||
- `1101` / `1102` 错误后能触发诊断 helper 或输出建议。
|
||||
- Account Usage helper 不进入 registry/selector。
|
||||
- CI 默认不打真实账户接口。
|
||||
|
||||
### 9.4 Callback 测试
|
||||
|
||||
覆盖：
|
||||
|
||||
- Classic callback_url 顶层透传。
|
||||
- Turbo callback_url 放入 `options.callback_url`。
|
||||
- ToolResult 记录 `callback_requested` 和 `polling_used`。
|
||||
- callback URL 基本校验。
|
||||
|
||||
## 10. 阶段验收清单
|
||||
|
||||
阶段 2 完成前逐项确认：
|
||||
|
||||
- Video Omni 支持多图、多视频、元素引用。
|
||||
- Image Omni 支持多图引用和 series 输出。
|
||||
- Omni prompt reference helper 有测试覆盖。
|
||||
- 多镜头 `multi_prompt` 有测试覆盖。
|
||||
- Elements helper 存在并能记录 element metadata。
|
||||
- Elements helper 不进入 registry/selector。
|
||||
- Account Usage helper 存在。
|
||||
- Account Usage helper 不进入 registry/selector。
|
||||
- Account Usage 有节流和缓存测试。
|
||||
- callback_url 能在 Classic/Turbo/Omni 路径正确透传。
|
||||
- callback 当前仍以 polling 为默认执行路径。
|
||||
- 成本估算覆盖多参考、多结果、4k、声音等高成本参数。
|
||||
- Omni paid 成功结果继续写入 `ToolResult.cost_usd`，Account Usage 可用时能记录校正信息。
|
||||
- 文档和 `kling-official` skill 已更新。
|
||||
- 阶段 1 的视频/图像基础测试仍通过。
|
||||
|
||||
## 11. 完成后进入下一阶段
|
||||
|
||||
只有当本阶段验收清单全部完成后，才能进入第三阶段：
|
||||
|
||||
```text
|
||||
docs/kling-official-phase-3-media-avatar-effects.md
|
||||
```
|
||||
|
||||
第三阶段会评估可灵官方在现有 OpenMontage capability 中还能补哪些 provider。没有现有 capability 或 pipeline 消费路径的端点默认不接。
|
||||
|
|
@ -0,0 +1,561 @@
|
|||
# 可灵官方 API 集成阶段 3：TTS、音效、数字人、口型与视频特效
|
||||
|
||||
状态：已按本阶段边界完成 OpenMontage provider 接入；真实可灵 API 调用仍需显式 live QA 或人工端到端测试。
|
||||
|
||||
来源：从 `docs/kling-official-integration-plan.md` 拆分而来。本阶段对应原计划中的 P3「音频、TTS、数字人、口型、特效」。
|
||||
|
||||
执行顺序：必须在以下两个阶段完成后执行：
|
||||
|
||||
1. `docs/kling-official-phase-1-core.md`
|
||||
2. `docs/kling-official-phase-2-omni-operations.md`
|
||||
|
||||
## 1. 阶段目标
|
||||
|
||||
本阶段目标是在已有 OpenMontage capability 中继续增加 `kling_official` provider 覆盖，而不是新增产品功能：
|
||||
|
||||
- 可灵 TTS：接入现有 `tts` capability 和 `tts_selector`。
|
||||
- 可灵数字人：作为可选 provider 工具接入现有 `avatar` capability，不自动替代 `talking_head.py`。
|
||||
- 可灵口型：作为可选 provider 工具接入现有 `avatar` capability，与已有 `lip_sync.py` 并存，不自动替代本地口型工具。
|
||||
- 可灵音效：仅当能自然映射到现有 `music_generation` 或已有音频后期流程时才接；默认不新增 `sound_effects` capability。
|
||||
- 可灵视频特效：默认不接入普通 `video_generation`，除非已有 pipeline 明确消费该类 operation；默认不新增 `video_effects` capability。
|
||||
|
||||
本阶段的重点是“给已有槽位增加可灵官方供应商”，不是把官方 API 的所有端点都产品化。
|
||||
|
||||
当前实现记录：
|
||||
|
||||
- 已新增 `tools/audio/kling_tts.py`，注册到 `tts` capability，并可由 `tts_selector` 通过 `preferred_provider="kling_official"` 选中。
|
||||
- 已新增 `tools/avatar/kling_avatar.py`，注册到 `avatar` capability，与本地 `talking_head.py` 并存。
|
||||
- 已新增 `tools/avatar/kling_lip_sync.py`，注册到 `avatar` capability，与本地 `lip_sync.py` 并存，并保留多人脸人工确认出口。
|
||||
- 未新增 `kling_audio` 或 `kling_effects`，避免为当前 pipeline 引入未设计的 `sound_effects` / `video_effects` 能力面。
|
||||
- 已更新 `docs/PROVIDERS.md`、`docs/ARCHITECTURE.md`、`README.md`、`skills/INDEX.md`、`.agents/skills/kling-official/SKILL.md` 和 contract tests。
|
||||
|
||||
注意：仓库当前只有 `tts_selector`、`image_selector`、`video_selector` 三类 selector；没有 `avatar_selector`。`avatar-spokesperson` 和 `localization-dub` pipeline 目前通过 manifest/director 显式列出 `talking_head` / `lip_sync`。因此 `kling_avatar` / `kling_lip_sync` 被 registry 发现并不等于现有 avatar pipeline 会自动消费它们。如需让现有 pipeline 使用，只能在对应 pipeline 的 `tools_available`、`optional_tools` 和 stage director tool plan 中做最小供应商选项更新，不新增 stage、canonical artifact 或新的 selector。
|
||||
|
||||
## 2. 进入条件
|
||||
|
||||
开始前必须确认：
|
||||
|
||||
- 阶段 1 的官方视频和图像 provider 已完成并验收。
|
||||
- 阶段 2 的 Omni、Elements helper、Account Usage helper、Callback 已完成并验收。
|
||||
- `tools/_kling/` client 能复用到音频、头像、口型和特效端点。
|
||||
- `kling-official` skill 已覆盖任务协议、错误处理、成本治理和 Omni 引用。
|
||||
- 当前官方 schema fixture 已刷新，并包含本阶段端点的核心字段。
|
||||
- CI 仍默认不打真实可灵 API。
|
||||
|
||||
## 3. 全局规则
|
||||
|
||||
本阶段新增的每个工具都必须遵守：
|
||||
|
||||
- 继承 `BaseTool`。
|
||||
- 使用 `provider="kling_official"`。
|
||||
- 声明 `dependencies = ["env:KLING_API_KEY"]`。
|
||||
- 声明 `runtime = ToolRuntime.API`。
|
||||
- `agent_skills` 至少包含 `kling-official`，并按能力补充对应 Layer 3 skill。
|
||||
- 显式实现 `estimate_cost()`，不能静默返回 `0.0`。
|
||||
- 成功的 paid ToolResult 必须写入 `cost_usd`。如果阶段 2 的 Account Usage 能提供实际用量，可记录估算成本和实际用量的校正信息。
|
||||
- 所有远端结果必须下载到本地 output path 或项目 artifacts。
|
||||
- ToolResult 必须包含 `task_id`、`provider`、`model`、`operation`、`output_path` 或等价字段。
|
||||
- 官方错误必须保留 `code`、`message`、`request_id`。
|
||||
- 真实 API 测试必须由显式环境变量开启。
|
||||
|
||||
Selector 规则：
|
||||
|
||||
- TTS 可以接入 `tts_selector`。
|
||||
- 音效不要伪装成长音乐生成，除非 capability 暂时只能挂到 `music_generation`，且 `best_for/not_good_for` 必须写清楚。
|
||||
- 数字人和口型可以挂到 `avatar` capability，但仓库当前没有 `avatar_selector`。现有 avatar pipeline 是显式工具槽位模式，不能假设新增 provider 会自动被 pipeline 选择。
|
||||
- 如果要让 `avatar-spokesperson` 或 `localization-dub` 使用 `kling_avatar` / `kling_lip_sync`，必须按现有 pipeline 规范显式更新 manifest 和 director skill 的工具选择规则；这只能是供应商选项更新，不能新增流程。
|
||||
- 视频特效不应挂到普通 `video_generation` 自动选择路径。
|
||||
- 本阶段默认不新增 capability。若确实需要 `sound_effects`、`video_effects` 这类新 capability，必须另写设计文档，并说明对应 pipeline、selector、artifact 和用户入口；不能混在“新增可灵供应商”任务里。
|
||||
|
||||
## 4. 工作流 A：可灵 TTS
|
||||
|
||||
### 4.1 建议文件
|
||||
|
||||
```text
|
||||
tools/audio/kling_tts.py
|
||||
```
|
||||
|
||||
### 4.2 Tool 契约
|
||||
|
||||
建议：
|
||||
|
||||
```python
|
||||
class KlingTTS(BaseTool):
|
||||
name = "kling_tts"
|
||||
version = "0.1.0"
|
||||
tier = ToolTier.GENERATE
|
||||
capability = "tts"
|
||||
provider = "kling_official"
|
||||
runtime = ToolRuntime.API
|
||||
dependencies = ["env:KLING_API_KEY"]
|
||||
agent_skills = ["kling-official", "text-to-speech"]
|
||||
```
|
||||
|
||||
### 4.3 API 范围
|
||||
|
||||
目标端点：
|
||||
|
||||
```text
|
||||
POST /v1/audio/tts
|
||||
GET /v1/audio/tts/{id}
|
||||
```
|
||||
|
||||
核心字段：
|
||||
|
||||
- `text`
|
||||
- `voice_id`
|
||||
- `voice_language`，例如 `zh`、`en`
|
||||
- `voice_speed`
|
||||
|
||||
结果路径：
|
||||
|
||||
```text
|
||||
data.task_result.audios[]
|
||||
```
|
||||
|
||||
### 4.4 实现要求
|
||||
|
||||
- 接入 `tts_selector`，确保 `preferred_provider="kling_official"` 可选中。
|
||||
- `text` 必填，并在调用前做长度校验。
|
||||
- `voice_language` 使用官方枚举，不要用自由文本。
|
||||
- `voice_speed` 做范围校验。
|
||||
- 如果没有传 `voice_id`，要么使用官方默认，要么返回清晰错误；不要硬编码不存在的 voice。
|
||||
- 输出音频必须下载到 `output_path`。
|
||||
- 如果返回多个音频，全部写入 artifacts，`data.output_path` 指向第一条。
|
||||
- 记录 `voice_id`、`voice_language`、`voice_speed`。
|
||||
|
||||
### 4.5 测试
|
||||
|
||||
覆盖：
|
||||
|
||||
- registry 能发现 `kling_tts`。
|
||||
- `capability="tts"`。
|
||||
- `provider="kling_official"`。
|
||||
- `tts_selector` 可通过 `preferred_provider="kling_official"` 选中。
|
||||
- payload 字段正确。
|
||||
- 成功结果下载音频。
|
||||
- `agent_skills` 包含 `kling-official`。
|
||||
- `estimate_cost()` 不静默返回 `0.0`。
|
||||
|
||||
## 5. 工作流 B：可灵音效
|
||||
|
||||
### 5.1 建议文件
|
||||
|
||||
```text
|
||||
tools/audio/kling_audio.py
|
||||
```
|
||||
|
||||
### 5.2 Capability 决策
|
||||
|
||||
实施前必须做一次“是否接入现有能力槽位”的决策：
|
||||
|
||||
| 选择 | 适用情况 | 要求 |
|
||||
|------|----------|------|
|
||||
| 接入 `music_generation` | 官方返回内容能满足现有音乐/音频生成 stage 的消费方式 | `best_for/not_good_for` 必须写明它偏音效，不是长音乐生成 |
|
||||
| 暂不接入 | 端点更像短音效或视频后期声音，不符合现有 pipeline 消费方式 | 只在 `kling-official` skill 和后续计划中记录，不新增工具 |
|
||||
|
||||
推荐：默认暂不接入，除非现有 pipeline 明确需要该 provider。不要在本阶段新增 `sound_effects` capability。
|
||||
|
||||
### 5.3 API 范围
|
||||
|
||||
文生音效：
|
||||
|
||||
```text
|
||||
POST /v1/audio/text-to-audio
|
||||
GET /v1/audio/text-to-audio/{id}
|
||||
```
|
||||
|
||||
核心字段：
|
||||
|
||||
- `prompt`
|
||||
- `duration`
|
||||
|
||||
视频生音效：
|
||||
|
||||
```text
|
||||
POST /v1/audio/video-to-audio
|
||||
GET /v1/audio/video-to-audio/{id}
|
||||
```
|
||||
|
||||
核心字段：
|
||||
|
||||
- `video_id` 或 `video_url`
|
||||
- `sound_effect_prompt`
|
||||
- `bgm_prompt`
|
||||
- `asmr_mode`
|
||||
|
||||
结果路径：
|
||||
|
||||
```text
|
||||
data.task_result.audios[]
|
||||
```
|
||||
|
||||
视频生音效也可能返回 videos，必须按当前 schema fixture 处理。
|
||||
|
||||
### 5.4 实现要求
|
||||
|
||||
- 使用 `operation` 区分 `text_to_audio` 和 `video_to_audio`。
|
||||
- 如果接入 `music_generation`，必须保证现有音乐 stage 能消费输出；否则不要接入 registry，只保留在官方 skill/后续计划里。
|
||||
- `text_to_audio` 必须有 `prompt`。
|
||||
- `video_to_audio` 必须有 `video_id` 或 `video_url`。
|
||||
- 如果输入本地视频而官方只接受 URL，必须明确要求 URL 或先实现官方支持的上传路径；不能静默使用其它 provider。
|
||||
- 输出音频必须下载到本地。
|
||||
- 如果返回视频，也要下载并放入 artifacts。
|
||||
- `duration` 必须进入成本估算。
|
||||
- `asmr_mode`、bgm、长时长属于更高成本/更强效果差异参数，默认不启用。
|
||||
|
||||
### 5.5 测试
|
||||
|
||||
覆盖：
|
||||
|
||||
- 如果接入现有 capability，registry 能发现音效 provider。
|
||||
- 如果暂不接入，文档明确“不接入当前 registry”的理由。
|
||||
- `text_to_audio` payload。
|
||||
- `video_to_audio` payload。
|
||||
- 无必需输入时报参数错误。
|
||||
- 多音频 artifacts。
|
||||
- 如返回视频，视频 artifacts。
|
||||
- 不被普通 TTS 或普通音乐流程误选；如果无法保证，则不接入 registry。
|
||||
- `estimate_cost()` 不静默返回 `0.0`。
|
||||
|
||||
## 6. 工作流 C：可灵数字人
|
||||
|
||||
### 6.1 建议文件
|
||||
|
||||
```text
|
||||
tools/avatar/kling_avatar.py
|
||||
```
|
||||
|
||||
### 6.2 Tool 契约
|
||||
|
||||
建议：
|
||||
|
||||
```python
|
||||
class KlingAvatar(BaseTool):
|
||||
name = "kling_avatar"
|
||||
version = "0.1.0"
|
||||
tier = ToolTier.GENERATE
|
||||
capability = "avatar"
|
||||
provider = "kling_official"
|
||||
runtime = ToolRuntime.API
|
||||
dependencies = ["env:KLING_API_KEY"]
|
||||
agent_skills = ["kling-official", "avatar-video"]
|
||||
```
|
||||
|
||||
### 6.3 API 范围
|
||||
|
||||
目标端点：
|
||||
|
||||
```text
|
||||
POST /v1/videos/avatar/image2video
|
||||
GET /v1/videos/avatar/image2video/{id}
|
||||
```
|
||||
|
||||
核心字段：
|
||||
|
||||
- `image`
|
||||
- `audio_id` 或 `sound_file`
|
||||
- `prompt`
|
||||
- `mode`，例如 `std`、`pro`
|
||||
|
||||
结果路径：
|
||||
|
||||
```text
|
||||
data.task_result.videos[]
|
||||
```
|
||||
|
||||
### 6.4 实现要求
|
||||
|
||||
- 与本地 `talking_head.py` 并存，不替代本地工具。
|
||||
- 官方工具适合云端高质量数字人；本地工具适合无 API 成本、离线可控。
|
||||
- 不修改 `talking_head.py` 的行为，不把 `kling_avatar` 包装成 `talking_head` 的内部分支。
|
||||
- 如果需要在 `avatar-spokesperson` 中启用，只在现有 pipeline 的工具列表和 director 决策里增加一个可选供应商路径；不改变 scene_plan、asset_manifest 或 checkpoint 契约。
|
||||
- 输入头像图片可接受 URL 或本地路径；本地路径按官方要求转 raw base64 或报清晰错误。
|
||||
- 音频可以使用 `audio_id` 或 `sound_file`，具体字段按当前 schema fixture。
|
||||
- `mode="pro"` 进入成本估算。
|
||||
- 输出视频必须下载到本地，并调用 `probe_output()`。
|
||||
- ToolResult 记录头像来源、音频来源、模式、任务 ID。
|
||||
|
||||
### 6.5 测试
|
||||
|
||||
覆盖：
|
||||
|
||||
- registry 能发现 `kling_avatar`。
|
||||
- `capability="avatar"`。
|
||||
- payload 使用头像和音频字段。
|
||||
- 缺少头像或音频时报参数错误。
|
||||
- 成功后下载视频并 probe。
|
||||
- 与本地 `talking_head.py` 不冲突。
|
||||
- 若更新了 `avatar-spokesperson`，测试或文档必须证明它是显式选择 `kling_avatar`，不是靠 registry 自动替换 `talking_head`。
|
||||
- `estimate_cost()` 不静默返回 `0.0`。
|
||||
|
||||
## 7. 工作流 D：可灵口型
|
||||
|
||||
### 7.1 建议文件
|
||||
|
||||
```text
|
||||
tools/avatar/kling_lip_sync.py
|
||||
```
|
||||
|
||||
### 7.2 流程
|
||||
|
||||
口型生成至少分两步：
|
||||
|
||||
1. 识别人脸：
|
||||
|
||||
```text
|
||||
POST /v1/videos/identify-face
|
||||
```
|
||||
|
||||
输入：
|
||||
|
||||
- `video_id` 或 `video_url`
|
||||
|
||||
输出：
|
||||
|
||||
- `data.session_id`
|
||||
- face 信息列表
|
||||
|
||||
2. 生成口型：
|
||||
|
||||
```text
|
||||
POST /v1/videos/advanced-lip-sync
|
||||
GET /v1/videos/advanced-lip-sync/{id}
|
||||
```
|
||||
|
||||
输入：
|
||||
|
||||
- `session_id`
|
||||
- `face_choose[]`
|
||||
- `audio_id` 或 `sound_file`
|
||||
|
||||
输出：
|
||||
|
||||
```text
|
||||
data.task_result.videos[]
|
||||
```
|
||||
|
||||
### 7.3 Tool 契约
|
||||
|
||||
建议：
|
||||
|
||||
```python
|
||||
class KlingLipSync(BaseTool):
|
||||
name = "kling_lip_sync"
|
||||
version = "0.1.0"
|
||||
tier = ToolTier.GENERATE
|
||||
capability = "avatar"
|
||||
provider = "kling_official"
|
||||
runtime = ToolRuntime.API
|
||||
dependencies = ["env:KLING_API_KEY"]
|
||||
agent_skills = ["kling-official", "avatar-video"]
|
||||
```
|
||||
|
||||
### 7.4 人脸选择规则
|
||||
|
||||
口型 face 选择可能影响输出人物，必须保留人工确认出口。
|
||||
|
||||
规则：
|
||||
|
||||
- 如果用户明确传入 `face_id` 或 `face_choose`，直接使用。
|
||||
- 如果未传入，工具可以返回 face list 并要求上层确认。
|
||||
- 如果实现自动选择，只能作为显式参数启用，例如 `auto_select_face=True`。
|
||||
- 自动策略应选择最大或最居中的 face，并在 ToolResult 中记录选择理由。
|
||||
- 不允许在多人视频中静默选择第一张脸。
|
||||
|
||||
### 7.5 Artifacts
|
||||
|
||||
建议记录：
|
||||
|
||||
```text
|
||||
projects/<project-name>/artifacts/kling_lip_sync_faces.json
|
||||
```
|
||||
|
||||
内容：
|
||||
|
||||
- `session_id`
|
||||
- face list
|
||||
- 每个 face 的位置、大小、置信度等官方字段
|
||||
- 选中的 face
|
||||
- 选择方式：`user_selected` 或 `auto_selected`
|
||||
|
||||
### 7.6 实现要求
|
||||
|
||||
- 识别人脸和生成口型可以是同一个工具的不同 operation，也可以拆 helper。
|
||||
- `identify_face` operation 可以只返回 face list，不生成视频。
|
||||
- `advanced_lip_sync` operation 必须有 `session_id` 和音频输入。
|
||||
- 一站式 operation 可以执行识别、选择、生成，但多人场景必须遵守人工确认规则。
|
||||
- 不修改 `lip_sync.py` 的行为，不把 `kling_lip_sync` 包装成 `lip_sync` 的内部分支。
|
||||
- 如果需要在 `localization-dub` 或 `avatar-spokesperson` 中启用，只在现有 pipeline 的工具列表和 director 决策里增加一个可选供应商路径；不改变 artifact schema 或 stage 顺序。
|
||||
- 输出视频下载到本地，并调用 `probe_output()`。
|
||||
- 成本估算要覆盖识别和生成两个步骤。
|
||||
|
||||
### 7.7 测试
|
||||
|
||||
覆盖：
|
||||
|
||||
- `identify_face` payload。
|
||||
- face list 解析。
|
||||
- 无 face 时报清晰错误。
|
||||
- 多 face 未启用自动选择时不静默继续。
|
||||
- `auto_select_face=True` 时记录选择理由。
|
||||
- `advanced_lip_sync` payload。
|
||||
- 成功后下载视频并 probe。
|
||||
- 若更新了 `localization-dub` 或 `avatar-spokesperson`，测试或文档必须证明它是显式选择 `kling_lip_sync`，不是靠 registry 自动替换 `lip_sync`。
|
||||
- `estimate_cost()` 不静默返回 `0.0`。
|
||||
|
||||
## 8. 工作流 E：可灵视频特效
|
||||
|
||||
### 8.1 建议文件
|
||||
|
||||
```text
|
||||
tools/video/kling_effects.py
|
||||
```
|
||||
|
||||
### 8.2 Capability 决策
|
||||
|
||||
视频特效不是普通 text-to-video，也不是通用 image-to-video。
|
||||
|
||||
本阶段默认不接入 `kling_effects`，原因是现有 `video_selector` 的标准 operation 是 `text_to_video`、`image_to_video`、`reference_to_video`、`rank`，视频特效没有稳定的 pipeline 消费路径。
|
||||
|
||||
只有满足以下条件时才允许接入：
|
||||
|
||||
- 已有 pipeline 或 stage 明确需要视频特效 operation。
|
||||
- 已定义该 operation 如何进入 stage artifact。
|
||||
- 不新增普通视频生成选择分支。
|
||||
- 不让 `video_selector` 在 `text_to_video` / `image_to_video` / `reference_to_video` 中自动选择它。
|
||||
|
||||
不要在本阶段新增 `video_effects` capability。若确实需要，应另写设计文档。
|
||||
|
||||
### 8.3 API 范围
|
||||
|
||||
目标端点：
|
||||
|
||||
```text
|
||||
POST /v1/videos/effects
|
||||
GET /v1/videos/effects/{id}
|
||||
```
|
||||
|
||||
核心字段：
|
||||
|
||||
- `effect_scene`
|
||||
- `input.image`
|
||||
- `input.images`
|
||||
|
||||
结果路径：
|
||||
|
||||
```text
|
||||
data.task_result.videos[]
|
||||
```
|
||||
|
||||
### 8.4 实现要求
|
||||
|
||||
默认不实现。若满足上面的接入条件：
|
||||
|
||||
- 使用 `operation="video_effect"` 或更具体的 effect operation。
|
||||
- `effect_scene` 必须是官方枚举或 fixture 中记录的有效值。
|
||||
- 必须有输入图片，除非官方某个 effect 明确不需要。
|
||||
- 支持单图和多图输入。
|
||||
- 本地图片转 raw base64 或按官方要求处理。
|
||||
- 输出视频下载到本地，并调用 `probe_output()`。
|
||||
- `best_for` 写清楚它适合特效模板，不适合普通视频生成。
|
||||
- `not_good_for` 写清楚不要用于普通叙事视频、解释器视频、连续镜头生成。
|
||||
|
||||
### 8.5 测试
|
||||
|
||||
覆盖：
|
||||
|
||||
- 如果实现，registry 路由方式被文档和测试固化。
|
||||
- 如果不实现，阶段验收中明确记录“不接入当前 registry”的理由。
|
||||
- 普通 `video_selector` text_to_video 不会误选 `kling_effects`。
|
||||
- effect payload。
|
||||
- 缺少图片或 effect_scene 时报参数错误。
|
||||
- 成功后下载视频并 probe。
|
||||
- `estimate_cost()` 不静默返回 `0.0`。
|
||||
|
||||
## 9. 文档和 Skill 更新
|
||||
|
||||
本阶段更新：
|
||||
|
||||
```text
|
||||
docs/PROVIDERS.md
|
||||
docs/ARCHITECTURE.md
|
||||
README.md
|
||||
skills/INDEX.md
|
||||
.agents/skills/kling-official/SKILL.md
|
||||
```
|
||||
|
||||
要求：
|
||||
|
||||
- `docs/PROVIDERS.md` 增加已接入的 TTS、数字人、口型 provider；音效和特效若不接入，只记录为官方 API 未映射端点。
|
||||
- `docs/ARCHITECTURE.md` 只标明现有 capability 的 provider 扩展；不要描述未落地的新 capability。
|
||||
- `README.md` provider key 列表不重复，只保留 `KLING_API_KEY` / `KLING_API_BASE_URL` 说明。
|
||||
- `skills/INDEX.md` 能让 agent 发现对应能力。
|
||||
- 若让现有 avatar pipeline 消费 `kling_avatar` / `kling_lip_sync`，对应 `pipeline_defs/` 和 `skills/pipelines/...` 只记录供应商选择规则，不新增 pipeline 阶段或 canonical artifact。
|
||||
- `.agents/skills/kling-official/SKILL.md` 增加：
|
||||
- TTS voice 参数注意事项。
|
||||
- 音效 prompt 注意事项，以及默认不新增 `sound_effects` capability 的原因。
|
||||
- 数字人头像/音频输入注意事项。
|
||||
- 口型 face 选择规则。
|
||||
- 视频特效默认不接入普通 `video_generation` 的警告。
|
||||
|
||||
## 10. 测试总要求
|
||||
|
||||
新增或更新：
|
||||
|
||||
```text
|
||||
tests/contracts/test_kling_tts.py
|
||||
tests/contracts/test_kling_avatar.py
|
||||
tests/contracts/test_kling_lip_sync.py
|
||||
```
|
||||
|
||||
可选测试：
|
||||
|
||||
- 如果 `kling_audio` 接入现有 capability，新增 `tests/contracts/test_kling_audio.py`。
|
||||
- 如果 `kling_effects` 有明确 pipeline 消费路径并被实现，新增 `tests/contracts/test_kling_effects.py`。
|
||||
|
||||
本阶段默认不新增 capability。如果后续单独设计了新 capability，该设计必须自带 registry、provider menu、selector 或非 selector 路由测试。
|
||||
|
||||
Live QA 必须显式开启：
|
||||
|
||||
```bash
|
||||
RUN_KLING_LIVE_TESTS=1 KLING_API_KEY=... pytest tests/qa/test_kling_official_live.py
|
||||
```
|
||||
|
||||
live smoke 限制：
|
||||
|
||||
- 每类能力只跑最小调用。
|
||||
- 不跑批量。
|
||||
- 不默认使用高成本模式。
|
||||
- 不在 CI 默认开启。
|
||||
|
||||
## 11. 阶段验收清单
|
||||
|
||||
阶段 3 完成前逐项确认：
|
||||
|
||||
- `kling_tts` 存在并接入 `tts_selector`。
|
||||
- `kling_audio` 只有在能映射到现有 capability 时才存在；否则验收记录为“暂不接入，避免新增功能面”。
|
||||
- `kling_avatar` 存在，并与本地 `talking_head.py` 并存；除非现有 pipeline 显式加入它，否则不要求 pipeline 自动消费。
|
||||
- `kling_lip_sync` 存在，并保留 face 人工选择出口；除非现有 pipeline 显式加入它，否则不要求 pipeline 自动消费。
|
||||
- `kling_effects` 默认不存在；若存在，必须证明不会被普通视频生成误选，并说明对应 pipeline 消费路径。
|
||||
- 每个新增工具都声明 `provider="kling_official"`。
|
||||
- 每个新增工具都声明 `dependencies = ["env:KLING_API_KEY"]`。
|
||||
- 每个新增工具都包含 `agent_skills = ["kling-official", ...]` 或等价配置。
|
||||
- 每个新增工具都实现非默认 `estimate_cost()`。
|
||||
- 每个新增 paid 工具的成功 ToolResult 都写入 `cost_usd`。
|
||||
- 每个新增工具都有 contract 测试。
|
||||
- 真实 API 测试默认不进 CI。
|
||||
- 文档、skill、provider table 已更新。
|
||||
- 阶段 1 和阶段 2 的测试仍通过。
|
||||
|
||||
## 12. 全项目完成标准
|
||||
|
||||
三阶段全部完成后，OpenMontage 的可灵官方 API 集成应达到：
|
||||
|
||||
- 官方可灵每个主要能力族都有明确实现或不接入理由。
|
||||
- 视频、图像和已接入现有 OpenMontage capability 的 TTS、数字人、口型等能力有实现和测试；未接入端点有清楚的不接入理由。
|
||||
- selector 或现有 stage routing 不会在不适合的 operation 中误选特效、口型、音效工具。
|
||||
- 付费调用默认不进 CI，live QA 需要显式环境变量开启。
|
||||
- 所有新增工具遵守 OpenMontage 的 BaseTool 契约、artifact 路径和项目目录约定。
|
||||
- fal.ai Kling 和 official Kling 在 provider 命名、文档、skill 和 selector 使用上清楚分离。
|
||||
- 未新增 pipeline、stage、canonical artifact 或未设计的新 capability。
|
||||
|
|
@ -0,0 +1,570 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Kling Official animated-explainer E2E smoke script.
|
||||
|
||||
This script validates the official Kling provider path through OpenMontage
|
||||
selectors and the animated-explainer asset/compose surface.
|
||||
|
||||
Default mode is a no-cost dry run. Use --live-tts for one paid TTS sample, or
|
||||
--live-full for TTS + image + image-to-video + local FFmpeg compose.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, Mapping, Sequence
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
if str(ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
|
||||
from lib.env_loader import load_env # noqa: E402
|
||||
|
||||
load_env(ROOT)
|
||||
|
||||
from lib.pipeline_loader import load_pipeline # noqa: E402
|
||||
from tools.tool_registry import registry # noqa: E402
|
||||
from tools.video.video_compose import VideoCompose # noqa: E402
|
||||
|
||||
|
||||
DEFAULT_PROJECT = "kling-animated-explainer-e2e"
|
||||
DEFAULT_VOICE_ID = "oversea_male1"
|
||||
DEFAULT_TTS_TEXT = (
|
||||
"Throughout my time in college, several memorable events left a significant impact on my life."
|
||||
)
|
||||
REQUIRED_LIVE_ENV = ("KLING_API_KEY",)
|
||||
RELEVANT_ENV = ("KLING_API_KEY", "KLING_API_BASE_URL", "FAL_KEY", "OPENAI_API_KEY")
|
||||
|
||||
|
||||
def _json_safe(value: Any) -> Any:
|
||||
if isinstance(value, Path):
|
||||
return str(value)
|
||||
if isinstance(value, dict):
|
||||
return {str(k): _json_safe(v) for k, v in value.items()}
|
||||
if isinstance(value, list):
|
||||
return [_json_safe(v) for v in value]
|
||||
return value
|
||||
|
||||
|
||||
def _write_json(path: Path, data: dict[str, Any]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(_json_safe(data), indent=2, ensure_ascii=False), encoding="utf-8")
|
||||
|
||||
|
||||
def _probe_media(path: Path) -> dict[str, Any]:
|
||||
cmd = [
|
||||
"ffprobe",
|
||||
"-v",
|
||||
"error",
|
||||
"-show_entries",
|
||||
"stream=codec_type,codec_name,width,height,sample_rate,channels,duration",
|
||||
"-show_entries",
|
||||
"format=format_name,duration,size,bit_rate",
|
||||
"-of",
|
||||
"json",
|
||||
str(path),
|
||||
]
|
||||
try:
|
||||
completed = subprocess.run(cmd, capture_output=True, text=True, check=True)
|
||||
return json.loads(completed.stdout or "{}")
|
||||
except Exception as exc:
|
||||
return {"error": str(exc)}
|
||||
|
||||
|
||||
def _env_status(environ: Mapping[str, str | None] | None = None) -> dict[str, dict[str, Any]]:
|
||||
source = environ if environ is not None else os.environ
|
||||
status: dict[str, dict[str, Any]] = {}
|
||||
for key in RELEVANT_ENV:
|
||||
value = source.get(key)
|
||||
present = bool(value)
|
||||
if not present:
|
||||
display = "<missing>"
|
||||
elif key == "KLING_API_BASE_URL":
|
||||
display = str(value)
|
||||
else:
|
||||
display = f"<set:{len(str(value))} chars>"
|
||||
status[key] = {"present": present, "length": len(str(value or "")), "display": display}
|
||||
return status
|
||||
|
||||
|
||||
def _missing_required_env() -> list[str]:
|
||||
return [key for key in REQUIRED_LIVE_ENV if not os.environ.get(key)]
|
||||
|
||||
|
||||
def _discover() -> None:
|
||||
registry.clear()
|
||||
registry.discover("tools")
|
||||
|
||||
|
||||
def _tool_statuses() -> dict[str, str]:
|
||||
names = [
|
||||
"tts_selector",
|
||||
"image_selector",
|
||||
"video_selector",
|
||||
"video_compose",
|
||||
"kling_tts",
|
||||
"kling_official_image",
|
||||
"kling_official_video",
|
||||
]
|
||||
statuses: dict[str, str] = {}
|
||||
for name in names:
|
||||
tool = registry.get(name)
|
||||
statuses[name] = tool.get_status().value if tool else "missing"
|
||||
return statuses
|
||||
|
||||
|
||||
def _capability_summary() -> dict[str, Any]:
|
||||
summary = registry.provider_menu_summary()
|
||||
wanted = {"tts", "image_generation", "video_generation", "video_post"}
|
||||
return {
|
||||
"composition_runtimes": summary.get("composition_runtimes", {}),
|
||||
"capabilities": [
|
||||
item for item in summary.get("capabilities", []) if item.get("capability") in wanted
|
||||
],
|
||||
"runtime_warnings": summary.get("runtime_warnings", []),
|
||||
}
|
||||
|
||||
|
||||
def _kling_entry(rank_result: dict[str, Any]) -> dict[str, Any] | None:
|
||||
for item in rank_result.get("rankings", []):
|
||||
if item.get("provider") == "kling_official":
|
||||
return item
|
||||
return None
|
||||
|
||||
|
||||
def _rank_selectors(voice_id: str, voice_language: str, voice_speed: float) -> dict[str, Any]:
|
||||
tts = registry.get("tts_selector")
|
||||
image = registry.get("image_selector")
|
||||
video = registry.get("video_selector")
|
||||
assert tts and image and video
|
||||
|
||||
tts_rank = tts.execute(
|
||||
{
|
||||
"operation": "rank",
|
||||
"allowed_providers": ["kling_official"],
|
||||
"text": "Kling official TTS selector smoke test.",
|
||||
"voice_id": voice_id,
|
||||
"voice_language": voice_language,
|
||||
"voice_speed": voice_speed,
|
||||
}
|
||||
).data
|
||||
image_rank = image.execute(
|
||||
{
|
||||
"operation": "rank",
|
||||
"allowed_providers": ["kling_official"],
|
||||
"prompt": "Clean minimal explainer visual about AI video production.",
|
||||
"api_family": "generation",
|
||||
"model_name": "kling-v3",
|
||||
}
|
||||
).data
|
||||
video_rank = video.execute(
|
||||
{
|
||||
"operation": "rank",
|
||||
"target_operation": "image_to_video",
|
||||
"allowed_providers": ["kling_official"],
|
||||
"prompt": "Slow camera push over a clean explainer visual.",
|
||||
"api_family": "classic",
|
||||
"model_name": "kling-v3",
|
||||
"duration": "3",
|
||||
"mode": "std",
|
||||
"sound": "off",
|
||||
}
|
||||
).data
|
||||
|
||||
return {
|
||||
"note": (
|
||||
"Rank mode is advisory. Live modes use preferred_provider and "
|
||||
"allowed_providers to force kling_official selection."
|
||||
),
|
||||
"kling_official_entries": {
|
||||
"tts": _kling_entry(tts_rank),
|
||||
"image": _kling_entry(image_rank),
|
||||
"video": _kling_entry(video_rank),
|
||||
},
|
||||
"tts_rank_all": tts_rank,
|
||||
"image_rank_all": image_rank,
|
||||
"video_rank_all": video_rank,
|
||||
}
|
||||
|
||||
|
||||
def _dry_run(voice_id: str, voice_language: str, voice_speed: float, text: str) -> dict[str, Any]:
|
||||
dry: dict[str, Any] = {}
|
||||
cases = {
|
||||
"kling_tts": {
|
||||
"text": text,
|
||||
"voice_id": voice_id,
|
||||
"voice_language": voice_language,
|
||||
"voice_speed": voice_speed,
|
||||
},
|
||||
"kling_official_image": {
|
||||
"prompt": "Clean minimal explainer visual about AI video production.",
|
||||
"api_family": "generation",
|
||||
"model_name": "kling-v3",
|
||||
"resolution": "1k",
|
||||
"aspect_ratio": "16:9",
|
||||
"n": 1,
|
||||
},
|
||||
"kling_official_video": {
|
||||
"prompt": "Slow camera push over a clean explainer visual.",
|
||||
"operation": "image_to_video",
|
||||
"api_family": "classic",
|
||||
"model_name": "kling-v3",
|
||||
"duration": "3",
|
||||
"mode": "std",
|
||||
"sound": "off",
|
||||
},
|
||||
}
|
||||
for name, payload in cases.items():
|
||||
tool = registry.get(name)
|
||||
dry[name] = tool.dry_run(payload) if tool else {"status": "missing"}
|
||||
return dry
|
||||
|
||||
|
||||
def _require_success(name: str, result: Any) -> None:
|
||||
if not result.success:
|
||||
raise RuntimeError(f"{name} failed: {result.error}")
|
||||
|
||||
|
||||
def _aligned_video_duration(requested_duration: str, narration_seconds: float | None) -> str:
|
||||
requested = int(requested_duration)
|
||||
if narration_seconds:
|
||||
requested = max(requested, int(math.ceil(narration_seconds)))
|
||||
return str(min(max(requested, 3), 15))
|
||||
|
||||
|
||||
def _announce_paid_call(tool: str, provider: str, model: str, reason: str, run_type: str) -> None:
|
||||
print(f"[paid:{run_type}] tool={tool} provider={provider} model={model}")
|
||||
print(f"[paid:{run_type}] reason={reason}")
|
||||
|
||||
|
||||
def _run_live_tts(
|
||||
project_dir: Path,
|
||||
*,
|
||||
voice_id: str,
|
||||
voice_language: str,
|
||||
voice_speed: float,
|
||||
text: str,
|
||||
timeout_seconds: int,
|
||||
poll_interval: float,
|
||||
include_account_usage: bool,
|
||||
) -> dict[str, Any]:
|
||||
tts = registry.get("tts_selector")
|
||||
assert tts
|
||||
|
||||
audio_dir = project_dir / "assets" / "audio"
|
||||
audio_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
_announce_paid_call(
|
||||
"tts_selector -> kling_tts",
|
||||
"kling_official",
|
||||
"kling-official-tts",
|
||||
"Validate official Kling TTS through OpenMontage selector routing.",
|
||||
"sample",
|
||||
)
|
||||
result = tts.execute(
|
||||
{
|
||||
"preferred_provider": "kling_official",
|
||||
"allowed_providers": ["kling_official"],
|
||||
"text": text,
|
||||
"voice_id": voice_id,
|
||||
"voice_language": voice_language,
|
||||
"voice_speed": voice_speed,
|
||||
"sample_mode": True,
|
||||
"include_account_usage": include_account_usage,
|
||||
"timeout_seconds": timeout_seconds,
|
||||
"poll_interval": poll_interval,
|
||||
"output_path": str(audio_dir / "narration.mp3"),
|
||||
}
|
||||
)
|
||||
_require_success("tts_selector", result)
|
||||
output_path = Path(result.data["output_path"])
|
||||
return {
|
||||
"result": result.data,
|
||||
"artifacts": {"narration": str(output_path)},
|
||||
"ffprobe": _probe_media(output_path),
|
||||
"estimated_cost_usd": float(result.cost_usd or 0),
|
||||
}
|
||||
|
||||
|
||||
def _run_live_full(
|
||||
project_dir: Path,
|
||||
*,
|
||||
voice_id: str,
|
||||
voice_language: str,
|
||||
voice_speed: float,
|
||||
text: str,
|
||||
timeout_seconds: int,
|
||||
poll_interval: float,
|
||||
include_account_usage: bool,
|
||||
video_duration: str,
|
||||
) -> dict[str, Any]:
|
||||
image = registry.get("image_selector")
|
||||
video = registry.get("video_selector")
|
||||
assert image and video
|
||||
|
||||
assets = project_dir / "assets"
|
||||
image_dir = assets / "images"
|
||||
video_dir = assets / "video"
|
||||
renders_dir = project_dir / "renders"
|
||||
for path in (image_dir, video_dir, renders_dir):
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
tts_data = _run_live_tts(
|
||||
project_dir,
|
||||
voice_id=voice_id,
|
||||
voice_language=voice_language,
|
||||
voice_speed=voice_speed,
|
||||
text=text,
|
||||
timeout_seconds=timeout_seconds,
|
||||
poll_interval=poll_interval,
|
||||
include_account_usage=include_account_usage,
|
||||
)
|
||||
narration_path = Path(tts_data["artifacts"]["narration"])
|
||||
target_video_duration = _aligned_video_duration(
|
||||
video_duration,
|
||||
tts_data.get("result", {}).get("audio_duration_seconds"),
|
||||
)
|
||||
|
||||
_announce_paid_call(
|
||||
"image_selector -> kling_official_image",
|
||||
"kling_official",
|
||||
"kling-v3",
|
||||
"Generate one reference frame for the animated-explainer E2E smoke.",
|
||||
"sample",
|
||||
)
|
||||
image_result = image.execute(
|
||||
{
|
||||
"preferred_provider": "kling_official",
|
||||
"allowed_providers": ["kling_official"],
|
||||
"prompt": (
|
||||
"A clean 16:9 animated-explainer hero frame: a luminous production "
|
||||
"pipeline diagram on a dark desk, small cards labeled script, voice, "
|
||||
"image, video, render, realistic yet crisp, no text artifacts."
|
||||
),
|
||||
"negative_prompt": "blurry, unreadable text, distorted interface, watermark",
|
||||
"api_family": "generation",
|
||||
"model_name": "kling-v3",
|
||||
"resolution": "1k",
|
||||
"aspect_ratio": "16:9",
|
||||
"n": 1,
|
||||
"output_path": str(image_dir / "hero_frame.png"),
|
||||
}
|
||||
)
|
||||
_require_success("image_selector", image_result)
|
||||
image_path = Path(image_result.data["output_path"])
|
||||
|
||||
_announce_paid_call(
|
||||
"video_selector -> kling_official_video",
|
||||
"kling_official",
|
||||
"kling-v3 classic image_to_video",
|
||||
"Animate the generated reference frame for a minimal provider E2E smoke.",
|
||||
"sample",
|
||||
)
|
||||
video_result = video.execute(
|
||||
{
|
||||
"preferred_provider": "kling_official",
|
||||
"allowed_providers": ["kling_official"],
|
||||
"prompt": (
|
||||
"A slow cinematic push-in over the explainer pipeline diagram. "
|
||||
"Cards glow softly in sequence, subtle parallax, stable camera, smooth motion."
|
||||
),
|
||||
"operation": "image_to_video",
|
||||
"api_family": "classic",
|
||||
"model_name": "kling-v3",
|
||||
"reference_image_path": str(image_path),
|
||||
"duration": target_video_duration,
|
||||
"mode": "std",
|
||||
"sound": "off",
|
||||
"output_path": str(video_dir / "kling_i2v_clip.mp4"),
|
||||
"timeout_seconds": max(timeout_seconds, 900),
|
||||
"poll_interval": poll_interval,
|
||||
}
|
||||
)
|
||||
_require_success("video_selector", video_result)
|
||||
clip_path = Path(video_result.data["output_path"])
|
||||
|
||||
print("[local] tool=video_compose runtime=ffmpeg reason=minimal one-clip provider smoke")
|
||||
edit_decisions = {
|
||||
"version": "1.0",
|
||||
"render_runtime": "ffmpeg",
|
||||
"renderer_family": "video_concat_smoke",
|
||||
"cuts": [
|
||||
{
|
||||
"id": "cut-001",
|
||||
"source": str(clip_path),
|
||||
"in_seconds": 0,
|
||||
"out_seconds": float(target_video_duration),
|
||||
"speed": 1.0,
|
||||
}
|
||||
],
|
||||
"subtitles": {"enabled": False},
|
||||
"metadata": {
|
||||
"pipeline": "animated-explainer",
|
||||
"compose_target": {"width": 1280, "height": 720, "fit": "pad"},
|
||||
"provider_smoke": True,
|
||||
"approved_runtime_reason": "Minimal provider integration smoke uses ffmpeg compose for one clip.",
|
||||
},
|
||||
}
|
||||
compose_result = VideoCompose().execute(
|
||||
{
|
||||
"operation": "compose",
|
||||
"edit_decisions": edit_decisions,
|
||||
"audio_path": str(narration_path),
|
||||
"output_path": str(renders_dir / "final_kling_e2e_smoke.mp4"),
|
||||
"profile": "youtube_landscape",
|
||||
"crf": 23,
|
||||
"preset": "medium",
|
||||
}
|
||||
)
|
||||
_require_success("video_compose", compose_result)
|
||||
final_path = Path(compose_result.data["output"])
|
||||
|
||||
return {
|
||||
"tts": tts_data,
|
||||
"image": image_result.data,
|
||||
"video": video_result.data,
|
||||
"compose": compose_result.data,
|
||||
"requested_video_duration": video_duration,
|
||||
"aligned_video_duration": target_video_duration,
|
||||
"artifacts": {
|
||||
"narration": str(narration_path),
|
||||
"image": str(image_path),
|
||||
"clip": str(clip_path),
|
||||
"final": str(final_path),
|
||||
},
|
||||
"ffprobe": {
|
||||
"narration": _probe_media(narration_path),
|
||||
"image": _probe_media(image_path),
|
||||
"clip": _probe_media(clip_path),
|
||||
"final": _probe_media(final_path),
|
||||
},
|
||||
"estimated_cost_usd": sum(
|
||||
float(getattr(result, "cost_usd", 0) or 0)
|
||||
for result in (image_result, video_result)
|
||||
)
|
||||
+ float(tts_data.get("estimated_cost_usd") or 0),
|
||||
}
|
||||
|
||||
|
||||
def _parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
live = parser.add_mutually_exclusive_group()
|
||||
live.add_argument("--live-tts", action="store_true", help="Run one paid Kling TTS sample.")
|
||||
live.add_argument(
|
||||
"--live",
|
||||
"--live-full",
|
||||
dest="live_full",
|
||||
action="store_true",
|
||||
help="Run paid Kling TTS, image, video, and local compose.",
|
||||
)
|
||||
parser.add_argument("--voice-id", default=DEFAULT_VOICE_ID)
|
||||
parser.add_argument("--voice-language", choices=["en", "zh"], default="en")
|
||||
parser.add_argument("--voice-speed", type=float, default=1.0)
|
||||
parser.add_argument("--text", default=DEFAULT_TTS_TEXT)
|
||||
parser.add_argument("--video-duration", choices=[str(v) for v in range(3, 16)], default="3")
|
||||
parser.add_argument("--timeout-seconds", type=int, default=300)
|
||||
parser.add_argument("--poll-interval", type=float, default=3.0)
|
||||
parser.add_argument("--include-account-usage", action="store_true")
|
||||
parser.add_argument("--project", default=DEFAULT_PROJECT)
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def _execution_mode(args: argparse.Namespace) -> str:
|
||||
if getattr(args, "live_tts", False):
|
||||
return "live_tts"
|
||||
if getattr(args, "live_full", False):
|
||||
return "live_full"
|
||||
return "dry_run"
|
||||
|
||||
|
||||
def main(argv: Sequence[str] | None = None) -> int:
|
||||
args = _parse_args(argv)
|
||||
mode = _execution_mode(args)
|
||||
project_dir = ROOT / "projects" / args.project
|
||||
report_path = project_dir / "artifacts" / "kling_official_animated_explainer_e2e_report.json"
|
||||
|
||||
manifest = load_pipeline("animated-explainer")
|
||||
_discover()
|
||||
|
||||
report: dict[str, Any] = {
|
||||
"timestamp": datetime.now(timezone.utc).isoformat(),
|
||||
"pipeline": manifest["name"],
|
||||
"purpose": "Kling official selector-level E2E smoke for animated-explainer assets + compose.",
|
||||
"mode": mode,
|
||||
"voice_id": args.voice_id,
|
||||
"voice_language": args.voice_language,
|
||||
"voice_speed": args.voice_speed,
|
||||
"project_dir": str(project_dir),
|
||||
"env_status": _env_status(),
|
||||
"tool_statuses": _tool_statuses(),
|
||||
"capability_summary": _capability_summary(),
|
||||
"selector_rankings": _rank_selectors(args.voice_id, args.voice_language, args.voice_speed),
|
||||
"dry_run": _dry_run(args.voice_id, args.voice_language, args.voice_speed, args.text),
|
||||
}
|
||||
|
||||
missing_env = _missing_required_env()
|
||||
if mode != "dry_run" and missing_env:
|
||||
report["blocked"] = {
|
||||
"reason": "missing required live environment variables",
|
||||
"missing_env": missing_env,
|
||||
}
|
||||
_write_json(report_path, report)
|
||||
print(f"blocked: missing required live env vars: {', '.join(missing_env)}")
|
||||
print(f"report: {report_path}")
|
||||
return 2
|
||||
|
||||
try:
|
||||
if mode == "live_tts":
|
||||
report["live_tts_result"] = _run_live_tts(
|
||||
project_dir,
|
||||
voice_id=args.voice_id,
|
||||
voice_language=args.voice_language,
|
||||
voice_speed=args.voice_speed,
|
||||
text=args.text,
|
||||
timeout_seconds=args.timeout_seconds,
|
||||
poll_interval=args.poll_interval,
|
||||
include_account_usage=args.include_account_usage,
|
||||
)
|
||||
elif mode == "live_full":
|
||||
report["live_full_result"] = _run_live_full(
|
||||
project_dir,
|
||||
voice_id=args.voice_id,
|
||||
voice_language=args.voice_language,
|
||||
voice_speed=args.voice_speed,
|
||||
text=args.text,
|
||||
timeout_seconds=args.timeout_seconds,
|
||||
poll_interval=args.poll_interval,
|
||||
include_account_usage=args.include_account_usage,
|
||||
video_duration=args.video_duration,
|
||||
)
|
||||
else:
|
||||
report["next_steps"] = [
|
||||
"Run with --live-tts to make one paid Kling TTS sample call.",
|
||||
"Run with --live-full to make paid Kling TTS/image/video calls and compose final_kling_e2e_smoke.mp4.",
|
||||
]
|
||||
except Exception as exc:
|
||||
report["failed"] = {"error": str(exc)}
|
||||
_write_json(report_path, report)
|
||||
print(f"failed: {exc}")
|
||||
print(f"report: {report_path}")
|
||||
return 1
|
||||
|
||||
_write_json(report_path, report)
|
||||
print(f"report: {report_path}")
|
||||
if mode == "live_tts":
|
||||
print(f"narration: {report['live_tts_result']['artifacts']['narration']}")
|
||||
elif mode == "live_full":
|
||||
print(f"final: {report['live_full_result']['artifacts']['final']}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
|
@ -312,5 +312,6 @@ Claude Code accesses them via symlinks in `.claude/skills/`.
|
|||
| **Animation** | `framer-motion`, `lottie-bodymovin` | `pproenca/dot-skills`, `dylantarre/animation-principles` |
|
||||
| **Design** | `tailwind-design-system`, `web-design-guidelines`, `vercel-react-best-practices`, `vercel-composition-patterns` | `wshobson/agents`, `vercel-labs/agent-skills` |
|
||||
| **AI Video (HeyGen)** | `heygen`, `avatar-video`, `create-video`, `faceswap`, `ai-video-gen`, `video-download`, `video-edit`, `video-translate`, `video-understand`, `visual-style` | `heygen-com/skills` |
|
||||
| **AI Video/Image/TTS/Avatar (Kling Official)** | `kling-official` - official direct API auth, Classic/Turbo/Omni task protocols, multi-reference Omni syntax, internal Elements/Account Usage helpers, callback notes, TTS voice parameters, avatar/lip-sync face selection, error handling, and cost governance for `kling_official_video` / `kling_official_image` / `kling_tts` / `kling_avatar` / `kling_lip_sync` | Local OpenMontage skill |
|
||||
| **AI Video (Premium)** | `seedance-2-0` — preferred premium default (cinematic, trailer, multi-shot, lip-sync, synced audio); accessed via `seedance_video` (fal.ai) or `heygen_video` Avatar Shots | Local OpenMontage skill |
|
||||
| **Infrastructure** | `acestep`, `ltx2`, `playwright-recording` | `digitalsamba/claude-code-video-toolkit` |
|
||||
|
|
|
|||
|
|
@ -22,6 +22,7 @@ For model-specific tips, see the linked guides below.
|
|||
| **HunyuanVideo 1.5** | [Tencent Prompt Handbook](https://github.com/Tencent-Hunyuan/HunyuanVideo-1.5/blob/main/assets/HunyuanVideo_1_5_Prompt_Handbook_EN.md) | Formula: Subject + Motion + Scene + [Shot] + [Camera] + [Lighting] + [Style] + [Atmosphere]. |
|
||||
| **Runway Gen-4** | [Runway Prompting Guide](https://help.runwayml.com/hc/en-us/articles/39789879462419-Gen-4-Video-Prompting-Guide) | "Focus on motion, not appearance." One scene per clip. Simplicity wins. |
|
||||
| **Kling 2.6** | [Kling Prompt Guide](https://fal.ai/learn/devs/kling-2-6-pro-prompt-guide) | 4-part structure. Supports `++emphasis++` syntax for key elements. |
|
||||
| **Kling Official** | Layer 3 `.agents/skills/kling-official/` | Direct official API. Use `provider="kling_official"` to distinguish it from fal.ai Kling. `api_family` selects Classic, Turbo, or Omni; Turbo image-to-video needs a URL reference image. |
|
||||
| **Wan 2.1 / CogVideoX** | Use this generic guide | No official prompt guide. Standard cinematographic vocabulary works well. |
|
||||
|
||||
## Order Matters
|
||||
|
|
|
|||
|
|
@ -0,0 +1,120 @@
|
|||
"""Contract tests for the Kling official avatar provider."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
from tools.avatar.kling_avatar import KlingAvatar
|
||||
from tools.avatar.talking_head import TalkingHead
|
||||
from tools.tool_registry import registry
|
||||
|
||||
|
||||
def test_registry_discovers_kling_avatar(monkeypatch):
|
||||
monkeypatch.delenv("KLING_API_KEY", raising=False)
|
||||
registry.clear()
|
||||
registry.discover("tools")
|
||||
tool = registry.get("kling_avatar")
|
||||
assert tool is not None
|
||||
assert tool.capability == "avatar"
|
||||
assert tool.provider == "kling_official"
|
||||
|
||||
|
||||
def test_avatar_schema_and_local_tool_are_distinct():
|
||||
tool = KlingAvatar()
|
||||
assert "anyOf" in tool.input_schema
|
||||
assert "allOf" in tool.input_schema
|
||||
assert "kling-official" in tool.agent_skills
|
||||
assert "avatar-video" in tool.agent_skills
|
||||
assert tool.runtime.value == "api"
|
||||
assert TalkingHead().provider == "sadtalker"
|
||||
assert TalkingHead().runtime.value == "local_gpu"
|
||||
|
||||
|
||||
def test_avatar_payload_uses_image_and_audio_paths(tmp_path):
|
||||
image_path = tmp_path / "avatar.png"
|
||||
audio_path = tmp_path / "voice.mp3"
|
||||
image_path.write_bytes(b"image")
|
||||
audio_path.write_bytes(b"audio")
|
||||
|
||||
request = KlingAvatar()._build_request(
|
||||
{
|
||||
"image_path": str(image_path),
|
||||
"audio_path": str(audio_path),
|
||||
"prompt": "warm presenter, subtle head motion",
|
||||
"mode": "pro",
|
||||
"callback_url": "https://example.com/kling/callback",
|
||||
}
|
||||
)
|
||||
|
||||
assert request["path"] == "/v1/videos/avatar/image2video"
|
||||
assert request["payload"]["image"] == base64.b64encode(b"image").decode("ascii")
|
||||
assert request["payload"]["sound_file"] == base64.b64encode(b"audio").decode("ascii")
|
||||
assert request["payload"]["mode"] == "pro"
|
||||
assert request["payload"]["callback_url"] == "https://example.com/kling/callback"
|
||||
assert request["audio_source"]["type"] == "sound_file"
|
||||
|
||||
|
||||
def test_avatar_requires_image_and_audio():
|
||||
tool = KlingAvatar()
|
||||
try:
|
||||
tool._build_request({"audio_id": "audio-a"})
|
||||
except ValueError as exc:
|
||||
assert "image_url or image_path" in str(exc)
|
||||
else:
|
||||
raise AssertionError("Kling avatar must require an image")
|
||||
|
||||
try:
|
||||
tool._build_request({"image_url": "https://example.com/avatar.png"})
|
||||
except ValueError as exc:
|
||||
assert "requires audio_id" in str(exc)
|
||||
else:
|
||||
raise AssertionError("Kling avatar must require audio input")
|
||||
|
||||
|
||||
def test_execute_downloads_avatar_video(monkeypatch, tmp_path):
|
||||
class FakeClient:
|
||||
def create_classic_task(self, path, payload):
|
||||
self.path = path
|
||||
self.payload = payload
|
||||
return "avatar-task-1"
|
||||
|
||||
def poll_classic(self, path, task_id, result_key, timeout_seconds, poll_interval):
|
||||
assert result_key == "videos"
|
||||
return [{"url": "https://example.com/avatar.mp4"}]
|
||||
|
||||
def download(self, url, output_path):
|
||||
output_path.write_bytes(b"video")
|
||||
return output_path
|
||||
|
||||
monkeypatch.setenv("KLING_API_KEY", "test-key")
|
||||
monkeypatch.setattr("tools.avatar.kling_avatar.KlingClient", lambda: FakeClient())
|
||||
monkeypatch.setattr("tools.avatar.kling_avatar.probe_output", lambda path: {"duration_seconds": 5.0})
|
||||
|
||||
result = KlingAvatar().execute(
|
||||
{
|
||||
"image_url": "https://example.com/avatar.png",
|
||||
"audio_id": "audio-a",
|
||||
"output_path": str(tmp_path / "avatar.mp4"),
|
||||
}
|
||||
)
|
||||
|
||||
assert result.success
|
||||
assert result.data["provider"] == "kling_official"
|
||||
assert result.data["task_id"] == "avatar-task-1"
|
||||
assert result.data["duration_seconds"] == 5.0
|
||||
assert Path(result.artifacts[0]).read_bytes() == b"video"
|
||||
assert result.cost_usd > 0
|
||||
|
||||
|
||||
def test_avatar_cost_estimate_is_not_zero():
|
||||
tool = KlingAvatar()
|
||||
base = tool.estimate_cost({"image_url": "x", "audio_id": "a"})
|
||||
pro = tool.estimate_cost({"image_url": "x", "audio_id": "a", "mode": "pro"})
|
||||
assert base > 0
|
||||
assert pro > base
|
||||
assert tool.dry_run({"image_url": "x", "audio_id": "a"})["cost_estimate_confidence"] == "low"
|
||||
|
|
@ -0,0 +1,233 @@
|
|||
"""Contract tests for the Kling official lip-sync provider."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
from tools.avatar.kling_lip_sync import KlingLipSync
|
||||
from tools.avatar.lip_sync import LipSync
|
||||
from tools.tool_registry import registry
|
||||
|
||||
|
||||
def test_registry_discovers_kling_lip_sync(monkeypatch):
|
||||
monkeypatch.delenv("KLING_API_KEY", raising=False)
|
||||
registry.clear()
|
||||
registry.discover("tools")
|
||||
tool = registry.get("kling_lip_sync")
|
||||
assert tool is not None
|
||||
assert tool.capability == "avatar"
|
||||
assert tool.provider == "kling_official"
|
||||
|
||||
|
||||
def test_lip_sync_schema_and_local_tool_are_distinct():
|
||||
tool = KlingLipSync()
|
||||
assert "kling-official" in tool.agent_skills
|
||||
assert "avatar-video" in tool.agent_skills
|
||||
assert tool.runtime.value == "api"
|
||||
assert LipSync().provider == "wav2lip"
|
||||
assert LipSync().runtime.value == "local_gpu"
|
||||
|
||||
|
||||
def test_identify_face_payload_and_local_video_rejection():
|
||||
tool = KlingLipSync()
|
||||
request = tool._build_identify_request({"video_url": "https://example.com/source.mp4"})
|
||||
assert request["path"] == "/v1/videos/identify-face"
|
||||
assert request["payload"] == {"video_url": "https://example.com/source.mp4"}
|
||||
|
||||
try:
|
||||
tool._build_identify_request({"video_path": "/tmp/local.mp4"})
|
||||
except ValueError as exc:
|
||||
assert "local video paths cannot be silently uploaded" in str(exc)
|
||||
else:
|
||||
raise AssertionError("Local video paths must not be silently uploaded")
|
||||
|
||||
|
||||
def test_advanced_lip_sync_payload_uses_face_and_audio_path(tmp_path):
|
||||
audio_path = tmp_path / "voice.mp3"
|
||||
audio_path.write_bytes(b"audio")
|
||||
|
||||
request = KlingLipSync()._build_advanced_request(
|
||||
{
|
||||
"session_id": "session-a",
|
||||
"face_id": "face-a",
|
||||
"audio_path": str(audio_path),
|
||||
"callback_url": "https://example.com/kling/callback",
|
||||
}
|
||||
)
|
||||
|
||||
assert request["path"] == "/v1/videos/advanced-lip-sync"
|
||||
assert request["payload"]["session_id"] == "session-a"
|
||||
assert request["payload"]["face_choose"] == [{"face_id": "face-a"}]
|
||||
assert request["payload"]["sound_file"] == base64.b64encode(b"audio").decode("ascii")
|
||||
assert request["payload"]["callback_url"] == "https://example.com/kling/callback"
|
||||
|
||||
|
||||
def test_identify_face_execute_writes_faces_artifact(monkeypatch, tmp_path):
|
||||
class FakeClient:
|
||||
def post(self, path, payload):
|
||||
assert path == "/v1/videos/identify-face"
|
||||
return {
|
||||
"code": 0,
|
||||
"data": {
|
||||
"session_id": "session-a",
|
||||
"faces": [{"face_id": "face-a", "bbox": [0, 0, 100, 100]}],
|
||||
},
|
||||
}
|
||||
|
||||
monkeypatch.setenv("KLING_API_KEY", "test-key")
|
||||
monkeypatch.setattr("tools.avatar.kling_lip_sync.KlingClient", lambda: FakeClient())
|
||||
|
||||
artifact_path = tmp_path / "faces.json"
|
||||
result = KlingLipSync().execute(
|
||||
{
|
||||
"operation": "identify_face",
|
||||
"video_url": "https://example.com/source.mp4",
|
||||
"faces_artifact_path": str(artifact_path),
|
||||
}
|
||||
)
|
||||
|
||||
assert result.success
|
||||
assert result.data["session_id"] == "session-a"
|
||||
data = json.loads(artifact_path.read_text())
|
||||
assert data["provider"] == "kling_official"
|
||||
assert data["face_count"] == 1
|
||||
|
||||
|
||||
def test_full_lip_sync_requires_confirmation_for_multiple_faces(monkeypatch, tmp_path):
|
||||
class FakeClient:
|
||||
def post(self, path, payload):
|
||||
return {
|
||||
"code": 0,
|
||||
"data": {
|
||||
"session_id": "session-a",
|
||||
"faces": [
|
||||
{"face_id": "face-small", "bbox": [0, 0, 50, 50]},
|
||||
{"face_id": "face-large", "bbox": [0, 0, 200, 200]},
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
monkeypatch.setenv("KLING_API_KEY", "test-key")
|
||||
monkeypatch.setattr("tools.avatar.kling_lip_sync.KlingClient", lambda: FakeClient())
|
||||
|
||||
result = KlingLipSync().execute(
|
||||
{
|
||||
"operation": "full_lip_sync",
|
||||
"video_url": "https://example.com/source.mp4",
|
||||
"audio_id": "audio-a",
|
||||
"faces_artifact_path": str(tmp_path / "faces.json"),
|
||||
}
|
||||
)
|
||||
|
||||
assert not result.success
|
||||
assert result.data["requires_face_selection"] is True
|
||||
assert len(result.data["faces"]) == 2
|
||||
assert "Multiple faces detected" in result.error
|
||||
assert Path(result.artifacts[0]).is_file()
|
||||
|
||||
|
||||
def test_full_lip_sync_auto_selects_largest_face_and_downloads(monkeypatch, tmp_path):
|
||||
class FakeClient:
|
||||
def post(self, path, payload):
|
||||
return {
|
||||
"code": 0,
|
||||
"data": {
|
||||
"session_id": "session-a",
|
||||
"faces": [
|
||||
{"face_id": "face-small", "bbox": [0, 0, 50, 50]},
|
||||
{"face_id": "face-large", "bbox": [0, 0, 200, 200]},
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
def create_classic_task(self, path, payload):
|
||||
self.path = path
|
||||
self.payload = payload
|
||||
assert payload["face_choose"] == [{"face_id": "face-large"}]
|
||||
return "lip-task-1"
|
||||
|
||||
def poll_classic(self, path, task_id, result_key, timeout_seconds, poll_interval):
|
||||
assert result_key == "videos"
|
||||
return [{"url": "https://example.com/lip.mp4"}]
|
||||
|
||||
def download(self, url, output_path):
|
||||
output_path.write_bytes(b"video")
|
||||
return output_path
|
||||
|
||||
monkeypatch.setenv("KLING_API_KEY", "test-key")
|
||||
monkeypatch.setattr("tools.avatar.kling_lip_sync.KlingClient", lambda: FakeClient())
|
||||
monkeypatch.setattr("tools.avatar.kling_lip_sync.probe_output", lambda path: {"duration_seconds": 4.0})
|
||||
|
||||
result = KlingLipSync().execute(
|
||||
{
|
||||
"operation": "full_lip_sync",
|
||||
"video_url": "https://example.com/source.mp4",
|
||||
"audio_id": "audio-a",
|
||||
"auto_select_face": True,
|
||||
"output_path": str(tmp_path / "lip.mp4"),
|
||||
}
|
||||
)
|
||||
|
||||
assert result.success
|
||||
assert result.data["task_id"] == "lip-task-1"
|
||||
assert result.data["face_selection"]["selection_method"] == "auto_selected"
|
||||
assert result.data["face_choose"] == [{"face_id": "face-large"}]
|
||||
assert Path(result.artifacts[0]).read_bytes() == b"video"
|
||||
faces_artifact = next(Path(path) for path in result.artifacts if Path(path).name == "kling_lip_sync_faces.json")
|
||||
assert json.loads(faces_artifact.read_text())["selection"]["selection_method"] == "auto_selected"
|
||||
assert result.cost_usd > 0
|
||||
|
||||
|
||||
def test_auto_select_face_area_avoids_position_inflation():
|
||||
tool = KlingLipSync()
|
||||
assert tool._face_area({"bbox": [10, 20, 110, 220]}) == 100 * 200
|
||||
assert tool._face_area({"bbox": [10, 20, 100, 200]}) == 90 * 180
|
||||
assert tool._face_area({"box": {"width": 80, "height": 90}}) == 80 * 90
|
||||
|
||||
|
||||
def test_advanced_lip_sync_execute_downloads_video(monkeypatch, tmp_path):
|
||||
class FakeClient:
|
||||
def create_classic_task(self, path, payload):
|
||||
assert path == "/v1/videos/advanced-lip-sync"
|
||||
assert payload["face_choose"] == [{"face_id": "face-a"}]
|
||||
return "lip-task-1"
|
||||
|
||||
def poll_classic(self, path, task_id, result_key, timeout_seconds, poll_interval):
|
||||
return [{"video_url": "https://example.com/lip.mp4"}]
|
||||
|
||||
def download(self, url, output_path):
|
||||
output_path.write_bytes(b"video")
|
||||
return output_path
|
||||
|
||||
monkeypatch.setenv("KLING_API_KEY", "test-key")
|
||||
monkeypatch.setattr("tools.avatar.kling_lip_sync.KlingClient", lambda: FakeClient())
|
||||
monkeypatch.setattr("tools.avatar.kling_lip_sync.probe_output", lambda path: {"duration_seconds": 4.0})
|
||||
|
||||
result = KlingLipSync().execute(
|
||||
{
|
||||
"operation": "advanced_lip_sync",
|
||||
"session_id": "session-a",
|
||||
"face_id": "face-a",
|
||||
"audio_id": "audio-a",
|
||||
"output_path": str(tmp_path / "lip.mp4"),
|
||||
}
|
||||
)
|
||||
|
||||
assert result.success
|
||||
assert result.data["provider"] == "kling_official"
|
||||
assert result.data["task_id"] == "lip-task-1"
|
||||
assert result.data["duration_seconds"] == 4.0
|
||||
assert result.cost_usd > 0
|
||||
|
||||
|
||||
def test_lip_sync_cost_estimate_is_not_zero():
|
||||
tool = KlingLipSync()
|
||||
assert tool.estimate_cost({"operation": "identify_face"}) > 0
|
||||
assert tool.estimate_cost({"operation": "advanced_lip_sync"}) > tool.estimate_cost({"operation": "identify_face"})
|
||||
assert tool.dry_run({"operation": "advanced_lip_sync"})["cost_estimate_confidence"] == "low"
|
||||
|
|
@ -0,0 +1,179 @@
|
|||
"""Contract tests for the Kling official shared client and schema snapshot."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
from tools._kling.client import KlingClient
|
||||
from tools._kling.errors import KlingAPIError, is_retryable_kling_error
|
||||
from tools._kling.schemas import DEFAULT_API_BASE_URL
|
||||
|
||||
|
||||
class FakeResponse:
|
||||
def __init__(self, data=None, status_code=200, content=b"data", text=""):
|
||||
self._data = data if data is not None else {"code": 0}
|
||||
self.status_code = status_code
|
||||
self.content = content
|
||||
self.text = text
|
||||
|
||||
def json(self):
|
||||
if isinstance(self._data, Exception):
|
||||
raise self._data
|
||||
return self._data
|
||||
|
||||
|
||||
class FakeSession:
|
||||
def __init__(self, responses):
|
||||
self.responses = list(responses)
|
||||
self.calls = []
|
||||
|
||||
def post(self, url, **kwargs):
|
||||
self.calls.append(("post", url, kwargs))
|
||||
return self.responses.pop(0)
|
||||
|
||||
def get(self, url, **kwargs):
|
||||
self.calls.append(("get", url, kwargs))
|
||||
return self.responses.pop(0)
|
||||
|
||||
|
||||
def test_missing_api_key_header_error(monkeypatch):
|
||||
monkeypatch.delenv("KLING_API_KEY", raising=False)
|
||||
client = KlingClient(session=FakeSession([]))
|
||||
with pytest.raises(KlingAPIError) as exc:
|
||||
_ = client.headers
|
||||
assert "KLING_API_KEY" in str(exc.value)
|
||||
|
||||
|
||||
def test_headers_use_bearer_api_key(monkeypatch):
|
||||
monkeypatch.setenv("KLING_API_KEY", "test-key")
|
||||
session = FakeSession([FakeResponse({"code": 0, "data": {"ok": True}})])
|
||||
client = KlingClient(session=session)
|
||||
client.post("/v1/test", {"prompt": "x"})
|
||||
headers = session.calls[0][2]["headers"]
|
||||
assert headers["Authorization"] == "Bearer test-key"
|
||||
assert headers["Content-Type"] == "application/json"
|
||||
|
||||
|
||||
def test_default_and_env_base_url(monkeypatch):
|
||||
monkeypatch.setenv("KLING_API_KEY", "test-key")
|
||||
monkeypatch.delenv("KLING_API_BASE_URL", raising=False)
|
||||
assert KlingClient().base_url == DEFAULT_API_BASE_URL
|
||||
monkeypatch.setenv("KLING_API_BASE_URL", "https://api-beijing.klingai.com")
|
||||
assert KlingClient().base_url == "https://api-beijing.klingai.com"
|
||||
|
||||
|
||||
def test_business_error_preserves_code_message_request_id(monkeypatch):
|
||||
monkeypatch.setenv("KLING_API_KEY", "test-key")
|
||||
session = FakeSession([FakeResponse({"code": 1200, "message": "bad parameter", "request_id": "req-1"})])
|
||||
client = KlingClient(session=session, max_retries=0)
|
||||
with pytest.raises(KlingAPIError) as exc:
|
||||
client.post("/v1/videos/text2video", {})
|
||||
assert exc.value.code == 1200
|
||||
assert exc.value.message == "bad parameter"
|
||||
assert exc.value.request_id == "req-1"
|
||||
|
||||
|
||||
def test_1303_retryable_message_mentions_concurrency(monkeypatch):
|
||||
monkeypatch.setenv("KLING_API_KEY", "test-key")
|
||||
session = FakeSession([FakeResponse({"code": 1303, "message": "parallel task over resource pack limit"})])
|
||||
client = KlingClient(session=session, max_retries=0)
|
||||
with pytest.raises(KlingAPIError) as exc:
|
||||
client.post("/v1/videos/text2video", {})
|
||||
assert is_retryable_kling_error(exc.value)
|
||||
assert "并发/资源包限制" in exc.value.message
|
||||
|
||||
|
||||
def test_classic_create_and_poll_parse_result_paths(monkeypatch):
|
||||
monkeypatch.setenv("KLING_API_KEY", "test-key")
|
||||
session = FakeSession(
|
||||
[
|
||||
FakeResponse({"code": 0, "data": {"task_id": "task-1"}}),
|
||||
FakeResponse(
|
||||
{
|
||||
"code": 0,
|
||||
"data": {
|
||||
"task_status": "succeed",
|
||||
"task_result": {"videos": [{"url": "https://example.com/out.mp4"}]},
|
||||
},
|
||||
}
|
||||
),
|
||||
]
|
||||
)
|
||||
client = KlingClient(session=session)
|
||||
task_id = client.create_classic_task("/v1/videos/text2video", {"prompt": "x"})
|
||||
outputs = client.poll_classic("/v1/videos/text2video", task_id, "videos")
|
||||
assert task_id == "task-1"
|
||||
assert outputs == [{"url": "https://example.com/out.mp4"}]
|
||||
|
||||
|
||||
def test_turbo_create_and_poll_parse_result_paths(monkeypatch):
|
||||
monkeypatch.setenv("KLING_API_KEY", "test-key")
|
||||
session = FakeSession(
|
||||
[
|
||||
FakeResponse({"code": 0, "data": {"id": "turbo-1"}}),
|
||||
FakeResponse(
|
||||
{
|
||||
"code": 0,
|
||||
"data": [
|
||||
{
|
||||
"id": "turbo-1",
|
||||
"status": "succeeded",
|
||||
"outputs": [{"url": "https://example.com/out.mp4"}],
|
||||
}
|
||||
],
|
||||
}
|
||||
),
|
||||
]
|
||||
)
|
||||
client = KlingClient(session=session)
|
||||
task_id = client.create_turbo("/text-to-video/kling-3.0-turbo", {"prompt": "x"})
|
||||
outputs = client.poll_turbo(task_id)
|
||||
assert task_id == "turbo-1"
|
||||
assert outputs == [{"url": "https://example.com/out.mp4"}]
|
||||
|
||||
|
||||
def test_schema_snapshot_contains_phase1_contract_fields():
|
||||
fixture = PROJECT_ROOT / "tests/fixtures/kling_official/schema_snapshot.json"
|
||||
data = json.loads(fixture.read_text())
|
||||
assert data["build_id"] == "97344324"
|
||||
assert "index-B9E4in0e.js" in data["chunk_names"]
|
||||
assert "document-navigation-nxVgwiS5.js" in data["chunk_names"]
|
||||
assert data["api_base"]["auth_env"] == "KLING_API_KEY"
|
||||
assert data["task_statuses"]["classic"] == ["submitted", "processing", "succeed", "failed"]
|
||||
assert data["task_statuses"]["turbo"] == ["submitted", "processing", "succeeded", "failed"]
|
||||
assert data["result_paths"]["classic_created_id"] == "data.task_id"
|
||||
assert data["result_paths"]["turbo_created_id"] == "data.id"
|
||||
assert "kling-v3" in data["models"]["video"]
|
||||
assert "kling-v3" in data["models"]["image"]
|
||||
assert data["endpoints"]["tts"]["path"] == "/v1/audio/tts"
|
||||
assert data["endpoints"]["avatar_image_to_video"]["path"] == "/v1/videos/avatar/image2video"
|
||||
assert data["endpoints"]["identify_face"]["path"] == "/v1/videos/identify-face"
|
||||
assert data["endpoints"]["advanced_lip_sync"]["path"] == "/v1/videos/advanced-lip-sync"
|
||||
assert data["endpoints"]["video_effects"]["path"] == "/v1/videos/effects"
|
||||
assert data["result_paths"]["classic_audio_results"] == "data.task_result.audios[]"
|
||||
assert data["result_paths"]["identify_face_session"] == "data.session_id"
|
||||
assert data["core_field_enums"]["tts_voice_language"] == ["zh", "en"]
|
||||
assert data["core_field_enums"]["avatar_mode"] == ["std", "pro"]
|
||||
|
||||
|
||||
def test_optional_live_doc_snapshot_check():
|
||||
if os.environ.get("RUN_KLING_DOC_LIVE_CHECK") != "1":
|
||||
pytest.skip("Set RUN_KLING_DOC_LIVE_CHECK=1 to compare fixture against current Kling docs HTML.")
|
||||
import re
|
||||
import urllib.request
|
||||
|
||||
fixture = PROJECT_ROOT / "tests/fixtures/kling_official/schema_snapshot.json"
|
||||
expected = json.loads(fixture.read_text())
|
||||
with urllib.request.urlopen("https://kling.ai/document-api/api/video/3-0-turbo/text-to-video", timeout=20) as response:
|
||||
html = response.read().decode("utf-8", errors="ignore")
|
||||
match = re.search(r'<meta name="buildId" content="([^"]+)"', html)
|
||||
assert match, "Kling official docs HTML no longer exposes buildId; refresh schema fixture."
|
||||
assert match.group(1) == expected["build_id"], "Kling official docs buildId changed; refresh schema fixture before implementation."
|
||||
|
|
@ -0,0 +1,88 @@
|
|||
"""Documentation and skill contract tests for Kling official integration."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
from tools.graphics.kling_official_image import KlingOfficialImage
|
||||
from tools.audio.kling_tts import KlingTTS
|
||||
from tools.avatar.kling_avatar import KlingAvatar
|
||||
from tools.avatar.kling_lip_sync import KlingLipSync
|
||||
from tools.video.kling_official_video import KlingOfficialVideo
|
||||
|
||||
|
||||
def read(path: str) -> str:
|
||||
return (PROJECT_ROOT / path).read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_env_example_documents_kling_official_keys():
|
||||
env = read(".env.example")
|
||||
assert "KLING_API_KEY=" in env
|
||||
assert "KLING_API_BASE_URL=" in env
|
||||
|
||||
|
||||
def test_provider_docs_distinguish_fal_and_official_kling():
|
||||
providers = read("docs/PROVIDERS.md")
|
||||
assert "Kling Official" in providers
|
||||
assert "kling_official_video" in providers
|
||||
assert "kling_official_image" in providers
|
||||
assert "kling_tts" in providers
|
||||
assert "kling_avatar" in providers
|
||||
assert "kling_lip_sync" in providers
|
||||
assert "fal.ai" in providers
|
||||
assert "provider=\"kling_official\"" in providers
|
||||
assert "provider=\"kling\"" in providers
|
||||
assert "Elements remain an internal Kling Official helper" in providers
|
||||
assert "Account Usage is available as a low-frequency diagnostic helper" in providers
|
||||
assert "callback_url" in providers
|
||||
assert "audio effects and video effects are documented but intentionally not registered" in providers
|
||||
|
||||
|
||||
def test_architecture_env_mapping_includes_kling_official():
|
||||
architecture = read("docs/ARCHITECTURE.md")
|
||||
assert "`KLING_API_KEY` | kling_official_video, kling_official_image, kling_tts, kling_avatar, kling_lip_sync" in architecture
|
||||
assert "`KLING_API_BASE_URL` | kling_official_video, kling_official_image, kling_tts, kling_avatar, kling_lip_sync" in architecture
|
||||
assert "Elements and Account" in architecture
|
||||
assert "not separate pipeline stages" in architecture
|
||||
assert "Kling Official Phase 3 adds provider tools only where OpenMontage already has a" in architecture
|
||||
|
||||
|
||||
def test_ai_video_skill_metadata_and_new_skill_link():
|
||||
ai_video = read(".agents/skills/ai-video-gen/SKILL.md")
|
||||
index = read("skills/INDEX.md")
|
||||
creative = read("skills/creative/video-gen-prompting.md")
|
||||
official_skill = PROJECT_ROOT / ".agents/skills/kling-official/SKILL.md"
|
||||
|
||||
assert "KLING_API_KEY" in ai_video
|
||||
assert "kling_official_video" in ai_video
|
||||
assert "kling_tts" in index
|
||||
assert "avatar/lip-sync face selection" in index
|
||||
assert ".agents/skills/kling-official/" in creative
|
||||
assert official_skill.is_file()
|
||||
official_skill_text = official_skill.read_text(encoding="utf-8")
|
||||
assert "Omni References" in official_skill_text
|
||||
assert "Callback Notes" in official_skill_text
|
||||
assert "TTS Parameters" in official_skill_text
|
||||
assert "Lip Sync Parameters" in official_skill_text
|
||||
assert "Audio Effects And Video Effects" in official_skill_text
|
||||
|
||||
|
||||
def test_provider_agent_skills_reference_kling_official():
|
||||
assert "kling-official" in KlingOfficialVideo().agent_skills
|
||||
assert "kling-official" in KlingOfficialImage().agent_skills
|
||||
assert "kling-official" in KlingTTS().agent_skills
|
||||
assert "kling-official" in KlingAvatar().agent_skills
|
||||
assert "kling-official" in KlingLipSync().agent_skills
|
||||
|
||||
|
||||
def test_phase3_does_not_register_audio_or_video_effect_tools():
|
||||
from tools.tool_registry import registry
|
||||
|
||||
registry.clear()
|
||||
registry.discover("tools")
|
||||
assert registry.get("kling_audio") is None
|
||||
assert registry.get("kling_effects") is None
|
||||
|
|
@ -0,0 +1,54 @@
|
|||
"""Contract tests for the Kling official E2E smoke script."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
|
||||
SCRIPT_PATH = PROJECT_ROOT / "scripts" / "kling_official_animated_explainer_e2e.py"
|
||||
|
||||
|
||||
def _load_script():
|
||||
spec = importlib.util.spec_from_file_location("kling_official_animated_explainer_e2e", SCRIPT_PATH)
|
||||
assert spec and spec.loader
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def test_env_status_redacts_secret_values():
|
||||
script = _load_script()
|
||||
|
||||
status = script._env_status(
|
||||
{
|
||||
"KLING_API_KEY": "secret-token",
|
||||
"KLING_API_BASE_URL": "https://api-beijing.klingai.com",
|
||||
"FAL_KEY": "",
|
||||
}
|
||||
)
|
||||
|
||||
assert status["KLING_API_KEY"]["present"] is True
|
||||
assert status["KLING_API_KEY"]["display"] == "<set:12 chars>"
|
||||
assert "secret-token" not in repr(status)
|
||||
assert status["KLING_API_BASE_URL"]["display"] == "https://api-beijing.klingai.com"
|
||||
assert status["FAL_KEY"]["present"] is False
|
||||
|
||||
|
||||
def test_cli_modes_are_explicit_and_non_paid_by_default():
|
||||
script = _load_script()
|
||||
|
||||
assert script._execution_mode(script._parse_args([])) == "dry_run"
|
||||
assert script._execution_mode(script._parse_args(["--live-tts"])) == "live_tts"
|
||||
assert script._execution_mode(script._parse_args(["--live-full"])) == "live_full"
|
||||
assert script._execution_mode(script._parse_args(["--live"])) == "live_full"
|
||||
|
||||
|
||||
def test_video_duration_aligns_to_narration_within_kling_limits():
|
||||
script = _load_script()
|
||||
|
||||
assert script._aligned_video_duration("3", 6.05) == "7"
|
||||
assert script._aligned_video_duration("10", 6.05) == "10"
|
||||
assert script._aligned_video_duration("3", None) == "3"
|
||||
assert script._aligned_video_duration("3", 30.0) == "15"
|
||||
|
|
@ -0,0 +1,166 @@
|
|||
"""Contract tests for Kling official Phase 2 helpers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
from tools._kling.account import get_account_costs, reset_account_usage_cache
|
||||
from tools._kling.client import KlingClient
|
||||
from tools._kling.elements import (
|
||||
get_custom_element,
|
||||
list_custom_elements,
|
||||
list_preset_elements,
|
||||
normalize_element_list,
|
||||
write_elements_artifact,
|
||||
)
|
||||
from tools.tool_registry import registry
|
||||
|
||||
|
||||
class FakeClient:
|
||||
def __init__(self, api_key="fake-key", base_url="https://api.example.test"):
|
||||
self.api_key = api_key
|
||||
self.base_url = base_url
|
||||
self.calls = []
|
||||
|
||||
def get(self, path, params=None):
|
||||
self.calls.append((path, params or {}))
|
||||
if path.startswith("/v1/general/advanced-custom-elements/"):
|
||||
return {"code": 0, "data": {"element_id": 123}}
|
||||
if path == "/v1/general/advanced-custom-elements":
|
||||
return {"code": 0, "data": [{"element_id": 456}]}
|
||||
if path == "/v1/general/advanced-presets-elements":
|
||||
return {"code": 0, "data": [{"element_id": 1}]}
|
||||
return {"code": 0, "data": {"resource_pack_subscribe_infos": [{"name": "pack-a"}]}}
|
||||
|
||||
|
||||
class FakeResponse:
|
||||
status_code = 200
|
||||
|
||||
def json(self):
|
||||
return {"code": 0, "data": {"resource_pack_subscribe_infos": [{"name": "pack-a"}]}}
|
||||
|
||||
|
||||
class FakeSession:
|
||||
def __init__(self):
|
||||
self.calls = []
|
||||
|
||||
def get(self, url, **kwargs):
|
||||
self.calls.append(("get", url, kwargs))
|
||||
return FakeResponse()
|
||||
|
||||
|
||||
def test_elements_helper_normalizes_and_records_metadata(tmp_path):
|
||||
assert normalize_element_list([123, {"element_id": "456"}]) == [
|
||||
{"element_id": 123},
|
||||
{"element_id": 456},
|
||||
]
|
||||
artifact = write_elements_artifact(
|
||||
tmp_path / "kling_elements.json",
|
||||
[{"element_id": 123, "kind": "character", "name": "main-presenter"}],
|
||||
)
|
||||
data = json.loads(artifact.read_text())
|
||||
assert data["provider"] == "kling_official"
|
||||
assert data["elements"][0]["element_id"] == 123
|
||||
|
||||
try:
|
||||
normalize_element_list([{"name": "missing-id"}])
|
||||
except ValueError as exc:
|
||||
assert "element_id" in str(exc)
|
||||
else:
|
||||
raise AssertionError("element_list items without element_id must be rejected")
|
||||
|
||||
|
||||
def test_elements_helper_read_only_endpoints_do_not_enter_registry():
|
||||
fake = FakeClient()
|
||||
assert get_custom_element(123, client=fake)["data"]["element_id"] == 123
|
||||
assert list_custom_elements(client=fake)["data"][0]["element_id"] == 456
|
||||
assert list_preset_elements(client=fake)["data"][0]["element_id"] == 1
|
||||
assert fake.calls == [
|
||||
("/v1/general/advanced-custom-elements/123", {}),
|
||||
("/v1/general/advanced-custom-elements", {}),
|
||||
("/v1/general/advanced-presets-elements", {}),
|
||||
]
|
||||
import tools._kling.elements as elements_module
|
||||
|
||||
assert not hasattr(elements_module, "create_element")
|
||||
assert not hasattr(elements_module, "delete_element")
|
||||
|
||||
registry.clear()
|
||||
registry.discover("tools")
|
||||
assert registry.get("kling_elements") is None
|
||||
assert registry.get("kling_account_usage") is None
|
||||
|
||||
|
||||
def test_account_usage_helper_uses_endpoint_cache_and_throttle():
|
||||
reset_account_usage_cache()
|
||||
fake = FakeClient()
|
||||
first = get_account_costs(
|
||||
start_time="2026-07-01",
|
||||
end_time="2026-07-03",
|
||||
client=fake,
|
||||
now=100.0,
|
||||
)
|
||||
second = get_account_costs(
|
||||
start_time="2026-07-01",
|
||||
end_time="2026-07-03",
|
||||
client=fake,
|
||||
now=101.0,
|
||||
)
|
||||
throttled = get_account_costs(
|
||||
resource_pack_name="different",
|
||||
client=fake,
|
||||
now=102.0,
|
||||
)
|
||||
|
||||
assert fake.calls == [
|
||||
("/account/costs", {"start_time": "2026-07-01", "end_time": "2026-07-03"})
|
||||
]
|
||||
assert first["throttle_status"] == "fresh"
|
||||
assert second["cached"] is True
|
||||
assert second["throttle_status"] == "cache_hit"
|
||||
assert throttled["throttle_status"] == "throttled_no_cache"
|
||||
|
||||
|
||||
def test_account_usage_cache_is_scoped_by_api_identity():
|
||||
reset_account_usage_cache()
|
||||
first_client = FakeClient(api_key="account-a")
|
||||
second_client = FakeClient(api_key="account-b")
|
||||
|
||||
get_account_costs(client=first_client, now=100.0)
|
||||
get_account_costs(client=second_client, now=111.0)
|
||||
cached = get_account_costs(client=second_client, now=112.0)
|
||||
|
||||
assert first_client.calls == [("/account/costs", {})]
|
||||
assert second_client.calls == [("/account/costs", {})]
|
||||
assert cached["cached"] is True
|
||||
|
||||
|
||||
def test_account_usage_helper_uses_kling_auth_header(monkeypatch):
|
||||
reset_account_usage_cache()
|
||||
monkeypatch.setenv("KLING_API_KEY", "test-key")
|
||||
session = FakeSession()
|
||||
client = KlingClient(session=session, max_retries=0)
|
||||
|
||||
result = get_account_costs(
|
||||
start_time="2026-07-01",
|
||||
end_time="2026-07-03",
|
||||
resource_pack_name="starter",
|
||||
client=client,
|
||||
now=200.0,
|
||||
)
|
||||
|
||||
assert result["resource_pack_subscribe_infos"][0]["name"] == "pack-a"
|
||||
method, url, kwargs = session.calls[0]
|
||||
assert method == "get"
|
||||
assert url.endswith("/account/costs")
|
||||
assert kwargs["headers"]["Authorization"] == "Bearer test-key"
|
||||
assert kwargs["params"] == {
|
||||
"start_time": "2026-07-01",
|
||||
"end_time": "2026-07-03",
|
||||
"resource_pack_name": "starter",
|
||||
}
|
||||
|
|
@ -0,0 +1,299 @@
|
|||
"""Contract tests for the Kling official image provider."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
from tools.graphics.image_selector import ImageSelector
|
||||
from tools.graphics.kling_official_image import KlingOfficialImage
|
||||
from tools.tool_registry import registry
|
||||
|
||||
|
||||
def test_registry_discovers_kling_official_image(monkeypatch):
|
||||
monkeypatch.delenv("KLING_API_KEY", raising=False)
|
||||
registry.clear()
|
||||
registry.discover("tools")
|
||||
tool = registry.get("kling_official_image")
|
||||
assert tool is not None
|
||||
assert tool.capability == "image_generation"
|
||||
assert tool.provider == "kling_official"
|
||||
|
||||
|
||||
def test_image_schema_and_skill():
|
||||
tool = KlingOfficialImage()
|
||||
props = tool.input_schema["properties"]
|
||||
assert "image_url" in props
|
||||
assert "api_family" in props
|
||||
assert "kling-official" in tool.agent_skills
|
||||
|
||||
|
||||
def test_generation_payload():
|
||||
tool = KlingOfficialImage()
|
||||
request = tool._build_request(
|
||||
{
|
||||
"prompt": "portrait of a launch engineer",
|
||||
"negative_prompt": "blurry",
|
||||
"api_family": "generation",
|
||||
"resolution": "2k",
|
||||
"n": 2,
|
||||
"watermark": False,
|
||||
}
|
||||
)
|
||||
assert request["path"] == "/v1/images/generations"
|
||||
assert request["payload"]["model_name"] == "kling-v3"
|
||||
assert request["payload"]["negative_prompt"] == "blurry"
|
||||
assert request["payload"]["resolution"] == "2k"
|
||||
assert request["payload"]["n"] == 2
|
||||
assert request["payload"]["watermark_info"] == {"enabled": False}
|
||||
|
||||
|
||||
def test_edit_payload_converts_image_path_to_base64(tmp_path):
|
||||
image_path = tmp_path / "subject.png"
|
||||
image_path.write_bytes(b"subject")
|
||||
tool = KlingOfficialImage()
|
||||
request = tool._build_request(
|
||||
{
|
||||
"prompt": "keep the subject, change background",
|
||||
"generation_mode": "edit",
|
||||
"image_path": str(image_path),
|
||||
"image_reference": "subject",
|
||||
}
|
||||
)
|
||||
assert request["path"] == "/v1/images/generations"
|
||||
assert request["payload"]["image"] == base64.b64encode(b"subject").decode("ascii")
|
||||
assert request["payload"]["image_reference"] == "subject"
|
||||
|
||||
|
||||
def test_omni_payload_uses_image_list(tmp_path):
|
||||
image_path = tmp_path / "ref.png"
|
||||
image_path.write_bytes(b"ref")
|
||||
tool = KlingOfficialImage()
|
||||
request = tool._build_request(
|
||||
{
|
||||
"prompt": "combine <<<image_1>>> with neon product lighting",
|
||||
"api_family": "omni",
|
||||
"image_urls": ["https://example.com/ref-a.png"],
|
||||
"image_paths": [str(image_path)],
|
||||
"result_type": "series",
|
||||
"series_amount": "3",
|
||||
}
|
||||
)
|
||||
assert request["path"] == "/v1/images/omni-image"
|
||||
assert request["payload"]["model_name"] == "kling-image-o1"
|
||||
assert request["payload"]["image_list"][0] == {"image": "https://example.com/ref-a.png"}
|
||||
assert request["payload"]["image_list"][1] == {"image": base64.b64encode(b"ref").decode("ascii")}
|
||||
assert request["payload"]["series_amount"] == "3"
|
||||
assert request["references_used"][0]["placeholder"] == "<<<image_1>>>"
|
||||
|
||||
|
||||
def test_omni_prompt_helper_adds_placeholders_and_validates_counts():
|
||||
tool = KlingOfficialImage()
|
||||
request = tool._build_request(
|
||||
{
|
||||
"prompt": "combine these into one scene",
|
||||
"api_family": "omni",
|
||||
"image_urls": ["https://example.com/a.png", "https://example.com/b.png"],
|
||||
}
|
||||
)
|
||||
assert "<<<image_1>>> <<<image_2>>>" in request["payload"]["prompt"]
|
||||
assert request["references_used"][1]["source"] == "https://example.com/b.png"
|
||||
|
||||
existing = tool._build_request(
|
||||
{
|
||||
"prompt": "keep <<<image_1>>> as the subject",
|
||||
"api_family": "omni",
|
||||
"image_urls": ["https://example.com/a.png"],
|
||||
}
|
||||
)
|
||||
assert existing["payload"]["prompt"].count("<<<image_1>>>") == 1
|
||||
|
||||
try:
|
||||
tool._build_request(
|
||||
{
|
||||
"prompt": "use <<<image_2>>>",
|
||||
"api_family": "omni",
|
||||
"image_urls": ["https://example.com/a.png"],
|
||||
}
|
||||
)
|
||||
except ValueError as exc:
|
||||
assert "only 1 image" in str(exc)
|
||||
else:
|
||||
raise AssertionError("Image Omni placeholders must match provided image count")
|
||||
|
||||
|
||||
def test_image_omni_element_list_and_callback_payload():
|
||||
tool = KlingOfficialImage()
|
||||
request = tool._build_request(
|
||||
{
|
||||
"prompt": "render with element",
|
||||
"api_family": "omni",
|
||||
"element_list": [{"element_id": "321"}],
|
||||
"callback_url": "https://example.com/callback",
|
||||
}
|
||||
)
|
||||
assert request["payload"]["element_list"] == [{"element_id": 321}]
|
||||
assert request["payload"]["callback_url"] == "https://example.com/callback"
|
||||
assert request["element_ids"] == [321]
|
||||
|
||||
try:
|
||||
tool._build_request(
|
||||
{
|
||||
"prompt": "bad callback",
|
||||
"api_family": "omni",
|
||||
"callback_url": "ftp://example.com/callback",
|
||||
}
|
||||
)
|
||||
except ValueError as exc:
|
||||
assert "callback_url" in str(exc)
|
||||
else:
|
||||
raise AssertionError("callback_url must be an absolute http(s) URL")
|
||||
|
||||
|
||||
def test_image_model_must_match_api_family():
|
||||
tool = KlingOfficialImage()
|
||||
try:
|
||||
tool._build_request(
|
||||
{
|
||||
"prompt": "generation with omni model",
|
||||
"api_family": "generation",
|
||||
"model_name": "kling-image-o1",
|
||||
}
|
||||
)
|
||||
except ValueError as exc:
|
||||
assert "api_family=generation" in str(exc)
|
||||
else:
|
||||
raise AssertionError("generation requests must reject omni image models")
|
||||
|
||||
try:
|
||||
tool._build_request(
|
||||
{
|
||||
"prompt": "omni with generation model",
|
||||
"api_family": "omni",
|
||||
"model_name": "kling-v3",
|
||||
}
|
||||
)
|
||||
except ValueError as exc:
|
||||
assert "api_family=omni" in str(exc)
|
||||
else:
|
||||
raise AssertionError("omni requests must reject generation image models")
|
||||
|
||||
|
||||
def test_execute_downloads_all_image_results(monkeypatch, tmp_path):
|
||||
class FakeClient:
|
||||
def create_classic_task(self, path, payload):
|
||||
return "img-task-1"
|
||||
|
||||
def poll_classic(self, path, task_id, result_key, timeout_seconds, poll_interval):
|
||||
return [
|
||||
{"url": "https://example.com/a.png"},
|
||||
{"url": "https://example.com/b.png"},
|
||||
]
|
||||
|
||||
def download(self, url, output_path):
|
||||
output_path.write_bytes(url.encode("utf-8"))
|
||||
return output_path
|
||||
|
||||
monkeypatch.setenv("KLING_API_KEY", "test-key")
|
||||
monkeypatch.setattr("tools.graphics.kling_official_image.KlingClient", lambda: FakeClient())
|
||||
output_path = tmp_path / "image.png"
|
||||
result = KlingOfficialImage().execute({"prompt": "x", "n": 2, "output_path": str(output_path)})
|
||||
assert result.success
|
||||
assert result.data["provider"] == "kling_official"
|
||||
assert result.data["task_id"] == "img-task-1"
|
||||
assert result.data["remote_outputs"][0]["url"].endswith("a.png")
|
||||
assert len(result.artifacts) == 2
|
||||
assert Path(result.artifacts[0]).read_bytes() == b"https://example.com/a.png"
|
||||
assert Path(result.artifacts[1]).read_bytes() == b"https://example.com/b.png"
|
||||
assert result.cost_usd > 0
|
||||
|
||||
|
||||
def test_execute_image_omni_series_records_references_callback_and_artifacts(monkeypatch, tmp_path):
|
||||
class FakeClient:
|
||||
def create_classic_task(self, path, payload):
|
||||
self.path = path
|
||||
self.payload = payload
|
||||
return "omni-img-task-1"
|
||||
|
||||
def poll_classic(self, path, task_id, result_key, timeout_seconds, poll_interval):
|
||||
return [
|
||||
{"url": "https://example.com/series-a.png"},
|
||||
{"url": "https://example.com/series-b.png"},
|
||||
]
|
||||
|
||||
def download(self, url, output_path):
|
||||
output_path.write_bytes(url.encode("utf-8"))
|
||||
return output_path
|
||||
|
||||
monkeypatch.setenv("KLING_API_KEY", "test-key")
|
||||
monkeypatch.setattr("tools.graphics.kling_official_image.KlingClient", lambda: FakeClient())
|
||||
result = KlingOfficialImage().execute(
|
||||
{
|
||||
"prompt": "series from references",
|
||||
"api_family": "omni",
|
||||
"image_urls": ["https://example.com/ref.png"],
|
||||
"element_list": [654],
|
||||
"result_type": "series",
|
||||
"series_amount": "2",
|
||||
"callback_url": "https://example.com/callback",
|
||||
"output_path": str(tmp_path / "series.png"),
|
||||
}
|
||||
)
|
||||
|
||||
assert result.success
|
||||
assert result.data["api_family"] == "omni"
|
||||
assert result.data["remote_outputs"][1]["url"].endswith("series-b.png")
|
||||
assert result.data["references_used"][0]["placeholder"] == "<<<image_1>>>"
|
||||
assert result.data["element_ids"] == [654]
|
||||
assert result.data["callback_requested"] is True
|
||||
assert result.data["polling_used"] is True
|
||||
assert len(result.artifacts) == 2
|
||||
assert Path(result.artifacts[1]).name == "series_2.png"
|
||||
|
||||
|
||||
def test_image_selector_prefers_official_provider(monkeypatch):
|
||||
monkeypatch.setenv("KLING_API_KEY", "test-key")
|
||||
registry.clear()
|
||||
registry.register(KlingOfficialImage())
|
||||
registry.register(ImageSelector())
|
||||
registry._discovered_packages.add("tools")
|
||||
|
||||
def fake_execute(self, inputs):
|
||||
from tools.base_tool import ToolResult
|
||||
|
||||
return ToolResult(success=True, data={"output_path": "out.png"}, artifacts=["out.png"])
|
||||
|
||||
monkeypatch.setattr(KlingOfficialImage, "execute", fake_execute)
|
||||
result = registry.get("image_selector").execute(
|
||||
{
|
||||
"prompt": "official image",
|
||||
"preferred_provider": "kling_official",
|
||||
"api_family": "omni",
|
||||
"image_reference": "subject",
|
||||
}
|
||||
)
|
||||
assert result.success
|
||||
assert result.data["selected_provider"] == "kling_official"
|
||||
|
||||
|
||||
def test_image_cost_estimate_is_not_zero():
|
||||
tool = KlingOfficialImage()
|
||||
assert tool.estimate_cost({"prompt": "x"}) > 0
|
||||
base = tool.estimate_cost({"prompt": "x", "api_family": "omni"})
|
||||
series = tool.estimate_cost(
|
||||
{
|
||||
"prompt": "x",
|
||||
"api_family": "omni",
|
||||
"result_type": "series",
|
||||
"series_amount": "3",
|
||||
"resolution": "4k",
|
||||
"image_urls": ["https://example.com/a.png", "https://example.com/b.png"],
|
||||
}
|
||||
)
|
||||
assert series > base
|
||||
dry_run = tool.dry_run({"prompt": "x"})
|
||||
assert dry_run["cost_estimate_confidence"] == "low"
|
||||
|
|
@ -0,0 +1,422 @@
|
|||
"""Contract tests for the Kling official video provider."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
from tools._kling.account import reset_account_usage_cache
|
||||
from tools.tool_registry import registry
|
||||
from tools._kling.errors import KlingAPIError
|
||||
from tools.video.kling_official_video import KlingOfficialVideo
|
||||
from tools.video.video_selector import VideoSelector
|
||||
|
||||
|
||||
def test_registry_discovers_kling_official_video(monkeypatch):
|
||||
monkeypatch.delenv("KLING_API_KEY", raising=False)
|
||||
registry.clear()
|
||||
registry.discover("tools")
|
||||
tool = registry.get("kling_official_video")
|
||||
assert tool is not None
|
||||
assert tool.capability == "video_generation"
|
||||
assert tool.provider == "kling_official"
|
||||
|
||||
|
||||
def test_video_schema_has_no_top_level_image_url_and_has_skill():
|
||||
tool = KlingOfficialVideo()
|
||||
props = tool.input_schema["properties"]
|
||||
assert "image_url" not in props
|
||||
assert "reference_image_url" in props
|
||||
assert "kling-official" in tool.agent_skills
|
||||
assert "ai-video-gen" in tool.agent_skills
|
||||
|
||||
|
||||
def test_classic_text_to_video_payload():
|
||||
tool = KlingOfficialVideo()
|
||||
request = tool._build_request(
|
||||
{
|
||||
"prompt": "cinematic robot walking through rain",
|
||||
"api_family": "classic",
|
||||
"operation": "text_to_video",
|
||||
"duration": "5",
|
||||
"aspect_ratio": "9:16",
|
||||
"watermark": False,
|
||||
}
|
||||
)
|
||||
assert request["path"] == "/v1/videos/text2video"
|
||||
assert request["protocol"] == "classic"
|
||||
assert request["payload"]["model_name"] == "kling-v3"
|
||||
assert request["payload"]["prompt"].startswith("cinematic")
|
||||
assert request["payload"]["aspect_ratio"] == "9:16"
|
||||
assert request["payload"]["watermark_info"] == {"enabled": False}
|
||||
|
||||
|
||||
def test_classic_image_to_video_uses_reference_image_path(tmp_path):
|
||||
image_path = tmp_path / "ref.png"
|
||||
image_path.write_bytes(b"fake-image")
|
||||
tool = KlingOfficialVideo()
|
||||
request = tool._build_request(
|
||||
{
|
||||
"prompt": "animate the frame",
|
||||
"api_family": "classic",
|
||||
"operation": "image_to_video",
|
||||
"reference_image_path": str(image_path),
|
||||
}
|
||||
)
|
||||
assert request["path"] == "/v1/videos/image2video"
|
||||
assert request["payload"]["image"] == base64.b64encode(b"fake-image").decode("ascii")
|
||||
assert "aspect_ratio" not in request["payload"]
|
||||
|
||||
|
||||
def test_turbo_payloads():
|
||||
tool = KlingOfficialVideo()
|
||||
text_request = tool._build_request(
|
||||
{
|
||||
"prompt": "fast product reveal",
|
||||
"api_family": "turbo",
|
||||
"operation": "text_to_video",
|
||||
"duration": "6",
|
||||
"resolution": "1080p",
|
||||
}
|
||||
)
|
||||
assert text_request["path"] == "/text-to-video/kling-3.0-turbo"
|
||||
assert text_request["payload"]["settings"] == {
|
||||
"resolution": "1080p",
|
||||
"duration": 6,
|
||||
"aspect_ratio": "16:9",
|
||||
}
|
||||
|
||||
image_request = tool._build_request(
|
||||
{
|
||||
"prompt": "animate the product",
|
||||
"api_family": "turbo",
|
||||
"operation": "image_to_video",
|
||||
"reference_image_url": "https://example.com/ref.png",
|
||||
}
|
||||
)
|
||||
assert image_request["path"] == "/image-to-video/kling-3.0-turbo"
|
||||
assert image_request["payload"]["contents"] == [
|
||||
{"type": "prompt", "text": "animate the product"},
|
||||
{"type": "first_frame", "url": "https://example.com/ref.png"},
|
||||
]
|
||||
|
||||
|
||||
def test_omni_reference_payload():
|
||||
tool = KlingOfficialVideo()
|
||||
request = tool._build_request(
|
||||
{
|
||||
"prompt": "match the motion and mood",
|
||||
"api_family": "omni",
|
||||
"operation": "reference_to_video",
|
||||
"video_list": [{"video_url": "https://example.com/ref.mp4", "refer_type": "base"}],
|
||||
}
|
||||
)
|
||||
assert request["path"] == "/v1/videos/omni-video"
|
||||
assert request["payload"]["model_name"] == "kling-video-o1"
|
||||
assert request["payload"]["video_list"][0]["video_url"].endswith("ref.mp4")
|
||||
|
||||
|
||||
def test_video_omni_payload_supports_multi_refs_elements_and_multi_prompt(tmp_path):
|
||||
image_path = tmp_path / "local.png"
|
||||
image_path.write_bytes(b"local-ref")
|
||||
tool = KlingOfficialVideo()
|
||||
request = tool._build_request(
|
||||
{
|
||||
"prompt": "two-shot brand reveal",
|
||||
"api_family": "omni",
|
||||
"operation": "reference_to_video",
|
||||
"image_list": [{"image_url": "https://example.com/start.png", "type": "first_frame"}],
|
||||
"reference_tail_image_path": str(image_path),
|
||||
"video_list": [
|
||||
{
|
||||
"video_url": "https://example.com/motion.mp4",
|
||||
"refer_type": "feature",
|
||||
"keep_original_sound": True,
|
||||
}
|
||||
],
|
||||
"element_list": [123, {"element_id": "456"}],
|
||||
"multi_shot": True,
|
||||
"shot_type": "customize",
|
||||
"multi_prompt": [
|
||||
{"prompt": "wide product intro", "duration": "5"},
|
||||
{"prompt": "close detail pass", "camera_control": {"type": "simple"}},
|
||||
],
|
||||
}
|
||||
)
|
||||
payload = request["payload"]
|
||||
assert payload["image_list"][0] == {"image_url": "https://example.com/start.png", "type": "first_frame"}
|
||||
assert payload["image_list"][1] == {
|
||||
"image_url": base64.b64encode(b"local-ref").decode("ascii"),
|
||||
"type": "end_frame",
|
||||
}
|
||||
assert payload["video_list"] == [
|
||||
{
|
||||
"video_url": "https://example.com/motion.mp4",
|
||||
"refer_type": "feature",
|
||||
"keep_original_sound": "yes",
|
||||
}
|
||||
]
|
||||
assert payload["element_list"] == [{"element_id": 123}, {"element_id": 456}]
|
||||
assert payload["multi_shot"] is True
|
||||
assert payload["multi_prompt"][1]["camera_control"] == {"type": "simple"}
|
||||
assert request["element_ids"] == [123, 456]
|
||||
assert any(item["kind"] == "element" for item in request["references_used"])
|
||||
|
||||
|
||||
def test_video_omni_requires_reference_input_and_rejects_local_video_paths():
|
||||
tool = KlingOfficialVideo()
|
||||
try:
|
||||
tool._build_request(
|
||||
{
|
||||
"prompt": "needs a reference",
|
||||
"api_family": "omni",
|
||||
"operation": "reference_to_video",
|
||||
}
|
||||
)
|
||||
except ValueError as exc:
|
||||
assert "requires image_list, video_list, element_list" in str(exc)
|
||||
else:
|
||||
raise AssertionError("reference_to_video must require at least one Omni reference")
|
||||
|
||||
try:
|
||||
tool._build_request(
|
||||
{
|
||||
"prompt": "local video",
|
||||
"api_family": "omni",
|
||||
"operation": "reference_to_video",
|
||||
"reference_video_path": "/tmp/ref.mp4",
|
||||
}
|
||||
)
|
||||
except ValueError as exc:
|
||||
assert "local video paths cannot be silently uploaded" in str(exc)
|
||||
else:
|
||||
raise AssertionError("Video Omni must not silently upload local videos")
|
||||
|
||||
|
||||
def test_video_callback_payloads_and_validation():
|
||||
tool = KlingOfficialVideo()
|
||||
classic = tool._build_request(
|
||||
{
|
||||
"prompt": "callback classic",
|
||||
"api_family": "classic",
|
||||
"operation": "text_to_video",
|
||||
"callback_url": "https://example.com/kling/callback",
|
||||
}
|
||||
)
|
||||
assert classic["payload"]["callback_url"] == "https://example.com/kling/callback"
|
||||
|
||||
turbo = tool._build_request(
|
||||
{
|
||||
"prompt": "callback turbo",
|
||||
"api_family": "turbo",
|
||||
"operation": "text_to_video",
|
||||
"callback_url": "https://example.com/kling/callback",
|
||||
}
|
||||
)
|
||||
assert turbo["payload"]["options"]["callback_url"] == "https://example.com/kling/callback"
|
||||
|
||||
omni = tool._build_request(
|
||||
{
|
||||
"prompt": "callback omni",
|
||||
"api_family": "omni",
|
||||
"operation": "reference_to_video",
|
||||
"video_list": [{"video_url": "https://example.com/ref.mp4"}],
|
||||
"callback_url": "https://example.com/kling/callback",
|
||||
}
|
||||
)
|
||||
assert omni["payload"]["callback_url"] == "https://example.com/kling/callback"
|
||||
|
||||
try:
|
||||
tool._build_request(
|
||||
{
|
||||
"prompt": "bad callback",
|
||||
"api_family": "classic",
|
||||
"operation": "text_to_video",
|
||||
"callback_url": "not-a-url",
|
||||
}
|
||||
)
|
||||
except ValueError as exc:
|
||||
assert "callback_url" in str(exc)
|
||||
else:
|
||||
raise AssertionError("callback_url must be validated before sending")
|
||||
|
||||
|
||||
def test_video_model_must_match_api_family():
|
||||
tool = KlingOfficialVideo()
|
||||
try:
|
||||
tool._build_request(
|
||||
{
|
||||
"prompt": "classic request with omni model",
|
||||
"api_family": "classic",
|
||||
"operation": "text_to_video",
|
||||
"model_name": "kling-video-o1",
|
||||
}
|
||||
)
|
||||
except ValueError as exc:
|
||||
assert "api_family=classic" in str(exc)
|
||||
else:
|
||||
raise AssertionError("classic requests must reject omni video models")
|
||||
|
||||
try:
|
||||
tool._build_request(
|
||||
{
|
||||
"prompt": "omni request with classic model",
|
||||
"api_family": "omni",
|
||||
"operation": "reference_to_video",
|
||||
"model_name": "kling-v3",
|
||||
"video_list": [{"video_url": "https://example.com/ref.mp4"}],
|
||||
}
|
||||
)
|
||||
except ValueError as exc:
|
||||
assert "api_family=omni" in str(exc)
|
||||
else:
|
||||
raise AssertionError("omni requests must reject classic video models")
|
||||
|
||||
|
||||
def test_execute_downloads_video_and_returns_artifact(monkeypatch, tmp_path):
|
||||
class FakeClient:
|
||||
def create_classic_task(self, path, payload):
|
||||
self.path = path
|
||||
self.payload = payload
|
||||
return "task-1"
|
||||
|
||||
def poll_classic(self, path, task_id, result_key, timeout_seconds, poll_interval):
|
||||
return [{"url": "https://example.com/out.mp4"}]
|
||||
|
||||
def download(self, url, output_path):
|
||||
output_path.write_bytes(b"video")
|
||||
return output_path
|
||||
|
||||
monkeypatch.setenv("KLING_API_KEY", "test-key")
|
||||
monkeypatch.setattr("tools.video.kling_official_video.KlingClient", lambda: FakeClient())
|
||||
monkeypatch.setattr("tools.video.kling_official_video.probe_output", lambda path: {"duration_seconds": 5.0})
|
||||
output_path = tmp_path / "out.mp4"
|
||||
result = KlingOfficialVideo().execute({"prompt": "x", "output_path": str(output_path)})
|
||||
assert result.success
|
||||
assert result.data["provider"] == "kling_official"
|
||||
assert result.data["task_id"] == "task-1"
|
||||
assert result.artifacts == [str(output_path)]
|
||||
assert output_path.read_bytes() == b"video"
|
||||
assert result.cost_usd > 0
|
||||
|
||||
|
||||
def test_execute_downloads_all_omni_video_outputs_and_records_metadata(monkeypatch, tmp_path):
|
||||
reset_account_usage_cache()
|
||||
|
||||
class FakeClient:
|
||||
def create_classic_task(self, path, payload):
|
||||
self.path = path
|
||||
self.payload = payload
|
||||
return "omni-task-1"
|
||||
|
||||
def poll_classic(self, path, task_id, result_key, timeout_seconds, poll_interval):
|
||||
return [
|
||||
{"url": "https://example.com/a.mp4"},
|
||||
{"url": "https://example.com/b.mp4"},
|
||||
]
|
||||
|
||||
def download(self, url, output_path):
|
||||
output_path.write_bytes(url.encode("utf-8"))
|
||||
return output_path
|
||||
|
||||
def get(self, path, params=None):
|
||||
assert path == "/account/costs"
|
||||
return {"code": 0, "data": {"resource_pack_subscribe_infos": [{"name": "pack-a"}]}}
|
||||
|
||||
monkeypatch.setenv("KLING_API_KEY", "test-key")
|
||||
monkeypatch.setattr("tools.video.kling_official_video.KlingClient", lambda: FakeClient())
|
||||
monkeypatch.setattr("tools.video.kling_official_video.probe_output", lambda path: {"duration_seconds": 5.0})
|
||||
result = KlingOfficialVideo().execute(
|
||||
{
|
||||
"prompt": "omni",
|
||||
"api_family": "omni",
|
||||
"operation": "reference_to_video",
|
||||
"video_list": [{"video_url": "https://example.com/ref.mp4", "refer_type": "base"}],
|
||||
"element_list": [789],
|
||||
"callback_url": "https://example.com/callback",
|
||||
"include_account_usage": True,
|
||||
"output_path": str(tmp_path / "out.mp4"),
|
||||
}
|
||||
)
|
||||
assert result.success
|
||||
assert result.data["api_family"] == "omni"
|
||||
assert result.data["remote_outputs"][1]["url"].endswith("b.mp4")
|
||||
assert result.data["element_ids"] == [789]
|
||||
assert result.data["callback_requested"] is True
|
||||
assert result.data["polling_used"] is True
|
||||
assert result.data["account_usage"]["resource_pack_subscribe_infos"][0]["name"] == "pack-a"
|
||||
assert result.data["cost_source"] == "estimate_with_account_usage_context"
|
||||
assert result.cost_usd > 0
|
||||
assert len(result.artifacts) == 2
|
||||
assert Path(result.artifacts[1]).name == "out_2.mp4"
|
||||
|
||||
|
||||
def test_video_selector_prefers_official_provider_without_fal_upload(monkeypatch, tmp_path):
|
||||
monkeypatch.setenv("KLING_API_KEY", "test-key")
|
||||
image_path = tmp_path / "ref.png"
|
||||
image_path.write_bytes(b"fake")
|
||||
registry.clear()
|
||||
registry.register(KlingOfficialVideo())
|
||||
registry.register(VideoSelector())
|
||||
registry._discovered_packages.add("tools")
|
||||
|
||||
seen = {}
|
||||
|
||||
def fake_execute(self, inputs):
|
||||
seen.update(inputs)
|
||||
from tools.base_tool import ToolResult
|
||||
|
||||
return ToolResult(success=True, data={"output_path": "out.mp4"}, artifacts=["out.mp4"])
|
||||
|
||||
def fail_upload(path):
|
||||
raise AssertionError("fal.ai upload should not be called for kling_official_video")
|
||||
|
||||
monkeypatch.setattr(KlingOfficialVideo, "execute", fake_execute)
|
||||
monkeypatch.setattr("tools.video._shared.upload_image_fal", fail_upload)
|
||||
result = registry.get("video_selector").execute(
|
||||
{
|
||||
"prompt": "animate",
|
||||
"operation": "image_to_video",
|
||||
"preferred_provider": "kling_official",
|
||||
"reference_image_path": str(image_path),
|
||||
}
|
||||
)
|
||||
assert result.success
|
||||
assert result.data["selected_provider"] == "kling_official"
|
||||
assert seen["reference_image_path"] == str(image_path)
|
||||
|
||||
|
||||
def test_video_cost_estimate_is_not_zero():
|
||||
tool = KlingOfficialVideo()
|
||||
assert tool.estimate_cost({"prompt": "x"}) > 0
|
||||
base = tool.estimate_cost({"prompt": "x", "api_family": "omni"})
|
||||
expensive = tool.estimate_cost(
|
||||
{
|
||||
"prompt": "x",
|
||||
"api_family": "omni",
|
||||
"mode": "4k",
|
||||
"sound": "on",
|
||||
"video_list": [{"video_url": "https://example.com/ref.mp4"}],
|
||||
"element_list": [1, 2],
|
||||
"multi_prompt": [{"prompt": "a"}, {"prompt": "b"}],
|
||||
}
|
||||
)
|
||||
assert expensive > base
|
||||
dry_run = tool.dry_run({"prompt": "x"})
|
||||
assert dry_run["cost_estimate_confidence"] == "low"
|
||||
|
||||
|
||||
def test_video_account_resource_error_includes_diagnostic(monkeypatch):
|
||||
class FakeClient:
|
||||
def create_classic_task(self, path, payload):
|
||||
raise KlingAPIError("resource pack exhausted", code=1102, request_id="req-1")
|
||||
|
||||
monkeypatch.setenv("KLING_API_KEY", "test-key")
|
||||
monkeypatch.setattr("tools.video.kling_official_video.KlingClient", lambda: FakeClient())
|
||||
result = KlingOfficialVideo().execute({"prompt": "x"})
|
||||
assert not result.success
|
||||
assert result.data["account_usage_diagnostic"]["reason"] == "account_balance_or_resource_pack"
|
||||
|
|
@ -0,0 +1,177 @@
|
|||
"""Contract tests for the Kling official TTS provider."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
from tools.audio.kling_tts import KlingTTS
|
||||
from tools.audio.tts_selector import TTSSelector
|
||||
from tools.tool_registry import registry
|
||||
|
||||
|
||||
def test_registry_discovers_kling_tts(monkeypatch):
|
||||
monkeypatch.delenv("KLING_API_KEY", raising=False)
|
||||
registry.clear()
|
||||
registry.discover("tools")
|
||||
tool = registry.get("kling_tts")
|
||||
assert tool is not None
|
||||
assert tool.capability == "tts"
|
||||
assert tool.provider == "kling_official"
|
||||
|
||||
|
||||
def test_tts_schema_and_skill_metadata():
|
||||
tool = KlingTTS()
|
||||
props = tool.input_schema["properties"]
|
||||
assert "voice_id" in props
|
||||
assert props["voice_language"]["enum"] == ["zh", "en"]
|
||||
assert "kling-official" in tool.agent_skills
|
||||
assert "text-to-speech" in tool.agent_skills
|
||||
assert tool.estimate_cost({"text": "hello", "voice_id": "voice-a"}) > 0
|
||||
assert tool.dry_run({"text": "hello", "voice_id": "voice-a"})["cost_estimate_confidence"] == "low"
|
||||
|
||||
|
||||
def test_tts_payload_and_validation():
|
||||
tool = KlingTTS()
|
||||
request = tool._build_request(
|
||||
{
|
||||
"text": "Hello from Kling",
|
||||
"voice_id": "voice-a",
|
||||
"voice_language": "en",
|
||||
"voice_speed": 1.2,
|
||||
"callback_url": "https://example.com/kling/callback",
|
||||
}
|
||||
)
|
||||
assert request["path"] == "/v1/audio/tts"
|
||||
assert request["payload"] == {
|
||||
"text": "Hello from Kling",
|
||||
"voice_id": "voice-a",
|
||||
"voice_language": "en",
|
||||
"voice_speed": 1.2,
|
||||
"callback_url": "https://example.com/kling/callback",
|
||||
}
|
||||
|
||||
for bad_inputs in (
|
||||
{"text": "missing voice"},
|
||||
{"text": "x", "voice_id": "voice-a", "voice_language": "fr"},
|
||||
{"text": "x", "voice_id": "voice-a", "voice_speed": 9},
|
||||
):
|
||||
try:
|
||||
tool._build_request(bad_inputs)
|
||||
except ValueError:
|
||||
pass
|
||||
else:
|
||||
raise AssertionError(f"Invalid TTS inputs should fail: {bad_inputs}")
|
||||
|
||||
|
||||
def test_execute_downloads_all_audio_results(monkeypatch, tmp_path):
|
||||
class FakeClient:
|
||||
def create_classic_task(self, path, payload):
|
||||
self.path = path
|
||||
self.payload = payload
|
||||
return "tts-task-1"
|
||||
|
||||
def poll_classic(self, path, task_id, result_key, timeout_seconds, poll_interval):
|
||||
assert result_key == "audios"
|
||||
return [
|
||||
{"url": "https://example.com/a.mp3"},
|
||||
{"audio_url": "https://example.com/b.wav"},
|
||||
]
|
||||
|
||||
def download(self, url, output_path):
|
||||
output_path.write_bytes(url.encode("utf-8"))
|
||||
return output_path
|
||||
|
||||
monkeypatch.setenv("KLING_API_KEY", "test-key")
|
||||
monkeypatch.setattr("tools.audio.kling_tts.KlingClient", lambda: FakeClient())
|
||||
monkeypatch.setattr("tools.audio.kling_tts.probe_duration", lambda path: 1.23)
|
||||
|
||||
output_path = tmp_path / "speech.mp3"
|
||||
result = KlingTTS().execute(
|
||||
{
|
||||
"text": "Hello",
|
||||
"voice_id": "voice-a",
|
||||
"output_path": str(output_path),
|
||||
}
|
||||
)
|
||||
|
||||
assert result.success
|
||||
assert result.data["provider"] == "kling_official"
|
||||
assert result.data["task_id"] == "tts-task-1"
|
||||
assert result.data["audio_duration_seconds"] == 1.23
|
||||
assert len(result.artifacts) == 2
|
||||
assert Path(result.artifacts[0]).name == "speech.mp3"
|
||||
assert Path(result.artifacts[1]).name == "speech_2.wav"
|
||||
assert result.cost_usd > 0
|
||||
|
||||
|
||||
def test_execute_accepts_synchronous_create_response(monkeypatch, tmp_path):
|
||||
class FakeClient:
|
||||
def post(self, path, payload):
|
||||
assert path == "/v1/audio/tts"
|
||||
assert payload["voice_id"] == "voice-a"
|
||||
return {
|
||||
"code": 0,
|
||||
"message": "SUCCEED",
|
||||
"request_id": "req-1",
|
||||
"data": {
|
||||
"task_id": "tts-task-sync",
|
||||
"task_status": "succeed",
|
||||
"task_result": {
|
||||
"audios": [
|
||||
{"url": "https://example.com/sync.mp3"},
|
||||
]
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
def poll_classic(self, *args, **kwargs):
|
||||
raise AssertionError("synchronous TTS response should not poll")
|
||||
|
||||
def download(self, url, output_path):
|
||||
output_path.write_bytes(url.encode("utf-8"))
|
||||
return output_path
|
||||
|
||||
monkeypatch.setenv("KLING_API_KEY", "test-key")
|
||||
monkeypatch.setattr("tools.audio.kling_tts.KlingClient", lambda: FakeClient())
|
||||
monkeypatch.setattr("tools.audio.kling_tts.probe_duration", lambda path: 2.5)
|
||||
|
||||
result = KlingTTS().execute(
|
||||
{
|
||||
"text": "Hello",
|
||||
"voice_id": "voice-a",
|
||||
"output_path": str(tmp_path / "sync.mp3"),
|
||||
}
|
||||
)
|
||||
|
||||
assert result.success
|
||||
assert result.data["task_id"] == "tts-task-sync"
|
||||
assert result.data["remote_outputs"] == [{"url": "https://example.com/sync.mp3"}]
|
||||
assert result.data["audio_duration_seconds"] == 2.5
|
||||
|
||||
|
||||
def test_tts_selector_prefers_kling_official(monkeypatch):
|
||||
monkeypatch.setenv("KLING_API_KEY", "test-key")
|
||||
registry.clear()
|
||||
registry.register(KlingTTS())
|
||||
registry.register(TTSSelector())
|
||||
registry._discovered_packages.add("tools")
|
||||
|
||||
def fake_execute(self, inputs):
|
||||
from tools.base_tool import ToolResult
|
||||
|
||||
return ToolResult(success=True, data={"output_path": "out.mp3"}, artifacts=["out.mp3"])
|
||||
|
||||
monkeypatch.setattr(KlingTTS, "execute", fake_execute)
|
||||
result = registry.get("tts_selector").execute(
|
||||
{
|
||||
"text": "official speech",
|
||||
"voice_id": "voice-a",
|
||||
"preferred_provider": "kling_official",
|
||||
}
|
||||
)
|
||||
assert result.success
|
||||
assert result.data["selected_provider"] == "kling_official"
|
||||
|
|
@ -143,7 +143,7 @@ class TestCapabilityMetadata:
|
|||
catalog = reg.capability_catalog()
|
||||
assert "tts" in catalog
|
||||
providers = {item["provider"] for item in catalog["tts"] if item["provider"] != "selector"}
|
||||
assert providers == {"doubao", "elevenlabs", "google_tts", "openai", "piper"}
|
||||
assert providers == {"doubao", "elevenlabs", "google_tts", "kling_official", "openai", "piper"}
|
||||
|
||||
|
||||
# ---- Animated Explainer Pipeline ----
|
||||
|
|
|
|||
|
|
@ -0,0 +1,226 @@
|
|||
{
|
||||
"build_id": "97344324",
|
||||
"source_urls": [
|
||||
"https://kling.ai/document-api/api/get-started/authentication",
|
||||
"https://kling.ai/document-api/api/get-started/error-codes",
|
||||
"https://kling.ai/document-api/api/get-started/concurrency-rules",
|
||||
"https://kling.ai/document-api/api/video/3-0-turbo/text-to-video",
|
||||
"https://kling.ai/document-api/api/video/3-0-turbo/image-to-video",
|
||||
"https://kling.ai/document-api/api/video/3-0-omni/text-to-video",
|
||||
"https://kling.ai/document-api/api/video/3-0-omni/image-to-video",
|
||||
"https://kling.ai/document-api/api/video/3-0-omni/video-omni",
|
||||
"https://kling.ai/document-api/api/image/3-0-omni/image-generation",
|
||||
"https://kling.ai/document-api/api/image/3-0-omni/image-omni",
|
||||
"https://kling.ai/document-api/api/video/audio-generation/text-to-audio",
|
||||
"https://kling.ai/document-api/api/video/audio-generation/video-to-audio",
|
||||
"https://kling.ai/document-api/api/video/avatar",
|
||||
"https://kling.ai/document-api/api/video/lip-sync",
|
||||
"https://kling.ai/document-api/api/video/effects"
|
||||
],
|
||||
"chunk_names": [
|
||||
"index-B9E4in0e.js",
|
||||
"document-navigation-nxVgwiS5.js"
|
||||
],
|
||||
"extracted_at": "2026-07-03T08:12:55Z",
|
||||
"api_base": {
|
||||
"default": "https://api-singapore.klingai.com",
|
||||
"env_override": "KLING_API_BASE_URL",
|
||||
"auth_env": "KLING_API_KEY",
|
||||
"auth_header": "Authorization: Bearer <KLING_API_KEY>"
|
||||
},
|
||||
"endpoints": {
|
||||
"classic_text_to_video": {
|
||||
"method": "POST",
|
||||
"path": "/v1/videos/text2video",
|
||||
"poll": "GET /v1/videos/text2video/{id}"
|
||||
},
|
||||
"classic_image_to_video": {
|
||||
"method": "POST",
|
||||
"path": "/v1/videos/image2video",
|
||||
"poll": "GET /v1/videos/image2video/{id}"
|
||||
},
|
||||
"turbo_text_to_video": {
|
||||
"method": "POST",
|
||||
"path": "/text-to-video/kling-3.0-turbo",
|
||||
"poll": "GET /tasks?task_ids=<id>"
|
||||
},
|
||||
"turbo_image_to_video": {
|
||||
"method": "POST",
|
||||
"path": "/image-to-video/kling-3.0-turbo",
|
||||
"poll": "GET /tasks?task_ids=<id>"
|
||||
},
|
||||
"video_omni": {
|
||||
"method": "POST",
|
||||
"path": "/v1/videos/omni-video",
|
||||
"poll": "GET /v1/videos/omni-video/{id}"
|
||||
},
|
||||
"image_generation": {
|
||||
"method": "POST",
|
||||
"path": "/v1/images/generations",
|
||||
"poll": "GET /v1/images/generations/{id}"
|
||||
},
|
||||
"image_omni": {
|
||||
"method": "POST",
|
||||
"path": "/v1/images/omni-image",
|
||||
"poll": "GET /v1/images/omni-image/{id}"
|
||||
},
|
||||
"tts": {
|
||||
"method": "POST",
|
||||
"path": "/v1/audio/tts",
|
||||
"poll": "GET /v1/audio/tts/{id}"
|
||||
},
|
||||
"text_to_audio": {
|
||||
"method": "POST",
|
||||
"path": "/v1/audio/text-to-audio",
|
||||
"poll": "GET /v1/audio/text-to-audio/{id}"
|
||||
},
|
||||
"video_to_audio": {
|
||||
"method": "POST",
|
||||
"path": "/v1/audio/video-to-audio",
|
||||
"poll": "GET /v1/audio/video-to-audio/{id}"
|
||||
},
|
||||
"avatar_image_to_video": {
|
||||
"method": "POST",
|
||||
"path": "/v1/videos/avatar/image2video",
|
||||
"poll": "GET /v1/videos/avatar/image2video/{id}"
|
||||
},
|
||||
"identify_face": {
|
||||
"method": "POST",
|
||||
"path": "/v1/videos/identify-face"
|
||||
},
|
||||
"advanced_lip_sync": {
|
||||
"method": "POST",
|
||||
"path": "/v1/videos/advanced-lip-sync",
|
||||
"poll": "GET /v1/videos/advanced-lip-sync/{id}"
|
||||
},
|
||||
"video_effects": {
|
||||
"method": "POST",
|
||||
"path": "/v1/videos/effects",
|
||||
"poll": "GET /v1/videos/effects/{id}"
|
||||
}
|
||||
},
|
||||
"models": {
|
||||
"video": [
|
||||
"kling-v1",
|
||||
"kling-v1-5",
|
||||
"kling-v1-6",
|
||||
"kling-v2-master",
|
||||
"kling-v2-1",
|
||||
"kling-v2-1-master",
|
||||
"kling-v2-5-turbo",
|
||||
"kling-v2-6",
|
||||
"kling-v3",
|
||||
"kling-video-o1",
|
||||
"kling-v3-omni"
|
||||
],
|
||||
"image": [
|
||||
"kling-v1",
|
||||
"kling-v1-5",
|
||||
"kling-v2",
|
||||
"kling-v2-new",
|
||||
"kling-v2-1",
|
||||
"kling-v3",
|
||||
"kling-image-o1",
|
||||
"kling-v3-omni"
|
||||
]
|
||||
},
|
||||
"task_statuses": {
|
||||
"classic": [
|
||||
"submitted",
|
||||
"processing",
|
||||
"succeed",
|
||||
"failed"
|
||||
],
|
||||
"turbo": [
|
||||
"submitted",
|
||||
"processing",
|
||||
"succeeded",
|
||||
"failed"
|
||||
]
|
||||
},
|
||||
"result_paths": {
|
||||
"classic_created_id": "data.task_id",
|
||||
"turbo_created_id": "data.id",
|
||||
"classic_video_results": "data.task_result.videos[]",
|
||||
"classic_image_results": "data.task_result.images[]",
|
||||
"classic_audio_results": "data.task_result.audios[]",
|
||||
"identify_face_session": "data.session_id",
|
||||
"turbo_results": "data[0].outputs[]"
|
||||
},
|
||||
"core_field_enums": {
|
||||
"aspect_ratio": [
|
||||
"16:9",
|
||||
"9:16",
|
||||
"1:1"
|
||||
],
|
||||
"video_duration": [
|
||||
"3",
|
||||
"4",
|
||||
"5",
|
||||
"6",
|
||||
"7",
|
||||
"8",
|
||||
"9",
|
||||
"10",
|
||||
"11",
|
||||
"12",
|
||||
"13",
|
||||
"14",
|
||||
"15"
|
||||
],
|
||||
"video_resolution": [
|
||||
"720p",
|
||||
"1080p"
|
||||
],
|
||||
"image_resolution": [
|
||||
"1k",
|
||||
"2k",
|
||||
"4k"
|
||||
],
|
||||
"mode": [
|
||||
"std",
|
||||
"pro",
|
||||
"4k"
|
||||
],
|
||||
"sound": [
|
||||
"on",
|
||||
"off"
|
||||
],
|
||||
"image_aspect_ratio": [
|
||||
"16:9",
|
||||
"9:16",
|
||||
"1:1",
|
||||
"4:3",
|
||||
"3:4",
|
||||
"3:2",
|
||||
"2:3",
|
||||
"21:9",
|
||||
"auto"
|
||||
],
|
||||
"image_reference": [
|
||||
"subject",
|
||||
"face"
|
||||
],
|
||||
"image_result_type": [
|
||||
"single",
|
||||
"series"
|
||||
],
|
||||
"tts_voice_language": [
|
||||
"zh",
|
||||
"en"
|
||||
],
|
||||
"avatar_mode": [
|
||||
"std",
|
||||
"pro"
|
||||
],
|
||||
"lip_sync_operation": [
|
||||
"identify_face",
|
||||
"advanced_lip_sync",
|
||||
"full_lip_sync"
|
||||
]
|
||||
},
|
||||
"notes": [
|
||||
"Official docs were fetched as a SPA on 2026-07-03. Current HTML exposes buildId 97344324.",
|
||||
"The current entry/navigation bundles no longer expose the older api-*.js chunk names as literal references, so this fixture records the current entry assets and the Phase 1 core schema facts used by implementation and tests."
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
"""Shared helpers for Kling official API providers."""
|
||||
|
||||
from .client import KlingClient
|
||||
from .errors import KlingAPIError, is_retryable_kling_error
|
||||
|
||||
__all__ = ["KlingAPIError", "KlingClient", "is_retryable_kling_error"]
|
||||
|
|
@ -0,0 +1,121 @@
|
|||
"""Account usage diagnostics for Kling official API.
|
||||
|
||||
This module is a low-frequency helper, not an OpenMontage registry tool.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from hashlib import sha256
|
||||
from typing import Any
|
||||
|
||||
from .client import KlingClient
|
||||
from .errors import KlingAPIError
|
||||
|
||||
_CACHE: dict[tuple[tuple[str, str], ...], dict[str, Any]] = {}
|
||||
_LAST_QUERY_AT = 0.0
|
||||
|
||||
|
||||
def reset_account_usage_cache() -> None:
|
||||
"""Clear in-process account usage cache. Intended for tests."""
|
||||
|
||||
global _LAST_QUERY_AT
|
||||
_CACHE.clear()
|
||||
_LAST_QUERY_AT = 0.0
|
||||
|
||||
|
||||
def get_account_costs(
|
||||
*,
|
||||
start_time: str | None = None,
|
||||
end_time: str | None = None,
|
||||
resource_pack_name: str | None = None,
|
||||
client: KlingClient | None = None,
|
||||
ttl_seconds: float = 300.0,
|
||||
min_interval_seconds: float = 10.0,
|
||||
now: float | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Read `/account/costs` with in-process cache and throttle protection."""
|
||||
|
||||
global _LAST_QUERY_AT
|
||||
|
||||
timestamp = time.time() if now is None else now
|
||||
params = {
|
||||
key: value
|
||||
for key, value in {
|
||||
"start_time": start_time,
|
||||
"end_time": end_time,
|
||||
"resource_pack_name": resource_pack_name,
|
||||
}.items()
|
||||
if value
|
||||
}
|
||||
api = client or KlingClient()
|
||||
key = _cache_key(api, params)
|
||||
cached = _CACHE.get(key)
|
||||
if cached and timestamp - float(cached["fetched_at"]) <= ttl_seconds:
|
||||
return {**cached["payload"], "cached": True, "throttle_status": "cache_hit"}
|
||||
|
||||
if _LAST_QUERY_AT and timestamp - _LAST_QUERY_AT < min_interval_seconds:
|
||||
if cached:
|
||||
return {**cached["payload"], "cached": True, "throttle_status": "throttled_cache"}
|
||||
return {
|
||||
"provider": "kling_official",
|
||||
"queried_range": {
|
||||
"start_time": start_time,
|
||||
"end_time": end_time,
|
||||
"resource_pack_name": resource_pack_name,
|
||||
},
|
||||
"cached": False,
|
||||
"throttle_status": "throttled_no_cache",
|
||||
"message": "Account Usage is rate-limited; retry after the local throttle window.",
|
||||
}
|
||||
|
||||
raw = api.get("/account/costs", params=params)
|
||||
data = raw.get("data") if isinstance(raw, dict) else {}
|
||||
if not isinstance(data, dict):
|
||||
data = {}
|
||||
payload = {
|
||||
"provider": "kling_official",
|
||||
"queried_range": {
|
||||
"start_time": start_time,
|
||||
"end_time": end_time,
|
||||
"resource_pack_name": resource_pack_name,
|
||||
},
|
||||
"resource_pack_subscribe_infos": data.get("resource_pack_subscribe_infos", []),
|
||||
"raw": raw,
|
||||
"cached": False,
|
||||
"throttle_status": "fresh",
|
||||
}
|
||||
_CACHE[key] = {"fetched_at": timestamp, "payload": payload}
|
||||
_LAST_QUERY_AT = timestamp
|
||||
return payload
|
||||
|
||||
|
||||
def _cache_key(client: Any, params: dict[str, Any]) -> tuple[tuple[str, str], ...]:
|
||||
"""Scope Account Usage cache by request params and account endpoint identity."""
|
||||
|
||||
api_key = getattr(client, "api_key", None) or ""
|
||||
api_key_hash = sha256(str(api_key).encode("utf-8")).hexdigest() if api_key else ""
|
||||
scope = {
|
||||
"base_url": getattr(client, "base_url", ""),
|
||||
"api_key_sha256": api_key_hash,
|
||||
**{name: str(value) for name, value in params.items()},
|
||||
}
|
||||
return tuple(sorted((name, str(value)) for name, value in scope.items()))
|
||||
|
||||
|
||||
def account_usage_hint_for_error(error: KlingAPIError) -> dict[str, Any]:
|
||||
"""Return a diagnostic hint for balance/resource-pack related errors."""
|
||||
|
||||
code = str(error.code) if error.code is not None else ""
|
||||
if code not in {"1101", "1102"}:
|
||||
return {}
|
||||
return {
|
||||
"provider": "kling_official",
|
||||
"reason": "account_balance_or_resource_pack",
|
||||
"message": (
|
||||
"Kling returned an account/resource-pack error. Use tools._kling.account.get_account_costs() "
|
||||
"for a low-frequency account usage diagnostic, or check the Kling Open Platform console."
|
||||
),
|
||||
"error_code": error.code,
|
||||
"request_id": error.request_id,
|
||||
}
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
"""Callback validation helpers for Kling official providers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from urllib.parse import urlparse
|
||||
|
||||
|
||||
def validate_callback_url(callback_url: str | None) -> str | None:
|
||||
"""Return a normalized callback URL or raise for obviously invalid input."""
|
||||
|
||||
if not callback_url:
|
||||
return None
|
||||
value = str(callback_url).strip()
|
||||
parsed = urlparse(value)
|
||||
if parsed.scheme not in {"http", "https"} or not parsed.netloc:
|
||||
raise ValueError("callback_url must be an absolute http(s) URL")
|
||||
return value
|
||||
|
|
@ -0,0 +1,216 @@
|
|||
"""HTTP client and task parsers for Kling official API providers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.parse import urljoin
|
||||
|
||||
import requests
|
||||
|
||||
from .errors import KlingAPIError, is_retryable_kling_error
|
||||
from .schemas import (
|
||||
CLASSIC_FAILURE_STATUS,
|
||||
CLASSIC_PENDING_STATUSES,
|
||||
CLASSIC_SUCCESS_STATUS,
|
||||
DEFAULT_API_BASE_URL,
|
||||
TURBO_FAILURE_STATUS,
|
||||
TURBO_PENDING_STATUSES,
|
||||
TURBO_SUCCESS_STATUS,
|
||||
)
|
||||
|
||||
|
||||
class KlingClient:
|
||||
"""Small synchronous client for the official Kling API."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
api_key: str | None = None,
|
||||
base_url: str | None = None,
|
||||
session: Any | None = None,
|
||||
max_retries: int = 2,
|
||||
) -> None:
|
||||
self.api_key = api_key if api_key is not None else os.environ.get("KLING_API_KEY")
|
||||
self.base_url = (base_url or os.environ.get("KLING_API_BASE_URL") or DEFAULT_API_BASE_URL).rstrip("/")
|
||||
self.session = session or requests.Session()
|
||||
self.max_retries = max_retries
|
||||
|
||||
@property
|
||||
def headers(self) -> dict[str, str]:
|
||||
if not self.api_key:
|
||||
raise KlingAPIError(
|
||||
"KLING_API_KEY is not set. Configure KLING_API_KEY for official Kling API access.",
|
||||
http_status=401,
|
||||
)
|
||||
return {
|
||||
"Authorization": f"Bearer {self.api_key}",
|
||||
"Accept": "application/json",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
def post(self, path: str, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
return self._request("post", path, json=payload)
|
||||
|
||||
def get(self, path: str, params: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
return self._request("get", path, params=params)
|
||||
|
||||
def download(self, url: str, output_path: Path, timeout: int = 180) -> Path:
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
response = self.session.get(url, timeout=timeout)
|
||||
self._raise_for_http_error(response)
|
||||
content = getattr(response, "content", None)
|
||||
if content is None and hasattr(response, "iter_content"):
|
||||
content = b"".join(chunk for chunk in response.iter_content(chunk_size=1024 * 128) if chunk)
|
||||
output_path.write_bytes(content or b"")
|
||||
return output_path
|
||||
|
||||
def create_classic_task(self, path: str, payload: dict[str, Any]) -> str:
|
||||
data = self.post(path, payload)
|
||||
task_id = ((data.get("data") or {}).get("task_id"))
|
||||
if not task_id:
|
||||
raise KlingAPIError(f"Kling Classic create response missing data.task_id: {data}")
|
||||
return str(task_id)
|
||||
|
||||
def poll_classic(
|
||||
self,
|
||||
path: str,
|
||||
task_id: str,
|
||||
result_key: str,
|
||||
timeout_seconds: int = 900,
|
||||
poll_interval: float = 5.0,
|
||||
) -> list[dict[str, Any]]:
|
||||
deadline = time.time() + timeout_seconds
|
||||
while time.time() < deadline:
|
||||
data = self.get(f"{path.rstrip('/')}/{task_id}")
|
||||
payload = data.get("data") or {}
|
||||
status = payload.get("task_status") or payload.get("status")
|
||||
if status == CLASSIC_SUCCESS_STATUS:
|
||||
task_result = payload.get("task_result") or {}
|
||||
outputs = task_result.get(result_key) or []
|
||||
if not isinstance(outputs, list):
|
||||
raise KlingAPIError(f"Kling Classic result path data.task_result.{result_key} is not a list")
|
||||
return outputs
|
||||
if status == CLASSIC_FAILURE_STATUS:
|
||||
message = payload.get("task_status_msg") or payload.get("message") or "Kling Classic task failed"
|
||||
raise KlingAPIError(str(message), code=payload.get("task_status"), response=data)
|
||||
if status not in CLASSIC_PENDING_STATUSES:
|
||||
raise KlingAPIError(f"Unexpected Kling Classic task status {status!r}", response=data)
|
||||
time.sleep(min(poll_interval, max(0.0, deadline - time.time())))
|
||||
raise TimeoutError(f"Kling Classic task {task_id} timed out after {timeout_seconds}s")
|
||||
|
||||
def create_turbo(self, path: str, payload: dict[str, Any]) -> str:
|
||||
data = self.post(path, payload)
|
||||
task_id = ((data.get("data") or {}).get("id"))
|
||||
if not task_id:
|
||||
raise KlingAPIError(f"Kling Turbo create response missing data.id: {data}")
|
||||
return str(task_id)
|
||||
|
||||
def poll_turbo(
|
||||
self,
|
||||
task_id: str,
|
||||
timeout_seconds: int = 900,
|
||||
poll_interval: float = 5.0,
|
||||
) -> list[dict[str, Any]]:
|
||||
deadline = time.time() + timeout_seconds
|
||||
while time.time() < deadline:
|
||||
data = self.get("/tasks", params={"task_ids": task_id})
|
||||
records = data.get("data") or []
|
||||
if not records:
|
||||
raise KlingAPIError(f"Kling Turbo poll response missing data[0]: {data}")
|
||||
record = records[0]
|
||||
status = record.get("status") or record.get("task_status")
|
||||
if status == TURBO_SUCCESS_STATUS:
|
||||
outputs = record.get("outputs") or []
|
||||
if not isinstance(outputs, list):
|
||||
raise KlingAPIError("Kling Turbo result path data[0].outputs is not a list")
|
||||
return outputs
|
||||
if status == TURBO_FAILURE_STATUS:
|
||||
message = record.get("message") or record.get("error") or "Kling Turbo task failed"
|
||||
raise KlingAPIError(str(message), code=record.get("code"), request_id=record.get("request_id"), response=data)
|
||||
if status not in TURBO_PENDING_STATUSES:
|
||||
raise KlingAPIError(f"Unexpected Kling Turbo task status {status!r}", response=data)
|
||||
time.sleep(min(poll_interval, max(0.0, deadline - time.time())))
|
||||
raise TimeoutError(f"Kling Turbo task {task_id} timed out after {timeout_seconds}s")
|
||||
|
||||
def _request(self, method: str, path: str, **kwargs: Any) -> dict[str, Any]:
|
||||
url = self._url(path)
|
||||
last_error: KlingAPIError | None = None
|
||||
for attempt in range(self.max_retries + 1):
|
||||
try:
|
||||
response = getattr(self.session, method)(url, headers=self.headers, timeout=30, **kwargs)
|
||||
self._raise_for_http_error(response)
|
||||
data = response.json()
|
||||
self._raise_for_business_error(data)
|
||||
return data
|
||||
except KlingAPIError as error:
|
||||
last_error = error
|
||||
if attempt >= self.max_retries or not is_retryable_kling_error(error):
|
||||
raise
|
||||
time.sleep(min(2.0 * (attempt + 1), 8.0))
|
||||
except requests.RequestException as exc:
|
||||
last_error = KlingAPIError(str(exc))
|
||||
if attempt >= self.max_retries:
|
||||
raise last_error from exc
|
||||
time.sleep(min(2.0 * (attempt + 1), 8.0))
|
||||
raise last_error or KlingAPIError("Kling API request failed")
|
||||
|
||||
def _url(self, path: str) -> str:
|
||||
if path.startswith("http://") or path.startswith("https://"):
|
||||
return path
|
||||
return urljoin(f"{self.base_url}/", path.lstrip("/"))
|
||||
|
||||
def _raise_for_http_error(self, response: Any) -> None:
|
||||
status = getattr(response, "status_code", None)
|
||||
if status is not None and 200 <= int(status) < 300:
|
||||
return
|
||||
code = None
|
||||
message = None
|
||||
request_id = None
|
||||
body: dict[str, Any] | None = None
|
||||
try:
|
||||
body = response.json()
|
||||
code = body.get("code")
|
||||
message = body.get("message") or body.get("msg")
|
||||
request_id = body.get("request_id") or body.get("requestId")
|
||||
except Exception:
|
||||
text = getattr(response, "text", "")
|
||||
message = text[:500] if text else f"HTTP {status}"
|
||||
raise self._format_error(
|
||||
code=code,
|
||||
message=message or f"HTTP {status}",
|
||||
request_id=request_id,
|
||||
http_status=int(status) if status is not None else None,
|
||||
response=body,
|
||||
)
|
||||
|
||||
def _raise_for_business_error(self, data: dict[str, Any]) -> None:
|
||||
code = data.get("code")
|
||||
if code in (None, 0, "0"):
|
||||
return
|
||||
raise self._format_error(
|
||||
code=code,
|
||||
message=str(data.get("message") or data.get("msg") or "Kling API returned an error"),
|
||||
request_id=data.get("request_id") or data.get("requestId"),
|
||||
response=data,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _format_error(
|
||||
*,
|
||||
code: str | int | None,
|
||||
message: str,
|
||||
request_id: str | None = None,
|
||||
http_status: int | None = None,
|
||||
response: dict[str, Any] | None = None,
|
||||
) -> KlingAPIError:
|
||||
if str(code) == "1303" and "并发/资源包限制" not in message:
|
||||
message = f"{message} (并发/资源包限制: parallel task over resource pack limit)"
|
||||
return KlingAPIError(
|
||||
message=message,
|
||||
code=code,
|
||||
request_id=request_id,
|
||||
http_status=http_status,
|
||||
response=response,
|
||||
)
|
||||
|
|
@ -0,0 +1,86 @@
|
|||
"""Element reference helpers for Kling official Omni providers.
|
||||
|
||||
These helpers intentionally do not inherit from BaseTool. Elements are an
|
||||
internal provider reference mechanism in Phase 2, not a registry capability.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from .client import KlingClient
|
||||
|
||||
|
||||
def normalize_element_list(element_list: Any | None) -> list[dict[str, int]]:
|
||||
"""Normalize official Kling element references to element_list objects."""
|
||||
|
||||
if not element_list:
|
||||
return []
|
||||
if not isinstance(element_list, list):
|
||||
raise ValueError("element_list must be a list of element ids or objects")
|
||||
|
||||
normalized: list[dict[str, int]] = []
|
||||
for item in element_list:
|
||||
raw_id: Any
|
||||
if isinstance(item, dict):
|
||||
raw_id = item.get("element_id", item.get("id"))
|
||||
else:
|
||||
raw_id = item
|
||||
if raw_id is None:
|
||||
raise ValueError("each element_list item must include element_id")
|
||||
try:
|
||||
element_id = int(raw_id)
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise ValueError(f"element_id must be an integer-compatible value: {raw_id!r}") from exc
|
||||
if element_id <= 0:
|
||||
raise ValueError("element_id must be positive")
|
||||
normalized.append({"element_id": element_id})
|
||||
return normalized
|
||||
|
||||
|
||||
def element_ids(element_list: Any | None) -> list[int]:
|
||||
"""Return normalized element ids from an element reference list."""
|
||||
|
||||
return [item["element_id"] for item in normalize_element_list(element_list)]
|
||||
|
||||
|
||||
def get_custom_element(element_id: int, client: KlingClient | None = None) -> dict[str, Any]:
|
||||
"""Fetch one custom element for validation or diagnostics."""
|
||||
|
||||
api = client or KlingClient()
|
||||
return api.get(f"/v1/general/advanced-custom-elements/{int(element_id)}")
|
||||
|
||||
|
||||
def list_custom_elements(
|
||||
client: KlingClient | None = None,
|
||||
params: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""List custom elements without creating or deleting assets."""
|
||||
|
||||
api = client or KlingClient()
|
||||
return api.get("/v1/general/advanced-custom-elements", params=params)
|
||||
|
||||
|
||||
def list_preset_elements(
|
||||
client: KlingClient | None = None,
|
||||
params: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""List official preset elements without entering the tool registry."""
|
||||
|
||||
api = client or KlingClient()
|
||||
return api.get("/v1/general/advanced-presets-elements", params=params)
|
||||
|
||||
|
||||
def write_elements_artifact(
|
||||
artifact_path: str | Path,
|
||||
elements: list[dict[str, Any]],
|
||||
) -> Path:
|
||||
"""Write element metadata in the Phase 2 reproducibility artifact shape."""
|
||||
|
||||
path = Path(artifact_path)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
payload = {"provider": "kling_official", "elements": elements}
|
||||
path.write_text(json.dumps(payload, indent=2, ensure_ascii=False), encoding="utf-8")
|
||||
return path
|
||||
|
|
@ -0,0 +1,46 @@
|
|||
"""Error types and retry policy for Kling official API calls."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
|
||||
@dataclass
|
||||
class KlingAPIError(Exception):
|
||||
"""Structured error returned by the Kling official API."""
|
||||
|
||||
message: str
|
||||
code: str | int | None = None
|
||||
request_id: str | None = None
|
||||
http_status: int | None = None
|
||||
response: dict[str, Any] | None = None
|
||||
|
||||
def __str__(self) -> str:
|
||||
parts = [self.message]
|
||||
if self.code is not None:
|
||||
parts.append(f"code={self.code}")
|
||||
if self.request_id:
|
||||
parts.append(f"request_id={self.request_id}")
|
||||
if self.http_status is not None:
|
||||
parts.append(f"http_status={self.http_status}")
|
||||
return " | ".join(parts)
|
||||
|
||||
|
||||
_RETRYABLE_CODES = {"1302", "1303", "5000", "5001", "5002"}
|
||||
_RETRYABLE_HTTP = {500, 503, 504}
|
||||
|
||||
|
||||
def _code_str(code: str | int | None) -> str | None:
|
||||
if code is None:
|
||||
return None
|
||||
return str(code)
|
||||
|
||||
|
||||
def is_retryable_kling_error(error: KlingAPIError) -> bool:
|
||||
"""Return whether an official Kling error is safe for limited retry."""
|
||||
|
||||
code = _code_str(error.code)
|
||||
if code in _RETRYABLE_CODES:
|
||||
return True
|
||||
return error.http_status in _RETRYABLE_HTTP
|
||||
|
|
@ -0,0 +1,111 @@
|
|||
"""Media normalization and download helpers for Kling official providers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import mimetypes
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlparse
|
||||
|
||||
|
||||
def strip_data_uri_prefix(value: str | None) -> str | None:
|
||||
"""Return raw base64/content by removing a data URI prefix if present."""
|
||||
|
||||
if value is None:
|
||||
return None
|
||||
marker = ";base64,"
|
||||
if value.startswith("data:") and marker in value:
|
||||
return value.split(marker, 1)[1]
|
||||
return value
|
||||
|
||||
|
||||
def image_file_to_raw_base64(path: str | Path) -> str:
|
||||
"""Read a local image file and return raw base64 without data URI prefix."""
|
||||
|
||||
image_path = Path(path)
|
||||
if not image_path.is_file():
|
||||
raise FileNotFoundError(f"Image not found: {image_path}")
|
||||
return base64.b64encode(image_path.read_bytes()).decode("ascii")
|
||||
|
||||
|
||||
def file_to_raw_base64(path: str | Path, *, label: str = "File") -> str:
|
||||
"""Read a local media file and return raw base64 without a data URI prefix."""
|
||||
|
||||
media_path = Path(path)
|
||||
if not media_path.is_file():
|
||||
raise FileNotFoundError(f"{label} not found: {media_path}")
|
||||
return base64.b64encode(media_path.read_bytes()).decode("ascii")
|
||||
|
||||
|
||||
def normalize_image_input(url: str | None = None, path: str | Path | None = None) -> str | None:
|
||||
"""Normalize a Kling image input to either URL or raw base64."""
|
||||
|
||||
if url:
|
||||
return strip_data_uri_prefix(url)
|
||||
if path:
|
||||
return image_file_to_raw_base64(path)
|
||||
return None
|
||||
|
||||
|
||||
def normalize_media_input(
|
||||
url: str | None = None,
|
||||
path: str | Path | None = None,
|
||||
value: str | None = None,
|
||||
*,
|
||||
label: str = "Media file",
|
||||
) -> str | None:
|
||||
"""Normalize a generic Kling media input to URL, raw base64, or raw provided value."""
|
||||
|
||||
if value:
|
||||
return strip_data_uri_prefix(value)
|
||||
if url:
|
||||
return strip_data_uri_prefix(url)
|
||||
if path:
|
||||
return file_to_raw_base64(path, label=label)
|
||||
return None
|
||||
|
||||
|
||||
def extension_from_url(url: str | None, default: str = ".png") -> str:
|
||||
"""Infer a file extension from a URL path."""
|
||||
|
||||
if not url:
|
||||
return default
|
||||
suffix = Path(urlparse(url).path).suffix.lower()
|
||||
if suffix in {
|
||||
".png",
|
||||
".jpg",
|
||||
".jpeg",
|
||||
".webp",
|
||||
".gif",
|
||||
".mp4",
|
||||
".mov",
|
||||
".m4v",
|
||||
".mp3",
|
||||
".wav",
|
||||
".m4a",
|
||||
".aac",
|
||||
".ogg",
|
||||
".opus",
|
||||
}:
|
||||
return suffix
|
||||
return default
|
||||
|
||||
|
||||
def extension_from_content_type(content_type: str | None, default: str = ".png") -> str:
|
||||
if not content_type:
|
||||
return default
|
||||
ext = mimetypes.guess_extension(content_type.split(";", 1)[0].strip())
|
||||
return ext or default
|
||||
|
||||
|
||||
def output_path_with_suffix(path: str | Path, suffix: str) -> Path:
|
||||
output_path = Path(path)
|
||||
if output_path.suffix:
|
||||
return output_path
|
||||
return output_path.with_suffix(suffix)
|
||||
|
||||
|
||||
def numbered_output_path(first_path: Path, index: int, suffix: str) -> Path:
|
||||
if index == 0:
|
||||
return output_path_with_suffix(first_path, suffix)
|
||||
return first_path.with_name(f"{first_path.stem}_{index + 1}{suffix}")
|
||||
|
|
@ -0,0 +1,40 @@
|
|||
"""Omni reference helpers for Kling official providers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
PLACEHOLDER_RE = re.compile(r"<<<image_(\d+)>>>")
|
||||
|
||||
|
||||
def build_image_prompt_references(
|
||||
prompt: str,
|
||||
image_list: list[dict[str, Any]],
|
||||
) -> tuple[str, list[dict[str, Any]]]:
|
||||
"""Bind image_list entries to stable Image Omni placeholders."""
|
||||
|
||||
references = [
|
||||
{
|
||||
"index": index,
|
||||
"placeholder": f"<<<image_{index}>>>",
|
||||
"source": item.get("source") or item.get("image") or item.get("image_url"),
|
||||
"source_type": item.get("source_type", "unknown"),
|
||||
}
|
||||
for index, item in enumerate(image_list, start=1)
|
||||
]
|
||||
if not references:
|
||||
return prompt, []
|
||||
|
||||
existing_numbers = [int(value) for value in PLACEHOLDER_RE.findall(prompt)]
|
||||
if existing_numbers:
|
||||
if max(existing_numbers) > len(references):
|
||||
raise ValueError(
|
||||
f"prompt references <<<image_{max(existing_numbers)}>>> but only {len(references)} image(s) were provided"
|
||||
)
|
||||
if min(existing_numbers) < 1:
|
||||
raise ValueError("Image Omni prompt placeholders must start at <<<image_1>>>")
|
||||
return prompt, references
|
||||
|
||||
placeholders = " ".join(item["placeholder"] for item in references)
|
||||
return f"{prompt}\nReferences: {placeholders}", references
|
||||
|
|
@ -0,0 +1,121 @@
|
|||
"""Lightweight schema constants for Kling official providers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from typing import Any
|
||||
|
||||
|
||||
DEFAULT_API_BASE_URL = "https://api-singapore.klingai.com"
|
||||
|
||||
|
||||
class KlingProtocol(str, Enum):
|
||||
CLASSIC = "classic"
|
||||
TURBO = "turbo"
|
||||
|
||||
|
||||
CLASSIC_PENDING_STATUSES = {"submitted", "processing"}
|
||||
CLASSIC_SUCCESS_STATUS = "succeed"
|
||||
CLASSIC_FAILURE_STATUS = "failed"
|
||||
CLASSIC_STATUSES = [
|
||||
"submitted",
|
||||
"processing",
|
||||
"succeed",
|
||||
"failed",
|
||||
]
|
||||
|
||||
TURBO_PENDING_STATUSES = {"submitted", "processing"}
|
||||
TURBO_SUCCESS_STATUS = "succeeded"
|
||||
TURBO_FAILURE_STATUS = "failed"
|
||||
TURBO_STATUSES = [
|
||||
"submitted",
|
||||
"processing",
|
||||
"succeeded",
|
||||
"failed",
|
||||
]
|
||||
|
||||
VIDEO_MODELS = [
|
||||
"kling-v1",
|
||||
"kling-v1-5",
|
||||
"kling-v1-6",
|
||||
"kling-v2-master",
|
||||
"kling-v2-1",
|
||||
"kling-v2-1-master",
|
||||
"kling-v2-5-turbo",
|
||||
"kling-v2-6",
|
||||
"kling-v3",
|
||||
"kling-video-o1",
|
||||
"kling-v3-omni",
|
||||
]
|
||||
CLASSIC_VIDEO_MODELS = [
|
||||
"kling-v1",
|
||||
"kling-v1-5",
|
||||
"kling-v1-6",
|
||||
"kling-v2-master",
|
||||
"kling-v2-1",
|
||||
"kling-v2-1-master",
|
||||
"kling-v2-5-turbo",
|
||||
"kling-v2-6",
|
||||
"kling-v3",
|
||||
]
|
||||
OMNI_VIDEO_MODELS = ["kling-video-o1", "kling-v3-omni"]
|
||||
|
||||
IMAGE_MODELS = [
|
||||
"kling-v1",
|
||||
"kling-v1-5",
|
||||
"kling-v2",
|
||||
"kling-v2-new",
|
||||
"kling-v2-1",
|
||||
"kling-v3",
|
||||
"kling-image-o1",
|
||||
"kling-v3-omni",
|
||||
]
|
||||
IMAGE_GENERATION_MODELS = [
|
||||
"kling-v1",
|
||||
"kling-v1-5",
|
||||
"kling-v2",
|
||||
"kling-v2-new",
|
||||
"kling-v2-1",
|
||||
"kling-v3",
|
||||
]
|
||||
OMNI_IMAGE_MODELS = ["kling-image-o1", "kling-v3-omni"]
|
||||
|
||||
VIDEO_DURATIONS = [str(value) for value in range(3, 16)]
|
||||
VIDEO_ASPECT_RATIOS = ["16:9", "9:16", "1:1"]
|
||||
VIDEO_RESOLUTIONS = ["720p", "1080p"]
|
||||
VIDEO_MODES = ["std", "pro", "4k"]
|
||||
SOUND_VALUES = ["on", "off"]
|
||||
|
||||
IMAGE_RESOLUTIONS = ["1k", "2k", "4k"]
|
||||
IMAGE_ASPECT_RATIOS = ["16:9", "9:16", "1:1", "4:3", "3:4", "3:2", "2:3", "21:9", "auto"]
|
||||
IMAGE_REFERENCE_TYPES = ["subject", "face"]
|
||||
IMAGE_RESULT_TYPES = ["single", "series"]
|
||||
|
||||
RESULT_PATHS = {
|
||||
"classic_video": "data.task_result.videos[]",
|
||||
"classic_image": "data.task_result.images[]",
|
||||
"classic_audio": "data.task_result.audios[]",
|
||||
"turbo": "data[0].outputs[]",
|
||||
}
|
||||
|
||||
TTS_LANGUAGES = ["zh", "en"]
|
||||
TTS_SPEED_MIN = 0.5
|
||||
TTS_SPEED_MAX = 2.0
|
||||
|
||||
AVATAR_MODES = ["std", "pro"]
|
||||
LIP_SYNC_OPERATIONS = ["identify_face", "advanced_lip_sync", "full_lip_sync"]
|
||||
|
||||
|
||||
@dataclass
|
||||
class ClassicTaskResult:
|
||||
task_id: str
|
||||
status: str
|
||||
outputs: list[dict[str, Any]]
|
||||
|
||||
|
||||
@dataclass
|
||||
class TurboTaskResult:
|
||||
task_id: str
|
||||
status: str
|
||||
outputs: list[dict[str, Any]]
|
||||
|
|
@ -0,0 +1,340 @@
|
|||
"""Kling official API text-to-speech provider."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from tools._kling.account import account_usage_hint_for_error, get_account_costs
|
||||
from tools._kling.callbacks import validate_callback_url
|
||||
from tools._kling.client import KlingClient
|
||||
from tools._kling.errors import KlingAPIError
|
||||
from tools._kling.media import extension_from_url, numbered_output_path, output_path_with_suffix
|
||||
from tools._kling.schemas import TTS_LANGUAGES, TTS_SPEED_MAX, TTS_SPEED_MIN
|
||||
from tools.analysis.audio_probe import probe_duration
|
||||
from tools.base_tool import (
|
||||
BaseTool,
|
||||
DependencyError,
|
||||
Determinism,
|
||||
ExecutionMode,
|
||||
ResourceProfile,
|
||||
RetryPolicy,
|
||||
ToolResult,
|
||||
ToolRuntime,
|
||||
ToolStability,
|
||||
ToolTier,
|
||||
)
|
||||
|
||||
|
||||
class KlingTTS(BaseTool):
|
||||
name = "kling_tts"
|
||||
version = "0.1.0"
|
||||
tier = ToolTier.VOICE
|
||||
capability = "tts"
|
||||
provider = "kling_official"
|
||||
stability = ToolStability.EXPERIMENTAL
|
||||
execution_mode = ExecutionMode.SYNC
|
||||
determinism = Determinism.STOCHASTIC
|
||||
runtime = ToolRuntime.API
|
||||
|
||||
dependencies = ["env:KLING_API_KEY"]
|
||||
install_instructions = (
|
||||
"Set KLING_API_KEY in .env for the official Kling API. "
|
||||
"Pass voice_id explicitly; OpenMontage does not guess Kling voice IDs."
|
||||
)
|
||||
agent_skills = ["kling-official", "text-to-speech"]
|
||||
|
||||
capabilities = ["text_to_speech", "voice_selection", "multilingual"]
|
||||
supports = {
|
||||
"multilingual": True,
|
||||
"voice_selection": True,
|
||||
"offline": False,
|
||||
"native_audio": True,
|
||||
}
|
||||
best_for = [
|
||||
"official Kling text-to-speech",
|
||||
"Chinese or English narration when a Kling voice_id is known",
|
||||
"keeping narration provider provenance inside the Kling official account",
|
||||
]
|
||||
not_good_for = [
|
||||
"fully offline narration",
|
||||
"voice cloning without a configured official voice_id",
|
||||
"auto-discovering voices",
|
||||
]
|
||||
fallback_tools = ["doubao_tts", "elevenlabs_tts", "openai_tts", "google_tts", "piper_tts"]
|
||||
|
||||
input_schema = {
|
||||
"type": "object",
|
||||
"required": ["text", "voice_id"],
|
||||
"properties": {
|
||||
"text": {"type": "string"},
|
||||
"voice_id": {
|
||||
"type": "string",
|
||||
"description": "Official Kling voice ID. Required; do not rely on an unknown default.",
|
||||
},
|
||||
"voice_language": {"type": "string", "enum": TTS_LANGUAGES, "default": "en"},
|
||||
"voice_speed": {
|
||||
"type": "number",
|
||||
"minimum": TTS_SPEED_MIN,
|
||||
"maximum": TTS_SPEED_MAX,
|
||||
"default": 1.0,
|
||||
},
|
||||
"callback_url": {"type": "string"},
|
||||
"external_task_id": {"type": "string"},
|
||||
"include_account_usage": {
|
||||
"type": "boolean",
|
||||
"default": False,
|
||||
"description": "Optional low-frequency account usage diagnostic; not used by default.",
|
||||
},
|
||||
"timeout_seconds": {"type": "integer", "default": 300},
|
||||
"poll_interval": {"type": "number", "default": 3.0},
|
||||
"output_path": {"type": "string"},
|
||||
},
|
||||
}
|
||||
|
||||
output_schema = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"output": {"type": "string"},
|
||||
"output_path": {"type": "string"},
|
||||
"audio_paths": {"type": "array"},
|
||||
"task_id": {"type": "string"},
|
||||
},
|
||||
}
|
||||
|
||||
resource_profile = ResourceProfile(
|
||||
cpu_cores=1, ram_mb=256, vram_mb=0, disk_mb=100, network_required=True
|
||||
)
|
||||
retry_policy = RetryPolicy(
|
||||
max_retries=2,
|
||||
backoff_seconds=2.0,
|
||||
retryable_errors=["1302", "1303", "5000", "5001", "5002"],
|
||||
)
|
||||
idempotency_key_fields = ["text", "voice_id", "voice_language", "voice_speed"]
|
||||
side_effects = ["paid remote generation via official Kling API", "writes audio file to output_path"]
|
||||
user_visible_verification = ["Listen to generated audio for voice, language, and pacing"]
|
||||
quality_score = 0.78
|
||||
latency_p50_seconds = 20.0
|
||||
|
||||
def estimate_cost(self, inputs: dict[str, Any]) -> float:
|
||||
text_length = len(str(inputs.get("text") or ""))
|
||||
return round(max(text_length, 1) * 0.000018, 4)
|
||||
|
||||
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
|
||||
return 30.0
|
||||
|
||||
def dry_run(self, inputs: dict[str, Any]) -> dict[str, Any]:
|
||||
result = super().dry_run(inputs)
|
||||
result.update(
|
||||
{
|
||||
"paid_api": True,
|
||||
"cost_estimate_confidence": "low",
|
||||
"cost_estimate_basis": "Conservative character-based OpenMontage estimate pending official account-usage reconciliation.",
|
||||
}
|
||||
)
|
||||
return result
|
||||
|
||||
def execute(self, inputs: dict[str, Any]) -> ToolResult:
|
||||
try:
|
||||
self.check_dependencies()
|
||||
except DependencyError as exc:
|
||||
return ToolResult(success=False, error=str(exc))
|
||||
|
||||
start = time.time()
|
||||
try:
|
||||
request = self._build_request(inputs)
|
||||
client = KlingClient()
|
||||
task_id, outputs = self._create_and_collect_audios(client, request, inputs)
|
||||
paths = self._download_audios(client, outputs, inputs)
|
||||
audio_duration = probe_duration(paths[0])
|
||||
except (KlingAPIError, TimeoutError, ValueError, KeyError, FileNotFoundError) as exc:
|
||||
data: dict[str, Any] = {"provider": self.provider}
|
||||
if isinstance(exc, KlingAPIError):
|
||||
data.update(
|
||||
{
|
||||
"error_code": exc.code,
|
||||
"request_id": exc.request_id,
|
||||
"http_status": exc.http_status,
|
||||
"account_usage_diagnostic": account_usage_hint_for_error(exc),
|
||||
}
|
||||
)
|
||||
return ToolResult(success=False, data=data, error=f"Kling official TTS failed: {exc}")
|
||||
except Exception as exc:
|
||||
return ToolResult(success=False, data={"provider": self.provider}, error=f"Kling official TTS failed: {exc}")
|
||||
|
||||
return ToolResult(
|
||||
success=True,
|
||||
data={
|
||||
"provider": self.provider,
|
||||
"model": "kling-official-tts",
|
||||
"task_id": task_id,
|
||||
"operation": "text_to_speech",
|
||||
"text_length": len(request["payload"]["text"]),
|
||||
"voice_id": request["payload"]["voice_id"],
|
||||
"voice_language": request["payload"].get("voice_language"),
|
||||
"voice_speed": request["payload"].get("voice_speed"),
|
||||
"remote_outputs": outputs,
|
||||
"output": str(paths[0]),
|
||||
"output_path": str(paths[0]),
|
||||
"audio_paths": [str(path) for path in paths],
|
||||
"format": paths[0].suffix.lstrip(".") or "mp3",
|
||||
"audio_duration_seconds": round(audio_duration, 2) if audio_duration else None,
|
||||
"cost_estimate_confidence": "low",
|
||||
"cost_estimate_basis": "Conservative estimate pending official account-usage reconciliation.",
|
||||
**self._account_usage_result(inputs, client),
|
||||
**self._callback_result_data(inputs, task_id),
|
||||
},
|
||||
artifacts=[str(path) for path in paths],
|
||||
cost_usd=self.estimate_cost(inputs),
|
||||
duration_seconds=round(time.time() - start, 2),
|
||||
model="kling-official-tts",
|
||||
)
|
||||
|
||||
def _build_request(self, inputs: dict[str, Any]) -> dict[str, Any]:
|
||||
text = str(inputs.get("text") or "").strip()
|
||||
if not text:
|
||||
raise ValueError("text is required")
|
||||
if len(text) > 5000:
|
||||
raise ValueError("text exceeds Kling TTS safety limit of 5000 characters")
|
||||
|
||||
voice_id = str(inputs.get("voice_id") or "").strip()
|
||||
if not voice_id:
|
||||
raise ValueError("voice_id is required for Kling official TTS")
|
||||
|
||||
voice_language = str(inputs.get("voice_language") or "en")
|
||||
if voice_language not in TTS_LANGUAGES:
|
||||
raise ValueError(f"voice_language must be one of: {', '.join(TTS_LANGUAGES)}")
|
||||
|
||||
voice_speed = float(inputs.get("voice_speed", 1.0))
|
||||
if voice_speed < TTS_SPEED_MIN or voice_speed > TTS_SPEED_MAX:
|
||||
raise ValueError(f"voice_speed must be between {TTS_SPEED_MIN} and {TTS_SPEED_MAX}")
|
||||
|
||||
payload: dict[str, Any] = {
|
||||
"text": text,
|
||||
"voice_id": voice_id,
|
||||
"voice_language": voice_language,
|
||||
"voice_speed": voice_speed,
|
||||
}
|
||||
self._copy_common_task_fields(inputs, payload)
|
||||
return {
|
||||
"protocol": "classic",
|
||||
"path": "/v1/audio/tts",
|
||||
"payload": payload,
|
||||
"operation": "text_to_speech",
|
||||
"model": "kling-official-tts",
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _create_and_collect_audios(
|
||||
client: KlingClient,
|
||||
request: dict[str, Any],
|
||||
inputs: dict[str, Any],
|
||||
) -> tuple[str, list[dict[str, Any]]]:
|
||||
"""Create a TTS task and return audio outputs.
|
||||
|
||||
Official TTS may return a completed task and task_result.audios[]
|
||||
directly from POST /v1/audio/tts. Older/async behavior still requires
|
||||
polling GET /v1/audio/tts/{task_id}, so support both shapes.
|
||||
"""
|
||||
if hasattr(client, "post"):
|
||||
data = client.post(request["path"], request["payload"])
|
||||
payload = data.get("data") or {}
|
||||
task_id = payload.get("task_id")
|
||||
if not task_id:
|
||||
raise KlingAPIError(f"Kling TTS create response missing data.task_id: {data}")
|
||||
|
||||
task_result = payload.get("task_result") or {}
|
||||
outputs = task_result.get("audios")
|
||||
if outputs is not None:
|
||||
if not isinstance(outputs, list):
|
||||
raise KlingAPIError("Kling TTS result path data.task_result.audios is not a list")
|
||||
return str(task_id), outputs
|
||||
|
||||
status = payload.get("task_status") or payload.get("status")
|
||||
if status == "failed":
|
||||
message = payload.get("task_status_msg") or payload.get("message") or "Kling TTS task failed"
|
||||
raise KlingAPIError(str(message), code=payload.get("task_status"), response=data)
|
||||
|
||||
return str(task_id), client.poll_classic(
|
||||
request["path"],
|
||||
str(task_id),
|
||||
"audios",
|
||||
timeout_seconds=int(inputs.get("timeout_seconds", 300)),
|
||||
poll_interval=float(inputs.get("poll_interval", 3.0)),
|
||||
)
|
||||
|
||||
task_id = client.create_classic_task(request["path"], request["payload"])
|
||||
return task_id, client.poll_classic(
|
||||
request["path"],
|
||||
task_id,
|
||||
"audios",
|
||||
timeout_seconds=int(inputs.get("timeout_seconds", 300)),
|
||||
poll_interval=float(inputs.get("poll_interval", 3.0)),
|
||||
)
|
||||
|
||||
def _download_audios(
|
||||
self,
|
||||
client: KlingClient,
|
||||
outputs: list[dict[str, Any]],
|
||||
inputs: dict[str, Any],
|
||||
) -> list[Path]:
|
||||
if not outputs:
|
||||
raise ValueError("Kling TTS response contained no audios")
|
||||
base_path = Path(inputs.get("output_path", "kling_tts.mp3"))
|
||||
paths: list[Path] = []
|
||||
for index, item in enumerate(outputs):
|
||||
url = self._output_url(item)
|
||||
suffix = extension_from_url(url, ".mp3")
|
||||
output_path = numbered_output_path(output_path_with_suffix(base_path, suffix), index, suffix)
|
||||
client.download(url, output_path)
|
||||
paths.append(output_path)
|
||||
return paths
|
||||
|
||||
@staticmethod
|
||||
def _output_url(item: dict[str, Any]) -> str:
|
||||
url = item.get("url") or item.get("audio_url") or item.get("resource_url")
|
||||
if url:
|
||||
return str(url)
|
||||
resource = item.get("resource") or {}
|
||||
if isinstance(resource, dict) and resource.get("url"):
|
||||
return str(resource["url"])
|
||||
raise ValueError(f"Kling TTS response item contained no downloadable URL: {item}")
|
||||
|
||||
@staticmethod
|
||||
def _copy_common_task_fields(inputs: dict[str, Any], payload: dict[str, Any]) -> None:
|
||||
callback_url = validate_callback_url(inputs.get("callback_url"))
|
||||
if callback_url:
|
||||
payload["callback_url"] = callback_url
|
||||
if inputs.get("external_task_id"):
|
||||
payload["external_task_id"] = inputs["external_task_id"]
|
||||
|
||||
@staticmethod
|
||||
def _callback_result_data(inputs: dict[str, Any], task_id: str) -> dict[str, Any]:
|
||||
callback_url = inputs.get("callback_url")
|
||||
if not callback_url:
|
||||
return {}
|
||||
return {
|
||||
"callback_url": str(callback_url),
|
||||
"callback_requested": True,
|
||||
"polling_used": True,
|
||||
"task_id": task_id,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _account_usage_result(inputs: dict[str, Any], client: KlingClient) -> dict[str, Any]:
|
||||
if not inputs.get("include_account_usage"):
|
||||
return {}
|
||||
try:
|
||||
usage = get_account_costs(client=client)
|
||||
return {
|
||||
"account_usage": usage,
|
||||
"cost_source": "estimate_with_account_usage_context",
|
||||
"reconciled_cost_usd": None,
|
||||
}
|
||||
except Exception as exc:
|
||||
return {
|
||||
"account_usage_error": str(exc),
|
||||
"cost_source": "estimate",
|
||||
"reconciled_cost_usd": None,
|
||||
}
|
||||
|
|
@ -45,6 +45,17 @@ class TTSSelector(BaseTool):
|
|||
"type": "string",
|
||||
"description": "Provider-specific voice ID. Passed through to the selected TTS provider.",
|
||||
},
|
||||
"voice_language": {
|
||||
"type": "string",
|
||||
"enum": ["zh", "en"],
|
||||
"description": "Kling official voice language. Passed through when selected provider supports it.",
|
||||
},
|
||||
"voice_speed": {
|
||||
"type": "number",
|
||||
"minimum": 0.5,
|
||||
"maximum": 2.0,
|
||||
"description": "Kling official voice speed. Use speed for OpenAI/ElevenLabs-style controls.",
|
||||
},
|
||||
"model_id": {
|
||||
"type": "string",
|
||||
"description": "TTS model to use (e.g. eleven_multilingual_v2). Passed through to provider.",
|
||||
|
|
|
|||
|
|
@ -0,0 +1,332 @@
|
|||
"""Kling official API avatar image-to-video provider."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from tools._kling.account import account_usage_hint_for_error, get_account_costs
|
||||
from tools._kling.callbacks import validate_callback_url
|
||||
from tools._kling.client import KlingClient
|
||||
from tools._kling.errors import KlingAPIError
|
||||
from tools._kling.media import (
|
||||
extension_from_url,
|
||||
normalize_image_input,
|
||||
normalize_media_input,
|
||||
numbered_output_path,
|
||||
output_path_with_suffix,
|
||||
)
|
||||
from tools._kling.schemas import AVATAR_MODES
|
||||
from tools.base_tool import (
|
||||
BaseTool,
|
||||
DependencyError,
|
||||
Determinism,
|
||||
ExecutionMode,
|
||||
ResourceProfile,
|
||||
RetryPolicy,
|
||||
ToolResult,
|
||||
ToolRuntime,
|
||||
ToolStability,
|
||||
ToolTier,
|
||||
)
|
||||
from tools.video._shared import probe_output
|
||||
|
||||
|
||||
class KlingAvatar(BaseTool):
|
||||
name = "kling_avatar"
|
||||
version = "0.1.0"
|
||||
tier = ToolTier.GENERATE
|
||||
capability = "avatar"
|
||||
provider = "kling_official"
|
||||
stability = ToolStability.EXPERIMENTAL
|
||||
execution_mode = ExecutionMode.SYNC
|
||||
determinism = Determinism.STOCHASTIC
|
||||
runtime = ToolRuntime.API
|
||||
|
||||
dependencies = ["env:KLING_API_KEY"]
|
||||
install_instructions = (
|
||||
"Set KLING_API_KEY in .env for the official Kling API. "
|
||||
"Provide an avatar image plus either audio_id or sound_file/audio_path."
|
||||
)
|
||||
agent_skills = ["kling-official", "avatar-video"]
|
||||
|
||||
capabilities = ["photo_to_video", "avatar_video", "audio_driven_avatar"]
|
||||
supports = {
|
||||
"photo_to_video": True,
|
||||
"audio_driven_animation": True,
|
||||
"offline": False,
|
||||
"cloud_render": True,
|
||||
}
|
||||
best_for = [
|
||||
"official Kling cloud avatar presenter clips",
|
||||
"high-quality image-to-video avatar generation from a supplied portrait",
|
||||
"projects already using a Kling official account and resource pack",
|
||||
]
|
||||
not_good_for = [
|
||||
"fully offline avatar generation",
|
||||
"free local drafts",
|
||||
"silently replacing the local talking_head provider",
|
||||
]
|
||||
fallback_tools = ["talking_head", "lip_sync"]
|
||||
|
||||
input_schema = {
|
||||
"type": "object",
|
||||
"anyOf": [
|
||||
{"required": ["image_url"]},
|
||||
{"required": ["image_path"]},
|
||||
],
|
||||
"allOf": [
|
||||
{
|
||||
"anyOf": [
|
||||
{"required": ["audio_id"]},
|
||||
{"required": ["sound_file"]},
|
||||
{"required": ["sound_file_url"]},
|
||||
{"required": ["sound_file_path"]},
|
||||
{"required": ["audio_path"]},
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"image_url": {"type": "string"},
|
||||
"image_path": {"type": "string"},
|
||||
"audio_id": {"type": "string"},
|
||||
"sound_file": {
|
||||
"type": "string",
|
||||
"description": "Official Kling sound_file value or raw base64 audio.",
|
||||
},
|
||||
"sound_file_url": {"type": "string"},
|
||||
"sound_file_path": {"type": "string"},
|
||||
"audio_path": {
|
||||
"type": "string",
|
||||
"description": "Alias for sound_file_path for compatibility with local avatar tools.",
|
||||
},
|
||||
"prompt": {"type": "string"},
|
||||
"mode": {"type": "string", "enum": AVATAR_MODES, "default": "std"},
|
||||
"callback_url": {"type": "string"},
|
||||
"external_task_id": {"type": "string"},
|
||||
"include_account_usage": {
|
||||
"type": "boolean",
|
||||
"default": False,
|
||||
"description": "Optional low-frequency account usage diagnostic; not used by default.",
|
||||
},
|
||||
"timeout_seconds": {"type": "integer", "default": 900},
|
||||
"poll_interval": {"type": "number", "default": 5.0},
|
||||
"output_path": {"type": "string"},
|
||||
},
|
||||
}
|
||||
|
||||
resource_profile = ResourceProfile(
|
||||
cpu_cores=1, ram_mb=512, vram_mb=0, disk_mb=500, network_required=True
|
||||
)
|
||||
retry_policy = RetryPolicy(
|
||||
max_retries=2,
|
||||
backoff_seconds=2.0,
|
||||
retryable_errors=["1302", "1303", "5000", "5001", "5002"],
|
||||
)
|
||||
idempotency_key_fields = ["image_url", "image_path", "audio_id", "sound_file", "sound_file_path", "mode"]
|
||||
side_effects = ["paid remote generation via official Kling API", "writes avatar video to output_path"]
|
||||
user_visible_verification = ["Watch generated avatar video for identity preservation and mouth motion"]
|
||||
quality_score = 0.82
|
||||
latency_p50_seconds = 240.0
|
||||
|
||||
def estimate_cost(self, inputs: dict[str, Any]) -> float:
|
||||
base = 0.35
|
||||
if inputs.get("mode") == "pro":
|
||||
base *= 1.7
|
||||
if inputs.get("sound_file_path") or inputs.get("audio_path"):
|
||||
base += 0.04
|
||||
return round(base, 4)
|
||||
|
||||
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
|
||||
return 240.0
|
||||
|
||||
def dry_run(self, inputs: dict[str, Any]) -> dict[str, Any]:
|
||||
result = super().dry_run(inputs)
|
||||
result.update(
|
||||
{
|
||||
"paid_api": True,
|
||||
"cost_estimate_confidence": "low",
|
||||
"cost_estimate_basis": "Conservative OpenMontage avatar estimate pending official account-usage reconciliation.",
|
||||
}
|
||||
)
|
||||
return result
|
||||
|
||||
def execute(self, inputs: dict[str, Any]) -> ToolResult:
|
||||
try:
|
||||
self.check_dependencies()
|
||||
except DependencyError as exc:
|
||||
return ToolResult(success=False, error=str(exc))
|
||||
|
||||
start = time.time()
|
||||
try:
|
||||
request = self._build_request(inputs)
|
||||
client = KlingClient()
|
||||
task_id = client.create_classic_task(request["path"], request["payload"])
|
||||
outputs = client.poll_classic(
|
||||
request["path"],
|
||||
task_id,
|
||||
"videos",
|
||||
timeout_seconds=int(inputs.get("timeout_seconds", 900)),
|
||||
poll_interval=float(inputs.get("poll_interval", 5.0)),
|
||||
)
|
||||
paths = self._download_videos(client, outputs, inputs)
|
||||
probed = probe_output(paths[0])
|
||||
except (KlingAPIError, TimeoutError, ValueError, KeyError, FileNotFoundError) as exc:
|
||||
data: dict[str, Any] = {"provider": self.provider}
|
||||
if isinstance(exc, KlingAPIError):
|
||||
data.update(
|
||||
{
|
||||
"error_code": exc.code,
|
||||
"request_id": exc.request_id,
|
||||
"http_status": exc.http_status,
|
||||
"account_usage_diagnostic": account_usage_hint_for_error(exc),
|
||||
}
|
||||
)
|
||||
return ToolResult(success=False, data=data, error=f"Kling official avatar generation failed: {exc}")
|
||||
except Exception as exc:
|
||||
return ToolResult(success=False, data={"provider": self.provider}, error=f"Kling official avatar generation failed: {exc}")
|
||||
|
||||
return ToolResult(
|
||||
success=True,
|
||||
data={
|
||||
"provider": self.provider,
|
||||
"model": "kling-official-avatar",
|
||||
"task_id": task_id,
|
||||
"operation": "image_to_avatar_video",
|
||||
"mode": request["payload"].get("mode"),
|
||||
"prompt": request["payload"].get("prompt"),
|
||||
"avatar_source": request["avatar_source"],
|
||||
"audio_source": request["audio_source"],
|
||||
"remote_outputs": outputs,
|
||||
"output": str(paths[0]),
|
||||
"output_path": str(paths[0]),
|
||||
"video_paths": [str(path) for path in paths],
|
||||
"format": "mp4",
|
||||
"cost_estimate_confidence": "low",
|
||||
"cost_estimate_basis": "Conservative estimate pending official account-usage reconciliation.",
|
||||
**self._account_usage_result(inputs, client),
|
||||
**self._callback_result_data(inputs, task_id),
|
||||
**probed,
|
||||
},
|
||||
artifacts=[str(path) for path in paths],
|
||||
cost_usd=self.estimate_cost(inputs),
|
||||
duration_seconds=round(time.time() - start, 2),
|
||||
model="kling-official-avatar",
|
||||
)
|
||||
|
||||
def _build_request(self, inputs: dict[str, Any]) -> dict[str, Any]:
|
||||
image = normalize_image_input(inputs.get("image_url"), inputs.get("image_path"))
|
||||
if not image:
|
||||
raise ValueError("Kling avatar requires image_url or image_path")
|
||||
|
||||
mode = str(inputs.get("mode") or "std")
|
||||
if mode not in AVATAR_MODES:
|
||||
raise ValueError(f"mode must be one of: {', '.join(AVATAR_MODES)}")
|
||||
|
||||
payload: dict[str, Any] = {
|
||||
"image": image,
|
||||
"mode": mode,
|
||||
}
|
||||
if inputs.get("prompt"):
|
||||
payload["prompt"] = str(inputs["prompt"])
|
||||
|
||||
audio_source = self._copy_audio_input(inputs, payload)
|
||||
self._copy_common_task_fields(inputs, payload)
|
||||
return {
|
||||
"protocol": "classic",
|
||||
"path": "/v1/videos/avatar/image2video",
|
||||
"payload": payload,
|
||||
"operation": "image_to_avatar_video",
|
||||
"model": "kling-official-avatar",
|
||||
"avatar_source": inputs.get("image_url") or inputs.get("image_path"),
|
||||
"audio_source": audio_source,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _copy_audio_input(inputs: dict[str, Any], payload: dict[str, Any]) -> dict[str, Any]:
|
||||
audio_id = str(inputs.get("audio_id") or "").strip()
|
||||
if audio_id:
|
||||
payload["audio_id"] = audio_id
|
||||
return {"type": "audio_id", "value": audio_id}
|
||||
|
||||
sound_path = inputs.get("sound_file_path") or inputs.get("audio_path")
|
||||
sound_file = normalize_media_input(
|
||||
url=inputs.get("sound_file_url"),
|
||||
path=sound_path,
|
||||
value=inputs.get("sound_file"),
|
||||
label="Avatar audio file",
|
||||
)
|
||||
if not sound_file:
|
||||
raise ValueError("Kling avatar requires audio_id, sound_file, sound_file_url, sound_file_path, or audio_path")
|
||||
payload["sound_file"] = sound_file
|
||||
return {
|
||||
"type": "sound_file",
|
||||
"source": inputs.get("sound_file_url") or sound_path or "inline",
|
||||
}
|
||||
|
||||
def _download_videos(
|
||||
self,
|
||||
client: KlingClient,
|
||||
outputs: list[dict[str, Any]],
|
||||
inputs: dict[str, Any],
|
||||
) -> list[Path]:
|
||||
if not outputs:
|
||||
raise ValueError("Kling avatar response contained no videos")
|
||||
base_path = Path(inputs.get("output_path", "kling_avatar.mp4"))
|
||||
paths: list[Path] = []
|
||||
for index, item in enumerate(outputs):
|
||||
url = self._output_url(item)
|
||||
suffix = extension_from_url(url, ".mp4")
|
||||
output_path = numbered_output_path(output_path_with_suffix(base_path, suffix), index, suffix)
|
||||
client.download(url, output_path)
|
||||
paths.append(output_path)
|
||||
return paths
|
||||
|
||||
@staticmethod
|
||||
def _output_url(item: dict[str, Any]) -> str:
|
||||
url = item.get("url") or item.get("video_url") or item.get("resource_url")
|
||||
if url:
|
||||
return str(url)
|
||||
resource = item.get("resource") or {}
|
||||
if isinstance(resource, dict) and resource.get("url"):
|
||||
return str(resource["url"])
|
||||
raise ValueError(f"Kling avatar response item contained no downloadable URL: {item}")
|
||||
|
||||
@staticmethod
|
||||
def _copy_common_task_fields(inputs: dict[str, Any], payload: dict[str, Any]) -> None:
|
||||
callback_url = validate_callback_url(inputs.get("callback_url"))
|
||||
if callback_url:
|
||||
payload["callback_url"] = callback_url
|
||||
if inputs.get("external_task_id"):
|
||||
payload["external_task_id"] = inputs["external_task_id"]
|
||||
|
||||
@staticmethod
|
||||
def _callback_result_data(inputs: dict[str, Any], task_id: str) -> dict[str, Any]:
|
||||
callback_url = inputs.get("callback_url")
|
||||
if not callback_url:
|
||||
return {}
|
||||
return {
|
||||
"callback_url": str(callback_url),
|
||||
"callback_requested": True,
|
||||
"polling_used": True,
|
||||
"task_id": task_id,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _account_usage_result(inputs: dict[str, Any], client: KlingClient) -> dict[str, Any]:
|
||||
if not inputs.get("include_account_usage"):
|
||||
return {}
|
||||
try:
|
||||
usage = get_account_costs(client=client)
|
||||
return {
|
||||
"account_usage": usage,
|
||||
"cost_source": "estimate_with_account_usage_context",
|
||||
"reconciled_cost_usd": None,
|
||||
}
|
||||
except Exception as exc:
|
||||
return {
|
||||
"account_usage_error": str(exc),
|
||||
"cost_source": "estimate",
|
||||
"reconciled_cost_usd": None,
|
||||
}
|
||||
|
|
@ -0,0 +1,551 @@
|
|||
"""Kling official API lip-sync provider."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from tools._kling.account import account_usage_hint_for_error, get_account_costs
|
||||
from tools._kling.callbacks import validate_callback_url
|
||||
from tools._kling.client import KlingClient
|
||||
from tools._kling.errors import KlingAPIError
|
||||
from tools._kling.media import (
|
||||
extension_from_url,
|
||||
normalize_media_input,
|
||||
numbered_output_path,
|
||||
output_path_with_suffix,
|
||||
)
|
||||
from tools._kling.schemas import LIP_SYNC_OPERATIONS
|
||||
from tools.base_tool import (
|
||||
BaseTool,
|
||||
DependencyError,
|
||||
Determinism,
|
||||
ExecutionMode,
|
||||
ResourceProfile,
|
||||
RetryPolicy,
|
||||
ToolResult,
|
||||
ToolRuntime,
|
||||
ToolStability,
|
||||
ToolTier,
|
||||
)
|
||||
from tools.video._shared import probe_output
|
||||
|
||||
|
||||
class KlingLipSync(BaseTool):
|
||||
name = "kling_lip_sync"
|
||||
version = "0.1.0"
|
||||
tier = ToolTier.GENERATE
|
||||
capability = "avatar"
|
||||
provider = "kling_official"
|
||||
stability = ToolStability.EXPERIMENTAL
|
||||
execution_mode = ExecutionMode.SYNC
|
||||
determinism = Determinism.STOCHASTIC
|
||||
runtime = ToolRuntime.API
|
||||
|
||||
dependencies = ["env:KLING_API_KEY"]
|
||||
install_instructions = (
|
||||
"Set KLING_API_KEY in .env for the official Kling API. "
|
||||
"Use identify_face first for multi-person clips, then pass face_choose or face_id."
|
||||
)
|
||||
agent_skills = ["kling-official", "avatar-video"]
|
||||
|
||||
capabilities = ["lip_sync", "identify_face", "audio_video_alignment"]
|
||||
supports = {
|
||||
"lip_sync": True,
|
||||
"face_selection": True,
|
||||
"offline": False,
|
||||
"cloud_render": True,
|
||||
}
|
||||
best_for = [
|
||||
"official Kling cloud lip-sync for existing presenter video",
|
||||
"dubbing workflows that can use Kling face identification",
|
||||
"manual or explicit automatic face selection before paid lip-sync generation",
|
||||
]
|
||||
not_good_for = [
|
||||
"fully offline lip-sync",
|
||||
"silent first-face selection in multi-person footage",
|
||||
"replacing local lip_sync behavior implicitly",
|
||||
]
|
||||
fallback_tools = ["lip_sync"]
|
||||
|
||||
input_schema = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"operation": {"type": "string", "enum": LIP_SYNC_OPERATIONS, "default": "advanced_lip_sync"},
|
||||
"video_id": {"type": "string"},
|
||||
"video_url": {"type": "string"},
|
||||
"video_path": {
|
||||
"type": "string",
|
||||
"description": "Not silently uploaded. Provide video_url unless an official upload path is added.",
|
||||
},
|
||||
"session_id": {"type": "string"},
|
||||
"face_id": {"type": "string"},
|
||||
"face_choose": {"type": "array"},
|
||||
"auto_select_face": {
|
||||
"type": "boolean",
|
||||
"default": False,
|
||||
"description": "Explicitly allow largest-face automatic selection after identify_face.",
|
||||
},
|
||||
"audio_id": {"type": "string"},
|
||||
"sound_file": {
|
||||
"type": "string",
|
||||
"description": "Official Kling sound_file value or raw base64 audio.",
|
||||
},
|
||||
"sound_file_url": {"type": "string"},
|
||||
"sound_file_path": {"type": "string"},
|
||||
"audio_path": {
|
||||
"type": "string",
|
||||
"description": "Alias for sound_file_path for compatibility with local lip_sync.",
|
||||
},
|
||||
"faces_artifact_path": {"type": "string"},
|
||||
"callback_url": {"type": "string"},
|
||||
"external_task_id": {"type": "string"},
|
||||
"include_account_usage": {
|
||||
"type": "boolean",
|
||||
"default": False,
|
||||
"description": "Optional low-frequency account usage diagnostic; not used by default.",
|
||||
},
|
||||
"timeout_seconds": {"type": "integer", "default": 900},
|
||||
"poll_interval": {"type": "number", "default": 5.0},
|
||||
"output_path": {"type": "string"},
|
||||
},
|
||||
}
|
||||
|
||||
resource_profile = ResourceProfile(
|
||||
cpu_cores=1, ram_mb=512, vram_mb=0, disk_mb=500, network_required=True
|
||||
)
|
||||
retry_policy = RetryPolicy(
|
||||
max_retries=2,
|
||||
backoff_seconds=2.0,
|
||||
retryable_errors=["1302", "1303", "5000", "5001", "5002"],
|
||||
)
|
||||
idempotency_key_fields = ["video_id", "video_url", "session_id", "face_id", "audio_id", "sound_file_path"]
|
||||
side_effects = [
|
||||
"paid remote generation via official Kling API",
|
||||
"writes face selection artifact",
|
||||
"writes lip-synced video to output_path",
|
||||
]
|
||||
user_visible_verification = ["Watch output video to verify the selected face matches the new audio"]
|
||||
quality_score = 0.80
|
||||
latency_p50_seconds = 240.0
|
||||
|
||||
def estimate_cost(self, inputs: dict[str, Any]) -> float:
|
||||
operation = str(inputs.get("operation", "advanced_lip_sync"))
|
||||
if operation == "identify_face":
|
||||
return 0.02
|
||||
if operation == "full_lip_sync":
|
||||
return 0.34
|
||||
return 0.32
|
||||
|
||||
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
|
||||
if inputs.get("operation") == "identify_face":
|
||||
return 15.0
|
||||
return 240.0
|
||||
|
||||
def dry_run(self, inputs: dict[str, Any]) -> dict[str, Any]:
|
||||
result = super().dry_run(inputs)
|
||||
result.update(
|
||||
{
|
||||
"paid_api": True,
|
||||
"cost_estimate_confidence": "low",
|
||||
"cost_estimate_basis": "Conservative OpenMontage estimate for identify-face plus advanced lip-sync.",
|
||||
}
|
||||
)
|
||||
return result
|
||||
|
||||
def execute(self, inputs: dict[str, Any]) -> ToolResult:
|
||||
try:
|
||||
self.check_dependencies()
|
||||
except DependencyError as exc:
|
||||
return ToolResult(success=False, error=str(exc))
|
||||
|
||||
operation = str(inputs.get("operation") or "advanced_lip_sync")
|
||||
start = time.time()
|
||||
client = KlingClient()
|
||||
try:
|
||||
if operation == "identify_face":
|
||||
identify = self._identify_faces(client, inputs)
|
||||
return self._identify_result(inputs, identify, start)
|
||||
if operation == "full_lip_sync":
|
||||
identify = self._identify_faces(client, inputs)
|
||||
artifact_path = self._write_faces_artifact(inputs, identify)
|
||||
face_choose, selection = self._face_selection(identify["faces"], inputs)
|
||||
artifact_path = self._write_faces_artifact(inputs, identify, selection=selection)
|
||||
if selection["selection_method"] == "requires_user_selection":
|
||||
return ToolResult(
|
||||
success=False,
|
||||
data={
|
||||
"provider": self.provider,
|
||||
"operation": operation,
|
||||
"session_id": identify["session_id"],
|
||||
"faces": identify["faces"],
|
||||
"requires_face_selection": True,
|
||||
"selection_reason": selection["selection_reason"],
|
||||
"faces_artifact_path": str(artifact_path),
|
||||
},
|
||||
artifacts=[str(artifact_path)],
|
||||
error="Multiple faces detected. Pass face_id/face_choose or set auto_select_face=True.",
|
||||
cost_usd=self.estimate_cost({"operation": "identify_face"}),
|
||||
duration_seconds=round(time.time() - start, 2),
|
||||
model="kling-official-lip-sync",
|
||||
)
|
||||
merged = {**inputs, "session_id": identify["session_id"], "face_choose": face_choose}
|
||||
request = self._build_advanced_request(merged)
|
||||
result = self._run_advanced_lip_sync(client, merged, request, start)
|
||||
result.data["faces_artifact_path"] = str(artifact_path)
|
||||
result.data["face_selection"] = selection
|
||||
result.artifacts.append(str(artifact_path))
|
||||
return result
|
||||
if operation == "advanced_lip_sync":
|
||||
request = self._build_advanced_request(inputs)
|
||||
return self._run_advanced_lip_sync(client, inputs, request, start)
|
||||
raise ValueError(f"Unsupported Kling lip-sync operation: {operation}")
|
||||
except (KlingAPIError, TimeoutError, ValueError, KeyError, FileNotFoundError) as exc:
|
||||
data: dict[str, Any] = {"provider": self.provider}
|
||||
if isinstance(exc, KlingAPIError):
|
||||
data.update(
|
||||
{
|
||||
"error_code": exc.code,
|
||||
"request_id": exc.request_id,
|
||||
"http_status": exc.http_status,
|
||||
"account_usage_diagnostic": account_usage_hint_for_error(exc),
|
||||
}
|
||||
)
|
||||
return ToolResult(success=False, data=data, error=f"Kling official lip-sync failed: {exc}")
|
||||
except Exception as exc:
|
||||
return ToolResult(success=False, data={"provider": self.provider}, error=f"Kling official lip-sync failed: {exc}")
|
||||
|
||||
def _identify_faces(self, client: KlingClient, inputs: dict[str, Any]) -> dict[str, Any]:
|
||||
request = self._build_identify_request(inputs)
|
||||
data = client.post(request["path"], request["payload"])
|
||||
payload = data.get("data") or {}
|
||||
session_id = payload.get("session_id")
|
||||
if not session_id:
|
||||
raise ValueError(f"Kling identify-face response missing data.session_id: {data}")
|
||||
faces = (
|
||||
payload.get("faces")
|
||||
or payload.get("face_list")
|
||||
or payload.get("face_infos")
|
||||
or payload.get("faces_info")
|
||||
or []
|
||||
)
|
||||
if not isinstance(faces, list):
|
||||
raise ValueError("Kling identify-face response face list is not a list")
|
||||
if not faces:
|
||||
raise ValueError("Kling identify-face response contained no faces")
|
||||
return {
|
||||
"session_id": str(session_id),
|
||||
"faces": faces,
|
||||
"raw_response": data,
|
||||
"request": request,
|
||||
}
|
||||
|
||||
def _identify_result(self, inputs: dict[str, Any], identify: dict[str, Any], start: float) -> ToolResult:
|
||||
artifact_path = self._write_faces_artifact(inputs, identify)
|
||||
return ToolResult(
|
||||
success=True,
|
||||
data={
|
||||
"provider": self.provider,
|
||||
"model": "kling-official-lip-sync",
|
||||
"operation": "identify_face",
|
||||
"session_id": identify["session_id"],
|
||||
"faces": identify["faces"],
|
||||
"face_count": len(identify["faces"]),
|
||||
"faces_artifact_path": str(artifact_path),
|
||||
},
|
||||
artifacts=[str(artifact_path)],
|
||||
cost_usd=self.estimate_cost({"operation": "identify_face"}),
|
||||
duration_seconds=round(time.time() - start, 2),
|
||||
model="kling-official-lip-sync",
|
||||
)
|
||||
|
||||
def _run_advanced_lip_sync(
|
||||
self,
|
||||
client: KlingClient,
|
||||
inputs: dict[str, Any],
|
||||
request: dict[str, Any],
|
||||
start: float,
|
||||
) -> ToolResult:
|
||||
task_id = client.create_classic_task(request["path"], request["payload"])
|
||||
outputs = client.poll_classic(
|
||||
request["path"],
|
||||
task_id,
|
||||
"videos",
|
||||
timeout_seconds=int(inputs.get("timeout_seconds", 900)),
|
||||
poll_interval=float(inputs.get("poll_interval", 5.0)),
|
||||
)
|
||||
paths = self._download_videos(client, outputs, inputs)
|
||||
probed = probe_output(paths[0])
|
||||
return ToolResult(
|
||||
success=True,
|
||||
data={
|
||||
"provider": self.provider,
|
||||
"model": "kling-official-lip-sync",
|
||||
"task_id": task_id,
|
||||
"operation": request["operation"],
|
||||
"session_id": request["payload"]["session_id"],
|
||||
"face_choose": request["payload"]["face_choose"],
|
||||
"audio_source": request["audio_source"],
|
||||
"remote_outputs": outputs,
|
||||
"output": str(paths[0]),
|
||||
"output_path": str(paths[0]),
|
||||
"video_paths": [str(path) for path in paths],
|
||||
"format": "mp4",
|
||||
"cost_estimate_confidence": "low",
|
||||
"cost_estimate_basis": "Conservative estimate pending official account-usage reconciliation.",
|
||||
**self._account_usage_result(inputs, client),
|
||||
**self._callback_result_data(inputs, task_id),
|
||||
**probed,
|
||||
},
|
||||
artifacts=[str(path) for path in paths],
|
||||
cost_usd=self.estimate_cost(inputs),
|
||||
duration_seconds=round(time.time() - start, 2),
|
||||
model="kling-official-lip-sync",
|
||||
)
|
||||
|
||||
def _build_identify_request(self, inputs: dict[str, Any]) -> dict[str, Any]:
|
||||
if inputs.get("video_path") and not (inputs.get("video_url") or inputs.get("video_id")):
|
||||
raise ValueError("Kling identify_face requires video_url or video_id; local video paths cannot be silently uploaded.")
|
||||
payload: dict[str, Any] = {}
|
||||
if inputs.get("video_id"):
|
||||
payload["video_id"] = str(inputs["video_id"])
|
||||
if inputs.get("video_url"):
|
||||
payload["video_url"] = str(inputs["video_url"])
|
||||
if not payload:
|
||||
raise ValueError("Kling identify_face requires video_id or video_url")
|
||||
return {
|
||||
"path": "/v1/videos/identify-face",
|
||||
"payload": payload,
|
||||
"operation": "identify_face",
|
||||
}
|
||||
|
||||
def _build_advanced_request(self, inputs: dict[str, Any]) -> dict[str, Any]:
|
||||
session_id = str(inputs.get("session_id") or "").strip()
|
||||
if not session_id:
|
||||
raise ValueError("advanced_lip_sync requires session_id")
|
||||
face_choose = self._normalize_face_choose(inputs)
|
||||
if not face_choose:
|
||||
raise ValueError("advanced_lip_sync requires face_choose or face_id")
|
||||
payload: dict[str, Any] = {
|
||||
"session_id": session_id,
|
||||
"face_choose": face_choose,
|
||||
}
|
||||
audio_source = self._copy_audio_input(inputs, payload)
|
||||
self._copy_common_task_fields(inputs, payload)
|
||||
return {
|
||||
"protocol": "classic",
|
||||
"path": "/v1/videos/advanced-lip-sync",
|
||||
"payload": payload,
|
||||
"operation": "advanced_lip_sync",
|
||||
"model": "kling-official-lip-sync",
|
||||
"audio_source": audio_source,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _normalize_face_choose(inputs: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
if inputs.get("face_choose"):
|
||||
raw = inputs["face_choose"]
|
||||
if isinstance(raw, dict):
|
||||
raw = [raw]
|
||||
if not isinstance(raw, list):
|
||||
raise ValueError("face_choose must be a list of face choice objects")
|
||||
normalized: list[dict[str, Any]] = []
|
||||
for item in raw:
|
||||
if isinstance(item, str):
|
||||
normalized.append({"face_id": item})
|
||||
elif isinstance(item, dict):
|
||||
if not (item.get("face_id") or item.get("id")):
|
||||
raise ValueError("face_choose items must include face_id")
|
||||
record = dict(item)
|
||||
if "face_id" not in record and record.get("id"):
|
||||
record["face_id"] = record.pop("id")
|
||||
normalized.append(record)
|
||||
else:
|
||||
raise ValueError("face_choose items must be strings or objects")
|
||||
return normalized
|
||||
if inputs.get("face_id"):
|
||||
return [{"face_id": str(inputs["face_id"])}]
|
||||
return []
|
||||
|
||||
def _face_selection(
|
||||
self,
|
||||
faces: list[dict[str, Any]],
|
||||
inputs: dict[str, Any],
|
||||
) -> tuple[list[dict[str, Any]], dict[str, Any]]:
|
||||
explicit = self._normalize_face_choose(inputs)
|
||||
if explicit:
|
||||
return explicit, {
|
||||
"selection_method": "user_selected",
|
||||
"selection_reason": "face_choose or face_id was provided",
|
||||
"selected_face": explicit,
|
||||
}
|
||||
if len(faces) == 1:
|
||||
choice = [self._face_to_choice(faces[0])]
|
||||
return choice, {
|
||||
"selection_method": "single_face",
|
||||
"selection_reason": "Only one face was returned by identify_face",
|
||||
"selected_face": choice,
|
||||
}
|
||||
if not inputs.get("auto_select_face"):
|
||||
return [], {
|
||||
"selection_method": "requires_user_selection",
|
||||
"selection_reason": "Multiple faces detected and auto_select_face was not enabled",
|
||||
"face_count": len(faces),
|
||||
}
|
||||
selected = max(faces, key=self._face_area)
|
||||
choice = [self._face_to_choice(selected)]
|
||||
return choice, {
|
||||
"selection_method": "auto_selected",
|
||||
"selection_reason": "auto_select_face=True selected the largest detected face area",
|
||||
"selected_face": choice,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _face_to_choice(face: dict[str, Any]) -> dict[str, Any]:
|
||||
face_id = face.get("face_id") or face.get("id")
|
||||
if not face_id:
|
||||
raise ValueError(f"Cannot select face without face_id/id: {face}")
|
||||
return {"face_id": str(face_id)}
|
||||
|
||||
@staticmethod
|
||||
def _face_area(face: dict[str, Any]) -> float:
|
||||
for key in ("bbox", "box"):
|
||||
value = face.get(key)
|
||||
if isinstance(value, list) and len(value) >= 4:
|
||||
third = float(value[2])
|
||||
fourth = float(value[3])
|
||||
width_height_area = max(third, 0.0) * max(fourth, 0.0)
|
||||
corner_width = third - float(value[0])
|
||||
corner_height = fourth - float(value[1])
|
||||
corner_area = (
|
||||
corner_width * corner_height
|
||||
if corner_width > 0 and corner_height > 0
|
||||
else 0.0
|
||||
)
|
||||
if corner_area and width_height_area:
|
||||
return min(corner_area, width_height_area)
|
||||
return corner_area or width_height_area
|
||||
if isinstance(value, dict):
|
||||
width = value.get("width") or value.get("w")
|
||||
height = value.get("height") or value.get("h")
|
||||
if width is not None and height is not None:
|
||||
return max(float(width), 0.0) * max(float(height), 0.0)
|
||||
width = face.get("width") or face.get("w")
|
||||
height = face.get("height") or face.get("h")
|
||||
if width is not None and height is not None:
|
||||
return max(float(width), 0.0) * max(float(height), 0.0)
|
||||
return 0.0
|
||||
|
||||
@staticmethod
|
||||
def _copy_audio_input(inputs: dict[str, Any], payload: dict[str, Any]) -> dict[str, Any]:
|
||||
audio_id = str(inputs.get("audio_id") or "").strip()
|
||||
if audio_id:
|
||||
payload["audio_id"] = audio_id
|
||||
return {"type": "audio_id", "value": audio_id}
|
||||
|
||||
sound_path = inputs.get("sound_file_path") or inputs.get("audio_path")
|
||||
sound_file = normalize_media_input(
|
||||
url=inputs.get("sound_file_url"),
|
||||
path=sound_path,
|
||||
value=inputs.get("sound_file"),
|
||||
label="Lip-sync audio file",
|
||||
)
|
||||
if not sound_file:
|
||||
raise ValueError("advanced_lip_sync requires audio_id, sound_file, sound_file_url, sound_file_path, or audio_path")
|
||||
payload["sound_file"] = sound_file
|
||||
return {
|
||||
"type": "sound_file",
|
||||
"source": inputs.get("sound_file_url") or sound_path or "inline",
|
||||
}
|
||||
|
||||
def _download_videos(
|
||||
self,
|
||||
client: KlingClient,
|
||||
outputs: list[dict[str, Any]],
|
||||
inputs: dict[str, Any],
|
||||
) -> list[Path]:
|
||||
if not outputs:
|
||||
raise ValueError("Kling lip-sync response contained no videos")
|
||||
base_path = Path(inputs.get("output_path", "kling_lip_sync.mp4"))
|
||||
paths: list[Path] = []
|
||||
for index, item in enumerate(outputs):
|
||||
url = self._output_url(item)
|
||||
suffix = extension_from_url(url, ".mp4")
|
||||
output_path = numbered_output_path(output_path_with_suffix(base_path, suffix), index, suffix)
|
||||
client.download(url, output_path)
|
||||
paths.append(output_path)
|
||||
return paths
|
||||
|
||||
@staticmethod
|
||||
def _output_url(item: dict[str, Any]) -> str:
|
||||
url = item.get("url") or item.get("video_url") or item.get("resource_url")
|
||||
if url:
|
||||
return str(url)
|
||||
resource = item.get("resource") or {}
|
||||
if isinstance(resource, dict) and resource.get("url"):
|
||||
return str(resource["url"])
|
||||
raise ValueError(f"Kling lip-sync response item contained no downloadable URL: {item}")
|
||||
|
||||
@staticmethod
|
||||
def _copy_common_task_fields(inputs: dict[str, Any], payload: dict[str, Any]) -> None:
|
||||
callback_url = validate_callback_url(inputs.get("callback_url"))
|
||||
if callback_url:
|
||||
payload["callback_url"] = callback_url
|
||||
if inputs.get("external_task_id"):
|
||||
payload["external_task_id"] = inputs["external_task_id"]
|
||||
|
||||
@staticmethod
|
||||
def _callback_result_data(inputs: dict[str, Any], task_id: str) -> dict[str, Any]:
|
||||
callback_url = inputs.get("callback_url")
|
||||
if not callback_url:
|
||||
return {}
|
||||
return {
|
||||
"callback_url": str(callback_url),
|
||||
"callback_requested": True,
|
||||
"polling_used": True,
|
||||
"task_id": task_id,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _account_usage_result(inputs: dict[str, Any], client: KlingClient) -> dict[str, Any]:
|
||||
if not inputs.get("include_account_usage"):
|
||||
return {}
|
||||
try:
|
||||
usage = get_account_costs(client=client)
|
||||
return {
|
||||
"account_usage": usage,
|
||||
"cost_source": "estimate_with_account_usage_context",
|
||||
"reconciled_cost_usd": None,
|
||||
}
|
||||
except Exception as exc:
|
||||
return {
|
||||
"account_usage_error": str(exc),
|
||||
"cost_source": "estimate",
|
||||
"reconciled_cost_usd": None,
|
||||
}
|
||||
|
||||
def _write_faces_artifact(
|
||||
self,
|
||||
inputs: dict[str, Any],
|
||||
identify: dict[str, Any],
|
||||
selection: dict[str, Any] | None = None,
|
||||
) -> Path:
|
||||
if inputs.get("faces_artifact_path"):
|
||||
artifact_path = Path(inputs["faces_artifact_path"])
|
||||
elif inputs.get("output_path"):
|
||||
artifact_path = Path(inputs["output_path"]).with_name("kling_lip_sync_faces.json")
|
||||
else:
|
||||
artifact_path = Path("kling_lip_sync_faces.json")
|
||||
artifact_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
artifact = {
|
||||
"provider": self.provider,
|
||||
"operation": "identify_face",
|
||||
"session_id": identify["session_id"],
|
||||
"faces": identify["faces"],
|
||||
"face_count": len(identify["faces"]),
|
||||
"selection": selection,
|
||||
}
|
||||
artifact_path.write_text(json.dumps(artifact, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
return artifact_path
|
||||
|
|
@ -61,6 +61,14 @@ class ImageSelector(BaseTool):
|
|||
"type": "string",
|
||||
"description": "Resolution tier for providers that support named resolutions.",
|
||||
},
|
||||
"api_family": {
|
||||
"type": "string",
|
||||
"description": "Provider-specific API family hint passed through when supported.",
|
||||
},
|
||||
"model_name": {
|
||||
"type": "string",
|
||||
"description": "Provider-specific model name passed through when supported.",
|
||||
},
|
||||
"generation_mode": {
|
||||
"type": "string",
|
||||
"enum": ["generate", "edit"],
|
||||
|
|
@ -79,6 +87,46 @@ class ImageSelector(BaseTool):
|
|||
"items": {"type": "string"},
|
||||
"description": "Multiple local source image paths for compositing edits.",
|
||||
},
|
||||
"image_list": {
|
||||
"type": "array",
|
||||
"description": "Provider-specific image reference list, e.g. Kling Official Image Omni.",
|
||||
},
|
||||
"element_list": {
|
||||
"type": "array",
|
||||
"description": "Provider-specific element references, e.g. Kling Official element_id objects.",
|
||||
},
|
||||
"image_reference": {
|
||||
"type": "string",
|
||||
"description": "Provider-specific reference type, e.g. subject or face.",
|
||||
},
|
||||
"image_fidelity": {
|
||||
"type": "number",
|
||||
"description": "Provider-specific reference image fidelity hint.",
|
||||
},
|
||||
"human_fidelity": {
|
||||
"type": "number",
|
||||
"description": "Provider-specific human or face fidelity hint.",
|
||||
},
|
||||
"result_type": {
|
||||
"type": "string",
|
||||
"description": "Provider-specific result type, e.g. single or series.",
|
||||
},
|
||||
"series_amount": {
|
||||
"type": "string",
|
||||
"description": "Provider-specific series amount for image series generation.",
|
||||
},
|
||||
"watermark": {
|
||||
"type": "boolean",
|
||||
"description": "Provider-specific watermark toggle passed through when supported.",
|
||||
},
|
||||
"callback_url": {
|
||||
"type": "string",
|
||||
"description": "Provider-specific callback URL. Current OpenMontage providers still poll by default.",
|
||||
},
|
||||
"external_task_id": {
|
||||
"type": "string",
|
||||
"description": "Provider-specific idempotency/provenance task id.",
|
||||
},
|
||||
"preferred_provider": {
|
||||
"type": "string",
|
||||
"description": "Provider name or 'auto'. Valid values are discovered at runtime from the registry.",
|
||||
|
|
@ -216,6 +264,18 @@ class ImageSelector(BaseTool):
|
|||
"image_path",
|
||||
"image_urls",
|
||||
"image_paths",
|
||||
"image_list",
|
||||
"element_list",
|
||||
"api_family",
|
||||
"model_name",
|
||||
"image_reference",
|
||||
"image_fidelity",
|
||||
"human_fidelity",
|
||||
"result_type",
|
||||
"series_amount",
|
||||
"watermark",
|
||||
"callback_url",
|
||||
"external_task_id",
|
||||
"workflow_json",
|
||||
"workflow_path",
|
||||
"output_node",
|
||||
|
|
|
|||
|
|
@ -0,0 +1,437 @@
|
|||
"""Kling official API image generation provider."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from tools._kling.account import account_usage_hint_for_error, get_account_costs
|
||||
from tools._kling.callbacks import validate_callback_url
|
||||
from tools._kling.client import KlingClient
|
||||
from tools._kling.elements import element_ids, normalize_element_list
|
||||
from tools._kling.errors import KlingAPIError
|
||||
from tools._kling.media import (
|
||||
extension_from_url,
|
||||
normalize_image_input,
|
||||
numbered_output_path,
|
||||
output_path_with_suffix,
|
||||
)
|
||||
from tools._kling.omni import build_image_prompt_references
|
||||
from tools._kling.schemas import (
|
||||
IMAGE_ASPECT_RATIOS,
|
||||
IMAGE_GENERATION_MODELS,
|
||||
IMAGE_MODELS,
|
||||
IMAGE_REFERENCE_TYPES,
|
||||
IMAGE_RESOLUTIONS,
|
||||
IMAGE_RESULT_TYPES,
|
||||
OMNI_IMAGE_MODELS,
|
||||
)
|
||||
from tools.base_tool import (
|
||||
BaseTool,
|
||||
DependencyError,
|
||||
Determinism,
|
||||
ExecutionMode,
|
||||
ResourceProfile,
|
||||
RetryPolicy,
|
||||
ToolResult,
|
||||
ToolRuntime,
|
||||
ToolStability,
|
||||
ToolTier,
|
||||
)
|
||||
|
||||
|
||||
class KlingOfficialImage(BaseTool):
|
||||
name = "kling_official_image"
|
||||
version = "0.1.0"
|
||||
tier = ToolTier.GENERATE
|
||||
capability = "image_generation"
|
||||
provider = "kling_official"
|
||||
stability = ToolStability.EXPERIMENTAL
|
||||
execution_mode = ExecutionMode.SYNC
|
||||
determinism = Determinism.STOCHASTIC
|
||||
runtime = ToolRuntime.API
|
||||
|
||||
dependencies = ["env:KLING_API_KEY"]
|
||||
install_instructions = (
|
||||
"Set KLING_API_KEY in .env for the official Kling API. "
|
||||
"Optionally set KLING_API_BASE_URL to override the default Singapore endpoint."
|
||||
)
|
||||
agent_skills = ["kling-official"]
|
||||
|
||||
capabilities = ["generate_image", "text_to_image", "image_edit"]
|
||||
supports = {
|
||||
"text_to_image": True,
|
||||
"image_edit": True,
|
||||
"negative_prompt": True,
|
||||
"aspect_ratio": True,
|
||||
}
|
||||
best_for = [
|
||||
"official Kling image generation",
|
||||
"subject or face reference generation",
|
||||
"Omni multi-reference image workflows",
|
||||
]
|
||||
not_good_for = ["offline generation", "free generation", "non-Kling model families"]
|
||||
fallback_tools = ["flux_image", "google_imagen", "openai_image", "recraft_image"]
|
||||
|
||||
input_schema = {
|
||||
"type": "object",
|
||||
"required": ["prompt"],
|
||||
"properties": {
|
||||
"prompt": {"type": "string"},
|
||||
"negative_prompt": {"type": "string"},
|
||||
"operation": {"type": "string", "enum": ["generate", "omni"], "default": "generate"},
|
||||
"generation_mode": {"type": "string", "enum": ["generate", "edit"], "default": "generate"},
|
||||
"api_family": {"type": "string", "enum": ["generation", "omni"], "default": "generation"},
|
||||
"model_name": {"type": "string", "enum": IMAGE_MODELS, "default": "kling-v3"},
|
||||
"image_url": {"type": "string"},
|
||||
"image_path": {"type": "string"},
|
||||
"image_urls": {"type": "array", "items": {"type": "string"}},
|
||||
"image_paths": {"type": "array", "items": {"type": "string"}},
|
||||
"image_list": {"type": "array"},
|
||||
"image_reference": {"type": "string", "enum": IMAGE_REFERENCE_TYPES},
|
||||
"image_fidelity": {"type": "number", "default": 0.5},
|
||||
"human_fidelity": {"type": "number", "default": 0.45},
|
||||
"resolution": {"type": "string", "enum": IMAGE_RESOLUTIONS, "default": "1k"},
|
||||
"aspect_ratio": {"type": "string", "enum": IMAGE_ASPECT_RATIOS, "default": "16:9"},
|
||||
"n": {"type": "integer", "default": 1},
|
||||
"result_type": {"type": "string", "enum": IMAGE_RESULT_TYPES, "default": "single"},
|
||||
"series_amount": {"type": "string"},
|
||||
"element_list": {"type": "array"},
|
||||
"watermark": {"type": "boolean", "default": False},
|
||||
"callback_url": {"type": "string"},
|
||||
"external_task_id": {"type": "string"},
|
||||
"include_account_usage": {
|
||||
"type": "boolean",
|
||||
"default": False,
|
||||
"description": "Optional low-frequency account usage diagnostic; not used by default.",
|
||||
},
|
||||
"timeout_seconds": {"type": "integer", "default": 600},
|
||||
"poll_interval": {"type": "number", "default": 3.0},
|
||||
"output_path": {"type": "string"},
|
||||
},
|
||||
}
|
||||
|
||||
resource_profile = ResourceProfile(
|
||||
cpu_cores=1, ram_mb=512, vram_mb=0, disk_mb=200, network_required=True
|
||||
)
|
||||
retry_policy = RetryPolicy(
|
||||
max_retries=2,
|
||||
backoff_seconds=2.0,
|
||||
retryable_errors=["1302", "1303", "5000", "5001", "5002"],
|
||||
)
|
||||
idempotency_key_fields = [
|
||||
"prompt",
|
||||
"api_family",
|
||||
"model_name",
|
||||
"image_url",
|
||||
"image_path",
|
||||
"aspect_ratio",
|
||||
"resolution",
|
||||
"n",
|
||||
]
|
||||
side_effects = [
|
||||
"paid remote generation via official Kling API",
|
||||
"writes image file(s) to output_path",
|
||||
]
|
||||
user_visible_verification = ["Inspect generated image for quality, prompt adherence, and reference fidelity"]
|
||||
|
||||
def estimate_cost(self, inputs: dict[str, Any]) -> float:
|
||||
n = int(inputs.get("n", 1) or 1)
|
||||
resolution = str(inputs.get("resolution", "1k"))
|
||||
api_family = str(inputs.get("api_family", "generation"))
|
||||
base = 0.04 if api_family == "generation" else 0.08
|
||||
if resolution == "2k":
|
||||
base *= 1.8
|
||||
if resolution == "4k":
|
||||
base *= 3.5
|
||||
if inputs.get("result_type") == "series":
|
||||
base *= 1.5
|
||||
amount = inputs.get("series_amount")
|
||||
if amount and str(amount).isdigit():
|
||||
base *= max(int(str(amount)), 1)
|
||||
if api_family == "omni":
|
||||
reference_count = sum(len(inputs.get(key) or []) for key in ("image_list", "image_urls", "image_paths", "element_list"))
|
||||
if inputs.get("image_url") or inputs.get("image_path"):
|
||||
reference_count += 1
|
||||
base *= 1 + (0.08 * reference_count)
|
||||
return round(base * max(n, 1), 4)
|
||||
|
||||
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
|
||||
return 90.0
|
||||
|
||||
def dry_run(self, inputs: dict[str, Any]) -> dict[str, Any]:
|
||||
result = super().dry_run(inputs)
|
||||
result.update(
|
||||
{
|
||||
"paid_api": True,
|
||||
"cost_estimate_confidence": "low",
|
||||
"cost_estimate_basis": "Conservative OpenMontage estimate; official account usage reconciliation is planned for Phase 2.",
|
||||
}
|
||||
)
|
||||
return result
|
||||
|
||||
def execute(self, inputs: dict[str, Any]) -> ToolResult:
|
||||
try:
|
||||
self.check_dependencies()
|
||||
except DependencyError as exc:
|
||||
return ToolResult(success=False, error=str(exc))
|
||||
|
||||
start = time.time()
|
||||
try:
|
||||
request = self._build_request(inputs)
|
||||
client = KlingClient()
|
||||
task_id = client.create_classic_task(request["path"], request["payload"])
|
||||
outputs = client.poll_classic(
|
||||
request["path"],
|
||||
task_id,
|
||||
"images",
|
||||
timeout_seconds=int(inputs.get("timeout_seconds", 600)),
|
||||
poll_interval=float(inputs.get("poll_interval", 3.0)),
|
||||
)
|
||||
paths = self._download_images(client, outputs, inputs)
|
||||
except (KlingAPIError, TimeoutError, ValueError, KeyError, FileNotFoundError) as exc:
|
||||
data: dict[str, Any] = {"provider": self.provider}
|
||||
if isinstance(exc, KlingAPIError):
|
||||
data.update(
|
||||
{
|
||||
"error_code": exc.code,
|
||||
"request_id": exc.request_id,
|
||||
"http_status": exc.http_status,
|
||||
}
|
||||
)
|
||||
data["account_usage_diagnostic"] = account_usage_hint_for_error(exc)
|
||||
return ToolResult(success=False, data=data, error=f"Kling official image generation failed: {exc}")
|
||||
except Exception as exc:
|
||||
return ToolResult(success=False, data={"provider": self.provider}, error=f"Kling official image generation failed: {exc}")
|
||||
|
||||
return ToolResult(
|
||||
success=True,
|
||||
data={
|
||||
"provider": self.provider,
|
||||
"model": request["model"],
|
||||
"task_id": task_id,
|
||||
"api_family": request["api_family"],
|
||||
"operation": request["operation"],
|
||||
"prompt": request["payload"]["prompt"],
|
||||
"remote_outputs": outputs,
|
||||
"output": str(paths[0]),
|
||||
"output_path": str(paths[0]),
|
||||
"image_paths": [str(path) for path in paths],
|
||||
"format": paths[0].suffix.lstrip(".") or "png",
|
||||
"references_used": request.get("references_used", []),
|
||||
"element_ids": request.get("element_ids", []),
|
||||
"cost_estimate_confidence": "low",
|
||||
"cost_estimate_basis": "Conservative estimate pending official account-usage reconciliation.",
|
||||
**self._account_usage_result(inputs, client),
|
||||
**self._callback_result_data(inputs, task_id),
|
||||
},
|
||||
artifacts=[str(path) for path in paths],
|
||||
cost_usd=self.estimate_cost(inputs),
|
||||
duration_seconds=round(time.time() - start, 2),
|
||||
model=request["model"],
|
||||
)
|
||||
|
||||
def _build_request(self, inputs: dict[str, Any]) -> dict[str, Any]:
|
||||
api_family = str(inputs.get("api_family", "generation"))
|
||||
if inputs.get("operation") == "omni":
|
||||
api_family = "omni"
|
||||
if api_family == "omni":
|
||||
return self._build_omni_request(inputs)
|
||||
return self._build_generation_request(inputs)
|
||||
|
||||
def _build_generation_request(self, inputs: dict[str, Any]) -> dict[str, Any]:
|
||||
prompt = self._prompt(inputs)
|
||||
model_name = str(inputs.get("model_name") or "kling-v3")
|
||||
if model_name not in IMAGE_GENERATION_MODELS:
|
||||
raise ValueError(f"model_name {model_name!r} is not supported for api_family=generation")
|
||||
payload: dict[str, Any] = {
|
||||
"model_name": model_name,
|
||||
"prompt": prompt,
|
||||
"resolution": inputs.get("resolution", "1k"),
|
||||
"n": int(inputs.get("n", 1) or 1),
|
||||
"aspect_ratio": inputs.get("aspect_ratio", "16:9"),
|
||||
}
|
||||
if len(prompt) > 2500:
|
||||
raise ValueError("prompt exceeds Kling image generation limit of 2500 characters")
|
||||
if inputs.get("negative_prompt"):
|
||||
payload["negative_prompt"] = inputs["negative_prompt"]
|
||||
image = normalize_image_input(inputs.get("image_url"), inputs.get("image_path"))
|
||||
if image:
|
||||
payload["image"] = image
|
||||
if inputs.get("image_reference"):
|
||||
payload["image_reference"] = inputs["image_reference"]
|
||||
for key in ("image_fidelity", "human_fidelity"):
|
||||
if inputs.get(key) is not None:
|
||||
payload[key] = inputs[key]
|
||||
elements = normalize_element_list(inputs.get("element_list"))
|
||||
if elements:
|
||||
payload["element_list"] = elements
|
||||
self._copy_common_task_fields(inputs, payload)
|
||||
return {
|
||||
"protocol": "classic",
|
||||
"path": "/v1/images/generations",
|
||||
"payload": payload,
|
||||
"api_family": "generation",
|
||||
"operation": "generate",
|
||||
"model": payload["model_name"],
|
||||
"references_used": self._reference_metadata_from_generation_payload(payload),
|
||||
"element_ids": element_ids(payload.get("element_list")),
|
||||
}
|
||||
|
||||
def _build_omni_request(self, inputs: dict[str, Any]) -> dict[str, Any]:
|
||||
model_name = str(inputs.get("model_name") or "kling-image-o1")
|
||||
if model_name not in OMNI_IMAGE_MODELS:
|
||||
raise ValueError(f"model_name {model_name!r} is not supported for api_family=omni")
|
||||
image_items, references_used = self._normalize_omni_image_list(inputs)
|
||||
prompt, prompt_references = build_image_prompt_references(self._prompt(inputs), image_items)
|
||||
references_used = prompt_references or references_used
|
||||
elements = normalize_element_list(inputs.get("element_list"))
|
||||
payload: dict[str, Any] = {
|
||||
"model_name": model_name,
|
||||
"prompt": prompt,
|
||||
"resolution": inputs.get("resolution", "1k"),
|
||||
"n": int(inputs.get("n", 1) or 1),
|
||||
"result_type": inputs.get("result_type", "single"),
|
||||
"aspect_ratio": inputs.get("aspect_ratio", "16:9"),
|
||||
}
|
||||
if image_items:
|
||||
payload["image_list"] = [{"image": item["image"]} for item in image_items]
|
||||
if elements:
|
||||
payload["element_list"] = elements
|
||||
if inputs.get("series_amount"):
|
||||
payload["series_amount"] = inputs["series_amount"]
|
||||
self._copy_common_task_fields(inputs, payload)
|
||||
return {
|
||||
"protocol": "classic",
|
||||
"path": "/v1/images/omni-image",
|
||||
"payload": payload,
|
||||
"api_family": "omni",
|
||||
"operation": "generate",
|
||||
"model": model_name,
|
||||
"references_used": references_used,
|
||||
"element_ids": [item["element_id"] for item in elements],
|
||||
}
|
||||
|
||||
def _normalize_omni_image_list(
|
||||
self,
|
||||
inputs: dict[str, Any],
|
||||
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
|
||||
image_list: list[dict[str, Any]] = []
|
||||
references_used: list[dict[str, Any]] = []
|
||||
|
||||
def add_image(value: str | None, *, source: str | None, source_type: str) -> None:
|
||||
if not value:
|
||||
return
|
||||
image_list.append({"image": value, "source": source or value, "source_type": source_type})
|
||||
references_used.append(
|
||||
{
|
||||
"kind": "image",
|
||||
"source": source or value,
|
||||
"source_type": source_type,
|
||||
"placeholder": f"<<<image_{len(image_list)}>>>",
|
||||
}
|
||||
)
|
||||
|
||||
for item in inputs.get("image_list") or []:
|
||||
if not isinstance(item, dict):
|
||||
raise ValueError("image_list items must be objects")
|
||||
source = item.get("image") or item.get("image_url") or item.get("image_path")
|
||||
value = normalize_image_input(item.get("image") or item.get("image_url"), item.get("image_path"))
|
||||
if not value:
|
||||
raise ValueError("image_list items must include image, image_url, or image_path")
|
||||
add_image(value, source=source, source_type="image_list")
|
||||
for url in inputs.get("image_urls") or []:
|
||||
add_image(normalize_image_input(url=url), source=url, source_type="image_urls")
|
||||
for path in inputs.get("image_paths") or []:
|
||||
add_image(normalize_image_input(path=path), source=str(path), source_type="image_paths")
|
||||
if inputs.get("image_url") or inputs.get("image_path"):
|
||||
add_image(
|
||||
normalize_image_input(inputs.get("image_url"), inputs.get("image_path")),
|
||||
source=inputs.get("image_url") or inputs.get("image_path"),
|
||||
source_type="image",
|
||||
)
|
||||
|
||||
return image_list, references_used
|
||||
|
||||
def _download_images(self, client: KlingClient, outputs: list[dict[str, Any]], inputs: dict[str, Any]) -> list[Path]:
|
||||
if not outputs:
|
||||
raise ValueError("Kling image response contained no images")
|
||||
base_path = Path(inputs.get("output_path", "kling_official_image.png"))
|
||||
paths: list[Path] = []
|
||||
for index, item in enumerate(outputs):
|
||||
url = self._output_url(item)
|
||||
suffix = extension_from_url(url, ".png")
|
||||
output_path = numbered_output_path(output_path_with_suffix(base_path, suffix), index, suffix)
|
||||
client.download(url, output_path)
|
||||
paths.append(output_path)
|
||||
return paths
|
||||
|
||||
@staticmethod
|
||||
def _output_url(item: dict[str, Any]) -> str:
|
||||
url = item.get("url") or item.get("image_url") or item.get("resource_url")
|
||||
if url:
|
||||
return str(url)
|
||||
resource = item.get("resource") or {}
|
||||
if isinstance(resource, dict) and resource.get("url"):
|
||||
return str(resource["url"])
|
||||
raise ValueError(f"Kling image response item contained no downloadable URL: {item}")
|
||||
|
||||
@staticmethod
|
||||
def _prompt(inputs: dict[str, Any]) -> str:
|
||||
prompt = str(inputs.get("prompt") or "").strip()
|
||||
if not prompt:
|
||||
raise ValueError("prompt is required")
|
||||
return prompt
|
||||
|
||||
@staticmethod
|
||||
def _copy_common_task_fields(inputs: dict[str, Any], payload: dict[str, Any]) -> None:
|
||||
if "watermark" in inputs:
|
||||
payload["watermark_info"] = {"enabled": bool(inputs.get("watermark"))}
|
||||
callback_url = validate_callback_url(inputs.get("callback_url"))
|
||||
if callback_url:
|
||||
payload["callback_url"] = callback_url
|
||||
if inputs.get("external_task_id"):
|
||||
payload["external_task_id"] = inputs["external_task_id"]
|
||||
|
||||
@staticmethod
|
||||
def _reference_metadata_from_generation_payload(payload: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
references: list[dict[str, Any]] = []
|
||||
if payload.get("image"):
|
||||
references.append({"kind": "image", "source_type": "image"})
|
||||
if payload.get("element_list"):
|
||||
references.extend(
|
||||
{"kind": "element", "element_id": item["element_id"]}
|
||||
for item in normalize_element_list(payload.get("element_list"))
|
||||
)
|
||||
return references
|
||||
|
||||
@staticmethod
|
||||
def _callback_result_data(inputs: dict[str, Any], task_id: str) -> dict[str, Any]:
|
||||
callback_url = inputs.get("callback_url")
|
||||
if not callback_url:
|
||||
return {}
|
||||
return {
|
||||
"callback_url": str(callback_url),
|
||||
"callback_requested": True,
|
||||
"polling_used": True,
|
||||
"task_id": task_id,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _account_usage_result(inputs: dict[str, Any], client: KlingClient) -> dict[str, Any]:
|
||||
if not inputs.get("include_account_usage"):
|
||||
return {}
|
||||
try:
|
||||
usage = get_account_costs(client=client)
|
||||
return {
|
||||
"account_usage": usage,
|
||||
"cost_source": "estimate_with_account_usage_context",
|
||||
"reconciled_cost_usd": None,
|
||||
}
|
||||
except Exception as exc:
|
||||
return {
|
||||
"account_usage_error": str(exc),
|
||||
"cost_source": "estimate",
|
||||
"reconciled_cost_usd": None,
|
||||
}
|
||||
|
|
@ -0,0 +1,684 @@
|
|||
"""Kling official API video generation provider."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from tools._kling.account import account_usage_hint_for_error, get_account_costs
|
||||
from tools._kling.callbacks import validate_callback_url
|
||||
from tools._kling.client import KlingClient
|
||||
from tools._kling.elements import element_ids, normalize_element_list
|
||||
from tools._kling.errors import KlingAPIError
|
||||
from tools._kling.media import (
|
||||
extension_from_url,
|
||||
normalize_image_input,
|
||||
numbered_output_path,
|
||||
output_path_with_suffix,
|
||||
)
|
||||
from tools._kling.schemas import (
|
||||
CLASSIC_VIDEO_MODELS,
|
||||
OMNI_VIDEO_MODELS,
|
||||
SOUND_VALUES,
|
||||
VIDEO_ASPECT_RATIOS,
|
||||
VIDEO_DURATIONS,
|
||||
VIDEO_MODES,
|
||||
VIDEO_MODELS,
|
||||
VIDEO_RESOLUTIONS,
|
||||
)
|
||||
from tools.base_tool import (
|
||||
BaseTool,
|
||||
DependencyError,
|
||||
Determinism,
|
||||
ExecutionMode,
|
||||
ResourceProfile,
|
||||
RetryPolicy,
|
||||
ToolResult,
|
||||
ToolRuntime,
|
||||
ToolStability,
|
||||
ToolTier,
|
||||
)
|
||||
from tools.video._shared import probe_output
|
||||
|
||||
|
||||
class KlingOfficialVideo(BaseTool):
|
||||
name = "kling_official_video"
|
||||
version = "0.1.0"
|
||||
tier = ToolTier.GENERATE
|
||||
capability = "video_generation"
|
||||
provider = "kling_official"
|
||||
stability = ToolStability.EXPERIMENTAL
|
||||
execution_mode = ExecutionMode.SYNC
|
||||
determinism = Determinism.STOCHASTIC
|
||||
runtime = ToolRuntime.API
|
||||
|
||||
dependencies = ["env:KLING_API_KEY"]
|
||||
install_instructions = (
|
||||
"Set KLING_API_KEY in .env for the official Kling API. "
|
||||
"Optionally set KLING_API_BASE_URL to override the default Singapore endpoint."
|
||||
)
|
||||
agent_skills = ["ai-video-gen", "kling-official"]
|
||||
|
||||
capabilities = ["text_to_video", "image_to_video", "reference_to_video"]
|
||||
supports = {
|
||||
"text_to_video": True,
|
||||
"image_to_video": True,
|
||||
"reference_to_video": True,
|
||||
"reference_image": True,
|
||||
"negative_prompt": True,
|
||||
"aspect_ratio": True,
|
||||
}
|
||||
best_for = [
|
||||
"official Kling direct API access",
|
||||
"text-to-video and image-to-video with Kling model controls",
|
||||
"projects that need provider provenance separate from fal.ai Kling",
|
||||
]
|
||||
not_good_for = ["offline generation", "free generation", "non-Kling model families"]
|
||||
fallback_tools = ["kling_video", "seedance_video", "veo_video", "minimax_video"]
|
||||
|
||||
input_schema = {
|
||||
"type": "object",
|
||||
"required": ["prompt"],
|
||||
"properties": {
|
||||
"prompt": {"type": "string"},
|
||||
"operation": {
|
||||
"type": "string",
|
||||
"enum": ["text_to_video", "image_to_video", "reference_to_video", "omni_video"],
|
||||
"default": "text_to_video",
|
||||
},
|
||||
"api_family": {
|
||||
"type": "string",
|
||||
"enum": ["classic", "turbo", "omni"],
|
||||
"default": "classic",
|
||||
},
|
||||
"model_name": {"type": "string", "enum": VIDEO_MODELS, "default": "kling-v3"},
|
||||
"model_variant": {"type": "string", "description": "Compatibility alias for model_name."},
|
||||
"duration": {"type": "string", "enum": VIDEO_DURATIONS, "default": "5"},
|
||||
"aspect_ratio": {"type": "string", "enum": VIDEO_ASPECT_RATIOS, "default": "16:9"},
|
||||
"resolution": {"type": "string", "enum": VIDEO_RESOLUTIONS, "default": "720p"},
|
||||
"mode": {"type": "string", "enum": VIDEO_MODES, "default": "std"},
|
||||
"sound": {"type": "string", "enum": SOUND_VALUES, "default": "off"},
|
||||
"negative_prompt": {"type": "string"},
|
||||
"cfg_scale": {"type": "number", "default": 0.5},
|
||||
"reference_image_url": {"type": "string"},
|
||||
"reference_image_path": {"type": "string"},
|
||||
"reference_tail_image_url": {"type": "string"},
|
||||
"reference_tail_image_path": {"type": "string"},
|
||||
"reference_image_urls": {"type": "array", "items": {"type": "string"}},
|
||||
"reference_image_paths": {"type": "array", "items": {"type": "string"}},
|
||||
"reference_video_url": {"type": "string"},
|
||||
"reference_video_path": {"type": "string"},
|
||||
"video_urls": {"type": "array", "items": {"type": "string"}},
|
||||
"video_paths": {"type": "array", "items": {"type": "string"}},
|
||||
"image_list": {"type": "array"},
|
||||
"video_list": {"type": "array"},
|
||||
"element_list": {"type": "array"},
|
||||
"multi_shot": {"type": "boolean"},
|
||||
"shot_type": {"type": "string", "enum": ["customize", "intelligence"]},
|
||||
"multi_prompt": {"type": "array"},
|
||||
"camera_control": {"type": "object"},
|
||||
"watermark": {"type": "boolean", "default": False},
|
||||
"callback_url": {"type": "string"},
|
||||
"external_task_id": {"type": "string"},
|
||||
"include_account_usage": {
|
||||
"type": "boolean",
|
||||
"default": False,
|
||||
"description": "Optional low-frequency account usage diagnostic; not used by default.",
|
||||
},
|
||||
"timeout_seconds": {"type": "integer", "default": 900},
|
||||
"poll_interval": {"type": "number", "default": 5.0},
|
||||
"output_path": {"type": "string"},
|
||||
},
|
||||
}
|
||||
|
||||
resource_profile = ResourceProfile(
|
||||
cpu_cores=1, ram_mb=512, vram_mb=0, disk_mb=500, network_required=True
|
||||
)
|
||||
retry_policy = RetryPolicy(
|
||||
max_retries=2,
|
||||
backoff_seconds=2.0,
|
||||
retryable_errors=["1302", "1303", "5000", "5001", "5002"],
|
||||
)
|
||||
idempotency_key_fields = [
|
||||
"prompt",
|
||||
"operation",
|
||||
"api_family",
|
||||
"model_name",
|
||||
"reference_image_url",
|
||||
"reference_image_path",
|
||||
"duration",
|
||||
"aspect_ratio",
|
||||
]
|
||||
side_effects = [
|
||||
"paid remote generation via official Kling API",
|
||||
"writes video file to output_path",
|
||||
]
|
||||
user_visible_verification = ["Watch generated clip for motion coherence and prompt adherence"]
|
||||
|
||||
def estimate_cost(self, inputs: dict[str, Any]) -> float:
|
||||
duration = int(str(inputs.get("duration", "5")))
|
||||
mode = str(inputs.get("mode", "std"))
|
||||
api_family = str(inputs.get("api_family", "classic"))
|
||||
base = 0.18
|
||||
if api_family == "turbo":
|
||||
base = 0.22
|
||||
if api_family == "omni":
|
||||
base = 0.30
|
||||
if mode == "pro":
|
||||
base *= 1.6
|
||||
if mode == "4k":
|
||||
base *= 3.0
|
||||
if inputs.get("sound") == "on":
|
||||
base += 0.05
|
||||
if api_family == "omni":
|
||||
reference_count = self._estimate_reference_count(inputs)
|
||||
base *= 1 + (0.12 * reference_count)
|
||||
multi_prompt = inputs.get("multi_prompt") or []
|
||||
if multi_prompt:
|
||||
base *= 1 + (0.10 * len(multi_prompt))
|
||||
return round(base * max(duration, 3) / 5, 4)
|
||||
|
||||
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
|
||||
return 180.0
|
||||
|
||||
def dry_run(self, inputs: dict[str, Any]) -> dict[str, Any]:
|
||||
result = super().dry_run(inputs)
|
||||
result.update(
|
||||
{
|
||||
"paid_api": True,
|
||||
"cost_estimate_confidence": "low",
|
||||
"cost_estimate_basis": "Conservative OpenMontage estimate; official account usage reconciliation is planned for Phase 2.",
|
||||
}
|
||||
)
|
||||
return result
|
||||
|
||||
def execute(self, inputs: dict[str, Any]) -> ToolResult:
|
||||
try:
|
||||
self.check_dependencies()
|
||||
except DependencyError as exc:
|
||||
return ToolResult(success=False, error=str(exc))
|
||||
|
||||
start = time.time()
|
||||
try:
|
||||
request = self._build_request(inputs)
|
||||
client = KlingClient()
|
||||
if request["protocol"] == "turbo":
|
||||
task_id = client.create_turbo(request["path"], request["payload"])
|
||||
outputs = client.poll_turbo(
|
||||
task_id,
|
||||
timeout_seconds=int(inputs.get("timeout_seconds", 900)),
|
||||
poll_interval=float(inputs.get("poll_interval", 5.0)),
|
||||
)
|
||||
else:
|
||||
task_id = client.create_classic_task(request["path"], request["payload"])
|
||||
outputs = client.poll_classic(
|
||||
request["path"],
|
||||
task_id,
|
||||
"videos",
|
||||
timeout_seconds=int(inputs.get("timeout_seconds", 900)),
|
||||
poll_interval=float(inputs.get("poll_interval", 5.0)),
|
||||
)
|
||||
paths = self._download_videos(client, outputs, inputs)
|
||||
video_url = self._first_output_url(outputs)
|
||||
probed = probe_output(paths[0])
|
||||
except (KlingAPIError, TimeoutError, ValueError, KeyError, FileNotFoundError) as exc:
|
||||
data: dict[str, Any] = {"provider": self.provider}
|
||||
if isinstance(exc, KlingAPIError):
|
||||
data.update(
|
||||
{
|
||||
"error_code": exc.code,
|
||||
"request_id": exc.request_id,
|
||||
"http_status": exc.http_status,
|
||||
}
|
||||
)
|
||||
data["account_usage_diagnostic"] = account_usage_hint_for_error(exc)
|
||||
return ToolResult(success=False, data=data, error=f"Kling official video generation failed: {exc}")
|
||||
except Exception as exc:
|
||||
return ToolResult(success=False, data={"provider": self.provider}, error=f"Kling official video generation failed: {exc}")
|
||||
|
||||
return ToolResult(
|
||||
success=True,
|
||||
data={
|
||||
"provider": self.provider,
|
||||
"model": request["model"],
|
||||
"task_id": task_id,
|
||||
"operation": request["operation"],
|
||||
"api_family": request["api_family"],
|
||||
"prompt": inputs["prompt"],
|
||||
"remote_url": video_url,
|
||||
"remote_outputs": outputs,
|
||||
"output": str(paths[0]),
|
||||
"output_path": str(paths[0]),
|
||||
"video_paths": [str(path) for path in paths],
|
||||
"format": "mp4",
|
||||
"references_used": request.get("references_used", []),
|
||||
"element_ids": request.get("element_ids", []),
|
||||
"cost_estimate_confidence": "low",
|
||||
"cost_estimate_basis": "Conservative estimate pending official account-usage reconciliation.",
|
||||
**self._account_usage_result(inputs, client),
|
||||
**self._callback_result_data(inputs, task_id),
|
||||
**probed,
|
||||
},
|
||||
artifacts=[str(path) for path in paths],
|
||||
cost_usd=self.estimate_cost(inputs),
|
||||
duration_seconds=round(time.time() - start, 2),
|
||||
model=request["model"],
|
||||
)
|
||||
|
||||
def _build_request(self, inputs: dict[str, Any]) -> dict[str, Any]:
|
||||
operation = str(inputs.get("operation", "text_to_video"))
|
||||
api_family = str(inputs.get("api_family", "classic"))
|
||||
if operation == "omni_video":
|
||||
operation = "reference_to_video"
|
||||
api_family = "omni"
|
||||
if api_family == "turbo":
|
||||
return self._build_turbo_request(inputs, operation)
|
||||
if api_family == "omni":
|
||||
return self._build_omni_request(inputs, operation)
|
||||
return self._build_classic_request(inputs, operation)
|
||||
|
||||
def _build_classic_request(self, inputs: dict[str, Any], operation: str) -> dict[str, Any]:
|
||||
if operation == "text_to_video":
|
||||
payload = self._base_classic_payload(inputs)
|
||||
payload["prompt"] = self._prompt(inputs)
|
||||
if inputs.get("negative_prompt"):
|
||||
payload["negative_prompt"] = inputs["negative_prompt"]
|
||||
if inputs.get("aspect_ratio"):
|
||||
payload["aspect_ratio"] = inputs.get("aspect_ratio", "16:9")
|
||||
self._copy_multi_shot_fields(inputs, payload)
|
||||
path = "/v1/videos/text2video"
|
||||
elif operation == "image_to_video":
|
||||
image = normalize_image_input(inputs.get("reference_image_url"), inputs.get("reference_image_path"))
|
||||
if not image:
|
||||
raise ValueError("image_to_video requires reference_image_url or reference_image_path")
|
||||
payload = self._base_classic_payload(inputs)
|
||||
payload["image"] = image
|
||||
if inputs.get("prompt"):
|
||||
payload["prompt"] = inputs["prompt"]
|
||||
if inputs.get("negative_prompt"):
|
||||
payload["negative_prompt"] = inputs["negative_prompt"]
|
||||
tail = normalize_image_input(inputs.get("reference_tail_image_url"), inputs.get("reference_tail_image_path"))
|
||||
if tail:
|
||||
payload["image_tail"] = tail
|
||||
if inputs.get("element_list"):
|
||||
payload["element_list"] = normalize_element_list(inputs.get("element_list"))
|
||||
self._copy_multi_shot_fields(inputs, payload)
|
||||
path = "/v1/videos/image2video"
|
||||
else:
|
||||
raise ValueError(f"Unsupported classic video operation: {operation}")
|
||||
return {
|
||||
"protocol": "classic",
|
||||
"path": path,
|
||||
"payload": payload,
|
||||
"operation": operation,
|
||||
"api_family": "classic",
|
||||
"model": payload["model_name"],
|
||||
"references_used": self._reference_metadata_from_classic_payload(payload),
|
||||
"element_ids": element_ids(payload.get("element_list")),
|
||||
}
|
||||
|
||||
def _build_turbo_request(self, inputs: dict[str, Any], operation: str) -> dict[str, Any]:
|
||||
settings = {
|
||||
"resolution": inputs.get("resolution", "720p"),
|
||||
"duration": int(str(inputs.get("duration", "5"))),
|
||||
}
|
||||
options = self._options_payload(inputs)
|
||||
if operation == "text_to_video":
|
||||
settings["aspect_ratio"] = inputs.get("aspect_ratio", "16:9")
|
||||
payload = {"prompt": self._prompt(inputs), "settings": settings}
|
||||
if options:
|
||||
payload["options"] = options
|
||||
path = "/text-to-video/kling-3.0-turbo"
|
||||
elif operation == "image_to_video":
|
||||
if inputs.get("reference_image_path") and not inputs.get("reference_image_url"):
|
||||
raise ValueError("Turbo image_to_video requires reference_image_url; local paths cannot be silently uploaded.")
|
||||
image_url = inputs.get("reference_image_url")
|
||||
if not image_url:
|
||||
raise ValueError("image_to_video requires reference_image_url for api_family=turbo")
|
||||
contents = [{"type": "prompt", "text": self._prompt(inputs)}, {"type": "first_frame", "url": image_url}]
|
||||
payload = {"contents": contents, "settings": settings}
|
||||
if options:
|
||||
payload["options"] = options
|
||||
path = "/image-to-video/kling-3.0-turbo"
|
||||
else:
|
||||
raise ValueError(f"Unsupported turbo video operation: {operation}")
|
||||
return {
|
||||
"protocol": "turbo",
|
||||
"path": path,
|
||||
"payload": payload,
|
||||
"operation": operation,
|
||||
"api_family": "turbo",
|
||||
"model": "kling-3.0-turbo",
|
||||
}
|
||||
|
||||
def _build_omni_request(self, inputs: dict[str, Any], operation: str) -> dict[str, Any]:
|
||||
explicit_model = inputs.get("model_name") or inputs.get("model_variant")
|
||||
model_name = str(explicit_model or "kling-video-o1")
|
||||
if model_name not in OMNI_VIDEO_MODELS:
|
||||
raise ValueError(f"model_name {model_name!r} is not supported for api_family=omni")
|
||||
payload, references_used, element_id_values = self._build_omni_payload(inputs, operation, model_name)
|
||||
return {
|
||||
"protocol": "classic",
|
||||
"path": "/v1/videos/omni-video",
|
||||
"payload": payload,
|
||||
"operation": operation,
|
||||
"api_family": "omni",
|
||||
"model": model_name,
|
||||
"references_used": references_used,
|
||||
"element_ids": element_id_values,
|
||||
}
|
||||
|
||||
def _build_omni_payload(
|
||||
self,
|
||||
inputs: dict[str, Any],
|
||||
operation: str,
|
||||
model_name: str,
|
||||
) -> tuple[dict[str, Any], list[dict[str, Any]], list[int]]:
|
||||
payload: dict[str, Any] = {
|
||||
"model_name": model_name,
|
||||
"prompt": self._prompt(inputs),
|
||||
"mode": inputs.get("mode", "pro"),
|
||||
"duration": str(inputs.get("duration", "5")),
|
||||
}
|
||||
if inputs.get("sound"):
|
||||
payload["sound"] = inputs["sound"]
|
||||
if inputs.get("aspect_ratio"):
|
||||
payload["aspect_ratio"] = inputs["aspect_ratio"]
|
||||
self._copy_common_task_fields(inputs, payload)
|
||||
self._copy_multi_shot_fields(inputs, payload)
|
||||
|
||||
references_used: list[dict[str, Any]] = []
|
||||
image_list, image_refs = self._normalize_omni_image_list(inputs)
|
||||
references_used.extend(image_refs)
|
||||
if image_list:
|
||||
payload["image_list"] = image_list
|
||||
|
||||
video_list, video_refs = self._normalize_omni_video_list(inputs)
|
||||
references_used.extend(video_refs)
|
||||
if video_list:
|
||||
payload["video_list"] = video_list
|
||||
|
||||
elements = normalize_element_list(inputs.get("element_list"))
|
||||
element_id_values = [item["element_id"] for item in elements]
|
||||
if elements:
|
||||
payload["element_list"] = elements
|
||||
references_used.extend(
|
||||
{"kind": "element", "element_id": item["element_id"]}
|
||||
for item in elements
|
||||
)
|
||||
if operation == "reference_to_video" and not any(payload.get(k) for k in ("image_list", "video_list", "element_list")):
|
||||
raise ValueError("reference_to_video with api_family=omni requires image_list, video_list, element_list, or reference image URLs.")
|
||||
return payload, references_used, element_id_values
|
||||
|
||||
def _base_classic_payload(self, inputs: dict[str, Any]) -> dict[str, Any]:
|
||||
model_name = str(inputs.get("model_name") or inputs.get("model_variant") or "kling-v3")
|
||||
if model_name not in CLASSIC_VIDEO_MODELS:
|
||||
raise ValueError(f"model_name {model_name!r} is not supported for api_family=classic")
|
||||
payload: dict[str, Any] = {
|
||||
"model_name": model_name,
|
||||
"duration": str(inputs.get("duration", "5")),
|
||||
"mode": inputs.get("mode", "std"),
|
||||
"sound": inputs.get("sound", "off"),
|
||||
}
|
||||
if inputs.get("cfg_scale") is not None:
|
||||
payload["cfg_scale"] = inputs["cfg_scale"]
|
||||
if inputs.get("camera_control"):
|
||||
payload["camera_control"] = inputs["camera_control"]
|
||||
self._copy_common_task_fields(inputs, payload)
|
||||
return payload
|
||||
|
||||
def _options_payload(self, inputs: dict[str, Any]) -> dict[str, Any]:
|
||||
options: dict[str, Any] = {}
|
||||
callback_url = validate_callback_url(inputs.get("callback_url"))
|
||||
if callback_url:
|
||||
options["callback_url"] = callback_url
|
||||
if inputs.get("external_task_id"):
|
||||
options["external_task_id"] = inputs["external_task_id"]
|
||||
if "watermark" in inputs:
|
||||
options["watermark_info"] = {"enabled": bool(inputs.get("watermark"))}
|
||||
return options
|
||||
|
||||
def _copy_common_task_fields(self, inputs: dict[str, Any], payload: dict[str, Any]) -> None:
|
||||
if "watermark" in inputs:
|
||||
payload["watermark_info"] = {"enabled": bool(inputs.get("watermark"))}
|
||||
callback_url = validate_callback_url(inputs.get("callback_url"))
|
||||
if callback_url:
|
||||
payload["callback_url"] = callback_url
|
||||
if inputs.get("external_task_id"):
|
||||
payload["external_task_id"] = inputs["external_task_id"]
|
||||
|
||||
def _copy_multi_shot_fields(self, inputs: dict[str, Any], payload: dict[str, Any]) -> None:
|
||||
if inputs.get("multi_shot") is None and not inputs.get("multi_prompt"):
|
||||
return
|
||||
payload["multi_shot"] = bool(inputs.get("multi_shot", True))
|
||||
shot_type = str(inputs.get("shot_type") or "customize")
|
||||
if shot_type not in {"customize", "intelligence"}:
|
||||
raise ValueError("shot_type must be one of: customize, intelligence")
|
||||
payload["shot_type"] = shot_type
|
||||
if inputs.get("multi_prompt"):
|
||||
if not isinstance(inputs["multi_prompt"], list):
|
||||
raise ValueError("multi_prompt must be a list")
|
||||
normalized: list[dict[str, Any]] = []
|
||||
for item in inputs["multi_prompt"]:
|
||||
if not isinstance(item, dict) or not item.get("prompt"):
|
||||
raise ValueError("each multi_prompt item must be an object with prompt")
|
||||
allowed = {
|
||||
key: item[key]
|
||||
for key in ("prompt", "duration", "camera_control", "image_refs", "element_refs")
|
||||
if key in item
|
||||
}
|
||||
normalized.append(allowed)
|
||||
payload["multi_prompt"] = normalized
|
||||
|
||||
def _normalize_omni_image_list(
|
||||
self,
|
||||
inputs: dict[str, Any],
|
||||
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
|
||||
image_list: list[dict[str, Any]] = []
|
||||
references_used: list[dict[str, Any]] = []
|
||||
|
||||
def add_image(value: str | None, *, kind: str, item_type: str | None = None) -> None:
|
||||
if not value:
|
||||
return
|
||||
record = {"image_url": value}
|
||||
if item_type:
|
||||
record["type"] = item_type
|
||||
image_list.append(record)
|
||||
references_used.append(
|
||||
{
|
||||
"kind": "image",
|
||||
"source": value,
|
||||
"source_type": kind,
|
||||
"type": item_type,
|
||||
}
|
||||
)
|
||||
|
||||
for item in inputs.get("image_list") or []:
|
||||
if not isinstance(item, dict):
|
||||
raise ValueError("image_list items must be objects")
|
||||
value = normalize_image_input(item.get("image_url") or item.get("image"), item.get("image_path"))
|
||||
if not value:
|
||||
raise ValueError("image_list items must include image_url, image, or image_path")
|
||||
record = {"image_url": value}
|
||||
if item.get("type"):
|
||||
record["type"] = item["type"]
|
||||
image_list.append(record)
|
||||
references_used.append(
|
||||
{
|
||||
"kind": "image",
|
||||
"source": item.get("image_url") or item.get("image_path") or item.get("image"),
|
||||
"source_type": "image_list",
|
||||
"type": item.get("type"),
|
||||
}
|
||||
)
|
||||
|
||||
add_image(
|
||||
normalize_image_input(inputs.get("reference_image_url"), inputs.get("reference_image_path")),
|
||||
kind="reference_image",
|
||||
item_type="first_frame",
|
||||
)
|
||||
add_image(
|
||||
normalize_image_input(inputs.get("reference_tail_image_url"), inputs.get("reference_tail_image_path")),
|
||||
kind="reference_tail_image",
|
||||
item_type="end_frame",
|
||||
)
|
||||
for url in inputs.get("reference_image_urls") or []:
|
||||
add_image(normalize_image_input(url=url), kind="reference_image_urls")
|
||||
for path in inputs.get("reference_image_paths") or []:
|
||||
add_image(normalize_image_input(path=path), kind="reference_image_paths")
|
||||
return image_list, references_used
|
||||
|
||||
def _normalize_omni_video_list(
|
||||
self,
|
||||
inputs: dict[str, Any],
|
||||
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
|
||||
if inputs.get("reference_video_path") or inputs.get("video_paths"):
|
||||
raise ValueError("Video Omni requires video URLs; local video paths cannot be silently uploaded.")
|
||||
|
||||
video_list: list[dict[str, Any]] = []
|
||||
references_used: list[dict[str, Any]] = []
|
||||
|
||||
def add_video(item: dict[str, Any], source_type: str) -> None:
|
||||
if item.get("video_path"):
|
||||
raise ValueError("Video Omni requires video URLs; local video paths cannot be silently uploaded.")
|
||||
video_url = item.get("video_url") or item.get("url")
|
||||
if not video_url:
|
||||
raise ValueError("video_list items must include video_url")
|
||||
record = {"video_url": video_url}
|
||||
if item.get("refer_type"):
|
||||
record["refer_type"] = item["refer_type"]
|
||||
if "keep_original_sound" in item:
|
||||
value = item["keep_original_sound"]
|
||||
record["keep_original_sound"] = "yes" if value is True else "no" if value is False else value
|
||||
video_list.append(record)
|
||||
references_used.append(
|
||||
{
|
||||
"kind": "video",
|
||||
"source": video_url,
|
||||
"source_type": source_type,
|
||||
"refer_type": record.get("refer_type"),
|
||||
"keep_original_sound": record.get("keep_original_sound"),
|
||||
}
|
||||
)
|
||||
|
||||
for item in inputs.get("video_list") or []:
|
||||
if not isinstance(item, dict):
|
||||
raise ValueError("video_list items must be objects")
|
||||
add_video(item, "video_list")
|
||||
if inputs.get("reference_video_url"):
|
||||
add_video({"video_url": inputs["reference_video_url"]}, "reference_video_url")
|
||||
for url in inputs.get("video_urls") or []:
|
||||
add_video({"video_url": url}, "video_urls")
|
||||
return video_list, references_used
|
||||
|
||||
def _download_videos(
|
||||
self,
|
||||
client: KlingClient,
|
||||
outputs: list[dict[str, Any]],
|
||||
inputs: dict[str, Any],
|
||||
) -> list[Path]:
|
||||
if not outputs:
|
||||
raise ValueError("Kling video response contained no videos")
|
||||
base_path = Path(inputs.get("output_path", "kling_official_video.mp4"))
|
||||
paths: list[Path] = []
|
||||
for index, item in enumerate(outputs):
|
||||
url = self._output_url(item)
|
||||
suffix = extension_from_url(url, ".mp4")
|
||||
output_path = numbered_output_path(output_path_with_suffix(base_path, suffix), index, suffix)
|
||||
client.download(url, output_path)
|
||||
paths.append(output_path)
|
||||
return paths
|
||||
|
||||
@staticmethod
|
||||
def _output_url(item: dict[str, Any]) -> str:
|
||||
url = item.get("url") or item.get("video_url") or item.get("resource_url")
|
||||
if url:
|
||||
return str(url)
|
||||
resource = item.get("resource") or {}
|
||||
if isinstance(resource, dict) and resource.get("url"):
|
||||
return str(resource["url"])
|
||||
raise ValueError(f"Kling video response contained no downloadable URL: {item}")
|
||||
|
||||
@staticmethod
|
||||
def _reference_metadata_from_classic_payload(payload: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
references: list[dict[str, Any]] = []
|
||||
if payload.get("image"):
|
||||
references.append({"kind": "image", "source_type": "reference_image"})
|
||||
if payload.get("image_tail"):
|
||||
references.append({"kind": "image", "source_type": "reference_tail_image"})
|
||||
if payload.get("element_list"):
|
||||
references.extend(
|
||||
{"kind": "element", "element_id": item["element_id"]}
|
||||
for item in normalize_element_list(payload.get("element_list"))
|
||||
)
|
||||
return references
|
||||
|
||||
@staticmethod
|
||||
def _callback_result_data(inputs: dict[str, Any], task_id: str) -> dict[str, Any]:
|
||||
callback_url = inputs.get("callback_url")
|
||||
if not callback_url:
|
||||
return {}
|
||||
return {
|
||||
"callback_url": str(callback_url),
|
||||
"callback_requested": True,
|
||||
"polling_used": True,
|
||||
"task_id": task_id,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _account_usage_result(inputs: dict[str, Any], client: KlingClient) -> dict[str, Any]:
|
||||
if not inputs.get("include_account_usage"):
|
||||
return {}
|
||||
try:
|
||||
usage = get_account_costs(client=client)
|
||||
return {
|
||||
"account_usage": usage,
|
||||
"cost_source": "estimate_with_account_usage_context",
|
||||
"reconciled_cost_usd": None,
|
||||
}
|
||||
except Exception as exc:
|
||||
return {
|
||||
"account_usage_error": str(exc),
|
||||
"cost_source": "estimate",
|
||||
"reconciled_cost_usd": None,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _estimate_reference_count(inputs: dict[str, Any]) -> int:
|
||||
count = 0
|
||||
for key in (
|
||||
"image_list",
|
||||
"video_list",
|
||||
"element_list",
|
||||
"reference_image_urls",
|
||||
"reference_image_paths",
|
||||
"video_urls",
|
||||
):
|
||||
count += len(inputs.get(key) or [])
|
||||
for key in (
|
||||
"reference_image_url",
|
||||
"reference_image_path",
|
||||
"reference_tail_image_url",
|
||||
"reference_tail_image_path",
|
||||
"reference_video_url",
|
||||
):
|
||||
if inputs.get(key):
|
||||
count += 1
|
||||
return count
|
||||
|
||||
@staticmethod
|
||||
def _prompt(inputs: dict[str, Any]) -> str:
|
||||
prompt = str(inputs.get("prompt") or "").strip()
|
||||
if not prompt:
|
||||
raise ValueError("prompt is required")
|
||||
return prompt
|
||||
|
||||
@staticmethod
|
||||
def _first_output_url(outputs: list[dict[str, Any]]) -> str:
|
||||
for item in outputs:
|
||||
try:
|
||||
return KlingOfficialVideo._output_url(item)
|
||||
except ValueError:
|
||||
continue
|
||||
raise ValueError(f"Kling video response contained no downloadable URL: {outputs}")
|
||||
|
|
@ -88,6 +88,38 @@ class VideoSelector(BaseTool):
|
|||
"items": {"type": "string"},
|
||||
"description": "Local reference image paths for providers that support reference-conditioned video.",
|
||||
},
|
||||
"reference_video_url": {
|
||||
"type": "string",
|
||||
"description": "Reference video URL for providers that support video-conditioned generation.",
|
||||
},
|
||||
"reference_video_path": {
|
||||
"type": "string",
|
||||
"description": "Local reference video path. Providers that require URLs should reject this clearly.",
|
||||
},
|
||||
"image_list": {
|
||||
"type": "array",
|
||||
"description": "Provider-specific list of image references, e.g. Kling Official Video Omni.",
|
||||
},
|
||||
"video_list": {
|
||||
"type": "array",
|
||||
"description": "Provider-specific list of video references, e.g. Kling Official Video Omni.",
|
||||
},
|
||||
"element_list": {
|
||||
"type": "array",
|
||||
"description": "Provider-specific element references, e.g. Kling Official element_id objects.",
|
||||
},
|
||||
"multi_shot": {
|
||||
"type": "boolean",
|
||||
"description": "Provider-specific multi-shot mode.",
|
||||
},
|
||||
"shot_type": {
|
||||
"type": "string",
|
||||
"description": "Provider-specific multi-shot type.",
|
||||
},
|
||||
"multi_prompt": {
|
||||
"type": "array",
|
||||
"description": "Structured multi-shot prompts; not inferred from prose.",
|
||||
},
|
||||
"image_url": {
|
||||
"type": "string",
|
||||
"description": "Alias for reference_image_url (used by some providers like Kling via fal.ai).",
|
||||
|
|
@ -96,6 +128,34 @@ class VideoSelector(BaseTool):
|
|||
"type": "string",
|
||||
"description": "Resolution hint for providers that support named output resolutions.",
|
||||
},
|
||||
"api_family": {
|
||||
"type": "string",
|
||||
"description": "Provider-specific API family hint passed through when supported, e.g. classic/turbo/omni.",
|
||||
},
|
||||
"model_name": {
|
||||
"type": "string",
|
||||
"description": "Provider-specific model name passed through when supported.",
|
||||
},
|
||||
"mode": {
|
||||
"type": "string",
|
||||
"description": "Provider-specific quality mode passed through when supported.",
|
||||
},
|
||||
"sound": {
|
||||
"type": "string",
|
||||
"description": "Provider-specific native audio toggle passed through when supported.",
|
||||
},
|
||||
"watermark": {
|
||||
"type": "boolean",
|
||||
"description": "Provider-specific watermark toggle passed through when supported.",
|
||||
},
|
||||
"callback_url": {
|
||||
"type": "string",
|
||||
"description": "Provider-specific callback URL. Current OpenMontage providers still poll by default.",
|
||||
},
|
||||
"external_task_id": {
|
||||
"type": "string",
|
||||
"description": "Provider-specific idempotency/provenance task id.",
|
||||
},
|
||||
"workflow_json": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
|
|
|
|||
Loading…
Reference in New Issue