Capabilities & Use Cases
This article describes how to call models via the Ark API to build GUI Agent applications that parse executable action intents from UI screenshots and prompts. Supported models are the Doubao Seed series. I. Task flow: the model operates the GUI like a human — look at a screenshot → decide the next step → execute the action → look at the new screen, looping until the task is complete. Output in plain text without <seed:tool_call> is treated as task completion. Two usage modes: Mode 1, GUI Agent executes a multi-turn loop of observe → decide → act → observe again; Mode 2, pure Grounding performs only single-step localization and outputs a click coordinate. II. Key conventions: when calling chat.completions.create, do not pass the tools parameter; otherwise it goes through OpenAI function calling, and the output format will be incompatible with this guide. The model is trained to write action blocks directly into message.content, with the chain of thought in the separate field message.reasoning_content. All click/drag coordinates are relative coordinates in the 0–1000 range, independent of screenshot resolution; the mapping formula is px = round(x / 1000 * width). III. Mode 1 implementation notes: the system prompt consists of two parts — TASK_SP (task framework) and FUNCTION_SP (action definitions), defining 8 discrete actions (click / drag / hotkey / left_double / right_single / scroll / type / wait); screenshots are converted to Base64 for transmission; action parsing calls the official parser parse_xml_action_65_with_validate_soft from the ui-tars package; an empty list returned means the model finished with plain text and the interaction has ended. IV. Mode 2, pure Grounding: one instruction plus one screenshot outputs one click action coordinate, with no loop, no end-condition check, and no actual execution of actions; coordinates support integer or one-decimal precision, with the origin at the top-left corner of the screenshot. V. Deployment and operations constraints: RGBA images should be converted to ("RGB") before encoding; the first multimodal request takes about 10~23s, with timeout set to ≥ 120; Ark has concurrency limits — for Grounding batch evaluation, a concurrency of 4–8 is recommended, --max-retries ≥ 3, and the batch evaluation script's --timeout defaults to 300s.
No verified item-by-item scores available; capability radar and overall ranking will not be displayed for now.
Data & Sources
- Data Provider
- 火山方舟
- Official Notes
- Verify official materials ↗
- On-site profile update
- 2026-11-22 02:01:23
On-site update time is not equivalent to the model release date. Official or platform benchmark scores are not displayed when test sets, dates, or original results are not provided.
Version History
Historical versions, release and discontinuation dates to be verified.
Access & Usage
Please confirm the API address, model identifier, authentication, permissions, and billing rules according to the official documentation.
Open Official Documentation