此语言仍有 24 条内容或界面任务待翻译;未完成部分显示中文原文。
AI Home / Model Library / Volcano Ark GUI Agent Capability

Volcano Ark GUI Agent Capability

AI 辅助整理
火山方舟GUI Agent界面截图解析动作意图输出单步 Grounding 定位多轮任务回环Enter Reference Price · Official Prices Prevail

Volcano Ark's GUI Agent capability documentation, based on the Doubao Seed series models, parses executable action intents from UI screenshots and prompts. Two usage modes are supported: GUI Agent outputs discrete actions (multi-turn observe-decide-act loop), and pure Grounding for single-step localization (one click coordinate per output). Calls must use the native OpenAI SDK without passing tools, and action parsing uses the official ui-tars parser. Suitable for building Agent applications that automate GUI operations, but note that coordinates are 0–1000 relative coordinates requiring manual pixel conversion, the first multimodal request takes about 10~23 seconds, and Ark has concurrency limits.

Context tokensPending Verification
Input / Million tokensPending Verification
Output / Million tokensPending Verification
Versiondoubao-seed-2.1-turbo
Deployment云端 API(Ark 推理接入点,OpenAI 兼容 chat.completions,不传 tools,思维链在 message.reasoning_content)

Capabilities & Use Cases

GUI Agent界面截图解析动作意图输出单步 Grounding 定位多轮任务回环

This article describes how to call models via the Ark API to build GUI Agent applications that parse executable action intents from UI screenshots and prompts. Supported models are the Doubao Seed series. I. Task flow: the model operates the GUI like a human — look at a screenshot → decide the next step → execute the action → look at the new screen, looping until the task is complete. Output in plain text without <seed:tool_call> is treated as task completion. Two usage modes: Mode 1, GUI Agent executes a multi-turn loop of observe → decide → act → observe again; Mode 2, pure Grounding performs only single-step localization and outputs a click coordinate. II. Key conventions: when calling chat.completions.create, do not pass the tools parameter; otherwise it goes through OpenAI function calling, and the output format will be incompatible with this guide. The model is trained to write action blocks directly into message.content, with the chain of thought in the separate field message.reasoning_content. All click/drag coordinates are relative coordinates in the 0–1000 range, independent of screenshot resolution; the mapping formula is px = round(x / 1000 * width). III. Mode 1 implementation notes: the system prompt consists of two parts — TASK_SP (task framework) and FUNCTION_SP (action definitions), defining 8 discrete actions (click / drag / hotkey / left_double / right_single / scroll / type / wait); screenshots are converted to Base64 for transmission; action parsing calls the official parser parse_xml_action_65_with_validate_soft from the ui-tars package; an empty list returned means the model finished with plain text and the interaction has ended. IV. Mode 2, pure Grounding: one instruction plus one screenshot outputs one click action coordinate, with no loop, no end-condition check, and no actual execution of actions; coordinates support integer or one-decimal precision, with the origin at the top-left corner of the screenshot. V. Deployment and operations constraints: RGBA images should be converted to ("RGB") before encoding; the first multimodal request takes about 10~23s, with timeout set to ≥ 120; Ark has concurrency limits — for Grounding batch evaluation, a concurrency of 4–8 is recommended, --max-retries ≥ 3, and the batch evaluation script's --timeout defaults to 300s.

Capability Comparison

No verified item-by-item scores available; capability radar and overall ranking will not be displayed for now.

Data & Sources

Data Provider
火山方舟
On-site profile update
2026-11-22 02:01:23

On-site update time is not equivalent to the model release date. Official or platform benchmark scores are not displayed when test sets, dates, or original results are not provided.

Version History

doubao-seed-2.1-turboCurrent Entry Version

Historical versions, release and discontinuation dates to be verified.

Access & Usage

Please confirm the API address, model identifier, authentication, permissions, and billing rules according to the official documentation.

Open Official Documentation

Platform Tested

No verified model test reports available. Model capability tags do not indicate completed platform testing.
Product Selection Assistant

让需求变成可修改的筛选条件

依据已录入参数生成条件,结果可查看和修改。帧率与性能需核对实测,不能仅凭算力数值保证。