chore: refactor tool handling (#4510)
# Tool System Refactor
- Centralizes tool definitions and execution in `core/src/tools/*`:
specs (`spec.rs`), handlers (`handlers/*`), router (`router.rs`),
registry/dispatch (`registry.rs`), and shared context (`context.rs`).
One registry now builds the model-visible tool list and binds handlers.
- Router converts model responses to tool calls; Registry dispatches
with consistent telemetry via `codex-rs/otel` and unified error
handling. Function, Local Shell, MCP, and experimental `unified_exec`
all flow through this path; legacy shell aliases still work.
- Rationale: reduce per‑tool boilerplate, keep spec/handler in sync, and
make adding tools predictable and testable.
Example: `read_file`
- Spec: `core/src/tools/spec.rs` (see `create_read_file_tool`,
registered by `build_specs`).
- Handler: `core/src/tools/handlers/read_file.rs` (absolute `file_path`,
1‑indexed `offset`, `limit`, `L#: ` prefixes, safe truncation).
- E2E test: `core/tests/suite/read_file.rs` validates the tool returns
the requested lines.
## Next steps:
- Decompose `handle_container_exec_with_params`
- Add parallel tool calls
2025-10-03 13:21:06 +01:00
|
|
|
use async_trait::async_trait;
|
|
|
|
|
use codex_protocol::models::ShellToolCallParams;
|
2025-10-05 17:10:49 +01:00
|
|
|
use std::sync::Arc;
|
chore: refactor tool handling (#4510)
# Tool System Refactor
- Centralizes tool definitions and execution in `core/src/tools/*`:
specs (`spec.rs`), handlers (`handlers/*`), router (`router.rs`),
registry/dispatch (`registry.rs`), and shared context (`context.rs`).
One registry now builds the model-visible tool list and binds handlers.
- Router converts model responses to tool calls; Registry dispatches
with consistent telemetry via `codex-rs/otel` and unified error
handling. Function, Local Shell, MCP, and experimental `unified_exec`
all flow through this path; legacy shell aliases still work.
- Rationale: reduce per‑tool boilerplate, keep spec/handler in sync, and
make adding tools predictable and testable.
Example: `read_file`
- Spec: `core/src/tools/spec.rs` (see `create_read_file_tool`,
registered by `build_specs`).
- Handler: `core/src/tools/handlers/read_file.rs` (absolute `file_path`,
1‑indexed `offset`, `limit`, `L#: ` prefixes, safe truncation).
- E2E test: `core/tests/suite/read_file.rs` validates the tool returns
the requested lines.
## Next steps:
- Decompose `handle_container_exec_with_params`
- Add parallel tool calls
2025-10-03 13:21:06 +01:00
|
|
|
|
2025-10-23 09:24:01 +01:00
|
|
|
use crate::apply_patch;
|
|
|
|
|
use crate::apply_patch::InternalApplyPatchInvocation;
|
|
|
|
|
use crate::apply_patch::convert_apply_patch_to_protocol;
|
chore: refactor tool handling (#4510)
# Tool System Refactor
- Centralizes tool definitions and execution in `core/src/tools/*`:
specs (`spec.rs`), handlers (`handlers/*`), router (`router.rs`),
registry/dispatch (`registry.rs`), and shared context (`context.rs`).
One registry now builds the model-visible tool list and binds handlers.
- Router converts model responses to tool calls; Registry dispatches
with consistent telemetry via `codex-rs/otel` and unified error
handling. Function, Local Shell, MCP, and experimental `unified_exec`
all flow through this path; legacy shell aliases still work.
- Rationale: reduce per‑tool boilerplate, keep spec/handler in sync, and
make adding tools predictable and testable.
Example: `read_file`
- Spec: `core/src/tools/spec.rs` (see `create_read_file_tool`,
registered by `build_specs`).
- Handler: `core/src/tools/handlers/read_file.rs` (absolute `file_path`,
1‑indexed `offset`, `limit`, `L#: ` prefixes, safe truncation).
- E2E test: `core/tests/suite/read_file.rs` validates the tool returns
the requested lines.
## Next steps:
- Decompose `handle_container_exec_with_params`
- Add parallel tool calls
2025-10-03 13:21:06 +01:00
|
|
|
use crate::codex::TurnContext;
|
|
|
|
|
use crate::exec::ExecParams;
|
|
|
|
|
use crate::exec_env::create_env;
|
|
|
|
|
use crate::function_tool::FunctionCallError;
|
|
|
|
|
use crate::tools::context::ToolInvocation;
|
|
|
|
|
use crate::tools::context::ToolOutput;
|
|
|
|
|
use crate::tools::context::ToolPayload;
|
2025-10-23 09:24:01 +01:00
|
|
|
use crate::tools::events::ToolEmitter;
|
|
|
|
|
use crate::tools::events::ToolEventCtx;
|
|
|
|
|
use crate::tools::orchestrator::ToolOrchestrator;
|
chore: refactor tool handling (#4510)
# Tool System Refactor
- Centralizes tool definitions and execution in `core/src/tools/*`:
specs (`spec.rs`), handlers (`handlers/*`), router (`router.rs`),
registry/dispatch (`registry.rs`), and shared context (`context.rs`).
One registry now builds the model-visible tool list and binds handlers.
- Router converts model responses to tool calls; Registry dispatches
with consistent telemetry via `codex-rs/otel` and unified error
handling. Function, Local Shell, MCP, and experimental `unified_exec`
all flow through this path; legacy shell aliases still work.
- Rationale: reduce per‑tool boilerplate, keep spec/handler in sync, and
make adding tools predictable and testable.
Example: `read_file`
- Spec: `core/src/tools/spec.rs` (see `create_read_file_tool`,
registered by `build_specs`).
- Handler: `core/src/tools/handlers/read_file.rs` (absolute `file_path`,
1‑indexed `offset`, `limit`, `L#: ` prefixes, safe truncation).
- E2E test: `core/tests/suite/read_file.rs` validates the tool returns
the requested lines.
## Next steps:
- Decompose `handle_container_exec_with_params`
- Add parallel tool calls
2025-10-03 13:21:06 +01:00
|
|
|
use crate::tools::registry::ToolHandler;
|
|
|
|
|
use crate::tools::registry::ToolKind;
|
2025-10-23 09:24:01 +01:00
|
|
|
use crate::tools::runtimes::apply_patch::ApplyPatchRequest;
|
|
|
|
|
use crate::tools::runtimes::apply_patch::ApplyPatchRuntime;
|
|
|
|
|
use crate::tools::runtimes::shell::ShellRequest;
|
|
|
|
|
use crate::tools::runtimes::shell::ShellRuntime;
|
|
|
|
|
use crate::tools::sandboxing::ToolCtx;
|
chore: refactor tool handling (#4510)
# Tool System Refactor
- Centralizes tool definitions and execution in `core/src/tools/*`:
specs (`spec.rs`), handlers (`handlers/*`), router (`router.rs`),
registry/dispatch (`registry.rs`), and shared context (`context.rs`).
One registry now builds the model-visible tool list and binds handlers.
- Router converts model responses to tool calls; Registry dispatches
with consistent telemetry via `codex-rs/otel` and unified error
handling. Function, Local Shell, MCP, and experimental `unified_exec`
all flow through this path; legacy shell aliases still work.
- Rationale: reduce per‑tool boilerplate, keep spec/handler in sync, and
make adding tools predictable and testable.
Example: `read_file`
- Spec: `core/src/tools/spec.rs` (see `create_read_file_tool`,
registered by `build_specs`).
- Handler: `core/src/tools/handlers/read_file.rs` (absolute `file_path`,
1‑indexed `offset`, `limit`, `L#: ` prefixes, safe truncation).
- E2E test: `core/tests/suite/read_file.rs` validates the tool returns
the requested lines.
## Next steps:
- Decompose `handle_container_exec_with_params`
- Add parallel tool calls
2025-10-03 13:21:06 +01:00
|
|
|
|
|
|
|
|
pub struct ShellHandler;
|
|
|
|
|
|
|
|
|
|
impl ShellHandler {
|
|
|
|
|
fn to_exec_params(params: ShellToolCallParams, turn_context: &TurnContext) -> ExecParams {
|
|
|
|
|
ExecParams {
|
|
|
|
|
command: params.command,
|
|
|
|
|
cwd: turn_context.resolve_path(params.workdir.clone()),
|
|
|
|
|
timeout_ms: params.timeout_ms,
|
|
|
|
|
env: create_env(&turn_context.shell_environment_policy),
|
|
|
|
|
with_escalated_permissions: params.with_escalated_permissions,
|
|
|
|
|
justification: params.justification,
|
2025-10-20 20:57:37 +01:00
|
|
|
arg0: None,
|
chore: refactor tool handling (#4510)
# Tool System Refactor
- Centralizes tool definitions and execution in `core/src/tools/*`:
specs (`spec.rs`), handlers (`handlers/*`), router (`router.rs`),
registry/dispatch (`registry.rs`), and shared context (`context.rs`).
One registry now builds the model-visible tool list and binds handlers.
- Router converts model responses to tool calls; Registry dispatches
with consistent telemetry via `codex-rs/otel` and unified error
handling. Function, Local Shell, MCP, and experimental `unified_exec`
all flow through this path; legacy shell aliases still work.
- Rationale: reduce per‑tool boilerplate, keep spec/handler in sync, and
make adding tools predictable and testable.
Example: `read_file`
- Spec: `core/src/tools/spec.rs` (see `create_read_file_tool`,
registered by `build_specs`).
- Handler: `core/src/tools/handlers/read_file.rs` (absolute `file_path`,
1‑indexed `offset`, `limit`, `L#: ` prefixes, safe truncation).
- E2E test: `core/tests/suite/read_file.rs` validates the tool returns
the requested lines.
## Next steps:
- Decompose `handle_container_exec_with_params`
- Add parallel tool calls
2025-10-03 13:21:06 +01:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#[async_trait]
|
|
|
|
|
impl ToolHandler for ShellHandler {
|
|
|
|
|
fn kind(&self) -> ToolKind {
|
|
|
|
|
ToolKind::Function
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn matches_kind(&self, payload: &ToolPayload) -> bool {
|
|
|
|
|
matches!(
|
|
|
|
|
payload,
|
|
|
|
|
ToolPayload::Function { .. } | ToolPayload::LocalShell { .. }
|
|
|
|
|
)
|
|
|
|
|
}
|
|
|
|
|
|
2025-10-05 17:10:49 +01:00
|
|
|
async fn handle(&self, invocation: ToolInvocation) -> Result<ToolOutput, FunctionCallError> {
|
chore: refactor tool handling (#4510)
# Tool System Refactor
- Centralizes tool definitions and execution in `core/src/tools/*`:
specs (`spec.rs`), handlers (`handlers/*`), router (`router.rs`),
registry/dispatch (`registry.rs`), and shared context (`context.rs`).
One registry now builds the model-visible tool list and binds handlers.
- Router converts model responses to tool calls; Registry dispatches
with consistent telemetry via `codex-rs/otel` and unified error
handling. Function, Local Shell, MCP, and experimental `unified_exec`
all flow through this path; legacy shell aliases still work.
- Rationale: reduce per‑tool boilerplate, keep spec/handler in sync, and
make adding tools predictable and testable.
Example: `read_file`
- Spec: `core/src/tools/spec.rs` (see `create_read_file_tool`,
registered by `build_specs`).
- Handler: `core/src/tools/handlers/read_file.rs` (absolute `file_path`,
1‑indexed `offset`, `limit`, `L#: ` prefixes, safe truncation).
- E2E test: `core/tests/suite/read_file.rs` validates the tool returns
the requested lines.
## Next steps:
- Decompose `handle_container_exec_with_params`
- Add parallel tool calls
2025-10-03 13:21:06 +01:00
|
|
|
let ToolInvocation {
|
|
|
|
|
session,
|
|
|
|
|
turn,
|
|
|
|
|
tracker,
|
|
|
|
|
call_id,
|
|
|
|
|
tool_name,
|
|
|
|
|
payload,
|
|
|
|
|
} = invocation;
|
|
|
|
|
|
|
|
|
|
match payload {
|
|
|
|
|
ToolPayload::Function { arguments } => {
|
|
|
|
|
let params: ShellToolCallParams =
|
|
|
|
|
serde_json::from_str(&arguments).map_err(|e| {
|
|
|
|
|
FunctionCallError::RespondToModel(format!(
|
|
|
|
|
"failed to parse function arguments: {e:?}"
|
|
|
|
|
))
|
|
|
|
|
})?;
|
2025-10-05 17:10:49 +01:00
|
|
|
let exec_params = Self::to_exec_params(params, turn.as_ref());
|
2025-10-23 09:24:01 +01:00
|
|
|
Self::run_exec_like(
|
chore: refactor tool handling (#4510)
# Tool System Refactor
- Centralizes tool definitions and execution in `core/src/tools/*`:
specs (`spec.rs`), handlers (`handlers/*`), router (`router.rs`),
registry/dispatch (`registry.rs`), and shared context (`context.rs`).
One registry now builds the model-visible tool list and binds handlers.
- Router converts model responses to tool calls; Registry dispatches
with consistent telemetry via `codex-rs/otel` and unified error
handling. Function, Local Shell, MCP, and experimental `unified_exec`
all flow through this path; legacy shell aliases still work.
- Rationale: reduce per‑tool boilerplate, keep spec/handler in sync, and
make adding tools predictable and testable.
Example: `read_file`
- Spec: `core/src/tools/spec.rs` (see `create_read_file_tool`,
registered by `build_specs`).
- Handler: `core/src/tools/handlers/read_file.rs` (absolute `file_path`,
1‑indexed `offset`, `limit`, `L#: ` prefixes, safe truncation).
- E2E test: `core/tests/suite/read_file.rs` validates the tool returns
the requested lines.
## Next steps:
- Decompose `handle_container_exec_with_params`
- Add parallel tool calls
2025-10-03 13:21:06 +01:00
|
|
|
tool_name.as_str(),
|
|
|
|
|
exec_params,
|
2025-10-23 09:24:01 +01:00
|
|
|
session,
|
|
|
|
|
turn,
|
|
|
|
|
tracker,
|
|
|
|
|
call_id,
|
chore: refactor tool handling (#4510)
# Tool System Refactor
- Centralizes tool definitions and execution in `core/src/tools/*`:
specs (`spec.rs`), handlers (`handlers/*`), router (`router.rs`),
registry/dispatch (`registry.rs`), and shared context (`context.rs`).
One registry now builds the model-visible tool list and binds handlers.
- Router converts model responses to tool calls; Registry dispatches
with consistent telemetry via `codex-rs/otel` and unified error
handling. Function, Local Shell, MCP, and experimental `unified_exec`
all flow through this path; legacy shell aliases still work.
- Rationale: reduce per‑tool boilerplate, keep spec/handler in sync, and
make adding tools predictable and testable.
Example: `read_file`
- Spec: `core/src/tools/spec.rs` (see `create_read_file_tool`,
registered by `build_specs`).
- Handler: `core/src/tools/handlers/read_file.rs` (absolute `file_path`,
1‑indexed `offset`, `limit`, `L#: ` prefixes, safe truncation).
- E2E test: `core/tests/suite/read_file.rs` validates the tool returns
the requested lines.
## Next steps:
- Decompose `handle_container_exec_with_params`
- Add parallel tool calls
2025-10-03 13:21:06 +01:00
|
|
|
)
|
2025-10-23 09:24:01 +01:00
|
|
|
.await
|
chore: refactor tool handling (#4510)
# Tool System Refactor
- Centralizes tool definitions and execution in `core/src/tools/*`:
specs (`spec.rs`), handlers (`handlers/*`), router (`router.rs`),
registry/dispatch (`registry.rs`), and shared context (`context.rs`).
One registry now builds the model-visible tool list and binds handlers.
- Router converts model responses to tool calls; Registry dispatches
with consistent telemetry via `codex-rs/otel` and unified error
handling. Function, Local Shell, MCP, and experimental `unified_exec`
all flow through this path; legacy shell aliases still work.
- Rationale: reduce per‑tool boilerplate, keep spec/handler in sync, and
make adding tools predictable and testable.
Example: `read_file`
- Spec: `core/src/tools/spec.rs` (see `create_read_file_tool`,
registered by `build_specs`).
- Handler: `core/src/tools/handlers/read_file.rs` (absolute `file_path`,
1‑indexed `offset`, `limit`, `L#: ` prefixes, safe truncation).
- E2E test: `core/tests/suite/read_file.rs` validates the tool returns
the requested lines.
## Next steps:
- Decompose `handle_container_exec_with_params`
- Add parallel tool calls
2025-10-03 13:21:06 +01:00
|
|
|
}
|
|
|
|
|
ToolPayload::LocalShell { params } => {
|
2025-10-05 17:10:49 +01:00
|
|
|
let exec_params = Self::to_exec_params(params, turn.as_ref());
|
2025-10-23 09:24:01 +01:00
|
|
|
Self::run_exec_like(
|
chore: refactor tool handling (#4510)
# Tool System Refactor
- Centralizes tool definitions and execution in `core/src/tools/*`:
specs (`spec.rs`), handlers (`handlers/*`), router (`router.rs`),
registry/dispatch (`registry.rs`), and shared context (`context.rs`).
One registry now builds the model-visible tool list and binds handlers.
- Router converts model responses to tool calls; Registry dispatches
with consistent telemetry via `codex-rs/otel` and unified error
handling. Function, Local Shell, MCP, and experimental `unified_exec`
all flow through this path; legacy shell aliases still work.
- Rationale: reduce per‑tool boilerplate, keep spec/handler in sync, and
make adding tools predictable and testable.
Example: `read_file`
- Spec: `core/src/tools/spec.rs` (see `create_read_file_tool`,
registered by `build_specs`).
- Handler: `core/src/tools/handlers/read_file.rs` (absolute `file_path`,
1‑indexed `offset`, `limit`, `L#: ` prefixes, safe truncation).
- E2E test: `core/tests/suite/read_file.rs` validates the tool returns
the requested lines.
## Next steps:
- Decompose `handle_container_exec_with_params`
- Add parallel tool calls
2025-10-03 13:21:06 +01:00
|
|
|
tool_name.as_str(),
|
|
|
|
|
exec_params,
|
2025-10-23 09:24:01 +01:00
|
|
|
session,
|
|
|
|
|
turn,
|
|
|
|
|
tracker,
|
|
|
|
|
call_id,
|
chore: refactor tool handling (#4510)
# Tool System Refactor
- Centralizes tool definitions and execution in `core/src/tools/*`:
specs (`spec.rs`), handlers (`handlers/*`), router (`router.rs`),
registry/dispatch (`registry.rs`), and shared context (`context.rs`).
One registry now builds the model-visible tool list and binds handlers.
- Router converts model responses to tool calls; Registry dispatches
with consistent telemetry via `codex-rs/otel` and unified error
handling. Function, Local Shell, MCP, and experimental `unified_exec`
all flow through this path; legacy shell aliases still work.
- Rationale: reduce per‑tool boilerplate, keep spec/handler in sync, and
make adding tools predictable and testable.
Example: `read_file`
- Spec: `core/src/tools/spec.rs` (see `create_read_file_tool`,
registered by `build_specs`).
- Handler: `core/src/tools/handlers/read_file.rs` (absolute `file_path`,
1‑indexed `offset`, `limit`, `L#: ` prefixes, safe truncation).
- E2E test: `core/tests/suite/read_file.rs` validates the tool returns
the requested lines.
## Next steps:
- Decompose `handle_container_exec_with_params`
- Add parallel tool calls
2025-10-03 13:21:06 +01:00
|
|
|
)
|
2025-10-23 09:24:01 +01:00
|
|
|
.await
|
chore: refactor tool handling (#4510)
# Tool System Refactor
- Centralizes tool definitions and execution in `core/src/tools/*`:
specs (`spec.rs`), handlers (`handlers/*`), router (`router.rs`),
registry/dispatch (`registry.rs`), and shared context (`context.rs`).
One registry now builds the model-visible tool list and binds handlers.
- Router converts model responses to tool calls; Registry dispatches
with consistent telemetry via `codex-rs/otel` and unified error
handling. Function, Local Shell, MCP, and experimental `unified_exec`
all flow through this path; legacy shell aliases still work.
- Rationale: reduce per‑tool boilerplate, keep spec/handler in sync, and
make adding tools predictable and testable.
Example: `read_file`
- Spec: `core/src/tools/spec.rs` (see `create_read_file_tool`,
registered by `build_specs`).
- Handler: `core/src/tools/handlers/read_file.rs` (absolute `file_path`,
1‑indexed `offset`, `limit`, `L#: ` prefixes, safe truncation).
- E2E test: `core/tests/suite/read_file.rs` validates the tool returns
the requested lines.
## Next steps:
- Decompose `handle_container_exec_with_params`
- Add parallel tool calls
2025-10-03 13:21:06 +01:00
|
|
|
}
|
|
|
|
|
_ => Err(FunctionCallError::RespondToModel(format!(
|
|
|
|
|
"unsupported payload for shell handler: {tool_name}"
|
|
|
|
|
))),
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
2025-10-23 09:24:01 +01:00
|
|
|
|
|
|
|
|
impl ShellHandler {
|
|
|
|
|
async fn run_exec_like(
|
|
|
|
|
tool_name: &str,
|
|
|
|
|
exec_params: ExecParams,
|
|
|
|
|
session: Arc<crate::codex::Session>,
|
|
|
|
|
turn: Arc<TurnContext>,
|
|
|
|
|
tracker: crate::tools::context::SharedTurnDiffTracker,
|
|
|
|
|
call_id: String,
|
|
|
|
|
) -> Result<ToolOutput, FunctionCallError> {
|
|
|
|
|
// Approval policy guard for explicit escalation in non-OnRequest modes.
|
|
|
|
|
if exec_params.with_escalated_permissions.unwrap_or(false)
|
|
|
|
|
&& !matches!(
|
|
|
|
|
turn.approval_policy,
|
|
|
|
|
codex_protocol::protocol::AskForApproval::OnRequest
|
|
|
|
|
)
|
|
|
|
|
{
|
|
|
|
|
return Err(FunctionCallError::RespondToModel(format!(
|
|
|
|
|
"approval policy is {policy:?}; reject command — you should not ask for escalated permissions if the approval policy is {policy:?}",
|
|
|
|
|
policy = turn.approval_policy
|
|
|
|
|
)));
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Intercept apply_patch if present.
|
|
|
|
|
match codex_apply_patch::maybe_parse_apply_patch_verified(
|
|
|
|
|
&exec_params.command,
|
|
|
|
|
&exec_params.cwd,
|
|
|
|
|
) {
|
|
|
|
|
codex_apply_patch::MaybeApplyPatchVerified::Body(changes) => {
|
|
|
|
|
match apply_patch::apply_patch(session.as_ref(), turn.as_ref(), &call_id, changes)
|
|
|
|
|
.await
|
|
|
|
|
{
|
|
|
|
|
InternalApplyPatchInvocation::Output(item) => {
|
|
|
|
|
// Programmatic apply_patch path; return its result.
|
|
|
|
|
let content = item?;
|
|
|
|
|
return Ok(ToolOutput::Function {
|
|
|
|
|
content,
|
[MCP] Render MCP tool call result images to the model (#5600)
It's pretty amazing we have gotten here without the ability for the
model to see image content from MCP tool calls.
This PR builds off of 4391 and fixes #4819. I would like @KKcorps to get
adequete credit here but I also want to get this fix in ASAP so I gave
him a week to update it and haven't gotten a response so I'm going to
take it across the finish line.
This test highlights how absured the current situation is. I asked the
model to read this image using the Chrome MCP
<img width="2378" height="674" alt="image"
src="https://github.com/user-attachments/assets/9ef52608-72a2-4423-9f5e-7ae36b2b56e0"
/>
After this change, it correctly outputs:
> Captured the page: image dhows a dark terminal-style UI labeled
`OpenAI Codex (v0.0.0)` with prompt `model: gpt-5-codex medium` and
working directory `/codex/codex-rs`
(and more)
Before this change, it said:
> Took the full-page screenshot you asked for. It shows a long,
horizontally repeating pattern of stylized people in orange, light-blue,
and mustard clothing, holding hands in alternating poses against a white
background. No text or other graphics-just rows of flat illustration
stretching off to the right.
Without this change, the Figma, Playwright, Chrome, and other visual MCP
servers are pretty much entirely useless.
I tested this change with the openai respones api as well as a third
party completions api
2025-10-27 14:55:57 -07:00
|
|
|
content_items: None,
|
2025-10-23 09:24:01 +01:00
|
|
|
success: Some(true),
|
|
|
|
|
});
|
|
|
|
|
}
|
|
|
|
|
InternalApplyPatchInvocation::DelegateToExec(apply) => {
|
|
|
|
|
let emitter = ToolEmitter::apply_patch(
|
|
|
|
|
convert_apply_patch_to_protocol(&apply.action),
|
|
|
|
|
!apply.user_explicitly_approved_this_action,
|
|
|
|
|
);
|
|
|
|
|
let event_ctx = ToolEventCtx::new(
|
|
|
|
|
session.as_ref(),
|
|
|
|
|
turn.as_ref(),
|
|
|
|
|
&call_id,
|
|
|
|
|
Some(&tracker),
|
|
|
|
|
);
|
|
|
|
|
emitter.begin(event_ctx).await;
|
|
|
|
|
|
|
|
|
|
let req = ApplyPatchRequest {
|
|
|
|
|
patch: apply.action.patch.clone(),
|
2025-10-23 17:00:48 +01:00
|
|
|
cwd: apply.action.cwd.clone(),
|
2025-10-23 09:24:01 +01:00
|
|
|
timeout_ms: exec_params.timeout_ms,
|
|
|
|
|
user_explicitly_approved: apply.user_explicitly_approved_this_action,
|
|
|
|
|
codex_exe: turn.codex_linux_sandbox_exe.clone(),
|
|
|
|
|
};
|
|
|
|
|
let mut orchestrator = ToolOrchestrator::new();
|
|
|
|
|
let mut runtime = ApplyPatchRuntime::new();
|
|
|
|
|
let tool_ctx = ToolCtx {
|
|
|
|
|
session: session.as_ref(),
|
|
|
|
|
turn: turn.as_ref(),
|
|
|
|
|
call_id: call_id.clone(),
|
|
|
|
|
tool_name: tool_name.to_string(),
|
|
|
|
|
};
|
|
|
|
|
let out = orchestrator
|
|
|
|
|
.run(&mut runtime, &req, &tool_ctx, &turn, turn.approval_policy)
|
|
|
|
|
.await;
|
|
|
|
|
let event_ctx = ToolEventCtx::new(
|
|
|
|
|
session.as_ref(),
|
|
|
|
|
turn.as_ref(),
|
|
|
|
|
&call_id,
|
|
|
|
|
Some(&tracker),
|
|
|
|
|
);
|
|
|
|
|
let content = emitter.finish(event_ctx, out).await?;
|
|
|
|
|
return Ok(ToolOutput::Function {
|
|
|
|
|
content,
|
[MCP] Render MCP tool call result images to the model (#5600)
It's pretty amazing we have gotten here without the ability for the
model to see image content from MCP tool calls.
This PR builds off of 4391 and fixes #4819. I would like @KKcorps to get
adequete credit here but I also want to get this fix in ASAP so I gave
him a week to update it and haven't gotten a response so I'm going to
take it across the finish line.
This test highlights how absured the current situation is. I asked the
model to read this image using the Chrome MCP
<img width="2378" height="674" alt="image"
src="https://github.com/user-attachments/assets/9ef52608-72a2-4423-9f5e-7ae36b2b56e0"
/>
After this change, it correctly outputs:
> Captured the page: image dhows a dark terminal-style UI labeled
`OpenAI Codex (v0.0.0)` with prompt `model: gpt-5-codex medium` and
working directory `/codex/codex-rs`
(and more)
Before this change, it said:
> Took the full-page screenshot you asked for. It shows a long,
horizontally repeating pattern of stylized people in orange, light-blue,
and mustard clothing, holding hands in alternating poses against a white
background. No text or other graphics-just rows of flat illustration
stretching off to the right.
Without this change, the Figma, Playwright, Chrome, and other visual MCP
servers are pretty much entirely useless.
I tested this change with the openai respones api as well as a third
party completions api
2025-10-27 14:55:57 -07:00
|
|
|
content_items: None,
|
2025-10-23 09:24:01 +01:00
|
|
|
success: Some(true),
|
|
|
|
|
});
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
codex_apply_patch::MaybeApplyPatchVerified::CorrectnessError(parse_error) => {
|
|
|
|
|
return Err(FunctionCallError::RespondToModel(format!(
|
|
|
|
|
"apply_patch verification failed: {parse_error}"
|
|
|
|
|
)));
|
|
|
|
|
}
|
|
|
|
|
codex_apply_patch::MaybeApplyPatchVerified::ShellParseError(error) => {
|
|
|
|
|
tracing::trace!("Failed to parse shell command, {error:?}");
|
|
|
|
|
// Fall through to regular shell execution.
|
|
|
|
|
}
|
|
|
|
|
codex_apply_patch::MaybeApplyPatchVerified::NotApplyPatch => {
|
|
|
|
|
// Fall through to regular shell execution.
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Regular shell execution path.
|
|
|
|
|
let emitter = ToolEmitter::shell(exec_params.command.clone(), exec_params.cwd.clone());
|
|
|
|
|
let event_ctx = ToolEventCtx::new(session.as_ref(), turn.as_ref(), &call_id, None);
|
|
|
|
|
emitter.begin(event_ctx).await;
|
|
|
|
|
|
|
|
|
|
let req = ShellRequest {
|
|
|
|
|
command: exec_params.command.clone(),
|
|
|
|
|
cwd: exec_params.cwd.clone(),
|
|
|
|
|
timeout_ms: exec_params.timeout_ms,
|
|
|
|
|
env: exec_params.env.clone(),
|
|
|
|
|
with_escalated_permissions: exec_params.with_escalated_permissions,
|
|
|
|
|
justification: exec_params.justification.clone(),
|
|
|
|
|
};
|
|
|
|
|
let mut orchestrator = ToolOrchestrator::new();
|
|
|
|
|
let mut runtime = ShellRuntime::new();
|
|
|
|
|
let tool_ctx = ToolCtx {
|
|
|
|
|
session: session.as_ref(),
|
|
|
|
|
turn: turn.as_ref(),
|
|
|
|
|
call_id: call_id.clone(),
|
|
|
|
|
tool_name: tool_name.to_string(),
|
|
|
|
|
};
|
|
|
|
|
let out = orchestrator
|
|
|
|
|
.run(&mut runtime, &req, &tool_ctx, &turn, turn.approval_policy)
|
|
|
|
|
.await;
|
|
|
|
|
let event_ctx = ToolEventCtx::new(session.as_ref(), turn.as_ref(), &call_id, None);
|
|
|
|
|
let content = emitter.finish(event_ctx, out).await?;
|
|
|
|
|
Ok(ToolOutput::Function {
|
|
|
|
|
content,
|
[MCP] Render MCP tool call result images to the model (#5600)
It's pretty amazing we have gotten here without the ability for the
model to see image content from MCP tool calls.
This PR builds off of 4391 and fixes #4819. I would like @KKcorps to get
adequete credit here but I also want to get this fix in ASAP so I gave
him a week to update it and haven't gotten a response so I'm going to
take it across the finish line.
This test highlights how absured the current situation is. I asked the
model to read this image using the Chrome MCP
<img width="2378" height="674" alt="image"
src="https://github.com/user-attachments/assets/9ef52608-72a2-4423-9f5e-7ae36b2b56e0"
/>
After this change, it correctly outputs:
> Captured the page: image dhows a dark terminal-style UI labeled
`OpenAI Codex (v0.0.0)` with prompt `model: gpt-5-codex medium` and
working directory `/codex/codex-rs`
(and more)
Before this change, it said:
> Took the full-page screenshot you asked for. It shows a long,
horizontally repeating pattern of stylized people in orange, light-blue,
and mustard clothing, holding hands in alternating poses against a white
background. No text or other graphics-just rows of flat illustration
stretching off to the right.
Without this change, the Figma, Playwright, Chrome, and other visual MCP
servers are pretty much entirely useless.
I tested this change with the openai respones api as well as a third
party completions api
2025-10-27 14:55:57 -07:00
|
|
|
content_items: None,
|
2025-10-23 09:24:01 +01:00
|
|
|
success: Some(true),
|
|
|
|
|
})
|
|
|
|
|
}
|
|
|
|
|
}
|