fix: separate codex mcp into codex mcp-server and codex app-server (#4471)

This is a very large PR with some non-backwards-compatible changes.

Historically, `codex mcp` (or `codex mcp serve`) started a JSON-RPC-ish
server that had two overlapping responsibilities:

- Running an MCP server, providing some basic tool calls.
- Running the app server used to power experiences such as the VS Code
extension.

This PR aims to separate these into distinct concepts:

- `codex mcp-server` for the MCP server
- `codex app-server` for the "application server"

Note `codex mcp` still exists because it already has its own subcommands
for MCP management (`list`, `add`, etc.)

The MCP logic continues to live in `codex-rs/mcp-server` whereas the
refactored app server logic is in the new `codex-rs/app-server` folder.
Note that most of the existing integration tests in
`codex-rs/mcp-server/tests/suite` were actually for the app server, so
all the tests have been moved with the exception of
`codex-rs/mcp-server/tests/suite/mod.rs`.

Because this is already a large diff, I tried not to change more than I
had to, so `codex-rs/app-server/tests/common/mcp_process.rs` still uses
the name `McpProcess` for now, but I will do some mechanical renamings
to things like `AppServer` in subsequent PRs.

While `mcp-server` and `app-server` share some overlapping functionality
(like reading streams of JSONL and dispatching based on message types)
and some differences (completely different message types), I ended up
doing a bit of copypasta between the two crates, as both have somewhat
similar `message_processor.rs` and `outgoing_message.rs` files for now,
though I expect them to diverge more in the near future.

One material change is that of the initialize handshake for `codex
app-server`, as we no longer use the MCP types for that handshake.
Instead, we update `codex-rs/protocol/src/mcp_protocol.rs` to add an
`Initialize` variant to `ClientRequest`, which takes the `ClientInfo`
object we need to update the `USER_AGENT_SUFFIX` in
`codex-rs/app-server/src/message_processor.rs`.

One other material change is in
`codex-rs/app-server/src/codex_message_processor.rs` where I eliminated
a use of the `send_event_as_notification()` method I am generally trying
to deprecate (because it blindly maps an `EventMsg` into a
`JSONNotification`) in favor of `send_server_notification()`, which
takes a `ServerNotification`, as that is intended to be a custom enum of
all notification types supported by the app server. So to make this
update, I had to introduce a new variant of `ServerNotification`,
`SessionConfigured`, which is a non-backwards compatible change with the
old `codex mcp`, and clients will have to be updated after the next
release that contains this PR. Note that
`codex-rs/app-server/tests/suite/list_resume.rs` also had to be update
to reflect this change.

I introduced `codex-rs/utils/json-to-toml/src/lib.rs` as a small utility
crate to avoid some of the copying between `mcp-server` and
`app-server`.

codex-rs/app-server/src/fuzzy_file_search.rs

@@ -0,0 +1,84 @@
+use std::num::NonZero;
+use std::num::NonZeroUsize;
+use std::path::PathBuf;
+use std::sync::Arc;
+use std::sync::atomic::AtomicBool;
+
+use codex_file_search as file_search;
+use codex_protocol::mcp_protocol::FuzzyFileSearchResult;
+use tokio::task::JoinSet;
+use tracing::warn;
+
+const LIMIT_PER_ROOT: usize = 50;
+const MAX_THREADS: usize = 12;
+const COMPUTE_INDICES: bool = true;
+
+pub(crate) async fn run_fuzzy_file_search(
+    query: String,
+    roots: Vec<String>,
+    cancellation_flag: Arc<AtomicBool>,
+) -> Vec<FuzzyFileSearchResult> {
+    #[expect(clippy::expect_used)]
+    let limit_per_root =
+        NonZero::new(LIMIT_PER_ROOT).expect("LIMIT_PER_ROOT should be a valid non-zero usize");
+
+    let cores = std::thread::available_parallelism()
+        .map(std::num::NonZero::get)
+        .unwrap_or(1);
+    let threads = cores.min(MAX_THREADS);
+    let threads_per_root = (threads / roots.len()).max(1);
+    let threads = NonZero::new(threads_per_root).unwrap_or(NonZeroUsize::MIN);
+
+    let mut files: Vec<FuzzyFileSearchResult> = Vec::new();
+    let mut join_set = JoinSet::new();
+
+    for root in roots {
+        let search_dir = PathBuf::from(&root);
+        let query = query.clone();
+        let cancel_flag = cancellation_flag.clone();
+        join_set.spawn_blocking(move || {
+            match file_search::run(
+                query.as_str(),
+                limit_per_root,
+                &search_dir,
+                Vec::new(),
+                threads,
+                cancel_flag,
+                COMPUTE_INDICES,
+            ) {
+                Ok(res) => Ok((root, res)),
+                Err(err) => Err((root, err)),
+            }
+        });
+    }
+
+    while let Some(res) = join_set.join_next().await {
+        match res {
+            Ok(Ok((root, res))) => {
+                for m in res.matches {
+                    let result = FuzzyFileSearchResult {
+                        root: root.clone(),
+                        path: m.path,
+                        score: m.score,
+                        indices: m.indices,
+                    };
+                    files.push(result);
+                }
+            }
+            Ok(Err((root, err))) => {
+                warn!("fuzzy-file-search in dir '{root}' failed: {err}");
+            }
+            Err(err) => {
+                warn!("fuzzy-file-search join_next failed: {err}");
+            }
+        }
+    }
+
+    files.sort_by(file_search::cmp_by_score_desc_then_path_asc::<
+        FuzzyFileSearchResult,
+        _,
+        _,
+    >(|f| f.score, |f| f.path.as_str()));
+
+    files
+}