valknar/llmx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: @mention files in codex (#701)

Browse Source 
Solves #700

## State of the World Before

Prior to this PR, when users wanted to share file contents with Codex,
they had two options:
- Manually copy and paste file contents into the chat
- Wait for the assistant to use the shell tool to view the file

The second approach required the assistant to:
1. Recognize the need to view a file
2. Execute a shell tool call
3. Wait for the tool call to complete
4. Process the file contents

This consumed extra tokens and reduced user control over which files
were shared with the model.

## State of the World After

With this PR, users can now:
- Reference files directly in their chat input using the `@path` syntax
- Have file contents automatically expanded into XML blocks before being
sent to the LLM

For example, users can type `@src/utils/config.js` in their message, and
the file contents will be included in context. Within the terminal chat
history, these file blocks will be collapsed back to `@path` format in
the UI for clean presentation.

Tag File suggestions:
<img width="857" alt="file-suggestions"
src="https://github.com/user-attachments/assets/397669dc-ad83-492d-b5f0-164fab2ff4ba"
/>

Tagging files in action:
<img width="858" alt="tagging-files"
src="https://github.com/user-attachments/assets/0de9d559-7b7f-4916-aeff-87ae9b16550a"
/>

Demo video of file tagging:
[![Demo video of file
tagging](https://img.youtube.com/vi/vL4LqtBnqt8/0.jpg)](https://www.youtube.com/watch?v=vL4LqtBnqt8)

## Implementation Details

This PR consists of 2 main components:

1. **File Tag Utilities**:
- New `file-tag-utils.ts` utility module that handles both expansion and
collapsing of file tags
- `expandFileTags()` identifies `@path` tokens and replaces them with
XML blocks containing file contents
- `collapseXmlBlocks()` reverses the process, converting XML blocks back
to `@path` format for UI display
- Tokens are only expanded if they point to valid files (directories are
ignored)
   - Expansion happens just before sending input to the model

2. **Terminal Chat Integration**:
- Leveraged the existing file system completion system for tabbing to
support the `@path` syntax
   - Added `updateFsSuggestions` helper to manage filesystem suggestions
- Added `replaceFileSystemSuggestion` to replace input with filesystem
suggestions
- Applied `collapseXmlBlocks` in the chat response rendering so that
tagged files are shown as simple `@path` tags

The PR also includes test coverage for both the UI and the file tag
utilities.

## Next Steps

Some ideas I'd like to implement if this feature gets merged:

- Line selection: `@path[50:80]` to grab specific sections of files
- Method selection: `@path#methodName` to grab just one function/class
- Visual improvements: highlight file tags in the UI to make them more
noticeable
This commit is contained in:
moppywhip
2025-04-30 19:19:55 -04:00
committed by GitHub
parent bd82101859
commit bc4e6db749
10 changed files with 771 additions and 62 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
codex-cli/src/components/chat/multiline-editor.tsx
View File

@@ -137,6 +137,9 @@ export interface MultilineTextEditorProps {
// Called when the internal text buffer updates.
readonly onChange?: (text: string) => void;
// Optional initial cursor position (character offset)
readonly initialCursorOffset?: number;
}
// Expose a minimal imperative API so parent components (e.g. TerminalChatInput)
@@ -169,6 +172,7 @@ const MultilineTextEditorInner = (
onSubmit,
focus = true,
onChange,
initialCursorOffset,
}: MultilineTextEditorProps,
ref: React.Ref<MultilineTextEditorHandle | null>,
): React.ReactElement => {
@@ -176,7 +180,7 @@ const MultilineTextEditorInner = (
// Editor State
// ---------------------------------------------------------------------------
const buffer = useRef(new TextBuffer(initialText));
const buffer = useRef(new TextBuffer(initialText, initialCursorOffset));
const [version, setVersion] = useState(0);
// Keep track of the current terminal size so that the editor grows/shrinks

195
codex-cli/src/components/chat/terminal-chat-input.tsx
View File

@@ -1,5 +1,6 @@
import type { MultilineTextEditorHandle } from "./multiline-editor";
import type { ReviewDecision } from "../../utils/agent/review.js";
import type { FileSystemSuggestion } from "../../utils/file-system-suggestions.js";
import type { HistoryEntry } from "../../utils/storage/command-history.js";
import type {
ResponseInputItem,
@@ -11,6 +12,7 @@ import { TerminalChatCommandReview } from "./terminal-chat-command-review.js";
import TextCompletions from "./terminal-chat-completions.js";
import { loadConfig } from "../../utils/config.js";
import { getFileSystemSuggestions } from "../../utils/file-system-suggestions.js";
import { expandFileTags } from "../../utils/file-tag-utils";
import { createInputItem } from "../../utils/input-utils.js";
import { log } from "../../utils/logger/log.js";
import { setSessionId } from "../../utils/session.js";
@@ -92,16 +94,120 @@ export default function TerminalChatInput({
const [historyIndex, setHistoryIndex] = useState<number | null>(null);
const [draftInput, setDraftInput] = useState<string>("");
const [skipNextSubmit, setSkipNextSubmit] = useState<boolean>(false);
const [fsSuggestions, setFsSuggestions] = useState<Array<string>>([]);
const [fsSuggestions, setFsSuggestions] = useState<
Array<FileSystemSuggestion>
>([]);
const [selectedCompletion, setSelectedCompletion] = useState<number>(-1);
// Multiline text editor key to force remount after submission
const [editorKey, setEditorKey] = useState(0);
const [editorState, setEditorState] = useState<{
key: number;
initialCursorOffset?: number;
}>({ key: 0 });
// Imperative handle from the multiline editor so we can query caret position
const editorRef = useRef<MultilineTextEditorHandle | null>(null);
// Track the caret row across keystrokes
const prevCursorRow = useRef<number | null>(null);
const prevCursorWasAtLastRow = useRef<boolean>(false);
// --- Helper for updating input, remounting editor, and moving cursor to end ---
const applyFsSuggestion = useCallback((newInputText: string) => {
setInput(newInputText);
setEditorState((s) => ({
key: s.key + 1,
initialCursorOffset: newInputText.length,
}));
}, []);
// --- Helper for updating file system suggestions ---
function updateFsSuggestions(
txt: string,
alwaysUpdateSelection: boolean = false,
) {
// Clear file system completions if a space is typed
if (txt.endsWith(" ")) {
setFsSuggestions([]);
setSelectedCompletion(-1);
} else {
// Determine the current token (last whitespace-separated word)
const words = txt.trim().split(/\s+/);
const lastWord = words[words.length - 1] ?? "";
const shouldUpdateSelection =
lastWord.startsWith("@") || alwaysUpdateSelection;
// Strip optional leading '@' for the path prefix
let pathPrefix: string;
if (lastWord.startsWith("@")) {
pathPrefix = lastWord.slice(1);
// If only '@' is typed, list everything in the current directory
pathPrefix = pathPrefix.length === 0 ? "./" : pathPrefix;
} else {
pathPrefix = lastWord;
}
if (shouldUpdateSelection) {
const completions = getFileSystemSuggestions(pathPrefix);
setFsSuggestions(completions);
if (completions.length > 0) {
setSelectedCompletion((prev) =>
prev < 0 || prev >= completions.length ? 0 : prev,
);
} else {
setSelectedCompletion(-1);
}
} else if (fsSuggestions.length > 0) {
// Token cleared → clear menu
setFsSuggestions([]);
setSelectedCompletion(-1);
}
}
}
/**
* Result of replacing text with a file system suggestion
*/
interface ReplacementResult {
/** The new text with the suggestion applied */
text: string;
/** The selected suggestion if a replacement was made */
suggestion: FileSystemSuggestion | null;
/** Whether a replacement was actually made */
wasReplaced: boolean;
}
// --- Helper for replacing input with file system suggestion ---
function getFileSystemSuggestion(
txt: string,
requireAtPrefix: boolean = false,
): ReplacementResult {
if (fsSuggestions.length === 0 || selectedCompletion < 0) {
return { text: txt, suggestion: null, wasReplaced: false };
}
const words = txt.trim().split(/\s+/);
const lastWord = words[words.length - 1] ?? "";
// Check if @ prefix is required and the last word doesn't have it
if (requireAtPrefix && !lastWord.startsWith("@")) {
return { text: txt, suggestion: null, wasReplaced: false };
}
const selected = fsSuggestions[selectedCompletion];
if (!selected) {
return { text: txt, suggestion: null, wasReplaced: false };
}
const replacement = lastWord.startsWith("@")
? `@${selected.path}`
: selected.path;
words[words.length - 1] = replacement;
return {
text: words.join(" "),
suggestion: selected,
wasReplaced: true,
};
}
// Load command history on component mount
useEffect(() => {
async function loadHistory() {
@@ -223,21 +329,12 @@ export default function TerminalChatInput({
}
if (_key.tab && selectedCompletion >= 0) {
const words = input.trim().split(/\s+/);
const selected = fsSuggestions[selectedCompletion];
if (words.length > 0 && selected) {
words[words.length - 1] = selected;
const newText = words.join(" ");
setInput(newText);
// Force remount of the editor with the new text
setEditorKey((k) => k + 1);
// We need to move the cursor to the end after editor remounts
setTimeout(() => {
editorRef.current?.moveCursorToEnd?.();
}, 0);
const { text: newText, wasReplaced } =
getFileSystemSuggestion(input);
// Only proceed if the text was actually changed
if (wasReplaced) {
applyFsSuggestion(newText);
setFsSuggestions([]);
setSelectedCompletion(-1);
}
@@ -277,7 +374,7 @@ export default function TerminalChatInput({
setInput(history[newIndex]?.command ?? "");
// Re-mount the editor so it picks up the new initialText
setEditorKey((k) => k + 1);
setEditorState((s) => ({ key: s.key + 1 }));
return; // handled
}
@@ -296,28 +393,23 @@ export default function TerminalChatInput({
if (newIndex >= history.length) {
setHistoryIndex(null);
setInput(draftInput);
setEditorKey((k) => k + 1);
setEditorState((s) => ({ key: s.key + 1 }));
} else {
setHistoryIndex(newIndex);
setInput(history[newIndex]?.command ?? "");
setEditorKey((k) => k + 1);
setEditorState((s) => ({ key: s.key + 1 }));
}
return; // handled
}
// Otherwise let it propagate
}
if (_key.tab) {
const words = input.split(/\s+/);
const mostRecentWord = words[words.length - 1];
if (mostRecentWord === undefined || mostRecentWord === "") {
return;
}
const completions = getFileSystemSuggestions(mostRecentWord);
setFsSuggestions(completions);
if (completions.length > 0) {
setSelectedCompletion(0);
}
// Defer filesystem suggestion logic to onSubmit if enter key is pressed
if (!_key.return) {
// Pressing tab should trigger the file system suggestions
const shouldUpdateSelection = _key.tab;
const targetInput = _key.delete ? input.slice(0, -1) : input + _input;
updateFsSuggestions(targetInput, shouldUpdateSelection);
}
}
@@ -599,7 +691,10 @@ export default function TerminalChatInput({
);
text = text.trim();
const inputItem = await createInputItem(text, images);
// Expand @file tokens into XML blocks for the model
const expandedText = await expandFileTags(text);
const inputItem = await createInputItem(expandedText, images);
submitInput([inputItem]);
// Get config for history persistence.
@@ -673,28 +768,30 @@ export default function TerminalChatInput({
setHistoryIndex(null);
}
setInput(txt);
// Clear tab completions if a space is typed
if (txt.endsWith(" ")) {
setFsSuggestions([]);
setSelectedCompletion(-1);
} else if (fsSuggestions.length > 0) {
// Update file suggestions as user types
const words = txt.trim().split(/\s+/);
const mostRecentWord =
words.length > 0 ? words[words.length - 1] : "";
if (mostRecentWord !== undefined) {
setFsSuggestions(getFileSystemSuggestions(mostRecentWord));
}
}
}}
key={editorKey}
key={editorState.key}
initialCursorOffset={editorState.initialCursorOffset}
initialText={input}
height={6}
focus={active}
onSubmit={(txt) => {
onSubmit(txt);
setEditorKey((k) => k + 1);
// If final token is an @path, replace with filesystem suggestion if available
const {
text: replacedText,
suggestion,
wasReplaced,
} = getFileSystemSuggestion(txt, true);
// If we replaced @path token with a directory, don't submit
if (wasReplaced && suggestion?.isDirectory) {
applyFsSuggestion(replacedText);
// Update suggestions for the new directory
updateFsSuggestions(replacedText, true);
return;
}
onSubmit(replacedText);
setEditorState((s) => ({ key: s.key + 1 }));
setInput("");
setHistoryIndex(null);
setDraftInput("");
@@ -741,7 +838,7 @@ export default function TerminalChatInput({
</Text>
) : fsSuggestions.length > 0 ? (
<TextCompletions
completions={fsSuggestions}
completions={fsSuggestions.map((suggestion) => suggestion.path)}
selectedCompletion={selectedCompletion}
displayLimit={5}
/>

3
codex-cli/src/components/chat/terminal-chat-response-item.tsx
View File

@@ -10,6 +10,7 @@ import type {
} from "openai/resources/responses/responses";
import { useTerminalSize } from "../../hooks/use-terminal-size";
import { collapseXmlBlocks } from "../../utils/file-tag-utils";
import { parseToolCall, parseToolCallOutput } from "../../utils/parsers";
import chalk, { type ForegroundColorName } from "chalk";
import { Box, Text } from "ink";
@@ -137,7 +138,7 @@ function TerminalChatResponseMessage({
: c.type === "refusal"
? c.refusal
: c.type === "input_text"
? c.text
? collapseXmlBlocks(c.text)
: c.type === "input_image"
? "<Image>"
: c.type === "input_file"

38
codex-cli/src/text-buffer.ts
View File

@@ -100,11 +100,14 @@ export default class TextBuffer {
private clipboard: string | null = null;
constructor(text = "") {
constructor(text = "", initialCursorIdx = 0) {
this.lines = text.split("\n");
if (this.lines.length === 0) {
this.lines = [""];
}
// No need to reset cursor on failure - class already default cursor position to 0,0
this.setCursorIdx(initialCursorIdx);
}
/* =======================================================================
@@ -122,6 +125,39 @@ export default class TextBuffer {
this.cursorCol = clamp(this.cursorCol, 0, this.lineLen(this.cursorRow));
}
/**
* Sets the cursor position based on a character offset from the start of the document.
* @param idx The character offset to move to (0-based)
* @returns true if successful, false if the index was invalid
*/
private setCursorIdx(idx: number): boolean {
// Reset preferred column since this is an explicit horizontal movement
this.preferredCol = null;
let remainingChars = idx;
let row = 0;
// Count characters line by line until we find the right position
while (row < this.lines.length) {
const lineLength = this.lineLen(row);
// Add 1 for the newline character (except for the last line)
const totalChars = lineLength + (row < this.lines.length - 1 ? 1 : 0);
if (remainingChars <= lineLength) {
this.cursorRow = row;
this.cursorCol = remainingChars;
return true;
}
// Move to next line, subtract this line's characters plus newline
remainingChars -= totalChars;
row++;
}
// If we get here, the index was too large
return false;
}
/* =====================================================================
* History helpers
* =================================================================== */

27
codex-cli/src/utils/file-system-suggestions.ts
View File

@@ -2,7 +2,24 @@ import fs from "fs";
import os from "os";
import path from "path";
export function getFileSystemSuggestions(pathPrefix: string): Array<string> {
/**
* Represents a file system suggestion with path and directory information
*/
export interface FileSystemSuggestion {
/** The full path of the suggestion */
path: string;
/** Whether the suggestion is a directory */
isDirectory: boolean;
}
/**
* Gets file system suggestions based on a path prefix
* @param pathPrefix The path prefix to search for
* @returns Array of file system suggestions
*/
export function getFileSystemSuggestions(
pathPrefix: string,
): Array<FileSystemSuggestion> {
if (!pathPrefix) {
return [];
}
@@ -31,10 +48,10 @@ export function getFileSystemSuggestions(pathPrefix: string): Array<string> {
.map((item) => {
const fullPath = path.join(readDir, item);
const isDirectory = fs.statSync(fullPath).isDirectory();
if (isDirectory) {
return path.join(fullPath, sep);
}
return fullPath;
return {
path: isDirectory ? path.join(fullPath, sep) : fullPath,
isDirectory,
};
});
} catch {
return [];

62
codex-cli/src/utils/file-tag-utils.ts Normal file
View File

@@ -0,0 +1,62 @@
import fs from "fs";
import path from "path";
/**
* Replaces @path tokens in the input string with <path>file contents</path> XML blocks for LLM context.
* Only replaces if the path points to a file; directories are ignored.
*/
export async function expandFileTags(raw: string): Promise<string> {
const re = /@([\w./~-]+)/g;
let out = raw;
type MatchInfo = { index: number; length: number; path: string };
const matches: Array<MatchInfo> = [];
for (const m of raw.matchAll(re) as IterableIterator<RegExpMatchArray>) {
const idx = m.index;
const captured = m[1];
if (idx !== undefined && captured) {
matches.push({ index: idx, length: m[0].length, path: captured });
}
}
// Process in reverse to avoid index shifting.
for (let i = matches.length - 1; i >= 0; i--) {
const { index, length, path: p } = matches[i]!;
const resolved = path.resolve(process.cwd(), p);
try {
const st = fs.statSync(resolved);
if (st.isFile()) {
const content = fs.readFileSync(resolved, "utf-8");
const rel = path.relative(process.cwd(), resolved);
const xml = `<${rel}>\n${content}\n</${rel}>`;
out = out.slice(0, index) + xml + out.slice(index + length);
}
} catch {
// If path invalid, leave token as is
}
}
return out;
}
/**
* Collapses <path>content</path> XML blocks back to @path format.
* This is the reverse operation of expandFileTags.
* Only collapses blocks where the path points to a valid file; invalid paths remain unchanged.
*/
export function collapseXmlBlocks(text: string): string {
return text.replace(
/<([^\n>]+)>([\s\S]*?)<\/\1>/g,
(match, path1: string) => {
const filePath = path.normalize(path1.trim());
try {
// Only convert to @path format if it's a valid file
return fs.statSync(path.resolve(process.cwd(), filePath)).isFile()
? "@" + filePath
: match;
} catch {
return match; // Keep XML block if path is invalid
}
},
);
}