Skip to content

Add agent sessions functionality and corresponding tests#298392

Draft
osortega wants to merge 32 commits intomainfrom
copilot-worktree-2026-02-27T00-36-54
Draft

Add agent sessions functionality and corresponding tests#298392
osortega wants to merge 32 commits intomainfrom
copilot-worktree-2026-02-27T00-36-54

Conversation

@osortega
Copy link
Contributor

No description provided.

- Implemented scenario parsing and execution using Playwright.
- Created a basic scenario for verifying the loading of the agent sessions window.
- Added necessary TypeScript files and configuration for E2E testing.
- Introduced a package.json for managing test scripts and dependencies.
- Developed session management utilities to launch and interact with the Electron app.
…ated tests

- Added support for preconditions in the scenario parser, allowing scenarios to specify prerequisites before executing steps.
- Updated the `Scenario` interface to include a `preconditions` array.
- Modified the parsing logic to correctly identify and store preconditions from `.scenario.md` files.
- Enhanced the test output to display preconditions when running scenarios.
- Created new scenario files to test various states including empty state, active workspace, and instruction creation.
- Removed outdated scenario files that are no longer relevant.
…ctory and replace '--new-window' with '--sessions' argument
…tion handling and disable built-in GitHub extensions
… workspace instruction creation, user instruction creation, and hook creation; add new scenario for repository picker on submit.
- Added functionality to capture a screenshot when a test step fails in both scenarios.spec.ts and scenarios.spec.js.
- Updated the error logging to include the path of the saved screenshot for easier debugging.
- Modified the scenario steps in 01-repo-picker-on-submit.scenario.md to include an initial button click before typing in the chat input.
- Added timestamp to log messages for Electron process launch, window openings, and other key events.
- Improved visibility of window count at launch and during waiting periods for the sessions window.
- Included DOM inspection logs to capture the state of the page and existence of the sessions workbench.
- Updated console error logging to provide more context during failures.
- Replace manual window collection with firstWindow() for improved reliability.
- Consolidate window tracking logic to handle additional windows more effectively.
- Simplify logic for determining the sessions workbench window.
- Enhance logging for better visibility during window management.
- Update wait logic for the sessions workbench to ensure visibility before proceeding.
…P support

- Added child_process module to spawn VS Code instead of using playwright._electron.launch.
- Implemented a random port selection for the Chrome DevTools Protocol (CDP) to avoid collisions during parallel runs.
- Introduced waitForCDP function to poll the CDP endpoint until it's ready.
- Added findSessionsPage function to locate the sessions workbench page among multiple browser contexts.
- Updated the launchSessionsWindow function to handle the new process management and page finding logic.
Copilot AI review requested due to automatic review settings February 28, 2026 00:53
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds automated coverage and test hooks for the Agent Sessions window, including a new smoke test area and a standalone Playwright-based E2E scenario runner, plus a CLI escape hatch to bypass the sessions welcome overlay during automation.

Changes:

  • Add Agent Sessions smoke tests and corresponding automation API surface (Workbench.agentSessions).
  • Introduce a standalone sessions E2E runner (Playwright over CDP) with natural-language scenario files and a mock chat/auth extension.
  • Add stable data-testid hooks in the sessions UI and a --skip-sessions-welcome CLI flag to skip the welcome overlay in tests.

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
test/smoke/src/main.ts Wires the new Agent Sessions smoke suite into the smoke test entrypoint.
test/smoke/src/areas/agentSessions/agentSessions.test.ts New smoke test validating the sessions window renders.
test/automation/src/workbench.ts Adds workbench.agentSessions to the automation Workbench API.
test/automation/src/index.ts Re-exports the new automation helper.
test/automation/src/agentSessions.ts New automation helper for interacting with the Agent Sessions window UI.
src/vs/sessions/test/e2e/tsconfig.json TypeScript config for sessions E2E runner compilation.
src/vs/sessions/test/e2e/sessionApp.ts Launches VS Code sessions window + connects Playwright via CDP + mocks Copilot endpoints.
src/vs/sessions/test/e2e/scenarios.spec.ts Scenario runner that discovers and executes natural-language steps.
src/vs/sessions/test/e2e/scenarioParser.ts Parses *.scenario.md files into structured scenarios.
src/vs/sessions/test/e2e/actionMap.ts Maps natural-language steps to Playwright actions/selectors.
src/vs/sessions/test/e2e/package.json Local scripts for compiling/running sessions E2E runner.
src/vs/sessions/test/e2e/README.md Documents architecture and how to author/run scenarios.
src/vs/sessions/test/e2e/.gitignore Ignores compiled output and failure screenshots.
src/vs/sessions/test/e2e/scenarios/01-repo-picker-on-submit.scenario.md Scenario: submitting without repo opens the repo picker.
src/vs/sessions/test/e2e/scenarios/02-cloud-disables-add-run-action.scenario.md Scenario: validates Add Run Action enablement/disablement across targets.
src/vs/sessions/test/e2e/extensions/mock-chat-provider/package.json Mock extension manifest providing fake auth + chat participant.
src/vs/sessions/test/e2e/extensions/mock-chat-provider/extension.js Mock auth provider + mock chat participant implementation.
src/vs/sessions/contrib/welcome/browser/welcome.contribution.ts Adds --skip-sessions-welcome early-exit for automation runs.
src/vs/sessions/contrib/chat/browser/sessionTargetPicker.ts Adds data-testid for the target picker.
src/vs/sessions/contrib/chat/browser/repoPicker.ts Adds data-testid for the repo picker trigger.
src/vs/sessions/contrib/chat/browser/newChatViewPane.ts Adds data-testid for the chat input container.
src/vs/sessions/browser/workbench.ts Adds data-testid to the sessions workbench root container.
src/vs/platform/environment/node/argv.ts Registers --skip-sessions-welcome in native argv parsing.
src/vs/platform/environment/common/argv.ts Adds skip-sessions-welcome to NativeParsedArgs typing.

Comment on lines 421 to +422
if (!opts.web) { setupChatTests(logger); }
if (!opts.web && !opts.remote) { setupAgentSessionsTests(logger); }
Copy link

Copilot AI Feb 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

setupAgentSessionsTests is wired into the default Electron smoke run, but the sessions UI is only opened when VS Code is launched with the --sessions CLI arg (see openFirstWindow handling). As written, the suite will start the normal workbench and then the Agent Sessions test will fail to find/switch to a sessions window. Consider launching a separate app instance for this suite with extraArgs: ['--sessions', '--skip-sessions-welcome'] (via installAllHandlers(..., opts => ({...opts, extraArgs: [...opts.extraArgs ?? [], ...]}))) or gate this suite behind an explicit flag so it only runs when --sessions is used.

Copilot uses AI. Check for mistakes.
Comment on lines +15 to +25
// Shared before/after handling
installAllHandlers(logger);

it('sessions window renders and accepts chat input', async function () {
const app = this.app as Application;

// The smoke test launches the standard workbench. The agent
// sessions window opens as a second Electron window, so we
// switch the Playwright driver to that window.
await app.workbench.agentSessions.switchToSessionsWindow();

Copy link

Copilot AI Feb 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This suite launches the app with the default smoke-test options, but the sessions workbench is only opened when VS Code is started with --sessions (it is not a second window by default). Without passing --sessions via installAllHandlers(logger, opts => ({ ...opts, extraArgs: [...(opts.extraArgs ?? []), '--sessions', '--skip-sessions-welcome'] })), switchToSessionsWindow() will reliably fail because no window URL will contain sessions.html.

Copilot uses AI. Check for mistakes.
await app.workbench.agentSessions.waitForWorkbench();

// Verify the chat input is present
await app.workbench.agentSessions.waitForChatInput();
Copy link

Copilot AI Feb 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test name says it "accepts chat input", but the assertions only verify that the workbench and chat input exist. To actually validate input works (and catch regressions), consider sending a short message (via app.workbench.agentSessions.sendMessage(...)) and asserting either that the input content changes or that a response element appears/finishes loading.

Suggested change
await app.workbench.agentSessions.waitForChatInput();
await app.workbench.agentSessions.waitForChatInput();
// Send a short message to verify that chat input accepts and submits text
await app.workbench.agentSessions.sendMessage('hello from agent sessions smoke test');

Copilot uses AI. Check for mistakes.
1. Click button "Cloud"
2. Verify the "Add Run Action..." button is disabled
3. Click button "Local"
4. Verify the "Add Run Action..." button is disabled
Copy link

Copilot AI Feb 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The scenario title says switching to Cloud disables the "Add Run Action..." button, but the last step also asserts the button is disabled after switching back to Local. Either the final assertion should verify the button becomes enabled again on Local, or the scenario title/steps should be updated to match the intended behavior.

Suggested change
4. Verify the "Add Run Action..." button is disabled
4. Verify the "Add Run Action..." button is enabled

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants