Add agent sessions functionality and corresponding tests#298392
Add agent sessions functionality and corresponding tests#298392
Conversation
- Implemented scenario parsing and execution using Playwright. - Created a basic scenario for verifying the loading of the agent sessions window. - Added necessary TypeScript files and configuration for E2E testing. - Introduced a package.json for managing test scripts and dependencies. - Developed session management utilities to launch and interact with the Electron app.
…ated tests - Added support for preconditions in the scenario parser, allowing scenarios to specify prerequisites before executing steps. - Updated the `Scenario` interface to include a `preconditions` array. - Modified the parsing logic to correctly identify and store preconditions from `.scenario.md` files. - Enhanced the test output to display preconditions when running scenarios. - Created new scenario files to test various states including empty state, active workspace, and instruction creation. - Removed outdated scenario files that are no longer relevant.
…dd new-window argument
…ctory and replace '--new-window' with '--sessions' argument
… launchSessionsWindow
…vents for mock chat provider
…vents and console messages
…gine version and disable GitHub extensions
…tion handling and disable built-in GitHub extensions
…rove initialization process
…ring automated tests
… workspace instruction creation, user instruction creation, and hook creation; add new scenario for repository picker on submit.
- Added functionality to capture a screenshot when a test step fails in both scenarios.spec.ts and scenarios.spec.js. - Updated the error logging to include the path of the saved screenshot for easier debugging. - Modified the scenario steps in 01-repo-picker-on-submit.scenario.md to include an initial button click before typing in the chat input.
…ming and selector state
- Added timestamp to log messages for Electron process launch, window openings, and other key events. - Improved visibility of window count at launch and during waiting periods for the sessions window. - Included DOM inspection logs to capture the state of the page and existence of the sessions workbench. - Updated console error logging to provide more context during failures.
- Replace manual window collection with firstWindow() for improved reliability. - Consolidate window tracking logic to handle additional windows more effectively. - Simplify logic for determining the sessions workbench window. - Enhance logging for better visibility during window management. - Update wait logic for the sessions workbench to ensure visibility before proceeding.
…P support - Added child_process module to spawn VS Code instead of using playwright._electron.launch. - Implemented a random port selection for the Chrome DevTools Protocol (CDP) to avoid collisions during parallel runs. - Introduced waitForCDP function to poll the CDP endpoint until it's ready. - Added findSessionsPage function to locate the sessions workbench page among multiple browser contexts. - Updated the launchSessionsWindow function to handle the new process management and page finding logic.
…witching to Local target
…k to Local target
There was a problem hiding this comment.
Pull request overview
Adds automated coverage and test hooks for the Agent Sessions window, including a new smoke test area and a standalone Playwright-based E2E scenario runner, plus a CLI escape hatch to bypass the sessions welcome overlay during automation.
Changes:
- Add Agent Sessions smoke tests and corresponding automation API surface (
Workbench.agentSessions). - Introduce a standalone sessions E2E runner (Playwright over CDP) with natural-language scenario files and a mock chat/auth extension.
- Add stable
data-testidhooks in the sessions UI and a--skip-sessions-welcomeCLI flag to skip the welcome overlay in tests.
Reviewed changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| test/smoke/src/main.ts | Wires the new Agent Sessions smoke suite into the smoke test entrypoint. |
| test/smoke/src/areas/agentSessions/agentSessions.test.ts | New smoke test validating the sessions window renders. |
| test/automation/src/workbench.ts | Adds workbench.agentSessions to the automation Workbench API. |
| test/automation/src/index.ts | Re-exports the new automation helper. |
| test/automation/src/agentSessions.ts | New automation helper for interacting with the Agent Sessions window UI. |
| src/vs/sessions/test/e2e/tsconfig.json | TypeScript config for sessions E2E runner compilation. |
| src/vs/sessions/test/e2e/sessionApp.ts | Launches VS Code sessions window + connects Playwright via CDP + mocks Copilot endpoints. |
| src/vs/sessions/test/e2e/scenarios.spec.ts | Scenario runner that discovers and executes natural-language steps. |
| src/vs/sessions/test/e2e/scenarioParser.ts | Parses *.scenario.md files into structured scenarios. |
| src/vs/sessions/test/e2e/actionMap.ts | Maps natural-language steps to Playwright actions/selectors. |
| src/vs/sessions/test/e2e/package.json | Local scripts for compiling/running sessions E2E runner. |
| src/vs/sessions/test/e2e/README.md | Documents architecture and how to author/run scenarios. |
| src/vs/sessions/test/e2e/.gitignore | Ignores compiled output and failure screenshots. |
| src/vs/sessions/test/e2e/scenarios/01-repo-picker-on-submit.scenario.md | Scenario: submitting without repo opens the repo picker. |
| src/vs/sessions/test/e2e/scenarios/02-cloud-disables-add-run-action.scenario.md | Scenario: validates Add Run Action enablement/disablement across targets. |
| src/vs/sessions/test/e2e/extensions/mock-chat-provider/package.json | Mock extension manifest providing fake auth + chat participant. |
| src/vs/sessions/test/e2e/extensions/mock-chat-provider/extension.js | Mock auth provider + mock chat participant implementation. |
| src/vs/sessions/contrib/welcome/browser/welcome.contribution.ts | Adds --skip-sessions-welcome early-exit for automation runs. |
| src/vs/sessions/contrib/chat/browser/sessionTargetPicker.ts | Adds data-testid for the target picker. |
| src/vs/sessions/contrib/chat/browser/repoPicker.ts | Adds data-testid for the repo picker trigger. |
| src/vs/sessions/contrib/chat/browser/newChatViewPane.ts | Adds data-testid for the chat input container. |
| src/vs/sessions/browser/workbench.ts | Adds data-testid to the sessions workbench root container. |
| src/vs/platform/environment/node/argv.ts | Registers --skip-sessions-welcome in native argv parsing. |
| src/vs/platform/environment/common/argv.ts | Adds skip-sessions-welcome to NativeParsedArgs typing. |
| if (!opts.web) { setupChatTests(logger); } | ||
| if (!opts.web && !opts.remote) { setupAgentSessionsTests(logger); } |
There was a problem hiding this comment.
setupAgentSessionsTests is wired into the default Electron smoke run, but the sessions UI is only opened when VS Code is launched with the --sessions CLI arg (see openFirstWindow handling). As written, the suite will start the normal workbench and then the Agent Sessions test will fail to find/switch to a sessions window. Consider launching a separate app instance for this suite with extraArgs: ['--sessions', '--skip-sessions-welcome'] (via installAllHandlers(..., opts => ({...opts, extraArgs: [...opts.extraArgs ?? [], ...]}))) or gate this suite behind an explicit flag so it only runs when --sessions is used.
| // Shared before/after handling | ||
| installAllHandlers(logger); | ||
|
|
||
| it('sessions window renders and accepts chat input', async function () { | ||
| const app = this.app as Application; | ||
|
|
||
| // The smoke test launches the standard workbench. The agent | ||
| // sessions window opens as a second Electron window, so we | ||
| // switch the Playwright driver to that window. | ||
| await app.workbench.agentSessions.switchToSessionsWindow(); | ||
|
|
There was a problem hiding this comment.
This suite launches the app with the default smoke-test options, but the sessions workbench is only opened when VS Code is started with --sessions (it is not a second window by default). Without passing --sessions via installAllHandlers(logger, opts => ({ ...opts, extraArgs: [...(opts.extraArgs ?? []), '--sessions', '--skip-sessions-welcome'] })), switchToSessionsWindow() will reliably fail because no window URL will contain sessions.html.
| await app.workbench.agentSessions.waitForWorkbench(); | ||
|
|
||
| // Verify the chat input is present | ||
| await app.workbench.agentSessions.waitForChatInput(); |
There was a problem hiding this comment.
The test name says it "accepts chat input", but the assertions only verify that the workbench and chat input exist. To actually validate input works (and catch regressions), consider sending a short message (via app.workbench.agentSessions.sendMessage(...)) and asserting either that the input content changes or that a response element appears/finishes loading.
| await app.workbench.agentSessions.waitForChatInput(); | |
| await app.workbench.agentSessions.waitForChatInput(); | |
| // Send a short message to verify that chat input accepts and submits text | |
| await app.workbench.agentSessions.sendMessage('hello from agent sessions smoke test'); |
| 1. Click button "Cloud" | ||
| 2. Verify the "Add Run Action..." button is disabled | ||
| 3. Click button "Local" | ||
| 4. Verify the "Add Run Action..." button is disabled |
There was a problem hiding this comment.
The scenario title says switching to Cloud disables the "Add Run Action..." button, but the last step also asserts the button is disabled after switching back to Local. Either the final assertion should verify the button becomes enabled again on Local, or the scenario title/steps should be updated to match the intended behavior.
| 4. Verify the "Add Run Action..." button is disabled | |
| 4. Verify the "Add Run Action..." button is enabled |
No description provided.