Skip to content

test-companion: fixes from running every job as the GitHub App - #96

Merged
francisf merged 31 commits into
browserstack:masterfrom
naimish-browse:test-companion-pilot-fixes
Nov 16, 2026
Merged

francisf merged 31 commits into
browserstack:masterfrom
naimish-browse:test-companion-pilot-fixes

Conversation

@naimish-browse

@naimish-browse naimish-browse commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Fixes from running every Test Companion job as the BrowserStack GitHub App on a pilot repository.

The App and its own comments

  • Comments written by bot accounts, the App's own included, are ignored before anything else. The starter workflow's filter skips them too, so no runner starts.
  • The App's commits count as ours even when a later run falls back to the workflow token, so the loop guard still holds.
  • Signing in as the App retries a network error, a 5xx or a 429 (up to three tries) before falling back to the workflow token, so one network blip doesn't cost a reply its BrowserStack identity.

What gets committed

  • Jobs that don't produce code (generate, rca, verdict, validate) never commit to the branch, including when the pipeline file picks the job for an event.
  • Tool by-products (browserstack.err, .browserstack*, logs, .env, report folders) are never committed. A handed-back fix or automate branch is always pushed as one squashed commit on the base, under our name, so the agent's own commits never reach GitHub; by-products are left out of it. Only .github/ files and symlinks still refuse a branch.
  • git reads every path literally, so an agent-chosen file name like :!x can't widen git add or reset to other files.
  • Commits made as the App are authored as its bot account, so they show its name and logo.

Pull requests handed back

  • A handed-back fix is squashed onto the commit the run started on, and only if the agent's branch builds on it. A branch that rewrote history (for example git reset past that commit) is refused, so a fix can never hand back a diff that undoes the pull request's own work.
  • A fix asked for on a pull request opens against that pull request's branch, as a draft, titled after the job. It falls back to the default branch if the run didn't start on that pull request's commit, and says so.
  • Repositories without draft pull requests get a normal one.
  • If an earlier run already pushed the branch: an open pull request is reported as done, a closed or merged one is reported and never reopened. A push rejected by a protection rule or permission is reported as such, never as "pushed".

Runs that break

  • A broken run no longer offers go, and its check completes.
  • The comment says why, using the CLI's own error first. Credentials (tokens, access keys, credentials in URLs) are hidden in everything sent to GitHub: comments, replies, checks and pull requests.
  • Lists of file names in comments are capped and safely quoted.

Replies and approvals

  • 👀 on a request once it's picked up.
  • Each request (an @browserstack comment, or a go) gets its own reply with that run's report. Results no longer overwrite each other in one shared comment. Automatic runs, which answer no request, keep one status comment.
  • Replying go approves the newest plan waiting. One approval settles every plan of that job on that commit that existed at the time, so neither a second go nor the nightly sweep acts on them; a plan asked for afterwards stays approvable. go #<id> on a settled plan says whether it was approved, dropped, or replaced by another approved plan.
  • A run a person asked for or approved can save on top of an earlier Test Companion commit. The loop guard still stops automatic runs.

Which failure is read

  • explain, fix and find the bad PR say which failed run they read, and list any other failures. A comment can name the run: its link, run <id>, or the workflow's name (@browserstack explain e2e; matrix suffixes like (linux) are ignored, and names made only of everyday words such as build or CI are asked for by run id). A named run must be in this repository, on one of the pull request's commits, and red. The "also failed" hint asks by run id. A failed GitHub lookup is reported as such, never as "nothing failed".

  • If nothing has failed yet but CI is still running, the reply says so and lists what is running, instead of "Nothing has failed".

  • explain and find the bad PR say they're read-only, instead of showing the preview banner.

  • The report's Asked by row reads @who | #<comment id>, linking to the comment that asked for the run.

Approving a generate plan saves exactly what was approved

  • A generate plan keeps the exact cases it listed as an artifact on its run (test-companion-cases, kept 7 days), and the plan records which run made it.
  • A plain go saves that file with tc tm add --from-file: no AI runs, so the cases reaching Test Management are the ones the reviewer read, in the project and folder the plan recorded. The reply lists each created case id. Before, go generated the cases again and saved a different set.
  • go with edits still generates again (edits need the AI), and the reply says so.
  • If the plan's cases can't be used (missing, expired, made for another repository or commit, or recording no destination), nothing is saved, the reply says why and asks for a fresh plan, and the plan stays approvable. A partial save lists what was saved and what failed, and the plan can't be approved twice. A plan with saved cases never falls back to generating again.
  • Needs Test Companion CLI 1.1.16 or later, the first release whose cases file records its destination. With an older CLI a plain go on a generate plan saves nothing and says why.

Tests: all 989 pass. The new ones call the real functions, including against real git repositories, and the starter's comment filter is evaluated rather than matched as text.

🤖 Generated with Claude Code

- Ignore comments from bots, our own included. The App's own comments
  quote "@browserstack go", so they started runs, got a "write access"
  reply, and could push a real request out of the queue. The starter
  workflow also skips bot comments, so no runner starts.
- Jobs whose output isn't code (generate, rca, verdict, validate) never
  commit to the branch, and BrowserStack tooling's by-products such as
  browserstack.err are never committed by any job.
- Work handed back as a pull request after a request on a pull request
  now targets that pull request's branch, as a draft, titled for it.
  Before, it targeted the default branch and carried the whole PR.
- A run that broke no longer offers "go" or leaves its check waiting for
  approval; it is reported like any other run, and the comment says why
  (the CLI's error line, e.g. a missing project or a credit limit).
- Acting as the App, commits are made as its bot account, so they show
  its name and logo, and the loop guard still recognises them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@naimish-browse
naimish-browse requested a review from a team as a code owner November 12, 2026 12:18
@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto (fallback from Default branch)
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Central YAML (base), Organization UI (inherited), Workspace UI (inherited); Enforced overrides: Organization
  • Review profile: ASSERTIVE
  • Plan: Enterprise
  • Run ID: b6048f46-4e52-4a3f-ba4d-484b22d9169f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

naimish-browse and others added 13 commits November 12, 2026 17:56
A fix or automate branch that also carried a file Test Companion never
commits, such as browserstack.err written by BrowserStack tooling, was
refused outright, so a real fix could be blocked by a by-product. Now
only files under .github/ and symlinks refuse the branch. Other
never-committed files are left out: one commit on top of the agent's
branch puts them back as they were, built in a scratch index so the
checkout is untouched, and the comment says what was left out. If they
were the branch's only changes, no pull request is opened.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s literally

- A handed-back branch with files Test Companion never commits now gets one
  squashed commit on the base, so those files are not in anything pushed.
  Before, the clean commit sat on top of the agent's commit, which still
  held them.
- git reads every path as a plain file name (GIT_LITERAL_PATHSPECS), so a
  file named like ":!x" can't make git add or reset match other files. The
  left-out paths go to git on stdin.
- The "jobs without code output never commit" rule is checked again with
  the job the pipeline chose, so event runs are covered too.
- The App's commits count as ours even when a later run can't act as the
  App, so it doesn't push in reaction to them.
- A pull request that can't be a draft is opened as a normal one; "already
  done" is only said when its pull request really is open.
- The pull request targets the requesting pull request's branch only when
  the run started on that pull request's commit, and its title names the
  job from the hand-off.
- File names in comments are capped at ten and rendered so a backtick in a
  name can't break out.
- Tests now call the real functions, against real git repositories, and the
  starter's comment filter is evaluated rather than searched for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mments

- The reason shown for a run that broke is the CLI's own error event first,
  then an "Error:" line from its stderr. "Error:" lines in the agent's test
  output on stdout are no longer mistaken for it.
- The reason and the answer are redacted before they reach a pull request
  comment: credential-named environment values (tokens, access keys) and
  user:password in URLs. GitHub masks secrets in logs, not in comments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…closed work

A handed-back fix or automate branch is now always pushed as one commit
on the base, under our name, even when nothing is left out. The agent's
own commits never reach GitHub, so a file it added and later deleted is
not in the pushed history, and what was checked is exactly what is
pushed.

Pushing the branch: only an ordinary rejection means an earlier run
already pushed it. A remote rejection (a protection rule, a missing
permission) is reported as "could not push", never "pushed". When the
branch is already there, an open pull request is reported as done, a
closed or merged one is reported and not reopened, and with none a pull
request is opened and said to be from the earlier run's branch.

Credentials are now hidden in everything sent to GitHub (comments,
replies, checks, pull requests), including the workflow token captured
before the App token replaces it. The squash commit message keeps
agent-chosen names on one line, so they can't add trailers, and the
scratch reset no longer re-reads the checkout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…after a failed lookup

A refused push now reports git's actual reason (the [remote rejected]
line, a remote error such as a ruleset, or a fatal line) instead of the
generic "failed to push some refs". A failed lookup for an earlier
run's pull request now stops with a plain message instead of being
read as "no pull request" and opening one, which could have reopened
closed work. leaveOut retries its reset without --no-refresh on a git
too old to know the option.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h build was read

- Each request (an @browserstack comment, or a `go`) now gets its own
  reply with that run's report, instead of rewriting one shared comment.
  Automatic runs, which answer no request, keep one status comment.
- A spent plan no longer blocks a newer plan for the same job on the
  same commit: spent markers now name the plan they spent. Older
  markers still spend the plans posted before them. `go #<id>` on a
  spent plan says it was already approved or dropped.
- A run a person asked for or approved may save on top of an earlier
  Test Companion commit; the loop guard now applies to automatic runs.
- explain, fix and find-the-bad-PR say which failed run they read and
  list the other failures. A comment can name a run: its link, `run
  <id>`, or a word from the workflow's name. A named run must be in
  this repository, on one of the pull request's commits, and red.
- When nothing has failed yet but CI is still running, the reply says
  so and lists what is running, instead of "Nothing has failed".
- Read-only jobs (explain, find the bad PR) no longer show the preview
  banner, which read as "nothing happened"; they say they're read-only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d on lookups

- Approving a plan now also spends every older plan of that job on the
  commit. Before, approving the newer of two plans left the older one
  waiting: a second "go" would run it, and the nightly sweep would drop
  it and turn the commit's check to "action required" over a green run.
- Naming a failed workflow matches its whole name first (the longest
  wins, so "Lint Checks" beats "Lint"), then a name whose every
  significant word appears; an incidental word no longer picks a run.
  The "also failed" hint now asks by run id, which is unambiguous.
- Workflow names in replies are code spans, so a name can't mention a
  person or team, link or format.
- A manual run may build on our own commit only when a human started
  it, not automation.
- Run lookups fail closed and say so: a GitHub failure is no longer
  reported as "nothing failed" or "no such run"; a run with no commit id
  never matches; a pull request too long to list fully says why.
- The read-only line is chosen from the job asked for, never from the
  agent-written summary, and says nothing the job does is saved.
- Removed latestFailedRun, unused since run selection moved to
  ciOnCommit; its tests now cover ciOnCommit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s don't name a workflow

Approving a plan now spends every plan of the same job on that commit
that already existed when the approval was recorded, older or newer.
Approving an older plan therefore also settles a newer one that was
waiting, so the nightly sweep can no longer drop it and flip the check
over the approved run. A plan posted after the approval stays
approvable. Dropping an expired plan still spends only it and older
plans, never a newer one that may still be waiting. A marker that names
a plan but no job spends only that plan. Old markers without a plan id
keep their meaning.

"go #<id>" on a plan that another approval settled now says a different
plan for this job was approved, instead of claiming this one ran.

Picking a failed workflow by name: a name made only of everyday words
(build, Test, CI, explain, fix) no longer matches, because comments use
those words anyway; such a run is asked for by run id, which is what the
bot's own hint suggests. Matrix suffixes in parentheses are ignored, so
"explain e2e" finds "E2E Tests (linux)" again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…history

Signing in as the App now retries a network error, a 5xx or a 429 up to
three times (1s, then 3s; a Retry-After wins, capped at 10s) before
falling back to the workflow token. 401, 403, 404 and 422 are never
retried. A single network blip had made one reply post as
github-actions[bot] instead of the App.

A handed-back fix is now squashed onto the commit the run started on,
and only if the agent's branch builds on that commit. An agent that ran
`git reset HEAD^` past it had handed back a branch whose diff, squashed
onto the older merge-base, would have undone the pull request's own fix
and tests. Such a branch is now refused with a plain reason, and no
branch is pushed and no pull request is opened.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a person with access comments a request to @browserstack, or a go
for a waiting plan, the Action adds an eyes reaction to that comment as
soon as it accepts it, so they know it was seen before the reply lands.
Bot comments, refused comments and a bare go with no plan to approve get
no reaction. The reaction is best effort: if it can't be added, the run
carries on with a notice. No new permissions: pull-requests: write
already covers reactions on pull request comments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The report table's Asked by row now reads `@who | #<comment id>`, the id
linking to the comment that asked for the run, so a reply can always be
traced back to its request. The pipe is escaped so the table keeps two
columns; without a comment link the row shows just who asked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mmit

- Each call made as the App now gives up after 10 seconds. Without a
  limit a stalled connection waited about five minutes, and each retry
  again, so the retry could hold a job for a quarter of an hour.
- Opening a handed-back fix no longer falls back to HEAD when the
  commit the run started on wasn't recorded: after the run the agent
  decides where HEAD points. It refuses instead.
- The Asked by comment link rejects URLs carrying a pipe, brackets or
  angle brackets, so it can never split the table cell.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed cases

A generate plan is a preview that lists the cases it would create. Replying
go used to start a fresh run that generated the cases again, so what reached
Test Management was not what the reviewer approved.

Now the plan's run keeps its cases as data (the CLI's result JSON, plus the
repo and commit it was made for) in a test-companion-cases artifact, and the
plan records its run id. A plain go downloads that artifact, checks it is for
this repository and this commit, and saves it with `tc tm add --from-file`,
which runs no AI. The reply lists each created case with its id from tm add's
own output, and any that failed.

A go with edits still generates again with the edits as guidance, and the
reply says so. If the cases can't be found, have expired, or Test Management
refuses them before saving any, nothing is saved and the reply asks for a
fresh plan; it never regenerates quietly. A partial save spends the plan, so
approving it again can't save the same cases twice.

A plan's report and result are now attached to its run too; before, a plan
attached nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n file

A plain go on a generate plan still regenerated in a live run. The exact save
was gated on the pipeline binding's tm_project and tm_folder, but the bindings
the CLI reports carry no inputs, so the gate never opened. The tests used a
binding shape the CLI never prints.

Now a plain go on a generate plan that recorded its run always takes the
exact-save path. The destination comes from the plan's cases file (its
top-level `destination`), and `tc tm add --from-file` is called without
--project/--folder so it uses that same destination. A plan whose file has no
destination saves nothing, says why, and stays approvable. Whatever goes
wrong on that path is reported; it never falls back to generating again. A
go with edits, and a plan made before plans recorded their run, behave as
before.

Tests now use the bindings JSON exactly as the CLI prints it and a
real-shaped cases file with and without a destination, including the case
that failed live.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@mdjakkariya mdjakkariya left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: not production-ready yet. 5 fixes needed, all small

I reviewed this from three angles: security, correctness and production readiness. The code is careful overall, and all 905 tests pass locally (node --test). But two items don't do what the description promises, and two security gaps let a prompt-injected agent or a fork influence what gets pushed or saved.

Must fix before merge

1. A file the agent stages itself bypasses the .github/ and symlink filter. Pre-existing code, but this PR relies on it.

  • Where: src/main.mjs:1088 / :1098.
  • What happens: saveWork filters the paths git status lists and then runs git add -- <push>. But git commit -m body takes no file list, so it commits the whole index.
  • Why the agent can do this: setfacl -R … rwX on the repo (src/sandbox.mjs:73) gives the agent user write access to .git/index.
  • Scenario:
    1. An agent that has been prompt-injected runs git add .github/actions/x/action.yml (or CODEOWNERS, a symlink, or .env) and does not commit.
    2. splitPushable puts the path in "skip", and the comment says "Not committed".
    3. The file is pushed to the PR branch anyway.
      .github/workflows/ is still refused by GitHub because neither token has the workflows permission. .github/actions/*, CODEOWNERS and dependabot.yml are not.
  • Fix: git commit --only -- <push>, or git reset -q before git add. Another option is to reuse the scratch-index approach that leaveOut uses for the hand-back path.

2. Approving a fix plan can fix a different failed run than the one the plan described.

  • Where: src/main.mjs:560, :726 and :764.
  • What happens: on the go run, run selection runs again on the request text, which is now just "go". parseRunReference and matchFailedByName both find nothing, so it falls back to ci.failed[0].
  • Example:
    1. Lint (newest) and E2E Tests (linux) are both red.
    2. @browserstack fix e2e produces a plan that says "Looked at E2E Tests (linux)".
    3. The reviewer replies go, and the run fixes Lint.
  • fix run <id> followed by go loses the named run in the same way, and so does any run that turns red between the plan and the go.
  • Fix: record the chosen run id in the proposal marker and reuse it on approval.

3. go on a generate plan is not bound to the cases the reviewer read.

  • Where: src/main.mjs:981-991, src/main.mjs:1598-1662 and src/approved-cases.mjs:42-54.
  • What happens: the proposal shows result.md, but the save reads result.md.json and its destination. The sandboxed agent writes both files. Nothing checks that they describe the same cases, and the proposal never shows the destination.
  • Impact: an agent prompt-injected by the PR's content can make go save different cases into any Test Management project the credentials can reach.
  • Fix:
    • Put a sha256 of result.md.json in the proposal marker (or in subject.json) and verify it before tm add.
    • Show the destination in the proposal.
    • Validate the destination against the pipeline file's tm_project / tm_folder from the default branch.

4. A fork's failed run can be the one that explain, fix or rca reads.

  • Where: src/runs.mjs:47-56 (ciOnCommit) and checkNamedRun.
  • What happens: neither function checks head_repository.
  • Scenario: an outside contributor opens a fork PR on the same public head commit, with a failing workflow named e.g. "CI" whose log carries prompt-injection text. A maintainer's @browserstack fix then reads that log, and the agent pushes code to the same-repo branch.
  • Fix: drop runs where r.head_repository?.full_name !== REPO, and fail closed when the field is missing.

5. Nothing checks the CLI version is at least 1.1.16, and older CLIs get a misleading reply.

  • Where: src/approved-cases.mjs:121-134.
  • What happens: with CLI 1.1.15 or older, a plain go on a generate plan hits no-destination. The reply tells the user to set tm_project / tm_folder. Users who already set them get stuck in a loop. --from-file being unknown is reported as "Test Management refused the cases".
  • Fix: probe for tm add --from-file with the existing cliSupports() (src/main.mjs:1156-1169) and reply "needs Test Companion CLI 1.1.16 or later".

Before release

  • README is out of date:
    • Line 7 still says "one comment"; each request now gets its own reply.
    • The hand-back PR changes aren't documented: draft by default, base is the PR's branch, one squashed bot commit, by-products dropped, history rewrites refused.
    • The GitHub App's required Actions: read permission isn't documented. gh run download needs it, so without it every exact save fails.
    • The CLI ≥ 1.1.16 requirement isn't documented, and the cli-version: "1.2.0" example points to a version that doesn't exist.
    • Existing users aren't told to re-copy the starter workflow. Without it, bot comments still start runners.
  • Tag: the starter and README pin @v1.1.0, but the newest tag is v1.0.6.

Minor / follow-ups

  • A tc tm add killed mid-save (300s timeout or spawn error) reports "refused before any were saved" and leaves the plan approvable. A second go can create duplicate cases (src/main.mjs:1662-1670).
  • Failure detail is lost:
    • A failed artifact lookup is reported as "missing" (src/main.mjs:1648).
    • The stderr from gh run download and tm add is thrown away. Log it after redaction.
  • Unescaped text in the exact-save reply: case titles and URLs from tm add go into the comment as-is, so they can @-mention people or inject links (src/approved-cases.mjs:150-160). Use codeSpan, and only link BrowserStack URLs.
  • subject.json can be forged: it is read from inside the artifact. In pickCasesArtifact, also require that the artifact's workflow_run has a matching head repo and head sha.
  • A PAT breaks the bot filters: if github-token is a personal access token, the bot-comment filter and isOurs (both based on Bot type) fail and the loop guard is disabled. Also skip comments by botLogin, or document that the token must be the App token or the Actions token.
  • Runs are not paginated: actions/runs?head_sha=…&per_page=100 misses failures past 100 runs.
  • Slow App sign-in fallback: the retries can take about 70–100s before falling back to the workflow token.
  • Checkout failure on pull_request: the squash is parented on the merge commit, so the hand-back PR carries the whole original PR.
  • Maintainability: main.mjs is about 1,740 lines. The orchestration (openHandedOffPr, saveApprovedCases, run selection in main()) has no behavioural tests, only regex checks on the source text. Item 2 slipped through that gap.

Checked and found safe

  • Triggering and approving: write access is required, fails closed, and fork PR comments are refused.
  • Command execution: no shell injection; everything uses spawnSync with an argument array.
  • Pathspecs: GIT_LITERAL_PATHSPECS=1 plus NUL-separated --pathspec-from-file.
  • Hand-back branch: a scratch-index squash, never force-pushed, history rewrites refused, push rejections classified correctly.
  • Secret redaction: applied to every ghRaw body. The regexes are free of ReDoS.

Suggestion

At about 3.1k lines across five feature areas, this PR would be safer to land split up. In particular, the generate exact-save part (items 3 and 5) would be better isolated.

naimish-browse and others added 9 commits November 16, 2026 12:49
… replace refs and grafts

The agent can write the checkout's index. A file it staged itself, without
committing (a .github/actions file, CODEOWNERS, a symlink, .env, or a
deletion), was filtered out of the list but still committed, because
`git commit` takes the whole index. The commit is now built in a scratch
index started from the commit the run began on, holding only the files
that passed the filter, and the push sends that commit.

The agent can also write .git/refs/replace and .git/info/grafts, which make
git read a commit as a different tree or with different parents. Our git
commands now ignore both (GIT_NO_REPLACE_OBJECTS, GIT_GRAFT_FILE=/dev/null).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ooked at

On a `go`, the failed run was chosen again from the reply, which names no
run, so it fell back to the newest red run. With Lint (newer) and E2E both
red, `@browserstack fix e2e` planned against E2E and `go` then fixed Lint;
`fix run <id>` lost the named run the same way.

The plan now records the run it read (`build=` in its marker). Approving
reads that run and only that run: if it can no longer be used (no longer
red, not on this pull request's commits) the reply says so, and a plan that
didn't record a run is refused. Neither ever falls back to another run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… plan showed

The plan comment showed the CLI's Markdown result, while approving saved the
JSON cases file. Both are written in the agent's folder, and nothing checked
they described the same cases, so an agent steered by the pull request's
content could make `go` save other cases, or send them elsewhere, without the
reviewer seeing it.

A generate plan is now shown from the same cases file approving saves,
including where it will be saved ("Will be saved to: ..."), and that file's
sha256 goes in our own plan comment's marker. On `go`, the downloaded file
must have that sha256 before anything is saved; otherwise nothing is saved
and nothing is generated in its place. Agent-written text in the plan is
shown as plain text, so it can't mention anyone or add links.

This also fixes the source-text test the previous commit broke.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A fork's pull request can carry the same commit as a branch here, and its
runs show up in this repository's run list. Neither the automatic choice of
failed run nor a run named in a comment checked where the run came from, so
`explain`, `fix` or `find the bad PR` could read a log a fork's author wrote
and act on it.

Both now keep only runs whose head repository is this repository (compared
without case). A run that doesn't say, or a caller that doesn't say which
repository it is, gets nothing: the check fails closed. Naming a fork's run
is refused with a reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…saved

Saving a plan's own cases needs CLI 1.1.16 or later. With an older CLI on the
approving run, `tm add --from-file` doesn't exist and the reply blamed Test
Management; with a plan made by CLI 1.1.15, the cases file has no destination
and the reply told users to set tm_project and tm_folder even when they had,
so they could loop.

The approving run now checks its CLI has `tm add --from-file` before doing
anything, and says it needs 1.1.16 or later (the plan stays approvable). The
plan's run records which CLI made it, so a missing destination from an older
CLI is named as that; a missing destination from a new enough CLI still
points at the pipeline file, and mentions the CLI as the other cause.

The step that runs the Action now receives the CLI version line; it was only
passed to the sign-in step, so the Action never had it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Each request gets its own reply; automatic runs keep one status comment.
- Hand-back pull requests: a draft, based on the pull request asked on, one
  squashed bot commit, by-products left out, history rewrites refused.
- The GitHub App's repository permissions, and why Actions needs read and
  write.
- Saving a generate plan's own cases needs CLI 1.1.16 or later; the pin
  example now uses a version that exists.
- Existing users should copy the starter again, so bot comments stop
  starting runs.
- github-token should be the workflow token or the App's, not a person's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…text, artifact origin)

- A `tc tm add` that stopped without a report (killed, timed out, or no
  output) may have saved some cases. The plan is now used up and the reply
  says to check Test Management before asking again, so a second `go` can't
  save them twice. Only the CLI's own "nothing was saved" leaves it
  approvable.
- A failed lookup of the plan's artifacts is reported as a lookup failure
  (the plan stays approvable), not as "the cases are missing". The error
  output of `gh run download` and `tm add` goes to the job log with secrets
  hidden, never to the comment.
- The exact-save reply shows titles, ids and errors as plain text, and links
  a case only when its address is https on browserstack.com.
- The cases artifact must come from a run in this repository that started
  from it (repository_id and head_repository_id). Its commit is not
  compared: a plan asked for in a comment runs on the default branch's
  commit; the plan's commit is checked through subject.json and the cases'
  fingerprint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…person's

Our comments quote "@browserstack go". The bot filters skip them by account
type, which works for the workflow token and the App, but a personal access
token posts as a person, so our own replies could start runs.

The Action now asks GitHub who github-token belongs to (only a person's
token can read GET /user) and, if it's a person, skips that account's
comments and warns that the workflow token or the App should be used. The
account we post as is skipped whatever its type. The README already says
github-token should be the workflow token or the App's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Choosing which failed run to read, and cancelling runs when a pull request
closes, both listed one page of 100 runs. A busy commit could have its
failure past the first page. Both now read every page, up to 1,000 runs. If
any page can't be read, the list counts as unreadable, so a partial list
can't make "nothing failed" look true.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
naimish-browse and others added 2 commits November 16, 2026 14:03
Each of the two App calls could take three 10-second tries plus waits, so a
GitHub outage delayed the fallback to the workflow token by 70 to 100
seconds. The sign-in now has one 30-second budget for every call, try and
wait: each try's time limit shrinks to what's left, no try starts after the
deadline, and no wait runs that would end past it. The retry rules are
otherwise unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e commit

On a pull_request event the job starts on GitHub's merge of the pull
request into its base. If moving to the pull request's own commit failed,
the run worked on the merge, the squash was parented on it, and the
hand-back pull request (based on the default branch) carried the whole
original pull request plus the fix.

That case is now refused before anything is squashed or pushed, and the
reply says why. Parenting the squash on the pull request's commit instead
would not be right: the run's files are the merge's, so it would pull in the
base branch's newer changes. A comment run that stayed on the default
branch keeps its documented fallback. The README says both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@naimish-browse

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review, @mdjakkariya. I checked every point against the code before changing anything. All five must-fix items were real and are fixed. Each fix is below with its commit. All 968 tests pass (node --test, no not ok lines). I haven't run these changes on a live pull request yet; I'll do that before asking you to look again.

Must fix before merge

1. Files the agent staged itself were committed. Confirmed: git commit took the whole index, and the agent can write it.

  • Fixed in ac2478a. The commit is now built in a scratch index, starting from the commit the run began on, with only the filtered paths. Whatever the agent staged is ignored.
  • Tested against a real repository. A pre-staged .github/actions/x/action.yml, CODEOWNERS, .env, a symlink and a staged deletion are all kept out. The real index is left untouched.
  • While checking .git more widely, I found two more ways the agent could mislead git: replace refs (.git/refs/replace/*) and grafts (.git/info/grafts). A replace ref could make the base commit appear to contain a .github file, and a graft could make the "builds on the reviewed commit" check pass falsely. The Action's git calls now ignore both. Tests show plain git follows them and the Action's git doesn't.
  • Hooks, fsmonitor, credential helpers and insteadOf were already neutralised. There's now a test for each.

2. go on a fix plan could fix a different failed run. Confirmed.

  • Fixed in 425eafe and e76f3f9. The plan's marker records which failed run it read. go reads exactly that run, after checking it is still red and on one of the pull request's commits. Otherwise the reply says why, and it never falls back to another run.
  • Tests cover fix e2e then go with a newer red Lint run, fix run <id> then go, and a recorded run that turned green.

3. go on a generate plan wasn't tied to the cases the reviewer read. Confirmed.

  • Fixed in e76f3f9. A generate plan is now drawn from the same cases file that gets saved, and shows where the cases will be saved (project and folder). Agent-written text in it is shown as plain text.
  • The file's sha256 is in the plan's marker. go saves only if the downloaded file matches it. Otherwise nothing is saved and nothing is generated in its place.
  • Not done: checking the destination against the pipeline file's tm_project / tm_folder. tc workflow bindings --json doesn't include a binding's inputs, so the Action would have to parse the YAML itself, and we'd rather not. The destination is now shown in the plan and covered by the hash, so what the reviewer approves is exactly what gets saved, and where.

4. A fork's failed run could be the one read. Confirmed.

  • Fixed in 40ef7c1. Only runs whose head repository is this repository are considered, compared without case. It fails closed if that field is missing. Naming a fork's run in a comment is refused with a reason.

5. No check for CLI 1.1.16, and misleading replies with older CLIs. Confirmed.

  • Fixed in d3e9c79. Before saving, the Action checks that tc tm add supports --from-file. If it doesn't, the reply says the save needs Test Companion CLI 1.1.16 or later, nothing is saved, and the plan stays approvable.
  • A plan made by an older CLI now gets its own message, instead of "set tm_project / tm_folder".
  • This also fixed a related gap: the CLI version wasn't being passed to the step that needs it.

Before release

  • README (3fc6bce):
    • Each request now gets its own reply; automatic runs keep one status comment.
    • Hand-back pull requests are documented: a draft, based on the pull request's branch, one squashed bot commit, by-products left out, history rewrites refused.
    • The App's permissions are listed with reasons, including why Actions needs read and write.
    • "Needs CLI 1.1.16 or later" is documented, and the cli-version example now uses 1.1.16.
    • Existing users are told to copy the starter workflow again.
  • Tag: @v1.1.0 is deliberate. It's the release this PR goes out in, so the README and the starter point at it ahead of time.

Minor / follow-ups

  • An interrupted tc tm add (killed, timed out, or no report printed) now counts as "may have saved some". The plan is used up, and the reply says to check Test Management before asking again. So a second go can't create duplicates (2dbf7a4).
  • Failure detail: a failed artifact lookup is reported as a lookup failure, not as "missing". Error output from gh run download and tm add goes to the job log with secrets hidden, never to the comment (2dbf7a4).
  • Unescaped text in the save reply: titles, IDs and errors are now plain text, and a case is linked only when its address is https on browserstack.com or a subdomain (2dbf7a4).
  • subject.json forgery: the artifact's run must now belong to this repository, using both repository_id and head_repository_id, and it fails closed (2dbf7a4).
    • I didn't add the head-commit check. A plan asked for in a comment runs on the default branch's commit, so requiring the pull request's commit would reject every comment-made plan.
    • The plan's commit is already tied in through subject.json, the sha256 in the plan's marker, and the marker being keyed to that commit.
  • A personal access token as github-token (045938c):
    • The Action now detects a personal token, skips that account's comments, and warns. It always skips the account it posts as.
    • The README says to use the workflow token or the App's.
    • One known limit, stated in the warning: with a personal token it can't recognise its own plans, because that check relies on a bot account.
  • Run listings: both run listings now read every page, up to 1,000 runs. A page that can't be read makes the whole list count as unreadable, so a partial list can't produce "nothing failed" (3c1fea8).
  • Slow App sign-in fallback: the whole sign-in now shares a 30-second budget across every call, try and wait (55b36cb).
  • Hand-back built on the merge commit: confirmed for a pull_request run where moving to the pull request's commit fails.
    • Parenting the squash on the PR's commit would pull the base branch's newer changes into the PR, so that case is now refused before anything is squashed or pushed, and the reply says why (e718ed0).
    • A comment run that stayed on the default branch keeps its documented fallback.
  • main.mjs size and orchestration tests: agreed that it's large. The new behaviour is covered by tests of the functions it calls, and item 2 now has behavioural tests. I'd rather split main.mjs in a follow-up than grow this PR further.

Splitting the PR

I'm keeping it as one PR, so the release has one reviewed unit. The fixes above are separate commits, so each can be reviewed on its own.

@naimish-browse

Copy link
Copy Markdown
Contributor Author

I've now run the two flows these fixes change, live on our internal pilot repository, using this PR's tip (e718ed0) and Test Companion CLI 1.1.16. Both behaved as intended.

generate, then go (items 3 and 5):

  • The plan showed "Will be saved to: in (, )" and listed its 11 cases from the cases file.
  • Its marker carried cases=<sha256>. I downloaded the run's test-companion-cases artifact and computed the hash myself: it matched exactly.
  • go saved from that file, with no new generation. The reply listed 11 new case IDs, and agent text in it was escaped.
  • I pulled all 11 cases back from Test Management. Title, precondition, priority and every step match the approved file, 11 of 11.

fix, then go (items 1 and 2):

  • The plan's marker recorded the failed run it read (build=<run id>).
  • The approved run's log shows it read exactly that run: "reading the failure from …/runs/".
  • The hand-back pull request is a draft, based on the pull request's branch. It holds one commit by browserstack[bot], whose parent is the reviewed commit, and it changes only the one file the fix needed.

Bot replies: the runs started by the bot's own replies were skipped before doing anything.

Ready for another look when you have time.

@mdjakkariya

Copy link
Copy Markdown

Re-review of the fixes (181fa6d..e718ed0): one item still needs fixing

Thanks for the thorough round. I checked each fix against the code. node --test gives 968 pass and 0 fail, and CI is green.

Verified fixed

  • 1. Agent-staged files are no longer committed.
    • commitPaths (src/push-work.mjs:495-509) builds the commit in a scratch index under RUNNER_TEMP, which the agent cannot write.
    • It starts with read-tree of headBeforeRun, adds only the filtered paths, then runs write-tree and commit-tree -p base.
    • GIT_NO_REPLACE_OBJECTS and GIT_GRAFT_FILE=/dev/null are set in gitEnvFrom, so every git call of the Action ignores replace refs and grafts.
  • 2. go on a fix plan uses the run the plan read.
    • The marker carries build=<id>, and go re-checks it (still red, on the PR's commits, not from a fork). It never falls back to another run.
    • The agent cannot plant a marker: isOurs and inertMarkers hold, and the last marker wins.
  • 4. Runs from forks are not read.
    • isOwnRepoRun is applied in ciOnCommit and checkNamedRun, compares without case, and fails closed.
    • The artifact's repository_id and head_repository_id are checked.
  • 5. CLIs older than 1.1.16 are detected. The tm add --help probe catches them and the reply names the version needed.
  • Minor items: all of the following are fixed:
    • interrupted saves now use up the plan;
    • a failed lookup is reported as a failure, not as "missing";
    • the link allowlist rejects evilbrowserstack.com and browserstack.com.evil.io;
    • a personal token as github-token is detected;
    • run listing is paginated, and a failed page fails the whole list closed;
    • App sign-in has a 30s budget;
    • a hand-back on the merge commit is refused;
    • the README is up to date.

Still needed before merge

3. A generate plan still doesn't show everything go saves.

The hash proves the file wasn't swapped after the plan was posted. But the reviewer still doesn't see all of what gets saved:

  • renderPlannedCases (src/approved-cases.mjs:63) shows only id, title, priority, preconditions and steps.
  • tc tm add --from-file saves more fields from the same file:
    • In ResultJSONCase (cli/pkg/cli/workflow/result_json.go): description, type, tags, references and customFields.
    • For BDD cases (format: "bdd"): feature, background and bddScenario. cli/pkg/cli/tm_add.go:364-373 sends custom fields and the BDD scenario.

What this means in practice:

  • BDD projects: the plan shows titles only, because BDD cases don't use steps. The reviewer approves scenarios they never read.
  • Prompt injection: an agent steered by text in the PR can put arbitrary text or links into description or references. They reach Test Management without anyone seeing them.

This undercuts the guarantee that "what is shown and what is saved can't differ".

Fix, either of:

  • Render every field the CLI saves in the plan, including feature, background and bddScenario for BDD cases. Escape them with plain(), as you already do for the shown fields.
  • Or, smaller: refuse the exact save, with a reason, when any case has a non-empty field that the plan doesn't render.

Optional (fine as follow-ups)

  • A staged rename becomes a copy. changedPaths keeps only the new path of R old -> new. The scratch index starts from the base commit, so after git mv a b the pushed commit contains both a and b. Adding the old path to the paths list records it as deleted.
  • A failure before anything is saved still uses up the plan (src/main.mjs:1785-1793). This covers tc tm add failures such as auth, connection or destination-resolution errors. The reply then says the cases "may already be saved". Erring on the side of no duplicates is fine, but the wording is wrong in this case.
  • Already-saved cases are listed as new. Cases that already have a folder (already in TM) appear as "will be saved". The CLI skips them, so add a note.
  • A misleading reply when the cases file couldn't be rendered. The plan falls back to the agent's markdown without cases=. Approving it then replies "an older version of Test Companion made it".
  • The object store is still a trust gap. The agent can still write .git/objects, so a forged pack could in theory change what the base tree reads as. The old code had the same exposure. You could close it by comparing ls-tree -r of the new commit against a listing taken before the agent ran.

Once item 3 is in, I'll approve.

naimish-browse and others added 5 commits November 16, 2026 15:32
The plan showed a case's id, title, priority, preconditions and steps, but
`tc tm add --from-file` also saves its description, type, tags, references
and custom fields, and for a BDD case its feature, background and scenario.
A reviewer could approve text and links they never saw.

- The plan now shows every one of those fields, each case in a collapsible
  block, all as plain text: multi-line text in a code block, references and
  custom fields unlinked, no mentions. A case already in Test Management is
  labelled as skipped.
- One list of shown fields drives both the plan and a check run just before
  saving: if any case carries a non-empty field the plan doesn't show (an
  unknown or newer field, or one of the wrong shape), nothing is saved and
  the reply says why. The plan warns about it too.
- A plan too large for one comment shows the cases that fit, says it can't
  be approved to save exactly as shown, and records that in its marker, so
  approving it is refused and asks for fewer cases. The run's own report
  above the cases is kept short so the cases get most of the room.
- A generate plan that had to fall back to the agent's text records that
  too, and approving it says the plan couldn't be checked against its cases
  file, rather than blaming an older version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`git status` reports a staged rename as `R  old -> new`, and only the new
path was kept. The commit is built from the base, so after `git mv a b` the
pushed commit had both a and b. The old path of a rename is now included, so
it is recorded as deleted. A copy (C) still keeps its source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y then

When `tc tm add` stopped without a report, the reply said cases "may
already be saved" even when it had failed before saving anything, such as a
sign-in, connection or folder lookup error.

The CLI prints its report as soon as it has started creating cases, so a
run that exited by itself with an error and no report failed before saving,
and one that never started saved nothing. Those now say nothing was saved
and leave the plan approvable. A kill, a timeout, a crash, or a clean exit
with no report may have stopped mid-save: those still use the plan up and
say to check Test Management first, so a second `go` can't save twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…re the run

The agent can write .git. Git checks the hash of every object it reads, so
a forged loose or packed object is refused by git itself. But some files are
trusted without that check, and the base the Action builds on is read after
the agent ran:

- Before the agent runs, the Action now lists the base's whole tree and
  reads .git/objects/info/alternates and http-alternates, keeping both in
  its own memory, which the agent can't reach.
- Before pushing a saved commit or a hand-back pull request, it refuses if
  the alternates files changed, and compares the new commit's whole tree
  with that listing: every path that differs must be one the Action chose
  (for a saved commit) and one it would ever commit.
- Its git calls no longer use the commit-graph or multi-pack-index files,
  which git reads without checking the objects (core.commitGraph=false,
  core.multiPackIndex=false). Variables that point git at other objects were
  already never passed on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@naimish-browse

Copy link
Copy Markdown
Contributor Author

Thanks, @mdjakkariya. Item 3 is in, and so are all five optional items. I checked each one against the Action's code and the CLI's source (result_json.go, plus resultCaseToTM and finishTmAdd in tm_add.go) before changing anything. The tip is now f2cb285, and 989 tests pass with no not ok lines.

3. The plan now shows everything go saves (fa37db7)

  • Every field shown. Each case is a collapsible block listing every field tc tm add --from-file sends: description, type, priority, preconditions, steps with expected results, tags, references and custom fields. BDD cases also get their feature, background and scenario. Everything is plain text. Multi-line text goes in a fence longer than any backticks inside it, nothing is clickable, and nothing can @-mention anyone. The "Will be saved to" line is unchanged.
  • One field list for showing and checking. A single list of shown fields drives both the plan and a check that runs right before tm add. If any case has a non-empty field the plan doesn't show, go saves nothing, says why, and leaves the plan approvable. That includes an unknown or newer field, or a field of the wrong type. The plan also warns about it up front.
  • Plans too big for one comment. The plan shows what fits and says it can't be approved for an exact save. The marker records shown=partial, so go refuses it and asks for fewer cases.
  • One limit the Action can't close. When a required Test Management field is left empty, the CLI fills it from the project's own defaults at save time. Those values come from the project, not the cases file, so the plan can't show them.

Optional items

  • Staged rename became a copy (52da1a1). The old path of a rename is now committed as deleted, while a copy keeps its source. Covered by a real-git git mv test.
  • Wrong "may already be saved" wording (9d0f22b).
    • The CLI prints its report once it starts creating cases. An error exit with no report therefore means it failed before saving anything: reading the file, signing in, or resolving the destination.
    • Those cases, and a CLI that never started, now say nothing was saved, and the plan stays approvable.
    • A kill, timeout, crash, or clean exit with no report still uses the plan up and says some may be saved, so a second go can't create duplicates.
  • Already-saved cases listed as new (fa37db7). They're now labelled "Already in Test Management: saving skips this case".
  • Misleading reply when the plan fell back to the agent's text (fa37db7). Such a plan records shown=markdown, and approving it says the plan couldn't be checked against its cases file. "An older version made it" is now used only for plans that really are older.
  • Object store (095fccb). Partly as described.
    • Git already rejects a forged object: both a swapped loose object and a hand-built pack whose index claims the base tree's id fail with "hash mismatch". There are tests for both.
    • The real gaps were three. The commit-graph file and the multi-pack-index aren't hash-checked, so they're now turned off for the Action's git calls (core.commitGraph=false, core.multiPackIndex=false). The alternates / http-alternates files could also feed objects in.
    • Before the agent runs, the Action records the base tree's full listing and the alternates files in its own memory.
    • Before any push, it refuses if the alternates changed. It also refuses if the new commit's tree differs from that listing anywhere other than the paths the Action chose to commit.
    • gitEnvFrom never passes on GIT_ALTERNATE_OBJECT_DIRECTORIES, GIT_OBJECT_DIRECTORY, GIT_DIR or GIT_INDEX_FILE.

Live check

I ran this on our internal pilot repository at f2cb285, with CLI 1.1.16.

  • The generate plan showed each case's type, priority, description, preconditions and steps, plus "Will be saved to". I checked its cases= hash against the downloaded artifact myself, and it matched.
  • go saved the 11 cases with no new generation.
  • I read them back from Test Management. Title, precondition, priority, type, every step and every description match the approved file, 11 of 11.

@mdjakkariya mdjakkariya left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-verified at f2cb285. All findings from my earlier review are addressed.

  • 3. The generate plan shows every saved field.
    • The plan's field list (CASE_FIELDS, src/approved-cases.mjs:96-126) matches every field the CLI's resultCaseToTM sends:
      • title, description, preconditions, priority, type, tags, references and customFields;
      • format, feature, background and bddScenario for BDD cases;
      • steps, each with its step and result.
    • Right before tm add, saveExactly checks the sha256 (src/main.mjs:1841), then refuses any non-empty field the plan didn't show, or a field of an unexpected type (:1849-1850).
    • Plans marked shown=partial or shown=markdown are refused.
    • Shown text is escaped: plain text, fences longer than any backticks inside, and summaries escaped for HTML with @-mentions neutralised.
  • Optional items:
    • A staged rename now records its old path as deleted.
    • A save is reported as having saved nothing only when that is certain.
    • Already-saved cases are labelled as such.
    • A plan that fell back to the agent's text gets its own message.
    • commit-graph and multi-pack-index are off, and the alternates files and the base-tree listing are checked before any push.
  • node --test gives 989 pass, 0 fail, and CI is green.

LGTM.

@francisf
francisf merged commit e1ceb20 into browserstack:master Nov 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants