Repository navigation
test-companion: fixes from running every job as the GitHub App - #96
Conversation
- Ignore comments from bots, our own included. The App's own comments quote "@browserstack go", so they started runs, got a "write access" reply, and could push a real request out of the queue. The starter workflow also skips bot comments, so no runner starts. - Jobs whose output isn't code (generate, rca, verdict, validate) never commit to the branch, and BrowserStack tooling's by-products such as browserstack.err are never committed by any job. - Work handed back as a pull request after a request on a pull request now targets that pull request's branch, as a draft, titled for it. Before, it targeted the default branch and carried the whole PR. - A run that broke no longer offers "go" or leaves its check waiting for approval; it is reported like any other run, and the comment says why (the CLI's error line, e.g. a missing project or a credit limit). - Acting as the App, commits are made as its bot account, so they show its name and logo, and the loop guard still recognises them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the
|
A fix or automate branch that also carried a file Test Companion never commits, such as browserstack.err written by BrowserStack tooling, was refused outright, so a real fix could be blocked by a by-product. Now only files under .github/ and symlinks refuse the branch. Other never-committed files are left out: one commit on top of the agent's branch puts them back as they were, built in a scratch index so the checkout is untouched, and the comment says what was left out. If they were the branch's only changes, no pull request is opened. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s literally - A handed-back branch with files Test Companion never commits now gets one squashed commit on the base, so those files are not in anything pushed. Before, the clean commit sat on top of the agent's commit, which still held them. - git reads every path as a plain file name (GIT_LITERAL_PATHSPECS), so a file named like ":!x" can't make git add or reset match other files. The left-out paths go to git on stdin. - The "jobs without code output never commit" rule is checked again with the job the pipeline chose, so event runs are covered too. - The App's commits count as ours even when a later run can't act as the App, so it doesn't push in reaction to them. - A pull request that can't be a draft is opened as a normal one; "already done" is only said when its pull request really is open. - The pull request targets the requesting pull request's branch only when the run started on that pull request's commit, and its title names the job from the hand-off. - File names in comments are capped at ten and rendered so a backtick in a name can't break out. - Tests now call the real functions, against real git repositories, and the starter's comment filter is evaluated rather than searched for. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mments - The reason shown for a run that broke is the CLI's own error event first, then an "Error:" line from its stderr. "Error:" lines in the agent's test output on stdout are no longer mistaken for it. - The reason and the answer are redacted before they reach a pull request comment: credential-named environment values (tokens, access keys) and user:password in URLs. GitHub masks secrets in logs, not in comments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…closed work A handed-back fix or automate branch is now always pushed as one commit on the base, under our name, even when nothing is left out. The agent's own commits never reach GitHub, so a file it added and later deleted is not in the pushed history, and what was checked is exactly what is pushed. Pushing the branch: only an ordinary rejection means an earlier run already pushed it. A remote rejection (a protection rule, a missing permission) is reported as "could not push", never "pushed". When the branch is already there, an open pull request is reported as done, a closed or merged one is reported and not reopened, and with none a pull request is opened and said to be from the earlier run's branch. Credentials are now hidden in everything sent to GitHub (comments, replies, checks, pull requests), including the workflow token captured before the App token replaces it. The squash commit message keeps agent-chosen names on one line, so they can't add trailers, and the scratch reset no longer re-reads the checkout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…after a failed lookup A refused push now reports git's actual reason (the [remote rejected] line, a remote error such as a ruleset, or a fatal line) instead of the generic "failed to push some refs". A failed lookup for an earlier run's pull request now stops with a plain message instead of being read as "no pull request" and opening one, which could have reopened closed work. leaveOut retries its reset without --no-refresh on a git too old to know the option. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h build was read - Each request (an @browserstack comment, or a `go`) now gets its own reply with that run's report, instead of rewriting one shared comment. Automatic runs, which answer no request, keep one status comment. - A spent plan no longer blocks a newer plan for the same job on the same commit: spent markers now name the plan they spent. Older markers still spend the plans posted before them. `go #<id>` on a spent plan says it was already approved or dropped. - A run a person asked for or approved may save on top of an earlier Test Companion commit; the loop guard now applies to automatic runs. - explain, fix and find-the-bad-PR say which failed run they read and list the other failures. A comment can name a run: its link, `run <id>`, or a word from the workflow's name. A named run must be in this repository, on one of the pull request's commits, and red. - When nothing has failed yet but CI is still running, the reply says so and lists what is running, instead of "Nothing has failed". - Read-only jobs (explain, find the bad PR) no longer show the preview banner, which read as "nothing happened"; they say they're read-only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d on lookups - Approving a plan now also spends every older plan of that job on the commit. Before, approving the newer of two plans left the older one waiting: a second "go" would run it, and the nightly sweep would drop it and turn the commit's check to "action required" over a green run. - Naming a failed workflow matches its whole name first (the longest wins, so "Lint Checks" beats "Lint"), then a name whose every significant word appears; an incidental word no longer picks a run. The "also failed" hint now asks by run id, which is unambiguous. - Workflow names in replies are code spans, so a name can't mention a person or team, link or format. - A manual run may build on our own commit only when a human started it, not automation. - Run lookups fail closed and say so: a GitHub failure is no longer reported as "nothing failed" or "no such run"; a run with no commit id never matches; a pull request too long to list fully says why. - The read-only line is chosen from the job asked for, never from the agent-written summary, and says nothing the job does is saved. - Removed latestFailedRun, unused since run selection moved to ciOnCommit; its tests now cover ciOnCommit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s don't name a workflow Approving a plan now spends every plan of the same job on that commit that already existed when the approval was recorded, older or newer. Approving an older plan therefore also settles a newer one that was waiting, so the nightly sweep can no longer drop it and flip the check over the approved run. A plan posted after the approval stays approvable. Dropping an expired plan still spends only it and older plans, never a newer one that may still be waiting. A marker that names a plan but no job spends only that plan. Old markers without a plan id keep their meaning. "go #<id>" on a plan that another approval settled now says a different plan for this job was approved, instead of claiming this one ran. Picking a failed workflow by name: a name made only of everyday words (build, Test, CI, explain, fix) no longer matches, because comments use those words anyway; such a run is asked for by run id, which is what the bot's own hint suggests. Matrix suffixes in parentheses are ignored, so "explain e2e" finds "E2E Tests (linux)" again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…history Signing in as the App now retries a network error, a 5xx or a 429 up to three times (1s, then 3s; a Retry-After wins, capped at 10s) before falling back to the workflow token. 401, 403, 404 and 422 are never retried. A single network blip had made one reply post as github-actions[bot] instead of the App. A handed-back fix is now squashed onto the commit the run started on, and only if the agent's branch builds on that commit. An agent that ran `git reset HEAD^` past it had handed back a branch whose diff, squashed onto the older merge-base, would have undone the pull request's own fix and tests. Such a branch is now refused with a plain reason, and no branch is pushed and no pull request is opened. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a person with access comments a request to @browserstack, or a go for a waiting plan, the Action adds an eyes reaction to that comment as soon as it accepts it, so they know it was seen before the reply lands. Bot comments, refused comments and a bare go with no plan to approve get no reaction. The reaction is best effort: if it can't be added, the run carries on with a notice. No new permissions: pull-requests: write already covers reactions on pull request comments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The report table's Asked by row now reads `@who | #<comment id>`, the id linking to the comment that asked for the run, so a reply can always be traced back to its request. The pipe is escaped so the table keeps two columns; without a comment link the row shows just who asked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mmit - Each call made as the App now gives up after 10 seconds. Without a limit a stalled connection waited about five minutes, and each retry again, so the retry could hold a job for a quarter of an hour. - Opening a handed-back fix no longer falls back to HEAD when the commit the run started on wasn't recorded: after the run the agent decides where HEAD points. It refuses instead. - The Asked by comment link rejects URLs carrying a pipe, brackets or angle brackets, so it can never split the table cell. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed cases A generate plan is a preview that lists the cases it would create. Replying go used to start a fresh run that generated the cases again, so what reached Test Management was not what the reviewer approved. Now the plan's run keeps its cases as data (the CLI's result JSON, plus the repo and commit it was made for) in a test-companion-cases artifact, and the plan records its run id. A plain go downloads that artifact, checks it is for this repository and this commit, and saves it with `tc tm add --from-file`, which runs no AI. The reply lists each created case with its id from tm add's own output, and any that failed. A go with edits still generates again with the edits as guidance, and the reply says so. If the cases can't be found, have expired, or Test Management refuses them before saving any, nothing is saved and the reply asks for a fresh plan; it never regenerates quietly. A partial save spends the plan, so approving it again can't save the same cases twice. A plan's report and result are now attached to its run too; before, a plan attached nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n file A plain go on a generate plan still regenerated in a live run. The exact save was gated on the pipeline binding's tm_project and tm_folder, but the bindings the CLI reports carry no inputs, so the gate never opened. The tests used a binding shape the CLI never prints. Now a plain go on a generate plan that recorded its run always takes the exact-save path. The destination comes from the plan's cases file (its top-level `destination`), and `tc tm add --from-file` is called without --project/--folder so it uses that same destination. A plan whose file has no destination saves nothing, says why, and stays approvable. Whatever goes wrong on that path is reported; it never falls back to generating again. A go with edits, and a plan made before plans recorded their run, behave as before. Tests now use the bindings JSON exactly as the CLI prints it and a real-shaped cases file with and without a destination, including the case that failed live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mdjakkariya
left a comment
There was a problem hiding this comment.
Review: not production-ready yet. 5 fixes needed, all small
I reviewed this from three angles: security, correctness and production readiness. The code is careful overall, and all 905 tests pass locally (node --test). But two items don't do what the description promises, and two security gaps let a prompt-injected agent or a fork influence what gets pushed or saved.
Must fix before merge
1. A file the agent stages itself bypasses the .github/ and symlink filter. Pre-existing code, but this PR relies on it.
- Where:
src/main.mjs:1088/:1098. - What happens:
saveWorkfilters the pathsgit statuslists and then runsgit add -- <push>. Butgit commit -m bodytakes no file list, so it commits the whole index. - Why the agent can do this:
setfacl -R … rwXon the repo (src/sandbox.mjs:73) gives the agent user write access to.git/index. - Scenario:
- An agent that has been prompt-injected runs
git add .github/actions/x/action.yml(orCODEOWNERS, a symlink, or.env) and does not commit. splitPushableputs the path in "skip", and the comment says "Not committed".- The file is pushed to the PR branch anyway.
.github/workflows/is still refused by GitHub because neither token has theworkflowspermission..github/actions/*,CODEOWNERSanddependabot.ymlare not.
- An agent that has been prompt-injected runs
- Fix:
git commit --only -- <push>, orgit reset -qbeforegit add. Another option is to reuse the scratch-index approach thatleaveOutuses for the hand-back path.
2. Approving a fix plan can fix a different failed run than the one the plan described.
- Where:
src/main.mjs:560,:726and:764. - What happens: on the
gorun, run selection runs again on the request text, which is now just "go".parseRunReferenceandmatchFailedByNameboth find nothing, so it falls back toci.failed[0]. - Example:
- Lint (newest) and
E2E Tests (linux)are both red. @browserstack fix e2eproduces a plan that says "Looked at E2E Tests (linux)".- The reviewer replies
go, and the run fixes Lint.
- Lint (newest) and
fix run <id>followed bygoloses the named run in the same way, and so does any run that turns red between the plan and thego.- Fix: record the chosen run id in the proposal marker and reuse it on approval.
3. go on a generate plan is not bound to the cases the reviewer read.
- Where:
src/main.mjs:981-991,src/main.mjs:1598-1662andsrc/approved-cases.mjs:42-54. - What happens: the proposal shows
result.md, but the save readsresult.md.jsonand itsdestination. The sandboxed agent writes both files. Nothing checks that they describe the same cases, and the proposal never shows the destination. - Impact: an agent prompt-injected by the PR's content can make
gosave different cases into any Test Management project the credentials can reach. - Fix:
- Put a sha256 of
result.md.jsonin the proposal marker (or insubject.json) and verify it beforetm add. - Show the destination in the proposal.
- Validate the destination against the pipeline file's
tm_project/tm_folderfrom the default branch.
- Put a sha256 of
4. A fork's failed run can be the one that explain, fix or rca reads.
- Where:
src/runs.mjs:47-56(ciOnCommit) andcheckNamedRun. - What happens: neither function checks
head_repository. - Scenario: an outside contributor opens a fork PR on the same public head commit, with a failing workflow named e.g. "CI" whose log carries prompt-injection text. A maintainer's
@browserstack fixthen reads that log, and the agent pushes code to the same-repo branch. - Fix: drop runs where
r.head_repository?.full_name !== REPO, and fail closed when the field is missing.
5. Nothing checks the CLI version is at least 1.1.16, and older CLIs get a misleading reply.
- Where:
src/approved-cases.mjs:121-134. - What happens: with CLI 1.1.15 or older, a plain
goon agenerateplan hitsno-destination. The reply tells the user to settm_project/tm_folder. Users who already set them get stuck in a loop.--from-filebeing unknown is reported as "Test Management refused the cases". - Fix: probe for
tm add --from-filewith the existingcliSupports()(src/main.mjs:1156-1169) and reply "needs Test Companion CLI 1.1.16 or later".
Before release
- README is out of date:
- Line 7 still says "one comment"; each request now gets its own reply.
- The hand-back PR changes aren't documented: draft by default, base is the PR's branch, one squashed bot commit, by-products dropped, history rewrites refused.
- The GitHub App's required Actions: read permission isn't documented.
gh run downloadneeds it, so without it every exact save fails. - The CLI ≥ 1.1.16 requirement isn't documented, and the
cli-version: "1.2.0"example points to a version that doesn't exist. - Existing users aren't told to re-copy the starter workflow. Without it, bot comments still start runners.
- Tag: the starter and README pin
@v1.1.0, but the newest tag isv1.0.6.
Minor / follow-ups
- A
tc tm addkilled mid-save (300s timeout or spawn error) reports "refused before any were saved" and leaves the plan approvable. A secondgocan create duplicate cases (src/main.mjs:1662-1670). - Failure detail is lost:
- A failed artifact lookup is reported as "missing" (
src/main.mjs:1648). - The stderr from
gh run downloadandtm addis thrown away. Log it after redaction.
- A failed artifact lookup is reported as "missing" (
- Unescaped text in the exact-save reply: case titles and URLs from
tm addgo into the comment as-is, so they can @-mention people or inject links (src/approved-cases.mjs:150-160). UsecodeSpan, and only link BrowserStack URLs. subject.jsoncan be forged: it is read from inside the artifact. InpickCasesArtifact, also require that the artifact'sworkflow_runhas a matching head repo and head sha.- A PAT breaks the bot filters: if
github-tokenis a personal access token, the bot-comment filter andisOurs(both based on Bot type) fail and the loop guard is disabled. Also skip comments bybotLogin, or document that the token must be the App token or the Actions token. - Runs are not paginated:
actions/runs?head_sha=…&per_page=100misses failures past 100 runs. - Slow App sign-in fallback: the retries can take about 70–100s before falling back to the workflow token.
- Checkout failure on
pull_request: the squash is parented on the merge commit, so the hand-back PR carries the whole original PR. - Maintainability:
main.mjsis about 1,740 lines. The orchestration (openHandedOffPr,saveApprovedCases, run selection inmain()) has no behavioural tests, only regex checks on the source text. Item 2 slipped through that gap.
Checked and found safe
- Triggering and approving: write access is required, fails closed, and fork PR comments are refused.
- Command execution: no shell injection; everything uses
spawnSyncwith an argument array. - Pathspecs:
GIT_LITERAL_PATHSPECS=1plus NUL-separated--pathspec-from-file. - Hand-back branch: a scratch-index squash, never force-pushed, history rewrites refused, push rejections classified correctly.
- Secret redaction: applied to every
ghRawbody. The regexes are free of ReDoS.
Suggestion
At about 3.1k lines across five feature areas, this PR would be safer to land split up. In particular, the generate exact-save part (items 3 and 5) would be better isolated.
… replace refs and grafts The agent can write the checkout's index. A file it staged itself, without committing (a .github/actions file, CODEOWNERS, a symlink, .env, or a deletion), was filtered out of the list but still committed, because `git commit` takes the whole index. The commit is now built in a scratch index started from the commit the run began on, holding only the files that passed the filter, and the push sends that commit. The agent can also write .git/refs/replace and .git/info/grafts, which make git read a commit as a different tree or with different parents. Our git commands now ignore both (GIT_NO_REPLACE_OBJECTS, GIT_GRAFT_FILE=/dev/null). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ooked at On a `go`, the failed run was chosen again from the reply, which names no run, so it fell back to the newest red run. With Lint (newer) and E2E both red, `@browserstack fix e2e` planned against E2E and `go` then fixed Lint; `fix run <id>` lost the named run the same way. The plan now records the run it read (`build=` in its marker). Approving reads that run and only that run: if it can no longer be used (no longer red, not on this pull request's commits) the reply says so, and a plan that didn't record a run is refused. Neither ever falls back to another run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… plan showed
The plan comment showed the CLI's Markdown result, while approving saved the
JSON cases file. Both are written in the agent's folder, and nothing checked
they described the same cases, so an agent steered by the pull request's
content could make `go` save other cases, or send them elsewhere, without the
reviewer seeing it.
A generate plan is now shown from the same cases file approving saves,
including where it will be saved ("Will be saved to: ..."), and that file's
sha256 goes in our own plan comment's marker. On `go`, the downloaded file
must have that sha256 before anything is saved; otherwise nothing is saved
and nothing is generated in its place. Agent-written text in the plan is
shown as plain text, so it can't mention anyone or add links.
This also fixes the source-text test the previous commit broke.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A fork's pull request can carry the same commit as a branch here, and its runs show up in this repository's run list. Neither the automatic choice of failed run nor a run named in a comment checked where the run came from, so `explain`, `fix` or `find the bad PR` could read a log a fork's author wrote and act on it. Both now keep only runs whose head repository is this repository (compared without case). A run that doesn't say, or a caller that doesn't say which repository it is, gets nothing: the check fails closed. Naming a fork's run is refused with a reason. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…saved Saving a plan's own cases needs CLI 1.1.16 or later. With an older CLI on the approving run, `tm add --from-file` doesn't exist and the reply blamed Test Management; with a plan made by CLI 1.1.15, the cases file has no destination and the reply told users to set tm_project and tm_folder even when they had, so they could loop. The approving run now checks its CLI has `tm add --from-file` before doing anything, and says it needs 1.1.16 or later (the plan stays approvable). The plan's run records which CLI made it, so a missing destination from an older CLI is named as that; a missing destination from a new enough CLI still points at the pipeline file, and mentions the CLI as the other cause. The step that runs the Action now receives the CLI version line; it was only passed to the sign-in step, so the Action never had it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Each request gets its own reply; automatic runs keep one status comment. - Hand-back pull requests: a draft, based on the pull request asked on, one squashed bot commit, by-products left out, history rewrites refused. - The GitHub App's repository permissions, and why Actions needs read and write. - Saving a generate plan's own cases needs CLI 1.1.16 or later; the pin example now uses a version that exists. - Existing users should copy the starter again, so bot comments stop starting runs. - github-token should be the workflow token or the App's, not a person's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…text, artifact origin) - A `tc tm add` that stopped without a report (killed, timed out, or no output) may have saved some cases. The plan is now used up and the reply says to check Test Management before asking again, so a second `go` can't save them twice. Only the CLI's own "nothing was saved" leaves it approvable. - A failed lookup of the plan's artifacts is reported as a lookup failure (the plan stays approvable), not as "the cases are missing". The error output of `gh run download` and `tm add` goes to the job log with secrets hidden, never to the comment. - The exact-save reply shows titles, ids and errors as plain text, and links a case only when its address is https on browserstack.com. - The cases artifact must come from a run in this repository that started from it (repository_id and head_repository_id). Its commit is not compared: a plan asked for in a comment runs on the default branch's commit; the plan's commit is checked through subject.json and the cases' fingerprint. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…person's Our comments quote "@browserstack go". The bot filters skip them by account type, which works for the workflow token and the App, but a personal access token posts as a person, so our own replies could start runs. The Action now asks GitHub who github-token belongs to (only a person's token can read GET /user) and, if it's a person, skips that account's comments and warns that the workflow token or the App should be used. The account we post as is skipped whatever its type. The README already says github-token should be the workflow token or the App's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Choosing which failed run to read, and cancelling runs when a pull request closes, both listed one page of 100 runs. A busy commit could have its failure past the first page. Both now read every page, up to 1,000 runs. If any page can't be read, the list counts as unreadable, so a partial list can't make "nothing failed" look true. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each of the two App calls could take three 10-second tries plus waits, so a GitHub outage delayed the fallback to the workflow token by 70 to 100 seconds. The sign-in now has one 30-second budget for every call, try and wait: each try's time limit shrinks to what's left, no try starts after the deadline, and no wait runs that would end past it. The retry rules are otherwise unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e commit On a pull_request event the job starts on GitHub's merge of the pull request into its base. If moving to the pull request's own commit failed, the run worked on the merge, the squash was parented on it, and the hand-back pull request (based on the default branch) carried the whole original pull request plus the fix. That case is now refused before anything is squashed or pushed, and the reply says why. Parenting the squash on the pull request's commit instead would not be right: the run's files are the merge's, so it would pull in the base branch's newer changes. A comment run that stayed on the default branch keeps its documented fallback. The README says both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the thorough review, @mdjakkariya. I checked every point against the code before changing anything. All five must-fix items were real and are fixed. Each fix is below with its commit. All 968 tests pass ( Must fix before merge1. Files the agent staged itself were committed. Confirmed:
2.
3.
4. A fork's failed run could be the one read. Confirmed.
5. No check for CLI 1.1.16, and misleading replies with older CLIs. Confirmed.
Before release
Minor / follow-ups
Splitting the PRI'm keeping it as one PR, so the release has one reviewed unit. The fixes above are separate commits, so each can be reviewed on its own. |
|
I've now run the two flows these fixes change, live on our internal pilot repository, using this PR's tip (e718ed0) and Test Companion CLI 1.1.16. Both behaved as intended.
Bot replies: the runs started by the bot's own replies were skipped before doing anything. Ready for another look when you have time. |
Re-review of the fixes (181fa6d..e718ed0): one item still needs fixingThanks for the thorough round. I checked each fix against the code. Verified fixed
Still needed before merge3. A The hash proves the file wasn't swapped after the plan was posted. But the reviewer still doesn't see all of what gets saved:
What this means in practice:
This undercuts the guarantee that "what is shown and what is saved can't differ". Fix, either of:
Optional (fine as follow-ups)
Once item 3 is in, I'll approve. |
The plan showed a case's id, title, priority, preconditions and steps, but `tc tm add --from-file` also saves its description, type, tags, references and custom fields, and for a BDD case its feature, background and scenario. A reviewer could approve text and links they never saw. - The plan now shows every one of those fields, each case in a collapsible block, all as plain text: multi-line text in a code block, references and custom fields unlinked, no mentions. A case already in Test Management is labelled as skipped. - One list of shown fields drives both the plan and a check run just before saving: if any case carries a non-empty field the plan doesn't show (an unknown or newer field, or one of the wrong shape), nothing is saved and the reply says why. The plan warns about it too. - A plan too large for one comment shows the cases that fit, says it can't be approved to save exactly as shown, and records that in its marker, so approving it is refused and asks for fewer cases. The run's own report above the cases is kept short so the cases get most of the room. - A generate plan that had to fall back to the agent's text records that too, and approving it says the plan couldn't be checked against its cases file, rather than blaming an older version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`git status` reports a staged rename as `R old -> new`, and only the new path was kept. The commit is built from the base, so after `git mv a b` the pushed commit had both a and b. The old path of a rename is now included, so it is recorded as deleted. A copy (C) still keeps its source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y then When `tc tm add` stopped without a report, the reply said cases "may already be saved" even when it had failed before saving anything, such as a sign-in, connection or folder lookup error. The CLI prints its report as soon as it has started creating cases, so a run that exited by itself with an error and no report failed before saving, and one that never started saved nothing. Those now say nothing was saved and leave the plan approvable. A kill, a timeout, a crash, or a clean exit with no report may have stopped mid-save: those still use the plan up and say to check Test Management first, so a second `go` can't save twice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…re the run The agent can write .git. Git checks the hash of every object it reads, so a forged loose or packed object is refused by git itself. But some files are trusted without that check, and the base the Action builds on is read after the agent ran: - Before the agent runs, the Action now lists the base's whole tree and reads .git/objects/info/alternates and http-alternates, keeping both in its own memory, which the agent can't reach. - Before pushing a saved commit or a hand-back pull request, it refuses if the alternates files changed, and compares the new commit's whole tree with that listing: every path that differs must be one the Action chose (for a saved commit) and one it would ever commit. - Its git calls no longer use the commit-graph or multi-pack-index files, which git reads without checking the objects (core.commitGraph=false, core.multiPackIndex=false). Variables that point git at other objects were already never passed on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks, @mdjakkariya. Item 3 is in, and so are all five optional items. I checked each one against the Action's code and the CLI's source ( 3. The plan now shows everything
|
mdjakkariya
left a comment
There was a problem hiding this comment.
Re-verified at f2cb285. All findings from my earlier review are addressed.
- 3. The generate plan shows every saved field.
- The plan's field list (
CASE_FIELDS,src/approved-cases.mjs:96-126) matches every field the CLI'sresultCaseToTMsends:- title, description, preconditions, priority, type, tags, references and customFields;
- format, feature, background and bddScenario for BDD cases;
- steps, each with its step and result.
- Right before
tm add,saveExactlychecks the sha256 (src/main.mjs:1841), then refuses any non-empty field the plan didn't show, or a field of an unexpected type (:1849-1850). - Plans marked
shown=partialorshown=markdownare refused. - Shown text is escaped: plain text, fences longer than any backticks inside, and summaries escaped for HTML with @-mentions neutralised.
- The plan's field list (
- Optional items:
- A staged rename now records its old path as deleted.
- A save is reported as having saved nothing only when that is certain.
- Already-saved cases are labelled as such.
- A plan that fell back to the agent's text gets its own message.
- commit-graph and multi-pack-index are off, and the alternates files and the base-tree listing are checked before any push.
node --testgives 989 pass, 0 fail, and CI is green.
LGTM.
Fixes from running every Test Companion job as the BrowserStack GitHub App on a pilot repository.
The App and its own comments
What gets committed
generate,rca,verdict,validate) never commit to the branch, including when the pipeline file picks the job for an event.browserstack.err,.browserstack*, logs,.env, report folders) are never committed. A handed-backfixorautomatebranch is always pushed as one squashed commit on the base, under our name, so the agent's own commits never reach GitHub; by-products are left out of it. Only.github/files and symlinks still refuse a branch.:!xcan't widengit addorresetto other files.Pull requests handed back
git resetpast that commit) is refused, so a fix can never hand back a diff that undoes the pull request's own work.Runs that break
go, and its check completes.Replies and approvals
@browserstackcomment, or ago) gets its own reply with that run's report. Results no longer overwrite each other in one shared comment. Automatic runs, which answer no request, keep one status comment.goapproves the newest plan waiting. One approval settles every plan of that job on that commit that existed at the time, so neither a secondgonor the nightly sweep acts on them; a plan asked for afterwards stays approvable.go #<id>on a settled plan says whether it was approved, dropped, or replaced by another approved plan.Which failure is read
explain,fixandfind the bad PRsay which failed run they read, and list any other failures. A comment can name the run: its link,run <id>, or the workflow's name (@browserstack explain e2e; matrix suffixes like(linux)are ignored, and names made only of everyday words such asbuildorCIare asked for by run id). A named run must be in this repository, on one of the pull request's commits, and red. The "also failed" hint asks by run id. A failed GitHub lookup is reported as such, never as "nothing failed".If nothing has failed yet but CI is still running, the reply says so and lists what is running, instead of "Nothing has failed".
explainandfind the bad PRsay they're read-only, instead of showing the preview banner.The report's Asked by row reads
@who | #<comment id>, linking to the comment that asked for the run.Approving a
generateplan saves exactly what was approvedgenerateplan keeps the exact cases it listed as an artifact on its run (test-companion-cases, kept 7 days), and the plan records which run made it.gosaves that file withtc tm add --from-file: no AI runs, so the cases reaching Test Management are the ones the reviewer read, in the project and folder the plan recorded. The reply lists each created case id. Before,gogenerated the cases again and saved a different set.gowith edits still generates again (edits need the AI), and the reply says so.goon agenerateplan saves nothing and says why.Tests: all 989 pass. The new ones call the real functions, including against real git repositories, and the starter's comment filter is evaluated rather than matched as text.
🤖 Generated with Claude Code