Rewrite Copilot instructions and move to AGENTS.md#53630
Rewrite Copilot instructions and move to AGENTS.md#53630lbussell wants to merge 6 commits intodotnet:mainfrom
Conversation
| dotnet test | ||
|
|
||
| # Run a single test by name | ||
| dotnet test --filter "FullyQualifiedName~TestMethodName" |
There was a problem hiding this comment.
It's missing the test binary. Is there enough context for it to find the test dlls as well as they build to a slightly different location than redist.
There was a problem hiding this comment.
I would also lean towards providing it the test binary location. The dotnetup copilot instructions use ./.dotnet/dotnet test path/to/project.csproj --filter "FullyQualifiedName~TestName which seemed to work well.
There was a problem hiding this comment.
@nagilson can you point me to the dnup copilot instructions?
There was a problem hiding this comment.
Absolutely! I haven't reviewed it super thoroughly, but I've got it working such that it helps the agent, at least for me. It doesn't follow some of the rules I mentioned this PR should follow, FWIW 😂
SDK (slightly modified version of the existing file in main, I was thinking of syncing it once I QA'd it) https://github.com/nagilson/sdk/blob/f3f4e89b3d9cd7c7d3823d4d45e3b6ca551ce7d3/.github/copilot-instructions.md
There was a problem hiding this comment.
@marcpopMSFT help me understand what the test binary is and when I'd use it. Does dotnet test not do The Right Thing automatically?
For example, if you made some changes to the dotnet CLI, what command would you use to run the dotnet CLI tests?
There was a problem hiding this comment.
I had copilot run one single test and this is what it came up with:
# from the repo root
source ./eng/dogfood.sh
# apparently that changes the working directory and you have to switch back
cd /path/to/dotnet/sdk/repo/
dotnet test test/dotnet.Tests/dotnet.Tests.csproj --filter "FullyQualifiedName~GivenNoSrcTestsFolder"There was a problem hiding this comment.
The SDK Repo contains a bootstrap SDK (.dotnet/dotnet, called Stage 0) — a complete, pre-built SDK downloaded during restore, pinned to a specific version in global.json. This Stage 0 SDK is used to compile the SDK source code. The stage 0 SDK tends to be based off daily builds and is newer than anything we've released, so without it other dotnet binaries will fail to build as they dont have awareness of the existence of say, 11.0 preview 3 as a valid target.
The build output is Stage 2 (artifacts/bin/redist/{Configuration}/dotnet/), which is created by overlaying freshly built SDK components (sdk/, host/, templates/) on top of Stage 0's runtime frameworks (shared/, packs/). This allows testing the changes in the SDK repo as if the SDK were the dotnet.exe. It assumes the code changes wer e minimal enough that copying and overwriting certain files over the daily build will work. (sometimes but rarely false)
dotnet test uses whatever is on the PATH which by default will not work... unless the PATH has the daily builds on it. So we use .dotnet/dotnet (Stage 0) to build and run tests, or source dogfood.sh to test CLI commands against Stage 2.
Sourcing the script will correctly edit the path but I've found sometimes the agent will fail to source the script or it will launch a new terminal / subagent and forget to source the script, so I prefer to give it the full local explicit path.
There was a problem hiding this comment.
So at the moment global.json has 11.0.100-preview.1.26104.118 in it (permalink). After a build, dotnet --list-sdks does correctly list 11.0.100-preview.1.26104.118 [<myrepo>/.dotnet/sdk]. It seems like dotnet is correctly picking up the SDK from the global.json. The test command I listed above appears to work correctly with this configuration.
And the dogfood script seems to work correctly as well. My understanding now is that you would not necessarily want to run the repo tests i.e. dotnet test under the dogfood SDK. You would only use the dogfood SDK only for smoke testing/verifying that you correctly fixed the issue that you are trying to solve. Is that correct?
nagilson
left a comment
There was a problem hiding this comment.
Thanks, I think that it's a smart idea to improve the instructions we have nowadays! Left some questions to ponder.
| dotnet test | ||
|
|
||
| # Run a single test by name | ||
| dotnet test --filter "FullyQualifiedName~TestMethodName" |
There was a problem hiding this comment.
I would also lean towards providing it the test binary location. The dotnetup copilot instructions use ./.dotnet/dotnet test path/to/project.csproj --filter "FullyQualifiedName~TestName which seemed to work well.
| By default (without `-pack`), the build skips crossgen and installers (`SkipUsingCrossgen=true`, `SkipBuildingInstallers=true`) to speed up inner-loop iteration. Build output goes to `artifacts/bin/redist/<Configuration>/dotnet/`. | ||
|
|
||
| Key flags: | ||
| - `-c Release` — Release configuration (default: Debug) |
There was a problem hiding this comment.
Not sure if -c Release is something we need to tell the AI
|
|
||
| ### Using the Built SDK (Dogfooding) | ||
|
|
||
| After building, set up your shell to use the locally-built SDK: |
There was a problem hiding this comment.
Skill authoring suggests writing in 3rd person - should we apply that here? https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices (this is 2nd person)
| - Changes should be minimal to resolve a problem in a clean way. | ||
| - User-visible changes to behavior should be considered carefully before committing. They should always be flagged. | ||
| - Only edit the files that are necessary to address the specific issue. Do not run `dotnet format` or make formatting changes to additional files. | ||
| - Prefer using file-based namespaces for new code. |
There was a problem hiding this comment.
Should we keep this instruction or leave it to the formatting?
There was a problem hiding this comment.
Here are code formatting options roughly in order of most to least token efficient:
- Do nothing - most efficient.
- Auto format on write/save/build and enforce in PR validation (minor impact to LLM input caching if formatting changes the shape of the code substantially).
- Agent instructions for formatting (takes up tokens in the context window).
- Enforce formatting in PR validation without any kind of automation or checks (most time wasted - you have to instruct your agent to fix the problems manually).
I chose option 1 here because the previous instructions made it clear that automated formatting/dotnet format would be destructive.
|
|
||
| ### Building | ||
|
|
||
| ```bash |
There was a problem hiding this comment.
might this confuse the agent since bash is more unixy?
| dotnet test --filter "ClassName=Microsoft.DotNet.Cli.SomeTests" | ||
| ``` | ||
|
|
||
| Test projects live in `test/` with naming conventions `<ProjectName>.Tests` (unit) and `<ProjectName>.IntegrationTests` (integration). |
There was a problem hiding this comment.
I like the straightforwardness of some of these prompts, definitely a nice improvement there.
| dotnet test --filter "ClassName=Microsoft.DotNet.Cli.SomeTests" | ||
| ``` | ||
|
|
||
| Test projects live in `test/` with naming conventions `<ProjectName>.Tests` (unit) and `<ProjectName>.IntegrationTests` (integration). |
There was a problem hiding this comment.
The one exception is the netanalyzer tests reside at https://github.com/dotnet/sdk/tree/main/src/Microsoft.CodeAnalysis.NetAnalyzers/tests
|
|
||
| This puts the built SDK on your PATH so `dotnet build`, `dotnet test`, etc. use your local changes. | ||
|
|
||
| ### Running Tests |
There was a problem hiding this comment.
One use case that I have encountered a number of times recently is simulating the Helix run environment. This is described in https://github.com/dotnet/sdk/blob/main/documentation/project-docs/repro-helix-failure.md. Scenarios in which this is important is to flush out dependencies on test-assets where the available content is limited. Thoughts on if this scenario should be called out?
|
|
||
| By default (without `-pack`), the build skips crossgen and installers (`SkipUsingCrossgen=true`, `SkipBuildingInstallers=true`) to speed up inner-loop iteration. Build output goes to `artifacts/bin/redist/<Configuration>/dotnet/`. | ||
|
|
||
| Key flags: |
There was a problem hiding this comment.
There are some secondary flags that can be important. e.g. BuildSdkDeb, BuildSdkRpm. I am sure there are others. Is it worth noting these or giving hints on how to self discover these?
There was a problem hiding this comment.
How do you discover these? Are they documented somewhere centrally? Or do you just need to inspect some of the msbuild files in the repo?
There was a problem hiding this comment.
I don't know of them being called out in any docs. I always look at the build definition to see what build options/parameters it uses. For the ones I called out, they are specified here - https://github.com/dotnet/sdk/blob/main/eng/pipelines/templates/jobs/sdk-job-matrix.yml#L26
The bulk of
.github/copilot-instructions.mdwas written before most frontier models were released. In my limited experience working with this repo, the existing instructions were mostly a waste of context/tokens and sometimes even reduced the quality of Copilot's output.I have rewritten the instructions based on Write effective instructions | Claude Code documentation.
I am new to this repo, so suggestions/improvements are welcome.
The new instructions file is ~1,200 tokens.