Doc rewrite, phase 1

Author: pcj

README.md

@@ -0,0 +1,9 @@
+
+- fix the issues related to `bazel mod tidy`
+- consider add better starlark example
+- consider nogo
+- test usage in bazel-aquery-differ
+- test usage in unity
+- check every file
+- fix manual tests, ideally
+- update documentation

TODO.md

@@ -1,18 +1,16 @@
 # cmd/gazelle
 
-This is essentially a copy of the files in bazelbuild/bazel-gazelle/cmd/gazelle.
+This is essentially a copy of the files in bazel-contrib/bazel-gazelle/cmd/gazelle.
 
 To upgrade gazelle, one must:
 
 - Compare changes in the source repo@version to the files here.  It's easiest to
   just copy over each file and see where the diffs are.  Make sure `langs.go`
-  includes the `github.com/stackb/rules_proto/language/protobuf`.  Internal
-  packages referenced must also be copied over (ugh).  There's probably a more
-  elegant solution to keeping a modified copy of gazelle binary here.
+  includes `github.com/stackb/rules_proto/language/protobuf`.
 - Since the `proto_gazelle.bzl` rule uses
   `@bazel_gazelle//internal:gazelle.bash.in`, changes there must remain
   compatible with proto_gazelle.  Look at the diff there and make sure the
-  proto_gazelle_impl is satifying the needs of that template.
+  `_proto_gazelle_impl` is satifying the needs of that template.
 - Remember that this `cmd/gazelle` must be buildable via the standard go
   toolchain (see proto_repository_tools.bzl). All deps must be in the vendor
   tree.

cmd/gazelle/README.md

@@ -0,0 +1,106 @@
+# API
+
+The go starlark API defines several core interfaces:
+
+## Plugin
+
+```go
+// Plugin implementations are capable of predicting the sources that are
+// generated by a particular protoc plugin.
+type Plugin interface {
+	// Name is the name of the plugin and its associated configuration.
+	Name() string
+	// Configure creates the plugin configuration.  If nil is returned, the
+	// plugin should be skipped for the current package/library.
+	Configure(ctx *PluginContext) *PluginConfiguration
+}
+```
+
+Let's look at the `bufbuild:es` plugin implementation.
+
+Plugins typically register themselves with a global object in an `init()` function:
+
+```go
+func init() {
+	protoc.Plugins().MustRegisterPlugin(&EsProto{})
+}
+```
+
+The `Name()` function returns the NAME that is specified in a
+`gazelle:proto_plugin implementation NAME`.
+
+```go
+// EsProto implements Plugin for the bufbuild/protoc-gen-es plugin.
+type EsProto struct{}
+
+// Name implements part of the Plugin interface.
+func (p *EsProto) Name() string {
+	return "bufbuild:es"
+}
+```
+
+In the following `Configure` function, the main task is to read the
+`proto_library` passed in via the context object, iterate over the `srcs`, and
+assemble the expected `tsFiles` that the actual `connect-es` protoc plugin will
+be expected to produce.  These are used for the `PluginConfiguration.Outputs`:
+
+```go
+// Configure implements part of the Plugin interface.
+func (p *EsProto) Configure(ctx *protoc.PluginContext) *protoc.PluginConfiguration {
+	flags := parseEsProtoOptions(p.Name(), ctx.PluginConfig.GetFlags())
+
+    var options = []string{"keep_empty_files=true", "target=ts"}
+	tsFiles := make([]string, 0)
+	for _, file := range ctx.ProtoLibrary.Files() {
+		// TODO: outputs should be conditional on which target= value is used
+		tsFile := file.Name + "_pb.ts"
+		if flags.excludeOutput[filepath.Base(tsFile)] {
+			continue
+		}
+		if ctx.Rel != "" {
+			tsFile = path.Join(ctx.Rel, tsFile)
+		}
+		tsFiles = append(tsFiles, tsFile)
+	}
+
+	pc := &protoc.PluginConfiguration{
+		Label:   label.New("build_stack_rules_proto", "plugin/bufbuild", "es"),
+		Outputs: protoc.DeduplicateAndSort(tsFiles),
+		Options: protoc.DeduplicateAndSort(options),
+	}
+	if len(pc.Outputs) == 0 {
+		pc.Outputs = nil
+	}
+
+    return pc
+}
+```
+
+This plugin happens to define a custom set of flags:
+
+```go
+// EsProtoOptions represents the parsed flag configuration for the
+// EsProto implementation.
+type EsProtoOptions struct {
+	excludeOutput map[string]bool
+}
+
+func parseEsProtoOptions(kindName string, args []string) *EsProtoOptions {
+	flags := flag.NewFlagSet(kindName, flag.ExitOnError)
+
+	var excludeOutput string
+	flags.StringVar(&excludeOutput, "exclude_output", "", "--exclude_output=foo.ts suppresses the file 'foo.ts' from the output list")
+
+	if err := flags.Parse(args); err != nil {
+		log.Fatalf("failed to parse flags for %q: %v", kindName, err)
+	}
+	config := &EsProtoOptions{
+		excludeOutput: make(map[string]bool),
+	}
+	for _, value := range strings.Split(excludeOutput, ",") {
+		config.excludeOutput[value] = true
+	}
+
+	return config
+}
+```

docs/API.md

@@ -0,0 +1,254 @@
+
+## Build Rules
+
+The core of `stackb/rules_proto` contains two build rules:
+
+| Rule            | Description                                             |
+|-----------------|---------------------------------------------------------|
+| `proto_compile` | Executes the `protoc` tool.                             |
+| `proto_plugin`  | Provides static `protoc` plugin-specific configuration. |
+
+### proto_compile
+
+Example:
+
+```python
+load("@rules_proto//proto:defs.bzl", "proto_library")
+load("@build_stack_rules_proto//rules:proto_compile.bzl", "proto_compile")
+
+proto_library(
+    name = "thing_proto",
+    srcs = ["thing.proto"],
+    deps = ["@com_google_protobuf//:timestamp_proto"],
+)
+
+proto_plugin(name = "cpp")
+
+proto_compile(
+    name = "person_cpp_compile",
+    outputs = [
+        "person.pb.cc",
+        "person.pb.h",
+    ],
+    plugins = [":cpp"],
+    proto = "person_proto",
+)
+```
+
+Takeaways:
+
+- A `proto_library` rule forms the basis for other language-specific derived
+  rules.
+- `proto_library` is provided by
+  [bazelbuild/rules_proto](https://github.com/bazelbuild/rules_proto).
+- A `proto_compile` rule references a single `proto_library` target.
+- The `plugins` attribute is a list of labels to `proto_plugin` targets.
+- The `outputs` attribute names the files that will be generated by the protoc
+  invocation.
+- The
+  [proto](https://github.com/bazelbuild/bazel-gazelle/blob/master/language/proto/lang.go)
+  extension provided by [bazel-gazelle] is responsible for generating
+  `proto_library`.
+
+### proto_plugin
+
+`proto_plugin` primarily provides the plugin tool executable. The example seen
+above is the simplest case where the plugin is builtin to `protoc` itself; no
+separate plugin tool is required. In this case the `proto_plugin` rule
+degenerates into just a `name`.
+
+It is possible to add additional plugin-specific
+`name = "foo", options = ["bar"]` on the `proto_plugin` rule, but the use-case
+for this is narrow. Generally it is preferred to say
+`# gazelle:proto_plugin foo option bar` such that the option can be interpreted
+during a gazelle run.
+
+### proto_compiled_sources
+
+`proto_compiled_sources` is used when you prefer to check the generated files
+into source control. This may be necessary for legacy reasons, during an initial
+Bazel migration, or to support better IDE integration.
+
+The shape of a `proto_compiled_sources` rule is essentially identical to
+`proto_compile` with one exception: generated source are named in the `srcs`
+attribute rather than `outputs`.
+
+For example, a `proto_compiled_sources` named `//example/thing:proto_go_sources`
+is a macro that generates three rules:
+
+1. `bazel build //example/thing:proto_go_sources` emits the generated files.
+2. `bazel run //example/thing:proto_go_sources.update` copies the generated
+   files back into the source package.
+3. `bazel test //example/thing:proto_go_sources_test` asserts the source files
+   are identical to generated files.
+
+In this scenario, `2.` is used to build the generated files (in the `bazel-bin/`
+output tree) and copy the `example/thing/thing.pb.go` back into place where it
+will be committed under source control. `3.` is used to prevent drift: if a
+developer modifies `thing.proto` and neglects to run the `.update` the test will
+fail in CI.
+
+### proto_compile_assets
+
+The macro `proto_compile_assets` aggregates a list of dependencies (which
+provide `ProtoCompileInfo`) into a single runnable target that copies files in
+bulk.
+
+For example, `bazel run //proto:assets` will copy all the generated `.pb.go`
+files back into the source tree:
+
+```py
+load("@build_stack_rules_proto//rules:proto_compile_assets.bzl", "proto_compile_assets")
+
+proto_compile_assets(
+    name = "assets",
+    deps = [,
+      "//proto/api/v1:proto_go_compile",
+      "//proto/api/v2:proto_go_compile",
+      "//proto/api/v3:proto_go_compile",
+    ],
+)
+```
+
+### The `output_mappings` attribute
+
+Consider the following rule within the package `example/thing`:
+
+```python
+proto_compile(
+    name = "thing_go_compile",
+    output_mappings = ["thing.pb.go=github.com/stackb/rules_proto/example/thing/thing.pb.go"],
+    outputs = ["thing.pb.go"],
+    plugins = ["@build_stack_rules_proto//plugin/golang/protobuf:protoc-gen-go"],
+    proto = "thing_proto",
+)
+```
+
+This rule is declaring that a file `bazel-bin/example/thing/thing.pb.go` will be
+output when the action is run. When we
+`bazel build //example/thing:thing_go_compile`, the file is indeed created.
+
+Let's temporarily comment out the `output_mappings` attribute and rebuild:
+
+```python
+proto_compile(
+    name = "thing_go_compile",
+    # output_mappings = ["thing.pb.go=github.com/stackb/rules_proto/example/thing/thing.pb.go"],
+    outputs = ["thing.pb.go"],
+    plugins = ["@build_stack_rules_proto//plugin/golang/protobuf:protoc-gen-go"],
+    proto = "thing_proto",
+)
+```
+
+```sh
+$ bazel build //example/thing:thing_go_compile
+ERROR: /github.com/stackb/rules_proto/example/thing/BUILD.bazel:54:14: output 'example/thing/thing.pb.go' was not created
+```
+
+What happened? Let's add a debugging attribute `verbose = True` on the rule:
+this will print debugging information and show the bazel sandbox before and
+after the `protoc` tool is invoked:
+
+```python
+proto_compile(
+    name = "thing_go_compile",
+    # output_mappings = ["thing.pb.go=github.com/stackb/rules_proto/example/thing/thing.pb.go"],
+    outputs = ["thing.pb.go"],
+    plugins = ["@build_stack_rules_proto//plugin/golang/protobuf:protoc-gen-go"],
+    proto = "thing_proto",
+    verbose = True,
+)
+```
+
+```sh
+$ bazel build //example/thing:thing_go_compile
+##### SANDBOX BEFORE RUNNING PROTOC
+./bazel-out/host/bin/external/com_google_protobuf/protoc
+./bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/com_github_golang_protobuf/protoc-gen-go/protoc-gen-go_/protoc-gen-go
+./bazel-out/darwin-fastbuild/bin/example/thing/thing_proto-descriptor-set.proto.bin
+./bazel-out/darwin-fastbuild/bin/external/com_google_protobuf/timestamp_proto-descriptor-set.proto.bin
+
+##### SANDBOX AFTER RUNNING PROTOC
+./bazel-out/darwin-fastbuild/bin/github.com/stackb/rules_proto/example/thing/thing.pb.go
+```
+
+So, the file was created, but not in the location we wanted. In this case the
+`protoc-gen-go` plugin is not "playing nice" with Bazel. Because this
+`thing.proto` has
+`option go_package = "github.com/stackb/rules_proto/example/thing;thing";`, the
+output location is no longer based on the `package`. This is a problem, because
+Bazel semantics disallow declaring a File outside its package boundary. As a
+result, we need to do a
+`mv ./bazel-out/darwin-fastbuild/bin/github.com/stackb/rules_proto/example/thing/thing.pb.go ./bazel-out/darwin-fastbuild/bin/example/thing/thing.pb.go`
+to relocate the file into its expected location before the action terminates.
+
+Therefore, the `output_mappings` attribute is a list of entries that map file
+locations `want=got` relative to the action execution root. It is required when
+the actual output location does not match the desired location. This can occur
+if the proto `package` statement does not match the Bazel package path, or in
+special circumstances specific to the plugin itself (like `go_package`).
+
+
+## proto_gazelle
+
+`proto_gazelle` is not a repository rule: it's just like the typical `gazelle`
+rule, but with extra deps resolution superpowers. But, we discuss it here since
+it works in conjunction with `proto_repository`:
+
+```python
+load("@build_stack_rules_proto//rules:proto_gazelle.bzl", "DEFAULT_LANGUAGES", "proto_gazelle")
+
+proto_gazelle(
+    name = "gazelle",
+    cfgs = ["//proto:config.yaml"],
+    command = "update",
+    gazelle = ":gazelle-protobuf",
+    imports = [
+        "@bazelapis//:imports.csv",
+        "@googleapis//:imports.csv",
+        "@protobufapis//:imports.csv",
+        "@remoteapis//:imports.csv",
+    ],
+)
+```
+
+In this example, we are again setting the base gazelle config using the YAML
+file (the same one used in for the `proto_repository` rules). We are also now
+importing resolve information from four external sources.
+
+With this setup, we can simply place an import statement like
+`import "src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto";`
+in a `foo.proto` file in the default workspace, and gazelle will automagically
+figure out the import dependency tree spanning `@bazelapis`, `@remoteapis`,
+`@googleapis`, and the well-known types from `@protobufapis`.
+
+This works for any `proto_language`, with any set of custom protoc plugins.
+
+## golden_filegroup
+
+`golden_filegroup` is a utility macro for golden file testing. It works like a
+native filegroup, but adds `.update` and `.test` targets. Example:
+
+```py
+load("@build_stack_rules_proto//rules:golden_filegroup.bzl", "golden_filegroup")
+
+# golden_filegroup asserts that generated files named in 'srcs' are
+# identical to the ones checked into source control.
+#
+# Usage:
+#
+# $ bazel build :golden        # not particularly useful, just a regular filegroup
+#
+# $ bazel test  :golden.test   # checks that generated files are identical to
+# ones in git (for CI)
+#
+# $ bazel run   :golden.update # copies the generated files into source tree
+# (then 'git add' to your PR if it looks good)
+golden_filegroup(
+    name = "golden",
+    srcs = [
+        ":some_generated_file1.json",
+        ":some_generated_file2.json",
+    ],
+)
+```