Create example for usage of Symbolic Macros (#518)

This adds a new directory containing a set of interrelated symbolic
macros that builds a narrative in inline comments. It's intended as
reference material to be linked to from the official Bazel docs. It
explains core symbolic macro features, including key aspects of the
visibility system, and has a brief example of a finalizer.

Work toward bazelbuild/bazel#23857.

README.md

@@ -66,6 +66,10 @@ Note that tutorials for other languages may be found under other repositories:
  * [Using mandatory providers to ensure your dependencies are of the right type](/rules/mandatory_provider)
  * [Using runfiles](/rules/runfiles)
 
+### [Macro writing]
+
+* [Using Symbolic Macros](/macros)
+
 
 CI:
 [![Build status](https://badge.buildkite.com/260bbace6a4067a3c60539a31fed1191d341a24cb0bfeb0e23.svg)](https://buildkite.com/bazel/bazel-bazel-examples)

macros/.bazelversion

@@ -0,0 +1 @@
+8.0.0

macros/README.md

@@ -0,0 +1,13 @@
+# Macros
+
+This workspace demonstrates the features of
+[Symbolic Macros](https://bazel.build/extending/macros), new to Bazel 8.0.
+
+It's recommended to start by reading the `//main` package's BUILD file, then
+`//word_counter`, `//fancy_word_counter`, and `//letter_metrics`. Each builds
+on the features of the previous, and the `//main` package ties them all
+together.
+
+To avoid additional dependencies, this example uses Python scripts as tools
+without encapsulating them in a `py_binary`. A `python3` interpreter must be
+available in the genrule environment to build the example.

macros/WORKSPACE

@@ -0,0 +1 @@
+# Marker that this folder is the root of the Bazel workspace

macros/data/BUILD

@@ -0,0 +1,9 @@
+exports_files(
+    srcs = [
+        "hamlet.txt",
+        "macbeth.txt",
+    ],
+    # Visible to the //main package, but consumed by targets created in macros
+    # that themselves have no intrinsic permission to view these files.
+    visibility = ["//main:__pkg__"],
+)

macros/data/hamlet.txt

@@ -0,0 +1,33 @@
+To be, or not to be, that is the question,
+Whether 'tis nobler in the mind to suffer
+The slings and arrows of outrageous fortune,
+Or to take arms against a sea of troubles,
+And by opposing end them? To die: to sleep;
+No more; and by a sleep to say we end
+The heart-ache and the thousand natural shocks
+That flesh is heir to, 'tis a consummation
+Devoutly to be wish'd. To die, to sleep;
+To sleep: perchance to dream: ay, there's the rub;
+For in that sleep of death what dreams may come
+When we have shuffled off this mortal coil,
+Must give us pause: there's the respect
+That makes calamity of so long life;
+For who would bear the whips and scorns of time,
+The oppressor's wrong, the proud man's contumely,
+The pangs of despised love, the law's delay,
+The insolence of office and the spurns
+That patient merit of the unworthy takes,
+When he himself might his quietus make
+With a bare bodkin? who would fardels bear,
+To grunt and sweat under a weary life,
+But that the dread of something after death,
+The undiscover'd country from whose bourn
+No traveller returns, puzzles the will
+And makes us rather bear those ills we have
+Than fly to others that we know not of?
+Thus conscience does make cowards of us all;
+And thus the native hue of resolution
+Is sicklied o'er with the pale cast of thought,
+And enterprises of great pith and moment
+With this regard their currents turn awry,
+And lose the name of action.

macros/data/macbeth.txt

@@ -0,0 +1,34 @@
+Double, double toil and trouble;
+Fire burn, and cauldron bubble.
+
+Fillet of a fenny snake,
+In the cauldron boil and bake;
+Eye of newt and toe of frog,
+Wool of bat and tongue of dog,
+Adder's fork and blind-worm's sting,
+Lizard's leg and owlet's wing,
+For a charm of powerful trouble,
+Like a hell-broth boil and bubble.
+
+Double, double toil and trouble;
+Fire burn and caldron bubble.
+
+Scale of dragon, tooth of wolf,
+Witches' mummy, maw and gulf
+Of the ravin'd salt-sea shark,
+Root of hemlock digg'd i' the dark,
+Liver of blaspheming Jew,
+Gall of goat, and slips of yew
+Silver'd in the moon's eclipse,
+Nose of Turk and Tartar's lips,
+Finger of birth-strangled babe
+Ditch-deliver'd by a drab,
+Make the gruel thick and slab:
+Add thereto a tiger's chaudron,
+For the ingredients of our cauldron.
+
+Double, double toil and trouble;
+Fire burn and cauldron bubble.
+
+Cool it with a baboon's blood,
+Then the charm is firm and good.

macros/fancy_word_counter/defs.bzl

@@ -0,0 +1,70 @@
+"""Example definition of a macro that wraps a submacro."""
+
+load("//word_counter:defs.bzl", "count_words")
+
+def _impl(name, visibility, srcs, out, make_it_loud, _shout_tool, **kwargs):
+    # If shouting is requested, make a genrule to create an intermediary file
+    # holding a capitalized version of the text, and feed that into the
+    # submacro in place of the original file.
+
+    # It's important that make_it_loud is non-configurable. Otherwise it would
+    # be a select() value that always evaluate to boolean True (#14506).
+    if make_it_loud:
+        native.genrule(
+            name = name + "_gen_shouted",
+            # No visibility passed in, so this genrule and the output file it
+            # creates are private to (macros defined in) this package.
+            # (We could've also written `visibility = ["//visibility:private"]`
+            # to be explicit.)
+            outs = [name + "_shouted"],
+            tools = [_shout_tool],
+            srcs = srcs,
+            cmd = "$(location %s) $(SRCS) > $@" % str(_shout_tool),
+        )
+
+    count_words(
+        name = name,
+        visibility = visibility,
+        # If we're forwarding our internal <name>_shouted target, count_words
+        # is able to instantiate dependencies on it because we passed the label
+        # in here as an attribute, even though package //word_counter is not in
+        # <name>_shouted's visibility.
+        srcs = [name + "_shouted"] if make_it_loud else srcs,
+        out = out,
+        # It's good practice to accept **kwargs and thread it through to the
+        # rule or submacro being wrapped. This helps mitigate friction when the
+        # schema of the wrapped object changes in ways that don't otherwise
+        # interact with the wrapping macro's logic.
+        #
+        # (`out` is an example of a parameter that could've been omitted from
+        # the implementation function and passed via **kwargs. We left it in
+        # for better readability.)
+        **kwargs
+    )
+
+fancy_count_words = macro(
+    implementation = _impl,
+    attrs = {
+        "srcs": attr.label_list(mandatory=True),
+        "out": attr.output(mandatory=True),
+        # `configurable=False` means it can't be set to a select() by the user,
+        # and won't be promoted to a select() when passed to the implementation
+        # function. Morally, this should be used whenever the value is consumed
+        # by the macro logic itself as opposed to being passed through to the
+        # underlying targets.
+        "make_it_loud": attr.bool(default=False, configurable=False),
+        "_shout_tool": attr.label(default="//shout:shout.py", configurable=False),
+    },
+    doc = """
+    Wraps the count_words macro, adding a shouting (capitalization) feature.
+    """,
+)
+
+# A word about select() promotion: An initial version of this macro had it
+# accepting a "src" attr.label() attribute, and instantiating count_words with
+# `srcs = [src]`. This failed because src was promoted to a select(), which is
+# not permitted as a list element. Had this been a legacy macro, we would not
+# discover this problem until a client attempted to pass in a select() for src.
+# Thanks to select() promotion, we found the problem immediately and could
+# address it by changing the schema to be a attr.label_list(). We could have
+# also kept it an attr.label() and set configurable=False.

macros/letter_metrics/BUILD.bazel

@@ -0,0 +1,5 @@
+# Script tool used by generate_letter_frequencies macro.
+exports_files(
+    srcs = ["aggregate.py"],
+    visibility = ["//visibility:private"],
+)

macros/letter_metrics/aggregate.py

@@ -0,0 +1,24 @@
+#!/usr/bin/env python3
+
+"""
+Script that aggregates frequency tables from each file on the command line and
+dumps out a combined table.
+"""
+
+import sys
+from collections import Counter
+
+
+counts = Counter()
+
+for fn in sys.argv[1:]:
+    with open(fn, 'rt') as input:
+        for line in input.readlines():
+            c, token = line.split()
+            c = int(c)
+            counts[token] += c
+
+by_frequency = sorted(counts.items(), key=lambda e: e[1], reverse=True)
+
+for k, v in by_frequency:
+    print("{:<3} {}".format(v, k))

macros/letter_metrics/defs.bzl

@@ -0,0 +1,63 @@
+"""
+Example definition of a macro that is granted friend visibility privileges to
+interoperate with another macro. Also demonstrates a finalizer that wraps it.
+"""
+
+def _impl(name, visibility, deps, out, _aggregate_tool):
+    native.genrule(
+        name = name,
+        visibility = visibility,
+        outs = [out],
+        tools = [_aggregate_tool],
+        # We generally try to avoid string manipulation on labels to derive
+        # other labels, but sometimes it's necessary. In this case we're
+        # obtaining the internal <name>_letter_freq targets of the given
+        # count_words macro instances. We can see these internal targets
+        # in this macro because our package is specifically granted visibility
+        # by those in the definition of count_words.
+        srcs = [str(d) + "_letter_freq" for d in deps],
+        cmd = "$(location %s) $(SRCS) > $@" % str(_aggregate_tool),
+    )
+
+generate_letter_frequencies = macro(
+    implementation = _impl,
+    attrs = {
+        "deps": attr.label_list(mandatory=True, configurable=False),
+        "out": attr.output(mandatory=True),
+        "_aggregate_tool": attr.label(default=":aggregate.py", configurable=False),
+    },
+    doc = """
+    Emits a file summarizing letter frequency data generated by the count_words
+    macro. Each dep is the label of a main target of a count_words macro
+    instance.
+    """,
+)
+
+
+def _gather_impl(name, visibility, out):
+    suffix = "_letter_freq"
+    deps = []
+    for target_name in native.existing_rules():
+        if target_name.endswith(suffix):
+            # Infer count_words macro name from matched target name.
+            deps.append(":" + target_name[:-len(suffix)])
+
+    generate_letter_frequencies(
+        name = name,
+        visibility = visibility,
+        deps = deps,
+        out = out,
+    )
+
+gather_all_letter_frequencies = macro(
+    implementation = _gather_impl,
+    attrs = {
+        "out": attr.output(mandatory=True),
+    },
+    finalizer = True,
+    doc = """
+    Convenience finalizer form of generate_letter_frequencies, that
+    automatically infers deps by grabbing targets named *_letter_freq
+    from the current package.
+    """
+)

macros/main/BUILD.bazel

@@ -0,0 +1,85 @@
+load("//word_counter:defs.bzl", "count_words")
+load("//fancy_word_counter:defs.bzl", "fancy_count_words")
+load("//letter_metrics:defs.bzl", "gather_all_letter_frequencies", "generate_letter_frequencies")
+
+# A simple symbolic macro.
+count_words(
+    name = "hamlet",
+    # This package has permission to see hamlet.txt, but the macro definition
+    # itself does not. Nonetheless, the build passes because we delegated our
+    # permission to the macro by passing hamlet.txt in a label attribute.
+    srcs = ["//data:hamlet.txt"],
+    # The output target, like any target created by a symbolic macro, is
+    # subject to a naming convention that requires it to suffix the macro name.
+    # If needed, we could declare an `alias` target in this file to make it
+    # available under an additional, unrestricted name. To avoid the
+    # possibility of the user specifying an invalid name here, we could have
+    # also eliminated the `out` attribute and just had the macro body create a
+    # target of form "<name>_word_freq".
+    out = "hamlet_word_freq",
+    # All symbolic macro targets take a visibility attribute. Any exported
+    # target (like :count.txt) created by the macro is visible to at least the
+    # contents of this attribute, and also to our own package. In this case,
+    # :count.txt is visible to //whatever and //main.
+    visibility = ["//whatever:__pkg__"],
+)
+
+# A more complex macro.
+fancy_count_words(
+    name = "loud_macbeth",
+    srcs = ["//data:macbeth.txt"],
+    out = "loud_macbeth_word_freq",
+    make_it_loud = True,
+)
+
+# A macro that processes targets produced internally by count_words, even
+# though those targets are not visible to the current package.
+generate_letter_frequencies(
+    name = "aggregate",
+    out = "aggregate_stats",
+    deps = [
+        ":hamlet",
+        ":loud_macbeth",
+    ],
+)
+
+# Same as above, but the deps are computed automatically using a finalizer.
+# (This call can appear anywhere in the BUILD file, not just at the end.)
+gather_all_letter_frequencies(
+    name = "auto_aggregate",
+    out = "auto_aggregate_stats",
+)
+
+# Examples of build failures:
+
+# hamlet_letter_freq is an internal target of the count_words macro, so it's
+# not visible to //main even though it lives in this package.
+#
+# alias(
+#     name = "cant_touch_this",
+#     actual = ":hamlet_letter_freq",
+# )
+
+# Calling a symbolic macro with the wrong attribute type is an error, just like
+# for a rule. (For legacy macros the error might not fail-fast.)
+#
+# count_words(
+#     name = "bad_count",
+#     srcs = ["//data:hamlet.txt"],
+#     out = 123,
+# )
+
+# Some attributes are non-configurable and trigger an error if you attempt to
+# pass in a select().
+#
+# config_setting(
+#     name = "config",
+#     values = {"cpu": "k8"},
+# )
+#
+# fancy_count_words(
+#     name = "bad_fancy_count",
+#     srcs = ["//data:macbeth.txt"],
+#     out = "bad_fancy_count.txt",
+#     make_it_loud = select({":config": True, "//conditions:default": False}),
+# )

macros/shout/BUILD.bazel

@@ -0,0 +1,7 @@
+# Declare the script file used by the fancy_count_words macro as a tool.
+exports_files(
+    srcs = ["shout.py"],
+    # But don't expose it to the world, just to fancy_count_word's package.
+    # (This BUILD file, and macros defined in this package, can still see it.)
+    visibility = ["//fancy_word_counter:__pkg__"],
+)

macros/shout/shout.py

@@ -0,0 +1,13 @@
+#!/usr/bin/env python3
+
+"""
+Script that echos the content of the files named by the command line, very
+loudly.
+"""
+
+import sys
+
+for filename in sys.argv[1:]:
+    with open(filename, 'rt') as input:
+        for line in input.readlines():
+            print(line.upper(), end='')

macros/word_counter/BUILD.bazel

@@ -0,0 +1,7 @@
+# Declare the script file used internally by the count_words macro as a tool.
+exports_files(
+    srcs = ["counter.py"],
+    # But don't expose it to the world. Private visibility still means visible
+    # to (macros defined in) this package.
+    visibility = ["//visibility:private"],
+)