Publish dockers to acr

[chrispsommers]

• Move all docker image storage to Azure Container Registry (ACR).
• Created second set of Git Action .yml scripts to build and publish docker images when doing push/PR to main repo.
• Relocate docker image names/tags macro definitions from dash-pipeline/Makefile to individual dockerfile/DOCKER_XXX_IMG.env files which are imported into the Makefile, allowing you to change image versions without editing the Makefile.
• Document detailed docker workflows

[mhanif]

Hi @chrispsommers, I have started looking at the changes from this PR in this branch publish-dockers-to-acr. Before going through some workflows, I was reading the dash-docker README and have some feedback on it:

1. In the volume mount section: Instead of saying "local", shouldn't we say "host" directory/sub-directory/development environment etc.? It reads a lot better. Especially when your figure refers to it as "Host Filesystem"
2. In the section Building Docker Images, Isn't it true that either creates or pulls the docker image? Currently it says that it creates it. We should clarify.
3. For this line Either you split the command in tow line or add semi-colon (before make p4) to split these two commands
4. In the diagram in workflow III, Shouldn't the "docker registry" after step 7 be "ACR" instead of "Dockerhub"?

Finally, what would you like me to try and help test these changes?

[chrispsommers]

Hi @chrispsommers, I have started looking at the changes from this PR in this branch publish-dockers-to-acr. Before going through some workflows, I was reading the dash-docker README and have some feedback on it:

1. In the volume mount section: Instead of saying "local", shouldn't we say "host" directory/sub-directory/development environment etc.? It reads a lot better. Especially when your figure refers to it as "Host Filesystem"
2. In the section Building Docker Images, Isn't it true that either creates or pulls the docker image? Currently it says that it creates it. We should clarify.
3. For this line Either you split the command in tow line or add semi-colon (before make p4) to split these two commands
4. In the diagram in workflow III, Shouldn't the "docker registry" after step 7 be "ACR" instead of "Dockerhub"?

Finally, what would you like me to try and help test these changes?

@mhanif, thanks for the very detailed review, I will try to address the documentation items soon. As far as what to try:

1. Try the normal build & test process to see if it goes well for you. It will pull all new docker images because they're from a different docker repo. This is kind of a sanity check.
2. If you want to spend more time and try some of the publishing workflows, it will require some coordination between us. The goal would be to make a new (possibly throwaway) docker image, try the publishing workflows and see if it works. For example, just change the tag (even with unchanged contents) and go through the whole process of one or more of the documented workflows. It would help prove it out and also "train" someone else on this process. Let me know what your appetite is and we can work together.

[mhanif]

Hi @chrispsommers, using your branch, I went on to try the dash-pipeline work. Did the "make all" which went successfully. Also, was able to do "make run-switch" and "make run-saithrift-server" successfully. However "make run-all-tests" fails. It fails the first time with the following error:

$ make run-all-tests
# Ensure P4Runtime server is listening
t=5; \
while [ ${t} -ge 1 ]; do \
	if sudo lsof -i:9559 | grep LISTEN >/dev/null; then \
		break; \
	else \
		sleep 1; \
		t=`expr $t - 1`; \
	fi; \
done; \
docker exec -w /tests/libsai/vnet_out simple_switch-mhanif ./vnet_out
lsof: no pwd entry for UID 4321
lsof: no pwd entry for UID 4321
GRPC call SetForwardingPipelineConfig 0.0.0.0:9559 => /etc/dash/dash_pipeline.json, /etc/dash/dash_pipeline_p4rt.txt
GRPC ERROR[7]: Not primary, GRPC call Write::INSERT ERROR:
table_id: 38960243 match { field_id: 1 exact { value: "\000\000<" } } action { action { action_id: 21912829 } }Failed to create Direction Lookup Entry
make: *** [Makefile:254: run-libsai-test] Error 1

I reran the same command and got the following output:

$ make run-all-tests
# Ensure P4Runtime server is listening
t=5; \
while [ ${t} -ge 1 ]; do \
	if sudo lsof -i:9559 | grep LISTEN >/dev/null; then \
		break; \
	else \
		sleep 1; \
		t=`expr $t - 1`; \
	fi; \
done; \
docker exec -w /tests/libsai/vnet_out simple_switch-mhanif ./vnet_out
lsof: no pwd entry for UID 4321
lsof: no pwd entry for UID 4321
GRPC call SetForwardingPipelineConfig 0.0.0.0:9559 => /etc/dash/dash_pipeline.json, /etc/dash/dash_pipeline_p4rt.txt
GRPC call Write::INSERT OKtable_id: 38960243 match { field_id: 1 exact { value: "\000\000<" } } action { action { action_id: 21912829 } }
GRPC call Write::INSERT OKtable_id: 45323240 match { field_id: 1 exact { value: "\000\001" } } action { action { action_id: 25655048 params { param_id: 1 value: "\000\000\000\000" } } }
GRPC call Write::INSERT OKtable_id: 45323240 match { field_id: 1 exact { value: "\000\002" } } action { action { action_id: 25655048 params { param_id: 1 value: "\000\000\000\000" } } }
GRPC call Write::INSERT OKtable_id: 34579306 match { field_id: 1 exact { value: "\000\003" } } action { action { action_id: 17187022 params { param_id: 1 value: "\000\000\t" } } }
GRPC call Write::INSERT OKtable_id: 47336097 match { field_id: 1 exact { value: "\000\004" } } action { action { action_id: 27167550 params { param_id: 1 value: "\000\000\'\020" } params { param_id: 2 value: "\000\001\206\240" } params { param_id: 3 value: "\000\001\206\240" } params { param_id: 4 value: "\001" } params { param_id: 5 value: "\254\020\003\001" } params { param_id: 6 value: "\000\000\t" } params { param_id: 7 value: "\000\003" } params { param_id: 27 value: "\000\000" } params { param_id: 26 value: "\000\000" } params { param_id: 25 value: "\000\000" } params { param_id: 24 value: "\000\000" } params { param_id: 23 value: "\000\000" } params { param_id: 17 value: "\000\000" } params { param_id: 16 value: "\000\000" } params { param_id: 15 value: "\000\000" } params { param_id: 14 value: "\000\000" } params { param_id: 13 value: "\000\000" } params { param_id: 22 value: "\000\002" } params { param_id: 21 value: "\000\002" } params { param_id: 20 value: "\000\002" } params { param_id: 19 value: "\000\002" } params { param_id: 18 value: "\000\002" } params { param_id: 12 value: "\000\001" } params { param_id: 11 value: "\000\001" } params { param_id: 10 value: "\000\001" } params { param_id: 9 value: "\000\001" } params { param_id: 8 value: "\000\001" } } }
GRPC call Write::INSERT OKtable_id: 38612462 match { field_id: 1 exact { value: "\252\314\314\314\314\314" } } action { action { action_id: 18228884 params { param_id: 1 value: "\000\004" } } }
GRPC call Write::DELETE OKtable_id: 38612462 match { field_id: 1 exact { value: "\252\314\314\314\314\314" } }
GRPC call Write::DELETE OKtable_id: 47336097 match { field_id: 1 exact { value: "\000\004" } } action { action { action_id: 27167550 params { param_id: 1 value: "\000\000\'\020" } params { param_id: 2 value: "\000\001\206\240" } params { param_id: 3 value: "\000\001\206\240" } params { param_id: 4 value: "\001" } params { param_id: 5 value: "\254\020\003\001" } params { param_id: 6 value: "\000\000\t" } params { param_id: 7 value: "\000\003" } params { param_id: 27 value: "\000\000" } params { param_id: 26 value: "\000\000" } params { param_id: 25 value: "\000\000" } params { param_id: 24 value: "\000\000" } params { param_id: 23 value: "\000\000" } params { param_id: 17 value: "\000\000" } params { param_id: 16 value: "\000\000" } params { param_id: 15 value: "\000\000" } params { param_id: 14 value: "\000\000" } params { param_id: 13 value: "\000\000" } params { param_id: 22 value: "\000\002" } params { param_id: 21 value: "\000\002" } params { param_id: 20 value: "\000\002" } params { param_id: 19 value: "\000\002" } params { param_id: 18 value: "\000\002" } params { param_id: 12 value: "\000\001" } params { param_id: 11 value: "\000\001" } params { param_id: 10 value: "\000\001" } params { param_id: 9 value: "\000\001" } params { param_id: 8 value: "\000\001" } } }
GRPC call Write::DELETE OKtable_id: 34579306 match { field_id: 1 exact { value: "\000\003" } } action { action { action_id: 17187022 params { param_id: 1 value: "\000\000\t" } } }
GRPC call Write::DELETE OKtable_id: 45323240 match { field_id: 1 exact { value: "\000\002" } } action { action { action_id: 25655048 params { param_id: 1 value: "\000\000\000\000" } } }
GRPC call Write::DELETE OKtable_id: 45323240 match { field_id: 1 exact { value: "\000\001" } } action { action { action_id: 25655048 params { param_id: 1 value: "\000\000\000\000" } } }
GRPC call Write::DELETE OKtable_id: 38960243 match { field_id: 1 exact { value: "\000\000<" } }
Done.
./disable_veth_ipv6.sh
Disable IPv6 to suppress Linux control plane noise
net.ipv6.conf.veth0.disable_ipv6 = 1
net.ipv6.conf.veth0.autoconf = 0
net.ipv6.conf.veth0.accept_ra = 0
net.ipv6.conf.veth0.accept_ra_pinfo = 0
net.ipv6.conf.veth0.router_solicitations = 0
net.ipv6.conf.veth1.disable_ipv6 = 1
net.ipv6.conf.veth1.autoconf = 0
net.ipv6.conf.veth1.accept_ra = 0
net.ipv6.conf.veth1.accept_ra_pinfo = 0
net.ipv6.conf.veth1.router_solicitations = 0
net.ipv6.conf.veth2.disable_ipv6 = 1
net.ipv6.conf.veth2.autoconf = 0
net.ipv6.conf.veth2.accept_ra = 0
net.ipv6.conf.veth2.accept_ra_pinfo = 0
net.ipv6.conf.veth2.router_solicitations = 0
net.ipv6.conf.veth3.disable_ipv6 = 1
net.ipv6.conf.veth3.autoconf = 0
net.ipv6.conf.veth3.accept_ra = 0
net.ipv6.conf.veth3.accept_ra_pinfo = 0
net.ipv6.conf.veth3.router_solicitations = 0
docker run -u root --network=host --rm -w /tests/ -it  \
-w /tests/saithrift/ptf \
local/dash-saithrift-client:latest \
./run-saithrift-ptftests.sh
/usr/local/lib/python3.8/dist-packages/ptf-0.9.3-py3.8.egg/EGG-INFO/scripts/ptf:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
Using packet manipulation module: ptf.packet_scapy
test_thrift_session.TestSaiThriftSession ... TestSaiThriftSession OK
ok

----------------------------------------------------------------------
Ran 1 test in 0.005s

OK
test_thrift_session.TestSaiThriftSaiHelper ... TestSaiThriftSaiHelper OK
ok

----------------------------------------------------------------------
Ran 1 test in 0.003s

OK
test_saithrift_vnet.TestSaiThrift_outbound_udp_pkt ...

Sending packet with wrong vip...

 <Ether  dst=00:00:02:03:04:05 src=00:00:05:06:06:06 type=IPv4 |<IP  ihl=None tos=0x0 id=1 flags= frag=0 ttl=64 proto=udp src=172.16.1.1 dst=172.16.100.100 |<UDP  sport=11638 dport=4789 chksum=0x0 |<VXLAN  flags=Instance reserved1=0 vni=0x64 reserved2=0x0 |<Ether  dst=02:02:02:02:02:02 src=00:cc:cc:cc:cc:cc type=IPv4 |<IP  ihl=None tos=0x0 id=1 flags= frag=0 ttl=64 proto=udp src=10.1.1.10 dst=10.1.2.50 |<UDP  sport=1234 dport=80 |<Raw  load='\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./0123456789' |>>>>>>>>

Verifying drop...

Sending packet with wrong dst CA IP to verify routing drop...

 <Ether  dst=00:00:02:03:04:05 src=00:00:05:06:06:06 type=IPv4 |<IP  ihl=None tos=0x0 id=1 flags= frag=0 ttl=64 proto=udp src=172.16.1.1 dst=172.16.1.100 |<UDP  sport=11638 dport=4789 chksum=0x0 |<VXLAN  flags=Instance reserved1=0 vni=0x64 reserved2=0x0 |<Ether  dst=02:02:02:02:02:02 src=00:cc:cc:cc:cc:cc type=IPv4 |<IP  ihl=None tos=0x0 id=1 flags= frag=0 ttl=64 proto=udp src=10.1.1.10 dst=10.200.2.50 |<UDP  sport=1234 dport=80 |<Raw  load='\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./0123456789' |>>>>>>>>

Verifying drop...

Sending packet with wrong dst CA IP to verify mapping drop...

 <Ether  dst=00:00:02:03:04:05 src=00:00:05:06:06:06 type=IPv4 |<IP  ihl=None tos=0x0 id=1 flags= frag=0 ttl=64 proto=udp src=172.16.1.1 dst=172.16.1.100 |<UDP  sport=11638 dport=4789 chksum=0x0 |<VXLAN  flags=Instance reserved1=0 vni=0x64 reserved2=0x0 |<Ether  dst=02:02:02:02:02:02 src=00:cc:cc:cc:cc:cc type=IPv4 |<IP  ihl=None tos=0x0 id=1 flags= frag=0 ttl=64 proto=udp src=10.1.1.10 dst=10.1.211.211 |<UDP  sport=1234 dport=80 |<Raw  load='\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./0123456789' |>>>>>>>>

Verifying drop...

Sending outbound packet...

 <Ether  dst=00:00:02:03:04:05 src=00:00:05:06:06:06 type=IPv4 |<IP  ihl=None tos=0x0 id=1 flags= frag=0 ttl=64 proto=udp src=172.16.1.1 dst=172.16.1.100 |<UDP  sport=11638 dport=4789 chksum=0x0 |<VXLAN  flags=Instance reserved1=0 vni=0x64 reserved2=0x0 |<Ether  dst=02:02:02:02:02:02 src=00:cc:cc:cc:cc:cc type=IPv4 |<IP  ihl=None tos=0x0 id=1 flags= frag=0 ttl=64 proto=udp src=10.1.1.10 dst=10.1.2.50 |<UDP  sport=1234 dport=80 |<Raw  load='\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./0123456789' |>>>>>>>>

Verifying packet...
 <Ether  dst=00:00:00:00:00:00 src=00:00:00:00:00:00 type=IPv4 |<IP  ihl=None tos=0x0 id=1 flags= frag=0 ttl=64 proto=udp chksum=0x0 src=172.16.1.100 dst=172.16.1.20 |<UDP  sport=0 dport=4789 chksum=0x0 |<VXLAN  flags= reserved1=0 vni=0x64 reserved2=0x0 |<Ether  dst=00:dd:dd:dd:dd:dd src=00:cc:cc:cc:cc:cc type=IPv4 |<IP  ihl=None tos=0x0 id=1 flags= frag=0 ttl=64 proto=udp src=10.1.1.10 dst=10.1.2.50 |<UDP  sport=1234 dport=80 |<Raw  load='\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./0123456789' |>>>>>>>>
test_sai_thrift_outbound_udp_pkt_test OK
ok

----------------------------------------------------------------------
Ran 1 test in 0.519s

OK
cd ../test/third-party/traffic_gen && ./deploy_ixiac.sh
.
WARNING: The DOCKER_REGISTRY variable is not set. Defaulting to a blank string.
WARNING: The IFC1 variable is not set. Defaulting to a blank string.
WARNING: The IFC2 variable is not set. Defaulting to a blank string.
Pulling controller (/ixia-c-controller:latest)...
ERROR: invalid reference format
make: *** [Makefile:506: deploy-ixiac] Error 1

docker ps and docker images have the following output:

$ docker ps
CONTAINER ID   IMAGE                                             COMMAND                  CREATED         STATUS         PORTS     NAMES
f789e5371199   sonicdash.azurecr.io/dash-saithrift-bldr:220819   "./saiserver"            9 minutes ago   Up 9 minutes             dash-saithrift-server-mhanif
3ca5717876cc   sonicdash.azurecr.io/dash-bmv2-bldr:220819        "env LD_LIBRARY_PATH…"   9 minutes ago   Up 9 minutes             simple_switch-mhanif

$ docker images
REPOSITORY                                        TAG       IMAGE ID       CREATED             SIZE
local/dash-saithrift-client                       latest    940694a2a346   15 minutes ago      520MB
sonicdash.azurecr.io/dash-saithrift-client-bldr   220819    0589d81cc9f8   About an hour ago   493MB
sonicdash.azurecr.io/dash-saithrift-bldr          220819    1b68871f29ba   About an hour ago   1.71GB
sonicdash.azurecr.io/dash-p4c-bmv2                220825    44671aa5a15e   About an hour ago   603MB
sonicdash.azurecr.io/dash-bmv2-bldr               220819    7bb37d7d476c   About an hour ago   992MB
<none>                                            <none>    7493ed3513e8   2 days ago          519MB
<none>                                            <none>    59d7b56cc7ec   2 days ago          519MB
<none>                                            <none>    804b5ce74492   2 days ago          519MB
<none>                                            <none>    e7bfd71f7f7a   3 days ago          519MB
<none>                                            <none>    d36c2fdba14d   5 days ago          519MB
<none>                                            <none>    4af948131dde   10 days ago         519MB
chrissommers/dash-p4c-bmv2                        220825    44e448102b8a   3 weeks ago         645MB
chrissommers/dash-saithrift-bldr                  220819    9e236a7d31e4   4 weeks ago         1.71GB
chrissommers/dash-bmv2-bldr                       220819    61e0fd3e4227   4 weeks ago         991MB
chrissommers/dash-saithrift-client-bldr           220723    f711a148d80d   2 months ago        492MB
nf                                                latest    f8df59d2a6d3   10 months ago       221MB
endpoint                                          latest    92f871c3fc13   10 months ago       223MB
hello-world                                       latest    feb5d9fea6a5   12 months ago       13.3kB
ubuntu                                            16.04     b6f507652425   12 months ago       135MB
ubuntu                                            trusty    13b66b487594   18 months ago       197MB

Shouldn't the new code clean-up older (chrissommers) docker images? Also, not sure what these images are?

[chrispsommers]

Hi @mhanif, thanks for the reporting. I see three things:

1. This error GRPC ERROR[7]: Not primary, GRPC call Write::INSERT ERROR: is our old friend make run-all-tests fails #218 which is still open. It remains to be solved and it seems to occur in your environment more than anywhere else. I saw it once, finally, in a CI pipeline run. I suspect it's a timing issue and suggested an experiment you could try in this comment: make run-all-tests fails #218 (comment)

2. This output seems to indicate missing or bad DASH/test/third-party/traffic_gen/deployment/.env which is weird. Could you check the file on your machine? I've never seen this before.

cd ../test/third-party/traffic_gen && ./deploy_ixiac.sh
.
WARNING: The DOCKER_REGISTRY variable is not set. Defaulting to a blank string.
WARNING: The IFC1 variable is not set. Defaulting to a blank string.
WARNING: The IFC2 variable is not set. Defaulting to a blank string.
Pulling controller (/ixia-c-controller:latest)...
ERROR: invalid reference format

The contents should be:

DOCKER_REGISTRY=ghcr.io/open-traffic-generator
CONTROLLER_VERSION=0.0.1-3383
TRAFFIC_ENGINE_VERSION=1.6.0.17
IFC1=veth1
IFC2=veth3
TCP_PORT_IFC1=5555
TCP_PORT_IFC2=5556
CPU_CORES_IFC1="0"
CPU_CORES_IFC2="1"
OPT_ENABLE_IPv6="No"

3. Shouldn't the new code clean-up older (chrissommers) docker images? Also, not sure what these images are?

These are simply the older versions of docker images used prior to this PR which used my docker account. The make scripts do not clean anything up, this is the responsibility of the developer to manage their local hosts' Docker registries. The scripts don't delete on their own, nor should they; that might actually cause lots of hardships for people who are switching back and forth or developing in multiple branches. In fact, if you were to go back to testing with main branch you'd need them again.

Deleting a few old images is easy, for example docker rmi 44e448102b8a or docker rmi chrissommers/dash-p4c-bmv2:220825 to delete one of the ones in your list. I'd hold off until this PR is merged though.

[chrispsommers]

@mhanif Hi, here are point-by-point responses to #225 (comment)

1. In the volume mount section: Instead of saying "local", shouldn't we say "host" directory/sub-directory/development environment etc.? It reads a lot better. Especially when your figure refers to it as "Host Filesystem"
   [cs] Agreed. Fixed - "local" replaced with "host" in many places. Thanks!

2. In the section Building Docker Images, Isn't it true that either creates or pulls the docker image? Currently it says that it creates it. We should clarify.
   [cs] No, the language is correct. All the make docker-XXX only build images. The docker-pull-xxx targets pull images on demand, only used in CI pipelines to provide granular testing. (Docker pulls are also done implicitly via any make targets invoking docker run but only if the image isn't already stored in the host's registry.)

3. For this line Either you split the command in tow line or add semi-colon (before make p4) to split these two commands
   [cs] No, the command is correct; the intent is to temporarily set an environment variable in the context of the make command in the same line, that was the point.

4. In the diagram in workflow III, Shouldn't the "docker registry" after step 7 be "ACR" instead of "Dockerhub"?
   [cs] Agreed, good catch! Fixed.

[mhanif]

2. This output seems to indicate missing or bad DASH/test/third-party/traffic_gen/deployment/.env which is weird. Could you check the file on your machine? I've never seen this before.

Hi @chrispsommers, here is the requested output:

$ more .env
DOCKER_REGISTRY=ghcr.io/open-traffic-generator
CONTROLLER_VERSION=0.0.1-2934
TRAFFIC_ENGINE_VERSION=1.4.1.26
IFC1=veth1
IFC2=veth3
TCP_PORT_IFC1=5555
TCP_PORT_IFC2=5556
CPU_CORES_IFC1="0"
CPU_CORES_IFC2="1"

[chrispsommers]

2. This output seems to indicate missing or bad DASH/test/third-party/traffic_gen/deployment/.env which is weird. Could you check the file on your machine? I've never seen this before.

Hi @chrispsommers, here is the requested output:

$ more .env
DOCKER_REGISTRY=ghcr.io/open-traffic-generator
CONTROLLER_VERSION=0.0.1-2934
TRAFFIC_ENGINE_VERSION=1.4.1.26
IFC1=veth1
IFC2=veth3
TCP_PORT_IFC1=5555
TCP_PORT_IFC2=5556
CPU_CORES_IFC1="0"
CPU_CORES_IFC2="1"

Thanks, I've no explanation for this behavior on your machine. Does it happen repeatably?

[mhanif]

Thanks, I've no explanation for this behavior on your machine. Does it happen repeatably?

I was not able to recreate this error. However, I frequently see GRPC ERROR[7]: Not primary, GRPC call Write::INSERT ERROR: Upon rerunning, I occasionally get the make run-all-tests to pass now. If you want me to try the publishing workflow, let me know how we can coordinate the work. Thanks!

[chrispsommers]

I was not able to recreate this error. However, I frequently see GRPC ERROR[7]: Not primary, GRPC call Write::INSERT ERROR: Upon rerunning, I occasionally get the make run-all-tests to pass now. If you want me to try the publishing workflow, let me know how we can coordinate the work. Thanks!

I'll try to explore this error soon, it puzzles me and impedes your efforts.

As far as an ACR workflow, which one would you like to try? I was thinking #2 or #3. I made a scratch branch acr-publish-test-branch in the main repo and you can use it to make PRs against. You can work at your own pace and ping me as much as needed.

[chrispsommers]

Merging per WG meeting on 10-5-2022