Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bot] Add prdoc generation #5331

Merged
merged 4 commits into from
Aug 13, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/commands-readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ The current available command actions are:

- [Command FMT](https://github.com/paritytech/polkadot-sdk/actions/workflows/command-fmt.yml)
- [Command Update UI](https://github.com/paritytech/polkadot-sdk/actions/workflows/command-update-ui.yml)
- [Command Prdoc](https://github.com/paritytech/polkadot-sdk/actions/workflows/command-prdoc.yml)
- [Command Sync](https://github.com/paritytech/polkadot-sdk/actions/workflows/command-sync.yml)
- [Command Bench](https://github.com/paritytech/polkadot-sdk/actions/workflows/command-bench.yml)
- [Command Bench All](https://github.com/paritytech/polkadot-sdk/actions/workflows/command-bench-all.yml)
Expand Down Expand Up @@ -235,6 +236,16 @@ You can use the following [`gh cli`](https://cli.github.com/) inside the repo:
gh workflow run command-bench-overheard.yml -f pr=1000 -f benchmark=substrate -f runtime=rococo -f target_dir=substrate
```

### PrDoc

Generate a PrDoc with the crates populated by all modified crates.

Options:
- `pr`: The PR number to generate the PrDoc for.
- `audience`: The audience of whom the changes may concern.
- `bump`: A default bump level for all crates. The PrDoc will likely need to be edited to reflect the actual changes after generation.
- `overwrite`: Whether to overwrite any existing PrDoc.

### Sync

Run sync and commit back results to PR.
Expand Down
110 changes: 110 additions & 0 deletions .github/scripts/generate-prdoc.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
"""
ggwpez marked this conversation as resolved.
Show resolved Hide resolved
Generate the PrDoc for a Pull Request with a specific number, audience and bump level.

It downloads and parses the patch from the GitHub API to opulate the prdoc with all modified crates.
This will delete any prdoc that already exists for the PR if `--force` is passed.

Usage:
python generate-prdoc.py --pr 1234 --audience "TODO" --bump "TODO"
"""

import argparse
import os
import re
import sys
import subprocess
import toml
import yaml
import requests

from github import Github
import whatthepatch
from cargo_workspace import Workspace

# Download the patch and pass the info into `create_prdoc`.
def from_pr_number(n, audience, bump, force):
print(f"Fetching PR '{n}' from GitHub")
g = Github()

repo = g.get_repo("paritytech/polkadot-sdk")
pr = repo.get_pull(n)

patch_url = pr.patch_url
patch = requests.get(patch_url).text

create_prdoc(n, audience, pr.title, pr.body, patch, bump, force)

def create_prdoc(pr, audience, title, description, patch, bump, force):
path = f"prdoc/pr_{pr}.prdoc"

if os.path.exists(path):
if force == True:
print(f"Overwriting existing PrDoc for PR {pr}")
else:
print(f"PrDoc already exists for PR {pr}. Use --force to overwrite.")
sys.exit(1)
else:
print(f"No preexisting PrDoc for PR {pr}")

prdoc = { "doc": [{}], "crates": [] }

prdoc["title"] = title
prdoc["doc"][0]["audience"] = audience
prdoc["doc"][0]["description"] = description

workspace = Workspace.from_path(".")

modified_paths = []
for diff in whatthepatch.parse_patch(patch):
modified_paths.append(diff.header.new_path)

modified_crates = {}
for p in modified_paths:
# Go up until we find a Cargo.toml
p = os.path.join(workspace.path, p)
while not os.path.exists(os.path.join(p, "Cargo.toml")):
p = os.path.dirname(p)

with open(os.path.join(p, "Cargo.toml")) as f:
manifest = toml.load(f)

if not "package" in manifest:
print(f"File was not in any crate: {p}")
continue

crate_name = manifest["package"]["name"]
if workspace.crate_by_name(crate_name).publish:
modified_crates[crate_name] = True
else:
print(f"Skipping unpublished crate: {crate_name}")

print(f"Modified crates: {modified_crates.keys()}")

for crate_name in modified_crates.keys():
entry = { "name": crate_name }

if bump == 'silent' or bump == 'ignore' or bump == 'no change':
entry["validate"] = False
else:
entry["bump"] = bump

print(f"Adding crate {entry}")
prdoc["crates"].append(entry)

# write the parsed PR documentation back to the file
with open(path, "w") as f:
yaml.dump(prdoc, f)

def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument("--pr", type=int, required=True)
parser.add_argument("--audience", type=str, default="TODO")
parser.add_argument("--bump", type=str, default="TODO")
parser.add_argument("--force", type=str)
return parser.parse_args()

if __name__ == "__main__":
args = parse_args()
force = True if args.force.lower() == "true" else False
print(f"Args: {args}, force: {force}")
from_pr_number(args.pr, args.audience, args.bump, force)
78 changes: 78 additions & 0 deletions .github/workflows/command-prdoc.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
name: Command PrDoc

on:
workflow_dispatch:
inputs:
pr:
type: number
description: Number of the Pull Request
required: true
bump:
type: choice
description: Default bump level for all crates
default: "TODO"
required: true
options:
- "TODO"
- "no change"
- "patch"
- "minor"
- "major"
audience:
type: choice
description: Audience of the PrDoc
default: "TODO"
required: true
options:
- "TODO"
- "Runtime Dev"
- "Runtime User"
- "Node Dev"
- "Node User"
overwrite:
type: choice
description: Overwrite existing PrDoc
default: "true"
required: true
options:
- "true"
- "false"

concurrency:
group: command-prdoc
cancel-in-progress: true

jobs:
cmd-prdoc:
runs-on: ubuntu-latest
timeout-minutes: 20
permissions:
contents: write
pull-requests: write
steps:
- name: Download repo
uses: actions/checkout@v4
- name: Install gh cli
id: gh
uses: ./.github/actions/set-up-gh
with:
pr-number: ${{ inputs.pr }}
GH_TOKEN: ${{ github.token }}
- name: Generate PrDoc
run: |
python3 -m pip install -q cargo-workspace PyGithub whatthepatch pyyaml toml

python3 .github/scripts/generate-prdoc.py --pr "${{ inputs.pr }}" --bump "${{ inputs.bump }}" --audience "${{ inputs.audience }}" --force "${{ inputs.overwrite }}"
Copy link
Contributor

@mordamax mordamax Aug 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

may be you could also alternatively add an option to start it with command?
like here https://github.com/polkadot-fellows/runtimes/pull/406/files#diff-38149785bc1a430631c477eabc9b0970804d2c9433504a67574d0b9d38461ba6R120

  issue_comment: # listen for comments on issues
    types: [created]

Or I will do later for all

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a type set on all inputs to either number or choice so i dont think that there can be any query injection.

But i would also prefer comments, as outsiders can probably not run CI actions. Not sure why the comment approach was deprecated?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it wasn't deprecated, we just tried to move to GHA faster/cheaper, and that was one of the fast and secure ways


- name: Report failure
if: ${{ failure() }}
run: gh pr comment ${{ inputs.pr }} --body "<h2>Command failed ❌</h2> Run by @${{ github.actor }} for <code>${{ github.workflow }}</code> failed. See logs <a href=\"$RUN\">here</a>."
env:
RUN: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
GH_TOKEN: ${{ github.token }}
- name: Push Commit
uses: stefanzweifel/git-auto-commit-action@v5
with:
commit_message: Add PrDoc (auto generated)
branch: ${{ steps.gh.outputs.branch }}
file_pattern: 'prdoc/*.prdoc'
Loading