Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(docs): re-write test gen doc flow with pytest plugin #801

Open
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

danceratopz
Copy link
Member

@danceratopz danceratopz commented Sep 16, 2024

🗒️ Description

This is a rewrite of the automatic test case doc generation as a pytest plugin (as apposed to a standalone script). A preview is available at https://ethereum.github.io/execution-spec-tests/pr-801-preview.

Doc Improvements

  1. Test module pages: Added a table that provides and overview of its test functions, e.g., tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1add.
  2. Added test function pages which include a table which tries to give an overview of its parametrized test cases, e.g., tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1add::test_valid.
  3. Added static html files for the tables of the function parameters overview, as they often don't play well within mkdocs styling.
  4. Markdown pages: Added a header with a path and ref link for the markdown file, e.g., tests/prague/eip7692_eof_v1/tracker.md

Technical Improvements

  1. Doc build fails if no function docstring is available or the test type (e.g. state_test) can not be determined (both used in the function overview tables on test module pages).
  2. Using a pytest plugin allows us to lever all the usual pytest flags for test item filtering and fork specification (great for debugging) and to introspect test items to easily grab test metadata.
  3. jinja2 is used for templating, which helps separate the content from its formatting. Jinja is a very powerful templating library which helps save a lot of boilerplate code.

🔗 Related Issues

Perhaps solved, plz check whether the styling is acceptable, e.g., test_bls12_g1add/test_valid.html.

Current issues:

Ideas for future improvements:

✅ Checklist

  • All: Set appropriate labels for the changes.
  • All: Considered squashing commits to improve commit history.
  • All: Added an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.
  • Tests: All converted JSON/YML tests from ethereum/tests have been added to converted-ethereum-tests.txt.
  • Tests: A PR with removal of converted JSON/YML tests from ethereum/tests have been opened.
  • Tests: Included the type and version of evm t8n tool used to locally execute test cases: e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
  • Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.

@danceratopz danceratopz added scope:docs Scope: Documentation type:feat type: Feature labels Sep 16, 2024
Copy link
Member

@marioevz marioevz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is absolutely amazing, great PR!
Just some minor comments. Thanks!

Comment on lines +205 to +211
test_types: List[str] = [
"state_test",
"state_test_only",
"blockchain_test",
"eof_test",
"eof_state_test",
]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can import SPEC_TYPES from ethereum_test_specs then do:

Suggested change
test_types: List[str] = [
"state_test",
"state_test_only",
"blockchain_test",
"eof_test",
"eof_state_test",
]
test_types: List[str] = [spec_type.pytest_parameter_name() for spec_type in SPEC_TYPES]

self.source_dir = Path("tests")
self.ref = get_current_commit_hash_or_tag()
self.top_level_nav_entry = "Test Case Reference"
self.skip_params = ["blockchain_test", "state_test", "eof_test", "fork"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
self.skip_params = ["blockchain_test", "state_test", "eof_test", "fork"]
self.skip_params = ["fork"] + [
spec_type.pytest_parameter_name() for spec_type in SPEC_TYPES
]

logger.warning(f"No docstring available for `{test_function_name}`.")
return f"[📖🐛 No docstring available]({github_issue_url})"
docstring = docstring.strip()
lines = docstring.splitlines()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realized on 7702 tests that I copy-pasted the docstrings of some of the tests.

I think this would be a good time to check for duplicate docstrings in different test functions:

Suggested change
lines = docstring.splitlines()
test_function_id = get_test_function_id(item)
if (
docstring in docstring_test_function_history
and docstring_test_function_history[docstring] != test_function_id
):
logger.warning(
f"Duplicate docstring for {test_function_id}: "
f"{docstring_test_function_history[docstring]} and {test_function_id}"
)
else:
docstring_test_function_history[docstring] = test_function_id
lines = docstring.splitlines()

docstring_test_function_history in this case is a simple global dictionary to keep track of the docstring texts of all test functions visited so far.

pytest_node_id: str
package_name: str

def get_template(self) -> str:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
def get_template(self) -> str:
@property
@abstractmethod
def template(self) -> str:
pass

And then define the attribute in each subclass:

@dataclass
class FunctionPageProps(PagePropsBase):
  ...
  template: str = "function.md.j2"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
scope:docs Scope: Documentation type:feat type: Feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants