-
Notifications
You must be signed in to change notification settings - Fork 72
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(docs): re-write test gen doc flow with pytest plugin #801
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is absolutely amazing, great PR!
Just some minor comments. Thanks!
test_types: List[str] = [ | ||
"state_test", | ||
"state_test_only", | ||
"blockchain_test", | ||
"eof_test", | ||
"eof_state_test", | ||
] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can import SPEC_TYPES
from ethereum_test_specs
then do:
test_types: List[str] = [ | |
"state_test", | |
"state_test_only", | |
"blockchain_test", | |
"eof_test", | |
"eof_state_test", | |
] | |
test_types: List[str] = [spec_type.pytest_parameter_name() for spec_type in SPEC_TYPES] |
self.source_dir = Path("tests") | ||
self.ref = get_current_commit_hash_or_tag() | ||
self.top_level_nav_entry = "Test Case Reference" | ||
self.skip_params = ["blockchain_test", "state_test", "eof_test", "fork"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
self.skip_params = ["blockchain_test", "state_test", "eof_test", "fork"] | |
self.skip_params = ["fork"] + [ | |
spec_type.pytest_parameter_name() for spec_type in SPEC_TYPES | |
] |
logger.warning(f"No docstring available for `{test_function_name}`.") | ||
return f"[📖🐛 No docstring available]({github_issue_url})" | ||
docstring = docstring.strip() | ||
lines = docstring.splitlines() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I realized on 7702 tests that I copy-pasted the docstrings of some of the tests.
I think this would be a good time to check for duplicate docstrings in different test functions:
lines = docstring.splitlines() | |
test_function_id = get_test_function_id(item) | |
if ( | |
docstring in docstring_test_function_history | |
and docstring_test_function_history[docstring] != test_function_id | |
): | |
logger.warning( | |
f"Duplicate docstring for {test_function_id}: " | |
f"{docstring_test_function_history[docstring]} and {test_function_id}" | |
) | |
else: | |
docstring_test_function_history[docstring] = test_function_id | |
lines = docstring.splitlines() |
docstring_test_function_history
in this case is a simple global dictionary to keep track of the docstring texts of all test functions visited so far.
pytest_node_id: str | ||
package_name: str | ||
|
||
def get_template(self) -> str: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
def get_template(self) -> str: | |
@property | |
@abstractmethod | |
def template(self) -> str: | |
pass |
And then define the attribute in each subclass:
@dataclass
class FunctionPageProps(PagePropsBase):
...
template: str = "function.md.j2"
🗒️ Description
This is a rewrite of the automatic test case doc generation as a pytest plugin (as apposed to a standalone script). A preview is available at https://ethereum.github.io/execution-spec-tests/pr-801-preview.
Doc Improvements
Technical Improvements
state_test
) can not be determined (both used in the function overview tables on test module pages).🔗 Related Issues
Perhaps solved, plz check whether the styling is acceptable, e.g., test_bls12_g1add/test_valid.html.
Current issues:
Ideas for future improvements:
✅ Checklist
mkdocs serve
locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.