Fix doctest files fetch issue #23277

ydshieh · 2023-05-10T20:42:41Z

Embarrassingly and unfortunately, the new job tests_pr_documientation_tests fails at the step Get files to test when the job is run on the main branch.

https://app.circleci.com/pipelines/github/huggingface/transformers/64235/workflows/54a99003-258e-4c2a-8366-b4461b3ec33f/jobs/794628/parallel-runs/0/steps/0-113

I will have to take a look - the log is not informative.

HuggingFaceDocBuilderDev · 2023-05-10T20:58:52Z

The documentation is not available anymore as the PR was closed or merged.

sgugger · 2023-05-10T21:56:28Z

As said offline I don't think we need to revert urgently, we can just ignore the red check on main.

sgugger · 2023-05-10T21:58:31Z

.circleci/create_circleci_config.py

-                "git remote add upstream https://github.com/huggingface/transformers.git && git fetch upstream \n"
-                "git diff --name-only --relative --diff-filter=AMR refs/remotes/upstream/main...HEAD | grep -E '\.(py|mdx)$' | grep -Ev '^\..*|/\.' | grep -Ev '__' > pr_documentation_tests.txt"


This is the stuff that probably needs to be different on main vs a PR branch. The test fetcher has this test that can be reused/adapted.

The problems comes from the fact grep gives exit code 1 if the input is empty.

circleci@9576b4afebd1:~/transformers$ echo "" | grep -E '\.(mdx)$' circleci@9576b4afebd1:~/transformers$ echo $? 1

Considering the chain of 3 grep in the 2nd line here, I think it's best to have a python function to do things.

ydshieh · 2023-05-11T09:59:24Z

.circleci/create_circleci_config.py

@@ -432,10 +432,11 @@ def job_name(self):
    tests_to_run="tests/repo_utils",
 )

-# At this moment, only the files that are in `utils/documentation_tests.txt` will be kept (together with a dummy file).
-py_command = 'import os; import json; fp = open("pr_documentation_tests.txt"); data_1 = fp.read().strip().split("\\n"); fp = open("utils/documentation_tests.txt"); data_2 = fp.read().strip().split("\\n"); to_test = [x for x in data_1 if x in set(data_2)] + ["dummy.py"]; to_test = " ".join(to_test); print(to_test)'
+


put the logic to some new functions in utils/tests_fetcher.py.

That's cleaner thanks!

ydshieh · 2023-05-11T10:00:02Z

utils/tests_fetcher.py

@@ -216,6 +216,66 @@ def get_modified_python_files(diff_with_last_commit=False):
        return get_diff(repo, repo.head.commit, parent_commits)


+def get_diff_for_py_and_mdx_files(repo, base_commit, commits):


just copies of existing functions with some modifications for doctest test files.

ydshieh · 2023-05-11T11:47:40Z

@sgugger FYI The doctest PR also has some problem running on our GitHub Actions runner. See error below.
I will take a look, but this PR could be merged (without fixing the following issue) once you think the changes are good.

_____ ERROR collecting src/transformers/generation/configuration_utils.py ______
import file mismatch:
imported module 'transformers.generation.configuration_utils' has this __file__ attribute:
  /transformers/src/transformers/generation/configuration_utils.py
which is not the same as the test file we want to collect:
  /__w/transformers/transformers/src/transformers/generation/configuration_utils.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules

sgugger

Thanks for the cleanup! I just have a question on why the dummy.py. Maybe the comment can be more explicit on why it's needed?

sgugger · 2023-05-11T12:58:23Z

.circleci/create_circleci_config.py

@@ -432,10 +432,11 @@ def job_name(self):
    tests_to_run="tests/repo_utils",
 )

-# At this moment, only the files that are in `utils/documentation_tests.txt` will be kept (together with a dummy file).
-py_command = 'import os; import json; fp = open("pr_documentation_tests.txt"); data_1 = fp.read().strip().split("\\n"); fp = open("utils/documentation_tests.txt"); data_2 = fp.read().strip().split("\\n"); to_test = [x for x in data_1 if x in set(data_2)] + ["dummy.py"]; to_test = " ".join(to_test); print(to_test)'
+


That's cleaner thanks!

.circleci/create_circleci_config.py

ydshieh · 2023-05-11T14:24:21Z

@sgugger FYI The doctest PR also has some problem running on our GitHub Actions runner. See error below. I will take a look, but this PR could be merged (without fixing the following issue) once you think the changes are good.
_____ ERROR collecting src/transformers/generation/configuration_utils.py ______
import file mismatch:
imported module 'transformers.generation.configuration_utils' has this __file__ attribute:
  /transformers/src/transformers/generation/configuration_utils.py
which is not the same as the test file we want to collect:
  /__w/transformers/transformers/src/transformers/generation/configuration_utils.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules

@sgugger A short fix for this issue is given in the last commit.

The reason is the file conftest.py has this line from transformers.testing_utils import HfDoctestModule, HfDocTestParser added in #23271. However, the transformers is installed during docker image build, which is different from the one when the CI is run.

This change should be applied to other workflow file, but it's rare that we have such imports in the codebase. I will do it in a separate PR.

sgugger · 2023-05-11T14:56:44Z

Thanks for all the explanation!

ydshieh · 2023-05-11T15:13:44Z

Going to merge as the failing tests are irrelevant and I have tried to re-run for a few times.

* fix --------- Co-authored-by: ydshieh <[email protected]>

ydshieh requested a review from sgugger May 10, 2023 20:43

sgugger reviewed May 10, 2023

View reviewed changes

ydshieh force-pushed the revert-23271-revert-23245-revert-22987-doct-tests branch from 67a269a to 7ccebe6 Compare May 11, 2023 07:52

ydshieh changed the title ~~Revert "Bring back the PR Refactor doctests + add CI to main"~~ Fix doctest files fetch issue May 11, 2023

ydshieh commented May 11, 2023

View reviewed changes

ydshieh requested a review from sgugger May 11, 2023 10:00

ydshieh added 6 commits May 11, 2023 12:47

fix

749b3e0

fix

043f7f1

fix

8a8fabf

fix

8339b33

fix

03c98b6

fix

863dcf5

ydshieh force-pushed the revert-23271-revert-23245-revert-22987-doct-tests branch from e8ee1c1 to 863dcf5 Compare May 11, 2023 10:47

sgugger approved these changes May 11, 2023

View reviewed changes

ydshieh added 2 commits May 11, 2023 15:23

fix

f699906

fix

bada5b3

ydshieh merged commit 6a6225b into main May 11, 2023

ydshieh deleted the revert-23271-revert-23245-revert-22987-doct-tests branch May 11, 2023 15:14

sheonhan pushed a commit to sheonhan/transformers that referenced this pull request May 15, 2023

Fix doctest files fetch issue (huggingface#23277)

df9d2b7

* fix --------- Co-authored-by: ydshieh <[email protected]>

ydshieh mentioned this pull request May 19, 2023

Fix confusing transformers installation in CI #23465

Merged

gojiteji pushed a commit to gojiteji/transformers that referenced this pull request Jun 5, 2023

Fix doctest files fetch issue (huggingface#23277)

8220fca

* fix --------- Co-authored-by: ydshieh <[email protected]>

novice03 pushed a commit to novice03/transformers that referenced this pull request Jun 23, 2023

Fix doctest files fetch issue (huggingface#23277)

35d56eb

* fix --------- Co-authored-by: ydshieh <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix doctest files fetch issue #23277

Fix doctest files fetch issue #23277

ydshieh commented May 10, 2023

HuggingFaceDocBuilderDev commented May 10, 2023 •

edited

Loading

sgugger commented May 10, 2023

sgugger May 10, 2023

ydshieh May 11, 2023

ydshieh May 11, 2023

sgugger May 11, 2023

ydshieh May 11, 2023

ydshieh commented May 11, 2023

sgugger left a comment

sgugger May 11, 2023

ydshieh commented May 11, 2023

sgugger commented May 11, 2023

ydshieh commented May 11, 2023

		"git remote add upstream https://github.com/huggingface/transformers.git && git fetch upstream \n"
		"git diff --name-only --relative --diff-filter=AMR refs/remotes/upstream/main...HEAD \| grep -E '\.(py\|mdx)$' \| grep -Ev '^\..*\|/\.' \| grep -Ev '__' > pr_documentation_tests.txt"

		@@ -216,6 +216,66 @@ def get_modified_python_files(diff_with_last_commit=False):
		return get_diff(repo, repo.head.commit, parent_commits)


		def get_diff_for_py_and_mdx_files(repo, base_commit, commits):

Fix doctest files fetch issue #23277

Fix doctest files fetch issue #23277

Conversation

ydshieh commented May 10, 2023

HuggingFaceDocBuilderDev commented May 10, 2023 • edited Loading

sgugger commented May 10, 2023

sgugger May 10, 2023

Choose a reason for hiding this comment

ydshieh May 11, 2023

Choose a reason for hiding this comment

ydshieh May 11, 2023

Choose a reason for hiding this comment

sgugger May 11, 2023

Choose a reason for hiding this comment

ydshieh May 11, 2023

Choose a reason for hiding this comment

ydshieh commented May 11, 2023

sgugger left a comment

Choose a reason for hiding this comment

sgugger May 11, 2023

Choose a reason for hiding this comment

ydshieh commented May 11, 2023

sgugger commented May 11, 2023

ydshieh commented May 11, 2023

HuggingFaceDocBuilderDev commented May 10, 2023 •

edited

Loading