Replace Py_INCREF()/Py_XINCREF() usage with Py_NewRef()/Py_XNewRef()

[vstinner]

I added Py_NewRef() and Py_XNewRef() to Python 3.10 C API. IMO using them make to code easier to read and make the code looks "more correct". Examples:

(A) Assign + INCREF:

-                result = Py_False;
-                Py_INCREF(result);
+                result = Py_NewRef(Py_False);

(B) INCREF + assign:

-        Py_INCREF(last);
-        self->last = last;
+        self->last = Py_NewRef(last);

(C) INCREF + return:

-    Py_XINCREF(result);
-    return result;
+    return Py_XNewRef(result);

While technically, Py_INCREF() and Py_XINCREF() modify the object in-place (increment their reference counter), for me Py_NewRef() makes me sense: it creates "a new reference".

The example (A) is weird: it assigns a variable to something, and only later creates a new reference. For me, the syntax with Py_NewRef() makes more sense.

Examples (B) and (C) are shorter with Py_NewRef(), and again, IMO makes more sense and are more readable.

• PR: gh-99300: Use Py_NewRef() in Python/ directory #99302

• PR: gh-99300: Use Py_NewRef() in Python/ directory #99317

• PR: gh-99300: Use Py_NewRef() in Python/ceval.c #99318

• PR: gh-99300: Use Py_NewRef() in Parser/ directory #99330

• PR: gh-99300: Use Py_NewRef() in Objects/ directory #99332

• PR: gh-99300: Use Py_NewRef() in Objects/dictobject.c #99333

• PR: gh-99300: Use Py_NewRef() in Objects/ directory #99335

• PR: gh-99300: Use Py_NewRef() in Objects/listobject.c #99336

• PR: gh-99300: Use Py_NewRef() in Objects/ directory #99351

• PR: gh-99300: Use Py_NewRef() in Objects/ directory #99354

• PR: gh-99300: Use Py_NewRef() in Modules/_ctypes/ #99436

• PR: gh-99300: Use Py_NewRef() in Modules/_elementtree.c #99438

• PR: gh-99300: Use Py_NewRef() in Modules/itertoolsmodule.c #99439

• PR: gh-99300: Use Py_NewRef() in Modules/ directory #99440

• PR: gh-99300: Use Py_NewRef() in Modules/_asynciomodule.c #99441

• PR: gh-99300: Use Py_NewRef() in Modules/_datetimemodule.c #99465

• PR: gh-99300: Use Py_NewRef() in Modules/ directory #99466

• PR: gh-99300: Use Py_NewRef() in Modules/ directory #99467

• PR: gh-99300: Use Py_NewRef() in Modules/ directory #99468

• PR: gh-99300: Use Py_NewRef() in Modules/ directory #99469

• PR: gh-99300: Use Py_NewRef() in Modules/ directory #99473

• PR: gh-99300: Use Py_NewRef() in PC/ directory #99479

• PR: gh-99300: Use Py_NewRef() in Doc/ directory  #99480

• PR: gh-99300: Use Py_NewRef() in Python/Python-ast.c #99499

• PR: gh-99300: Replace Py_INCREF() with Py_NewRef() #99513

• PR: gh-99300: Replace Py_INCREF() with Py_NewRef() #99530

• PR: gh-99300: Replace Py_INCREF() with Py_NewRef() in _elementtree.c #99696

[vstinner]

I like the new idiom but not so much that I think it was worth all the code churn. (I would never have noticed you weren't doing this if it hadn't been for some merge conflicts in bytecodes.c, so you can ignore this comment. :-)

Sorry for the conflicts :-( I know that it's unpleasant. I tried to group changes related to ceval.c in a single commit to not require people have to fix conflicts multiple times: it's a single shot.

If you have troubles to fix your merge conflicts, I can help you to fix them. I have a tool to replace Py_INCREF with Py_NewRef().

Since I got positive feedbacks on other changes, I continue the work on this issue. The last big piece is the Modules/ directory. Then there are only a few remaining files which need to be changed.

FYI I started to collect some patterns which look suspicious and I plan to investigate them. It seems like I found a few reference leaks, and replacing Py_INCREF with Py_NewRef helped me to identify these.

I'm also considering to replace some "DECREF; INCREF; assign" patterns with Py_SETREF() when applicable. But I will open separated issues for these.

[erlend-aasland]

For the record, see gh-23170 for the PR that converted sqlite3 to Py_NewRef/Py_XNewRef.

[vstinner]

26 commits later, I have converted all "INCREF + assign" and "INCREF + return" to NewRef. I am now closing the issue.