Replace _ignored_refs with explicit allow_refs

[philippjfr]

Use allow_refs definitions as ultimate source of truth for enabling/disabling allow_refs.

[codecov]

Codecov Report

Merging #5565 (8cf5a70) into main (bf09830) will decrease coverage by 42.66%.
Report is 2 commits behind head on main.
The diff coverage is 50.00%.

@@             Coverage Diff             @@
##             main    #5565       +/-   ##
===========================================
- Coverage   83.26%   40.61%   -42.66%     
===========================================
  Files         277      277               
  Lines       39601    39578       -23     
===========================================
- Hits        32975    16075    -16900     
- Misses       6626    23503    +16877     

Flag	Coverage Δ
ui-tests	40.61% <50.00%> (-0.03%)	⬇️
unitexamples-tests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
panel/config.py	54.98% <100.00%> (-14.26%)	⬇️
panel/pane/base.py	58.28% <100.00%> (-26.25%)	⬇️
panel/pane/holoviews.py	15.85% <ø> (-63.47%)	⬇️
panel/pane/ipywidget.py	48.75% <ø> (-0.64%)	⬇️
panel/pane/plot.py	41.73% <ø> (-27.10%)	⬇️
panel/param.py	24.40% <ø> (-61.17%)	⬇️
panel/pipeline.py	13.29% <ø> (-75.51%)	⬇️
panel/reactive.py	40.62% <100.00%> (-39.48%)	⬇️
panel/widgets/indicators.py	36.36% <ø> (-37.45%)	⬇️
panel/widgets/misc.py	56.29% <ø> (-36.36%)	⬇️
... and 1 more

... and 190 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[philippjfr]

Remaining failures are resolved in holoviz/holoviews#5918