Possible Hold bug

[ZXGuesser]

Edit: disregard this, I'm an idiot who can't count...

[rawles]

Sorry, just got around to this. If you're sure you're mistaken I'll close the bug. Feel free to re-open it if you later discover similar problems.

[ZXGuesser]

This isn't really the same bug but it is to do with held mosaics;

What is the reasoning behind ignoring character 0x1B (ESC) when considering held mosiacs? I understand that that control character isn't implemented to do anything, but even if it was it is set-after so the held character should be displayed I think. My TV does display it for what that is worth.

It also resets the held character to the space on the level 2.5+ size control codes (0x0E and 0x0F). It would be nice to have an option to select this behaviour, like with the allowing and ignoring of codes 0x00 and 0x10. I realise that this is all a bit academic to most people since you can't currently enter any of these control codes in the editor :)

I've discovered all these odd edge cases while assembling a better copy of the old version of the engineering test card as documented on http://www.pembers.freeserve.co.uk/Teletext/Photographs.html
I'm 99% confident I now have all the control codes correct as shown by the output of the TIFAX decoder: http://edit.tf/#0:QIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAYMS54fzJmixs-YCDToOLOjyZ0WLSkzo6AkcGHuZUeRHlB4dgyAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgGDP9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9_YNCho8sImSoACAAlRp0FUy8-iChhz5UCA4MPEuZYgTAFyAFg1_f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f2DYCAAgAIACAAgAIACAHkliwffHngIACAAgAIACAAgAIACAYN_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_39g4AgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgGDn9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9_YsAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIBix_f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f2LICAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAYs_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_39i0AgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgGLX9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9_Ytq-jT0yg7OXZs39w0Pzh3Ao_LLl3BZuHPl3dMIGllyBIWzrlLmkKJGTSJUycsoUqlZJYtXLzLBiyZlWjVs3IuHLp2UePXz9AgQokaBIlTJ0ChSqVoFi1cvQMGLJmgaNWzdA4cunaB49fP0ECDChoIkWNHQSJMqWgmTZ09BQo0qaCpVrV0FizatoLl29fQYMOLGgyZc2dBo06taDZt3b0HDjy5oOnXt3QePPr2g-ff38pgw4sZHJlzZyujTq1ktm3dvNcOPLmW6de3cn48-vZf59_fwZiHv3Y8uHYIjbMPPQDVCxcLf4E0-mXDk8mI-_dlFCn5a9_QIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECA

Edit: updated url to fix error on row 7

[rawles]

Thanks, I'll look into this later. In the meantime, here's the engineering test card that I've been using:

http://edit.tf/#0:QIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECAueH8yZosePmAg06Dizo8mdFi0pM6OgJHBh7mVHER5QeHYMi54fzJmix4-YCDToOLOjyZ0WLSkzo6AkcGHuZUcRHlB4dgy_f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f2DQoaPcyJkqAKgRqAKdBVMvPogoYc-VAgODDxLmWMEwBcwBYNSho9zImSoAqBGoAp0FUy8-iChhz5UCA4MPEuZYwTAFzAFg1AgAKAAgAnkB5AXWEyZYsSJEipUiRKFCiBAUAAQAEABAAWDf9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9_YOAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIBg5_f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f2LACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAYsf3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_39iyAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgGLP9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9_YtAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIBi1_f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f2Lavo09MoOzl2bN_cND84dwKPyy5dwWbhz5d3TCBpZcgSFs65S5pCiRk0iVMnLKFKpWSWLVy8ywYsmZVo1bNyLhy6dlHj18_QIEKJGgSJUydAoUqlaBYtXL0DBiyZoGjVs3QOHLp2gePXz9BAgwoaCJFjR0EiTKloJk2dPQUKNKmgqVa1dBYs2raC5dvX0GDDixoMmXNnQaNOrWg2bd29Bw48uaDp17d0Hjz69oPn39_KYMOLGRyZc2cro06tZLZt3bzXDjy5lunXt3J-PPr2X-ff38GYh792PLh2CI2zDz0A1QsXC3-BNPplw5PJiPv3ZRQp-WvfwUXdn07suXlp3Z0AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECA

[ZXGuesser]

Yeah I saw that in the readme. I'm not sure where that has come from because it starts a row early and has extra text at the bottom that I've never seen on any photographs or screenshots. I'm trying to get hold of a proper dump of that version of the test page as transmitted and if so I'll put together a proper version of that one too as I'm interested to see whether it avoids using the problematic control codes that stop the original from displaying on modern TV sets that respond to some level 2.5+ codes.

[ZXGuesser]

I've spent more time researching testcards and have improved my copy of the circa 1976 testcard:
http://edit.tf/#0:QIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAYMS54fzJmixs-YCDToOLOjyZ0WLSkzo6AkcGHuZUeRHlB4dgyAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgGDP9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9_YNCho9zImSoACAAlRp0FUy8-iChhz5UCA4MPEuZYgTAFyAFg1_f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f2DYCAAgAIACAAgAIACAHkliwffHngIACAAgAIACAAgAIACAYN_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_39g4AgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgGDn9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9_YsAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIBix_f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f2LICAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAYs_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_39i0AgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgGLX9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9_Ytq-jT0yg7OXZs39w0Pzh3Ao_LLl3BZuHPl3dMIGllyBIWzrlLmkKJGTSJUycsoUqlZJYtXLzLBiyZlWjVs3IuHLp2UePXz9AgQokaBIlTJ0ChSqVoFi1cvQMGLJmgaNWzdA4cunaB49fP0ECDChoIkWNHQSJMqWgmTZ09BQo0qaCpVrV0FizatoLl29fQYMOLGgyZc2dBo06taDZt3b0HDjy5oOnXt3QePPr2g-ff38pgw4sZHJlzZyujTq1ktm3dvNcOPLmW6de3cn48-vZf59_fwZiHv3Y8uHYIjbMPPQDVCxcLf4E0-mXDk8mI-_dlFCn5a9_QIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECA

I've also done a "probably accurate" recreation of the later version with the colour stripes, based on screenshots and old page captures including the one your example is derived from:
http://edit.tf/#0:QIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAYMS54fzJmixs-YCDToOLOjyZ0WLSkzo6AkcGHuZUeRHlB4dgyAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgGDP9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9_YNCho9zImSoAqBGoAp0FUy8-iChhz5UCA4MPEuZYwTAFzAFg1_f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f2DYCAAgAIACAAgC55YTHlh5IePKjyI8oPAgAIACAAgAIACAYN_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_39g4AgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgGDn9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9_YsAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIBix_f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f2LICAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAYs_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_39i0AgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgAIACAAgGLX9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9__f_3_9_Ytq-jT0yg7OXZs39w0Pzh3Ao_LLl3BZuHPl3dMIGllyBIWzrlLmkKJGTSJUycsoUqlZJYtXLzLBiyZlWjVs3IuHLp2UePXz9AgQokaBIlTJ0ChSqVoFi1cvQMGLJmgaNWzdA4cunaB49fP0ECDChoIkWNHQSJMqWgmTZ09BQo0qaCpVrV0FizatoLl29fQYMOLGgyZc2dBo06taDZt3b0HDjy5oOnXt3QePPr2g-ff38pgw4sZHJlzZyujTq1ktm3dvNcOPLmW6de3cn48-vZf59_fwZiHv3Y8uHYIjbMPPQDVCxcLf4E0-mXDk8mI-_dlFCn5a9_QIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECA

The example posted above and in the readme is a dump from a BBC micro which explains the repeated double height rows. The green text at the bottom was added by the micro and was not part of the transmitted page.

[rawles]

I've had a quick look through the SAA5050 and ETS300706 specs and they seem to imply (but not explicitly state) that any character between 0x00 and 0x1F is a spacing/control character, which I think is your opinion too. I'll try to implement this soon.

About the testcards, it would be useful to have a definitive testcard that also includes all the really fiendish corner cases. I was wondering how you had constructed the test cards that you linked?

[ZXGuesser]

Well the very early one comes from this photograph: https://web.archive.org/web/20160117232136/http://www.pembers.freeserve.co.uk/Teletext/Ceefax1-Engineering2.jpg
The later one is based on a combination of that, and a couple of data captures people have sent me.
I created everything in edit-tf and added the few unavailable codes by setting a breakpoint in the debugger and changing values manually.

[rawles]

After having looked at the code I noticed the actual behaviour we're after is to only ignore these codes (0xA, 0xB, 0xE, 0xF, 0x1B) if we're in text mode. In graphics mode we go through to the held graphics handling, which should fix this. If you have a chance, perhaps you can verify that this works in the way you'd expect.

About the testcards, I think it might be useful to make a small collection of them in a subdirectory with the expected display as a graphics file. What do you think?

[ZXGuesser]

What should actually happen is kind of undefined for level 1. 0x0E and 0x0F have meanings at level 2.5+ but are "interpreted by some existing level 1 and level 1.5 decoders" as with alpha black and mosaics black.

0x0A and 0x0B have no effect unless a page is a newsflash or subtitles page but held mosaics should be displayed for them.

0x1B toggles character set so that accents can be displayed etc. but again that's something that's not implemented so it can just be left as a no-op with held mosaics displayed.

My TV makes a mess of displaying the test card, and the BBC's in vision decoder has at least one held mosaics bug too as far as I can make out. A gallery of testcards along with expected output is a good idea though. Sort of an Acid rendering test for teletext. I've been meaning to do that on my website for a while and still not got around to it.

[rawles]

Thanks for the information. It sounds like things are at least improved with the fixed today. The following frame demonstrates that all the control characters display held mosaics now, well, except for the for 'new background' (0x1D), which is 'set at'.

http://edit.tf/#8:QIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIDzDwwYILpfMAQIA7DwxYILpfMQQIECBAgQIECBAgQIECBAgPMPDBigul8wFAgDsPDFigul8xFAgQIECBAgQIECBAgQIECA8w8MGSC6XzAkCAOw8MWSC6XzEkCBAgQIECBAgQIECBAgQIDzDwwZoLpfMDQIA7DwxZoLpfMTQIECBAgQIECBAgQIECBAgPMPDBogul8wRAgDsPDFogul8xRAgQIECBAgQIECBAgQIECA8w8MGqC6XzBUCAOw8MWqC6XzFUCBAgQIECBAgQIECBAgQIDzDwwbILpfMGQIA7DwxbILpfMWQIECBAgQIECBAgQIECBAgPMPDBugul8wdAgDsPDFugul8xdAgQIECBAgQIECBAgQIECA8w8MHCC6XzCEAkOw8MXCC6XzGECBAgQIECBAgQIECBAgQIDzDwwcoLpfMJQIA7DwxcoLpfMZQIECBAgQIECBAgQIECBAgPMPDDCgul8wpAgDsPDHCgul8xpAgQIECBAgQIECBAgQIECA8w8MMSC6XzC0CAOw8McSC6XzG0CBAgQIECBAgQIECBAgQIDzDwwxoLpfMMQIA7DwxxoLpfMcQIECBAgQIECBAgQIECBAgPMPDDIgul8w1AgDsPDHIgul8x1AgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIDzDwwyoLpfMOQIA7DwxyoLpfMeQIECBAgQIECBAgQIECBAgPMPDDMgul8w9AgDsPDHMgul8x9AgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECA

Still not sure what to do about 0x0E and 0x0F. I've read documentation that says they can also be interpreted by some decoders as double height. I don't think they should do these things in edit-tf, but since they are both set-after, I think it makes sense to substitute mosaic graphics as in the other cases. Do you agree?

The gallery of testcards would be very useful, I think.

[ZXGuesser]

What about Conceal? The specification isn't explicit about what happens to concealed held mosaics.
In my rendering software I don't display the held mosaics when concealed. This was also the behaviour of the BBC's in-vision decoder: https://i.ytimg.com/vi/b1ZKzh7LgF4/hqdefault.jpg

0x0E and 0x0F do double width and double size at level 2.5 so presumably out of scope for edit.tf. The spec says that some decoders interpret 0x0F as double height, but that's just a warning note, I don't think that should be implemented.

The other question is whether the double width and double size codes would count as a "change of size" to a level 1 decoder and thus clear the held mosaic. Again the specification doesn't say. I think on balance I would lean towards them not doing, and just treating them as a no-op.
My rubbish "Level 1 with foreground black" TV does reset the hold graphic on these, but the BBC in vision decoder apparently did not.

[rawles]

As it stands, in the editor you can set a held graphics character in concealed text and then use it outside of the concealed block (which you'd need to terminate with a mosaic control character). The following frame shows this behaviour, and also happily appears to show compliance with the European standard, at least for some aspects of held graphics.

http://edit.tf/#0:CHQY2Ny37UEWpTZsGDdg2dDECBAgQIECBRww58qBwwUoECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIKmjKgRSMuzItm7-eHTjRIEGPRh5YcfTLyQaeaDll55eiDpvRU6EGHFRIMPRB00ZUCDn0w8uiDfmQIMuHHoQct_dYg37kGHHow7s-VBvzIMOzhow7uu3Ly040C9Bt388OnHzQbd-TKg38t-5BhQY9GHdnyoN-ZBz0-sq5Agk9EGnmg3b-iDll55eiDF55ZdO7Nv5Y8u3Lu6IECDfmQIOmjKgQIMvjTz6ad2dAg56fWXnl6dNO7OuQIJPRBp5oN2_og5ZeeXogxeUGFBj0Yd2fKg07pG_ZkQTd_PDpx80G3fkyrkCBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECA8RWEiYdAgQU-mHzp3Z0Gncg27-eHTj5oO-_cn6IMeHdjy7DxFYSAkw6BBD0Yd2fTuzoOm9B0y-OiDbvyZUGPDux5dnNAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECA8RWEhhMOgQUsundm38sendnQc9PrKgyb8vPcn6IMeHdjy7Bp4isJDSYdBSy6d2bfyx6d2dBEkIMm_Lz3J-iDHh3Y8uxAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECA8RWEhpMOgQQ9GHdnyoN-ZB0y-OiDnp9ZUGTfl5oMeHdjy7ECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECA8RWHyR4mHQVNGVBoy7MiDPyw8NGnHzQY9GHlhx9MvJBp5oECBAgQIECBB13Y9GHdny5EGLygwoMejDuz5UGncgkR19KOuQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECAIdByMO7Js07s6DHv3Y8uHZlyIJEdBj0YeWHH0y8ubpAgQICJg8sWLCRMPU0ZUEiOgx6MPLDj6ZeSDTzQc8vRBp3IOmjKgx792PLh2ZciDpl8dEGXtl3INOZBp6INPNBu39EHPRv77lyA

[ZXGuesser]

That all looks good to me. The case that the specification isn't clear about is what gets displayed for something like this:

http://edit.tf/#0:QIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECCdv5bcOxBoy6c-jog688uRBh5oOGHJk07s6DpvQZNPPhsw-dGXZkQbd_PDpxoMejDyw4-mXkuQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECDHvyZebpBHkIIc6dOnTkE6dOnR506cgQIECBAgQIECBAgQIECBAgQIC9KO5nyJEiRIYyJEiRZkSJCBAgQIECBAgQIECBAgQIECBAgQETzkwMGDBgxiMGDBhMYMGIECBAgQIECBAgQIECBAgQIECBAgQXryBAgQILyBAgQXkCBAgQIECBAgQIECBAgQIECBAgQIECBA6dIECBAgdIECBA6QIECBAgQIECBAgQIECBAgQIECBAgQIEDp0gQIECB0gQIECtagx792PLh2IN-bMgQIECBAgQIECBAgQOnSBAgQIHSBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBA6dIECBAgVrUGjLsyINu_nh040GPRh3Z8vNAgQIECBAgQIEDp0gQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQOla1Bj37seXDsQb9yBAgQIECBAgQIECBAgQIECBAgQIECBA6QIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECtag0ZdmRBt388OnGg55eiBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECBAgQIECA

[rawles]

Nice! Thanks for exposing all these bugs!

The ETS spec says "At a screen location where substitution is permitted, the 'Held-Mosaic' character inserted is the most recent mosaics character with bit 6 = '1' in its code on that row.". I guess that's whether the character is concealed or not.

Should those NH characters be showing up in the block of concealed characters, though?

[ZXGuesser]

That's the thing. It also says "The following characters up to the end of the row, or until a Colour Code attribute is encountered, are to be displayed as SPACES until revealed by a decoder or user operation."
Nowhere does it say which of those statements has priority or whether the SPACE substituted by conceal still counts as a "SPACE corresponding to a control character" as a "location where substitution is permitted"
If it was anywhere I'd expect it to be in Annex G where it gives the example of held mosaics not being affected by switching to separated mosaics etc.

[rawles]

I could test these cases out on a BBC Micro this week and see what the SAA5050 does, report back here, and try to match its behaviour in the editor.

I think it might be useful for people implementing editors, frame renderers, etc., to have a page somewhere detailing what various decoders do in these cases. We could make a mini-project for this - like you say, an Acid rendering test for teletext.