Ledger produces an invalid flat term

[dnadales]

When checking that the CBOR produced by Consensus is valid I ran into the error shown below, which can be reproduced by running:

cabal new-run ouroboros-consensus-cardano:cardano-test -- -p '/Roundtrip.SerialiseNodeToClient.roundtrip Result/'

on this branch.

          WithVersion (HardForkNodeToClientEnabled HardForkSpecificNodeToClientVersion2 (EraNodeToClientEnabled ByronNodeToClientVersion1 :* EraNodeToClientEnabled ShelleyNodeToClientVersion4 :* EraNodeToClientEnabled ShelleyNodeToClientVersion3 :* EraNodeToClientEnabled ShelleyNodeToClientVersion2 :* EraNodeToClientEnabled ShelleyNodeToClientVersion1 :* EraNodeToClientEnabled ShelleyNodeToClientVersion8 :* EraNodeToClientEnabled ShelleyNodeToClientVersion8 :* Nil)) Right (PParams (AlonzoPParams {appMinFeeA = Coin 731653, appMinFeeB = Coin 510467, appMaxBBSize = 2, appMaxTxSize = 1, appMaxBHSize = 1, appKeyDeposit = Coin 608263, appPoolDeposit = Coin 341615, appEMax = EpochNo 19, appNOpt = 7, appA0 = 7327765607742746987 % 1000, appRho = 3 % 5, appTau = 1813 % 100000, appD = 2449668107080885149 % 2500000000000000000, appExtraEntropy = Nonce "c697f75851eac9612d7161af6196ed3402da04dc407238184e75fd4bdd4bd65b", appProtocolVersion = ProtVer {pvMajor = Version 10, pvMinor = 6}, appMinPoolCost = Coin 500529, appCoinsPerUTxOWord = Coin 495448, appCostModels = CostModels {costModelsValid = fromList [(PlutusV1,CostModel PlutusV1 [4,-2,0,-8,-2,-5,7,-3,6,-1,0,1,1,-8,3,7,-1,8,8,-7,-1,-6,-3,4,1,0,3,-6,0,4,6,-1,-6,5,7,4,-2,7,-4,-7,-2,8,-5,1,5,5,-5,7,4,3,0,6,6,-2,5,3,6,8,7,-2,-1,-4,-1,4,-8,-4,3,0,-3,3,-8,-8,2,-8,3,2,-1,2,-8,6,8,-1,-7,2,2,5,-1,-1,8,-2,2,0,0,-2,-8,-2,5,2,5,-1,-6,4,8,-1,-8,-1,3,3,6,0,3,-2,-2,3,0,-4,-3,-5,3,3,8,7,-7,0,0,-4,5,-3,-2,-5,-7,5,2,-1,2,-5,-6,-5,-3,-5,5,-7,-7,-7,0,-2,-3,8,-3,-5,-2,1,8,1,-2,-3,-2,-2,-5,5,-2,8,-2,-3,-5,-5]),(PlutusV2,CostModel PlutusV2 [-4,-1,-3,-2,-6,7,-3,5,-3,0,4,7,-4,-4,0,8,-3,-7,-6,-1,-1,-3,4,2,-5,3,-4,0,0,6,-5,-3,-8,-1,-8,-1,1,-4,2,-4,5,6,-1,-5,3,-2,-5,-7,-6,-4,-6,-7,-4,0,2,7,1,5,-4,-5,8,-2,-2,-7,1,0,-6,-5,-6,-6,0,4,-2,1,2,6,-8,-5,1,-7,8,-1,5,1,6,1,-2,1,-5,0,4,8,-1,8,5,5,5,1,8,-8,-1,6,-8,-6,2,8,-1,-6,-6,8,5,5,-2,-5,-4,8,5,7,3,7,-5,-1,-5,5,-8,-2,1,-3,-4,3,-1,1,3,-2,5,4,4,-7,5,-7,5,-1,2,-8,4,3,5,-8,-1,-8,8,4,6,7,2,0,-4,4,4,-4,7,8,5,-4,5,7,0,5,-8,0,1,0,6,7,0]),(PlutusV3,CostModel PlutusV3 [-3,6,8,6,-6,4,8,8,-8,-4,1,-3,-1,3,7,-7,7,-4,1,8,-5,0,-7,-8,-1,-2,-5,1,4,2,-4,7,-5,5,7,8,6,7,6,7,-3,8,1,7,4,5,3,0,-4,7,-3,-3,-8,-3,2,8,-7,-8,4,-4,8,4,5,7,-1,6,7,4,-8,7,-1,5,-4,-5,4,-2,-2,-3,-7,7,1,-5,0,1,4,-1,8,2,-7,-8,8,2,-4,2,4,-3,-2,-1,-5,8,-4,-4,6,-5,7,8,7,-6,7,-2,-5,-5,-5,-4,-6,-8,3,-3,1,-2,1,0,1,-3,1,1,1,1,-6,1,-2,2,-2,0,-5,-3,-1,8,8,-1,-6,6,2,-4,-3,-2,6,3,0,-6,-7,-1,-8,1,2,-4,5,-7,4,6,-4,1,-4,3,-4,4,7,-3,-4,-7,-7,6,1,6,-1,-5,3,-2,-1,-8,4,8,5,-7,2,0,-7,2,0,8,-2,-6,6,-5,2,-6,-6,7,0,-6,2,0,-2,6,-2,2,-1,-1,6,-6,0,-5,-2,8,6,2,3,3,7,0,-2,4,8])], costModelsErrors = fromList [], costModelsUnknown = fromList []}, appPrices = Prices {prMem = 390445784865860073 % 25000000000, prSteps = 4691156197783094469 % 5000}, appMaxTxExUnits = WrapExUnits {unWrapExUnits = ExUnits' {exUnitsMem' = 8175950039059856398, exUnitsSteps' = 7517193088755259942}}, appMaxBlockExUnits = WrapExUnits {unWrapExUnits = ExUnits' {exUnitsMem' = 6460320455910312492, exUnitsSteps' = 4127550193106455665}}, appMaxValSize = 2, appCollateralPercentage = 6, appMaxCollateralInputs = 2}))
          Encoded flat term is not valid: [TkListLen 1,TkListLen 24,TkInt 731653,TkInt 510467,TkInt 2,TkInt 1,TkInt 1,TkInt 608263,TkInt 341615,TkInt 19,TkInt 7,TkTag 30,TkListLen 2,TkInt 7327765607742746987,TkInt 1000,TkTag 30,TkListLen 2,TkInt 3,TkInt 5,TkTag 30,TkListLen 2,TkInt 1813,TkInt 100000,TkTag 30,TkListLen 2,TkInt 2449668107080885149,TkInt 2500000000000000000,TkListLen 2,TkInt 1,TkBytes "\198\151\247XQ\234\201a-qa\175a\150\237\&4\STX\218\EOT\220@r8\CANNu\253K\221K\214[",TkInt 10,TkInt 6,TkInt 500529,TkInt 495448,TkMapLen 3,TkInt 0,TkListBegin,TkInt 4,TkInt (-2),TkInt 0,TkInt (-8),TkInt (-2),TkInt (-5),TkInt 7,TkInt (-3),TkInt 6,TkInt (-1),TkInt 0,TkInt 1,TkInt 1,TkInt (-8),TkInt 3,TkInt 7,TkInt (-1),TkInt 8,TkInt 8,TkInt (-7),TkInt (-1),TkInt (-6),TkInt (-3),TkInt 4,TkInt 1,TkInt 0,TkInt 3,TkInt (-6),TkInt 0,TkInt 4,TkInt 6,TkInt (-1),TkInt (-6),TkInt 5,TkInt 7,TkInt 4,TkInt (-2),TkInt 7,TkInt (-4),TkInt (-7),TkInt (-2),TkInt 8,TkInt (-5),TkInt 1,TkInt 5,TkInt 5,TkInt (-5),TkInt 7,TkInt 4,TkInt 3,TkInt 0,TkInt 6,TkInt 6,TkInt (-2),TkInt 5,TkInt 3,TkInt 6,TkInt 8,TkInt 7,TkInt (-2),TkInt (-1),TkInt (-4),TkInt (-1),TkInt 4,TkInt (-8),TkInt (-4),TkInt 3,TkInt 0,TkInt (-3),TkInt 3,TkInt (-8),TkInt (-8),TkInt 2,TkInt (-8),TkInt 3,TkInt 2,TkInt (-1),TkInt 2,TkInt (-8),TkInt 6,TkInt 8,TkInt (-1),TkInt (-7),TkInt 2,TkInt 2,TkInt 5,TkInt (-1),TkInt (-1),TkInt 8,TkInt (-2),TkInt 2,TkInt 0,TkInt 0,TkInt (-2),TkInt (-8),TkInt (-2),TkInt 5,TkInt 2,TkInt 5,TkInt (-1),TkInt (-6),TkInt 4,TkInt 8,TkInt (-1),TkInt (-8),TkInt (-1),TkInt 3,TkInt 3,TkInt 6,TkInt 0,TkInt 3,TkInt (-2),TkInt (-2),TkInt 3,TkInt 0,TkInt (-4),TkInt (-3),TkInt (-5),TkInt 3,TkInt 3,TkInt 8,TkInt 7,TkInt (-7),TkInt 0,TkInt 0,TkInt (-4),TkInt 5,TkInt (-3),TkInt (-2),TkInt (-5),TkInt (-7),TkInt 5,TkInt 2,TkInt (-1),TkInt 2,TkInt (-5),TkInt (-6),TkInt (-5),TkInt (-3),TkInt (-5),TkInt 5,TkInt (-7),TkInt (-7),TkInt (-7),TkInt 0,TkInt (-2),TkInt (-3),TkInt 8,TkInt (-3),TkInt (-5),TkInt (-2),TkInt 1,TkInt 8,TkInt 1,TkInt (-2),TkInt (-3),TkInt (-2),TkInt (-2),TkInt (-5),TkInt 5,TkInt (-2),TkInt 8,TkInt (-2),TkInt (-3),TkInt (-5),TkInt (-5),TkBreak,TkInt 1,TkListBegin,TkInt (-4),TkInt (-1),TkInt (-3),TkInt (-2),TkInt (-6),TkInt 7,TkInt (-3),TkInt 5,TkInt (-3),TkInt 0,TkInt 4,TkInt 7,TkInt (-4),TkInt (-4),TkInt 0,TkInt 8,TkInt (-3),TkInt (-7),TkInt (-6),TkInt (-1),TkInt (-1),TkInt (-3),TkInt 4,TkInt 2,TkInt (-5),TkInt 3,TkInt (-4),TkInt 0,TkInt 0,TkInt 6,TkInt (-5),TkInt (-3),TkInt (-8),TkInt (-1),TkInt (-8),TkInt (-1),TkInt 1,TkInt (-4),TkInt 2,TkInt (-4),TkInt 5,TkInt 6,TkInt (-1),TkInt (-5),TkInt 3,TkInt (-2),TkInt (-5),TkInt (-7),TkInt (-6),TkInt (-4),TkInt (-6),TkInt (-7),TkInt (-4),TkInt 0,TkInt 2,TkInt 7,TkInt 1,TkInt 5,TkInt (-4),TkInt (-5),TkInt 8,TkInt (-2),TkInt (-2),TkInt (-7),TkInt 1,TkInt 0,TkInt (-6),TkInt (-5),TkInt (-6),TkInt (-6),TkInt 0,TkInt 4,TkInt (-2),TkInt 1,TkInt 2,TkInt 6,TkInt (-8),TkInt (-5),TkInt 1,TkInt (-7),TkInt 8,TkInt (-1),TkInt 5,TkInt 1,TkInt 6,TkInt 1,TkInt (-2),TkInt 1,TkInt (-5),TkInt 0,TkInt 4,TkInt 8,TkInt (-1),TkInt 8,TkInt 5,TkInt 5,TkInt 5,TkInt 1,TkInt 8,TkInt (-8),TkInt (-1),TkInt 6,TkInt (-8),TkInt (-6),TkInt 2,TkInt 8,TkInt (-1),TkInt (-6),TkInt (-6),TkInt 8,TkInt 5,TkInt 5,TkInt (-2),TkInt (-5),TkInt (-4),TkInt 8,TkInt 5,TkInt 7,TkInt 3,TkInt 7,TkInt (-5),TkInt (-1),TkInt (-5),TkInt 5,TkInt (-8),TkInt (-2),TkInt 1,TkInt (-3),TkInt (-4),TkInt 3,TkInt (-1),TkInt 1,TkInt 3,TkInt (-2),TkInt 5,TkInt 4,TkInt 4,TkInt (-7),TkInt 5,TkInt (-7),TkInt 5,TkInt (-1),TkInt 2,TkInt (-8),TkInt 4,TkInt 3,TkInt 5,TkInt (-8),TkInt (-1),TkInt (-8),TkInt 8,TkInt 4,TkInt 6,TkInt 7,TkInt 2,TkInt 0,TkInt (-4),TkInt 4,TkInt 4,TkInt (-4),TkInt 7,TkInt 8,TkInt 5,TkInt (-4),TkInt 5,TkInt 7,TkInt 0,TkInt 5,TkInt (-8),TkInt 0,TkInt 1,TkInt 0,TkInt 6,TkInt 7,TkInt 0,TkBreak,TkInt 2,TkListBegin,TkInt (-3),TkInt 6,TkInt 8,TkInt 6,TkInt (-6),TkInt 4,TkInt 8,TkInt 8,TkInt (-8),TkInt (-4),TkInt 1,TkInt (-3),TkInt (-1),TkInt 3,TkInt 7,TkInt (-7),TkInt 7,TkInt (-4),TkInt 1,TkInt 8,TkInt (-5),TkInt 0,TkInt (-7),TkInt (-8),TkInt (-1),TkInt (-2),TkInt (-5),TkInt 1,TkInt 4,TkInt 2,TkInt (-4),TkInt 7,TkInt (-5),TkInt 5,TkInt 7,TkInt 8,TkInt 6,TkInt 7,TkInt 6,TkInt 7,TkInt (-3),TkInt 8,TkInt 1,TkInt 7,TkInt 4,TkInt 5,TkInt 3,TkInt 0,TkInt (-4),TkInt 7,TkInt (-3),TkInt (-3),TkInt (-8),TkInt (-3),TkInt 2,TkInt 8,TkInt (-7),TkInt (-8),TkInt 4,TkInt (-4),TkInt 8,TkInt 4,TkInt 5,TkInt 7,TkInt (-1),TkInt 6,TkInt 7,TkInt 4,TkInt (-8),TkInt 7,TkInt (-1),TkInt 5,TkInt (-4),TkInt (-5),TkInt 4,TkInt (-2),TkInt (-2),TkInt (-3),TkInt (-7),TkInt 7,TkInt 1,TkInt (-5),TkInt 0,TkInt 1,TkInt 4,TkInt (-1),TkInt 8,TkInt 2,TkInt (-7),TkInt (-8),TkInt 8,TkInt 2,TkInt (-4),TkInt 2,TkInt 4,TkInt (-3),TkInt (-2),TkInt (-1),TkInt (-5),TkInt 8,TkInt (-4),TkInt (-4),TkInt 6,TkInt (-5),TkInt 7,TkInt 8,TkInt 7,TkInt (-6),TkInt 7,TkInt (-2),TkInt (-5),TkInt (-5),TkInt (-5),TkInt (-4),TkInt (-6),TkInt (-8),TkInt 3,TkInt (-3),TkInt 1,TkInt (-2),TkInt 1,TkInt 0,TkInt 1,TkInt (-3),TkInt 1,TkInt 1,TkInt 1,TkInt 1,TkInt (-6),TkInt 1,TkInt (-2),TkInt 2,TkInt (-2),TkInt 0,TkInt (-5),TkInt (-3),TkInt (-1),TkInt 8,TkInt 8,TkInt (-1),TkInt (-6),TkInt 6,TkInt 2,TkInt (-4),TkInt (-3),TkInt (-2),TkInt 6,TkInt 3,TkInt 0,TkInt (-6),TkInt (-7),TkInt (-1),TkInt (-8),TkInt 1,TkInt 2,TkInt (-4),TkInt 5,TkInt (-7),TkInt 4,TkInt 6,TkInt (-4),TkInt 1,TkInt (-4),TkInt 3,TkInt (-4),TkInt 4,TkInt 7,TkInt (-3),TkInt (-4),TkInt (-7),TkInt (-7),TkInt 6,TkInt 1,TkInt 6,TkInt (-1),TkInt (-5),TkInt 3,TkInt (-2),TkInt (-1),TkInt (-8),TkInt 4,TkInt 8,TkInt 5,TkInt (-7),TkInt 2,TkInt 0,TkInt (-7),TkInt 2,TkInt 0,TkInt 8,TkInt (-2),TkInt (-6),TkInt 6,TkInt (-5),TkInt 2,TkInt (-6),TkInt (-6),TkInt 7,TkInt 0,TkInt (-6),TkInt 2,TkInt 0,TkInt (-2),TkInt 6,TkInt (-2),TkInt 2,TkInt (-1),TkInt (-1),TkInt 6,TkInt (-6),TkInt 0,TkInt (-5),TkInt (-2),TkInt 8,TkInt 6,TkInt 2,TkInt 3,TkInt 3,TkInt 7,TkInt 0,TkInt (-2),TkInt 4,TkInt 8,TkBreak,TkListLen 2,TkTag 30,TkListLen 2,TkInt 390445784865860073,TkInt 25000000000,TkTag 30,TkListLen 2,TkInt 4691156197783094469,TkInt 5000,TkListLen 2,TkInt 8175950039059856398,TkInt 7517193088755259942,TkListLen 2,TkInt 6460320455910312492,TkInt 4127550193106455665,TkInt 2,TkInt 6,TkInt 2]
          Use --quickcheck-replay=26530 to reproduce.

[lehins]

It is actually consensus that produces this invalid CBOR 😄
Implementation is in this module: https://github.com/input-output-hk/ouroboros-consensus/blob/51da3876c01edc2eec250fdc998f6cb33cdc4367/ouroboros-consensus-cardano/src/shelley/Ouroboros/Consensus/Shelley/Ledger/Query/PParamsLegacyEncoder.hs#L100

All of our roundtrip tests now also validate through FlatTerm since this PR: #3769

So, unless we missed a test for a type defined in ledger, such failures should not be possible

[lehins]

I'd like to emphasis that I wanted to fix that bug in the NodeToClient protocol, but many people spoke against it, so we brought back the bug

[dnadales]

It is actually consensus that produces this invalid CBOR 😄

🤦 Thanks! Let me check that.

All of our roundtrip tests now also validate through FlatTerm since this PR: #3769

🎉 Great news!

I'd like to emphasis that I wanted to fix that bug in the NodeToClient protocol, but many people spoke against it, so we brought back the bug

:) Sorry, which bug are you referring to?

[dnadales]

which bug are you referring to?

I guess it's related to this

[lehins]

It is actually related to this issue right here: CardanoSolutions/ogmios#314 (comment)

In other words I have fixed this issue that you are experiencing, but then it broke people's tools and we had to backport a fix that reintroduced a bug. So there is nothing we can really do, except deprecating the old query (or maybe just the NoteToClient protocol version) that relies on this bug.

In any case, I don't think there is really nothing to do about this on the ledger side, thus closing the issue.

[dnadales]

Thanks @lehins. We'll try to workaround this in Consensus.