Not a valid type: “x” with byte array element in Headers #58

ghost · 2015-12-15T19:40:20Z

I am trying to fetch messages from queue using ruby amqp gems and getting error.

I have one byte array in header (i.e. actual record) which is must from development prospective and type cannot be changed. I have tried

qParser.pop do | payload| and consumer2.consume.on_delivery do |metadata, payload|

but same error in both issues. I found amqp protocol -> table raise this exception. The byte array type (header) should be supported as rabittMQ supports it.

Here is the error

ArgumentError: Not a valid type: "x"
Data: "\x00\x00\x064\fChannel-TypeS\x00\x00\x00\x03PHN\aApi-KeyS\x00\x00\x00\x03RTM\x0EContent-Lengthl\x00\x00\x00\x00\x00\x00\x01l\rActual-Recordx\x00\x00\x04{{\"custRoleTypeCd\":\"\",\"msgDlvryProcTs\":\"11/06/2015 05:16:22\",\"chnlTypeCd\":\"TMSG\",\"dlvryId\":\"0003713221\",\"mktgInd\":\"N\",\"vendrId\":\"11\",\"cntctPntAdrTxt\":\"3022000023\",\"sorId\":\"7\",\"elctrMsgFrmtDesc\":\"SMS\",\"msgCustId\":\"1000000165146\",\"msgLangCd\":\"\",\"quietPerdTznCd\":\"\",\"prodDesc\":\"MASTERCARD PLATINUM\",\"rplyToAdr\":\"\",\"cycleDt\":\"20151014\",\"last4DgtsTxt\":\"3599\",\"thldValTxt\":\"\",\"substitutionKeyValueData\":{\"AlertCode\":\"PAYPST\",\"CustomerFirstName\":\"Crysta\",\"VARBL_5_FCL_TXT\":\"298,35\",\"PaymentPostedDate\":\"OCT 14, 2015\",\"PaymentAmount\":\"298.35\",\"VARBL_9_ENG_TXT\":\"lloA7T%2FXblrp1Bc7aoIK0A%3D%3D\",\"productdescription\":\"MASTERCARD PLATINUM\",\"AccountEndingIn\":\"3599\",\"VARBL_10_FCL_TXT\":\"14 octobre 2015\",\"VARBL_10_ENG_TXT\":\"XqLuxBTABbIDvpw56ba34hF0imQ5McLi4fDdnotsiaY%3D\"},\"alrtTypeCd\":\"PAYPST\",\"quietPerdEndTm\":\"\",\"srcSysUnqId\":\"\",\"acctId\":\"10000371367\",\"evtRsltTypeDesc\":\"The system is preparing a message for delivery.\",\"tmplId\":\"10001\",\"loadTs\":\"11/06/2015 05:16:22\",\"fromAdr\":\"\",\"msgLobCd\":\"CARD\",\"wirlsPlanId\":\"\",\"quietPerdStrtTm\":\"\",\"elctrAdrTypeCd\":\"PHN\",\"subjLine\":\"\",\"oldDlvryId\":\"\",\"evtRsltReasId\":\"6020\",\"recProcId\":\"N\",\"sorCustId\":\"850784\"}\vcontentTypeS\x00\x00\x00\ntext/plain\nSet-CookieS\x00\x00\x00bJSESSIONID=91EED106D2925239197C35CF201727F4; Path=/int-dms-content-assembly-web/; Secure; HttpOnly\x06ServerS\x00\x00\x00\nCapitalOne\rCache-ControlS\x00\x00\x00.no-cache, no-store, max-age=0, must-revalidate\x06PragmaS\x00\x00\x00\bno-cache\x04Datel\x00\x00\x01Q\xA1\xF5\xB80\x13originalContentTypeS\x00\x00\x00\x10application/json\x0Fhttp_statusCodeS\x00\x00\x00\x03200\fContent-TypeS\x00\x00\x00\x10application/json"
Unprocessed data: "\x00\x00\x04{{\"custRoleTypeCd\":\"\",\"msgDlvryProcTs\":\"11/06/2015 05:16:22\",\"chnlTypeCd\":\"TMSG\",\"dlvryId\":\"400000000003713221\",\"mktgInd\":\"N\",\"vendrId\":\"11\",\"cntctPntAdrTxt\":\"3022000023\",\"sorId\":\"7\",\"elctrMsgFrmtDesc\":\"SMS\",\"msgCustId\":\"1000000165146\",\"msgLangCd\":\"\",\"quietPerdTznCd\":\"\",\"prodDesc\":\"MASTERCARD PLATINUM\",\"rplyToAdr\":\"\",\"cycleDt\":\"20151014\",\"last4DgtsTxt\":\"3599\",\"thldValTxt\":\"\",\"substitutionKeyValueData\":{\"AlertCode\":\"PAYPST\",\"CustomerFirstName\":\"Crysta\",\"VARBL_5_FCL_TXT\":\"298,35\",\"PaymentPostedDate\":\"OCT 14, 2015\",\"PaymentAmount\":\"298.35\",\"VARBL_9_ENG_TXT\":\"lloA7T%2FXblrp1Bc7aoIK0A%3D%3D\",\"productdescription\":\"MASTERCARD PLATINUM\",\"AccountEndingIn\":\"3599\",\"VARBL_10_FCL_TXT\":\"14 octobre 2015\",\"VARBL_10_ENG_TXT\":\"XqLuxBTABbIDvpw56ba34hF0imQ5McLi4fDdnotsiaY%3D\"},\"alrtTypeCd\":\"PAYPST\",\"quietPerdEndTm\":\"\",\"srcSysUnqId\":\"\",\"acctId\":\"10000371367\",\"evtRsltTypeDesc\":\"The system is preparing a message for delivery.\",\"tmplId\":\"10001\",\"loadTs\":\"11/06/2015 05:16:22\",\"fromAdr\":\"\",\"msgLobCd\":\"CARD\",\"wirlsPlanId\":\"\",\"quietPerdStrtTm\":\"\",\"elctrAdrTypeCd\":\"PHN\",\"subjLine\":\"\",\"oldDlvryId\":\"\",\"evtRsltReasId\":\"6020\",\"recProcId\":\"N\",\"sorCustId\":\"850784\"}\vcontentTypeS\x00\x00\x00\ntext/plain\nSet-CookieS\x00\x00\x00bJSESSIONID=91EED106D2925239197C35CF201727F4; Path=/int-dms-content-assembly-web/; Secure; HttpOnly\x06ServerS\x00\x00\x00\ne\rCache-ControlS\x00\x00\x00.no-cache, no-store, max-age=0, must-revalidate\x06PragmaS\x00\x00\x00\bno-cache\x04Datel\x00\x00\x01Q\xA1\xF5\xB80\x13originalContentTypeS\x00\x00\x00\x10application/json\x0Fhttp_statusCodeS\x00\x00\x00\x03200\fContent-TypeS\x00\x00\x00\x10application/json"
Offset: 80
Total size: 1588
Processed data: {"Channel-Type"=>"PHN", "Api-Key"=>"RTM", "Content-Length"=>364}

The text was updated successfully, but these errors were encountered:

michaelklishin · 2015-12-15T20:05:16Z

First of all, there are no byte arrays in Ruby. Rather, Ruby values are strings that are passed around as byte arrays. So that part is supported in the most straightforward way possible.

You are not providing any code examples but the issue can be value-specific.

Please post a code sample we can run to reproduce.

michaelklishin · 2015-12-15T20:05:45Z

Oh, and knowing what exact version of both gems you use would be useful, too.

michaelklishin · 2015-12-15T20:07:55Z

A quick glance at value encoder, value decoder, and table type constants suggests that the payload comes from a different client. amq-protocol indeed does not use TYPE_BYTE_ARRAY for serialisation.

So what client is that?

ghost · 2015-12-15T20:30:56Z

Here is the code:

MoreInfo.xlsx

Attached file has more info..

Versions:

amq-protocol (2.0.0, 1.9.2)
amqp (1.5.1)

I have attached a sample message from my queue...

ghost · 2015-12-15T20:39:14Z

The above code snippet worked fine with simple queue which did not have Actual-Record element. However with actual record header is processed till channel type and then raise exception...

i.e.

Api-Key:    RTM
Channel-Type:   EML
Actual-Record:  {"msgDlvryProcTs":"11/06/2015 05:16:22","custRoleTypeCd":"","chnlTypeCd":"","dlvryId":"03713247","mktgInd":"N","vendrId":"1","sorId":"7","cntctPntAdrTxt":"[email protected]","elctrMsgFrmtDesc":"TEXT","msgCustId":"1000000165146","msgLangCd":"","quietPerdTznCd":"","prodDesc":"MASTERCARD PLATINUM","cycleDt":"20151014","rplyToAdr":"","last4DgtsTxt":"3599","thldValTxt":"","substitutionKeyValueData":{"AlertCode":"PAYPST","CustomerFirstName":"Crysta","VARBL_5_FCL_TXT":"298,35","PaymentPostedDate":"OCT 14, 2015","PaymentAmount":"298.35","VARBL_9_ENG_TXT":"lloA7T%2FXblrp1Bc7aoIK0A%3D%3D","productdescription":"MASTERCARD PLATINUM","AccountEndingIn":"3599","VARBL_10_FCL_TXT":"14 octobre 2015","VARBL_10_ENG_TXT":"XqLuxBTABbIDvpw56ba34hF0imQ5McLi4fDdnotsiaY%3D"},"alrtTypeCd":"PAYPST","quietPerdEndTm":"","srcSysUnqId":"","acctId":"10000371367","evtRsltTypeDesc":"The system is preparing a message for delivery.","tmplId":"10001","loadTs":"11/30/2015 05:16:22","fromAdr":"","wirlsPlanId":"","msgLobCd":"CARD","quietPerdStrtTm":"","elctrAdrTypeCd":"EML","subjLine":"","oldDlvryId":"","evtRsltReasId":"6020","recProcId":"N","sorCustId":"850784"}
contentType:    text/plain
Content-Type:   application/json

michaelklishin · 2015-12-15T21:43:26Z

@mohitjagga this is the code you consume with. What code publishes messages?

ghost · 2015-12-15T21:47:14Z

@michaelklishin that code is springXD code. Developed by the Java developers and I am consuming the streams created by their modules by creating taps on streams.

michaelklishin · 2015-12-15T21:54:11Z

@mohitjagga alright. Last question: what do you expect to come out on the Ruby end where Java publishers use byte arrays? Ruby doesn't have byte arrays.

ghost · 2015-12-15T22:06:16Z

@michaelklishin Correct me if I am wrong... You are saying there is no concept of byte array in Ruby and Ruby will not handle it.

However I am using queue.pop(payload)... I want some way I can fetch payload without even bother about headers. What is happening => it inspect headers and throws exception. Is it possible to fetch payload in this scenario where we have byte array in header... as of now all amqp methods fails with same error and I am stuck with as even though I do not need headers...

michaelklishin · 2015-12-15T22:11:01Z

There is no data structure for byte and byte array the same way they are in Java. Byte arrays can be turned into strings in Ruby land. The question is what most Ruby users would expect.

You can't consume a message without consuming (and parsing) its headers.

ghost · 2015-12-15T22:12:23Z

Thanks a lot for your help... I will see if I can use Java for this

michaelklishin · 2015-12-15T22:14:58Z

Hold on. There is a missing deserialisation feature in this library: most people never hit it because many popular programming languages don't operate byte buffers, and thus don't pass them around, or at least not in headers.

I gather the best way to treat byte buffers in Ruby would be to turn them into strings in binary encoding.

ghost · 2015-12-15T22:16:13Z

I am sure Ruby users will need some other way to handle byte array. Converting into string will increase the size of header. We are using byte array as size of actual-record header is big and string cannot handle that.

ghost · 2015-12-15T22:21:38Z

@michaelklishin that would be awesome :-)

michaelklishin · 2015-12-15T22:37:23Z

Most Ruby users wouldn't even say with confidence what a byte array is. I'm going to add byte array deserialisation that produces strings (much like message bodies are strings, even though they really are byte streams, and byte[]s in Java, which is quite close).

garyrussell · 2015-12-15T23:37:35Z

I guess the only issue is if the ruby app publishes a reply, echoing the headers, the Java app would see the echoed header as a String instead of a byte[].

But I see there's not really much that can be done about that.

michaelklishin · 2015-12-16T00:24:52Z

@mohitjagga please try with amq-protocol master. While you are at it, give Bunny a try, since amqp gem use is highly discouraged for new projects these days.

ghost · 2015-12-18T20:30:59Z

@michaelklishin It worked like a charm. We are doing the same in out app.. byte array is just for transfer data and then we convert it to string.. byte array helps in transporting heavy data which string cant :-).. Thanks a lot for your help!

PS: Any idea when this fix will be available in rubygems.org?

michaelklishin · 2015-12-18T20:38:54Z

It already is.

## Changes between 2.0.0 and 2.1.0 ### Ruby Warnings Squashed Contributed by Akira Matsuda. GitHub issue: [#62](ruby-amqp/amq-protocol#62) ### Byte Array Decoding Byte array values in types now can be decoded (to the extent Ruby type system permits) by this library. GitHub issue: [#58](ruby-amqp/amq-protocol#58)

michaelklishin changed the title ~~Ruby AMQP Error : ArgumentError: Not a valid type: “x” with byte array element in Headers~~ Not a valid type: “x” with byte array element in Headers Dec 15, 2015

michaelklishin self-assigned this Dec 15, 2015

ghost closed this as completed Dec 15, 2015

michaelklishin reopened this Dec 15, 2015

michaelklishin closed this as completed in d50a116 Dec 16, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Not a valid type: “x” with byte array element in Headers #58

Not a valid type: “x” with byte array element in Headers #58

ghost commented Dec 15, 2015

michaelklishin commented Dec 15, 2015

michaelklishin commented Dec 15, 2015

michaelklishin commented Dec 15, 2015

ghost commented Dec 15, 2015 •

edited by ghost

Loading

ghost commented Dec 15, 2015

michaelklishin commented Dec 15, 2015

ghost commented Dec 15, 2015

michaelklishin commented Dec 15, 2015

ghost commented Dec 15, 2015

michaelklishin commented Dec 15, 2015

ghost commented Dec 15, 2015

michaelklishin commented Dec 15, 2015

ghost commented Dec 15, 2015

ghost commented Dec 15, 2015

michaelklishin commented Dec 15, 2015

garyrussell commented Dec 15, 2015

michaelklishin commented Dec 16, 2015

ghost commented Dec 18, 2015

michaelklishin commented Dec 18, 2015

Not a valid type: “x” with byte array element in Headers #58

Not a valid type: “x” with byte array element in Headers #58

Comments

ghost commented Dec 15, 2015

michaelklishin commented Dec 15, 2015

michaelklishin commented Dec 15, 2015

michaelklishin commented Dec 15, 2015

ghost commented Dec 15, 2015 • edited by ghost Loading

ghost commented Dec 15, 2015

michaelklishin commented Dec 15, 2015

ghost commented Dec 15, 2015

michaelklishin commented Dec 15, 2015

ghost commented Dec 15, 2015

michaelklishin commented Dec 15, 2015

ghost commented Dec 15, 2015

michaelklishin commented Dec 15, 2015

ghost commented Dec 15, 2015

ghost commented Dec 15, 2015

michaelklishin commented Dec 15, 2015

garyrussell commented Dec 15, 2015

michaelklishin commented Dec 16, 2015

ghost commented Dec 18, 2015

michaelklishin commented Dec 18, 2015

ghost commented Dec 15, 2015 •

edited by ghost

Loading