Question: parsing void *

[urban-1]

Hi all,

Quick question: I am using librdkafka to receive messages whose payload is JSON string. The library returns the payload in a non-null terminated void* and provides a message->len() to indicate how many characters/bytes are valid. Is it possible to call something like: json::parse(void*, const size_t&)?

I see that the parser supports char* type but it overruns the string's length causing an overflow ...

The solutions I see are:

1. Provide a json::parse(void*, const size_t&) signature maybe by extending/overloading parse(T(&) array[N] ...)
2. Enable the parser to detect the closing } character and stop

At the moment I am using strncpy to populate a pre-allocated buffer and terminate it properly before passing it to parse. This is an extra copy operation which could be avoided...

Many thanks for this awesome library - let me know what you think,

Andreas

[nlohmann]

Are you sure that the memory pointed to by the void pointer is really a sequence of characters of the proper length?

[urban-1]

I suspect that the void* is being reused internally because strlen and message->len() report different values...

[nlohmann]

Can you try json::parse((char*)ptr, strlen((char*)ptr))?

[urban-1]

Give me an hour and I will post here

[urban-1]

(sry about that... meetings...)

I have tried the above but instead of using strlen (which I know is wrong) I use the message->len(). In any case though there is no such signature/function:

src/KafkaJsonInput.cpp:152:103: error: no matching function for call to ‘nlohmann::basic_json<>::parse(char*, size_t)’
                 e->json = json::parse(static_cast<char *>(message->payload()),  (size_t)message->len());

This is the feature I propose

[nlohmann]

Sorry, the second parameter has to be another pointer. This would be an example:

#include <iostream>
#include "json.hpp"

using json = nlohmann::json;

int main() {
    const char* text = "{\"foo\": 12}";
    void *input = (void*)text;
    
    json j = json::parse((char*)input, ((char*)input) + strlen((char*)input));
    std::cout << j << std::endl;
}

[urban-1]

Perfect! I completely missed that ...

char * payload = static_cast<char *>(message->payload());
json::parse(payload, payload + message->len());

Works nicely - I think this can close. Thanks a lot for your help!

[nlohmann]

Thanks for reporting back!

[ibc]

Sorry for commenting in a closed issue. May I know where in the API is documented this signature used above?:

json::parse((char*)begin, (char*)end);

This is exactly what I was looking for (I receive a continuos stream via TCP and I do know the JSON string message boundaries, so I know the begin and end pointers. However I'd like to understand where this API signature is defined. In the official doc I read:

This function reads from an iterator range of a container with contiguous storage of 1-byte values. Compatible container types include std::vector, std::string, std::array, std::valarray, and std::initializer_list. Furthermore, C-style arrays can be used with std::begin()/std::end(). User-defined containers can be used as long as they implement random-access iterators and a contiguous storage.

BTW I don't know what "C-style arrays can be used with std::begin()/std::end()" means.

[nlohmann]

This would be a special case of an iterator range, which is not documented. :-/

For a char[] array, you can use std::begin() and std::end() to get an iterator range:

#include "json.hpp"
#include <iostream>

using json = nlohmann::json;

int main() {
    char input[] = {'t', 'r', 'u', 'e'};
    json j = json::parse(std::begin(input), std::end(input));
    std::cout << j << std::endl;
}

Output: true.

I shall update the documentation together with #1405.

[ibc]

Thanks. As a side note, I think that parsing from a stream or a specific range in a char array should not be a "special case". At least in my experience this is a very common usage :)

[nlohmann]

I don't mean it is a special case at per se - it's rather a special case for an iterator range...

[ibc]

ah ok :)