Consuming TCP Stream

[mistal-distal]

This is a question not an issue. I saw various issues back from this time last year about streams. Would the stream parser be able to parse/consume a TCP stream's buffer?

For example, it parses to the end of the buffer and the completing JSON isn't queued yet so it can rewind on buffer when new data has been queued.

[nlohmann]

Did you find a solution?

[mistal-distal]

I didn't I accidentally closed this without noticing. What would you recommend?

Right now I have a tcp stream coming into a char[ ] buffer

[nlohmann]

Could you describe your program a bit more? What is the structure you are parsing? How does it look like when you call the parser? How would further data be added?

[mistal-distal]

Long story short the server is continually sending JSON over TCP, which does not guarantee everything gets sent as whole. The buffer contains the sent json message parts and appends them to a string (std::string).

I'd like the client to be able to parse the json on the string and delete things it has parsed. For example here's the receive code block from the TCP stream.

while ((nDataLength = recv(Socket, buffer, 10000, 0)) > 0) {
		
		}

On a console test program I am able to parse my test strings appropriately. For example:
Test String:

{"ssid":"test", "power":1,"channel":20}

json j;
std::cin >> j;

	parseMe::WiFi w {
		j["ssid"].get<std::string>(),
		j["power"].get<int>(),
		j["channel"].get<int>()
	};

std::cout << w.ssid <<" " << w.power << " " <<  w.channel << std::endl;


namespace parseMe {

	struct WiFi {
		std::string ssid;
		int power;
		int channel;

	};
}

Ignore that last edit I made.

[jaredgrubb]

Are you receiving back-to-back json ("{...} {...} {....}") and you need to pull them apart, are you getting one very-large json ("[a,b,c,........]") and you want to consume "a" before you get the closing ']'?

[mistal-distal]

Back to back json exactly. Right now what I implemented is a size byte in between the json structures. Read size, buffer up a single json statement, parse, delete from buffer, rinse and repeat. It works well so far but I'm wondering if there's cleaner solutions.

[jaredgrubb]

I actually did this a couple years ago but used the "json11" library:

std::string parseError;
for(; iter != end; ) {
    size_t stopPos = 0;
    auto object = json11::Json::parse({iter, end-iter), stopPos, parseError);
    if (!parseError.empty()) { ...  break; }
    iter += stopPos;
}

I don't think nlohmann::json currently does this, but it probably would not be hard to add a form of parse that:

• allows a JSON object to end early
• return an iterator to the next unread position
• allow caller to distinguish between "early end-of-stream" failure from all the other parse errors; the first is expected as you are waiting for bytes off the socket, but you need to be able to detect the latter for when the JSON stream is really broken and you will never be able to recover.

[mistal-distal]

I've managed to get it working myself, I'm passing multiple different types of JSON structures and converting them to their arbitrary types. All is well

[g-arjones]

@DOGFIVE Could you please share your solution?

[madf]

@DOGFIVE Could you please share your solution?

I also faced this problem and found out that there is only one JSON parser suitable for this task: https://github.com/lloyd/yajl

It allows partial input and multiple root objects.

[manuelgavidia]

I've managed to get it working myself, I'm passing multiple different types of JSON structures and converting them to their arbitrary types. All is well

Too old thread, but same issue here today ... @mistal-distal the solution that worked for you was that shared by @jaredgrubb ?