Efficient partial reads of optional fields near the start of a large message

[Allybag]

So, I have a use case where I want to extract two strings from large JSON messages, where one will always be present, and the other is only present about 1% of the time. I think basically what I would want is something like

struct partial_struct
{
   std::string always{};
   std::optional<std::string> maybe{};
};
partial_struct obj{};
std::string s = R"({"always":"present","rest":"All sorts of other information and fields I don't care about...})";
glz::read<glz::opts{.partial_read = true}>(obj, std::string_view{s}.substr(0, N));

Where N is a number which I know will be < the min size of S and > the final index of the values of both always and maybe. The issue is that of course the first N chars of S are very unlikely to be valid JSON as it might end in the middle of some element or whatever.

[stephenberry]

Look into the docs for glz::lazy_json. I think this will suit you the best. If you have more questions, let me know!

[Allybag]

My understanding is that partial_read will short circuit and be fast in the case where both always and maybe are present, and that lazy_json will also be fast if I access maybe and then always and they are both present and I know that maybe will come before always, but if the field is not present in the JSON string then both will scan the entire large string right?
I guess if I know that there is another field X that I know will always be present and come after both maybe and always then I can iterate through a lazy_json till I see that field and then break?

[stephenberry]

I see where you are coming from and your initial question better now.

partial_read should be faster because you won't need multiple passes.

Why are you only passing in part of the buffer? Partial reading will short circuit after those two fields are read even if you have the whole buffer?

Are you trying to avoid parsing the whole buffer in the case where the maybe key doesn't exist? Do you know confidentiality the max sub string length?

[Allybag]

Yeah that's exactly my situation, I want to avoid parsing the whole buffer when the maybe key doesn't exist and I know that I will always have these fields within the first 100 chars if present.

[stephenberry]

I think you're best off doing two lazy_json passes on the truncated buffer (first 100 chars). The lazy parsing will skip until it finds the key of interest and then only parse that value. Since you only have a few chars to look through, this should be extremely fast even though part of the buffer will be parsed twice.

If you knew that the maybe key would always come after the always key, then you could use a single lazy_json pass that first parses the always and then searches for maybe in the truncated buffer.