Skip to content

Parse fast - #241

Open
fr-an-k wants to merge 3 commits into
zserge:masterfrom
fr-an-k:parse_fast
Open

Parse fast#241
fr-an-k wants to merge 3 commits into
zserge:masterfrom
fr-an-k:parse_fast

Conversation

@fr-an-k

@fr-an-k fr-an-k commented Mar 5, 2025

Copy link
Copy Markdown

Greatly accellerates parsing by memoizing toksuper per depth level.
Some tests were also fixed due to conflicts.
I can't spend anymore time on this.

Comment thread jsmn.h
jsmntok_t *tokens, const unsigned int num_tokens) {
unsigned int max_depth = JSMN_MAX_FAST_DEPTH;
unsigned int toksupers[JSMN_MAX_FAST_DEPTH];
jsmn_parse_fast(parser, js, len, tokens, num_tokens, &max_depth, toksupers);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is missing the return

@pt300

pt300 commented Mar 19, 2025

Copy link
Copy Markdown
Collaborator

So instead of parent_links in the tokens themselves, we have a separate side-array used only during parsing to store the "parents stack". Cool idea. There's probably a cache argument to be made, but I don't think it's a concern for JSMN.

@fr-an-k

fr-an-k commented Mar 19, 2025

Copy link
Copy Markdown
Author

If you profile the previous code (with a larger JSON) almost 100% of the time is spent on looking for the parent, but the parent is already known so that's unnecessary work.
A linked list could be used instead of an array on the stack or heap, I didn't think of that in the moment.
If you utilize the token pool for that, then there is no need for a separate API function, so the the final PR would be very minimal.
I guess you just need to link to the last encountered opening { or [ token and they should link to their parent.

@pt300

pt300 commented Mar 19, 2025

Copy link
Copy Markdown
Collaborator

Did you try #define JSMN_PARENT_LINKS? How does your solution compare to that? That is the same thing, but done in the token pool itself. Its existence is not really well documented if at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants