fix(crw): skip non-dict data rows from fastCRW - #2008
Closed
Bartok9 wants to merge 1 commit into
Closed
Conversation
Comprehension called .get on every data[] item; non-dict rows crashed the full search. Require list+dict shape before reading url/markdown.
Owner
Bartok9
pushed a commit
to Bartok9/gpt-researcher
that referenced
this pull request
Aug 23, 2026
Consolidates 20 single-file PRs from @Bartok9 into one reviewable change. Every provider parsed its JSON response with strict indexing, so one malformed row from an upstream API aborted the whole search rather than skipping that row: non-dict entries, missing link/href/url keys, non-list result containers, and null nested objects. Squashed from assafelovic#1839 assafelovic#1852 assafelovic#1917 assafelovic#1918 assafelovic#1921 assafelovic#1922 assafelovic#1923 assafelovic#1931 assafelovic#1932 assafelovic#1933 assafelovic#1934 assafelovic#1937 assafelovic#1938 assafelovic#1940 assafelovic#1946 assafelovic#1947 assafelovic#1958 assafelovic#1999 assafelovic#2007 assafelovic#2013. Each was applied and run against the full suite individually before being stacked. Duplicates that targeted the same file (assafelovic#1914 assafelovic#1920 assafelovic#1962 assafelovic#2001 on searchapi, assafelovic#1971 on serper, assafelovic#2006 on serpapi, assafelovic#2008 on crw, assafelovic#2009 on getxapi, assafelovic#1995 on xquik, assafelovic#1998 on bing, assafelovic#2000 on bocha) are superseded by the version kept here. Co-Authored-By: Bartok9 <noreply@github.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
fastCRW
datalist entries are assumed dicts; non-dict rows crashed result normalization.Verification