Fix SearchTimeline 404 + defensive parsing fixes#419
Open
Fix SearchTimeline 404 + defensive parsing fixes#419
Conversation
X rotated the SearchTimeline GraphQL endpoint; the previous `flaR-PUMshxFWZWPNpq4zA` doc_id now returns 404 for every request. Replacing it with the current doc_id `R0u1RWRf748KzyGBXvOYRA` is not enough on its own — X also expanded the accepted feature set and added a new field to the variables. Without all three changes in sync the endpoint responds 404 (not a partial shape), which made `client.search_tweet()` completely unusable. Changes: - `constants.py`: add `SEARCH_TIMELINE_FEATURES` (37 flags), the current set captured from a live x.com DevTools request. - `gql.py::Endpoint.SEARCH_TIMELINE`: point at the new doc_id. - `gql.py::GQLClient.search_timeline`: add `withGrokTranslatedBio: True` to variables and use `SEARCH_TIMELINE_FEATURES` instead of the generic `FEATURES` constant. Introducing a dedicated feature constant (instead of extending `FEATURES` in place) avoids side effects on every other endpoint that still consumes the original set.
X serves two shapes for the trailing cursor entry in TweetDetail: - legacy: `entries[-1].content.itemContent.value` - current: `entries[-1].content.value` (TimelineTimelineCursor without an itemContent wrapper) The current code reads the legacy path unconditionally and raises `KeyError: 'itemContent'` for any tweet returned with the new shape. That aborts the whole `get_tweet_by_id` call before `tweet.replies` is populated, so callers see an exception even though the reply entries were already parsed successfully above — pagination of *further* replies is the only thing that actually needs the cursor. Read both shapes, and fall back to `_fetch_more_replies = None` when neither is present. The function now returns successfully in all three cases (legacy cursor, new cursor, no cursor), and only pagination is disabled when the shape is unknown.
When `request()` sees a 429 it calls `_get_user_state()` to distinguish `TooManyRequests` from `AccountSuspended`. But `_get_user_state()` itself makes an HTTP call routed back through `request()`, and X rate limits the *account* (not per-endpoint), so the nested call also returns 429 — we re-enter the same branch and recurse until Python raises `RecursionError`. The end result is that an ordinary rate-limit gets masked by a `RecursionError` traceback from deep inside the library, which is misleading and hard to diagnose in user code. Trap any exception from the nested `v11.user_state()` call and fall back to 'normal'. The outer `request()` still raises `TooManyRequests` for the original 429 — callers see the correct exception. The only cost is that we may miss a suspension signal during a rate-limit window, but the next non-throttled call will detect it correctly.
ClientTransaction.init() fails with "Couldn't get KEY_BYTE indices" on
every run against the live x.com homepage, which in turn causes the
generator to fall back to a dummy X-Client-Transaction-Id header. The
dummy value is accepted by some endpoints (HomeTimeline, UserTweets)
but rejected with HTTP 404 by others (notably SearchTimeline) — a
difference which is invisible from the traceback and very hard to
debug in user code.
Root cause: webpack bundle layout change. The old layout shipped
`"ondemand.s":"HASH"` as a contiguous key/value pair, which the old
regex matched directly. The current layout splits that into two
entries keyed by a chunk id:
,123:"ondemand.s"...
...
,123:"7a3c9e1b"
Two-step lookup matches the chunk id from the `ondemand.s` label, then
finds the hash that was emitted against the same id. Mirrors the
strategy used by the `x-client-transaction-id` PyPI package against
the same bundle.
With this change, combined with the earlier SearchTimeline refresh,
`client.search_tweet()` returns results again on live X.
Reviewer's GuideRestores the SearchTimeline GraphQL endpoint, hardens tweet detail parsing, prevents recursive rate-limit handling in user state checks, and updates X client transaction hash extraction to match the current webpack bundle, collectively making search_tweet() and get_tweet_by_id() robust against recent x.com changes. Sequence diagram for updated search_timeline GraphQL requestsequenceDiagram
actor Developer
participant Client
participant ClientTransaction
participant XAPI as X_GraphQL_API
Developer->>Client: search_timeline(query, product, count, cursor)
Client->>Client: Build variables
Client->>Client: Set withGrokTranslatedBio = True
Client->>Client: Select SEARCH_TIMELINE_FEATURES
Client->>ClientTransaction: init() / generate_transaction_id()
ClientTransaction->>ClientTransaction: get_indices(home_page_response)
ClientTransaction->>XAPI: GET ondemand.s.<hash>a.js
XAPI-->>ClientTransaction: ondemand script with key byte indices
ClientTransaction-->>Client: X-Client-Transaction-Id
Client->>XAPI: gql_get(Endpoint.SEARCH_TIMELINE, variables, SEARCH_TIMELINE_FEATURES)
XAPI-->>Client: SearchTimeline response
Client-->>Developer: Parsed search results
Updated class diagram for Client and ClientTransaction methodsclassDiagram
class Client {
+async get_tweet_by_id(tweet_id)
+async search_timeline(query, product, count, cursor)
+async _get_user_state() Literal_normal_bounced_suspended
+async _get_more_replies(tweet_id, cursor)
}
class ClientTransaction {
+home_page_response
+async get_indices(home_page_response, session, headers)
+async init()
+generate_transaction_id()
}
class V11API {
+async user_state()
}
Client --> V11API : uses
Client --> ClientTransaction : uses for X_Client_Transaction_Id
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughAdded a Changes
Sequence Diagram(s)sequenceDiagram
participant Client as ClientTransaction
participant Page as Page HTML
participant HashSearch as Hash Resolver
participant CDN as CDN / ondemand.s.{HASH}a.js
Client->>Page: fetch page response text
Page-->>Client: response_text
Client->>HashSearch: extract numeric chunk id (ON_DEMAND_FILE_REGEX)
HashSearch-->>Client: chunk id
Client->>HashSearch: search response_text for hash (ON_DEMAND_HASH_PATTERN with id)
HashSearch-->>Client: hash (if found)
alt hash found
Client->>CDN: fetch ondemand.s.{HASH}a.js
CDN-->>Client: JS bundle
Client->>Client: parse indices from bundle (INDICES_REGEX)
else no hash
Client-->>Client: abort indices fetch / return no indices
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 1 issue, and left some high level feedback:
- The broad
except Exceptionin_get_user_staterisks silently treating non-rate-limit failures (e.g. JSON shape changes, network errors, or auth issues) as'normal'; consider narrowing the exception to the specific 429 path (or HTTP/transport errors) and at least logging unexpected exceptions for visibility. - In
get_tweet_by_id, you can further harden the cursor parsing by usingentries[-1].get('entryId', '')instead of indexing['entryId']directly, which will avoid aKeyErrorif X ever emits a cursor-like entry without that key. - The new
SEARCH_TIMELINE_FEATURESlargely overlaps with the globalFEATURES; consider building it from a shared base (e.g.{**FEATURES, **overrides}) to reduce the chance of configuration drift when feature flags change in other endpoints.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The broad `except Exception` in `_get_user_state` risks silently treating non-rate-limit failures (e.g. JSON shape changes, network errors, or auth issues) as `'normal'`; consider narrowing the exception to the specific 429 path (or HTTP/transport errors) and at least logging unexpected exceptions for visibility.
- In `get_tweet_by_id`, you can further harden the cursor parsing by using `entries[-1].get('entryId', '')` instead of indexing `['entryId']` directly, which will avoid a `KeyError` if X ever emits a cursor-like entry without that key.
- The new `SEARCH_TIMELINE_FEATURES` largely overlaps with the global `FEATURES`; consider building it from a shared base (e.g. `{**FEATURES, **overrides}`) to reduce the chance of configuration drift when feature flags change in other endpoints.
## Individual Comments
### Comment 1
<location path="twikit/client/client.py" line_range="4344-4347" />
<code_context>
+ # Trap any exception from the nested call and fall back to
+ # 'normal'. The original 429 is still raised by the outer
+ # `request()`; we just avoid turning it into a recursive crash.
+ try:
+ response, _ = await self.v11.user_state()
+ return response['userState']
+ except Exception:
+ return 'normal'
</code_context>
<issue_to_address>
**issue (bug_risk):** Catching `Exception` broadly in `_get_user_state` can hide unrelated bugs and runtime issues.
The recursion fix is good, but this `except Exception` will also hide programming and runtime errors (e.g. unexpected response structure, JSON parsing issues) that should fail loudly. Consider catching only the expected failure modes (e.g. rate-limit/network exceptions, `RecursionError`) or re-raising non-network exceptions so that genuine bugs aren’t masked:
```python
try:
response, _ = await self.v11.user_state()
return response['userState']
except ExpectedRateLimitErrors:
return 'normal'
```
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
twikit/client/client.py (1)
4332-4348: Recursion guard looks correct; consider narrowing theexcept.The fallback-to-
'normal'cleanly breaks the recursion whenv11.user_state()itself trips the 429 path insiderequest(), and the outerrequest()still re-raises the originalTooManyRequests. Behavior LGTM.Two optional notes (not blockers):
- The blanket
except Exception(RuffBLE001) will also swallow programmer errors such asKeyError: 'userState'if the response schema drifts, silently classifying a suspended account as'normal'. Consider narrowing to the exceptions you actually expect (TwitterException,httpx.HTTPError,KeyError,RecursionError) so genuine bugs surface.- A simple in-flight reentrancy guard (e.g. an
asyncio.Lockor a boolean flag set onself) would prevent the recursion at the source rather than relying on it raising — but the current approach is already sufficient for the stated bug.♻️ Optional narrower exception handling
- try: - response, _ = await self.v11.user_state() - return response['userState'] - except Exception: - return 'normal' + try: + response, _ = await self.v11.user_state() + return response['userState'] + except (TwitterException, RecursionError): + # Nested call hit a rate limit / recursion; assume normal so the + # outer request() raises the original TooManyRequests. + return 'normal'🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@twikit/client/client.py` around lines 4332 - 4348, The current broad except in _get_user_state swallows unrelated errors; change the handler in async def _get_user_state(self) to catch only expected exceptions (e.g., TwitterException, httpx.HTTPError, KeyError, RecursionError) instead of bare Exception so schema or programmer errors surface; locate the call to self.v11.user_state() inside _get_user_state and replace the blanket except with an except tuple listing those specific exception types and return 'normal' only for those cases.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@twikit/client/client.py`:
- Around line 1633-1650: get_tweet_by_id added defensive cursor extraction for
two cursor shapes but _get_more_replies still assumes the legacy shape and
unconditionally accesses entries[-1]['content']['itemContent']['value'], causing
KeyError for the newer flat shape; update _get_more_replies to mirror the same
logic used in get_tweet_by_id: inspect entries[-1].get('content') or {}, check
for content.get('itemContent') being a dict with 'value' and fall back to
content['value'] if present, then use that extracted value as next_cursor before
proceeding (refer to _get_more_replies, get_tweet_by_id, entries,
reply_next_cursor).
In `@twikit/x_client_transaction/transaction.py`:
- Around line 20-22: The regexes require a leading comma and the hash pattern
only allows double quotes; update ON_DEMAND_FILE_REGEX and
ON_DEMAND_HASH_PATTERN so the leading comma is optional (so entries at the start
like {123:"ondemand.s"} match) and both single and double quotes are accepted
for the hash; specifically change ON_DEMAND_FILE_REGEX to make the leading comma
optional and to use ['"] for quotes around ondemand.s, and change
ON_DEMAND_HASH_PATTERN to make its leading comma optional/allow start-of-object
and to accept either single or double quotes around the captured hex group
(while still interpolating the id placeholder {}).
---
Nitpick comments:
In `@twikit/client/client.py`:
- Around line 4332-4348: The current broad except in _get_user_state swallows
unrelated errors; change the handler in async def _get_user_state(self) to catch
only expected exceptions (e.g., TwitterException, httpx.HTTPError, KeyError,
RecursionError) instead of bare Exception so schema or programmer errors
surface; locate the call to self.v11.user_state() inside _get_user_state and
replace the blanket except with an except tuple listing those specific exception
types and return 'normal' only for those cases.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 677cb0db-3165-4a71-a268-5a09f8e82874
📒 Files selected for processing (4)
twikit/client/client.pytwikit/client/gql.pytwikit/constants.pytwikit/x_client_transaction/transaction.py
Follow-up to the earlier `get_tweet_by_id` fix. The new flatter
`content.value` cursor shape bypasses the parent call just fine, but
`_fetch_more_replies` is still bound to `_get_more_replies`, and that
method did an unconditional legacy-shape access — so the very first
`await tweet.replies.next()` would reintroduce
`KeyError: 'itemContent'`, defeating the point of the earlier commit.
Mirror the same two-shape handling here, and also harden the entryId
read on the main call with `.get('entryId', '')` for good measure.
Two robustness issues with the initial webpack-layout fix, both raised
in review:
- The leading-boundary was a hard comma, which misses valid chunk
maps where `ondemand.s` happens to be the first key
(`{123:"ondemand.s",...}`). Widening to `[,{]` covers both.
- The hash-lookup pattern only accepted double-quoted values, while
the file-label pattern already accepted both quote styles. A
mismatch would have silently skipped the hash on single-quoted
builds. Accept `["']` on both sides.
Also escape the literal `{` in the hash pattern as `{{` and use a
named placeholder (`{chunk_id}`) so `str.format()` doesn't try to
parse the character class as an unnamed field (which raises
`ValueError: unexpected '{' in field name`).
Re-verified against live x.com: KEY_BYTE indices still extracted
correctly (`[22, 37, 13]`).
Review feedback: the bare `except Exception` was too wide — it would
also swallow genuine bugs (unexpected JSON shape, programming errors,
auth failures), making them indistinguishable from the rate-limit
scenario the guard is supposed to address.
Keep the trap targeted at the actual recursion-causing paths:
* `TooManyRequests` — the nested v11 call hits the same
account-level throttle and would otherwise re-enter this branch.
* `RecursionError` — belt-and-braces if a different code path ever
reintroduces the loop.
* `HTTPError` — transport-level failures on the nested request
should also fall back to 'normal' rather than masking the caller's
original 429 with a networking traceback.
Unexpected exceptions keep propagating, so real bugs surface instead
of being silently converted into a 'normal' user state.
Author
|
Thanks for the reviews, both points were valid. Pushed three follow-up commits:
Didn't adopt the |
Contributor
|
Tip For best results, initiate chat on the files or code changes.
On [approve] |
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
twikit/client/client.py (1)
195-198:⚠️ Potential issue | 🟠 MajorAvoid the recursive 429 loop before it reaches
RecursionError.Line 4360 still calls
V11Client.user_state(), which routes throughClient.request(). If that nested request also returns 429, Line 196 re-enters_get_user_state()repeatedly; the newRecursionErrorcatch only recovers after the recursion limit and many extra HTTP calls.🐛 Proposed fix: disable account-state rechecks for the nested user-state request
async def request( self, method: str, url: str, auto_unlock: bool = True, raise_exception: bool = True, + check_user_state: bool = True, **kwargs ) -> tuple[dict | Any, Response]: @@ elif status_code == 429: - if await self._get_user_state() == 'suspended': + if check_user_state and await self._get_user_state() == 'suspended': raise AccountSuspended(message, headers=response.headers) raise TooManyRequests(message, headers=response.headers) @@ try: - response, _ = await self.v11.user_state() + response, _ = await self.v11.user_state(check_user_state=False) return response['userState'] - except (TooManyRequests, RecursionError, HTTPError): + except (TooManyRequests, HTTPError): return 'normal'And pass the flag through the v11 wrapper:
-async def user_state(self): +async def user_state(self, **kwargs): return await self.base.get( Endpoint.USER_STATE, - headers=self.base._base_headers + headers=self.base._base_headers, + **kwargs )Also applies to: 4359-4363
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@twikit/client/client.py` around lines 195 - 198, The error path in Client.request -> status_code 429 calls self._get_user_state(), which itself uses V11Client.user_state() that routes back through Client.request and can cause a recursive loop of 429-handling calls; modify _get_user_state (and the V11Client.user_state wrapper) to accept and pass a flag like allow_account_state_recheck=False so that when you call V11Client.user_state from within the 429 handling path you pass allow_account_state_recheck=False to prevent re-invoking Client.request’s 429 handler; update the V11 wrapper signature and calls accordingly so nested user-state lookups do not trigger another account-state recheck and thus avoid recursion.
🧹 Nitpick comments (1)
twikit/x_client_transaction/transaction.py (1)
75-87: Consider distinguishing the "no hash found" failure mode.When
on_demand_filematches buthash_matchisNone(e.g., the bundle layout shifts again or the chunk id appears in the name map but its hash is emitted from a different chunk file), execution silently falls through to the genericraise Exception("Couldn't get KEY_BYTE indices")at line 87. Since this exception propagates frominit()straight up throughrequest()inclient.pywithout any fallback, a more specific message (e.g., including the resolvedchunk_id) would make future bundle-layout regressions much easier to diagnose from user bug reports.♻️ Suggested diagnostic improvement
if hash_match: on_demand_file_url = ( f"https://abs.twimg.com/responsive-web/client-web/" f"ondemand.s.{hash_match.group(1)}a.js" ) on_demand_file_response = await session.request( method="GET", url=on_demand_file_url, headers=headers) key_byte_indices_match = INDICES_REGEX.finditer( str(on_demand_file_response.text)) for item in key_byte_indices_match: key_byte_indices.append(item.group(2)) + else: + raise Exception( + f"Couldn't resolve ondemand.s hash for chunk id {chunk_id}; " + "webpack bundle layout may have changed again." + )🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@twikit/x_client_transaction/transaction.py` around lines 75 - 87, The current flow can silently hit the generic "Couldn't get KEY_BYTE indices" when on_demand_file exists but hash_match is None; update the logic in init() where hash_match is used (around on_demand_file_url and key_byte_indices collection) to detect the hash_match==None case and raise a more specific exception that includes the chunk_id (and optionally the on_demand_file_url) so callers like request() in client.py receive a clear diagnostic; ensure you still raise the original "Couldn't get KEY_BYTE indices" only if no key_byte_indices were collected after attempting to fetch and parse with INDICES_REGEX.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Outside diff comments:
In `@twikit/client/client.py`:
- Around line 195-198: The error path in Client.request -> status_code 429 calls
self._get_user_state(), which itself uses V11Client.user_state() that routes
back through Client.request and can cause a recursive loop of 429-handling
calls; modify _get_user_state (and the V11Client.user_state wrapper) to accept
and pass a flag like allow_account_state_recheck=False so that when you call
V11Client.user_state from within the 429 handling path you pass
allow_account_state_recheck=False to prevent re-invoking Client.request’s 429
handler; update the V11 wrapper signature and calls accordingly so nested
user-state lookups do not trigger another account-state recheck and thus avoid
recursion.
---
Nitpick comments:
In `@twikit/x_client_transaction/transaction.py`:
- Around line 75-87: The current flow can silently hit the generic "Couldn't get
KEY_BYTE indices" when on_demand_file exists but hash_match is None; update the
logic in init() where hash_match is used (around on_demand_file_url and
key_byte_indices collection) to detect the hash_match==None case and raise a
more specific exception that includes the chunk_id (and optionally the
on_demand_file_url) so callers like request() in client.py receive a clear
diagnostic; ensure you still raise the original "Couldn't get KEY_BYTE indices"
only if no key_byte_indices were collected after attempting to fetch and parse
with INDICES_REGEX.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: dbc5109c-0e1b-48c8-b654-e1261f89c8dc
📒 Files selected for processing (2)
twikit/client/client.pytwikit/x_client_transaction/transaction.py
ctrl-alt-raccoon
pushed a commit
to ctrl-alt-raccoon/twikit
that referenced
this pull request
Apr 18, 2026
- PATCHES.md documents the cherry-picks that diverge this branch from upstream d60/twikit:main. Anyone landing here can see at a glance which commits are ours and why each exists. - .github/workflows/drift-check.yml runs weekly, checks whether any of the upstream PRs listed in PATCHES.md have been merged, and opens an issue here when one has so we know to retire the cherry-pick. Cherry-picks currently carried (see PATCHES.md for detail): - d60#419 (7 commits) — SearchTimeline queryId refresh + defensive parsing + 429 recursion guard + ondemand.s extractor. - d60#418 (1 commit) — .get() for optional fields in User.__init__ and Client.request.
Follow-up to review feedback. The earlier `RecursionError` trap catches the crash, but only *after* Python has walked the whole recursion limit — so we still do ~1000 redundant HTTP calls against X every time an account gets rate-limited. That's both slow and a great way to earn an extra throttling window. Cleaner: add a `check_user_state` keyword to `Client.request()`, default `True`. The 429 recovery branch consults it before calling `_get_user_state()`. When `_get_user_state()` itself dispatches the nested `v11.user_state()` GET, it sets `check_user_state=False` so that if the nested call also returns 429, `request()` raises `TooManyRequests` straight away — no recursion, no retries, correct exception bubbles up on the first round trip. The `HTTPError` trap stays: a transport-level failure on the nested GET still shouldn't mask the original 429 for the caller. The `RecursionError` trap is gone — that code path no longer exists once the flag breaks the loop at the source. Threading the flag through required widening `V11Client.user_state` to accept and forward `**kwargs` to `base.get`.
Nitpick follow-up. The previous two-step flow silently fell through to "Couldn't get KEY_BYTE indices" when the label regex matched but the hash pattern didn't, conflating two very different failure modes: - label missing → webpack bundle layout drifted, both regexes need a look. - label found, hash missing → only the hash pattern needs adjusting (typically a quote-style or chunk-id format change). Splitting the two lets a future X-side rotation be diagnosed from the traceback alone, instead of having to instrument `get_indices`. The chunk id is included in the message so it's clear which mapping to inspect in the page source. The generic "Couldn't get KEY_BYTE indices" stays for the case where indices parsing itself fails (bundle contents changed).
Author
|
Thanks @coderabbitai. Both valid, pushed:
Both re-verified live: |
Contributor
|
Tip For best results, initiate chat on the files or code changes.
Both look good — approving. [approve] |
|
Can someone merge these fixes? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four independent fixes that together restore
client.search_tweet()and hardenget_tweet_by_id()against schema drift on live x.com. Each commit is self-contained and can be reviewed / reverted independently.Refresh SearchTimeline queryId/features/variables— X rotated the SearchTimeline GraphQL endpoint. Old doc_id (flaR-PUMshxFWZWPNpq4zA) now 404s, and X also expanded the accepted feature set and added a newwithGrokTranslatedBiovariable. Without all three in sync the endpoint returns 404 wholesale (not a partial shape), sosearch_tweet()has been completely non-functional on the current x.com. Adds a dedicatedSEARCH_TIMELINE_FEATURESconstant instead of extending the globalFEATURES— avoids side effects on every other endpoint.Defensively parse trailing cursor in get_tweet_by_id— X serves two shapes for the trailing cursor entry in TweetDetail: legacycontent.itemContent.valueand the newer flattercontent.value. The current code reads the legacy path unconditionally and raisesKeyError: 'itemContent'for any tweet served with the new shape — which aborts the wholeget_tweet_by_id()call beforetweet.repliesis populated, even though the reply entries themselves were parsed fine. Read both shapes; fall back to_fetch_more_replies=Nonewhen neither is present (pagination of further replies is the only thing actually affected).Guard against recursion in _get_user_state on rate-limit—request()calls_get_user_state()on 429 to distinguishTooManyRequestsfromAccountSuspended. But that call routes back throughrequest(), and X rate-limits the account (not per-endpoint), so the nested call also 429s and we recurse until Python raisesRecursionError. The real 429 is thus masked by an unrelated crash. Trap exceptions from the nestedv11.user_state()call and fall back to'normal'; the outerrequest()still raisesTooManyRequestscorrectly.Update ondemand.s hash extraction for current webpack bundle—ClientTransaction.init()fails withCouldn't get KEY_BYTE indiceson every run, causing the generator to fall back to a dummyX-Client-Transaction-Id. The dummy is accepted by HomeTimeline/UserTweets but rejected with 404 by SearchTimeline (selective endpoint enforcement, nearly invisible from the traceback). Root cause is a webpack bundle layout change:\"ondemand.s\":\"HASH\"is no longer a contiguous pair; the chunk id and the hash are keyed separately. Two-step lookup mirrors the strategy from thex-client-transaction-idPyPI package. After this fix, combined with (1), real search queries come back with real results.Test plan
KeyError('itemContent')fromget_tweet_by_id(); it now returns successfully with all replies parsed.ClientTransaction.init()against live x.com — indices are extracted correctly andgenerate_transaction_id()returns a valid 94-char token.client.search_tweet(\"streetwear\", \"Top\"),client.search_tweet(\"japanese fashion\", \"Top\"), and four more queries — each returns populated results where previously every query returned HTTP 404.RecursionError; post-fix the expectedTooManyRequestsis raised instead.No new dependencies. No breaking API changes.
Scope note
These are all defensive / data-plane fixes. No changes to public method signatures, no new features, no migrations.
Summary by Sourcery
Restore search timeline functionality against current x.com schema and harden tweet retrieval and client transaction handling against recent platform changes.
Bug Fixes:
Enhancements:
Summary by CodeRabbit
Bug Fixes
New Features