Trying this for monitoring requests
@browser()
def scrape_responses_task(driver: Driver, data):
# Define a handler function that will be called after a response is received
def after_response_handler(
request_id: str,
response: cdp.network.Response,
event: cdp.network.ResponseReceived,
):
# Extract URL, status, and headers from the response
url = response.url
status = response.status
headers = response.headers
# Print the response details
print(
"after_response_handler",
{
"request_id": request_id,
"url": url,
"status": status,
"headers": headers,
},
)
# Append the request ID to the driver's responses list
driver.responses.append(request_id)
# Register the after_response_handler to be called after each response is received
driver.after_response_received(after_response_handler)
# Navigate to the specified URL
driver.get("https://example.com/")
# Collect all the responses that were appended during the navigation
collected_responses = driver.responses.collect()
# Save it in output/scrape_responses_task.json
return collected_responses
Looked at whats available in the response object, the response payload/body/text is not available ?
would really appreciate any help how to get the response data ?
Trying this for monitoring requests
Looked at whats available in the response object, the response payload/body/text is not available ?
would really appreciate any help how to get the response data ?