Recipes/Python requests

Extract video data with Python

Extract video data with Python: a requests example that submits a batch of videos, polls the job and saves each video's structured rows.

The code

import os, time, requests
 
API = "https://api.tubeextract.dev/v1"
H = {"X-API-Key": os.environ['TUBEEXTRACT_KEY']}
 
schema = [
    {"name": "topic", "type": "string",
     "description": "A subject the host discusses at length"},
]
 
# one call for the whole list, up to 200 videos
res = requests.post(API, headers=H,
                    json={"videos": urls, "multiple": True, "schema": schema})
res.raise_for_status()
job_id = res.json()["job_id"]
 
# Rows stay None until every video has settled. Every poll counts
# against the per-minute limit (10 on Free), so do not poll faster.
while True:
    time.sleep(8)
    res = requests.get(f"{API}/{job_id}", headers=H)
    if res.status_code == 429:
        time.sleep(int(res.headers.get("Retry-After", 60)))
        continue
    res.raise_for_status()
    job = res.json()
    if job["status"] not in ("queued", "processing"):
        break
 
# handed over once, so persist before anything else
for video in job["videos"]:
    if video["status"] == "done":
        save(video["video_id"], video["rows"])
    else:
        log(video["video_id"], video["error"]["code"])

Notes

One id covers the whole list, so there is one thing to poll however many videos went in. While it runs, counts on the same response shows how many have finished. Failures are logged instead of retried, since most job errors will not succeed a second time; only captions_fetch_failed and internal_error are worth resubmitting.

Was this page useful?Tell us what was missing

Node fetch

Go net/http