I've watched developers waste two or three weeks on the wrong problem. They rotate User-Agents, set Accept-Language headers, maybe add a random sleep between requests. Then TikTok or Instagram blocks them within a day anyway and they can't figure out why. The reason is that real bot detection stopped caring about headers a long time ago. It happens at the TLS layer now, and the JavaScript execution layer, and sometimes both simultaneously.
Here's what actually matters in 2026.
Your TLS handshake is giving you away
Every HTTPS connection starts with a TLS handshake. During that handshake, your client announces which cipher suites, extensions, and compression methods it supports. That list is called a TLS fingerprint, and it varies by client. Chrome has one. Firefox has a different one. Python's requests library using urllib3 has its own - and it looks absolutely nothing like a browser.
Video platforms have seen billions of requests. They know what a requests fingerprint looks like. Cloudflare databases it. So does TikTok's in-house detection stack.
The practical fix is curl_cffi. It wraps curl-impersonate, which replays a real browser's TLS handshake at the C level. One line of change:
from curl_cffi import requests as cf_requests resp = cf_requests.get(url, impersonate="chrome120")
Honestly, this alone gets you past a lot of platforms. But TikTok after mid-2023, and Instagram now, have added a second layer on top of it.
JavaScript challenges are the actual wall
The hard platforms don't just check your TLS fingerprint. They run JavaScript challenges that probe things no HTTP header can fake: canvas rendering output, WebGL renderer string, installed font list, screen size vs window size ratio, whether specific browser APIs behave exactly the way a real browser would.
Raw Playwright without stealth patches fails these. The page loads, the challenge script runs, it finds something off, and instead of a clean 403 you get quietly redirected to a login screen or a CAPTCHA. Soft blocks are harder to detect in your code than hard ones.
Two things that help:
playwright-stealth patches the most obvious automation tells in Playwright - navigator.webdriver being set to true, missing Chrome-specific globals, broken permissions API behavior. It's not comprehensive but it handles the easy wins.
Camoufox is the more serious option. It's a Firefox fork built for stealth - patches go into the browser internals, not just JavaScript properties. If you need reliable evasion without forking your own browser, Camoufox is the best thing available right now.
Speed kills scraper sessions
Real people don't make 60 requests a minute. They barely make 6. Someone browsing TikTok on their phone loads maybe 2-3 pages per minute with long idle gaps in between while they actually watch the video.
Random delays aren't just a courtesy. They're a disguise. At minimum, time.sleep(random.uniform(2, 8)) between requests. Every 10-15 requests, a longer pause - 30 to 60 seconds. Yes, it feels painfully slow. Do it anyway. Some platforms track cumulative velocity across a session, not just per-request timing, so even perfectly spaced requests at too-high a rate will flag you eventually.
Residential proxies vs datacenter proxies
IP bans happen, but they're usually not the first response - behavioral flags come first. Once you are IP-banned though, the range stays burned for a while.
Datacenter IPs don't work well for video platforms. AWS, GCP, Hetzner, DigitalOcean - their entire IP blocks are publicly listed. Platforms preemptively block them. Residential proxies come from real home ISP connections, which look like actual users. The detection systems were built to allow residential traffic through, not block it.
The tradeoff is cost. Residential proxy bandwidth runs $5-15 per GB, and scraping video metadata can burn through more than you'd expect. For a few hundred requests a day, a single reliable residential IP is fine. At scale you need rotation - either your own pool or a proxy service with built-in rotation logic.
Cookie harvesting and session warming
Fresh sessions get worse results. That's just the reality on TikTok especially. A brand new IP with no cookies and no session history gets different API responses than a session that's browsed around for a bit, accepted a cookie banner, has some watch history, and generally looks like an actual account.
Session warming means running a headless browser, actually navigating the platform for a few minutes - scroll, click through a couple videos, let the page load properly - then exporting those cookies and injecting them into your actual scraping session. It's annoying to build. It works.
Rotate the cookie sets. Don't send the same session cookie through 5,000 requests or it ages out and gets flagged. Treat them like short-lived credentials.
Patterns that get you caught fast
From what I've seen, the fastest path to a ban isn't aggressive request rates. It's robotic behavioral patterns.
Visiting pages in the exact same order every time. Never backtracking. Never pausing mid-session. Dismissing cookie consent dialogs immediately (real users sit on those for 5-20 seconds). No mouse movement events before navigation. No scroll events. Headless browsers with no user interaction produce clean, sparse event logs that are trivially different from a real browser session.
Some randomness in your navigation path goes a long way. It doesn't have to be perfect. It just has to not be a perfectly deterministic sequence.
When to use the private API instead
Most platforms have a private mobile API their official app uses. It's almost always less aggressively defended than the web frontend - partly because it wasn't designed to handle adversarial clients, partly because it's harder to instrument without the browser JavaScript environment.
Getting those endpoints means intercepting the official app's traffic with something like mitmproxy or Charles Proxy. More upfront work, but the requests are faster, simpler, and they look exactly like official app traffic because they are.
For video specifically - yt-dlp has already reverse-engineered the extraction logic for dozens of platforms and actively maintains it as APIs change. If you need the video file, use yt-dlp as a library. Build your own extractor only when you need data it doesn't provide: comments, engagement signals, recommendation graphs. That's where it's actually worth the effort.