4.5B Posts Scraped from TikTok

What this is

Almost every TikTok scraper you will find drives a headless browser or hits the public web endpoints. Both are the wrong layer: slow, fragile, and missing most of the interesting fields. The Android app does not use either. It talks to a private HTTP+JSON API, the same one com.zhiliaoapp.musically hits when you scroll, and that API is fast, stable, and returns far more.

Getting into it is the hard part, and it is hard in a specific way. Four completely unrelated things have to be right at once: a device credential TikTok issued, a valid request signature, the correct regional host, and a TLS handshake that looks like a phone.

Get any one of them wrong and you receive the identical response: a clean HTTP 200 with an empty body. No error message. No status code. Your HTTP client reports success, your logs stay green, and your database fills with nothing. There is no signal telling you which of the four you are standing at.

This article walks through all four, then documents the 24 endpoints that come out the other side. It names the primitives, shows the real pipeline and gives measured numbers rather than claims.

None of the four has a feedback loop. A wrong rotation constant, a wrong byte order, a wrong host and a wrong cipher suite in the handshake all produce the same well-formed request and the same empty response, so there is no error to bisect on and no partial credit.

Anatomy of a request

Before anything else, here is what one of these requests actually looks like. This is a real call, with the identifying values shortened:

GET /aweme/v1/aweme/post/
    ?# ── what you are asking for ──────────────────────────────
     source=0
    &user_id=6744630345964389381
    &count=20
    &max_cursor=1751028792000
    &sort_type=0
    &# ── who is asking: 38 params, order matters ─────────────
     ts=1788361402&ac=mobile&ac2=lte
    &aid=473824                         # app id: TikTok Lite
    &iid=7680617333853718293            # install id  ← from register
    &device_id=7680616891110524437      # device id   ← from register
    &cdid=4a1d...                       # client-generated uuid
    &openudid=8f2c...                    # client-generated 16-hex
    &device_brand=Samsung&device_type=SM-A136U&os_version=12&os_api=30
    &resolution=1080*2280&dpi=440&host_abi=arm64-v8a
    &region=SG&carrier_region=SG&sys_region=SG&mcc_mnc=52506
    &language=ja&app_language=ja&locale=ja-SG&timezone_name=Asia%2FSingapore
    &version_name=32.8.2&version_code=320820&manifest_version_code=320820
    &_rticket=1788361402193&channel=googleplay&app_type=normal

Headers:
  user-agent:      com.ss.android.ugc.tiktok.lite/320802 (Linux; U; Android 12; ...)
  x-tt-trace-id:   00-6a9f...-6a9f...-01
  x-ss-req-ticket: 1788361402193
  x-khronos:       1788361402                    # timestamp
  x-ladon:         XKp9...                       # Speck-128/256
  x-argus:         cQqbRZm8k1x...                # the hard one
  x-gorgon:        0404b0d30000...               # legacy digest

Three things to notice, because each one bites later:

• Two thirds of the URL is device identity. Thirty-eight common parameters describe the handset, the carrier, the region and the app build. They are not decoration. The signature covers them.
• device_id and iid are issued by TikTok, not chosen by you. cdid and openudid you generate and submit at registration. Getting the distinction wrong is the first wall.
• Parameter order is fixed. The signature hashes the query string as a literal, so url.Values.Encode(), which sorts keys alphabetically, silently produces an invalid signature. In Go you have to build the query by hand.

The vocabulary, since it recurs throughout:

The first empty 200

A correctly implemented signer produces output that verifies against captured traffic, with parameters matching byte for byte. The response is still this:

$ curl -sD- -o /tmp/body "https://api16-normal-c-alisg.tiktokv.com/aweme/v1/user/profile/other/?..."
HTTP/1.1 200 OK
content-type: application/json
content-length: 0
x-tt-logid: 2026090117...
server: TLB

$ wc -c /tmp/body
0 /tmp/body

This is TikTok's soft block, and it is the single most important thing to understand about this API. It is not a 403. It is not a 429. It is not a challenge page. It is a successful HTTP response containing nothing.

Which means this code, which is what everyone writes first, is silently broken:

res = requests.get(url, headers=signed)
if res.ok:                      # True. Always true.
    store(res.json())           # {} stored, no exception

# six hours later: 400,000 rows in the database, all empty,
# nothing in the error log, dashboard green

It is expensive to debug because four unrelated failures produce it:

1. Your device was never activated (§ activation)
2. Your signature is wrong (§ X-Argus)
3. You are talking to the wrong regional host (§ regions)
4. Your TLS handshake looks like a server, not a phone (§ JA3)

There is nothing in the response to tell you which. You cannot bisect it by reading errors, because there are none. The only way through is to fix all four and measure each one in isolation.

Where device IDs come from

You cannot invent a device_id. TikTok issues it, from /service/2/device_register/ on its logging host, in exchange for a plausible handset.

The request body is a JSON document (app header, device header, custom block) encrypted with TTEncrypt (TikTok's own body cipher, a simple byte-level transform with a fixed key schedule) and posted as application/octet-stream;tt-data=a. It goes out with the full signature set, so you need working signing before you can get a device, and the signing needs a device. You bootstrap with the client-generated fields and zeros where the issued ones go.

The body's shape, with the parts that matter:

{
  "magic_tag": "ss_app_log",
  "header": {
    // app identity: must agree with the aid in the query string
    "aid": 473824, "package": "com.ss.android.ugc.tiktok.lite",
    "app_version": "32.8.2", "version_code": 320820,
    "sdk_version": "...", "git_hash": "...", "sig_hash": "...",

    // hardware: every field here has to be internally consistent
    "device_model": "SM-A136U", "device_brand": "Samsung",
    "device_manufacturer": "samsung", "cpu_abi": "arm64-v8a",
    "os_version": "12", "os_api": 30,
    "resolution": "2280*1080", "density_dpi": 440,
    "rom": "...", "rom_version": "...",

    // identity you generate and are about to trade in
    "cdid": "<uuid4>", "openudid": "<16 hex>",
    "clientudid": "<uuid4>", "google_aid": "<uuid4>",

    // region: carrier must plausibly exist in this country
    "region": "SG", "sim_region": "sg", "carrier": "Singtel",
    "mcc_mnc": "52506", "tz_name": "Asia/Singapore", "tz_offset": 25200,

    "custom": {
      "screen_width_dp": 408, "screen_height_dp": 883,
      "web_ua": "Dalvik/2.1.0 (Linux; U; Android 12; SM-A136U Build/...)",
      "apk_last_update_time": 1788361409271
    },
    "apk_first_install_time": 1788360902118
  },
  "_gen_time": 1788361402240
}

Every field there is checked against the others. A Samsung SM-A136U has a specific screen resolution, a specific DPI, a specific ABI, and shipped with a specific range of Android versions. It is sold on carriers in some countries and not others. A flagship handset on a network that never carried it is not a real phone, and the registration is refused.

Rather than generating these procedurally, I build them from a catalogue of ~250 real Android device profiles crossed with a carrier table of MCC/MNC pairs (roughly 2,000 rows, derived from public numbering-plan data). Pick a handset, pick a carrier that actually exists in the target country, fill in the coherent values.

A successful registration comes back with the two ids you needed:

{
  "device_id_str":  "7680616891110524437",
  "install_id_str": "7680617333853718293",
  "new_user": 1
}

Most implementations stop here.

The activation call

With registration working, most endpoints answered. Video listings, search, hashtags, sounds, all fine. But /aweme/v1/user/profile/other/, the full profile record, returned the empty 200 every single time, on every device I made, forever.

The obvious suspect is the signature, and it is the wrong one. The tell is that an older pool of devices, generated months earlier by different code, worked fine on that same endpoint with the same signer and the same parameters. The only difference was in how the devices had been created, and it came down to one extra HTTP call:

GET /service/2/app_alert_check/?<common params>
    &cronet_version=...&ttnet_version=...
    &tt_info=<base64url(TTEncrypt(<60-field key=value blob>))>

→ {"message":"success"}

That is it. It returns nothing you need. It looks like telemetry, and functionally it is telemetry. It is the call the real app makes on launch, before it requests any data.

That is what the call is for. A device that registered and then immediately started querying the API is, from ByteDance's side, an install that never launched. Registration alone does not make you a running app. The startup call does.

The tt_info blob is the interesting part of the request: about sixty key=value pairs (GAID, timezone, install id, device id, carrier, screen, ABI, locale, a request UUID) TTEncrypt-ed and base64url-encoded. It is the app reporting its full environment on startup. My guess, and it is only a guess, is that this is where the device gets marked as a real install rather than a bare registration; I have not tried to prove it, because the empirical result is unambiguous.

Proving a device before you use it

The activation fixed the profile endpoint, but it introduced a second-order problem: activation itself sometimes fails silently, and a device that failed activation looks exactly like a device that succeeded until you use it.

So generation does not end at activation. It ends with a real read against a known creator. If real content comes back, the device joins the pool. If not, it is thrown away. Not retried, not quarantined. Discarded.

func GenerateDevice(client *http.Client, country string) (map[string]any, error) {
    tmpl, err := NewAndroidTemplate()          // handset × carrier
    ...
    if err := registerDevice(client, tmpl); err != nil {
        return nil, fmt.Errorf("register: %w", err)
    }
    // Without this TikTok will not serve profile detail to a fresh device.
    if err := appAlertCheck(client, tmpl); err != nil {
        return nil, fmt.Errorf("activate: %w", err)
    }
    // Survivorship filter: only provably-capable devices enter the pool.
    if !profileCapable(client, tmpl) {
        return nil, errors.New("profile probe failed: device not capable")
    }
    return tmpl, nil
}

The three-stage pipeline. Roughly 60-95% of attempts survive it, depending almost entirely on proxy quality.

Without the filter you get a pool that is a mixture of working and quietly dead devices, and because dead devices return the empty 200, the same as every other failure, the pool degrades invisibly. Your success rate drifts down over days and there is nothing in the logs to explain it.

With the filter, the pool is uniformly capable by construction. Live health is visible from the running server:

$ curl -s localhost:8080/v1/devices | jq
{
  "live": 43,
  "generated_total": 43,
  "rejected_total": 2,
  "evicted_total": 0,
  "success_total": 177,
  "failure_total": 74,
  "generation_survival_rate": 0.9555
}

Inside X-Argus

X-Argus is not a hash of a string. Its plaintext is a protobuf message in proto3 wire format, varints and length-delimited fields, which is then run through a two-stage encryption pipeline.

The message carries, among other fields:

type Argus struct {
    Magic          int32      // fixed marker
    Version        int32
    Rand           int64      // per-request random, 0x10000000..0xFFFFFFFF
    MsAppID        string     // "1233" / "473824"
    LicenseID      string
    DeviceID       string
    SdkVersion     int32
    SdkVersionStr  string
    AppVersion     string
    EnvCode        []byte
    CreateTime     int64      // X-Khronos, again, inside the blob
    BodyHash       []byte     // SM3 of the body (16 zero bytes on GET)
    QueryHash      []byte     // SM3 of the literal query string
    AlgorithmCount struct {
        SignCount    int32   // how many signatures this install has made
        ReportCount  int32
        SettingCount int32
        Timestamp    int64
    }
    SecDeviceToken string
    IsAppLicense   int64
    PskHash        []byte
    CallType       int32
    ChannelInfo    struct { PhoneInfo, Channel string; ... }
}

The subset the signer actually populates. Establishing the field numbering is most of the reverse-engineering work.

AlgorithmCount.SignCount is a counter of how many requests this install has signed. A real phone's counter climbs steadily over the life of the install. A scraper that emits a constant, or resets to zero on every request, is producing a statistically obvious pattern even when every individual signature verifies. I seed it randomly per device in a plausible range and it has never been a problem, but it is the kind of field that exists specifically so that naive replay is detectable in aggregate rather than at the individual request.

The pipeline

Once the protobuf is serialised, it goes through this, in order:

1.  pb        = proto3_serialize(Argus{...})
2.  padded    = pkcs7(pb, 16)

    // key derivation: the signing key is a per-aid 32-byte constant
3.  xmKey     = SM3( signKey[0:32] || f(rand_lo, rand_hi) || signKey[0:32] )

4.  enc1      = Simon-128/256-ECB( key = xmKey, padded )
5.  enc1      = reverse_bytes(enc1)
6.  enc1      = xor_mix(enc1, derived_from(rand))      // bit-level, order-sensitive

    // framing: a version byte, entropy, and a 3-byte marker built from
    // the first bytes of two separate SM3 digests
7.  framed    = hexFirstByte(aid) || rand_bytes || append_array || enc1

8.  enc2      = AES-128-CBC( key = MD5(signKey[0:16]),
                          iv  = MD5(signKey[16:32]), framed )

9.  X-Argus   = base64( rand_lo || enc2 )

Two encryption layers with different primitives and different key derivations, with a byte reversal and an XOR mix sandwiched between them. None of the individual steps is hard. The difficulty is entirely that there is no feedback . Get step 6 wrong and you produce a perfectly well-formed, correctly-sized, base64-clean header that TikTok answers with an empty 200.

Which is why the implementation ships with independent test vectors for every primitive. You verify Simon, Speck, SM3 and TTEncrypt separately against known input/output pairs, so that when a request fails you already know the crypto is right and the bug is in composition.

Simon, Speck and SM3

The choice of primitives is deliberate, and it says something about the threat model.

ARX ciphers

Simon and Speck are lightweight block ciphers published by the NSA in 2013. Both are ARX constructions, built entirely from modular Addition, bitwise Rotation and Xor. No S-boxes. No lookup tables. No multiplication.

Speck's round function, in full, is two lines:

x = (ROR(x, α) + y) ⊕ k
y =  ROL(y, β)     ⊕ x

// for the 128-bit block size: α = 8, β = 3, 64-bit words
// 128/256 configuration: 256-bit key, 34 rounds

Simon is the same idea with the addition swapped for AND, which makes it cheaper in hardware and slightly more expensive in software:

x' = y ⊕ (ROL(x,1) & ROL(x,8)) ⊕ ROL(x,2) ⊕ k
y' = x

// 128/256 configuration: 256-bit key, 128-bit block, 72 rounds,
// round constants from the Z4 sequence (a 62-bit LFSR period)

Why these and not AES? Three reasons, and they all point the same way:

• They compile to almost nothing. A few hundred bytes of ARM, no tables, no data-dependent memory access. That matters when the code lives inside an obfuscated native library that has to be small and has to avoid cache-timing side channels that would make it easy to locate.
• They are not in your standard library. AES is a function call in every language. Simon and Speck you have to implement, and the parameterisation space is large (block size, key size, round count, rotation constants, key schedule) so a wrong guess produces plausible ciphertext and no error.
• They are easy to get subtly wrong. Speck's key schedule reuses the round function itself. Get the word order or the endianness wrong and you get 32 valid-looking round keys that are all incorrect.

Note that AES-128-CBC is in the pipeline, as the outer layer. The interesting design choice is that the inner layer, the one actually protecting the protobuf, is the one you can't just call.

SM3

SM3 is the Chinese national cryptographic hash standard (GB/T 32905-2016). 256-bit output, 512-bit blocks, Merkle-Damgård construction with a compression function structurally similar to SHA-256 but with two parallel message expansion schedules and a different round function:

// two boolean functions, switching at round 16
FF(x,y,z) = x ⊕ y ⊕ z                      // j < 16
          = (x&y) | (x&z) | (y&z)          // j ≥ 16
GG(x,y,z) = x ⊕ y ⊕ z                      // j < 16
          = (x&y) | (~x&z)                 // j ≥ 16

// IV
7380166F 4914B2B9 172442D7 DA8A0600 A96F30BC 163138AA E38DEE4D B0FB0E4E

SM3 shows up in ByteDance's stack for the obvious reason. It is also, usefully for them, absent from every Western standard library. And the two message expansion arrays (W and W') are trivially transposable, so a large fraction of the reference implementations floating around are wrong in ways that only show up on certain inputs.

TTEncrypt

The body cipher, used for the registration payload and the activation blob. Not a standard construction, just a fixed-key byte transform with a small table. It is not cryptographically serious and is not meant to be; it exists to stop casual traffic inspection, and it is the easiest of the four to reimplement.

X-Ladon

Much simpler than Argus, and worth showing in full because it is a good illustration of how these schemes are layered: a cheap gate in front of an expensive one.

plaintext = "<khronos>-<license_id>-<aid>"
key       = ascii_hex( MD5( rand_bytes(4) || aid ) )   // 32 bytes
cipher    = Speck-128/256-ECB( key, pkcs7(plaintext) )
X-Ladon   = base64( rand_bytes || cipher )

Four random bytes, an MD5, and a Speck encryption of a dash-joined string. The random bytes are prepended to the output so the server can rederive the key. That is the whole construction.

It filters out anyone who hasn't looked at the app at all, and costs approximately nothing to verify at scale. Argus is the expensive check that runs after.

Version gating

A correct signature is necessary and not sufficient. Some endpoints are gated on the client build, and the gate is server-side.

The clearest case is comments. Same device, same signer, same second, same everything. Only version_code differs:

The version bump also unlocked comment replies and follower listing. It is not that the older build's signature is rejected, because it verifies fine. It is that the endpoint is simply not served to that client version.

Practically this means the app version is a per-endpoint property, not a global setting. In my catalogue each endpoint records the build it needs and the server swaps the four version fields transparently before signing:

func WithAppVersion(dev *DevInfo, version, code string) *DevInfo {
    c := *dev
    c.App.AppVersion          = version
    c.App.AppVersionCode      = code
    c.App.ManifestVersionCode = code
    c.App.UpdateVersionCode   = code
    return &c
}

Pinning the newest build everywhere is not the answer, because newer builds tighten other checks. The catalogue exists so that each endpoint sits on the build that works for it.

Region partitioning

The third gate, and the one with nothing to go on: no error, no redirect, no hint in the response.

TikTok does not run one API. It runs several regional data centres: alisg (Singapore), useast1a, useast5 and others. And they do not serve the same endpoints to the same devices.

With activation fixed, profile detail still failed on freshly generated devices while working on an older pool. Same code, same signer. The difference turns out to be the host:

Same second, same credential, same signed request, three hosts, one answer. And music/detail is the reverse: it answers on useast1a and returns nothing on the Singapore host that serves almost everything else.

So the host is part of the endpoint definition. Not a global base URL but a per-route property, established by measurement, because there is no documentation to consult:

{
    ID: "user.info", Route: "/v1/user/info",
    Host: tiktok.HostUSEast5,    // the ONLY host that serves this to fresh devices
    Path: "/aweme/v1/user/profile/other/",
    ...
},
{
    ID: "music.info", Route: "/v1/music/info",
    Host: tiktok.HostUSEast1A,   // and this one is the only host for THIS
    Path: "/aweme/v1/music/detail/",
    ...
},

There is a useful second-order effect here. The device's registered region also influences content on the region-scoped endpoints: trending sounds and trending category shelves. Running one pool registered in US and another in BR gives you genuinely different charts from the identical call, which is how you get per-country data without any per-country code.

The TLS fingerprint

The fourth gate, and the one that is invisible at every layer an application developer normally inspects.

Before any of your bytes arrive, your TLS client sends a ClientHello. Everything in it, and crucially the order of everything in it, is a fingerprint. JA3, the standard way of capturing this, is an MD5 of five comma-joined fields:

TLSVersion , Ciphers , Extensions , EllipticCurves , ECPointFormats

771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,
0-23-65281-10-11-35-16-5-13-18-51-45-43-27-21,29-23-24,0
       ↓ MD5
cd08e31494f9531f560d64c695473da9

A JA3 string and its hash. The cipher list and the extension list are ordered, and libraries order them differently.

That fingerprint identifies your TLS library, and often its version, with high precision. OpenSSL, BoringSSL, NSS, Go's crypto/tls, Java's JSSE are all distinguishable, before a single byte of HTTP is exchanged.

Go's crypto/tls has a very distinctive one. And no Android app has ever emitted it, because Android apps use BoringSSL through OkHttp. TikTok's useast5 edge checks.

The experiment that isolated it

The same request works from Python and fails from Go. Signature byte-identical, parameters byte-identical, cookies irrelevant (it works with and without). Dump the exact headers Python just used, replay them from Go, and hold everything else constant. Same URL, same signature, same device, same second:

// identical request, three clients, back to back

python  urllib3 / OpenSSL     →  9,517 bytes
curl    OpenSSL               →  9,519 bytes
go      crypto/tls            →      0 bytes   ← HTTP 200

Nothing about the request was different. The handshake was.

The fix is uTLS, which lets you specify the exact ClientHello to emit instead of accepting the one Go builds for you:

cfg := &utls.Config{ServerName: host, NextProtos: []string{"http/1.1"}}
conn := utls.UClient(raw, cfg, utls.HelloAndroid_11_OkHttp)
if err := conn.HandshakeContext(ctx); err != nil {
    return nil, fmt.Errorf("utls handshake: %w", err)
}

One line of profile selection. Profile detail on fresh devices went from 0% to 100%.

NextProtos is pinned to http/1.1 on purpose. The Android profile advertises h2, but the transport underneath this is HTTP/1.1 only. Negotiate h2 and you get a connection nothing can speak on.

The Go proxy trap

Short, and specific to Go.

Wire up uTLS, test it directly, confirm the fingerprint has changed, then put it behind the rotating proxy. The failures come straight back.

The reason is that http.Transport ignores DialTLSContext when Proxy is set. It dials the proxy, issues CONNECT itself, and then runs its own standard-library handshake over the resulting tunnel. Your custom dialer is silently discarded. No error, no warning, no log line.

You have to do the tunnel by hand:

dialTLS := func(ctx context.Context, network, addr string) (net.Conn, error) {
    // 1. plain TCP to the proxy
    raw, err := d.DialContext(ctx, "tcp", proxyURL.Host)
    ...
    // 2. CONNECT by hand: this is the part Transport would have done
    req := &http.Request{Method: "CONNECT", URL: &url.URL{Opaque: addr}, Host: addr, ...}
    req.Write(raw)
    resp, _ := http.ReadResponse(bufio.NewReader(raw), req)
    if resp.StatusCode != 200 { return nil, fmt.Errorf("CONNECT: %s", resp.Status) }

    // 3. NOW run the uTLS handshake over the tunnel
    u := utls.UClient(raw, cfg, utls.HelloAndroid_11_OkHttp)
    return u, u.HandshakeContext(ctx)
}

tr := &http.Transport{
    DialTLSContext:    dialTLS,
    DisableKeepAlives: true,   // see the next section
    // note: NO Proxy field. Setting it would bypass all of the above.
}

The Proxy field is deliberately absent from the transport. Setting it is what silently discards the dialer.

Detecting the empty 200

With all four gates passed you still need to know, per response, whether you actually got data. Status codes will not tell you. The check has to be on content:

if resp.StatusCode != http.StatusOK {
    return body, fmt.Errorf("upstream HTTP %d", resp.StatusCode)
}
if len(body) < minBodyBytes {                    // 64
    // The soft block: 200 with (almost) nothing in it.
    return body, fmt.Errorf("empty upstream body (%d bytes)", len(body))
}
var probe map[string]json.RawMessage
if err := json.Unmarshal(body, &probe); err != nil {
    // HTML, usually a proxy error page rather than TikTok
    return body, errors.New("upstream body is not a JSON object")
}
if raw, ok := probe["status_code"]; ok {
    var n int
    if json.Unmarshal(raw, &n) == nil && n != 0 {
        return body, fmt.Errorf("upstream status_code %d", n)
    }
}

Four conditions, in order: HTTP status, length floor, parseable JSON object, clean internal status_code. Anything that fails one is retried against a different device from a different IP.

Except when retrying is pointless

Some non-zero status_code values are TikTok answering rather than refusing. Retrying those four times is a waste of four devices and four IPs:

Which surfaces to the caller as a real answer instead of a gateway failure:

$ curl -s localhost:8080/v1/user/following?user_id=6744630345964389381 | jq
{
  "error": {
    "code": "hidden_by_user",
    "message": "This creator has hidden their following list. Most accounts do; there is no way around it.",
    "upstream_status_code": 3002060,
    "upstream_status_msg": "Profile user is hiding following list",
    "retried": false,
    "retry_would_not_help": true
  }
}

Everything else keeps its full retry budget, and when it exhausts it you get the per-attempt breakdown rather than a generic failure, which is what makes this debuggable in production:

{
  "error": {
    "code": "upstream_failed",
    "attempts": 4,
    "attempt_failures": [
      {"attempt": 1, "reason": "empty upstream body (0 bytes)"},
      {"attempt": 2, "reason": "empty upstream body (0 bytes)"},
      {"attempt": 3, "reason": "transport: ... EOF"},
      {"attempt": 4, "reason": "upstream HTTP 429"}
    ]
  }
}

Real output from the weakest endpoint in the catalogue. Two soft blocks, a dropped connection, and an honest rate limit.

Keep-alive pins the exit IP

With all four gates passed, the highest-volume endpoint ran at 88.2% over 174 million attempts. Good, and at that volume the missing 12% is twenty million lost records.

The obvious move is more retries. It does nothing, because of how rate limiting and connection reuse interact.

Rate limiting here is per exit IP. A rotating proxy gateway assigns an exit IP per TCP connection. HTTP keep-alive, which every client does by default and which is normally exactly what you want, pins you to one exit IP for the life of that connection.

So the retry went out from the address that had just been refused. And the next one. And the next:

// keep-alive on a rotating proxy
attempt 1  →  exit 203.0.113.44  →  empty 200
attempt 2  →  exit 203.0.113.44  →  empty 200     ← same IP
attempt 3  →  exit 203.0.113.44  →  empty 200     ← same IP
attempt 4  →  exit 203.0.113.44  →  empty 200     ← same IP

// fresh connection per attempt
attempt 1  →  exit 203.0.113.44  →  empty 200
attempt 2  →  exit 198.51.100.7  →  ok

Four attempts, one IP, four identical failures. The retry budget bought nothing at all. It was structurally incapable of helping.

Why the naive fix stalls at 96%

Setting DisableKeepAlives: true everywhere took it to 96.2% and then stopped. The cause is that at full concurrency you are now paying a TLS handshake for every attempt, including the ~88% that were going to succeed first time. The proxy gateway, not TikTok, became the bottleneck and started refusing tunnels:

{"attempt": 1, "reason": "transport: proxy CONNECT: 466 Too Many Requests"}

The failure had moved, not gone. The production shape is a hybrid of the two, which comes down to two pools and one policy switch:

// First-attempt pool: keep-alive, so the common case costs no handshake.
r.proxyPool, _      = httpclient.New(httpclient.Config{ProxyURL: cfg.ProxyURL})

// Retry pool: DisableKeepAlives => fresh TCP => NEW exit IP.
r.proxyPoolFresh, _ = httpclient.New(httpclient.Config{
    ProxyURL: cfg.ProxyURL, DisableKeepAlives: true,
})

// ...and in the request path:
pool := r.proxyPool
if attempt > 0 && r.proxyPoolFresh != nil {
    pool = r.proxyPoolFresh          // rotation exactly where it matters
}

Measured over hundreds of millions of calls across four shards. The self-hosted server described below keeps the simpler always-fresh form, because a single instance is nowhere near the load where the second bottleneck appears.

Proxies: the one running cost

Everything up to here is a software problem you solve once. The proxy is the single external dependency and the only recurring cost, and the requirement for one is structural rather than incidental.

What a proxy is, briefly

A proxy is a machine that makes the request on your behalf. You connect to it, it connects to TikTok, and TikTok sees the proxy's IP address instead of yours. A rotating gateway is one where each new connection comes out of a different address in a large pool, which is the property that matters here.

Why it is mandatory rather than recommended

Two reasons, and the second is the one people underestimate.

Rate limiting is per exit IP. One address gets a budget and it is not a large one. Without a proxy every request in your system shares a single address, and you exhaust it in minutes.

Retries are structurally useless without rotation. This is the point from the previous section. When a request is soft blocked, the retry has to leave from a different address or it fails identically. Rotation per connection is the entire mechanism behind the jump from 88% to 99.3%. A static proxy gives you one IP and therefore gives you nothing.

So the requirement is a rotating gateway, and you can verify yours actually rotates in one line before you commit to anything:

$ for i in 1 2 3; do curl -s --proxy "$PROXY_URL" https://api.ipify.org; echo; done
203.0.113.44
198.51.100.7      # different IP each time = rotating, good
192.0.2.19

If the same address comes back three times, your retry budget is decorative and your success rate will sit near the first-try rate no matter what you set MAX_ATTEMPTS to.

What it actually costs

The first thing to know is that you should not be paying by the gigabyte. Metered plans are the default recommendation in this space and they are the wrong shape for this workload, because the endpoints that return the most useful data are the ones measured in megabytes. A page of videos is 1.1 MB. A page of recommended creators is 2.9 MB. Metered billing turns every one of those into a line item.

Flat monthly subscriptions exist for both proxy types, and they are what you want. Two tiers cover essentially everyone:

For enriching a few hundred thousand creators, tracking sounds daily, or mapping a niche, the $150 tier is sufficient rather than a compromise: a rotating datacenter pool registers devices, passes activation and sustains the success rates in the table further down.

The residential tier is what you escalate to once you are saturating the datacenter one.

What the $150 tier buys

On a flat plan the two limits are threads (how many requests can be in flight) and line speed (how many bytes per second). Which one binds depends entirely on response size, and the endpoints here differ by two orders of magnitude: a profile is 9.5 KB, a page of videos is 1.1 MB.

The figures below are arithmetic from the measured response sizes and latencies in the benchmark, at 100 threads and 200 Mbit/s. They are ceilings at full saturation, so treat them as an upper bound rather than a promise.

On the small endpoints you run out of threads long before bandwidth, so the fix is a higher thread count. On the video endpoints you saturate the line at around 22 requests a second, and more threads buy you nothing at all. That is the number MAX_CONCURRENT exists to control, and setting it above what your plan can carry produces proxy CONNECT: 466 Too Many Requests in the attempt failures rather than more throughput.

What to look for when buying one

• Rotating, with a single gateway endpoint. Verify rotation with the loop above before you pay for a month.
• A published thread limit. If it is not stated, assume it is low. This is the number you actually plan around.
• Flat rate over metered, unless you know your volume is small and stays small. Metered plans punish exactly the endpoints that return the most useful data.
• Country targeting, if you want regional charts. The device region and the exit region should agree.
• A trial or one month first. Registration success rate is the real test and it varies between providers advertising the same product. Generate 30 devices and read generation_survival_rate before committing.

Measured success rates

Every endpoint ships with a real success rate rather than a claim: 100 calls each, at most 4 attempts, against a freshly generated pool over a rotating proxy gateway. Seeds are discovered live by walking the API rather than hardcoded, which changes the numbers. The note below explains why.

Average attempts is the more informative column. A 100% endpoint at 1.00 attempts succeeds first time, every time. A 92% endpoint at 2.12 attempts is being soft-blocked on roughly half its first tries and recovering on retry, which means a wider budget moves it, whereas nothing moves a first-try-clean endpoint because there is nothing to move.

feed.recommended is genuinely weak, at 10-25% across runs, and it is dominated by honest 429s rather than soft blocks. An anonymous device with no watch history asking for a personalised feed is precisely the traffic shape TikTok most wants to throttle. It ships documented as weak with the two 100% alternatives named in its place.

The 24 endpoints

All of the above is packaged as a self-hosted Go service. One binary, no database, no queue, no emulator, no native library. Reference data is compiled in.

$ cp .env.example .env        # set PROXY_URL
$ docker compose up -d
$ docker compose logs -f

TikTok Open API 1.0.0 starting
config: port=8080 country=SG pool=15/30 attempts=4 concurrency=32 proxy=http://***@gw:9000 auth=true
pool: 0 device(s) live, filling to 30 ...
listening on http://0.0.0.0:8080  (GET /healthz, GET /v1/endpoints)
pool: initial fill complete, 43 device(s) live

Real startup output. Cold start is 15 to 60 seconds; the pool persists to disk so restarts after that are instant.

All 24 routes are GET, all take query parameters, all return TikTok's JSON unmodified.

Creators

Videos

Sounds

Hashtags and search

Discovery

A real response

Trimmed to the interesting fields. The raw object has several hundred keys:

$ curl -s "localhost:8080/v1/user/posts?user_id=6744630345964389381&count=20" \
    | jq '{has_more, max_cursor, first: (.aweme_list[0] | {aweme_id, desc, statistics, music, author})}'
{
  "has_more": 1,
  "max_cursor": 1751028792000,
  "first": {
    "aweme_id": "7678101694902832397",
    "desc": "Who Remembers 2022? #fortnite #piececontrolkyle #dogwater",
    "statistics": {
      "play_count": 19438,  "digg_count": 2461,
      "comment_count": 39, "share_count": 176
    },
    "music": {
      "id_str": "7245172246876227585",
      "title": "Need 2 (Instrumental)"
    },
    "author": {
      "uid": "6744630345964389381",
      "unique_id": "freakynaughty",
      "nickname": "freaky",
      "follower_count": 1277258
    }
  }
}

Note that the author object is embedded in every video. One request gives you twenty videos and the full creator record. On the web that is twenty-one requests.

Request metadata comes back in headers rather than polluting the body:

HTTP/1.1 200 OK
X-Endpoint-Id:     user.posts
X-Attempts:        2                 ← first try was soft-blocked
X-Elapsed-Ms:      1874
X-Upstream-Region: sg
X-Device-Region:   SG

Fields the web does not give you

The last row was measured across 579 creators on both endpoints that could plausibly carry it. The outbound profile link is a web-surface field only.

Tutorial: a sound-trend detector

Concrete worked example, because the endpoint list on its own does not tell you what the data is good for. The goal: find sounds that are taking off right now, before they are obviously trending.

The signal is music.user_count, how many videos have been made with a sound. The absolute number tells you a sound is big. The rate of change tells you it is moving, which is the part you want.

Step 1. Snapshot the chart

curl -s "$API/v1/music/trending?count=50" \
  | jq -r '.music_list[] | [.id_str, .user_count, .title] | @tsv' \
  > "sounds-$(date +%s).tsv"

Run it hourly from cron. Each row is id, uses, title.

Step 2. Diff consecutive snapshots

import glob, csv, collections

snaps = sorted(glob.glob("sounds-*.tsv"))[-2:]
prev, curr = [{r[0]: (int(r[1]), r[2])
               for r in csv.reader(open(f), delimiter="\t")} for f in snaps]

movers = []
for mid, (n, title) in curr.items():
    was = prev.get(mid, (0, title))[0]
    if was > 0:
        movers.append((n / was - 1, n - was, title, mid))

for growth, delta, title, mid in sorted(movers, reverse=True)[:10]:
    print(f"{growth:6.1%}  +{delta:>8,}  {title[:40]:<40} {mid}")

Step 3. Confirm it is actually accelerating

Growth in the chart is a candidate, not a confirmation. The check that separates a real acceleration from a chart-placement artefact is the time spread of recent videos. Pull the newest videos using that sound and look at how tightly their upload times cluster:

curl -s "$API/v1/music/posts/fresh?music_id=$MID&count=30" \
  | jq '[.aweme_list[].create_time] | (max - min) / 3600'
2.4

Thirty videos in a 2.4-hour window means thirty people picked up that sound this afternoon. Compare with the popular ordering, which tells you whether it has already landed:

curl -s "$API/v1/music/posts?music_id=$MID&count=30" \
  | jq '[.aweme_list[].statistics.play_count] | add'

High fresh-clustering plus low cumulative plays is the interesting quadrant. Lots of people using it, not much accumulated reach yet. That is a sound on the way up rather than one on the way down.

Step 4. Find who is driving it

curl -s "$API/v1/music/posts/fresh?music_id=$MID&count=30" \
  | jq -r '.aweme_list[].author | [.follower_count, .unique_id] | @tsv' \
  | sort -rn | head

Because the author object is embedded, this costs no extra requests. If one large account is at the top and everyone else is small, you are looking at a sound that one creator kicked off, which is a different (and usually shorter-lived) phenomenon than organic uptake across many mid-sized accounts.

Step 5. Widen it

music.trending is region-scoped to the device. Run a second instance with POOL_COUNTRY=US and a third with POOL_COUNTRY=BR and you get three independent charts from identical code. Sounds frequently break in one market days before another.

Getting the code

Everything described here is a private Go repository. One-time payment, permanent access, complete source.

Before you run it

The one hard requirement is a rotating proxy gateway. Rate limiting is per exit IP and the retry design assumes a new connection gets a new address, so a static proxy is no better than none. It is the single external dependency and it is not optional at volume.

Maintenance is yours once you self-host. The repository is structured so that when something moves it is usually one struct literal in one table, but it is your struct literal.

If you would rather skip the setup entirely, there is a done-for-you tier where I build it on your server and hand it over running.

Done for you

The repository is one component of a collection system, and on its own it answers one request at a time. Getting from there to billions of records is a different piece of work: deciding what to fetch next, keeping the queue moving through failures, landing the results somewhere that is still queryable at that size, and running the whole thing across enough shards and proxy capacity to sustain the rate.

This tier is that system, built on your infrastructure and handed over running. Not a demo pointed at a few creators. The same shape as the one that produced the figures at the top of this page, sized to what you are collecting.

Storage, if you are keeping the data

Collecting the data is what the repository solves. Keeping it queryable once there are billions of rows is a separate problem with its own failure modes, and it is the half that decides whether the collection was worth doing. If you are building a store rather than running a one-off pull, that design is part of the handover: table engines, partition and sort keys, the update path for records that change, and the denormalisation that keeps a creator-to-video-to-sound question answerable without a join across billions of rows.

Four decisions that determine whether it holds, each of them measured on a production ClickHouse store at billion-row scale:

• Partition keys. A bare modulus of an id looks even and is not, because platform ids are not uniformly distributed. One table partitioned on author_id % 8 ended up with a 43x spread between its largest and smallest partition, the largest heading for the size where merges stop keeping up. Hashing the id before the modulus flattens it.
• The update path. ReplacingMergeTree keeps the newest whole row, not the newest value per column. Write a partial update and every field you left out is silently blanked. Nothing errors, and you find out much later.
• Duplicate control. Re-collecting the same creator is normal and the storage cost of that compounds quietly. One table was carrying three times the rows it needed before anything made it visible, and rebuilding it returned several terabytes.
• The database's own logs. ClickHouse writes text_log and trace_log by default with no retention. They reached 243 GB on one instance and took a 7 TB volume to full, which stops writes for everything sharing it. A TTL on the system tables is not optional at this scale.

How it goes

1. You tell me what you are collecting. Creators in a niche, sounds on a schedule, comments on a set of videos. This determines the proxy tier and the pool size, so it is worth being concrete.
2. You provide a server and a proxy subscription. The collection layer is light and runs comfortably on two cores. If you are storing what you pull, the database is the part that needs real disk and real memory, and sizing it is part of step one rather than a surprise later.
3. I deploy and tune it. Usually the same week. You get the running instance, the repository access, and the reasoning behind each setting rather than just the settings.
4. We run the integration test together. You watch 24 endpoints come back with live data on your own hardware before you consider it delivered.

Questions