How does Nitter Polling work? ← Dashboard

Technical Documentation · February 2026

1 · Architecture

A self-hosted Nitter Docker instance runs on heissa.de:9495 and mirrors Twitter/X content as RSS feeds. A Python script (nitter_poll.py) fetches these feeds regularly via cron and stores new posts in a MariaDB table (nitter_posts). PHP pages visualise the collected posts. 

X.com / Twitter
      │  (Twitter-API intern)
      ▼
Nitter-Docker  heissa.de:9495
      │  RSS-Feed  /<account>/rss
      ▼
nitter_poll.py  (Cron, stündlich)
      │  INSERT IGNORE
      ▼
MariaDB · wagodb · nitter_posts
      │
      ▼
SZwanglos.php / Impf_Info.php / StHomburg.php

2 · How long does a full cycle take?

To avoid overloading Nitter and Twitter, only 20 accounts are polled per cron run. Between each request the script waits 1.5 – 4.5 seconds randomly (natural timing). The three feeds run offset from each other to avoid overlap:

Feed Following accounts Batch / hour Runs until complete Total duration
@SZwanglos 947 20 · Cron :00 48 runs ≈ 48 hours (2 days)
@Impf_Info 173 20 · Cron :15 9 runs ≈ 9 hours
@StHomburg 3 20 · Cron :30 1 run ≈ 1 hour

A single batch (20 accounts) takes 30 – 90 seconds net. The cron job then finishes; at the next hourly tick the next 20 accounts are polled seamlessly (rotating batch position, stored in ~/.nitter_batch_SZwanglos).

Note: "Full cycle" means every account has been checked at least once for new posts. In practice, fresh posts from active accounts are available after the first cycle – depending on when the account appears in the rotation order.

3 · Method: fetching only new posts (Incremental Polling)

The script does not fetch all posts of an account every time. Instead it remembers the ID of the last seen post. On the next run it stops processing as soon as that ID appears – only newer entries are processed.

Step by step

1
Fetch RSS feed
GET http://heissa.de:9495/<account>/rss
Nitter returns the latest ~20 tweets as an Atom/RSS feed.
2
Load last known ID
The key following:SZwanglos:<account> is read from the state file ~/.nitter_seen.json. On the first run this is empty → all feed entries are imported.
3
Process only new entries
Feed entries are iterated from top (newest) to bottom. As soon as an entry matches the stored ID the loop breaks – everything below is already known.
4
Write to database
New posts are saved via INSERT IGNORE into nitter_posts. The IGNORE prevents duplicates if the same post ID appears twice (e.g. during batch overlap).
5
Save new last ID
The ID of the newest post is written as the new checkpoint to ~/.nitter_seen.json and used as the starting point on the next cron run.
Result: After the first full cycle (~2 days for SZwanglos) each hourly run fetches only the genuinely new posts since the last check – typically 0–5 posts per account. Traffic against Nitter and Twitter remains minimal.

Fallback on rate-limiting (HTTP 429)

If Nitter returns 429 Too Many Requests, the script automatically switches to Playwright: a headless Chromium browser opens the corresponding x.com page with a stored login session (~/.auth_state.json) and extracts posts directly from the rendered HTML – without RSS.

4 · Database structure

nitter_posts
├── id            BIGINT  AUTO_INCREMENT
├── post_id       VARCHAR(200)  UNIQUE   ← Twitter Snowflake ID or URL
├── for_user      VARCHAR(100)           ← SZwanglos / Impf_Info / StHomburg
├── account       VARCHAR(100)           ← polled account
├── author        VARCHAR(100)           ← display name
├── title         TEXT                   ← Tweet-Text (120 Zeichen)
├── link          VARCHAR(500)           ← Nitter URL to tweet
├── post_time     VARCHAR(20)            ← "DD.MM HH:MM" (from RSS)
├── published_at  DATETIME               ← reconstructed from Snowflake ID
├── content       TEXT                   ← Tweet-Text (200 Zeichen)
└── created_at    TIMESTAMP              ← import timestamp

The published_at timestamp is calculated from the Twitter Snowflake ID:
published_at = (snowflake_id >> 22) + 1288834974657 ms since epoch
This enables correct chronological sorting regardless of the import time.

5 · File overview

~/python/nitter_poll.py          Polling script (RSS + Playwright fallback)
~/.nitter_seen.json              Checkpoint: last known post ID per account
~/.nitter_batch_SZwanglos        Batch position: where rotation continues
~/.nitter_following/SZwanglos.json  Following list cache (24h TTL)
~/python/nitter_notify.py        Bot: @mention tweets about new feed page

← Back to Dashboard  ·  🇩🇪 Deutsche Version