Cinema Watch has its own indexing spider. It crawls the web for movies and TV shows, finds titles we're missing, and fills in the details: release dates, runtimes, posters, genres and trailers. Then a person checks every change before it reaches the catalog.
The spider is a guest on every site it visits. It takes its time on purpose, and we have no intention of ever slowing down or bringing down anyone's website.
Six steps from a page on the web to a title you can log, rate and add to a list.
The spider crawls the world wide web, looking for titles we don't have yet and gaps in the ones we do, like a missing poster or runtime.
Every site gets its own speed limit, shared by all our crawlers. If a site asks us to slow down, the spider waits and keeps the slower pace from then on.
Each find is matched to a title by a standard film identifier where there is one, otherwise by title and year. Editions, "4K" labels and punctuation are ignored, and every match gets a confidence score.
The spider never writes to the catalog. It turns what it found into proposed changes, each showing the current value, the new one, and the page it came from.
A person on our team checks every proposal and approves or rejects it. Low-confidence matches and anything that would replace an existing value are flagged.
Approved changes go live straight away. Images are resized and optimised first, and we record where each value came from.
Facts about titles, never anything about people who use the sites it visits.
Raw data from the web is messy. We may use AI prompts to tidy up and improve what the spider brings back, but only using what it actually found, and always with a person checking the result.
Turn scraped synopses into short, original descriptions in our own words, so we don't copy anyone's text.
Normalise messy values: dates in odd formats, runtimes like "2h 28m", stray symbols and leftover markup.
Map the many ways sites describe genres and ratings onto our own list, and flag anything that doesn't fit.
Spot likely mistakes before review, like a release year that doesn't fit the description or two titles that look like the same film.
AI suggestions go through the same review as everything else: nothing an AI writes reaches the catalog until a person approves it, and it never replaces details our members have set.
Every change waits for a person to approve it. A broken page or a bad match can't reach the catalog.
If a member or moderator set a value, the spider leaves it alone. Replacing it takes a deliberate choice in review.
We keep a record of who or what set each field and when, with a link back to the page it came from.
A named User-Agent, per-site rate limits, backing off when asked, and robots.txt checks before fetching pages.
Our spider identifies itself with cinema-watch-crawler in its User-Agent. To keep it off your site, or part of it, add a rule for that name to your robots.txt. It's picked up within an hour.
Crawling too fast, or something else we should know? Get in touch and we'll adjust or stop.
# Block the spider everywhere
User-agent: cinema-watch-crawler
Disallow: /
# …or just one section
User-agent: cinema-watch-crawler
Disallow: /private/Suggest a title, a release date or a better poster. Moderators check members' fixes too, and the spider won't overwrite them.