af_fulltext

Full-text extraction for Tiny Tiny RSS, driven by fivefilters/ftr-site-config rules, with an optional Firecrawl renderer for pages that only assemble themselves under JavaScript.

It replaces the older arrangement of subscribing to feed URLs that point at a self-hosted Full-Text RSS proxy. Feeds go back to being their real URLs, and extraction happens inside tt-rss where you can see and fix it.

Why not just Readability, or just Firecrawl

Both were measured against a real 34-feed subscription before this was written.

• Readability alone is good on prose and useless on image-only pages. A webcomic has no text to score, so it returns the blog note under the strip and drops the strip.
• Firecrawl alone is a superb fetcher and a weak extractor. Asked for a Cloudflare blog post with onlyMainContent: true it returned 73KB still containing skip-links, analytics markup and the language-picker footer.

So this plugin uses Firecrawl (when you select it) purely as a renderer, asks it for rawHtml, and does the extraction itself with the community rule set — roughly 2000 site configs that encode where the article actually is.

Rules

Standard ftr-site-config syntax. Supported directives:

title body author date strip strip_id_or_class strip_image_src dissolve single_page_link next_page_link find_string replace_string http_header(...) wrap_in(...) autodetect_on_failure test_url

Resolution order for a host, nearest match first:

site_config/custom/<host>.txt        your rules, edited locally
site_config/custom/.<host>.txt
site_config/standard/<host>.txt      the community set
                                     ... then without a leading "www."
                                     ... then wildcard parents, .example.com.txt
site_config/standard/global.txt      merged underneath whatever matched

prune and tidy are parsed but not applied. A body: rule is already an explicit selection, and running a second heuristic pass over it caused more surprises than it solved. This is reported in the extraction summary rather than being silent.

Stale rules are reported, not hidden

This is the reason the plugin exists in this shape. A site redesign turns a body: rule into a no-op; extraction then falls through to Readability and the article still looks plausible, so nothing appears broken. A rule for one of these comics had been dead long enough that nobody could say when it broke.

So: every extraction records which rule file matched, whether it selected anything, and whether it fell back. The settings pane says how many feeds are currently degraded, and each article carries an HTML comment naming its rule.

The first run of bin/audit.php over that subscription found five community rules that no longer matched anything, including the one for this project's own test target.

Command line

The engine in lib/ has no tt-rss dependencies, so rules can be written and checked without a running instance.

# Extract one URL and print the text
php bin/extract.php https://example.com/article

# ... as HTML, through Firecrawl, against a draft rule file
FIRECRAWL_URL=http://127.0.0.1:3002 \
  php bin/extract.php https://example.com/article --backend=firecrawl --rule=draft.txt --html

# Extraction health across a whole subscription list
php bin/audit.php feeds.txt

bin/php-podman.sh runs any of these in a pinned PHP container if you have no PHP on the machine.

Install

cd /var/www/html/tt-rss/plugins.local
git clone <this repo> af_fulltext
git clone https://github.com/fivefilters/ftr-site-config.git af_fulltext/site_config/standard

Then add af_fulltext to TTRSS_PLUGINS and enable it per feed in the feed editor. tt-rss's own startup script keeps git-backed plugins in plugins.local up to date, so redeploying is a pull and a restart.

Settings

env var	default	meaning
TTRSS_AF_FULLTEXT_FIRECRAWL_URL	(unset)	Firecrawl base URL; the Firecrawl backend is unavailable without it
TTRSS_AF_FULLTEXT_FIRECRAWL_KEY	(unset)	bearer token, if your Firecrawl requires one
TTRSS_AF_FULLTEXT_TIMEOUT	20	per-fetch timeout in seconds
TTRSS_AF_FULLTEXT_RULES_REPO	ftr-site-config	git remote the community rules come from
TTRSS_AF_FULLTEXT_RULES_REFRESH_HOURS	24	how often to fast-forward them; 0 disables

Rules refresh from the update daemon (HOOK_HOUSE_KEEPING), so there is no cron entry or sidecar to remember. The refresh is fast-forward only — a divergent checkout is left alone and reported.

Security

Article fetches go through tt-rss's UrlHelper, keeping its SSRF protection: feeds are attacker-influenced input, and a plugin that fetches whatever a feed points at must not be the hole in that. The Firecrawl backend is the deliberate exception, since it talks to one operator-configured endpoint that UrlHelper would refuse precisely for being private.

Licence

GPL-3.0-or-later, as required for tt-rss plugins. Vendored dependencies keep their own licences; local patches to them are recorded in patches/.