# af_fulltext

Full-text extraction for [Tiny Tiny RSS](https://github.com/tt-rss/tt-rss), driven
by [fivefilters/ftr-site-config](https://github.com/fivefilters/ftr-site-config)
rules, with an optional [Firecrawl](https://github.com/firecrawl/firecrawl) renderer
for pages that only assemble themselves under JavaScript.

It replaces the older arrangement of subscribing to feed URLs that point at a
self-hosted Full-Text RSS proxy. Feeds go back to being their real URLs, and
extraction happens inside tt-rss where you can see and fix it.

## Why not just Readability, or just Firecrawl

Both were measured against a real 34-feed subscription before this was written.

- **Readability alone** is good on prose and useless on image-only pages. A
  webcomic has no text to score, so it returns the blog note under the strip and
  drops the strip.
- **Firecrawl alone** is a superb fetcher and a weak extractor. Asked for a
  Cloudflare blog post with `onlyMainContent: true` it returned 73KB still
  containing skip-links, analytics markup and the language-picker footer.

So this plugin uses Firecrawl (when you select it) purely as a *renderer*, asks it
for `rawHtml`, and does the extraction itself with the community rule set —
roughly 2000 site configs that encode where the article actually is.

## Rules

Standard ftr-site-config syntax. Supported directives:

`title` `body` `author` `date` `strip` `strip_id_or_class` `strip_image_src`
`dissolve` `single_page_link` `next_page_link` `find_string` `replace_string`
`http_header(...)` `wrap_in(...)` `autodetect_on_failure` `test_url`

Resolution order for a host, nearest match first:

```
site_config/custom/<host>.txt        your rules, edited locally
site_config/custom/.<host>.txt
site_config/standard/<host>.txt      the community set
                                     ... then without a leading "www."
                                     ... then wildcard parents, .example.com.txt
site_config/standard/global.txt      merged underneath whatever matched
```

`prune` and `tidy` are parsed but not applied. A `body:` rule is already an explicit
selection, and running a second heuristic pass over it caused more surprises than it
solved. This is reported in the extraction summary rather than being silent.

## Stale rules are reported, not hidden

This is the reason the plugin exists in this shape. A site redesign turns a `body:`
rule into a no-op; extraction then falls through to Readability and the article
still looks plausible, so nothing appears broken. A rule for one of these comics had
been dead long enough that nobody could say when it broke.

So: every extraction records which rule file matched, whether it selected anything,
and whether it fell back. The settings pane says how many feeds are currently
degraded, and each article carries an HTML comment naming its rule.

The first run of `bin/audit.php` over that subscription found **five** community
rules that no longer matched anything, including the one for this project's own
test target.

## Command line

The engine in `lib/` has no tt-rss dependencies, so rules can be written and
checked without a running instance.

```sh
# Extract one URL and print the text
php bin/extract.php https://example.com/article

# ... as HTML, through Firecrawl, against a draft rule file
FIRECRAWL_URL=http://127.0.0.1:3002 \
  php bin/extract.php https://example.com/article --backend=firecrawl --rule=draft.txt --html

# Extraction health across a whole subscription list
php bin/audit.php feeds.txt
```

`bin/php-podman.sh` runs any of these in a pinned PHP container if you have no PHP
on the machine.

## Install

```sh
cd /var/www/html/tt-rss/plugins.local
git clone <this repo> af_fulltext
git clone https://github.com/fivefilters/ftr-site-config.git af_fulltext/site_config/standard
```

Then add `af_fulltext` to `TTRSS_PLUGINS` and enable it per feed in the feed editor.
tt-rss's own startup script keeps git-backed plugins in `plugins.local` up to date,
so redeploying is a pull and a restart.

### Settings

| env var | default | meaning |
|---|---|---|
| `TTRSS_AF_FULLTEXT_FIRECRAWL_URL` | *(unset)* | Firecrawl base URL; the Firecrawl backend is unavailable without it |
| `TTRSS_AF_FULLTEXT_FIRECRAWL_KEY` | *(unset)* | bearer token, if your Firecrawl requires one |
| `TTRSS_AF_FULLTEXT_TIMEOUT` | `20` | per-fetch timeout in seconds |
| `TTRSS_AF_FULLTEXT_RULES_REPO` | ftr-site-config | git remote the community rules come from |
| `TTRSS_AF_FULLTEXT_RULES_REFRESH_HOURS` | `24` | how often to fast-forward them; `0` disables |

Rules refresh from the update daemon (`HOOK_HOUSE_KEEPING`), so there is no cron
entry or sidecar to remember. The refresh is fast-forward only — a divergent
checkout is left alone and reported.

## Security

Article fetches go through tt-rss's `UrlHelper`, keeping its SSRF protection: feeds
are attacker-influenced input, and a plugin that fetches whatever a feed points at
must not be the hole in that. The Firecrawl backend is the deliberate exception,
since it talks to one operator-configured endpoint that `UrlHelper` would refuse
precisely for being private.

## Licence

GPL-3.0-or-later, as required for tt-rss plugins. Vendored dependencies keep their
own licences; local patches to them are recorded in `patches/`.