michal/ttrss-plugin-af-fulltext
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

af_fulltext: rule-driven extraction with a pluggable renderer

Browse Source 
Replaces subscribing feeds through a self-hosted Full-Text RSS proxy. Feed URLs
go back to being real feed URLs and extraction happens inside tt-rss, using the
same ftr-site-config rules the proxy used.

The engine in lib/ has no tt-rss dependencies, so rules can be developed and
audited from the command line; init.php is a thin adapter over it.

Two findings from measuring the real subscription first, both of which shaped
the design:

- Firecrawl's own onlyMainContent is far too coarse to extract with (73KB of
  chrome on a Cloudflare post), but it is an excellent renderer. So it is used
  for rawHtml only and the rule engine does the extraction.
- A body rule that stops matching after a redesign falls through to Readability
  and still produces a plausible article, so the breakage is invisible. Every
  extraction now records which rule matched and whether it fell back; auditing
  the 34 live feeds surfaced five community rules that match nothing.

Custom rules included for the sites that needed them, including three comics
where the article is an image and text-scoring extractors return the wrong thing
or nothing at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sSJdftQx5bW5HZHgUKF3i
This commit is contained in:
Michal
2026-08-25 00:16:16 +01:00
commit 9bc854ec44
146 changed files with 30221 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
.gitignore vendored Normal file
View File

@@ -0,0 +1,7 @@
# Community rules are a separate upstream checkout: the init container clones them
# and the plugin keeps them fast-forwarded (HOOK_HOUSE_KEEPING). Vendoring 2000
# files that someone else maintains would only make them stale.
/site_config/standard/
# Local feed lists used with bin/audit.php.
/feeds.txt

674
LICENSE Normal file
View File

@@ -0,0 +1,674 @@
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU General Public License is a free, copyleft license for
software and other kinds of works.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
the GNU General Public License is intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users. We, the Free Software Foundation, use the
GNU General Public License for most of our software; it applies also to
any other work released this way by its authors. You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights. Therefore, you have
certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must pass on to the recipients the same
freedoms that you received. You must make sure that they, too, receive
or can get the source code. And you must show them these terms so they
know their rights.
Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
giving you legal permission to copy, distribute and/or modify it.
For the developers' and authors' protection, the GPL clearly explains
that there is no warranty for this free software. For both users' and
authors' sake, the GPL requires that modified versions be marked as
changed, so that their problems will not be attributed erroneously to
authors of previous versions.
Some devices are designed to deny users access to install or run
modified versions of the software inside them, although the manufacturer
can do so. This is fundamentally incompatible with the aim of
protecting users' freedom to change the software. The systematic
pattern of such abuse occurs in the area of products for individuals to
use, which is precisely where it is most unacceptable. Therefore, we
have designed this version of the GPL to prohibit the practice for those
products. If such problems arise substantially in other domains, we
stand ready to extend this provision to those domains in future versions
of the GPL, as needed to protect the freedom of users.
Finally, every program is threatened constantly by software patents.
States should not allow patents to restrict development and use of
software on general-purpose computers, but in those that do, we wish to
avoid the special danger that patents applied to a free program could
make it effectively proprietary. To prevent this, the GPL assures that
patents cannot be used to render the program non-free.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Use with the GNU Affero General Public License.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU Affero General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the special requirements of the GNU Affero General Public License,
section 13, concerning interaction through a network will apply to the
combination as such.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If the program does terminal interaction, make it output a short
notice like this when it starts in an interactive mode:
<program> Copyright (C) <year> <name of author>
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, your program's commands
might be different; for a GUI interface, you would use an "about box".
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU GPL, see
<https://www.gnu.org/licenses/>.
The GNU General Public License does not permit incorporating your program
into proprietary programs. If your program is a subroutine library, you
may consider it more useful to permit linking proprietary applications with
the library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License. But first, please read
<https://www.gnu.org/licenses/why-not-lgpl.html>.

122
README.md Normal file
View File

@@ -0,0 +1,122 @@
# af_fulltext
Full-text extraction for [Tiny Tiny RSS](https://github.com/tt-rss/tt-rss), driven
by [fivefilters/ftr-site-config](https://github.com/fivefilters/ftr-site-config)
rules, with an optional [Firecrawl](https://github.com/firecrawl/firecrawl) renderer
for pages that only assemble themselves under JavaScript.
It replaces the older arrangement of subscribing to feed URLs that point at a
self-hosted Full-Text RSS proxy. Feeds go back to being their real URLs, and
extraction happens inside tt-rss where you can see and fix it.
## Why not just Readability, or just Firecrawl
Both were measured against a real 34-feed subscription before this was written.
- **Readability alone** is good on prose and useless on image-only pages. A
webcomic has no text to score, so it returns the blog note under the strip and
drops the strip.
- **Firecrawl alone** is a superb fetcher and a weak extractor. Asked for a
Cloudflare blog post with `onlyMainContent: true` it returned 73KB still
containing skip-links, analytics markup and the language-picker footer.
So this plugin uses Firecrawl (when you select it) purely as a *renderer*, asks it
for `rawHtml`, and does the extraction itself with the community rule set —
roughly 2000 site configs that encode where the article actually is.
## Rules
Standard ftr-site-config syntax. Supported directives:
`title` `body` `author` `date` `strip` `strip_id_or_class` `strip_image_src`
`dissolve` `single_page_link` `next_page_link` `find_string` `replace_string`
`http_header(...)` `wrap_in(...)` `autodetect_on_failure` `test_url`
Resolution order for a host, nearest match first:
```
site_config/custom/<host>.txt your rules, edited locally
site_config/custom/.<host>.txt
site_config/standard/<host>.txt the community set
... then without a leading "www."
... then wildcard parents, .example.com.txt
site_config/standard/global.txt merged underneath whatever matched
```
`prune` and `tidy` are parsed but not applied. A `body:` rule is already an explicit
selection, and running a second heuristic pass over it caused more surprises than it
solved. This is reported in the extraction summary rather than being silent.
## Stale rules are reported, not hidden
This is the reason the plugin exists in this shape. A site redesign turns a `body:`
rule into a no-op; extraction then falls through to Readability and the article
still looks plausible, so nothing appears broken. A rule for one of these comics had
been dead long enough that nobody could say when it broke.
So: every extraction records which rule file matched, whether it selected anything,
and whether it fell back. The settings pane says how many feeds are currently
degraded, and each article carries an HTML comment naming its rule.
The first run of `bin/audit.php` over that subscription found **five** community
rules that no longer matched anything, including the one for this project's own
test target.
## Command line
The engine in `lib/` has no tt-rss dependencies, so rules can be written and
checked without a running instance.
```sh
# Extract one URL and print the text
php bin/extract.php https://example.com/article
# ... as HTML, through Firecrawl, against a draft rule file
FIRECRAWL_URL=http://127.0.0.1:3002 \
php bin/extract.php https://example.com/article --backend=firecrawl --rule=draft.txt --html
# Extraction health across a whole subscription list
php bin/audit.php feeds.txt
```
`bin/php-podman.sh` runs any of these in a pinned PHP container if you have no PHP
on the machine.
## Install
```sh
cd /var/www/html/tt-rss/plugins.local
git clone <this repo> af_fulltext
git clone https://github.com/fivefilters/ftr-site-config.git af_fulltext/site_config/standard
```
Then add `af_fulltext` to `TTRSS_PLUGINS` and enable it per feed in the feed editor.
tt-rss's own startup script keeps git-backed plugins in `plugins.local` up to date,
so redeploying is a pull and a restart.
### Settings
| env var | default | meaning |
|---|---|---|
| `TTRSS_AF_FULLTEXT_FIRECRAWL_URL` | *(unset)* | Firecrawl base URL; the Firecrawl backend is unavailable without it |
| `TTRSS_AF_FULLTEXT_FIRECRAWL_KEY` | *(unset)* | bearer token, if your Firecrawl requires one |
| `TTRSS_AF_FULLTEXT_TIMEOUT` | `20` | per-fetch timeout in seconds |
| `TTRSS_AF_FULLTEXT_RULES_REPO` | ftr-site-config | git remote the community rules come from |
| `TTRSS_AF_FULLTEXT_RULES_REFRESH_HOURS` | `24` | how often to fast-forward them; `0` disables |
Rules refresh from the update daemon (`HOOK_HOUSE_KEEPING`), so there is no cron
entry or sidecar to remember. The refresh is fast-forward only — a divergent
checkout is left alone and reported.
## Security
Article fetches go through tt-rss's `UrlHelper`, keeping its SSRF protection: feeds
are attacker-influenced input, and a plugin that fetches whatever a feed points at
must not be the hole in that. The Firecrawl backend is the deliberate exception,
since it talks to one operator-configured endpoint that `UrlHelper` would refuse
precisely for being private.
## Licence
GPL-3.0-or-later, as required for tt-rss plugins. Vendored dependencies keep their
own licences; local patches to them are recorded in `patches/`.

16
autoload.php Normal file
View File

@@ -0,0 +1,16 @@
<?php
/**
* Autoloader for the plugin's own classes plus the vendored dependencies.
*
* Kept separate from init.php so the extraction engine can be exercised from the
* CLI (bin/extract.php) without bootstrapping tt-rss.
*/
require_once __DIR__ . '/vendor/autoload.php';
spl_autoload_register(function (string $class): void {
if (!str_starts_with($class, 'AfFulltext\\')) return;
$path = __DIR__ . '/lib/' . str_replace('\\', '/', substr($class, strlen('AfFulltext\\'))) . '.php';
if (is_readable($path)) require_once $path;
});

117
bin/audit.php Executable file
View File

@@ -0,0 +1,117 @@
#!/usr/bin/env php
<?php
/**
* Extraction health report across a list of feeds.
*
* Reads feed URLs (one per line) on stdin or from a file, pulls the newest item
* from each, extracts it, and reports what happened. The interesting column is
* `how`: a feed showing STALE has a rule that no longer matches, which is the
* failure that otherwise degrades silently.
*
* php bin/audit.php feeds.txt [--backend=direct|firecrawl] [--items=1]
*/
require_once __DIR__ . '/../autoload.php';
use AfFulltext\{CurlFetcher, Extractor, FirecrawlFetcher, RuleSet};
$args = array_slice($argv, 1);
$backend = 'direct';
$items = 1;
$file = 'php://stdin';
foreach ($args as $arg) {
if (str_starts_with($arg, '--backend=')) $backend = substr($arg, 10);
elseif (str_starts_with($arg, '--items=')) $items = max(1, (int) substr($arg, 8));
elseif (!str_starts_with($arg, '--')) $file = $arg;
}
$base = dirname(__DIR__);
$rules = new RuleSet([$base . '/site_config/custom', $base . '/site_config/standard']);
$extractor = new Extractor($rules);
$fetcher = $backend === 'firecrawl'
? new FirecrawlFetcher(getenv('FIRECRAWL_URL') ?: 'http://127.0.0.1:13002')
: new CurlFetcher();
$feed_fetcher = new CurlFetcher();
/** Newest item links in a feed, RSS or Atom. @return string[] */
function item_links(string $xml, int $limit): array {
$prev = libxml_use_internal_errors(true);
$doc = simplexml_load_string($xml);
libxml_clear_errors();
libxml_use_internal_errors($prev);
if ($doc === false) return [];
$out = [];
// RSS 2.0 / RDF
foreach ($doc->xpath('//item') ?: [] as $item) {
$link = trim((string) $item->link);
if ($link !== '') $out[] = $link;
if (count($out) >= $limit) return $out;
}
// Atom
$doc->registerXPathNamespace('a', 'http://www.w3.org/2005/Atom');
foreach ($doc->xpath('//a:entry') ?: [] as $entry) {
$entry->registerXPathNamespace('a', 'http://www.w3.org/2005/Atom');
foreach ($entry->xpath('a:link[not(@rel) or @rel="alternate"]') ?: [] as $link) {
$href = trim((string) $link['href']);
if ($href !== '') { $out[] = $href; break; }
}
if (count($out) >= $limit) break;
}
return array_slice($out, 0, $limit);
}
$lines = array_filter(array_map('trim', file($file) ?: []), fn($l) => $l !== '' && !str_starts_with($l, '#'));
printf("%-34s %-9s %-30s %7s %5s %s\n", 'FEED HOST', 'BACKEND', 'HOW', 'BYTES', 'IMGS', 'NOTES');
printf("%s\n", str_repeat('-', 130));
$totals = ['ok' => 0, 'stale' => 0, 'readability' => 0, 'failed' => 0];
foreach ($lines as $feed_url) {
$host = (string) parse_url($feed_url, PHP_URL_HOST);
$feed = $feed_fetcher->fetch($feed_url);
if (!$feed->ok()) {
printf("%-34s %-9s %-30s %7s %5s %s\n", substr($host, 0, 34), '-', 'FEED FETCH FAILED', '-', '-', $feed->error);
$totals['failed']++;
continue;
}
$links = item_links($feed->html, $items);
if (!$links) {
printf("%-34s %-9s %-30s %7s %5s %s\n", substr($host, 0, 34), '-', 'NO ITEMS', '-', '-', '');
$totals['failed']++;
continue;
}
foreach ($links as $link) {
$r = $extractor->extract($link, $fetcher);
$how = match (true) {
$r->rule_matched => 'rule: ' . basename($r->rule_sources[0] ?? '?'),
$r->rule_stale => 'STALE: ' . basename($r->rule_sources[0] ?? '?'),
$r->ok() => 'readability',
default => 'FAILED',
};
if ($r->rule_matched) $totals['ok']++;
elseif ($r->rule_stale) $totals['stale']++;
elseif ($r->ok()) $totals['readability']++;
else $totals['failed']++;
printf("%-34s %-9s %-30s %7d %5d %s\n",
substr((string) parse_url($link, PHP_URL_HOST), 0, 34),
$r->backend, substr($how, 0, 30), strlen($r->html),
substr_count($r->html, '<img'), implode('; ', array_slice($r->errors, 0, 1)));
}
}
printf("\n%d rule-matched, %d STALE rules, %d readability-only, %d failed\n",
$totals['ok'], $totals['stale'], $totals['readability'], $totals['failed']);

60
bin/extract.php Executable file
View File

@@ -0,0 +1,60 @@
#!/usr/bin/env php
<?php
/**
* Standalone harness for the extraction engine.
*
* The whole point of keeping lib/ free of tt-rss dependencies: rules can be
* written and checked against real URLs without a running instance.
*
* php bin/extract.php <url> [--backend=direct|firecrawl] [--rule=FILE] [--html] [--quiet]
*/
require_once __DIR__ . '/../autoload.php';
use AfFulltext\{CurlFetcher, Extractor, FirecrawlFetcher, RuleSet};
$args = array_slice($argv, 1);
$opts = ['backend' => 'direct', 'rule' => null, 'html' => false, 'quiet' => false];
$url = null;
foreach ($args as $arg) {
if (str_starts_with($arg, '--backend=')) $opts['backend'] = substr($arg, 10);
elseif (str_starts_with($arg, '--rule=')) $opts['rule'] = substr($arg, 7);
elseif ($arg === '--html') $opts['html'] = true;
elseif ($arg === '--quiet') $opts['quiet'] = true;
elseif (!str_starts_with($arg, '--')) $url = $arg;
}
if (!$url) {
fwrite(STDERR, "usage: extract.php <url> [--backend=direct|firecrawl] [--rule=FILE] [--html] [--quiet]\n");
exit(2);
}
$base = dirname(__DIR__);
$dirs = [$base . '/site_config/custom', $base . '/site_config/standard'];
// --rule points at a single file to try, overriding whatever the site config says.
if ($opts['rule']) {
$tmp = sys_get_temp_dir() . '/af_fulltext_rule_' . getmypid();
@mkdir($tmp, 0700, true);
$host = strtolower((string) parse_url($url, PHP_URL_HOST));
copy($opts['rule'], "$tmp/$host.txt");
array_unshift($dirs, $tmp);
}
$fetcher = $opts['backend'] === 'firecrawl'
? new FirecrawlFetcher(getenv('FIRECRAWL_URL') ?: 'http://127.0.0.1:13002')
: new CurlFetcher();
$result = (new Extractor(new RuleSet($dirs)))->extract($url, $fetcher);
if (!$opts['quiet']) {
fwrite(STDERR, $result->summary() . "\n");
foreach ($result->rule_sources as $s) fwrite(STDERR, " rule: $s\n");
foreach ($result->errors as $e) fwrite(STDERR, " error: $e\n");
if ($result->title) fwrite(STDERR, " title: {$result->title}\n");
}
if ($opts['html']) echo $result->html, "\n";
else echo trim(preg_replace('/\s+/u', ' ', strip_tags($result->html)) ?? ''), "\n";
exit($result->ok() ? 0 : 1);

5
bin/php-podman.sh Executable file
View File

@@ -0,0 +1,5 @@
#!/usr/bin/env bash
# Run the harness inside a pinned PHP image so results do not depend on whatever
# PHP happens to be on the host (there is none on this workstation).
exec podman run --rm --network=host -v "$(cd "$(dirname "$0")/.." && pwd)":/plugin:z -w /plugin \
docker.io/library/php:8.4-cli php "$@"

30
composer.json Normal file
View File

@@ -0,0 +1,30 @@
{
"name": "michal/ttrss-plugin-af-fulltext",
"description": "Full-text extraction for Tiny Tiny RSS using ftr-site-config rules, with an optional Firecrawl renderer",
"license": "GPL-3.0-or-later",
"minimum-stability": "dev",
"prefer-stable": true,
"repositories": [
{
"name": "fivefilters/readability.php",
"type": "vcs",
"url": "https://github.com/tt-rss/tt-rss-readability-php.git"
},
{
"name": "masterminds/html5",
"type": "vcs",
"url": "https://github.com/tt-rss/tt-rss-html5-php.git"
}
],
"require": {
"php": ">=8.2",
"ext-dom": "*",
"ext-libxml": "*",
"ext-mbstring": "*",
"ext-curl": "*",
"fivefilters/readability.php": "dev-main"
},
"autoload": {
"psr-4": { "AfFulltext\\": "lib/" }
}
}

6
feeds.example.txt Normal file
View File

@@ -0,0 +1,6 @@
# One feed URL per line; blank lines and # comments are ignored.
# Used by bin/audit.php to report extraction health across a subscription list.
https://blog.cloudflare.com/rss/
https://www.oglaf.com/feeds/rss/
https://loadingartist.com/index.xml
https://xkcd.com/atom.xml

48
init.js Normal file
View File

@@ -0,0 +1,48 @@
/* global xhr, App, Plugins, Article, Notify */
Plugins.Af_Fulltext = {
orig_attr_name: 'data-af-fulltext-orig-content',
/**
* Toggle extracted content in place for one article.
*
* Doubles as the rule-debugging loop: edit a rule, hit the button, see the
* result immediately -- and the notification names the rule that produced it,
* so a rule that quietly stopped matching is visible rather than merely
* disappointing.
*/
embed: function(id) {
const content = document.querySelector(App.isCombinedMode() ?
`.cdm[data-article-id="${id}"] .content-inner` :
`.post[data-article-id="${id}"] .content`);
if (!content) return;
if (content.hasAttribute(this.orig_attr_name)) {
content.innerHTML = content.getAttribute(this.orig_attr_name);
content.removeAttribute(this.orig_attr_name);
if (App.isCombinedMode()) Article.cdmMoveToId(id);
return;
}
Notify.progress("Extracting, please wait...");
xhr.json("backend.php", App.getPhArgs("af_fulltext", "embed", {id: id}), (reply) => {
if (reply && reply.content) {
content.setAttribute(this.orig_attr_name, content.innerHTML);
content.innerHTML = reply.content;
Notify.info(reply.summary || "Extracted");
if (App.isCombinedMode()) Article.cdmMoveToId(id);
} else {
const why = reply && reply.errors && reply.errors.length ?
reply.errors[0] : "no content extracted";
Notify.error("af_fulltext: " + why);
}
});
}
};

475
init.php Normal file
View File

@@ -0,0 +1,475 @@
<?php
/**
* af_fulltext -- rule-driven full-text extraction for Tiny Tiny RSS.
*
* Replaces the usual arrangement of subscribing to a separate Full-Text RSS proxy
* with extraction inside tt-rss, driven by the same fivefilters/ftr-site-config
* rules that proxy used. Two fetch backends: tt-rss's own HTTP stack, or a
* Firecrawl instance when a page only assembles itself under JavaScript.
*
* The extraction engine lives in lib/ and has no tt-rss dependencies, so rules can
* be developed and audited from the command line (bin/extract.php, bin/audit.php).
*/
require_once __DIR__ . '/autoload.php';
use AfFulltext\{Extractor, FirecrawlFetcher, RuleSet, TtrssFetcher};
use AfFulltext\Fetcher;
use AfFulltext\ExtractResult;
class Af_Fulltext extends Plugin {
private const BACKEND_DIRECT = 'direct';
private const BACKEND_FIRECRAWL = 'firecrawl';
/** Endpoint of a Firecrawl instance, e.g. http://firecrawl-api.firecrawl.svc.cluster.local:3002 */
private const CONF_FIRECRAWL_URL = 'AF_FULLTEXT_FIRECRAWL_URL';
private const CONF_FIRECRAWL_KEY = 'AF_FULLTEXT_FIRECRAWL_KEY';
private const CONF_TIMEOUT = 'AF_FULLTEXT_TIMEOUT';
/** Git repo the community rules are refreshed from. */
private const CONF_RULES_REPO = 'AF_FULLTEXT_RULES_REPO';
/** Hours between refreshes; 0 disables. */
private const CONF_RULES_REFRESH_HOURS = 'AF_FULLTEXT_RULES_REFRESH_HOURS';
/** @var PluginHost $host */
private $host;
function about() {
return array(null,
'Full-text extraction using ftr-site-config rules, with an optional Firecrawl renderer',
'michal');
}
function flags() {
return array('needs_curl' => true);
}
function api_version() {
return 2;
}
function init($host) {
$this->host = $host;
Config::add(self::CONF_FIRECRAWL_URL, '', Config::T_STRING);
Config::add(self::CONF_FIRECRAWL_KEY, '', Config::T_STRING);
Config::add(self::CONF_TIMEOUT, '20', Config::T_INT);
Config::add(self::CONF_RULES_REPO, 'https://github.com/fivefilters/ftr-site-config.git', Config::T_STRING);
Config::add(self::CONF_RULES_REFRESH_HOURS, '24', Config::T_INT);
$host->add_hook($host::HOOK_ARTICLE_FILTER, $this);
$host->add_hook($host::HOOK_PREFS_TAB, $this);
$host->add_hook($host::HOOK_PREFS_EDIT_FEED, $this);
$host->add_hook($host::HOOK_PREFS_SAVE_FEED, $this);
$host->add_hook($host::HOOK_ARTICLE_BUTTON, $this);
$host->add_hook($host::HOOK_HOUSE_KEEPING, $this);
// Installed unconditionally: init() runs before plugin storage is loaded,
// so the enable flag cannot be consulted here.
$host->add_hook($host::HOOK_GET_FULL_TEXT, $this);
$host->add_filter_action($this, 'action_inline', __('Extract full text'));
$host->add_filter_action($this, 'action_inline_append', __('Append full text'));
}
function get_js() {
return file_get_contents(__DIR__ . '/init.js');
}
// ---------------------------------------------------------------- extraction
private function rules_dirs(): array {
return [__DIR__ . '/site_config/custom', __DIR__ . '/site_config/standard'];
}
private function extractor(): Extractor {
return new Extractor(new RuleSet($this->rules_dirs()));
}
private function fetcher(string $backend): Fetcher {
$timeout = (int) Config::get(self::CONF_TIMEOUT);
if ($backend === self::BACKEND_FIRECRAWL) {
$endpoint = (string) Config::get(self::CONF_FIRECRAWL_URL);
if ($endpoint !== '')
return new FirecrawlFetcher($endpoint, max($timeout, 60),
((string) Config::get(self::CONF_FIRECRAWL_KEY)) ?: null);
// Configured per feed but unavailable globally: fall back rather than
// silently producing nothing.
user_error('af_fulltext: firecrawl requested but ' . self::CONF_FIRECRAWL_URL . ' is unset', E_USER_WARNING);
}
return new TtrssFetcher($timeout);
}
/** Backend selected for a feed, defaulting to direct. */
private function backend_for(int $feed_id): string {
$map = $this->host->get_array($this, 'backend_feeds');
return ($map[$feed_id] ?? self::BACKEND_DIRECT) === self::BACKEND_FIRECRAWL
? self::BACKEND_FIRECRAWL
: self::BACKEND_DIRECT;
}
private function extract(string $url, string $backend): ExtractResult {
return $this->extractor()->extract($url, $this->fetcher($backend));
}
/**
* Record what happened for a feed.
*
* A `body:` rule that stops matching after a site redesign still yields a
* plausible-looking article via the Readability fallback, so the degradation is
* invisible in the reader. Keeping the last outcome per feed is what lets the
* settings pane say so out loud.
*
* @return void
*/
private function record_health(int $feed_id, ExtractResult $result) {
$health = $this->host->get_array($this, 'health');
$health[$feed_id] = [
'when' => time(),
'summary' => $result->summary(),
'stale' => $result->rule_stale,
'fell_back' => $result->fell_back,
'bytes' => strlen($result->html),
'error' => $result->errors[0] ?? null,
];
$this->host->set($this, 'health', $health);
}
/**
* @param array<string,mixed> $article
* @return array<string,mixed>
*/
function process_article(array $article, bool $append_mode, ?string $backend = null): array {
$link = $article['link'] ?? '';
if (!$link) return $article;
$feed_id = (int) ($article['feed']['id'] ?? 0);
$backend ??= $this->backend_for($feed_id);
$result = $this->extract($link, $backend);
if ($feed_id) $this->record_health($feed_id, $result);
// Only replace the feed's own summary if we actually got something.
if (!$this->has_content($result->html)) return $article;
$content = $result->html . $this->provenance_comment($result);
if ($append_mode) $article['content'] .= '<hr/>' . $content;
else $article['content'] = $content;
return $article;
}
/**
* An HTML comment naming the rule that produced the article.
*
* Invisible while reading, and the first thing worth looking at when an
* article comes out wrong.
*/
private function provenance_comment(ExtractResult $result): string {
return "\n<!-- af_fulltext: " . htmlspecialchars($result->summary(), ENT_QUOTES) . " -->";
}
function hook_article_filter($article) {
$enabled = $this->host->get_array($this, 'enabled_feeds');
$append = $this->host->get_array($this, 'append_feeds');
$feed_id = $article['feed']['id'] ?? null;
if ($feed_id === null || !in_array($feed_id, $enabled)) return $article;
return $this->process_article($article, in_array($feed_id, $append));
}
function hook_article_filter_action($article, $action) {
return match ($action) {
'action_inline' => $this->process_article($article, false),
'action_inline_append' => $this->process_article($article, true),
default => $article,
};
}
function hook_get_full_text($link) {
if (!$this->host->get($this, 'enable_share_anything')) return false;
$result = $this->extract($link, self::BACKEND_DIRECT);
return $this->has_content($result->html) ? $result->html : false;
}
/**
* Does this survive sanitising as something worth showing?
*
* Sanitizer::sanitize returns false on failure, so its result cannot be passed
* straight to strip_tags. An image-only article has no text at all and must
* still count -- for a webcomic the picture IS the content.
*/
private function has_content(string $html): bool {
if ($html === '') return false;
$clean = Sanitizer::sanitize($html);
if (!is_string($clean)) return false;
if (trim(strip_tags($clean)) !== '') return true;
return (bool) preg_match('/<(img|video|audio|iframe|picture|source|svg|embed)\b/i', $clean);
}
// ------------------------------------------------------------- rule refresh
/**
* Keep the community rules current.
*
* Runs from the update daemon rather than a separate scheduled job, so the
* rules travel with the plugin instead of being someone else's cron entry to
* remember. Refreshing is a fast-forward pull only: a conflict means someone
* edited the checkout by hand, and overwriting it silently would be worse than
* leaving it stale and saying so.
*
* @return void
*/
function hook_house_keeping() {
$hours = (int) Config::get(self::CONF_RULES_REFRESH_HOURS);
if ($hours <= 0) return;
$last = (int) $this->host->get($this, 'rules_refreshed_at');
if ($last && time() - $last < $hours * 3600) return;
$dir = __DIR__ . '/site_config/standard';
$repo = (string) Config::get(self::CONF_RULES_REPO);
if (!is_dir("$dir/.git")) {
Debug::log("af_fulltext: $dir is not a git checkout, skipping rule refresh", Debug::LOG_VERBOSE);
return;
}
$cmd = sprintf('cd %s && git fetch --quiet --depth 1 origin 2>&1 && git reset --quiet --hard FETCH_HEAD 2>&1',
escapeshellarg($dir));
$output = [];
$rc = 0;
exec($cmd, $output, $rc);
// Stamp the attempt either way, so a persistently unreachable remote does
// not retry on every single housekeeping pass.
$this->host->set($this, 'rules_refreshed_at', time());
if ($rc !== 0) {
$this->host->set($this, 'rules_refresh_error', implode(' ', array_slice($output, 0, 3)));
Debug::log('af_fulltext: rule refresh failed: ' . implode(' ', $output), Debug::LOG_VERBOSE);
return;
}
$this->host->set($this, 'rules_refresh_error', '');
Debug::log("af_fulltext: refreshed site rules from $repo", Debug::LOG_VERBOSE);
}
// --------------------------------------------------------------------- UI
function hook_article_button($line) {
return "<i class='material-icons' onclick=\"Plugins.Af_Fulltext.embed(" . $line['id'] . ")\"
style='cursor : pointer' title=\"" . __('Extract full text') . "\">article</i>";
}
function embed(): void {
$article_id = (int) $_REQUEST['id'];
$sth = $this->pdo->prepare('SELECT link, feed_id FROM ttrss_entries e, ttrss_user_entries ue
WHERE e.id = ? AND ue.ref_id = e.id AND ue.owner_uid = ?');
$sth->execute([$article_id, $_SESSION['uid']]);
$ret = [];
if ($row = $sth->fetch()) {
$result = $this->extract($row['link'], $this->backend_for((int) $row['feed_id']));
$ret['content'] = (string) Sanitizer::sanitize($result->html);
$ret['summary'] = $result->summary();
$ret['errors'] = $result->errors;
}
print json_encode($ret);
}
/** @return void */
function save() {
$this->host->set($this, 'enable_share_anything',
checkbox_to_sql_bool($_POST['enable_share_anything'] ?? ''));
echo __('Data saved.');
}
function hook_prefs_edit_feed($feed_id) {
$enabled = $this->host->get_array($this, 'enabled_feeds');
$append = $this->host->get_array($this, 'append_feeds');
$backend = $this->backend_for((int) $feed_id);
?>
<header><?= __('Full-text extraction') ?></header>
<section>
<fieldset>
<label class='checkbox'>
<?= \Controls\checkbox_tag('af_fulltext_enabled', in_array($feed_id, $enabled)) ?>
<?= __('Extract full article content') ?>
</label>
</fieldset>
<fieldset>
<label class='checkbox'>
<?= \Controls\checkbox_tag('af_fulltext_append', in_array($feed_id, $append)) ?>
<?= __('Append to the summary instead of replacing it') ?>
</label>
</fieldset>
<fieldset>
<label><?= __('Fetch using') ?></label>
<?= \Controls\select_hash('af_fulltext_backend', $backend, [
self::BACKEND_DIRECT => __('Direct (fast)'),
self::BACKEND_FIRECRAWL => __('Firecrawl (renders JavaScript)'),
]) ?>
</fieldset>
</section>
<?php
}
function hook_prefs_save_feed($feed_id) {
$enabled = $this->toggle($this->host->get_array($this, 'enabled_feeds'), $feed_id,
(bool) checkbox_to_sql_bool($_POST['af_fulltext_enabled'] ?? ''));
$append = $this->toggle($this->host->get_array($this, 'append_feeds'), $feed_id,
(bool) checkbox_to_sql_bool($_POST['af_fulltext_append'] ?? ''));
$backends = $this->host->get_array($this, 'backend_feeds');
$chosen = $_POST['af_fulltext_backend'] ?? self::BACKEND_DIRECT;
if ($chosen === self::BACKEND_FIRECRAWL) $backends[$feed_id] = self::BACKEND_FIRECRAWL;
else unset($backends[$feed_id]);
$this->host->set($this, 'enabled_feeds', $enabled);
$this->host->set($this, 'append_feeds', $append);
$this->host->set($this, 'backend_feeds', $backends);
}
/**
* @param array<int> $list
* @return array<int>
*/
private function toggle(array $list, $feed_id, bool $on): array {
$key = array_search($feed_id, $list);
if ($on && $key === false) $list[] = $feed_id;
elseif (!$on && $key !== false) unset($list[$key]);
return array_values($list);
}
function hook_prefs_tab($args) {
if ($args != 'prefFeeds') return;
$enable_share_anything = sql_bool_to_bool($this->host->get($this, 'enable_share_anything'));
$enabled = $this->filter_unknown_feeds($this->host->get_array($this, 'enabled_feeds'));
$this->host->set($this, 'enabled_feeds', $enabled);
$append = $this->host->get_array($this, 'append_feeds');
$health = $this->host->get_array($this, 'health');
$refreshed = (int) $this->host->get($this, 'rules_refreshed_at');
$refresh_error = (string) $this->host->get($this, 'rules_refresh_error');
$rule_count = count(glob(__DIR__ . '/site_config/standard/*.txt') ?: [])
+ count(glob(__DIR__ . '/site_config/custom/*.txt') ?: []);
$degraded = array_filter($health, fn($h) => !empty($h['stale']));
?>
<div dojoType='dijit.layout.AccordionPane'
title="<i class='material-icons'>article</i> <?= __('Full-text extraction (af_fulltext)') ?>">
<?= format_notice('Enable per feed in the feed editor. ' . $rule_count . ' site rules loaded'
. ($refreshed ? ', refreshed ' . date('Y-m-d H:i', $refreshed) : ', never refreshed') . '.') ?>
<?php if ($refresh_error) { ?>
<?= format_warning('Rule refresh failed: ' . htmlspecialchars($refresh_error)) ?>
<?php } ?>
<?php if ($degraded) { ?>
<?= format_warning(sprintf(
'%d feed(s) have a site rule that no longer matches anything and are falling back to Readability. '
. 'That usually means the site was redesigned and the rule needs updating.',
count($degraded))) ?>
<?php } ?>
<form dojoType='dijit.form.Form'>
<?= \Controls\pluginhandler_tags($this, 'save') ?>
<script type="dojo/method" event="onSubmit" args="evt">
evt.preventDefault();
if (this.validate()) {
Notify.progress('Saving data...', true);
xhr.post("backend.php", this.getValues(), (reply) => {
Notify.info(reply);
})
}
</script>
<fieldset>
<label class='checkbox'>
<?= \Controls\checkbox_tag('enable_share_anything', $enable_share_anything) ?>
<?= __('Provide full-text services to core code (bookmarklets) and other plugins') ?>
</label>
</fieldset>
<hr/>
<?= \Controls\submit_tag(__('Save')) ?>
</form>
<?php if (count($enabled) > 0) { ?>
<hr/>
<h3><?= __('Currently enabled for (click to edit):') ?></h3>
<ul class='panel panel-scrollable list list-unstyled'>
<?php foreach ($enabled as $f) {
$h = $health[$f] ?? null;
?>
<li>
<?php if (Feeds::_has_icon($f)) { ?>
<img src='<?= Feeds::_get_icon_url($f) ?>' style="max-height: 20px" />
<?php } else { ?> <i class='material-icons'>rss_feed</i> <?php } ?>
<a href='#' onclick="CommonDialogs.editFeed(<?= $f ?>)">
<?= Feeds::_get_title($f, $this->host->get_owner_uid()) ?>
</a>
<?= in_array($f, $append) ? ' ' . __('(append)') : '' ?>
<?php if ($h) { ?>
<small style="opacity: .7"><?= htmlspecialchars($h['summary'] ?? '') ?></small>
<?php } ?>
</li>
<?php } ?>
</ul>
<?php } ?>
</div>
<?php
}
/**
* @param array<int> $feeds
* @return array<int>
*/
private function filter_unknown_feeds(array $feeds): array {
$out = [];
foreach ($feeds as $feed) {
$sth = $this->pdo->prepare('SELECT id FROM ttrss_feeds WHERE id = ? AND owner_uid = ?');
$sth->execute([$feed, $_SESSION['uid']]);
if ($sth->fetch()) $out[] = $feed;
}
return $out;
}
}

74
lib/CurlFetcher.php Normal file
View File

@@ -0,0 +1,74 @@
<?php
namespace AfFulltext;
/**
* Plain HTTP GET via cURL.
*
* Used by the CLI harness. Inside tt-rss the `direct` backend is TtrssFetcher
* instead, so article fetches keep tt-rss's own SSRF guard (UrlHelper::validate).
*/
final class CurlFetcher implements Fetcher {
public function __construct(
private readonly int $timeout = 20,
private readonly string $user_agent = 'Mozilla/5.0 (compatible; af_fulltext/1.0; +https://tt-rss.org/)',
) {}
public function name(): string { return 'direct'; }
public function fetch(string $url, array $headers = []): FetchResult {
$ch = curl_init();
$hdr = [];
foreach ($headers as $k => $v) $hdr[] = "$k: $v";
curl_setopt_array($ch, [
CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_MAXREDIRS => 8,
CURLOPT_TIMEOUT => $this->timeout,
CURLOPT_CONNECTTIMEOUT => 10,
CURLOPT_ENCODING => '',
CURLOPT_USERAGENT => $headers['user-agent'] ?? $this->user_agent,
CURLOPT_HTTPHEADER => $hdr,
]);
$body = curl_exec($ch);
$err = curl_error($ch);
$status = (int) curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
$effective = (string) curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
$content_type = (string) curl_getinfo($ch, CURLINFO_CONTENT_TYPE);
curl_close($ch);
if ($err !== '' || !is_string($body))
return new FetchResult(error: $err !== '' ? $err : 'empty response', status: $status, effective_url: $effective ?: $url);
if ($status >= 400)
return new FetchResult(error: "HTTP $status", status: $status, effective_url: $effective ?: $url);
return new FetchResult(
html: self::to_utf8($body, $content_type),
effective_url: $effective ?: $url,
status: $status,
);
}
/**
* Normalise to UTF-8. Several of these feeds are Polish, and a page served as
* ISO-8859-2 that is treated as UTF-8 loses every accented character.
*/
public static function to_utf8(string $body, string $content_type): string {
$charset = '';
if (preg_match('/charset=["\']?([\w-]+)/i', $content_type, $m)) $charset = $m[1];
if ($charset === '' && preg_match('/<meta[^>]+charset=["\']?([\w-]+)/i', substr($body, 0, 4096), $m))
$charset = $m[1];
if ($charset === '' || preg_match('/^utf-?8$/i', $charset)) return $body;
$converted = @mb_convert_encoding($body, 'UTF-8', $charset);
return is_string($converted) && $converted !== '' ? $converted : $body;
}
}

57
lib/ExtractResult.php Normal file
View File

@@ -0,0 +1,57 @@
<?php
namespace AfFulltext;
/**
* The outcome of one extraction, diagnostics included.
*
* The diagnostics are the point. A site redesign turns a `body:` rule into a
* silent no-op that falls through to Readability, and the article still looks
* plausible -- that is how a broken webcomic rule went unnoticed here for years.
* Every consumer of this class can see exactly which rule ran and whether it
* actually matched anything.
*/
final class ExtractResult {
public string $html = '';
public ?string $title = null;
public ?string $author = null;
public ?string $date = null;
public string $backend = '';
public string $effective_url = '';
/** Rule files that contributed, nearest first. @var string[] */
public array $rule_sources = [];
/** True when a `body:` rule existed and selected at least one node. */
public bool $rule_matched = false;
/** True when the content came from Readability rather than a rule. */
public bool $fell_back = false;
/** True when a rule existed but matched nothing -- the stale-rule signal. */
public bool $rule_stale = false;
public int $pages = 1;
public int $timing_ms = 0;
/** @var string[] */
public array $errors = [];
public function ok(): bool {
return $this->html !== '';
}
/** One-line summary for logs and the prefs UI. */
public function summary(): string {
$rule = $this->rule_sources ? basename($this->rule_sources[0]) : 'none';
$how = match (true) {
$this->rule_matched => "rule=$rule",
$this->rule_stale => "rule=$rule STALE->readability",
default => 'readability',
};
return sprintf('%s %s %db %dms%s', $this->backend, $how, strlen($this->html),
$this->timing_ms, $this->pages > 1 ? " pages={$this->pages}" : '');
}
}

361
lib/Extractor.php Normal file
View File

@@ -0,0 +1,361 @@
<?php
namespace AfFulltext;
use fivefilters\Readability\Configuration;
use fivefilters\Readability\Readability;
/**
* Turns a URL into clean article HTML.
*
* Rule-driven where a site config exists, Readability where one does not. The
* rule format is FiveFilters' own, so the ~2000 community rules in
* fivefilters/ftr-site-config apply unmodified and anything written here can go
* back upstream as a pull request.
*/
final class Extractor {
private const MAX_PAGES = 5;
public function __construct(
private readonly RuleSet $rules,
private readonly int $max_bytes = 4_000_000,
) {}
public function extract(string $url, Fetcher $fetcher): ExtractResult {
$started = microtime(true);
$result = new ExtractResult();
$result->backend = $fetcher->name();
$result->effective_url = $url;
$host = strtolower((string) parse_url($url, PHP_URL_HOST));
$rule = $this->rules->find($host);
if ($rule) $result->rule_sources = $rule->sources;
$fetched = $fetcher->fetch($url, $rule?->http_headers ?? []);
if (!$fetched->ok()) {
$result->errors[] = $fetched->error ?? 'fetch failed';
$result->timing_ms = (int) round((microtime(true) - $started) * 1000);
return $result;
}
if (strlen($fetched->html) > $this->max_bytes)
$result->errors[] = sprintf('page is %dKB, truncating parse', strlen($fetched->html) / 1024);
$result->effective_url = $fetched->effective_url ?: $url;
$doc = Html::parse(substr($fetched->html, 0, $this->max_bytes));
$base = Html::base_url($doc, $result->effective_url);
// A print/single-page view, where one exists, is both cleaner and cheaper
// to extract than following next_page_link through N requests.
if ($rule && $rule->single_page_links) {
$single = $this->first_url($doc, $rule->single_page_links, $base);
if ($single && $single !== $result->effective_url) {
$again = $fetcher->fetch($single, $rule->http_headers);
if ($again->ok()) {
$doc = Html::parse(substr($again->html, 0, $this->max_bytes));
$result->effective_url = $again->effective_url ?: $single;
$base = Html::base_url($doc, $result->effective_url);
} else {
$result->errors[] = 'single_page_link: ' . ($again->error ?? 'failed');
}
}
}
$html = $this->extract_from_doc($doc, $base, $rule, $result);
// Multi-page articles: keep appending until the chain ends or we hit the cap.
if ($rule && $rule->next_page_links) {
$seen = [$result->effective_url => true];
$current = $doc;
$current_base = $base;
while ($result->pages < self::MAX_PAGES) {
$next = $this->first_url($current, $rule->next_page_links, $current_base);
if (!$next || isset($seen[$next])) break;
$seen[$next] = true;
$page = $fetcher->fetch($next, $rule->http_headers);
if (!$page->ok()) { $result->errors[] = 'next_page_link: ' . ($page->error ?? 'failed'); break; }
$current = Html::parse(substr($page->html, 0, $this->max_bytes));
$current_base = Html::base_url($current, $page->effective_url ?: $next);
$more = $this->extract_from_doc($current, $current_base, $rule, new ExtractResult());
if ($more === '') break;
$html .= "\n" . $more;
$result->pages++;
}
}
if ($rule) {
$html = $this->apply_replacements($html, $rule);
$html = $this->apply_wrap_in($html, $rule);
}
$result->html = trim($html);
$result->timing_ms = (int) round((microtime(true) - $started) * 1000);
return $result;
}
/**
* Run the rule pipeline over one parsed document.
*
* Order matters: lazy images are resolved and URLs made absolute before
* anything is removed or selected, so every later step sees the same, final
* attribute values.
*/
private function extract_from_doc(\DOMDocument $doc, string $base, ?Rule $rule, ExtractResult $result): string {
Html::unlazy($doc);
Html::absolutize($doc, $base);
$xpath = new \DOMXPath($doc);
if ($rule) {
$this->apply_removals($doc, $xpath, $rule);
$this->apply_dissolve($xpath, $rule);
$result->title ??= $this->first_string($xpath, $rule->titles);
$result->author ??= $this->first_string($xpath, $rule->authors);
$result->date ??= $this->first_string($xpath, $rule->dates);
}
if ($rule && $rule->bodies) {
$nodes = $this->select_body($xpath, $rule->bodies);
if ($nodes) {
$result->rule_matched = true;
$out = '';
foreach ($nodes as $node) {
Html::sanitize($node);
$out .= Html::outer_html($node) . "\n";
}
return trim($out);
}
// A rule exists and selected nothing. Almost always a site redesign.
$result->rule_stale = true;
$result->errors[] = 'body rule matched no nodes (' . implode(', ', $rule->bodies) . ')';
if (!$rule->autodetect_on_failure) return '';
}
$result->fell_back = true;
return $this->readability($doc, $base, $result);
}
/** @return \DOMNode[] */
private function select_body(\DOMXPath $xpath, array $expressions): array {
$nodes = [];
foreach ($expressions as $expr) {
$found = @$xpath->query($expr);
if ($found === false) continue;
foreach ($found as $node) {
if (Html::text_length($node) > 0 || $this->has_media($node)) $nodes[] = $node;
}
}
return $nodes;
}
/**
* A node with no text is not necessarily empty -- on a webcomic the entire
* article is a single <img>, and discarding it for having no words is exactly
* the bug this plugin exists to fix.
*/
private const MEDIA_TAGS = ['img', 'video', 'audio', 'iframe', 'picture', 'source', 'svg', 'embed', 'object'];
private function has_media(\DOMNode $node): bool {
if (!$node instanceof \DOMElement && !$node instanceof \DOMDocument) return false;
// The node may BE the media. `body: //img[@id='strip']` is a perfectly
// ordinary rule for an image-only comic, and only looking at descendants
// throws that selection away.
if ($node instanceof \DOMElement && in_array(strtolower($node->tagName), self::MEDIA_TAGS, true))
return true;
$doc = $node instanceof \DOMDocument ? $node : $node->ownerDocument;
if (!$doc) return false;
$query = implode(' | ', array_map(fn(string $t) => ".//$t", self::MEDIA_TAGS));
$found = @(new \DOMXPath($doc))->query($query, $node);
return $found !== false && $found->length > 0;
}
private function apply_removals(\DOMDocument $doc, \DOMXPath $xpath, Rule $rule): void {
foreach ($rule->strips as $expr) {
$found = @$xpath->query($expr);
if ($found === false) continue;
foreach (iterator_to_array($found) as $node) $node->parentNode?->removeChild($node);
}
foreach ($rule->strip_id_or_class as $needle) {
$q = $this->contains_lower('@id', $needle) . ' or ' . $this->contains_lower('@class', $needle);
$found = @$xpath->query("//*[$q]");
if ($found === false) continue;
foreach (iterator_to_array($found) as $node) $node->parentNode?->removeChild($node);
}
foreach ($rule->strip_image_src as $needle) {
$found = @$xpath->query('//img[' . $this->contains_lower('@src', $needle) . ']');
if ($found === false) continue;
foreach (iterator_to_array($found) as $node) $node->parentNode?->removeChild($node);
}
}
/** Case-insensitive substring test, spelled out because XPath 1.0 has no lower-case(). */
private function contains_lower(string $attr, string $needle): string {
$upper = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ';
$lower = 'abcdefghijklmnopqrstuvwxyz';
$q = $this->xpath_literal(strtolower($needle));
return "contains(translate($attr, '$upper', '$lower'), $q)";
}
/** Quote a string for XPath 1.0, which has no escape syntax. */
private function xpath_literal(string $s): string {
if (!str_contains($s, "'")) return "'$s'";
if (!str_contains($s, '"')) return "\"$s\"";
return 'concat(' . implode(", \"'\", ", array_map(fn($p) => "'$p'", explode("'", $s))) . ')';
}
/** Replace matched elements with their own children. */
private function apply_dissolve(\DOMXPath $xpath, Rule $rule): void {
foreach ($rule->dissolve as $expr) {
$found = @$xpath->query($expr);
if ($found === false) continue;
foreach (iterator_to_array($found) as $node) {
$parent = $node->parentNode;
if (!$parent) continue;
while ($node->firstChild) $parent->insertBefore($node->firstChild, $node);
$parent->removeChild($node);
}
}
}
/**
* Readability fallback.
*
* The engine emits notices on malformed pages (null property reads on
* documents with no discernible body). Those must not reach output: inside
* tt-rss this runs mid-request and a stray warning corrupts the response. So
* diagnostics are captured into the result instead of being printed, and the
* previous handler is always restored.
*/
private function readability(\DOMDocument $doc, string $base, ExtractResult $result): string {
$config = new Configuration();
$config->setOriginalURL($base);
$config->setFixRelativeURLs(true);
$config->setParser('html5');
// Class names carry meaning for downstream styling and for anyone writing
// a rule off the extracted output.
$config->setKeepClasses(true);
$notices = [];
set_error_handler(function (int $no, string $msg) use (&$notices): bool {
$notices[] = $msg;
return true;
});
try {
$readability = new Readability($config);
$readability->parse((string) $doc->saveHTML());
$result->title ??= $readability->getTitle();
$result->author ??= $readability->getAuthor();
$content = (string) $readability->getContent();
} catch (\Throwable $e) {
$result->errors[] = 'readability: ' . $e->getMessage();
$content = '';
} finally {
restore_error_handler();
}
if ($notices)
$result->errors[] = sprintf('readability emitted %d notice(s): %s',
count($notices), $notices[0]);
return $content;
}
private function apply_replacements(string $html, Rule $rule): string {
foreach ($rule->find_strings as $i => $find) {
$replace = $rule->replace_strings[$i] ?? '';
if ($find !== '') $html = str_replace($find, $replace, $html);
}
return $html;
}
private function apply_wrap_in(string $html, Rule $rule): string {
// wrap_in targets nodes in the source document; applying it to the
// already-serialized result would need a reparse for little gain, so v1
// only honours a document-wide wrapper.
foreach ($rule->wrap_in as $expr => $spec) {
if ($expr !== '//*' && $expr !== '/') continue;
[$tag, $class] = array_pad(explode('.', $spec, 2), 2, null);
$tag = $tag ?: 'div';
$attr = $class ? ' class="' . htmlspecialchars($class, ENT_QUOTES) . '"' : '';
$html = "<$tag$attr>$html</$tag>";
}
return $html;
}
/** First non-empty string value produced by any of the expressions. */
private function first_string(\DOMXPath $xpath, array $expressions): ?string {
foreach ($expressions as $expr) {
$value = @$xpath->evaluate($expr);
if (is_string($value) && trim($value) !== '') return trim($value);
if ($value instanceof \DOMNodeList && $value->length > 0) {
$text = trim((string) $value->item(0)?->textContent);
if ($text !== '') return $text;
}
}
return null;
}
/** First absolute URL produced by any of the expressions. */
private function first_url(\DOMDocument $doc, array $expressions, string $base): ?string {
$xpath = new \DOMXPath($doc);
foreach ($expressions as $expr) {
$value = @$xpath->evaluate($expr);
$raw = null;
if (is_string($value) && trim($value) !== '') {
$raw = trim($value);
} elseif ($value instanceof \DOMNodeList && $value->length > 0) {
$node = $value->item(0);
$raw = $node instanceof \DOMElement ? ($node->getAttribute('href') ?: trim($node->textContent))
: trim((string) $node?->nodeValue);
}
if ($raw) return Html::resolve($raw, $base);
}
return null;
}
}

29
lib/Fetcher.php Normal file
View File

@@ -0,0 +1,29 @@
<?php
namespace AfFulltext;
final class FetchResult {
public function __construct(
public readonly string $html = '',
public readonly string $effective_url = '',
public readonly int $status = 0,
public readonly ?string $error = null,
) {}
public function ok(): bool {
return $this->error === null && $this->html !== '';
}
}
/**
* How a page's HTML is obtained. Kept behind an interface so the extraction
* pipeline is identical whether the markup came from a plain HTTP GET or from a
* headless browser -- and so the engine can be exercised from the CLI without
* tt-rss present.
*/
interface Fetcher {
/** @param array<string,string> $headers extra request headers (lowercased names) */
public function fetch(string $url, array $headers = []): FetchResult;
/** Short identifier used in logs and the prefs UI. */
public function name(): string;
}

80
lib/FirecrawlFetcher.php Normal file
View File

@@ -0,0 +1,80 @@
<?php
namespace AfFulltext;
/**
* Fetch through a Firecrawl instance so the page arrives with its JavaScript
* already run.
*
* We ask for `rawHtml` on purpose. Firecrawl's own `onlyMainContent` extraction
* was measured against these feeds and is far too coarse -- on a Cloudflare blog
* post it returned 73KB still containing skip-links, analytics markup and the
* language-picker footer. Firecrawl is the better *renderer*; the rule engine in
* this plugin is the better *extractor*. So: browser here, extraction ours.
*/
final class FirecrawlFetcher implements Fetcher {
public function __construct(
private readonly string $endpoint,
private readonly int $timeout = 60,
private readonly ?string $api_key = null,
) {}
public function name(): string { return 'firecrawl'; }
public function fetch(string $url, array $headers = []): FetchResult {
$payload = [
'url' => $url,
'formats' => ['rawHtml'],
// Firecrawl's own boilerplate stripping is deliberately off; see above.
'onlyMainContent' => false,
'timeout' => $this->timeout * 1000,
];
if ($headers) $payload['headers'] = $headers;
$request_headers = ['Content-Type: application/json'];
if ($this->api_key) $request_headers[] = 'Authorization: Bearer ' . $this->api_key;
$ch = curl_init();
curl_setopt_array($ch, [
CURLOPT_URL => rtrim($this->endpoint, '/') . '/v1/scrape',
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => json_encode($payload, JSON_UNESCAPED_SLASHES),
CURLOPT_HTTPHEADER => $request_headers,
CURLOPT_RETURNTRANSFER => true,
// The browser render dominates; allow slack over the scrape timeout.
CURLOPT_TIMEOUT => $this->timeout + 15,
CURLOPT_CONNECTTIMEOUT => 10,
]);
$body = curl_exec($ch);
$err = curl_error($ch);
$status = (int) curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
curl_close($ch);
if ($err !== '' || !is_string($body))
return new FetchResult(error: 'firecrawl: ' . ($err !== '' ? $err : 'empty response'), status: $status);
$json = json_decode($body, true);
if (!is_array($json))
return new FetchResult(error: 'firecrawl: unparseable response', status: $status);
if (empty($json['success'])) {
$msg = is_string($json['error'] ?? null) ? $json['error'] : "HTTP $status";
return new FetchResult(error: "firecrawl: $msg", status: $status);
}
$data = $json['data'] ?? [];
$html = $data['rawHtml'] ?? $data['html'] ?? '';
if (!is_string($html) || $html === '')
return new FetchResult(error: 'firecrawl: no html in response', status: $status);
$effective = $data['metadata']['sourceURL'] ?? $data['metadata']['url'] ?? $url;
return new FetchResult(
html: $html,
effective_url: is_string($effective) ? $effective : $url,
status: $status,
);
}
}

259
lib/Html.php Normal file
View File

@@ -0,0 +1,259 @@
<?php
namespace AfFulltext;
use Masterminds\HTML5;
/** DOM helpers shared by the extractor. No tt-rss dependencies. */
final class Html {
/**
* Parse a document. Masterminds' HTML5 parser handles modern markup that
* libxml mangles (unquoted attributes, <picture>, custom elements); libxml is
* the fallback because it copes with badly broken pages that make the HTML5
* parser give up.
*/
public static function parse(string $html): \DOMDocument {
$html = self::strip_bom($html);
try {
$doc = (new HTML5(['disable_html_ns' => true]))->loadHTML($html);
if ($doc->documentElement !== null) return $doc;
} catch (\Throwable) {
// fall through
}
$doc = new \DOMDocument();
$prev = libxml_use_internal_errors(true);
// Force UTF-8: libxml assumes ISO-8859-1 without a meta charset, which
// silently mojibakes every non-ASCII article (most of these feeds are Polish).
$doc->loadHTML('<?xml encoding="utf-8" ?>' . $html, LIBXML_NOWARNING | LIBXML_NOERROR);
libxml_clear_errors();
libxml_use_internal_errors($prev);
return $doc;
}
private static function strip_bom(string $s): string {
return str_starts_with($s, "\xEF\xBB\xBF") ? substr($s, 3) : $s;
}
/** Effective base URL for a document: its <base href> if present, else $url. */
public static function base_url(\DOMDocument $doc, string $url): string {
$base = (new \DOMXPath($doc))->query('//base[@href]');
if ($base && $base->length > 0) {
$href = trim(($base->item(0) instanceof \DOMElement) ? $base->item(0)->getAttribute('href') : '');
if ($href !== '') return self::resolve($href, $url);
}
return $url;
}
/** Resolve a possibly-relative URL against a base. */
public static function resolve(string $href, string $base): string {
$href = trim($href);
if ($href === '') return $base;
if (preg_match('#^[a-z][a-z0-9+.-]*:#i', $href) || str_starts_with($href, '#')) return $href;
$b = parse_url($base);
if (!$b || empty($b['scheme']) || empty($b['host'])) return $href;
$origin = $b['scheme'] . '://' . $b['host'] . (isset($b['port']) ? ':' . $b['port'] : '');
if (str_starts_with($href, '//')) return $b['scheme'] . ':' . $href;
if (str_starts_with($href, '/')) return $origin . self::normalize_path($href);
$dir = isset($b['path']) ? preg_replace('#/[^/]*$#', '/', $b['path']) : '/';
return $origin . self::normalize_path(($dir ?: '/') . $href);
}
/** Collapse ./ and ../ segments. */
private static function normalize_path(string $path): string {
[$path, $tail] = array_pad(explode('?', $path, 2), 2, null);
$out = [];
foreach (explode('/', $path) as $seg) {
if ($seg === '.' || $seg === '') continue;
if ($seg === '..') { array_pop($out); continue; }
$out[] = $seg;
}
$result = '/' . implode('/', $out);
if (str_ends_with($path, '/') && !str_ends_with($result, '/')) $result .= '/';
return $tail !== null ? $result . '?' . $tail : $result;
}
/** Rewrite every relative URL in the tree to an absolute one. */
public static function absolutize(\DOMNode $ctx, string $base): void {
$xpath = new \DOMXPath(self::owner($ctx));
foreach (['href', 'src', 'poster', 'data-src', 'longdesc'] as $attr) {
foreach ($xpath->query(".//*[@$attr]", $ctx) ?: [] as $el) {
if (!$el instanceof \DOMElement) continue;
$v = $el->getAttribute($attr);
if ($v !== '' && !str_starts_with($v, 'data:'))
$el->setAttribute($attr, self::resolve($v, $base));
}
}
foreach ($xpath->query('.//*[@srcset]', $ctx) ?: [] as $el) {
if (!$el instanceof \DOMElement) continue;
$el->setAttribute('srcset', self::absolutize_srcset($el->getAttribute('srcset'), $base));
}
}
private static function absolutize_srcset(string $srcset, string $base): string {
$out = [];
foreach (explode(',', $srcset) as $part) {
$part = trim($part);
if ($part === '') continue;
$bits = preg_split('/\s+/', $part, 2);
$url = self::resolve($bits[0], $base);
$out[] = isset($bits[1]) ? "$url {$bits[1]}" : $url;
}
return implode(', ', $out);
}
/**
* Promote lazy-loading placeholders to a real src.
*
* This is what makes webcomics work: the panel is routinely a 1x1 gif or a
* data: URI until the site's JS swaps in data-src, so an extractor that only
* reads @src produces an article with no picture in it.
*/
public static function unlazy(\DOMNode $ctx): void {
$xpath = new \DOMXPath(self::owner($ctx));
foreach ($xpath->query('.//img', $ctx) ?: [] as $img) {
if (!$img instanceof \DOMElement) continue;
$src = trim($img->getAttribute('src'));
if (!self::is_placeholder($src)) {
// A real src, but a srcset may still offer a larger rendition.
if ($src === '' && ($best = self::best_from_srcset($img->getAttribute('srcset'))))
$img->setAttribute('src', $best);
continue;
}
$replacement = '';
foreach (['data-src', 'data-original', 'data-lazy-src', 'data-url', 'data-full-src'] as $attr) {
$v = trim($img->getAttribute($attr));
if ($v !== '' && !self::is_placeholder($v)) { $replacement = $v; break; }
}
if ($replacement === '')
foreach (['data-srcset', 'srcset', 'data-lazy-srcset'] as $attr)
if ($best = self::best_from_srcset($img->getAttribute($attr))) { $replacement = $best; break; }
if ($replacement !== '') $img->setAttribute('src', $replacement);
self::drop_lazy_attrs($img);
}
// <picture><source srcset> with no usable <img src> underneath.
foreach ($xpath->query('.//picture', $ctx) ?: [] as $pic) {
if (!$pic instanceof \DOMElement) continue;
$imgs = (new \DOMXPath(self::owner($ctx)))->query('.//img', $pic);
$img = ($imgs && $imgs->length) ? $imgs->item(0) : null;
if (!$img instanceof \DOMElement || !self::is_placeholder(trim($img->getAttribute('src')))) continue;
foreach ((new \DOMXPath(self::owner($ctx)))->query('.//source[@srcset]', $pic) ?: [] as $source) {
if (!$source instanceof \DOMElement) continue;
if ($best = self::best_from_srcset($source->getAttribute('srcset'))) {
$img->setAttribute('src', $best);
break;
}
}
}
}
/**
* Drop the lazy-loading attributes once their value has been promoted.
*
* Left in place they roughly double the stored article -- tapas.io signs every
* panel URL, so each image carries its full token twice.
*/
private static function drop_lazy_attrs(\DOMElement $img): void {
foreach (['data-src', 'data-original', 'data-lazy-src', 'data-url',
'data-full-src', 'data-srcset', 'data-lazy-srcset'] as $attr)
if ($img->hasAttribute($attr)) $img->removeAttribute($attr);
}
private static function is_placeholder(string $src): bool {
if ($src === '') return true;
if (str_starts_with($src, 'data:')) return true;
return (bool) preg_match('#(^|/)(blank|spacer|placeholder|transparent|lazy|1x1|pixel)[^/]*\.(gif|png|svg|webp)$#i', $src);
}
/** Largest candidate in a srcset, by width or pixel density. */
private static function best_from_srcset(string $srcset): ?string {
$best = null;
$best_score = -1.0;
foreach (explode(',', $srcset) as $part) {
$part = trim($part);
if ($part === '') continue;
$bits = preg_split('/\s+/', $part, 2);
$url = $bits[0] ?? '';
if ($url === '' || self::is_placeholder($url)) continue;
$score = 1.0;
if (isset($bits[1]) && preg_match('/^([\d.]+)([wx])$/', trim($bits[1]), $m))
$score = $m[2] === 'w' ? (float) $m[1] : (float) $m[1] * 1000;
if ($score > $best_score) { $best_score = $score; $best = $url; }
}
return $best;
}
/** Remove scripts, styles, inline event handlers and other non-content noise. */
public static function sanitize(\DOMNode $ctx): void {
$xpath = new \DOMXPath(self::owner($ctx));
foreach ($xpath->query('.//script | .//style | .//noscript | .//template | .//link | .//meta', $ctx) ?: [] as $el)
$el->parentNode?->removeChild($el);
foreach ($xpath->query('.//@*', $ctx) ?: [] as $attr) {
if (!$attr instanceof \DOMAttr) continue;
$name = strtolower($attr->name);
$owner = $attr->ownerElement;
if (!$owner) continue;
if (str_starts_with($name, 'on') || $name === 'style')
$owner->removeAttribute($attr->name);
if (in_array($name, ['href', 'src', 'action'], true) && preg_match('/^\s*javascript:/i', $attr->value))
$owner->removeAttribute($attr->name);
}
}
/** Serialize a node's children (its inner HTML). */
public static function inner_html(\DOMNode $node): string {
$doc = self::owner($node);
$out = '';
foreach ($node->childNodes as $child)
$out .= $doc->saveHTML($child);
return trim($out);
}
public static function outer_html(\DOMNode $node): string {
return trim((string) self::owner($node)->saveHTML($node));
}
public static function text_length(\DOMNode $node): int {
return mb_strlen(trim(preg_replace('/\s+/u', ' ', $node->textContent) ?? ''));
}
private static function owner(\DOMNode $node): \DOMDocument {
return $node instanceof \DOMDocument ? $node : ($node->ownerDocument ?? new \DOMDocument());
}
}

68
lib/Rule.php Normal file
View File

@@ -0,0 +1,68 @@
<?php
namespace AfFulltext;
/**
* A parsed FiveFilters site-config rule.
*
* Field names track the ftr-site-config directive names so a rule file and this
* object read the same way. Directives that may repeat accumulate into arrays,
* exactly as Full-Text RSS treats them.
*/
final class Rule {
/** @var string[] */ public array $titles = [];
/** @var string[] */ public array $bodies = [];
/** @var string[] */ public array $authors = [];
/** @var string[] */ public array $dates = [];
/** @var string[] */ public array $strips = [];
/** @var string[] */ public array $strip_id_or_class = [];
/** @var string[] */ public array $strip_image_src = [];
/** @var string[] */ public array $dissolve = [];
/** @var string[] */ public array $single_page_links = [];
/** @var string[] */ public array $next_page_links = [];
/** @var string[] */ public array $find_strings = [];
/** @var string[] */ public array $replace_strings = [];
/** @var string[] */ public array $test_urls = [];
/** @var array<string,string> */ public array $http_headers = [];
/** @var array<string,string> map of xpath => wrapper spec e.g. "div.foo" */
public array $wrap_in = [];
public bool $prune = true;
public bool $tidy = true;
public bool $autodetect_on_failure = true;
/**
* Files this rule was assembled from, nearest-match first. Surfaced in the
* UI so a stale rule is visible rather than silently falling through to
* Readability -- the exact failure mode that hid a broken comic rule for
* years.
*
* @var string[]
*/
public array $sources = [];
public function is_empty(): bool {
return !$this->bodies && !$this->strips && !$this->strip_id_or_class
&& !$this->strip_image_src && !$this->titles && !$this->dissolve;
}
/** Merge $other UNDER $this: existing scalars win, list directives concatenate. */
public function merge_under(self $other): self {
$m = clone $this;
foreach (['titles', 'bodies', 'authors', 'dates', 'single_page_links', 'next_page_links'] as $f)
if (!$m->$f) $m->$f = $other->$f;
foreach (['strips', 'strip_id_or_class', 'strip_image_src', 'dissolve', 'test_urls'] as $f)
$m->$f = array_values(array_unique([...$m->$f, ...$other->$f]));
// find/replace are positional pairs -- appending keeps them aligned.
$m->find_strings = [...$m->find_strings, ...$other->find_strings];
$m->replace_strings = [...$m->replace_strings, ...$other->replace_strings];
$m->http_headers = $m->http_headers + $other->http_headers;
$m->wrap_in = $m->wrap_in + $other->wrap_in;
$m->sources = [...$m->sources, ...$other->sources];
return $m;
}
}

148
lib/RuleSet.php Normal file
View File

@@ -0,0 +1,148 @@
<?php
namespace AfFulltext;
/**
* Loads ftr-site-config rule files and resolves the one that applies to a host.
*
* Two directories are searched, custom first: user rules written by the prefs UI
* override the community set, and both use the same on-disk format so a rule
* refined here can be sent upstream to fivefilters/ftr-site-config as-is.
*/
final class RuleSet {
/** @var string[] directories searched in order (custom before standard) */
private array $dirs;
/** @var array<string,?Rule> */
private array $cache = [];
/** @param string[] $dirs */
public function __construct(array $dirs) {
$this->dirs = array_values(array_filter($dirs, 'is_dir'));
}
/**
* Resolve the rule for a hostname, or null if nothing matches.
*
* Order follows Full-Text RSS: exact host, then the same host without a
* leading "www.", then a wildcard file (".example.com.txt") for each parent
* domain. global.txt is merged underneath whatever matched, so its shared
* strip rules apply everywhere.
*/
public function find(string $host): ?Rule {
$host = strtolower(trim($host));
if ($host === '') return null;
if (array_key_exists($host, $this->cache))
return $this->cache[$host];
$rule = null;
foreach ($this->candidates($host) as $name) {
if ($found = $this->load($name)) { $rule = $found; break; }
}
if ($global = $this->load('global')) {
$rule = $rule ? $rule->merge_under($global) : $global;
}
return $this->cache[$host] = $rule;
}
/** Filenames to try, nearest match first. @return string[] */
public function candidates(string $host): array {
$out = [$host];
if (str_starts_with($host, 'www.'))
$out[] = substr($host, 4);
// .example.com.txt applies to every subdomain of example.com.
$parts = explode('.', $out[count($out) - 1]);
while (count($parts) > 1) {
$out[] = '.' . implode('.', $parts);
array_shift($parts);
}
return array_values(array_unique($out));
}
/** Load one rule by bare name (no .txt), searching custom then standard. */
public function load(string $name): ?Rule {
// Rule names come from feed hostnames; keep them off the filesystem.
if ($name === '' || str_contains($name, '/') || str_contains($name, "\0") || str_contains($name, '..'))
return null;
foreach ($this->dirs as $dir) {
$path = $dir . '/' . $name . '.txt';
if (is_readable($path)) {
$rule = self::parse((string) file_get_contents($path));
$rule->sources[] = $path;
return $rule;
}
}
return null;
}
/** Parse ftr-site-config text into a Rule. */
public static function parse(string $text): Rule {
$rule = new Rule();
foreach (preg_split('/\R/', $text) ?: [] as $line) {
$line = trim($line);
if ($line === '' || $line[0] === '#') continue;
$pos = strpos($line, ':');
if ($pos === false) continue;
$key = strtolower(trim(substr($line, 0, $pos)));
$val = trim(substr($line, $pos + 1));
if ($val === '') continue;
// http_header(user-agent) / wrap_in(div.foo) carry an argument.
$arg = null;
if (preg_match('/^([a-z_]+)\((.*)\)$/', $key, $m)) {
$key = $m[1];
$arg = trim($m[2]);
}
switch ($key) {
case 'title': $rule->titles[] = $val; break;
case 'body': $rule->bodies[] = $val; break;
case 'author': $rule->authors[] = $val; break;
case 'date': $rule->dates[] = $val; break;
case 'strip': $rule->strips[] = $val; break;
case 'strip_id_or_class': $rule->strip_id_or_class[] = $val; break;
case 'strip_image_src': $rule->strip_image_src[] = $val; break;
case 'dissolve': $rule->dissolve[] = $val; break;
case 'single_page_link': $rule->single_page_links[] = $val; break;
case 'next_page_link': $rule->next_page_links[] = $val; break;
case 'find_string': $rule->find_strings[] = $val; break;
case 'replace_string': $rule->replace_strings[] = $val; break;
case 'test_url': $rule->test_urls[] = $val; break;
case 'prune': $rule->prune = self::truthy($val); break;
case 'tidy': $rule->tidy = self::truthy($val); break;
case 'autodetect_on_failure': $rule->autodetect_on_failure = self::truthy($val); break;
case 'http_header':
if ($arg !== null && $arg !== '') $rule->http_headers[strtolower($arg)] = $val;
break;
case 'wrap_in':
if ($arg !== null && $arg !== '') $rule->wrap_in[$val] = $arg;
break;
// Deliberately ignored in v1: if_page_contains, replace_string
// variants with regex, convert_to_format, parser, src_lazy_load_attr
// (we resolve lazy images unconditionally -- see Html::unlazy).
default: break;
}
}
return $rule;
}
private static function truthy(string $v): bool {
return in_array(strtolower($v), ['yes', 'true', '1', 'on'], true);
}
}

48
lib/TtrssFetcher.php Normal file
View File

@@ -0,0 +1,48 @@
<?php
namespace AfFulltext;
/**
* The `direct` backend inside tt-rss.
*
* Deliberately goes through UrlHelper rather than cURL directly, so article
* fetches keep tt-rss's own SSRF protection (UrlHelper::validate refuses private
* and link-local addresses). Feeds are attacker-influenced input; a plugin that
* fetches whatever a feed points at must not be the hole in that.
*
* FirecrawlFetcher is the deliberate exception -- it talks to one operator-configured
* in-cluster endpoint, which UrlHelper would refuse precisely because it is private.
*/
final class TtrssFetcher implements Fetcher {
public function __construct(private readonly int $timeout = 20) {}
public function name(): string { return 'direct'; }
public function fetch(string $url, array $headers = []): FetchResult {
if (!class_exists('\UrlHelper'))
return new FetchResult(error: 'UrlHelper unavailable (not running inside tt-rss)');
$options = [
'url' => $url,
'http_accept' => 'text/*',
'type' => 'text/html',
'timeout' => $this->timeout,
];
// Site configs routinely set a user-agent to get the article rather than a
// consent wall; UrlHelper takes it as a first-class option.
if (isset($headers['user-agent'])) $options['useragent'] = $headers['user-agent'];
$body = \UrlHelper::fetch($options);
if (!is_string($body) || $body === '') {
$error = \UrlHelper::$fetch_last_error ?: 'fetch returned nothing';
return new FetchResult(error: $error, status: (int) \UrlHelper::$fetch_last_error_code);
}
return new FetchResult(
html: $body,
effective_url: \UrlHelper::$fetch_effective_url ?: $url,
status: 200,
);
}
}

12
patches/README.md Normal file
View File

@@ -0,0 +1,12 @@
# Local patches to the vendored dependencies
`vendor/` is committed (tt-rss plugins ship their dependencies; there is no
composer in the app container), so upstream fixes we cannot wait for are applied
here and recorded below.
## fivefilters/readability.php
- `src/Nodes/NodeTrait.php:371` — `callable $filterFn = null` → `?callable
$filterFn = null`. Implicitly nullable parameters are deprecated as of PHP 8.4,
and the tt-rss app image is on PHP 8.5, so without this every extraction emits a
deprecation notice.

10
site_config/custom/andrzejrysuje.pl.txt Normal file
View File

@@ -0,0 +1,10 @@
# Multi-panel strips are laid out as a Bootstrap carousel; taking carousel-inner
# keeps every panel in order. The <figure> alone would give only the first one.
body: //div[contains(concat(' ', normalize-space(@class), ' '), ' carousel-inner ')]
strip_id_or_class: carousel-control
strip_id_or_class: carousel-indicators
autodetect_on_failure: no
test_url: https://andrzejrysuje.pl/tradycja/

12
site_config/custom/blog.cloudflare.com.txt Normal file
View File

@@ -0,0 +1,12 @@
# The community rule still says //div[@class='post-content']. That class survives
# the site's redesign but is now one token among many Tailwind classes, so an
# exact-equality match never fires. article-content is the tighter target anyway.
body: //div[contains(concat(' ', normalize-space(@class), ' '), ' article-content ')]
title: //h1
author: //a[contains(@href, '/author/')]
strip_id_or_class: post-email
strip_id_or_class: image-lightbox
test_url: https://blog.cloudflare.com/bot-preference-sync/

8
site_config/custom/dobreprogramy.pl.txt Normal file
View File

@@ -0,0 +1,8 @@
# The old #phContent_divMetaBody id predates at least two rewrites of this site.
body: //div[contains(concat(' ', normalize-space(@class), ' '), ' article-body-grid ')]
strip_id_or_class: article-bottom
strip_id_or_class: article-img-placeholder
strip_id_or_class: adunit
test_url: https://www.dobreprogramy.pl/windows-11-26h2-coraz-blizej-mala-aktualizacja-z-szybka-instalacja,7321906159909056a

4
site_config/custom/ianlewis.org.txt Normal file
View File

@@ -0,0 +1,4 @@
# Now an h-entry microformat: "post e-content" rather than the old post-content.
body: //div[contains(concat(' ', normalize-space(@class), ' '), ' e-content ')]
test_url: https://www.ianlewis.org/en/2026-new-year-reflections

12
site_config/custom/loadingartist.com.txt Normal file
View File

@@ -0,0 +1,12 @@
# The site was redesigned: the old rule (body: //div[@class='comic']) selects
# nothing, so extraction silently fell through to Readability and dropped the
# comic entirely -- the panel is the article, and Readability keeps text.
body: //div[contains(@class, 'main-image-container')]
body: //div[@class='post-content']
strip: //nav
strip_id_or_class: pagination
autodetect_on_failure: no
test_url: https://loadingartist.com/comic/be-write-back/

7
site_config/custom/oglaf.com.txt Normal file
View File

@@ -0,0 +1,7 @@
# Image-only page: the strip is the article. Readability returns nothing at all
# here because there is no prose to score, which is why this needs a rule.
body: //img[@id='strip']
autodetect_on_failure: no
test_url: https://www.oglaf.com/

16
site_config/custom/securelist.com.txt Normal file
View File

@@ -0,0 +1,16 @@
# Redesigned onto a "c-" component prefix; the community rule's #primary wrapper
# is long gone.
body: //article[contains(concat(' ', normalize-space(@class), ' '), ' c-article ')]
# Sidebar widgets ("From the same authors", "In the same category", webinar
# promos) and the collapsible table of contents all live inside c-article.
strip_id_or_class: c-widget
strip_id_or_class: js-sticky-widget
strip_id_or_class: c-highlight
strip_id_or_class: c-article__authors
strip_id_or_class: c-card
strip_id_or_class: c-share
strip_id_or_class: comment
strip_id_or_class: akismet
test_url: https://securelist.com/android-head-unit-malware/121106/

10
site_config/custom/skeletonclaw.com.txt Normal file
View File

@@ -0,0 +1,10 @@
# Tumblr: the strip sits in the post body; everything else on the page is site
# furniture (logo, banner, Patreon button) that Readability happily keeps.
body: //div[contains(concat(' ', normalize-space(@class), ' '), ' post-content ')]
strip_id_or_class: post-notes
strip_id_or_class: tumblr_controls
autodetect_on_failure: no
test_url: https://www.skeletonclaw.com/post/768403912930836480

15
site_config/custom/tapas.io.txt Normal file
View File

@@ -0,0 +1,15 @@
# Panels are lazy-loaded: @src is a base64 placeholder gif and the real image is
# in @data-src, which the extractor promotes. Without that the episode extracts
# as a column of blank pixels.
#
# Note: tapas signs image URLs with an expiry, so a cached article's panels stop
# resolving after a few weeks. Nothing to be done about that from this end.
body: //div[contains(concat(' ', normalize-space(@class), ' '), ' js-episode-viewer ')]
strip_id_or_class: loading-indicator
strip_id_or_class: episode-info
strip_id_or_class: js-message-body
autodetect_on_failure: no
test_url: https://tapas.io/episode/2549532

22
vendor/autoload.php vendored Normal file
View File

@@ -0,0 +1,22 @@
<?php
// autoload.php @generated by Composer
if (PHP_VERSION_ID < 50600) {
if (!headers_sent()) {
header('HTTP/1.1 500 Internal Server Error');
}
$err = 'Composer 2.3.0 dropped support for autoloading on PHP <5.6 and you are running '.PHP_VERSION.', please upgrade PHP or use Composer 2.2 LTS via "composer self-update --2.2". Aborting.'.PHP_EOL;
if (!ini_get('display_errors')) {
if (PHP_SAPI === 'cli' || PHP_SAPI === 'phpdbg') {
fwrite(STDERR, $err);
} elseif (!headers_sent()) {
echo $err;
}
}
throw new RuntimeException($err);
}
require_once __DIR__ . '/composer/autoload_real.php';
return ComposerAutoloaderInitb44cc79a0eaef9cd9c2f2ac697cbe9c0::getLoader();

579
vendor/composer/ClassLoader.php vendored Normal file
View File

@@ -0,0 +1,579 @@
<?php
/*
* This file is part of Composer.
*
* (c) Nils Adermann <naderman@naderman.de>
* Jordi Boggiano <j.boggiano@seld.be>
*
* For the full copyright and license information, please view the LICENSE
* file that was distributed with this source code.
*/
namespace Composer\Autoload;
/**
* ClassLoader implements a PSR-0, PSR-4 and classmap class loader.
*
* $loader = new \Composer\Autoload\ClassLoader();
*
* // register classes with namespaces
* $loader->add('Symfony\Component', __DIR__.'/component');
* $loader->add('Symfony', __DIR__.'/framework');
*
* // activate the autoloader
* $loader->register();
*
* // to enable searching the include path (eg. for PEAR packages)
* $loader->setUseIncludePath(true);
*
* In this example, if you try to use a class in the Symfony\Component
* namespace or one of its children (Symfony\Component\Console for instance),
* the autoloader will first look for the class under the component/
* directory, and it will then fallback to the framework/ directory if not
* found before giving up.
*
* This class is loosely based on the Symfony UniversalClassLoader.
*
* @author Fabien Potencier <fabien@symfony.com>
* @author Jordi Boggiano <j.boggiano@seld.be>
* @see https://www.php-fig.org/psr/psr-0/
* @see https://www.php-fig.org/psr/psr-4/
*/
class ClassLoader
{
/** @var \Closure(string):void */
private static $includeFile;
/** @var string|null */
private $vendorDir;
// PSR-4
/**
* @var array<string, array<string, int>>
*/
private $prefixLengthsPsr4 = array();
/**
* @var array<string, list<string>>
*/
private $prefixDirsPsr4 = array();
/**
* @var list<string>
*/
private $fallbackDirsPsr4 = array();
// PSR-0
/**
* List of PSR-0 prefixes
*
* Structured as array('F (first letter)' => array('Foo\Bar (full prefix)' => array('path', 'path2')))
*
* @var array<string, array<string, list<string>>>
*/
private $prefixesPsr0 = array();
/**
* @var list<string>
*/
private $fallbackDirsPsr0 = array();
/** @var bool */
private $useIncludePath = false;
/**
* @var array<string, string>
*/
private $classMap = array();
/** @var bool */
private $classMapAuthoritative = false;
/**
* @var array<string, bool>
*/
private $missingClasses = array();
/** @var string|null */
private $apcuPrefix;
/**
* @var array<string, self>
*/
private static $registeredLoaders = array();
/**
* @param string|null $vendorDir
*/
public function __construct($vendorDir = null)
{
$this->vendorDir = $vendorDir;
self::initializeIncludeClosure();
}
/**
* @return array<string, list<string>>
*/
public function getPrefixes()
{
if (!empty($this->prefixesPsr0)) {
return call_user_func_array('array_merge', array_values($this->prefixesPsr0));
}
return array();
}
/**
* @return array<string, list<string>>
*/
public function getPrefixesPsr4()
{
return $this->prefixDirsPsr4;
}
/**
* @return list<string>
*/
public function getFallbackDirs()
{
return $this->fallbackDirsPsr0;
}
/**
* @return list<string>
*/
public function getFallbackDirsPsr4()
{
return $this->fallbackDirsPsr4;
}
/**
* @return array<string, string> Array of classname => path
*/
public function getClassMap()
{
return $this->classMap;
}
/**
* @param array<string, string> $classMap Class to filename map
*
* @return void
*/
public function addClassMap(array $classMap)
{
if ($this->classMap) {
$this->classMap = array_merge($this->classMap, $classMap);
} else {
$this->classMap = $classMap;
}
}
/**
* Registers a set of PSR-0 directories for a given prefix, either
* appending or prepending to the ones previously set for this prefix.
*
* @param string $prefix The prefix
* @param list<string>|string $paths The PSR-0 root directories
* @param bool $prepend Whether to prepend the directories
*
* @return void
*/
public function add($prefix, $paths, $prepend = false)
{
$paths = (array) $paths;
if (!$prefix) {
if ($prepend) {
$this->fallbackDirsPsr0 = array_merge(
$paths,
$this->fallbackDirsPsr0
);
} else {
$this->fallbackDirsPsr0 = array_merge(
$this->fallbackDirsPsr0,
$paths
);
}
return;
}
$first = $prefix[0];
if (!isset($this->prefixesPsr0[$first][$prefix])) {
$this->prefixesPsr0[$first][$prefix] = $paths;
return;
}
if ($prepend) {
$this->prefixesPsr0[$first][$prefix] = array_merge(
$paths,
$this->prefixesPsr0[$first][$prefix]
);
} else {
$this->prefixesPsr0[$first][$prefix] = array_merge(
$this->prefixesPsr0[$first][$prefix],
$paths
);
}
}
/**
* Registers a set of PSR-4 directories for a given namespace, either
* appending or prepending to the ones previously set for this namespace.
*
* @param string $prefix The prefix/namespace, with trailing '\\'
* @param list<string>|string $paths The PSR-4 base directories
* @param bool $prepend Whether to prepend the directories
*
* @throws \InvalidArgumentException
*
* @return void
*/
public function addPsr4($prefix, $paths, $prepend = false)
{
$paths = (array) $paths;
if (!$prefix) {
// Register directories for the root namespace.
if ($prepend) {
$this->fallbackDirsPsr4 = array_merge(
$paths,
$this->fallbackDirsPsr4
);
} else {
$this->fallbackDirsPsr4 = array_merge(
$this->fallbackDirsPsr4,
$paths
);
}
} elseif (!isset($this->prefixDirsPsr4[$prefix])) {
// Register directories for a new namespace.
$length = strlen($prefix);
if ('\\' !== $prefix[$length - 1]) {
throw new \InvalidArgumentException("A non-empty PSR-4 prefix must end with a namespace separator.");
}
$this->prefixLengthsPsr4[$prefix[0]][$prefix] = $length;
$this->prefixDirsPsr4[$prefix] = $paths;
} elseif ($prepend) {
// Prepend directories for an already registered namespace.
$this->prefixDirsPsr4[$prefix] = array_merge(
$paths,
$this->prefixDirsPsr4[$prefix]
);
} else {
// Append directories for an already registered namespace.
$this->prefixDirsPsr4[$prefix] = array_merge(
$this->prefixDirsPsr4[$prefix],
$paths
);
}
}
/**
* Registers a set of PSR-0 directories for a given prefix,
* replacing any others previously set for this prefix.
*
* @param string $prefix The prefix
* @param list<string>|string $paths The PSR-0 base directories
*
* @return void
*/
public function set($prefix, $paths)
{
if (!$prefix) {
$this->fallbackDirsPsr0 = (array) $paths;
} else {
$this->prefixesPsr0[$prefix[0]][$prefix] = (array) $paths;
}
}
/**
* Registers a set of PSR-4 directories for a given namespace,
* replacing any others previously set for this namespace.
*
* @param string $prefix The prefix/namespace, with trailing '\\'
* @param list<string>|string $paths The PSR-4 base directories
*
* @throws \InvalidArgumentException
*
* @return void
*/
public function setPsr4($prefix, $paths)
{
if (!$prefix) {
$this->fallbackDirsPsr4 = (array) $paths;
} else {
$length = strlen($prefix);
if ('\\' !== $prefix[$length - 1]) {
throw new \InvalidArgumentException("A non-empty PSR-4 prefix must end with a namespace separator.");
}
$this->prefixLengthsPsr4[$prefix[0]][$prefix] = $length;
$this->prefixDirsPsr4[$prefix] = (array) $paths;
}
}
/**
* Turns on searching the include path for class files.
*
* @param bool $useIncludePath
*
* @return void
*/
public function setUseIncludePath($useIncludePath)
{
$this->useIncludePath = $useIncludePath;
}
/**
* Can be used to check if the autoloader uses the include path to check
* for classes.
*
* @return bool
*/
public function getUseIncludePath()
{
return $this->useIncludePath;
}
/**
* Turns off searching the prefix and fallback directories for classes
* that have not been registered with the class map.
*
* @param bool $classMapAuthoritative
*
* @return void
*/
public function setClassMapAuthoritative($classMapAuthoritative)
{
$this->classMapAuthoritative = $classMapAuthoritative;
}
/**
* Should class lookup fail if not found in the current class map?
*
* @return bool
*/
public function isClassMapAuthoritative()
{
return $this->classMapAuthoritative;
}
/**
* APCu prefix to use to cache found/not-found classes, if the extension is enabled.
*
* @param string|null $apcuPrefix
*
* @return void
*/
public function setApcuPrefix($apcuPrefix)
{
$this->apcuPrefix = function_exists('apcu_fetch') && filter_var(ini_get('apc.enabled'), FILTER_VALIDATE_BOOLEAN) ? $apcuPrefix : null;
}
/**
* The APCu prefix in use, or null if APCu caching is not enabled.
*
* @return string|null
*/
public function getApcuPrefix()
{
return $this->apcuPrefix;
}
/**
* Registers this instance as an autoloader.
*
* @param bool $prepend Whether to prepend the autoloader or not
*
* @return void
*/
public function register($prepend = false)
{
spl_autoload_register(array($this, 'loadClass'), true, $prepend);
if (null === $this->vendorDir) {
return;
}
if ($prepend) {
self::$registeredLoaders = array($this->vendorDir => $this) + self::$registeredLoaders;
} else {
unset(self::$registeredLoaders[$this->vendorDir]);
self::$registeredLoaders[$this->vendorDir] = $this;
}
}
/**
* Unregisters this instance as an autoloader.
*
* @return void
*/
public function unregister()
{
spl_autoload_unregister(array($this, 'loadClass'));
if (null !== $this->vendorDir) {
unset(self::$registeredLoaders[$this->vendorDir]);
}
}
/**
* Loads the given class or interface.
*
* @param string $class The name of the class
* @return true|null True if loaded, null otherwise
*/
public function loadClass($class)
{
if ($file = $this->findFile($class)) {
$includeFile = self::$includeFile;
$includeFile($file);
return true;
}
return null;
}
/**
* Finds the path to the file where the class is defined.
*
* @param string $class The name of the class
*
* @return string|false The path if found, false otherwise
*/
public function findFile($class)
{
// class map lookup
if (isset($this->classMap[$class])) {
return $this->classMap[$class];
}
if ($this->classMapAuthoritative || isset($this->missingClasses[$class])) {
return false;
}
if (null !== $this->apcuPrefix) {
$file = apcu_fetch($this->apcuPrefix.$class, $hit);
if ($hit) {
return $file;
}
}
$file = $this->findFileWithExtension($class, '.php');
// Search for Hack files if we are running on HHVM
if (false === $file && defined('HHVM_VERSION')) {
$file = $this->findFileWithExtension($class, '.hh');
}
if (null !== $this->apcuPrefix) {
apcu_add($this->apcuPrefix.$class, $file);
}
if (false === $file) {
// Remember that this class does not exist.
$this->missingClasses[$class] = true;
}
return $file;
}
/**
* Returns the currently registered loaders keyed by their corresponding vendor directories.
*
* @return array<string, self>
*/
public static function getRegisteredLoaders()
{
return self::$registeredLoaders;
}
/**
* @param string $class
* @param string $ext
* @return string|false
*/
private function findFileWithExtension($class, $ext)
{
// PSR-4 lookup
$logicalPathPsr4 = strtr($class, '\\', DIRECTORY_SEPARATOR) . $ext;
$first = $class[0];
if (isset($this->prefixLengthsPsr4[$first])) {
$subPath = $class;
while (false !== $lastPos = strrpos($subPath, '\\')) {
$subPath = substr($subPath, 0, $lastPos);
$search = $subPath . '\\';
if (isset($this->prefixDirsPsr4[$search])) {
$pathEnd = DIRECTORY_SEPARATOR . substr($logicalPathPsr4, $lastPos + 1);
foreach ($this->prefixDirsPsr4[$search] as $dir) {
if (file_exists($file = $dir . $pathEnd)) {
return $file;
}
}
}
}
}
// PSR-4 fallback dirs
foreach ($this->fallbackDirsPsr4 as $dir) {
if (file_exists($file = $dir . DIRECTORY_SEPARATOR . $logicalPathPsr4)) {
return $file;
}
}
// PSR-0 lookup
if (false !== $pos = strrpos($class, '\\')) {
// namespaced class name
$logicalPathPsr0 = substr($logicalPathPsr4, 0, $pos + 1)
. strtr(substr($logicalPathPsr4, $pos + 1), '_', DIRECTORY_SEPARATOR);
} else {
// PEAR-like class name
$logicalPathPsr0 = strtr($class, '_', DIRECTORY_SEPARATOR) . $ext;
}
if (isset($this->prefixesPsr0[$first])) {
foreach ($this->prefixesPsr0[$first] as $prefix => $dirs) {
if (0 === strpos($class, $prefix)) {
foreach ($dirs as $dir) {
if (file_exists($file = $dir . DIRECTORY_SEPARATOR . $logicalPathPsr0)) {
return $file;
}
}
}
}
}
// PSR-0 fallback dirs
foreach ($this->fallbackDirsPsr0 as $dir) {
if (file_exists($file = $dir . DIRECTORY_SEPARATOR . $logicalPathPsr0)) {
return $file;
}
}
// PSR-0 include paths.
if ($this->useIncludePath && $file = stream_resolve_include_path($logicalPathPsr0)) {
return $file;
}
return false;
}
/**
* @return void
*/
private static function initializeIncludeClosure()
{
if (self::$includeFile !== null) {
return;
}
/**
* Scope isolated include.
*
* Prevents access to $this/self from included files.
*
* @param string $file
* @return void
*/
self::$includeFile = \Closure::bind(static function($file) {
include $file;
}, null, null);
}
}

396
vendor/composer/InstalledVersions.php vendored Normal file
View File

@@ -0,0 +1,396 @@
<?php
/*
* This file is part of Composer.
*
* (c) Nils Adermann <naderman@naderman.de>
* Jordi Boggiano <j.boggiano@seld.be>
*
* For the full copyright and license information, please view the LICENSE
* file that was distributed with this source code.
*/
namespace Composer;
use Composer\Autoload\ClassLoader;
use Composer\Semver\VersionParser;
/**
* This class is copied in every Composer installed project and available to all
*
* See also https://getcomposer.org/doc/07-runtime.md#installed-versions
*
* To require its presence, you can require `composer-runtime-api ^2.0`
*
* @final
*/
class InstalledVersions
{
/**
* @var string|null if set (by reflection by Composer), this should be set to the path where this class is being copied to
* @internal
*/
private static $selfDir = null;
/**
* @var mixed[]|null
* @psalm-var array{root: array{name: string, pretty_version: string, version: string, reference: string|null, type: string, install_path: string, aliases: string[], dev: bool}, versions: array<string, array{pretty_version?: string, version?: string, reference?: string|null, type?: string, install_path?: string, aliases?: string[], dev_requirement: bool, replaced?: string[], provided?: string[]}>}|array{}|null
*/
private static $installed;
/**
* @var bool
*/
private static $installedIsLocalDir;
/**
* @var bool|null
*/
private static $canGetVendors;
/**
* @var array[]
* @psalm-var array<string, array{root: array{name: string, pretty_version: string, version: string, reference: string|null, type: string, install_path: string, aliases: string[], dev: bool}, versions: array<string, array{pretty_version?: string, version?: string, reference?: string|null, type?: string, install_path?: string, aliases?: string[], dev_requirement: bool, replaced?: string[], provided?: string[]}>}>
*/
private static $installedByVendor = array();
/**
* Returns a list of all package names which are present, either by being installed, replaced or provided
*
* @return string[]
* @psalm-return list<string>
*/
public static function getInstalledPackages()
{
$packages = array();
foreach (self::getInstalled() as $installed) {
$packages[] = array_keys($installed['versions']);
}
if (1 === \count($packages)) {
return $packages[0];
}
return array_keys(array_flip(\call_user_func_array('array_merge', $packages)));
}
/**
* Returns a list of all package names with a specific type e.g. 'library'
*
* @param string $type
* @return string[]
* @psalm-return list<string>
*/
public static function getInstalledPackagesByType($type)
{
$packagesByType = array();
foreach (self::getInstalled() as $installed) {
foreach ($installed['versions'] as $name => $package) {
if (isset($package['type']) && $package['type'] === $type) {
$packagesByType[] = $name;
}
}
}
return $packagesByType;
}
/**
* Checks whether the given package is installed
*
* This also returns true if the package name is provided or replaced by another package
*
* @param string $packageName
* @param bool $includeDevRequirements
* @return bool
*/
public static function isInstalled($packageName, $includeDevRequirements = true)
{
foreach (self::getInstalled() as $installed) {
if (isset($installed['versions'][$packageName])) {
return $includeDevRequirements || !isset($installed['versions'][$packageName]['dev_requirement']) || $installed['versions'][$packageName]['dev_requirement'] === false;
}
}
return false;
}
/**
* Checks whether the given package satisfies a version constraint
*
* e.g. If you want to know whether version 2.3+ of package foo/bar is installed, you would call:
*
* Composer\InstalledVersions::satisfies(new VersionParser, 'foo/bar', '^2.3')
*
* @param VersionParser $parser Install composer/semver to have access to this class and functionality
* @param string $packageName
* @param string|null $constraint A version constraint to check for, if you pass one you have to make sure composer/semver is required by your package
* @return bool
*/
public static function satisfies(VersionParser $parser, $packageName, $constraint)
{
$constraint = $parser->parseConstraints((string) $constraint);
$provided = $parser->parseConstraints(self::getVersionRanges($packageName));
return $provided->matches($constraint);
}
/**
* Returns a version constraint representing all the range(s) which are installed for a given package
*
* It is easier to use this via isInstalled() with the $constraint argument if you need to check
* whether a given version of a package is installed, and not just whether it exists
*
* @param string $packageName
* @return string Version constraint usable with composer/semver
*/
public static function getVersionRanges($packageName)
{
foreach (self::getInstalled() as $installed) {
if (!isset($installed['versions'][$packageName])) {
continue;
}
$ranges = array();
if (isset($installed['versions'][$packageName]['pretty_version'])) {
$ranges[] = $installed['versions'][$packageName]['pretty_version'];
}
if (array_key_exists('aliases', $installed['versions'][$packageName])) {
$ranges = array_merge($ranges, $installed['versions'][$packageName]['aliases']);
}
if (array_key_exists('replaced', $installed['versions'][$packageName])) {
$ranges = array_merge($ranges, $installed['versions'][$packageName]['replaced']);
}
if (array_key_exists('provided', $installed['versions'][$packageName])) {
$ranges = array_merge($ranges, $installed['versions'][$packageName]['provided']);
}
return implode(' || ', $ranges);
}
throw new \OutOfBoundsException('Package "' . $packageName . '" is not installed');
}
/**
* @param string $packageName
* @return string|null If the package is being replaced or provided but is not really installed, null will be returned as version, use satisfies or getVersionRanges if you need to know if a given version is present
*/
public static function getVersion($packageName)
{
foreach (self::getInstalled() as $installed) {
if (!isset($installed['versions'][$packageName])) {
continue;
}
if (!isset($installed['versions'][$packageName]['version'])) {
return null;
}
return $installed['versions'][$packageName]['version'];
}
throw new \OutOfBoundsException('Package "' . $packageName . '" is not installed');
}
/**
* @param string $packageName
* @return string|null If the package is being replaced or provided but is not really installed, null will be returned as version, use satisfies or getVersionRanges if you need to know if a given version is present
*/
public static function getPrettyVersion($packageName)
{
foreach (self::getInstalled() as $installed) {
if (!isset($installed['versions'][$packageName])) {
continue;
}
if (!isset($installed['versions'][$packageName]['pretty_version'])) {
return null;
}
return $installed['versions'][$packageName]['pretty_version'];
}
throw new \OutOfBoundsException('Package "' . $packageName . '" is not installed');
}
/**
* @param string $packageName
* @return string|null If the package is being replaced or provided but is not really installed, null will be returned as reference
*/
public static function getReference($packageName)
{
foreach (self::getInstalled() as $installed) {
if (!isset($installed['versions'][$packageName])) {
continue;
}
if (!isset($installed['versions'][$packageName]['reference'])) {
return null;
}
return $installed['versions'][$packageName]['reference'];
}
throw new \OutOfBoundsException('Package "' . $packageName . '" is not installed');
}
/**
* @param string $packageName
* @return string|null If the package is being replaced or provided but is not really installed, null will be returned as install path. Packages of type metapackages also have a null install path.
*/
public static function getInstallPath($packageName)
{
foreach (self::getInstalled() as $installed) {
if (!isset($installed['versions'][$packageName])) {
continue;
}
return isset($installed['versions'][$packageName]['install_path']) ? $installed['versions'][$packageName]['install_path'] : null;
}
throw new \OutOfBoundsException('Package "' . $packageName . '" is not installed');
}
/**
* @return array
* @psalm-return array{name: string, pretty_version: string, version: string, reference: string|null, type: string, install_path: string, aliases: string[], dev: bool}
*/
public static function getRootPackage()
{
$installed = self::getInstalled();
return $installed[0]['root'];
}
/**
* Returns the raw installed.php data for custom implementations
*
* @deprecated Use getAllRawData() instead which returns all datasets for all autoloaders present in the process. getRawData only returns the first dataset loaded, which may not be what you expect.
* @return array[]
* @psalm-return array{root: array{name: string, pretty_version: string, version: string, reference: string|null, type: string, install_path: string, aliases: string[], dev: bool}, versions: array<string, array{pretty_version?: string, version?: string, reference?: string|null, type?: string, install_path?: string, aliases?: string[], dev_requirement: bool, replaced?: string[], provided?: string[]}>}
*/
public static function getRawData()
{
@trigger_error('getRawData only returns the first dataset loaded, which may not be what you expect. Use getAllRawData() instead which returns all datasets for all autoloaders present in the process.', E_USER_DEPRECATED);
if (null === self::$installed) {
// only require the installed.php file if this file is loaded from its dumped location,
// and not from its source location in the composer/composer package, see https://github.com/composer/composer/issues/9937
if (substr(__DIR__, -8, 1) !== 'C') {
self::$installed = include __DIR__ . '/installed.php';
} else {
self::$installed = array();
}
}
return self::$installed;
}
/**
* Returns the raw data of all installed.php which are currently loaded for custom implementations
*
* @return array[]
* @psalm-return list<array{root: array{name: string, pretty_version: string, version: string, reference: string|null, type: string, install_path: string, aliases: string[], dev: bool}, versions: array<string, array{pretty_version?: string, version?: string, reference?: string|null, type?: string, install_path?: string, aliases?: string[], dev_requirement: bool, replaced?: string[], provided?: string[]}>}>
*/
public static function getAllRawData()
{
return self::getInstalled();
}
/**
* Lets you reload the static array from another file
*
* This is only useful for complex integrations in which a project needs to use
* this class but then also needs to execute another project's autoloader in process,
* and wants to ensure both projects have access to their version of installed.php.
*
* A typical case would be PHPUnit, where it would need to make sure it reads all
* the data it needs from this class, then call reload() with
* `require $CWD/vendor/composer/installed.php` (or similar) as input to make sure
* the project in which it runs can then also use this class safely, without
* interference between PHPUnit's dependencies and the project's dependencies.
*
* @param array[] $data A vendor/composer/installed.php data set
* @return void
*
* @psalm-param array{root: array{name: string, pretty_version: string, version: string, reference: string|null, type: string, install_path: string, aliases: string[], dev: bool}, versions: array<string, array{pretty_version?: string, version?: string, reference?: string|null, type?: string, install_path?: string, aliases?: string[], dev_requirement: bool, replaced?: string[], provided?: string[]}>} $data
*/
public static function reload($data)
{
self::$installed = $data;
self::$installedByVendor = array();
// when using reload, we disable the duplicate protection to ensure that self::$installed data is
// always returned, but we cannot know whether it comes from the installed.php in __DIR__ or not,
// so we have to assume it does not, and that may result in duplicate data being returned when listing
// all installed packages for example
self::$installedIsLocalDir = false;
}
/**
* @return string
*/
private static function getSelfDir()
{
if (self::$selfDir === null) {
self::$selfDir = strtr(__DIR__, '\\', '/');
}
return self::$selfDir;
}
/**
* @return array[]
* @psalm-return list<array{root: array{name: string, pretty_version: string, version: string, reference: string|null, type: string, install_path: string, aliases: string[], dev: bool}, versions: array<string, array{pretty_version?: string, version?: string, reference?: string|null, type?: string, install_path?: string, aliases?: string[], dev_requirement: bool, replaced?: string[], provided?: string[]}>}>
*/
private static function getInstalled()
{
if (null === self::$canGetVendors) {
self::$canGetVendors = method_exists('Composer\Autoload\ClassLoader', 'getRegisteredLoaders');
}
$installed = array();
$copiedLocalDir = false;
if (self::$canGetVendors) {
$selfDir = self::getSelfDir();
foreach (ClassLoader::getRegisteredLoaders() as $vendorDir => $loader) {
$vendorDir = strtr($vendorDir, '\\', '/');
if (isset(self::$installedByVendor[$vendorDir])) {
$installed[] = self::$installedByVendor[$vendorDir];
} elseif (is_file($vendorDir.'/composer/installed.php')) {
/** @var array{root: array{name: string, pretty_version: string, version: string, reference: string|null, type: string, install_path: string, aliases: string[], dev: bool}, versions: array<string, array{pretty_version?: string, version?: string, reference?: string|null, type?: string, install_path?: string, aliases?: string[], dev_requirement: bool, replaced?: string[], provided?: string[]}>} $required */
$required = require $vendorDir.'/composer/installed.php';
self::$installedByVendor[$vendorDir] = $required;
$installed[] = $required;
if (self::$installed === null && $vendorDir.'/composer' === $selfDir) {
self::$installed = $required;
self::$installedIsLocalDir = true;
}
}
if (self::$installedIsLocalDir && $vendorDir.'/composer' === $selfDir) {
$copiedLocalDir = true;
}
}
}
if (null === self::$installed) {
// only require the installed.php file if this file is loaded from its dumped location,
// and not from its source location in the composer/composer package, see https://github.com/composer/composer/issues/9937
if (substr(__DIR__, -8, 1) !== 'C') {
/** @var array{root: array{name: string, pretty_version: string, version: string, reference: string|null, type: string, install_path: string, aliases: string[], dev: bool}, versions: array<string, array{pretty_version?: string, version?: string, reference?: string|null, type?: string, install_path?: string, aliases?: string[], dev_requirement: bool, replaced?: string[], provided?: string[]}>} $required */
$required = require __DIR__ . '/installed.php';
self::$installed = $required;
} else {
self::$installed = array();
}
}
if (self::$installed !== array() && !$copiedLocalDir) {
$installed[] = self::$installed;
}
return $installed;
}
}

21
vendor/composer/LICENSE vendored Normal file
View File

@@ -0,0 +1,21 @@
Copyright (c) Nils Adermann, Jordi Boggiano
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is furnished
to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

61
vendor/composer/autoload_classmap.php vendored Normal file
View File

@@ -0,0 +1,61 @@
<?php
// autoload_classmap.php @generated by Composer
$vendorDir = dirname(__DIR__);
$baseDir = dirname($vendorDir);
return array(
'Composer\\InstalledVersions' => $vendorDir . '/composer/InstalledVersions.php',
'Masterminds\\HTML5' => $vendorDir . '/masterminds/html5/src/HTML5.php',
'Masterminds\\HTML5\\Elements' => $vendorDir . '/masterminds/html5/src/HTML5/Elements.php',
'Masterminds\\HTML5\\Entities' => $vendorDir . '/masterminds/html5/src/HTML5/Entities.php',
'Masterminds\\HTML5\\Exception' => $vendorDir . '/masterminds/html5/src/HTML5/Exception.php',
'Masterminds\\HTML5\\InstructionProcessor' => $vendorDir . '/masterminds/html5/src/HTML5/InstructionProcessor.php',
'Masterminds\\HTML5\\Parser\\CharacterReference' => $vendorDir . '/masterminds/html5/src/HTML5/Parser/CharacterReference.php',
'Masterminds\\HTML5\\Parser\\DOMTreeBuilder' => $vendorDir . '/masterminds/html5/src/HTML5/Parser/DOMTreeBuilder.php',
'Masterminds\\HTML5\\Parser\\EventHandler' => $vendorDir . '/masterminds/html5/src/HTML5/Parser/EventHandler.php',
'Masterminds\\HTML5\\Parser\\FileInputStream' => $vendorDir . '/masterminds/html5/src/HTML5/Parser/FileInputStream.php',
'Masterminds\\HTML5\\Parser\\InputStream' => $vendorDir . '/masterminds/html5/src/HTML5/Parser/InputStream.php',
'Masterminds\\HTML5\\Parser\\ParseError' => $vendorDir . '/masterminds/html5/src/HTML5/Parser/ParseError.php',
'Masterminds\\HTML5\\Parser\\Scanner' => $vendorDir . '/masterminds/html5/src/HTML5/Parser/Scanner.php',
'Masterminds\\HTML5\\Parser\\StringInputStream' => $vendorDir . '/masterminds/html5/src/HTML5/Parser/StringInputStream.php',
'Masterminds\\HTML5\\Parser\\Tokenizer' => $vendorDir . '/masterminds/html5/src/HTML5/Parser/Tokenizer.php',
'Masterminds\\HTML5\\Parser\\TreeBuildingRules' => $vendorDir . '/masterminds/html5/src/HTML5/Parser/TreeBuildingRules.php',
'Masterminds\\HTML5\\Parser\\UTF8Utils' => $vendorDir . '/masterminds/html5/src/HTML5/Parser/UTF8Utils.php',
'Masterminds\\HTML5\\Serializer\\HTML5Entities' => $vendorDir . '/masterminds/html5/src/HTML5/Serializer/HTML5Entities.php',
'Masterminds\\HTML5\\Serializer\\OutputRules' => $vendorDir . '/masterminds/html5/src/HTML5/Serializer/OutputRules.php',
'Masterminds\\HTML5\\Serializer\\RulesInterface' => $vendorDir . '/masterminds/html5/src/HTML5/Serializer/RulesInterface.php',
'Masterminds\\HTML5\\Serializer\\Traverser' => $vendorDir . '/masterminds/html5/src/HTML5/Serializer/Traverser.php',
'Psr\\Log\\AbstractLogger' => $vendorDir . '/psr/log/Psr/Log/AbstractLogger.php',
'Psr\\Log\\InvalidArgumentException' => $vendorDir . '/psr/log/Psr/Log/InvalidArgumentException.php',
'Psr\\Log\\LogLevel' => $vendorDir . '/psr/log/Psr/Log/LogLevel.php',
'Psr\\Log\\LoggerAwareInterface' => $vendorDir . '/psr/log/Psr/Log/LoggerAwareInterface.php',
'Psr\\Log\\LoggerAwareTrait' => $vendorDir . '/psr/log/Psr/Log/LoggerAwareTrait.php',
'Psr\\Log\\LoggerInterface' => $vendorDir . '/psr/log/Psr/Log/LoggerInterface.php',
'Psr\\Log\\LoggerTrait' => $vendorDir . '/psr/log/Psr/Log/LoggerTrait.php',
'Psr\\Log\\NullLogger' => $vendorDir . '/psr/log/Psr/Log/NullLogger.php',
'Psr\\Log\\Test\\DummyTest' => $vendorDir . '/psr/log/Psr/Log/Test/DummyTest.php',
'Psr\\Log\\Test\\LoggerInterfaceTest' => $vendorDir . '/psr/log/Psr/Log/Test/LoggerInterfaceTest.php',
'Psr\\Log\\Test\\TestLogger' => $vendorDir . '/psr/log/Psr/Log/Test/TestLogger.php',
'fivefilters\\Readability\\Configuration' => $vendorDir . '/fivefilters/readability.php/src/Configuration.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMAttr' => $vendorDir . '/fivefilters/readability.php/src/Nodes/DOM/DOMAttr.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMCdataSection' => $vendorDir . '/fivefilters/readability.php/src/Nodes/DOM/DOMCdataSection.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMCharacterData' => $vendorDir . '/fivefilters/readability.php/src/Nodes/DOM/DOMCharacterData.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMComment' => $vendorDir . '/fivefilters/readability.php/src/Nodes/DOM/DOMComment.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMDocument' => $vendorDir . '/fivefilters/readability.php/src/Nodes/DOM/DOMDocument.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMDocumentFragment' => $vendorDir . '/fivefilters/readability.php/src/Nodes/DOM/DOMDocumentFragment.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMDocumentType' => $vendorDir . '/fivefilters/readability.php/src/Nodes/DOM/DOMDocumentType.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMElement' => $vendorDir . '/fivefilters/readability.php/src/Nodes/DOM/DOMElement.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMEntity' => $vendorDir . '/fivefilters/readability.php/src/Nodes/DOM/DOMEntity.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMEntityReference' => $vendorDir . '/fivefilters/readability.php/src/Nodes/DOM/DOMEntityReference.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMNode' => $vendorDir . '/fivefilters/readability.php/src/Nodes/DOM/DOMNode.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMNodeList' => $vendorDir . '/fivefilters/readability.php/src/Nodes/DOM/DOMNodeList.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMNotation' => $vendorDir . '/fivefilters/readability.php/src/Nodes/DOM/DOMNotation.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMProcessingInstruction' => $vendorDir . '/fivefilters/readability.php/src/Nodes/DOM/DOMProcessingInstruction.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMText' => $vendorDir . '/fivefilters/readability.php/src/Nodes/DOM/DOMText.php',
'fivefilters\\Readability\\Nodes\\NodeTrait' => $vendorDir . '/fivefilters/readability.php/src/Nodes/NodeTrait.php',
'fivefilters\\Readability\\Nodes\\NodeUtility' => $vendorDir . '/fivefilters/readability.php/src/Nodes/NodeUtility.php',
'fivefilters\\Readability\\ParseException' => $vendorDir . '/fivefilters/readability.php/src/ParseException.php',
'fivefilters\\Readability\\Readability' => $vendorDir . '/fivefilters/readability.php/src/Readability.php',
);

9
vendor/composer/autoload_namespaces.php vendored Normal file
View File

@@ -0,0 +1,9 @@
<?php
// autoload_namespaces.php @generated by Composer
$vendorDir = dirname(__DIR__);
$baseDir = dirname($vendorDir);
return array(
);

12
vendor/composer/autoload_psr4.php vendored Normal file
View File

@@ -0,0 +1,12 @@
<?php
// autoload_psr4.php @generated by Composer
$vendorDir = dirname(__DIR__);
$baseDir = dirname($vendorDir);
return array(
'fivefilters\\Readability\\' => array($vendorDir . '/fivefilters/readability.php/src'),
'Psr\\Log\\' => array($vendorDir . '/psr/log/Psr/Log'),
'Masterminds\\' => array($vendorDir . '/masterminds/html5/src'),
);

38
vendor/composer/autoload_real.php vendored Normal file
View File

@@ -0,0 +1,38 @@
<?php
// autoload_real.php @generated by Composer
class ComposerAutoloaderInitb44cc79a0eaef9cd9c2f2ac697cbe9c0
{
private static $loader;
public static function loadClassLoader($class)
{
if ('Composer\Autoload\ClassLoader' === $class) {
require __DIR__ . '/ClassLoader.php';
}
}
/**
* @return \Composer\Autoload\ClassLoader
*/
public static function getLoader()
{
if (null !== self::$loader) {
return self::$loader;
}
require __DIR__ . '/platform_check.php';
spl_autoload_register(array('ComposerAutoloaderInitb44cc79a0eaef9cd9c2f2ac697cbe9c0', 'loadClassLoader'), true, true);
self::$loader = $loader = new \Composer\Autoload\ClassLoader(\dirname(__DIR__));
spl_autoload_unregister(array('ComposerAutoloaderInitb44cc79a0eaef9cd9c2f2ac697cbe9c0', 'loadClassLoader'));
require __DIR__ . '/autoload_static.php';
call_user_func(\Composer\Autoload\ComposerStaticInitb44cc79a0eaef9cd9c2f2ac697cbe9c0::getInitializer($loader));
$loader->register(true);
return $loader;
}
}

103
vendor/composer/autoload_static.php vendored Normal file
View File

@@ -0,0 +1,103 @@
<?php
// autoload_static.php @generated by Composer
namespace Composer\Autoload;
class ComposerStaticInitb44cc79a0eaef9cd9c2f2ac697cbe9c0
{
public static $prefixLengthsPsr4 = array (
'f' =>
array (
'fivefilters\\Readability\\' => 24,
),
'P' =>
array (
'Psr\\Log\\' => 8,
),
'M' =>
array (
'Masterminds\\' => 12,
),
);
public static $prefixDirsPsr4 = array (
'fivefilters\\Readability\\' =>
array (
0 => __DIR__ . '/..' . '/fivefilters/readability.php/src',
),
'Psr\\Log\\' =>
array (
0 => __DIR__ . '/..' . '/psr/log/Psr/Log',
),
'Masterminds\\' =>
array (
0 => __DIR__ . '/..' . '/masterminds/html5/src',
),
);
public static $classMap = array (
'Composer\\InstalledVersions' => __DIR__ . '/..' . '/composer/InstalledVersions.php',
'Masterminds\\HTML5' => __DIR__ . '/..' . '/masterminds/html5/src/HTML5.php',
'Masterminds\\HTML5\\Elements' => __DIR__ . '/..' . '/masterminds/html5/src/HTML5/Elements.php',
'Masterminds\\HTML5\\Entities' => __DIR__ . '/..' . '/masterminds/html5/src/HTML5/Entities.php',
'Masterminds\\HTML5\\Exception' => __DIR__ . '/..' . '/masterminds/html5/src/HTML5/Exception.php',
'Masterminds\\HTML5\\InstructionProcessor' => __DIR__ . '/..' . '/masterminds/html5/src/HTML5/InstructionProcessor.php',
'Masterminds\\HTML5\\Parser\\CharacterReference' => __DIR__ . '/..' . '/masterminds/html5/src/HTML5/Parser/CharacterReference.php',
'Masterminds\\HTML5\\Parser\\DOMTreeBuilder' => __DIR__ . '/..' . '/masterminds/html5/src/HTML5/Parser/DOMTreeBuilder.php',
'Masterminds\\HTML5\\Parser\\EventHandler' => __DIR__ . '/..' . '/masterminds/html5/src/HTML5/Parser/EventHandler.php',
'Masterminds\\HTML5\\Parser\\FileInputStream' => __DIR__ . '/..' . '/masterminds/html5/src/HTML5/Parser/FileInputStream.php',
'Masterminds\\HTML5\\Parser\\InputStream' => __DIR__ . '/..' . '/masterminds/html5/src/HTML5/Parser/InputStream.php',
'Masterminds\\HTML5\\Parser\\ParseError' => __DIR__ . '/..' . '/masterminds/html5/src/HTML5/Parser/ParseError.php',
'Masterminds\\HTML5\\Parser\\Scanner' => __DIR__ . '/..' . '/masterminds/html5/src/HTML5/Parser/Scanner.php',
'Masterminds\\HTML5\\Parser\\StringInputStream' => __DIR__ . '/..' . '/masterminds/html5/src/HTML5/Parser/StringInputStream.php',
'Masterminds\\HTML5\\Parser\\Tokenizer' => __DIR__ . '/..' . '/masterminds/html5/src/HTML5/Parser/Tokenizer.php',
'Masterminds\\HTML5\\Parser\\TreeBuildingRules' => __DIR__ . '/..' . '/masterminds/html5/src/HTML5/Parser/TreeBuildingRules.php',
'Masterminds\\HTML5\\Parser\\UTF8Utils' => __DIR__ . '/..' . '/masterminds/html5/src/HTML5/Parser/UTF8Utils.php',
'Masterminds\\HTML5\\Serializer\\HTML5Entities' => __DIR__ . '/..' . '/masterminds/html5/src/HTML5/Serializer/HTML5Entities.php',
'Masterminds\\HTML5\\Serializer\\OutputRules' => __DIR__ . '/..' . '/masterminds/html5/src/HTML5/Serializer/OutputRules.php',
'Masterminds\\HTML5\\Serializer\\RulesInterface' => __DIR__ . '/..' . '/masterminds/html5/src/HTML5/Serializer/RulesInterface.php',
'Masterminds\\HTML5\\Serializer\\Traverser' => __DIR__ . '/..' . '/masterminds/html5/src/HTML5/Serializer/Traverser.php',
'Psr\\Log\\AbstractLogger' => __DIR__ . '/..' . '/psr/log/Psr/Log/AbstractLogger.php',
'Psr\\Log\\InvalidArgumentException' => __DIR__ . '/..' . '/psr/log/Psr/Log/InvalidArgumentException.php',
'Psr\\Log\\LogLevel' => __DIR__ . '/..' . '/psr/log/Psr/Log/LogLevel.php',
'Psr\\Log\\LoggerAwareInterface' => __DIR__ . '/..' . '/psr/log/Psr/Log/LoggerAwareInterface.php',
'Psr\\Log\\LoggerAwareTrait' => __DIR__ . '/..' . '/psr/log/Psr/Log/LoggerAwareTrait.php',
'Psr\\Log\\LoggerInterface' => __DIR__ . '/..' . '/psr/log/Psr/Log/LoggerInterface.php',
'Psr\\Log\\LoggerTrait' => __DIR__ . '/..' . '/psr/log/Psr/Log/LoggerTrait.php',
'Psr\\Log\\NullLogger' => __DIR__ . '/..' . '/psr/log/Psr/Log/NullLogger.php',
'Psr\\Log\\Test\\DummyTest' => __DIR__ . '/..' . '/psr/log/Psr/Log/Test/DummyTest.php',
'Psr\\Log\\Test\\LoggerInterfaceTest' => __DIR__ . '/..' . '/psr/log/Psr/Log/Test/LoggerInterfaceTest.php',
'Psr\\Log\\Test\\TestLogger' => __DIR__ . '/..' . '/psr/log/Psr/Log/Test/TestLogger.php',
'fivefilters\\Readability\\Configuration' => __DIR__ . '/..' . '/fivefilters/readability.php/src/Configuration.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMAttr' => __DIR__ . '/..' . '/fivefilters/readability.php/src/Nodes/DOM/DOMAttr.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMCdataSection' => __DIR__ . '/..' . '/fivefilters/readability.php/src/Nodes/DOM/DOMCdataSection.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMCharacterData' => __DIR__ . '/..' . '/fivefilters/readability.php/src/Nodes/DOM/DOMCharacterData.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMComment' => __DIR__ . '/..' . '/fivefilters/readability.php/src/Nodes/DOM/DOMComment.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMDocument' => __DIR__ . '/..' . '/fivefilters/readability.php/src/Nodes/DOM/DOMDocument.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMDocumentFragment' => __DIR__ . '/..' . '/fivefilters/readability.php/src/Nodes/DOM/DOMDocumentFragment.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMDocumentType' => __DIR__ . '/..' . '/fivefilters/readability.php/src/Nodes/DOM/DOMDocumentType.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMElement' => __DIR__ . '/..' . '/fivefilters/readability.php/src/Nodes/DOM/DOMElement.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMEntity' => __DIR__ . '/..' . '/fivefilters/readability.php/src/Nodes/DOM/DOMEntity.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMEntityReference' => __DIR__ . '/..' . '/fivefilters/readability.php/src/Nodes/DOM/DOMEntityReference.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMNode' => __DIR__ . '/..' . '/fivefilters/readability.php/src/Nodes/DOM/DOMNode.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMNodeList' => __DIR__ . '/..' . '/fivefilters/readability.php/src/Nodes/DOM/DOMNodeList.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMNotation' => __DIR__ . '/..' . '/fivefilters/readability.php/src/Nodes/DOM/DOMNotation.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMProcessingInstruction' => __DIR__ . '/..' . '/fivefilters/readability.php/src/Nodes/DOM/DOMProcessingInstruction.php',
'fivefilters\\Readability\\Nodes\\DOM\\DOMText' => __DIR__ . '/..' . '/fivefilters/readability.php/src/Nodes/DOM/DOMText.php',
'fivefilters\\Readability\\Nodes\\NodeTrait' => __DIR__ . '/..' . '/fivefilters/readability.php/src/Nodes/NodeTrait.php',
'fivefilters\\Readability\\Nodes\\NodeUtility' => __DIR__ . '/..' . '/fivefilters/readability.php/src/Nodes/NodeUtility.php',
'fivefilters\\Readability\\ParseException' => __DIR__ . '/..' . '/fivefilters/readability.php/src/ParseException.php',
'fivefilters\\Readability\\Readability' => __DIR__ . '/..' . '/fivefilters/readability.php/src/Readability.php',
);
public static function getInitializer(ClassLoader $loader)
{
return \Closure::bind(function () use ($loader) {
$loader->prefixLengthsPsr4 = ComposerStaticInitb44cc79a0eaef9cd9c2f2ac697cbe9c0::$prefixLengthsPsr4;
$loader->prefixDirsPsr4 = ComposerStaticInitb44cc79a0eaef9cd9c2f2ac697cbe9c0::$prefixDirsPsr4;
$loader->classMap = ComposerStaticInitb44cc79a0eaef9cd9c2f2ac697cbe9c0::$classMap;
}, null, ClassLoader::class);
}
}

208
vendor/composer/installed.json vendored Normal file
View File

@@ -0,0 +1,208 @@
{
"packages": [
{
"name": "fivefilters/readability.php",
"version": "dev-main",
"version_normalized": "dev-main",
"source": {
"type": "git",
"url": "https://github.com/tt-rss/tt-rss-readability-php.git",
"reference": "3bc7e81ae758642292ce20df08c56cd42cefd9d8"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/tt-rss/tt-rss-readability-php/zipball/3bc7e81ae758642292ce20df08c56cd42cefd9d8",
"reference": "3bc7e81ae758642292ce20df08c56cd42cefd9d8",
"shasum": ""
},
"require": {
"ext-dom": "*",
"ext-mbstring": "*",
"ext-xml": "*",
"masterminds/html5": "dev-main",
"php": ">=7.3.0",
"psr/log": "^1.0"
},
"require-dev": {
"monolog/monolog": "^2.3",
"phpunit/phpunit": "^9"
},
"suggest": {
"monolog/monolog": "Allow logging debug information"
},
"time": "2026-08-04T01:49:31+00:00",
"default-branch": true,
"type": "library",
"installation-source": "source",
"autoload": {
"psr-4": {
"fivefilters\\Readability\\": "src/"
}
},
"autoload-dev": {
"psr-4": {
"fivefilters\\Readability\\Test\\": "test"
}
},
"license": [
"Apache-2.0"
],
"authors": [
{
"name": "Andres Rey",
"email": "andreskrey@gmail.com",
"role": "Original Developer"
},
{
"name": "Keyvan Minoukadeh",
"email": "keyvan@fivefilters.org",
"homepage": "https://www.fivefilters.org",
"role": "Developer/Maintainer"
}
],
"description": "A PHP port of Readability.js",
"homepage": "https://github.com/fivefilters/readability.php",
"keywords": [
"html",
"readability"
],
"support": {
"source": "https://github.com/tt-rss/tt-rss-readability-php/tree/main",
"issues": "https://github.com/tt-rss/tt-rss-readability-php/issues"
},
"install-path": "../fivefilters/readability.php"
},
{
"name": "masterminds/html5",
"version": "dev-main",
"version_normalized": "dev-main",
"source": {
"type": "git",
"url": "https://github.com/tt-rss/tt-rss-html5-php.git",
"reference": "d2c79ada2a87bb7eaafe1a39e4e3bb37853099aa"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/tt-rss/tt-rss-html5-php/zipball/d2c79ada2a87bb7eaafe1a39e4e3bb37853099aa",
"reference": "d2c79ada2a87bb7eaafe1a39e4e3bb37853099aa",
"shasum": ""
},
"require": {
"ext-ctype": "*",
"ext-dom": "*",
"ext-libxml": "*",
"php": ">=5.3.0"
},
"require-dev": {
"phpunit/phpunit": "^4.8.35 || ^5.7.21 || ^6 || ^7"
},
"time": "2022-12-11T19:41:09+00:00",
"default-branch": true,
"type": "library",
"extra": {
"branch-alias": {
"dev-master": "2.7-dev"
}
},
"installation-source": "source",
"autoload": {
"psr-4": {
"Masterminds\\": "src"
}
},
"autoload-dev": {
"psr-4": {
"Masterminds\\HTML5\\Tests\\": "test/HTML5"
}
},
"license": [
"MIT"
],
"authors": [
{
"name": "Matt Butcher",
"email": "technosophos@gmail.com"
},
{
"name": "Matt Farina",
"email": "matt@mattfarina.com"
},
{
"name": "Asmir Mustafic",
"email": "goetas@gmail.com"
}
],
"description": "An HTML5 parser and serializer.",
"homepage": "http://masterminds.github.io/html5-php",
"keywords": [
"dom",
"html",
"html5",
"parser",
"querypath",
"serializer",
"xml"
],
"support": {
"source": "https://github.com/tt-rss/tt-rss-html5-php/tree/main",
"issues": "https://github.com/tt-rss/tt-rss-html5-php/issues"
},
"install-path": "../masterminds/html5"
},
{
"name": "psr/log",
"version": "1.1.4",
"version_normalized": "1.1.4.0",
"source": {
"type": "git",
"url": "https://github.com/php-fig/log.git",
"reference": "d49695b909c3b7628b6289db5479a1c204601f11"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/php-fig/log/zipball/d49695b909c3b7628b6289db5479a1c204601f11",
"reference": "d49695b909c3b7628b6289db5479a1c204601f11",
"shasum": ""
},
"require": {
"php": ">=5.3.0"
},
"time": "2021-05-03T11:20:27+00:00",
"type": "library",
"extra": {
"branch-alias": {
"dev-master": "1.1.x-dev"
}
},
"installation-source": "dist",
"autoload": {
"psr-4": {
"Psr\\Log\\": "Psr/Log/"
}
},
"notification-url": "https://packagist.org/downloads/",
"license": [
"MIT"
],
"authors": [
{
"name": "PHP-FIG",
"homepage": "https://www.php-fig.org/"
}
],
"description": "Common interface for logging libraries",
"homepage": "https://github.com/php-fig/log",
"keywords": [
"log",
"psr",
"psr-3"
],
"support": {
"source": "https://github.com/php-fig/log/tree/1.1.4"
},
"install-path": "../psr/log"
}
],
"dev": true,
"dev-package-names": []
}

54
vendor/composer/installed.php vendored Normal file
View File

@@ -0,0 +1,54 @@
<?php return array(
'root' => array(
'name' => '__root__',
'pretty_version' => 'dev-main',
'version' => 'dev-main',
'reference' => 'fce528aa69c2a7193fb7eb3a3cd9dd17885d6ab6',
'type' => 'library',
'install_path' => __DIR__ . '/../../',
'aliases' => array(),
'dev' => true,
),
'versions' => array(
'__root__' => array(
'pretty_version' => 'dev-main',
'version' => 'dev-main',
'reference' => 'fce528aa69c2a7193fb7eb3a3cd9dd17885d6ab6',
'type' => 'library',
'install_path' => __DIR__ . '/../../',
'aliases' => array(),
'dev_requirement' => false,
),
'fivefilters/readability.php' => array(
'pretty_version' => 'dev-main',
'version' => 'dev-main',
'reference' => '3bc7e81ae758642292ce20df08c56cd42cefd9d8',
'type' => 'library',
'install_path' => __DIR__ . '/../fivefilters/readability.php',
'aliases' => array(
0 => '9999999-dev',
),
'dev_requirement' => false,
),
'masterminds/html5' => array(
'pretty_version' => 'dev-main',
'version' => 'dev-main',
'reference' => 'd2c79ada2a87bb7eaafe1a39e4e3bb37853099aa',
'type' => 'library',
'install_path' => __DIR__ . '/../masterminds/html5',
'aliases' => array(
0 => '9999999-dev',
),
'dev_requirement' => false,
),
'psr/log' => array(
'pretty_version' => '1.1.4',
'version' => '1.1.4.0',
'reference' => 'd49695b909c3b7628b6289db5479a1c204601f11',
'type' => 'library',
'install_path' => __DIR__ . '/../psr/log',
'aliases' => array(),
'dev_requirement' => false,
),
),
);

25
vendor/composer/platform_check.php vendored Normal file
View File

@@ -0,0 +1,25 @@
<?php
// platform_check.php @generated by Composer
$issues = array();
if (!(PHP_VERSION_ID >= 70300)) {
$issues[] = 'Your Composer dependencies require a PHP version ">= 7.3.0". You are running ' . PHP_VERSION . '.';
}
if ($issues) {
if (!headers_sent()) {
header('HTTP/1.1 500 Internal Server Error');
}
if (!ini_get('display_errors')) {
if (PHP_SAPI === 'cli' || PHP_SAPI === 'phpdbg') {
fwrite(STDERR, 'Composer detected issues in your platform:' . PHP_EOL.PHP_EOL . implode(PHP_EOL, $issues) . PHP_EOL.PHP_EOL);
} elseif (!headers_sent()) {
echo 'Composer detected issues in your platform:' . PHP_EOL.PHP_EOL . str_replace('You are running '.PHP_VERSION.'.', '', implode(PHP_EOL, $issues)) . PHP_EOL.PHP_EOL;
}
}
throw new \RuntimeException(
'Composer detected issues in your platform: ' . implode(' ', $issues)
);
}

2
vendor/fivefilters/readability.php/.gitattributes vendored Normal file
View File

@@ -0,0 +1,2 @@
test/* linguist-language=PHP
* text=auto eol=lf

42
vendor/fivefilters/readability.php/.github/workflows/main.yml vendored Normal file
View File

@@ -0,0 +1,42 @@
# This is a basic workflow to help you get started with Actions
name: CI
# Controls when the workflow will run
on:
# Triggers the workflow on push or pull request events but only for the master branch
push:
branches: [ master ]
pull_request:
branches: [ master ]
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "build"
build:
# The type of runner that the job will run on
runs-on: ubuntu-latest
strategy:
matrix:
php: ['7.3', '7.4', '8']
libxml: ['2.9.4', '2.9.5', '2.9.10', '2.9.12']
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout@v2
# Runs a single command using the runners shell
#- name: Run a one-line script
# run: echo Hello, world!
# Runs a set of commands using the runners shell
- name: Run a multi-line script
run: |
composer install
docker build --build-arg PHP_VERSION=${{matrix.php}} --build-arg LIBXML_VERSION=${{matrix.libxml}} -t gh-action - < ./docker/php/Dockerfile
docker run --volume $PWD:/app --workdir="/app" --env XDEBUG_MODE=coverage gh-action php ./vendor/bin/phpunit --coverage-clover /app/test/clover.xml

5
vendor/fivefilters/readability.php/.gitignore vendored Normal file
View File

@@ -0,0 +1,5 @@
.idea/
vendor
composer.lock
/test.*
/test/changed/

14
vendor/fivefilters/readability.php/AUTHORS.md vendored Normal file
View File

@@ -0,0 +1,14 @@
# Authors
Readability.php developed by **Andres Rey**.
Based on Arc90's readability.js (1.7.1) script available at: http://code.google.com/p/arc90labs-readability.
Copyright (c) 2010 Arc90 Inc
The AUTHORS/Contributors are (and/or have been):
* Andres Rey
* Sergiy Lavryk
* Pedro Amorim
* Malu Decks
* Keyvan Minoukadeh

145
vendor/fivefilters/readability.php/CHANGELOG.md vendored Normal file
View File

@@ -0,0 +1,145 @@
# Change Log
All notable changes to this project will be documented in this file.
## Unreleased
## [v2.1.0](https://github.com/andreskrey/readability.php/releases/tag/v2.1.0)
- Avoid overwriting extracted metadata with similarly named keys (like `og:image` and `og:image:width`)
- Imported new `getSiteName()` feature from JS version as of [21 Dec 2018](https://github.com/mozilla/readability/pull/504)
- Added getFirstElementChild function to NodeTrait + test case (Issue #83)
- Reworked the test suit to use TestPage objects and give more hints about what failed
- Removed getWordThreshold and setWordThreshold configuration functions
- Added NodeUtility::filterTextNodes and deprecated NodeTrait getChildren()
- Added new DOMNodeList fake class that mimics the original DOMNodeList class but allows to add new nodes to the list
- Added new Dockerfiles that pulls different versions of PHP and libxml. Now we are supporting 4 versions of PHP and 6 versions of libxml!
## [v2.0.1](https://github.com/andreskrey/readability.php/releases/tag/v2.0.1)
- Fixed small issue that prevented the main image from showing up in the results
## [v2.0.0](https://github.com/andreskrey/readability.php/releases/tag/v2.0.0)
- [BREAKING CHANGE] Bumped the minimum supported version of PHP to 7.0
- Clean `<aside>` tags during `prepArticle()`.
- Merged PR #58: Fix notice non-object on $parentOfTopCandidate for tumblr.com
- Fixed issue #63: Division by zero
- Housekeeping:
- Removed $parseSuccessful flag that wasn't needed anymore
- Rename wordThreshold to charThreshold and throw deprecation notices. WordThreshold will be removed in version 3.0.
- Added "-ad-" as unlikely candidate
- Added Docker containers with PHP 7.0, 7.1, and 7.2 and makefile to trigger the tests.
- Imported new code from the JS version as of [19 Nov 2018](https://github.com/mozilla/readability/commit/876c81f710711ba2afb36dd83889d4c5b4fc2743), which includes the following changes:
- Move phrasing contents [into paragraphs](https://github.com/mozilla/readability/commit/9f2c5cb42ee9635f091178271d66888cbb47e5dc)
- Improved the title detection
- Remove [single cell tables](https://github.com/mozilla/readability/commit/ea4165721f9105d8f1e53cfecdcfdafceaf3e4bf)
- Improved the detection of video related elements
- New test cases
- Various minor fixes
## [v1.2.0](https://github.com/andreskrey/readability.php/releases/tag/v1.2.0)
- Merged PR#49 (Missing object when calling `->getContent()`)
- Imported all changes from Readability.js as of 2 March 2018 ([8525c6a](https://github.com/mozilla/readability/commit/8525c6af36d3badbe27c4672a6f2dd99ddb4097f)):
- Check for `<base>` elements before converting URLs to absolute.
- Clean `<link>` tags on `prepArticle()`
- Attempt to return at least some text if all the algorithm runs fail (Check PR [#423](https://github.com/mozilla/readability/pull/423) on JS version)
- Add new test cases for the previous changes
- And all other changes reflected [in this diff](https://github.com/mozilla/readability/compare/c3ff1a2d2c94c1db257b2c9aa88a4b8fbeb221c5...8525c6af36d3badbe27c4672a6f2dd99ddb4097f)
## [v1.1.1](https://github.com/andreskrey/readability.php/releases/tag/v1.1.1)
- Switched from assertEquals to assertSame on unit testing to avoid weak comparisons.
- Added a safe check to avoid sending the DOMDocument as a node when scanning for node ancestors.
- Fix issue #45: Small mistake in documentation
- Fix issue #46: Added `data-src` as a image source path
- Fixed bug when extracting all the image of the article (Was extracting images from the original DOM instead of the parsed one)
- Added the `->getDOMDocument()` getter to retrieve the fully parsed DOMDocument
- Merged PR #48 that allows passing an array as configuration (@topotru)
## [v1.1.0](https://github.com/andreskrey/readability.php/releases/tag/v1.1.0)
- Added 'data-orig' as an URL source for images
- Removed 'modal' as a negative property from classes
- Added option to inject a logger
- Removed all references to the `data-readability` tags that don't apply anymore to the new structure
- Merged PR #38 (Missing DOMEntityReference)
## [v1.0.0](https://github.com/andreskrey/readability.php/releases/tag/v1.0.0)
- Node encapsulation is gone. Pre v1 all nodes where encapsulated in a Readability class, which created lots of trouble with dependencies, responsibilities, and properties. Now all the encapsulation is gone: all the DOMNodes inside the Readability class are extensions of the original DOM classes, which allows the system to take advantage of the functions and properties of DOMDocument.
- HTMLParser is gone, Readability is the new main class. Switched things a bit for this release. Pre v1 you had to create an HTMLParser class to parse the HTML. Now you have to create a Readability class, feed it the text, and check the result.
- No more dumb arrays as a result. If you want to get the title, content, images, or anything else you'll have to use the getters of the Readability class.
- Environment class is gone. Now you have to create a configuration class and use setters to set your configuration options.
- Exceptions. Make sure you wrap your Readability class in a try catch block, because if it fails to parse your HTML, it will throw a `ParseException`.
- Minimum PHP version bumped to 5.6.
## [v0.3.1](https://github.com/andreskrey/readability.php/releases/tag/v0.3.1)
- Trim titles when detecting hierarchical separators to avoid false negatives on strings with spaces.
- Fix issue when converting divs to p nodes and never rating them (issue #29)
- Fix "Unsupported operand types" (PR #31)
- Fix division by zero when no title was found (issue #32)
- New function to retrieve all images at once (PR #30)
- Get the title from the `<title>` tag before searching on the `<meta>` tags
## [v0.3.0](https://github.com/andreskrey/readability.php/releases/tag/v0.3.0)
- Merged PR #24. Fixes notice when trying to extract `og:image`
- Up to date to commit [eb221c5](https://github.com/mozilla/readability/commit/c3ff1a2d2c94c1db257b2c9aa88a4b8fbeb221c5) (2017-10-16), which includes the following changes:
- New tags added to the unlikelyCandidates regex
- Detection and removal of hierarchical separators in titles
- Added more tags to clean after parsing the article (`button`, `textarea`, `select`, etc.)
- New way to detect empty nodes (including a edge case where a node with a `&nsbp;` was detected as a node with content)
- Better approach to find a top candidate (specially when a top candidate is the only child of a parent node, which allows a more accurate joining of sibling elements)
- Detect text direction (`ltr` or `rtl`)
- Detect and mark data tables to avoid removing them during final clean up
- Major fixes when scanning and deleting nodes (no need to traverse backwards anymore)
- Node cleaning via regex matches
- Clean table attributes during final clean up.
- Added license
Next release after this one will be v1 and will be a major refactor around Readability and HTMLParser methods and responsibilities.
## [v0.2.2](https://github.com/andreskrey/readability.php/releases/tag/v0.2.2)
- Added a safecheck for really nasty HTML
- Added summonCthulhu option, to remove all script tags via regex
## [v0.2.1](https://github.com/andreskrey/readability.php/releases/tag/v0.2.1)
- Added `normalizeEntities` flag to convert UTF-8 characters to its HTML Entity equivalent. Fixes bugs on htmls with mixed encoding.
- Added more information to the readme.md file
- New way to create a backup DOM: not creating a backup. In the previous version, the system cloned the $this->dom object to keep it as a backup in order to restart the algorithm with other flags, if needed. This seemed to work until I realized that *sometimes* the backup changes even if we are not touching it. Seems that the `dom` and `backupdom` objects are linked and *some* changes on the dom object reach the bakcupdom object. The new approach consists in deleting the backupdom object and recreating from scratch the dom object. Of course this has a performance impact, but seems to be quite low.
## [v0.2.0](https://github.com/andreskrey/readability.php/releases/tag/v0.2.0)
100% complete port of Readability.js!
- Every test unit passes
- Readability.php produces the same exact output as Readability.js
- I'm happy :)
### Fixed
- Lots of bugs
- Merged PR by DavidFricker to avoid exceptions while grabbing the document content
### Added
- substituteEntities flag, to avoid replacing especial characters with HTML entities. There's nothing we can do about `&nbsp;`, that entity is replaced by libxml and there's no way to disable it.
- Named data sets so it's easier to detect which test case is failing.
### Removed
- Couple of test cases that involved broken JS. There's nothing we can do about JS spilling onto the text.
## [0.0.3-alpha](https://github.com/andreskrey/readability.php/releases/tag/v0.0.3v-alpha)
We are getting closer to be a 100% complete port of Readability.js!
- Added prepArticle to remove junk after selecting the top candidates.
- Added a function to restore score after selecting top candidates. This basically works by scanning the data-readability tag and restoring the score to the contentScore variable. This is an horrible hack and should be removed once we ditch the Element interface of html-to-markdown and start extending the DOMDocument object.
- Switched all strlen functions to mb_strlen
- Fixed lots of bugs and pretty sure that introduced a bunch of new ones.
## [0.0.2-alpha](https://github.com/andreskrey/readability.php/releases/tag/v0.0.2-alpha)
- Last version I'm using master as the main development branch. All unreleased changes and main development will happen in the develop branch.
## [0.0.1-alpha](https://github.com/andreskrey/readability.php/releases/tag/v0.0.1-alpha)
- Initial release

31
vendor/fivefilters/readability.php/CONTRIBUTING.md vendored Normal file
View File

@@ -0,0 +1,31 @@
# Contributing
Contributions are **welcome** and will be fully **credited**.
When it comes to the core article-extraction functionality, please contribute to [Mozilla's Readability](https://github.com/mozilla/readability/) repository, as we're trying to mirror that here.
For anything else, we accept contributions via Pull Requests on [Github](https://github.com/fivefilters/readability.php/).
## Pull Requests
- **Document any change in behaviour** - Make sure the `README.md` and any other relevant documentation are kept up-to-date.
- **Add tests!** - Your patch won't be accepted if it doesn't have tests.
- **Create feature branches** - Don't ask us to pull from your master branch.
- **One pull request per feature** - If you want to do more than one thing, send multiple pull requests.
- **Send coherent history** - Make sure each individual commit in your pull request is meaningful. If you had to make multiple intermediate commits while developing, please [squash them](http://www.git-scm.com/book/en/v2/Git-Tools-Rewriting-History#Changing-Multiple-Commit-Messages) before submitting.
- **Don't forget to add yourself to AUTHORS.md** - If you want to be credited, make sure you add your information (whatever you want to include) in `AUTHORS.md`.
## Running Tests
``` bash
$ make test-all #requires docker and docker-compose
```
**Happy coding**!

201
vendor/fivefilters/readability.php/LICENSE vendored Normal file
View File

@@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

27
vendor/fivefilters/readability.php/Makefile vendored Normal file
View File

@@ -0,0 +1,27 @@
.PHONY: test-all
test-all: start test-7.3 test-7.4 test-8 stop
test-7.3:
docker-compose exec php-7.3-libxml-2.9.10 php /app/vendor/phpunit/phpunit/phpunit --configuration /app/phpunit.xml
test-7.4:
docker-compose exec php-7.4-libxml-2.9.10 php /app/vendor/phpunit/phpunit/phpunit --configuration /app/phpunit.xml
test-8:
docker-compose exec php-8-libxml-2.9.10 php /app/vendor/phpunit/phpunit/phpunit --configuration /app/phpunit.xml
start:
docker-compose up -d php-7.3-libxml-2.9.10 php-7.4-libxml-2.9.10 php-8-libxml-2.9.10
stop:
docker-compose stop
test-all-versions:
for php_version in 7.3 7.4 8; do \
for libxml_version in 2.9.4 2.9.5 2.9.10 2.9.12; do \
docker-compose up -d php-$$php_version-libxml-$$libxml_version; \
docker-compose exec php-$$php_version-libxml-$$libxml_version php /app/vendor/phpunit/phpunit/phpunit --configuration /app/phpunit.xml; \
done \
done
docker-compose stop

251
vendor/fivefilters/readability.php/README.md vendored Normal file
View File

@@ -0,0 +1,251 @@
# Readability.php
## News (August 2021)
Andres Rey, the [original developer](https://github.com/andreskrey/readability.php) of Readability.php has kindly let us take over maintenance and development of the project.
Please bear with us while we catch up with [Readability.js](https://github.com/mozilla/readability) changes. There'll be a new release (3.0.0) when we're ready.
For the changes we've made so far in this repository, please see our [blog post](https://www.fivefilters.org/2021/readability/).
## About
[![Latest Stable Version](https://poser.pugx.org/fivefilters/readability.php/v/stable)](https://packagist.org/packages/fivefilters/readability.php) [![Tests](https://github.com/fivefilters/readability.php/actions/workflows/main.yml/badge.svg?branch=master)](https://github.com/fivefilters/readability.php/actions/workflows/main.yml)
PHP port of *Mozilla's* **[Readability.js](https://github.com/mozilla/readability)**. Parses html text (usually news and other articles) and returns **title**, **author**, **main image** and **text content** without nav bars, ads, footers, or anything that isn't the main body of the text. Analyzes each node, gives them a score, and determines what's relevant and what can be discarded.
![Screenshot](https://raw.githubusercontent.com/fivefilters/readability.php/assets/screenshot.png)
The project aim is to be a 1 to 1 port of Mozilla's version and to follow closely all changes introduced there, but there are some major differences on the structure. Most of the code is a 1:1 copy –even the comments were imported– but some functions and structures were adapted to suit better the PHP language.
**Original Developer**: Andres Rey
**Developer/Maintainer**: FiveFilters.org
## Code porting
Master branch - Up to date on 26 August 2021, with the exception of a [piece of code](https://github.com/fivefilters/readability.php/commit/1c662465bded2ab3acf3b975a1315c8c45f0bf73#diff-b9b31807b1a39caec18ddc293e9c52931ba8b55191c61e6b77a623d699a599ffR1899) which doesn't produce the same results in PHP for us compard to the JS version. Perhaps there's an error, or some difference in the underlying code that affects this. If you know what's wrong, please feel free to drop us a note or submit a pull request. :)
Version 2.1.0 - Up to date with Readability.js up to [19 Nov 2018](https://github.com/mozilla/readability/commit/876c81f710711ba2afb36dd83889d4c5b4fc2743).
## Requirements
PHP 7.3+, ext-dom, ext-xml, and ext-mbstring. To install these dependencies (in the rare case your system does not have them already), you could try something like this in *nix like environments:
`$ sudo apt-get install php7.4-xml php7.4-mbstring`
## How to use it
First you have to require the library using composer:
`composer require fivefilters/readability.php`
Then, create a Readability class and pass a Configuration class, feed the `parse()` function with your HTML and echo the variable:
```php
use fivefilters\Readability\Readability;
use fivefilters\Readability\Configuration;
use fivefilters\Readability\ParseException;
$readability = new Readability(new Configuration());
$html = file_get_contents('http://your.favorite.newspaper/article.html');
try {
$readability->parse($html);
echo $readability;
} catch (ParseException $e) {
echo sprintf('Error processing text: %s', $e->getMessage());
}
```
Your script will output the parsed text or inform about any errors. You should always wrap the `->parse` call in a try/catch block because if the HTML cannot be parsed correctly, a `ParseException` will be thrown.
If you want to have a finer control on the output, just call the properties one by one, wrapping it with your own HTML.
```php
<h1><?= $readability->getTitle(); ?></h1>
<h2>By <?= $readability->getAuthor(); ?></h2>
<div class="content"><?= $readability->getContent(); ?></div>
```
Here's a list of the available properties:
- Article title: `->getTitle();`
- Article content: `->getContent();`
- Excerpt: `->getExcerpt();`
- Main image: `->getImage();`
- All images: `->getImages();`
- Author: `->getAuthor();`
- Text direction (ltr or rtl): `->getDirection();`
If you need to tweak the final HTML you can get the DOMDocument of the result by calling `->getDOMDocument()`.
## Options
You can change the behaviour of Readability via the Configuration object. For example, if you want to fix relative URLs and declare the original URL, you could set up the configuration like this:
```php
$configuration = new Configuration();
$configuration
->setFixRelativeURLs(true)
->setOriginalURL('http://my.newspaper.url/article/something-interesting-to-read.html');
```
Also you can pass an array of configuration parameters to the constructor:
```php
$configuration = new Configuration([
'fixRelativeURLs' => true,
'originalURL' => 'http://my.newspaper.url/article/something-interesting-to-read.html',
// other parameters ... listing below
]);
```
Then you pass this Configuration object to Readability. The following options are available. Remember to prepend `set` when calling them using native setters.
- **MaxTopCandidates**: default value `5`, max amount of top level candidates.
- **CharThreshold**: default value `500`, minimum amount of characters to consider that the article was parsed successful.
- **ArticleByLine**: default value `false`, search for the article byline and remove it from the text. It will be moved to the article metadata.
- **StripUnlikelyCandidates**: default value `true`, remove nodes that are unlikely to have relevant information. Useful for debugging or parsing complex or non-standard articles.
- **CleanConditionally**: default value `true`, remove certain nodes after parsing to return a cleaner result.
- **WeightClasses**: default value `true`, weight classes during the rating phase.
- **FixRelativeURLs**: default value `false`, convert relative URLs to absolute. Like `/test` to `http://host/test`.
- **SubstituteEntities**: default value `false`, disables the `substituteEntities` flag of libxml. Will avoid substituting HTML entities. Like `&aacute;` to á.
- **NormalizeEntities**: default value `false`, converts UTF-8 characters to its HTML Entity equivalent. Useful to parse HTML with mixed encoding.
- **OriginalURL**: default value `http://fakehost`, original URL from the article used to fix relative URLs.
- **KeepClasses**: default value `false`, which removes all `class="..."` attribute values from HTML elements.
- **Parser**: default value `html5`, which uses HTML5-PHP for parsing. Set to `libxml` to use that instead (not recommended for modern HTML documents).
- **SummonCthulhu**: default value `false`, remove all `<script>` nodes via regex. This is not ideal as it might break things, but if you've set the parser to libxml (see above), it might be the only solution to [libxml problems with unescaped javascript](https://github.com/fivefilters/readability.php#known-libxml-parsing-issues).
- **ExtraIgnoredElements**: default value `[]`, additional DOM elements that should be ignored. `noscript` and `script` are always ignored.
### Debug log
Logging is optional and you will have to inject your own logger to save all the debugging messages. To do so, use a logger that implements the [PSR-3 logging interface](https://github.com/php-fig/log) and pass it to the configuration object. For example:
```php
// Using monolog
$log = new Logger('Readability');
$log->pushHandler(new StreamHandler('path/to/my/log.txt'));
$configuration->setLogger($log);
```
In the log you will find information about the parsed nodes, why they were removed, and why they were considered relevant to the final article.
## Limitations
Of course the main limitation is PHP. Websites that load the content through lazy loading, AJAX, or any type of javascript fueled call will be ignored (actually, *not ran*) and the resulting text will be incorrect, compared to the readability.js results. All the articles you want to parse with readability.php need to be complete and all the content should be in the HTML already.
## Known libxml parsing issues
Readability.php as of version 3.0.0 uses a HTML5 parser. Earlier versions used libxml. The issues below apply to libxml parsing, so if you're using an earlier version of Readability.php (pre 3.0.0), or if you've set the parser to libxml in the configuration, read on...
### Javascript spilling into the text body
DOMDocument has some issues while parsing javascript with unescaped HTML on strings. Consider the following code:
```html
<div> <!-- Offending div without closing tag -->
<script type="text/javascript">
var test = '</div>';
// I should not appear on the result
</script>
```
If you would like to remove the scripts of the HTML (like readability does), you would expect ending up with just one div and one comment on the final HTML. The problem is that libxml takes that closing div tag inside the javascript string as a HTML tag, effectively closing the unclosed tag and leaving the rest of the javascript as a string within a P tag. If you save that node, the final HTML will end up like this:
```html
<div> <!-- Offending div without closing tag -->
<p>';
// I should not appear on the result
</p></div>
```
This is a libxml issue and not a Readability.php bug.
There's a workaround for this: using the `summonCthulhu` option. This will remove all script tags **via regex**, which is not ideal because you may end up summoning [the lord of darkness](https://stackoverflow.com/a/1732454).
### &nbsp entities disappearing
`&nbsp` entities are converted to spaces automatically by libxml and there's no way to disable it.
### Self closing tags rendering as fully expanded tags
Self closing tags like `<br />` get automatically expanded to `<br></br`. No way to disable it in libxml.
## Dependencies
Readability.php uses
* [HTML5-PHP](https://github.com/Masterminds/html5-php) to parse and serialise HTML.
* [PSR Log](https://github.com/php-fig/log) interface to define the allowed type of loggers.
* [Monolog](https://github.com/Seldaek/monolog) is only required on development installations. (`--dev` option during `composer install`).
## To-do
- Keep up with Readability.js changes
- Add a small template engine for the __toString() method, instead of using a hardcoded one.
- Replace all the `iterator_to_array` calls with a custom PHP generator that keeps track of the removed or altered nodes.
## How it works
Readability parses all the text with DOMDocument, scans the text nodes and gives the a score, based on the amount of words, links and type of element. Then it selects the highest scoring element and creates a new DOMDocument with all its siblings. Each sibling is scored to discard useless elements, like nav bars, empty nodes, etc.
## Security
If you're going to use Readability with untrusted input (whether in HTML or DOM form), we **strongly** recommend you use a sanitizer library like [HTML Purifier](https://github.com/ezyang/htmlpurifier) to avoid script injection when you use
the output of Readability. We would also recommend using [CSP](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) to add further defense-in-depth
restrictions to what you allow the resulting content to do. The Firefox integration of
reader mode uses both of these techniques itself. Sanitizing unsafe content out of the input is explicitly not something we aim to do as part of Readability itself - there are other good sanitizer libraries out there, use them!
## Testing
Any version of PHP from 7.3 and above installed locally should be enough to develop new features and add new test cases. If you want to be 100% sure that your change doesn't create any issues with other versions of PHP, you can use the provided Docker containers to test currently in 7.3, 7.4, and 8.0.
You'll need Docker and Docker Compose for this. To run all the tests in the three PHP versions above, just type the following command:
```bash
make test-all
```
This will start all the containers and run all the tests on every supported version of PHP. If you want to test against a specific version, you can use `make test-7.3`, `make test-7.4`, or `make test-8`.
### Different versions of libxml
If you want to test against supported versions of PHP *AND* multiple versions of libxml, run `test-all-versions`. This will test against PHP versions 7.3 to 8 and libxml versions 2.9.4, 2.9.5, 2.9.10, and 2.9.12. Normally you won't need to do this unless you think you've found a bug on an specific version of libxml.
### Updating the expected tests
If you've made an improvement to the code, you'll probably want to examine the Readability.php output for the test cases here. To do that, run the following command first from the root of the project folder:
docker-compose up -d php-7.4-libxml-2.9.10
You should now have a docker image running with the project root folder mapped to /app/ on your Docker instance (see `docker-compose.yml`). Any changes to these files will be accessible from the Docker instance from now on.
Next, create a folder in tests/ called /changed, then run the following command to run the test suite:
docker-compose exec -e output-changes=1 -e output-diff=1 php-7.4-libxml-2.9.10 php /app/vendor/phpunit/phpunit/phpunit --configuration /app/phpunit.xml
The two environment variables (`output-changes=1` and `output-diff=1`) will result in new output for any failing test (along with a diff of changes) being written to the changed/ folder.
If you're happy the changes are okay, set `output-diff=0` and the diff files will no longer be written, making it easier to copy the new expected output files over to their corresponding locations in test-pages\.
## License
Based on Arc90's readability.js (1.7.1) script available at: http://code.google.com/p/arc90labs-readability
Copyright (c) 2010 Arc90 Inc
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

51
vendor/fivefilters/readability.php/composer.json vendored Normal file
View File

@@ -0,0 +1,51 @@
{
"name": "fivefilters/readability.php",
"type": "library",
"description": "A PHP port of Readability.js",
"keywords": ["readability", "html"],
"homepage": "https://github.com/fivefilters/readability.php",
"license": "Apache-2.0",
"repositories": [
{
"name": "masterminds/html5",
"type": "vcs",
"url": "https://github.com/tt-rss/tt-rss-html5-php.git"
}
],
"authors": [
{
"name": "Andres Rey",
"email": "andreskrey@gmail.com",
"role": "Original Developer"
},
{
"name": "Keyvan Minoukadeh",
"email": "keyvan@fivefilters.org",
"homepage": "https://www.fivefilters.org",
"role": "Developer/Maintainer"
}
],
"autoload": {
"psr-4": {
"fivefilters\\Readability\\": "src/"
}
},
"autoload-dev": {
"psr-4": {"fivefilters\\Readability\\Test\\": "test"}
},
"require": {
"php": ">=7.3.0",
"ext-dom": "*",
"ext-xml": "*",
"ext-mbstring": "*",
"psr/log": "^1.0",
"masterminds/html5": "dev-main"
},
"require-dev": {
"phpunit/phpunit": "^9",
"monolog/monolog": "^2.3"
},
"suggest": {
"monolog/monolog": "Allow logging debug information"
}
}

100
vendor/fivefilters/readability.php/docker-compose.yml vendored Normal file
View File

@@ -0,0 +1,100 @@
version: '3'
services:
php-7.3-libxml-2.9.4: &template
build:
context: ./docker/php
args:
LIBXML_VERSION: 2.9.4
PHP_VERSION: 7.3
volumes:
- ./:/app
tty: true
php-7.3-libxml-2.9.5:
<<: *template
build:
context: ./docker/php
args:
LIBXML_VERSION: 2.9.5
PHP_VERSION: 7.3
php-7.3-libxml-2.9.10:
<<: *template
build:
context: ./docker/php
args:
LIBXML_VERSION: 2.9.10
PHP_VERSION: 7.3
php-7.3-libxml-2.9.12:
<<: *template
build:
context: ./docker/php
args:
LIBXML_VERSION: 2.9.12
PHP_VERSION: 7.3
php-7.4-libxml-2.9.4:
<<: *template
build:
context: ./docker/php
args:
LIBXML_VERSION: 2.9.4
PHP_VERSION: 7.4
php-7.4-libxml-2.9.5:
<<: *template
build:
context: ./docker/php
args:
LIBXML_VERSION: 2.9.5
PHP_VERSION: 7.4
php-7.4-libxml-2.9.10:
<<: *template
build:
context: ./docker/php
args:
LIBXML_VERSION: 2.9.10
PHP_VERSION: 7.4
php-7.4-libxml-2.9.12:
<<: *template
build:
context: ./docker/php
args:
LIBXML_VERSION: 2.9.12
PHP_VERSION: 7.4
php-8-libxml-2.9.4:
<<: *template
build:
context: ./docker/php
args:
LIBXML_VERSION: 2.9.4
PHP_VERSION: 8
php-8-libxml-2.9.5:
<<: *template
build:
context: ./docker/php
args:
LIBXML_VERSION: 2.9.5
PHP_VERSION: 8
php-8-libxml-2.9.10:
<<: *template
build:
context: ./docker/php
args:
LIBXML_VERSION: 2.9.10
PHP_VERSION: 8
php-8-libxml-2.9.12:
<<: *template
build:
context: ./docker/php
args:
LIBXML_VERSION: 2.9.12
PHP_VERSION: 8

16
vendor/fivefilters/readability.php/docker/php/Dockerfile vendored Normal file
View File

@@ -0,0 +1,16 @@
# Load pre-built image of PHP (php-cli) and libxml.
# See https://hub.docker.com/r/fivefilters/php-libxml for supported versions
# Use build.Dockerfile to compile new versions of PHP/libxml
# For reference, default package versions for Ubuntu are:
# Ubuntu 18.04 - php 7.2, libxml 2.9.4
# Ubuntu 20.04 - php 7.4, libxml 2.9.10
# Ubuntu 20.10 - php 7.4, libxml 2.9.10
# Ubuntu 21.04 - php 7.4, libxml 2.9.10
# Ubuntu 21.10 - php 8.0, libxml 2.9.10
ARG PHP_VERSION
ARG LIBXML_VERSION
FROM fivefilters/php-libxml:php-${PHP_VERSION}-libxml-${LIBXML_VERSION}
RUN apt-get update

56
vendor/fivefilters/readability.php/docker/php/build.Dockerfile vendored Normal file
View File

@@ -0,0 +1,56 @@
# Use this file to build a Docker image using the versions of PHP and Libxml specified.
# We have pre-built images at https://hub.docker.com/r/fivefilters/php-libxml which are faster to load than building from this file.
# To build using this file, use the following command from the root project folder (replace version of PHP/Libxml with the ones you want to use):
# docker build --build-arg PHP_VERSION=7.4 --build-arg LIBXML_VERSION=2.9.12 -t php-libxml -f ./docker/php/Dockerfile .
# To upload the image to Docker Hub, the tag (-t) value should be something like org/repo:tag, e.g. for us, fivefilters/php-libxml:php-8-libxml-2.9.12
# The tag can be applied afterwards too, e.g. docker tag php-libxml org/repo:tag
ARG PHP_VERSION=8
FROM php:${PHP_VERSION}-cli
# Install sqlite and libonig-dev (required for building PHP 7.4)
RUN apt-get update && apt-get install -y libsqlite3-dev libonig-dev
# Install libsodium (package doesn't work for some reason)
RUN curl https://download.libsodium.org/libsodium/releases/LATEST.tar.gz -o /tmp/libsodium.tar.gz && \
cd /tmp && \
tar -xzf libsodium.tar.gz && \
cd libsodium-stable/ && \
./configure && \
make && make check && \
make install
# Install custom version of libxml2
RUN apt-get install -y automake libtool unzip libssl-dev
# Remove current version
RUN apt-get remove -y libxml2
# Download new version, configure and compile
ARG LIBXML_VERSION=2.9.12
RUN curl https://gitlab.gnome.org/GNOME/libxml2/-/archive/v$LIBXML_VERSION/libxml2-v$LIBXML_VERSION.zip -o /tmp/libxml.zip && \
cd /tmp && \
unzip libxml.zip && \
cd libxml2-v$LIBXML_VERSION && \
./autogen.sh --libdir=/usr/lib/x86_64-linux-gnu && \
make && \
make install
# Recompile PHP with the new libxml2 library
RUN docker-php-source extract && \
cd /usr/src/php && \
./configure \
--with-libxml \
--enable-mbstring \
--with-openssl \
--with-config-file-path=/usr/local/etc/php \
--with-config-file-scan-dir=/usr/local/etc/php/conf.d && \
make && make install && \
docker-php-source delete
RUN apt-get update
#RUN pecl install libsodium
# Check if there's a pinned version of Xdebug for compatibility reasons
ARG XDEBUG_VERSION
RUN pecl install xdebug$(if [ ! ${XDEBUG_VERSION} = '' ]; then echo -${XDEBUG_VERSION} ; fi) && docker-php-ext-enable xdebug
# Required by coveralls
RUN apt-get install git -y

16
vendor/fivefilters/readability.php/phpunit.xml vendored Normal file
View File

@@ -0,0 +1,16 @@
<?xml version="1.0" encoding="UTF-8"?>
<phpunit bootstrap="vendor/autoload.php"
colors="true"
stopOnFailure="false"
stopOnError="false">
<testsuites>
<testsuite name="Readability.php Test Suite">
<directory>./test/</directory>
</testsuite>
</testsuites>
<filter>
<whitelist>
<directory suffix=".php">src/</directory>
</whitelist>
</filter>
</phpunit>

448
vendor/fivefilters/readability.php/src/Configuration.php vendored Normal file
View File

@@ -0,0 +1,448 @@
<?php
namespace fivefilters\Readability;
use Psr\Log\LoggerAwareTrait;
use Psr\Log\LoggerInterface;
use Psr\Log\NullLogger;
/**
* Class Configuration.
*/
class Configuration
{
use LoggerAwareTrait;
/**
* @var int
*/
protected $maxTopCandidates = 5;
/**
* @var int
*/
protected $charThreshold = 500;
/**
* @var bool
*/
protected $articleByLine = false;
/**
* @var bool
*/
protected $stripUnlikelyCandidates = true;
/**
* @var bool
*/
protected $cleanConditionally = true;
/**
* @var bool
*/
protected $weightClasses = true;
/**
* @var bool
*/
protected $fixRelativeURLs = false;
/**
* @var bool
*/
protected $substituteEntities = false;
/**
* @var bool
*/
protected $normalizeEntities = false;
/**
* @var bool
*/
protected $summonCthulhu = false;
/**
* @var string
*/
protected $originalURL = 'http://fakehost';
/**
* @var string
*/
protected $parser = 'html5';
/**
* @var bool
*/
protected $keepClasses = false;
/**
* @var bool
*/
protected $disableJSONLD = false;
/**
* @var array<string>
*/
protected $extraIgnoredElements = [];
/**
* Configuration constructor.
*
* @param array $params
*/
public function __construct(array $params = [])
{
foreach ($params as $key => $value) {
$setter = sprintf('set%s', $key);
if (method_exists($this, $setter)) {
call_user_func([$this, $setter], $value);
}
}
}
/**
* Returns an array-representation of configuration.
*
* @return array
*/
public function toArray()
{
$out = [];
foreach ($this as $key => $value) {
$getter = sprintf('get%s', $key);
if (!is_object($value) && method_exists($this, $getter)) {
$out[$key] = call_user_func([$this, $getter]);
}
}
return $out;
}
/**
* @return LoggerInterface
*/
public function getLogger()
{
// If no logger has been set, just return a null logger
if ($this->logger === null) {
return new NullLogger();
}
return $this->logger;
}
/**
* @param LoggerInterface $logger
*
* @return Configuration
*/
public function setLogger(LoggerInterface $logger)
{
$this->logger = $logger;
return $this;
}
/**
* @return int
*/
public function getMaxTopCandidates()
{
return $this->maxTopCandidates;
}
/**
* @param int $maxTopCandidates
*
* @return $this
*/
public function setMaxTopCandidates($maxTopCandidates)
{
$this->maxTopCandidates = $maxTopCandidates;
return $this;
}
/**
* @return int
*/
public function getCharThreshold()
{
return $this->charThreshold;
}
/**
* @param int $charThreshold
*
* @return $this
*/
public function setCharThreshold($charThreshold)
{
$this->charThreshold = $charThreshold;
return $this;
}
/**
* @return bool
*/
public function getArticleByLine()
{
return $this->articleByLine;
}
/**
* @param bool $articleByLine
*
* @return $this
*/
public function setArticleByLine($articleByLine)
{
$this->articleByLine = $articleByLine;
return $this;
}
/**
* @return bool
*/
public function getStripUnlikelyCandidates()
{
return $this->stripUnlikelyCandidates;
}
/**
* @param bool $stripUnlikelyCandidates
*
* @return $this
*/
public function setStripUnlikelyCandidates($stripUnlikelyCandidates)
{
$this->stripUnlikelyCandidates = $stripUnlikelyCandidates;
return $this;
}
/**
* @return bool
*/
public function getCleanConditionally()
{
return $this->cleanConditionally;
}
/**
* @param bool $cleanConditionally
*
* @return $this
*/
public function setCleanConditionally($cleanConditionally)
{
$this->cleanConditionally = $cleanConditionally;
return $this;
}
/**
* @return bool
*/
public function getWeightClasses()
{
return $this->weightClasses;
}
/**
* @param bool $weightClasses
*
* @return $this
*/
public function setWeightClasses($weightClasses)
{
$this->weightClasses = $weightClasses;
return $this;
}
/**
* @return bool
*/
public function getFixRelativeURLs()
{
return $this->fixRelativeURLs;
}
/**
* @param bool $fixRelativeURLs
*
* @return $this
*/
public function setFixRelativeURLs($fixRelativeURLs)
{
$this->fixRelativeURLs = $fixRelativeURLs;
return $this;
}
/**
* @return bool
*/
public function getSubstituteEntities()
{
return $this->substituteEntities;
}
/**
* @param bool $substituteEntities
*
* @return $this
*/
public function setSubstituteEntities($substituteEntities)
{
$this->substituteEntities = $substituteEntities;
return $this;
}
/**
* @return bool
*/
public function getNormalizeEntities()
{
return $this->normalizeEntities;
}
/**
* @param bool $normalizeEntities
*
* @return $this
*/
public function setNormalizeEntities($normalizeEntities)
{
$this->normalizeEntities = $normalizeEntities;
return $this;
}
/**
* @return string
*/
public function getOriginalURL()
{
return $this->originalURL;
}
/**
* @param string $originalURL
*
* @return $this
*/
public function setOriginalURL($originalURL)
{
$this->originalURL = $originalURL;
return $this;
}
/**
* @return string
*/
public function getParser()
{
return $this->parser;
}
/**
* @param string $parser
*
* @return $this
*/
public function setParser($parser)
{
$this->parser = $parser;
return $this;
}
/**
* @return bool
*/
public function getKeepClasses()
{
return $this->keepClasses;
}
/**
* @param bool $keepClasses
*
* @return $this
*/
public function setKeepClasses($keepClasses)
{
$this->keepClasses = $keepClasses;
return $this;
}
/**
* @return bool
*/
public function getDisableJSONLD()
{
return $this->disableJSONLD;
}
/**
* @param bool $disableJSONLD
*
* @return $this
*/
public function setDisableJSONLD($disableJSONLD)
{
$this->disableJSONLD = $disableJSONLD;
return $this;
}
/**
* @return bool
*/
public function getSummonCthulhu()
{
return $this->summonCthulhu;
}
/**
* @param bool $summonCthulhu
*
* @return $this
*/
public function setSummonCthulhu($summonCthulhu)
{
$this->summonCthulhu = $summonCthulhu;
return $this;
}
/**
* @return array<string>
*/
public function getExtraIgnoredElements()
{
return $this->extraIgnoredElements;
}
/**
* @param array<string>
*
* @return $this
*/
public function setExtraIgnoredElements($extraIgnoredElements)
{
$this->extraIgnoredElements = $extraIgnoredElements;
return $this;
}
}

10
vendor/fivefilters/readability.php/src/Nodes/DOM/DOMAttr.php vendored Normal file
View File

@@ -0,0 +1,10 @@
<?php
namespace fivefilters\Readability\Nodes\DOM;
use fivefilters\Readability\Nodes\NodeTrait;
class DOMAttr extends \DOMAttr
{
use NodeTrait;
}

10
vendor/fivefilters/readability.php/src/Nodes/DOM/DOMCdataSection.php vendored Normal file
View File

@@ -0,0 +1,10 @@
<?php
namespace fivefilters\Readability\Nodes\DOM;
use fivefilters\Readability\Nodes\NodeTrait;
class DOMCdataSection extends \DOMCdataSection
{
use NodeTrait;
}

10
vendor/fivefilters/readability.php/src/Nodes/DOM/DOMCharacterData.php vendored Normal file
View File

@@ -0,0 +1,10 @@
<?php
namespace fivefilters\Readability\Nodes\DOM;
use fivefilters\Readability\Nodes\NodeTrait;
class DOMCharacterData extends \DOMCharacterData
{
use NodeTrait;
}

10
vendor/fivefilters/readability.php/src/Nodes/DOM/DOMComment.php vendored Normal file
View File

@@ -0,0 +1,10 @@
<?php
namespace fivefilters\Readability\Nodes\DOM;
use fivefilters\Readability\Nodes\NodeTrait;
class DOMComment extends \DOMComment
{
use NodeTrait;
}

30
vendor/fivefilters/readability.php/src/Nodes/DOM/DOMDocument.php vendored Normal file
View File

@@ -0,0 +1,30 @@
<?php
namespace fivefilters\Readability\Nodes\DOM;
use fivefilters\Readability\Nodes\NodeTrait;
class DOMDocument extends \DOMDocument
{
use NodeTrait;
public function __construct($version, $encoding)
{
parent::__construct($version, $encoding);
$this->registerNodeClass('DOMAttr', DOMAttr::class);
$this->registerNodeClass('DOMCdataSection', DOMCdataSection::class);
$this->registerNodeClass('DOMCharacterData', DOMCharacterData::class);
$this->registerNodeClass('DOMComment', DOMComment::class);
$this->registerNodeClass('DOMDocument', self::class);
$this->registerNodeClass('DOMDocumentFragment', DOMDocumentFragment::class);
$this->registerNodeClass('DOMDocumentType', DOMDocumentType::class);
$this->registerNodeClass('DOMElement', DOMElement::class);
$this->registerNodeClass('DOMEntity', DOMEntity::class);
$this->registerNodeClass('DOMEntityReference', DOMEntityReference::class);
$this->registerNodeClass('DOMNode', DOMNode::class);
$this->registerNodeClass('DOMNotation', DOMNotation::class);
$this->registerNodeClass('DOMProcessingInstruction', DOMProcessingInstruction::class);
$this->registerNodeClass('DOMText', DOMText::class);
}
}

10
vendor/fivefilters/readability.php/src/Nodes/DOM/DOMDocumentFragment.php vendored Normal file
View File

@@ -0,0 +1,10 @@
<?php
namespace fivefilters\Readability\Nodes\DOM;
use fivefilters\Readability\Nodes\NodeTrait;
class DOMDocumentFragment extends \DOMDocumentFragment
{
use NodeTrait;
}

10
vendor/fivefilters/readability.php/src/Nodes/DOM/DOMDocumentType.php vendored Normal file
View File

@@ -0,0 +1,10 @@
<?php
namespace fivefilters\Readability\Nodes\DOM;
use fivefilters\Readability\Nodes\NodeTrait;
class DOMDocumentType extends \DOMDocumentType
{
use NodeTrait;
}

46
vendor/fivefilters/readability.php/src/Nodes/DOM/DOMElement.php vendored Normal file
View File

@@ -0,0 +1,46 @@
<?php
namespace fivefilters\Readability\Nodes\DOM;
use fivefilters\Readability\Nodes\NodeTrait;
class DOMElement extends \DOMElement
{
use NodeTrait;
/**
* Returns the child elements of this element.
*
* To get all child nodes, including non-element nodes like text and comment nodes, use childNodes.
*
* @return DOMNodeList
*/
public function children()
{
$newList = new DOMNodeList();
foreach ($this->childNodes as $node) {
if ($node->nodeType === XML_ELEMENT_NODE) {
$newList->add($node);
}
}
return $newList;
}
/**
* Returns the Element immediately prior to the specified one in its parent's children list, or null if the specified element is the first one in the list.
*
* @see https://wiki.php.net/rfc/dom_living_standard_api
* @return DOMElement|null
*/
public function previousElementSibling()
{
$previous = $this->previousSibling;
while ($previous) {
if ($previous->nodeType === XML_ELEMENT_NODE) {
return $previous;
}
$previous = $previous->previousSibling;
}
return null;
}
}

10
vendor/fivefilters/readability.php/src/Nodes/DOM/DOMEntity.php vendored Normal file
View File

@@ -0,0 +1,10 @@
<?php
namespace fivefilters\Readability\Nodes\DOM;
use fivefilters\Readability\Nodes\NodeTrait;
class DOMEntity extends \DOMEntity
{
use NodeTrait;
}

10
vendor/fivefilters/readability.php/src/Nodes/DOM/DOMEntityReference.php vendored Normal file
View File

@@ -0,0 +1,10 @@
<?php
namespace fivefilters\Readability\Nodes\DOM;
use fivefilters\Readability\Nodes\NodeTrait;
class DOMEntityReference extends \DOMEntityReference
{
use NodeTrait;
}

14
vendor/fivefilters/readability.php/src/Nodes/DOM/DOMNode.php vendored Normal file
View File

@@ -0,0 +1,14 @@
<?php
namespace fivefilters\Readability\Nodes\DOM;
use fivefilters\Readability\Nodes\NodeTrait;
/**
* @method getAttribute($attribute)
* @method hasAttribute($attribute)
*/
class DOMNode extends \DOMNode
{
use NodeTrait;
}

82
vendor/fivefilters/readability.php/src/Nodes/DOM/DOMNodeList.php vendored Normal file
View File

@@ -0,0 +1,82 @@
<?php
namespace fivefilters\Readability\Nodes\DOM;
/**
* Class DOMNodeList.
*
* This is a fake DOMNodeList class that allows adding items to the list. The original class is static and the nodes
* are defined automagically when instantiating it. This fake version behaves exactly the same way but adds the function
* add() that allows to insert new DOMNodes into the DOMNodeList.
*
* It cannot extend the original DOMNodeList class because the functionality behind the property ->length is hidden
* from the user and cannot be extended, changed, or tweaked.
*/
class DOMNodeList implements \Countable, \IteratorAggregate
{
/**
* @var array
*/
protected $items = [];
/**
* @var int
*/
protected $length = 0;
/**
* To allow access to length in the same way that DOMNodeList allows.
*
* {@inheritdoc}
*/
public function __get($name)
{
switch ($name) {
case 'length':
return $this->length;
default:
trigger_error(sprintf('Undefined property: %s::%s', static::class, $name));
}
}
/**
* @param DOMNode|DOMElement|DOMComment $node
*
* @return DOMNodeList
*/
public function add($node)
{
$this->items[] = $node;
$this->length++;
return $this;
}
/**
* @param int $offset
*
* @return DOMNode|DOMElement|DOMComment
*/
public function item(int $offset)
{
return $this->items[$offset];
}
/**
* @return int|void
*/
public function count(): int
{
return $this->length;
}
/**
* To make it compatible with iterator_to_array() function.
*
* {@inheritdoc}
*/
public function getIterator(): \ArrayIterator
{
return new \ArrayIterator($this->items);
}
}

10
vendor/fivefilters/readability.php/src/Nodes/DOM/DOMNotation.php vendored Normal file
View File

@@ -0,0 +1,10 @@
<?php
namespace fivefilters\Readability\Nodes\DOM;
use fivefilters\Readability\Nodes\NodeTrait;
class DOMNotation extends \DOMNotation
{
use NodeTrait;
}

10
vendor/fivefilters/readability.php/src/Nodes/DOM/DOMProcessingInstruction.php vendored Normal file
View File

@@ -0,0 +1,10 @@
<?php
namespace fivefilters\Readability\Nodes\DOM;
use fivefilters\Readability\Nodes\NodeTrait;
class DOMProcessingInstruction extends \DOMProcessingInstruction
{
use NodeTrait;
}

10
vendor/fivefilters/readability.php/src/Nodes/DOM/DOMText.php vendored Normal file
View File

@@ -0,0 +1,10 @@
<?php
namespace fivefilters\Readability\Nodes\DOM;
use fivefilters\Readability\Nodes\NodeTrait;
class DOMText extends \DOMText
{
use NodeTrait;
}

566
vendor/fivefilters/readability.php/src/Nodes/NodeTrait.php vendored Normal file
View File

@@ -0,0 +1,566 @@
<?php
namespace fivefilters\Readability\Nodes;
use fivefilters\Readability\Nodes\DOM\DOMDocument;
use fivefilters\Readability\Nodes\DOM\DOMElement;
use fivefilters\Readability\Nodes\DOM\DOMNode;
use fivefilters\Readability\Nodes\DOM\DOMText;
use DOMNodeList;
/**
* @method \DOMNode removeAttribute($name)
*/
trait NodeTrait
{
/**
* Content score of the node. Used to determine the value of the content.
*
* @var int
*/
public $contentScore = 0;
/**
* Flag for initialized status.
*
* @var bool
*/
private $initialized = false;
/**
* Flag data tables.
*
* @var bool
*/
private $readabilityDataTable = false;
/**
* @var array
*/
private $divToPElements = [
'blockquote',
'dl',
'div',
'img',
'ol',
'p',
'pre',
'table',
'ul'
];
/**
* The commented out elements qualify as phrasing content but tend to be
* removed by readability when put into paragraphs, so we ignore them here.
*
* @var array
*/
private $phrasing_elems = [
// 'CANVAS', 'IFRAME', 'SVG', 'VIDEO',
'abbr', 'audio', 'b', 'bdo', 'br', 'button', 'cite', 'code', 'data',
'datalist', 'dfn', 'em', 'embed', 'i', 'img', 'input', 'kbd', 'label',
'mark', 'math', 'meter', 'noscript', 'object', 'output', 'progress', 'q',
'ruby', 'samp', 'script', 'select', 'small', 'span', 'strong', 'sub',
'sup', 'textarea', 'time', 'var', 'wbr'
];
/**
* initialized getter.
*
* @return bool
*/
public function isInitialized()
{
return $this->initialized;
}
/**
* @return bool
*/
public function isReadabilityDataTable()
{
/*
* This is a workaround that I'd like to remove in the future.
* Seems that although we are extending the base DOMElement and adding custom properties (like this one,
* 'readabilityDataTable'), these properties get lost when you search for elements with getElementsByTagName.
* This means that even if we mark the tables in a previous step, when we want to retrieve that information,
* all the custom properties are in their default values. Somehow we need to find a way to make these properties
* permanent across the whole DOM.
*
* @see https://stackoverflow.com/questions/35654709/php-registernodeclass-and-reusing-variable-names
*/
return $this->hasAttribute('readabilityDataTable')
&& $this->getAttribute('readabilityDataTable') === '1';
// return $this->readabilityDataTable;
}
/**
* @param bool $param
*/
public function setReadabilityDataTable($param)
{
// Can't be "true" because DOMDocument casts it to "1"
$this->setAttribute('readabilityDataTable', $param ? '1' : '0');
// $this->readabilityDataTable = $param;
}
/**
* Initializer. Calculates the current score of the node and returns a full Readability object.
*
* @ TODO: I don't like the weightClasses param. How can we get the config here?
*
* @param $weightClasses bool Weight classes?
*
* @return static
*/
public function initializeNode($weightClasses)
{
if (!$this->isInitialized()) {
$contentScore = 0;
switch ($this->nodeName) {
case 'div':
$contentScore += 5;
break;
case 'pre':
case 'td':
case 'blockquote':
$contentScore += 3;
break;
case 'address':
case 'ol':
case 'ul':
case 'dl':
case 'dd':
case 'dt':
case 'li':
case 'form':
$contentScore -= 3;
break;
case 'h1':
case 'h2':
case 'h3':
case 'h4':
case 'h5':
case 'h6':
case 'th':
$contentScore -= 5;
break;
}
$this->contentScore = $contentScore + ($weightClasses ? $this->getClassWeight() : 0);
$this->initialized = true;
}
return $this;
}
/**
* Override for native getAttribute method. Some nodes have the getAttribute method, some don't, so we need
* to check first the existence of the attributes property.
*
* @param $attributeName string Attribute to retrieve
*
* @return string
*/
#[\ReturnTypeWillChange]
public function getAttribute($attributeName)
{
if (!is_null($this->attributes)) {
return parent::getAttribute($attributeName);
}
return '';
}
/**
* Override for native hasAttribute.
*
* @param $attributeName
*
* @return bool
*
* @see getAttribute
*/
#[\ReturnTypeWillChange]
public function hasAttribute($attributeName)
{
if (!is_null($this->attributes)) {
return parent::hasAttribute($attributeName);
}
return false;
}
/**
* Get the ancestors of the current node.
*
* @param int|bool $maxLevel Max amount of ancestors to get. False for all of them
*
* @return array
*/
public function getNodeAncestors($maxLevel = 3)
{
$ancestors = [];
$level = 0;
$node = $this->parentNode;
while ($node && !($node instanceof DOMDocument)) {
$ancestors[] = $node;
$level++;
if ($level === $maxLevel) {
break;
}
$node = $node->parentNode;
}
return $ancestors;
}
/**
* Returns all links from the current element.
*
* @return array
*/
public function getAllLinks()
{
return iterator_to_array($this->getElementsByTagName('a'));
}
/**
* Get the density of links as a percentage of the content
* This is the amount of text that is inside a link divided by the total text in the node.
*
* @return int
*/
public function getLinkDensity()
{
$textLength = mb_strlen($this->getTextContent(true));
if ($textLength === 0) {
return 0;
}
$linkLength = 0;
$links = $this->getAllLinks();
if ($links) {
/** @var DOMElement $link */
foreach ($links as $link) {
$href = $link->getAttribute('href');
$coefficient = ($href && preg_match(NodeUtility::$regexps['hashUrl'], $href)) ? 0.3 : 1;
$linkLength += mb_strlen($link->getTextContent(true)) * $coefficient;
}
}
return $linkLength / $textLength;
}
/**
* Calculates the weight of the class/id of the current element.
*
* @return int
*/
public function getClassWeight()
{
$weight = 0;
// Look for a special classname
$class = $this->getAttribute('class');
if (trim($class)) {
if (preg_match(NodeUtility::$regexps['negative'], $class)) {
$weight -= 25;
}
if (preg_match(NodeUtility::$regexps['positive'], $class)) {
$weight += 25;
}
}
// Look for a special ID
$id = $this->getAttribute('id');
if (trim($id) !== '') {
if (preg_match(NodeUtility::$regexps['negative'], $id)) {
$weight -= 25;
}
if (preg_match(NodeUtility::$regexps['positive'], $id)) {
$weight += 25;
}
}
return $weight;
}
/**
* Returns the full text of the node.
*
* @param bool $normalize Normalize white space?
*
* @return string
*/
public function getTextContent($normalize = true)
{
$nodeValue = trim($this->textContent);
if ($normalize) {
$nodeValue = preg_replace(NodeUtility::$regexps['normalize'], ' ', $nodeValue);
}
return $nodeValue;
}
/**
* Return an array indicating how many rows and columns this table has.
*
* @return array
*/
public function getRowAndColumnCount()
{
$rows = $columns = 0;
$trs = $this->getElementsByTagName('tr');
foreach ($trs as $tr) {
/** @var \DOMElement $tr */
$rowspan = $tr->getAttribute('rowspan');
$rows += ($rowspan || 1);
// Now look for column-related info
$columnsInThisRow = 0;
$cells = $tr->getElementsByTagName('td');
foreach ($cells as $cell) {
/** @var \DOMElement $cell */
$colspan = $cell->getAttribute('colspan');
$columnsInThisRow += ($colspan || 1);
}
$columns = max($columns, $columnsInThisRow);
}
return ['rows' => $rows, 'columns' => $columns];
}
/**
* Creates a new node based on the text content of the original node.
*
* @param $originalNode DOMNode
* @param $tagName string
*
* @return DOMElement
*/
public function createNode($originalNode, $tagName)
{
$text = $originalNode->getTextContent(false);
$newNode = $originalNode->ownerDocument->createElement($tagName, $text);
return $newNode;
}
/**
* Check if a given node has one of its ancestor tag name matching the
* provided one.
*
* @param string $tagName
* @param int $maxDepth
* @param callable $filterFn
*
* @return bool
*/
public function hasAncestorTag($tagName, $maxDepth = 3, ?callable $filterFn = null)
{
$depth = 0;
$node = $this;
while ($node->parentNode) {
if ($maxDepth > 0 && $depth > $maxDepth) {
return false;
}
if ($node->parentNode->nodeName === $tagName && (!$filterFn || $filterFn($node->parentNode))) {
return true;
}
$node = $node->parentNode;
$depth++;
}
return false;
}
/**
* Check if this node has only whitespace and a single element with given tag
* or if it contains no element with given tag or more than 1 element.
*
* @param $tag string Name of tag
*
* @return bool
*/
public function hasSingleTagInsideElement($tag)
{
// There should be exactly 1 element child with given tag
if (count($children = NodeUtility::filterTextNodes($this->childNodes)) !== 1 || $children->item(0)->nodeName !== $tag) {
return false;
}
// And there should be no text nodes with real content
return array_reduce(iterator_to_array($children), function ($carry, $child) {
if (!$carry === false) {
return false;
}
/* @var DOMNode $child */
return !($child->nodeType === XML_TEXT_NODE && preg_match(NodeUtility::$regexps['hasContent'], $child->textContent));
});
}
/**
* Check if the current element has a single child block element.
* Block elements are the ones defined in the divToPElements array.
*
* @return bool
*/
public function hasSingleChildBlockElement()
{
$result = false;
if ($this->hasChildNodes()) {
foreach ($this->childNodes as $child) {
if (in_array($child->nodeName, $this->divToPElements)) {
$result = true;
} else {
// If any of the hasSingleChildBlockElement calls return true, return true then.
/** @var $child DOMElement */
$result = ($result || $child->hasSingleChildBlockElement());
}
}
}
return $result;
}
/**
* Determines if a node has no content or it is just a bunch of dividing lines and/or whitespace.
*
* @return bool
*/
public function isElementWithoutContent()
{
return $this instanceof DOMElement &&
mb_strlen(preg_replace(NodeUtility::$regexps['onlyWhitespace'], '', $this->textContent)) === 0 &&
($this->childNodes->length === 0 ||
$this->childNodes->length === $this->getElementsByTagName('br')->length + $this->getElementsByTagName('hr')->length
/*
* Special PHP DOMDocument case: We also need to count how many DOMText we have inside the node.
* If there's an empty tag with an space inside and a BR (for example "<p> <br/></p>) counting only BRs and
* HRs will will say that the example has 2 nodes, instead of one. This happens because in DOMDocument,
* DOMTexts are also nodes (which doesn't happen in JS). So we need to also count how many DOMText we
* are dealing with (And at this point we know they are empty or are just whitespace, because of the
* mb_strlen in this chain of checks).
*/
+ count(array_filter(iterator_to_array($this->childNodes), function ($child) {
return $child instanceof DOMText;
}))
);
}
/**
* Determine if a node qualifies as phrasing content.
* https://developer.mozilla.org/en-US/docs/Web/Guide/HTML/Content_categories#Phrasing_content.
*
* @return bool
*/
public function isPhrasingContent()
{
return $this->nodeType === XML_TEXT_NODE || in_array($this->nodeName, $this->phrasing_elems) !== false ||
(!is_null($this->childNodes) &&
($this->nodeName === 'a' || $this->nodeName === 'del' || $this->nodeName === 'ins') &&
array_reduce(iterator_to_array($this->childNodes), function ($carry, $node) {
return $node->isPhrasingContent() && $carry;
}, true)
);
}
/**
* In the original JS project they check if the node has the style display=none, which unfortunately
* in our case we have no way of knowing that. So we just check for the attribute hidden or "display: none".
*
* @return bool
*/
public function isProbablyVisible()
{
return !preg_match('/display:( )?none/i', $this->getAttribute('style')) &&
!$this->hasAttribute('hidden') &&
//check for "fallback-image" so that wikimedia math images are displayed
(!$this->hasAttribute('aria-hidden') || $this->getAttribute('aria-hidden') !== 'true' || ($this->hasAttribute('class') && strpos($this->getAttribute('class'), 'fallback-image') !== false));
}
/**
* @return bool
*/
public function isWhitespace()
{
return ($this->nodeType === XML_TEXT_NODE && mb_strlen(trim($this->textContent)) === 0) ||
($this->nodeType === XML_ELEMENT_NODE && $this->nodeName === 'br');
}
/**
* This is a hack that overcomes the issue of node shifting when scanning and removing nodes.
*
* In the JS version of getElementsByTagName, if you remove a node it will not appear during the
* foreach. This does not happen in PHP DOMDocument, because if you remove a node, it will still appear but as an
* orphan node and will give an exception if you try to do anything with it.
*
* Shifting also occurs when converting parent nodes (like a P to a DIV), which in that case the found nodes are
* removed from the foreach "pool" but the internal index of the foreach is not aware and skips over nodes that
* never looped over. (index is at position 5, 2 nodes are removed, next one should be node 3, but the foreach tries
* to access node 6)
*
* This function solves this by searching for the nodes on every loop and keeping track of the count differences.
* Because on every loop we call getElementsByTagName again, this could cause a performance impact and should be
* used only when the results of the search are going to be used to remove the nodes.
*
* @param string $tag
*
* @return \Generator
*/
public function shiftingAwareGetElementsByTagName($tag)
{
/** @var $nodes DOMNodeList */
$nodes = $this->getElementsByTagName($tag);
$count = $nodes->length;
for ($i = 0; $i < $count; $i = max(++$i, 0)) {
yield $nodes->item($i);
// Search for all the nodes again
$nodes = $this->getElementsByTagName($tag);
// Subtract the amount of nodes removed from the current index
$i -= $count - $nodes->length;
// Subtract the amount of nodes removed from the current count
$count -= ($count - $nodes->length);
}
}
/**
* Mimics JS's firstElementChild property. PHP only has firstChild which could be any type of DOMNode. Use this
* function to get the first one that is an DOMElement node.
*
* @return \DOMElement|null
*/
public function getFirstElementChild()
{
if ($this->childNodes instanceof \Traversable) {
foreach ($this->childNodes as $node) {
if ($node instanceof \DOMElement) {
return $node;
}
}
}
return null;
}
}

192
vendor/fivefilters/readability.php/src/Nodes/NodeUtility.php vendored Normal file
View File

@@ -0,0 +1,192 @@
<?php
namespace fivefilters\Readability\Nodes;
use fivefilters\Readability\Nodes\DOM\DOMDocument;
use fivefilters\Readability\Nodes\DOM\DOMElement;
use fivefilters\Readability\Nodes\DOM\DOMNode;
use fivefilters\Readability\Nodes\DOM\DOMNodeList;
/**
* Class NodeUtility.
*/
class NodeUtility
{
/**
* Collection of regexps to check the node usability.
*
* @var array
*/
public static $regexps = [
'unlikelyCandidates' => '/-ad-|ai2html|banner|breadcrumbs|combx|comment|community|cover-wrap|disqus|extra|footer|gdpr|header|legends|menu|related|remark|replies|rss|shoutbox|sidebar|skyscraper|social|sponsor|supplemental|ad-break|agegate|pagination|pager|popup|yom-remote/i',
'okMaybeItsACandidate' => '/and|article|body|column|content|main|shadow/i',
'extraneous' => '/print|archive|comment|discuss|e[\-]?mail|share|reply|all|login|sign|single|utility/i',
'byline' => '/byline|author|dateline|writtenby|p-author/i',
'replaceFonts' => '/<(\/?)font[^>]*>/i',
'normalize' => '/\s{2,}/',
'videos' => '/\/\/(www\.)?((dailymotion|youtube|youtube-nocookie|player\.vimeo|v\.qq)\.com|(archive|upload\.wikimedia)\.org|player\.twitch\.tv)/i',
'shareElements' => '/(\b|_)(share|sharedaddy)(\b|_)/i',
'nextLink' => '/(next|weiter|continue|>([^\|]|$)|»([^\|]|$))/i',
'prevLink' => '/(prev|earl|old|new|<|«)/i',
'tokenize' => '/\W+/',
'whitespace' => '/^\s*$/',
'hasContent' => '/\S$/',
'positive' => '/article|body|content|entry|hentry|h-entry|main|page|pagination|post|text|blog|story/i',
'negative' => '/-ad-|hidden|^hid$| hid$| hid |^hid |banner|combx|comment|com-|contact|foot|footer|footnote|gdpr|masthead|media|meta|outbrain|promo|related|scroll|share|shoutbox|sidebar|skyscraper|sponsor|shopping|tags|tool|widget/i',
// \x{00A0} is the unicode version of &nbsp;
'onlyWhitespace' => '/\x{00A0}|\s+/u',
'hashUrl' => '/^#.+/',
'srcsetUrl' => '/(\S+)(\s+[\d.]+[xw])?(\s*(?:,|$))/',
'b64DataUrl' => '/^data:\s*([^\s;,]+)\s*;\s*base64\s*,/i',
// See: https://schema.org/Article
'jsonLdArticleTypes' => '/^Article|AdvertiserContentArticle|NewsArticle|AnalysisNewsArticle|AskPublicNewsArticle|BackgroundNewsArticle|OpinionNewsArticle|ReportageNewsArticle|ReviewNewsArticle|Report|SatiricalArticle|ScholarlyArticle|MedicalScholarlyArticle|SocialMediaPosting|BlogPosting|LiveBlogPosting|DiscussionForumPosting|TechArticle|APIReference$/'
];
/**
* Finds the next node, starting from the given node, and ignoring
* whitespace in between. If the given node is an element, the same node is
* returned.
*
* Imported from the Element class on league\html-to-markdown.
*
* @param $node
*
* @return DOMNode
*/
public static function nextNode($node)
{
$next = $node;
while ($next
&& $next->nodeType !== XML_ELEMENT_NODE
&& $next->isWhitespace()) {
$next = $next->nextSibling;
}
return $next;
}
/**
* Changes the node tag name. Since tagName on DOMElement is a read only value, this must be done creating a new
* element with the new tag name and importing it to the main DOMDocument.
*
* @param DOMNode $node
* @param string $value
* @param bool $importAttributes
*
* @return DOMNode
*/
public static function setNodeTag($node, $value, $importAttributes = true)
{
$new = new DOMDocument('1.0', 'utf-8');
$new->appendChild($new->createElement($value));
$children = $node->childNodes;
/** @var $children \DOMNodeList $i */
for ($i = 0; $i < $children->length; $i++) {
$import = $new->importNode($children->item($i), true);
$new->firstChild->appendChild($import);
}
if ($importAttributes) {
// Import attributes from the original node.
foreach ($node->attributes as $attribute) {
$new->firstChild->setAttribute($attribute->nodeName, $attribute->nodeValue);
}
}
// The import must be done on the firstChild of $new, since $new is a DOMDocument and not a DOMElement.
$import = $node->ownerDocument->importNode($new->firstChild, true);
$node->parentNode->replaceChild($import, $node);
return $import;
}
/**
* Removes the current node and returns the next node to be parsed (child, sibling or parent).
*
* @param DOMNode $node
*
* @return DOMNode
*/
public static function removeAndGetNext($node)
{
$nextNode = self::getNextNode($node, true);
$node->parentNode->removeChild($node);
return $nextNode;
}
/**
* Remove the selected node.
*
* @param $node DOMElement
*
* @return void
**/
public static function removeNode($node)
{
$parent = $node->parentNode;
if ($parent) {
$parent->removeChild($node);
}
}
/**
* Returns the next node. First checks for children (if the flag allows it), then for siblings, and finally
* for parents.
*
* @param DOMNode $originalNode
* @param bool $ignoreSelfAndKids
*
* @return DOMNode
*/
public static function getNextNode($originalNode, $ignoreSelfAndKids = false)
{
/*
* Traverse the DOM from node to node, starting at the node passed in.
* Pass true for the second parameter to indicate this node itself
* (and its kids) are going away, and we want the next node over.
*
* Calling this in a loop will traverse the DOM depth-first.
*/
// First check for kids if those aren't being ignored
if (!$ignoreSelfAndKids && $originalNode->firstChild) {
return $originalNode->firstChild;
}
// Then for siblings...
if ($originalNode->nextSibling) {
return $originalNode->nextSibling;
}
// And finally, move up the parent chain *and* find a sibling
// (because this is depth-first traversal, we will have already
// seen the parent nodes themselves).
do {
$originalNode = $originalNode->parentNode;
} while ($originalNode && !$originalNode->nextSibling);
return ($originalNode) ? $originalNode->nextSibling : $originalNode;
}
/**
* Remove all empty DOMNodes from DOMNodeLists.
*
* @param \DOMNodeList $list
*
* @return DOMNodeList
*/
public static function filterTextNodes(\DOMNodeList $list)
{
$newList = new DOMNodeList();
foreach ($list as $node) {
if ($node->nodeType !== XML_TEXT_NODE || mb_strlen(trim($node->nodeValue))) {
$newList->add($node);
}
}
return $newList;
}
}

7
vendor/fivefilters/readability.php/src/ParseException.php vendored Normal file
View File

@@ -0,0 +1,7 @@
<?php
namespace fivefilters\Readability;
class ParseException extends \Exception
{
}

2484
vendor/fivefilters/readability.php/src/Readability.php vendored Normal file
View File

File diff suppressed because it is too large Load Diff

8
vendor/masterminds/html5/.gitattributes vendored Normal file
View File

@@ -0,0 +1,8 @@
.gitattributes export-ignore
.gitignore export-ignore
.php_cs.dist export-ignore
.scrutinizer.yml export-ignore
.travis.yml export-ignore
example.php export-ignore
phpunit.xml.dist export-ignore
/test export-ignore

5
vendor/masterminds/html5/.gitignore vendored Normal file
View File

@@ -0,0 +1,5 @@
vendor/
scratch.php
composer.lock
build/
.php_cs.cache

14
vendor/masterminds/html5/.php_cs.dist vendored Normal file
View File

@@ -0,0 +1,14 @@
<?php
$finder = PhpCsFixer\Finder::create()
->in(__DIR__)
;
return PhpCsFixer\Config::create()
->setRules(array(
'@Symfony' => true,
'concat_space' => array('spacing' => 'one'),
'phpdoc_annotation_without_dot' => false,
))
->setFinder($finder)
;

41
vendor/masterminds/html5/.scrutinizer.yml vendored Normal file
View File

@@ -0,0 +1,41 @@
tools:
php_mess_detector: true
php_analyzer:
config:
parameter_reference_check: { enabled: false }
checkstyle: { enabled: false, no_trailing_whitespace: true, naming: { enabled: true, local_variable: '^[a-z][a-zA-Z0-9]*$', abstract_class_name: ^Abstract|Factory$, utility_class_name: 'Utils?$', constant_name: '^[A-Z][A-Z0-9]*(?:_[A-Z0-9]+)*$', property_name: '^[a-z][a-zA-Z0-9]*$', method_name: '^(?:[a-z]|__)[a-zA-Z0-9]*$', parameter_name: '^[a-z][a-zA-Z0-9]*$', interface_name: '^[A-Z][a-zA-Z0-9]*Interface$', type_name: '^[A-Z][a-zA-Z0-9]*$', exception_name: '^[A-Z][a-zA-Z0-9]*Exception$', isser_method_name: '^(?:is|has|should|may|supports)' } }
unreachable_code: { enabled: false }
check_access_control: { enabled: false }
typo_checks: { enabled: false }
check_variables: { enabled: false }
check_calls: { enabled: true, too_many_arguments: true, missing_argument: true, argument_type_checks: lenient }
suspicious_code: { enabled: false, overriding_parameter: false, overriding_closure_use: false, parameter_closure_use_conflict: false, parameter_multiple_times: false, non_existent_class_in_instanceof_check: false, non_existent_class_in_catch_clause: false, assignment_of_null_return: false, non_commented_switch_fallthrough: false, non_commented_empty_catch_block: false, overriding_private_members: false, use_statement_alias_conflict: false, precedence_in_condition_assignment: false }
dead_assignments: { enabled: false }
verify_php_doc_comments: { enabled: false, parameters: false, return: false, suggest_more_specific_types: false, ask_for_return_if_not_inferrable: false, ask_for_param_type_annotation: false }
loops_must_use_braces: { enabled: false }
check_usage_context: { enabled: true, foreach: { value_as_reference: true, traversable: true } }
simplify_boolean_return: { enabled: false }
phpunit_checks: { enabled: false }
reflection_checks: { enabled: false }
precedence_checks: { enabled: true, assignment_in_condition: true, comparison_of_bit_result: true }
basic_semantic_checks: { enabled: false }
unused_code: { enabled: false }
deprecation_checks: { enabled: false }
useless_function_calls: { enabled: false }
metrics_lack_of_cohesion_methods: { enabled: false }
metrics_coupling: { enabled: true, stable_code: { namespace_prefixes: { }, classes: { } } }
doctrine_parameter_binding: { enabled: false }
doctrine_entity_manager_injection: { enabled: false }
symfony_request_injection: { enabled: false }
doc_comment_fixes: { enabled: false }
reflection_fixes: { enabled: false }
use_statement_fixes: { enabled: true, remove_unused: true, preserve_multiple: false, preserve_blanklines: false, order_alphabetically: false }
php_code_sniffer: true
sensiolabs_security_checker: true
php_cpd: true
php_loc: true
php_pdepend: true
external_code_coverage: true
filter:
paths:
- src/*

47
vendor/masterminds/html5/.travis.yml vendored Normal file
View File

@@ -0,0 +1,47 @@
language: php
dist: trusty
sudo: false
matrix:
include:
- php: 5.3
dist: precise
env: COMPOSER_FLAGS="--prefer-lowest"
- php: 5.4
- php: 5.5
- php: 5.6
env: SCRUTINIZER=1 PHPUNIT_FLAGS="--coverage-clover=coverage.xml"
- php: 7.0
env: COMPOSER_FLAGS="--prefer-lowest"
- php: 7.1
env: CS_FIXER=1
- php: 7.2
- php: 7.3
env: BENCHMARK=1
- php: 7.4
- php: 8.0
dist: bionic
fast_finish: true
cache:
directories:
- $HOME/.composer/cache
before_script:
- if [[ SCRUTINIZER != '1' ]]; then phpenv config-rm xdebug.ini || true; fi
- composer self-update
- if [[ ${TRAVIS_PHP_VERSION:0:1} == "8" ]]; then composer config platform.php 7.4; fi
- composer update $COMPOSER_FLAGS --prefer-dist
script:
- stty cols 120
- if [ "$CS_FIXER" == 1 ]; then wget https://github.com/FriendsOfPHP/PHP-CS-Fixer/releases/download/v2.13.1/php-cs-fixer.phar && php php-cs-fixer.phar fix --dry-run --diff; fi
- mkdir -p build/logs
- ./vendor/bin/phpunit $PHPUNIT_FLAGS
after_script:
- if [ "$SCRUTINIZER" == 1 ]; then wget https://scrutinizer-ci.com/ocular.phar && php ocular.phar code-coverage:upload --format=php-clover coverage.xml; fi
- if [ "$BENCHMARK" == 1 ]; then php test/benchmark/run.php 10; fi
notifications:
irc: "irc.freenode.net#masterminds"

11
vendor/masterminds/html5/CREDITS vendored Normal file
View File

@@ -0,0 +1,11 @@
Matt Butcher [technosophos] <technosophos@gmail.com> (lead)
Matt Farina [mattfarina] <matt@mattfarina.com> (lead)
Asmir Mustafic [goetas] <goetas@lignano.it> (contributor)
Edward Z. Yang [ezyang] <ezyang@mit.edu> (contributor)
Geoffrey Sneddon [gsnedders] <geoffers@gmail.com> (contributor)
Kukhar Vasily [ngreduce] <ngreduce@gmail.com> (contributor)
Rune Christensen [MrElectronic] <mrelectronic@example.com> (contributor)
Mišo Belica [miso-belica] <miso-belica@example.com> (contributor)
Asmir Mustafic [goetas] <goetas@example.com> (contributor)
KITAITI Makoto [KitaitiMakoto] <KitaitiMakoto@example.com> (contributor)
Jacob Floyd [cognifloyd] <cognifloyd@gmail.com> (contributor)

19
vendor/masterminds/html5/Jenkinsfile vendored Normal file
View File

@@ -0,0 +1,19 @@
pipeline {
agent any
options {
buildDiscarder(logRotator(numToKeepStr: '5'))
}
stages {
stage('phpstan') {
steps {
sh """
docker run --rm -v ${env.WORKSPACE}:/src/tt-rss/lib/local \
--workdir /src/tt-rss registry.fakecake.org/cthulhoo/ttrss-fpm-pgsql-static:latest\
php81 ./vendor/bin/phpstan analyse lib/local --memory-limit=1G
"""
}
}
}
}

66
vendor/masterminds/html5/LICENSE.txt vendored Normal file
View File

@@ -0,0 +1,66 @@
## HTML5-PHP License
Copyright (c) 2013 The Authors of HTML5-PHP
Matt Butcher - mattbutcher@google.com
Matt Farina - matt@mattfarina.com
Asmir Mustafic - goetas@gmail.com
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
## HTML5Lib License
Portions of this are based on html5lib's PHP version, which was a
sub-project of html5lib. The following is the list of contributors from
html5lib:
html5lib:
Copyright (c) 2006-2009 The Authors
Contributors:
James Graham - jg307@cam.ac.uk
Anne van Kesteren - annevankesteren@gmail.com
Lachlan Hunt - lachlan.hunt@lachy.id.au
Matt McDonald - kanashii@kanashii.ca
Sam Ruby - rubys@intertwingly.net
Ian Hickson (Google) - ian@hixie.ch
Thomas Broyer - t.broyer@ltgt.net
Jacques Distler - distler@golem.ph.utexas.edu
Henri Sivonen - hsivonen@iki.fi
Adam Barth - abarth@webkit.org
Eric Seidel - eric@webkit.org
The Mozilla Foundation (contributions from Henri Sivonen since 2008)
David Flanagan (Mozilla) - dflanagan@mozilla.com
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

254
vendor/masterminds/html5/README.md vendored Normal file
View File

@@ -0,0 +1,254 @@
# HTML5-PHP
HTML5 is a standards-compliant HTML5 parser and writer written entirely in PHP.
It is stable and used in many production websites, and has
well over [five million downloads](https://packagist.org/packages/masterminds/html5).
HTML5 provides the following features.
- An HTML5 serializer
- Support for PHP namespaces
- Composer support
- Event-based (SAX-like) parser
- A DOM tree builder
- Interoperability with [QueryPath](https://github.com/technosophos/querypath)
- Runs on **PHP** 5.3.0 or newer
[![Build Status](https://travis-ci.org/Masterminds/html5-php.png?branch=master)](https://travis-ci.org/Masterminds/html5-php)
[![Latest Stable Version](https://poser.pugx.org/masterminds/html5/v/stable.png)](https://packagist.org/packages/masterminds/html5)
[![Code Coverage](https://scrutinizer-ci.com/g/Masterminds/html5-php/badges/coverage.png?b=master)](https://scrutinizer-ci.com/g/Masterminds/html5-php/?branch=master)
[![Scrutinizer Code Quality](https://scrutinizer-ci.com/g/Masterminds/html5-php/badges/quality-score.png?b=master)](https://scrutinizer-ci.com/g/Masterminds/html5-php/?branch=master)
[![Stability: Sustained](https://masterminds.github.io/stability/sustained.svg)](https://masterminds.github.io/stability/sustained.html)
## Installation
Install HTML5-PHP using [composer](http://getcomposer.org/).
By adding the `masterminds/html5` dependency to your `composer.json` file:
```json
{
"require" : {
"masterminds/html5": "^2.0"
},
}
```
By invoking require command via composer executable:
```bash
composer require masterminds/html5
```
## Basic Usage
HTML5-PHP has a high-level API and a low-level API.
Here is how you use the high-level `HTML5` library API:
```php
<?php
// Assuming you installed from Composer:
require "vendor/autoload.php";
use Masterminds\HTML5;
// An example HTML document:
$html = <<< 'HERE'
<html>
<head>
<title>TEST</title>
</head>
<body id='foo'>
<h1>Hello World</h1>
<p>This is a test of the HTML5 parser.</p>
</body>
</html>
HERE;
// Parse the document. $dom is a DOMDocument.
$html5 = new HTML5();
$dom = $html5->loadHTML($html);
// Render it as HTML5:
print $html5->saveHTML($dom);
// Or save it to a file:
$html5->save($dom, 'out.html');
```
The `$dom` created by the parser is a full `DOMDocument` object. And the
`save()` and `saveHTML()` methods will take any DOMDocument.
### Options
It is possible to pass in an array of configuration options when loading
an HTML5 document.
```php
// An associative array of options
$options = array(
'option_name' => 'option_value',
);
// Provide the options to the constructor
$html5 = new HTML5($options);
$dom = $html5->loadHTML($html);
```
The following options are supported:
* `encode_entities` (boolean): Indicates that the serializer should aggressively
encode characters as entities. Without this, it only encodes the bare
minimum.
* `disable_html_ns` (boolean): Prevents the parser from automatically
assigning the HTML5 namespace to the DOM document. This is for
non-namespace aware DOM tools.
* `target_document` (\DOMDocument): A DOM document that will be used as the
destination for the parsed nodes.
* `implicit_namespaces` (array): An assoc array of namespaces that should be
used by the parser. Name is tag prefix, value is NS URI.
## The Low-Level API
This library provides the following low-level APIs that you can use to
create more customized HTML5 tools:
- A SAX-like event-based parser that you can hook into for special kinds
of parsing.
- A flexible error-reporting mechanism that can be tuned to document
syntax checking.
- A DOM implementation that uses PHP's built-in DOM library.
The unit tests exercise each piece of the API, and every public function
is well-documented.
### Parser Design
The parser is designed as follows:
- The `Scanner` handles scanning on behalf of the parser.
- The `Tokenizer` requests data off of the scanner, parses it, clasifies
it, and sends it to an `EventHandler`. It is a *recursive descent parser.*
- The `EventHandler` receives notifications and data for each specific
semantic event that occurs during tokenization.
- The `DOMBuilder` is an `EventHandler` that listens for tokenizing
events and builds a document tree (`DOMDocument`) based on the events.
### Serializer Design
The serializer takes a data structure (the `DOMDocument`) and transforms
it into a character representation -- an HTML5 document.
The serializer is broken into three parts:
- The `OutputRules` contain the rules to turn DOM elements into strings. The
rules are an implementation of the interface `RulesInterface` allowing for
different rule sets to be used.
- The `Traverser`, which is a special-purpose tree walker. It visits
each node node in the tree and uses the `OutputRules` to transform the node
into a string.
- `HTML5` manages the `Traverser` and stores the resultant data
in the correct place.
The serializer (`save()`, `saveHTML()`) follows the
[section 8.9 of the HTML 5.0 spec](http://www.w3.org/TR/2012/CR-html5-20121217/syntax.html#serializing-html-fragments).
So tags are serialized according to these rules:
- A tag with children: &lt;foo&gt;CHILDREN&lt;/foo&gt;
- A tag that cannot have content: &lt;foo&gt; (no closing tag)
- A tag that could have content, but doesn't: &lt;foo&gt;&lt;/foo&gt;
## Known Issues (Or, Things We Designed Against the Spec)
Please check the issue queue for a full list, but the following are
issues known issues that are not presently on the roadmap:
- Namespaces: HTML5 only [supports a selected list of namespaces](http://www.w3.org/TR/html5/infrastructure.html#namespaces)
and they do not operate in the same way as XML namespaces. A `:` has no special
meaning.
By default the parser does not support XML style namespaces via `:`;
to enable the XML namespaces see the [XML Namespaces section](#xml-namespaces)
- Scripts: This parser does not contain a JavaScript or a CSS
interpreter. While one may be supplied, not all features will be
supported.
- Rentrance: The current parser is not re-entrant. (Thus you can't pause
the parser to modify the HTML string mid-parse.)
- Validation: The current tree builder is **not** a validating parser.
While it will correct some HTML, it does not check that the HTML
conforms to the standard. (Should you wish, you can build a validating
parser by extending DOMTree or building your own EventHandler
implementation.)
* There is limited support for insertion modes.
* Some autocorrection is done automatically.
* Per the spec, many legacy tags are admitted and correctly handled,
even though they are technically not part of HTML5.
- Attribute names and values: Due to the implementation details of the
PHP implementation of DOM, attribute names that do not follow the
XML 1.0 standard are not inserted into the DOM. (Effectively, they
are ignored.) If you've got a clever fix for this, jump in!
- Processor Instructions: The HTML5 spec does not allow processor
instructions. We do. Since this is a server-side library, we think
this is useful. And that means, dear reader, that in some cases you
can parse the HTML from a mixed PHP/HTML document. This, however,
is an incidental feature, not a core feature.
- HTML manifests: Unsupported.
- PLAINTEXT: Unsupported.
- Adoption Agency Algorithm: Not yet implemented. (8.2.5.4.7)
## XML Namespaces
To use XML style namespaces you have to configure well the main `HTML5` instance.
```php
use Masterminds\HTML5;
$html = new HTML5(array(
"xmlNamespaces" => true
));
$dom = $html->loadHTML('<t:tag xmlns:t="http://www.example.com"/>');
$dom->documentElement->namespaceURI; // http://www.example.com
```
You can also add some default prefixes that will not require the namespace declaration,
but its elements will be namespaced.
```php
use Masterminds\HTML5;
$html = new HTML5(array(
"implicitNamespaces"=>array(
"t"=>"http://www.example.com"
)
));
$dom = $html->loadHTML('<t:tag/>');
$dom->documentElement->namespaceURI; // http://www.example.com
```
## Thanks to...
The dedicated (and patient) contributors of patches small and large,
who have already made this library better.See the CREDITS file for
a list of contributors.
We owe a huge debt of gratitude to the original authors of html5lib.
While not much of the original parser remains, we learned a lot from
reading the html5lib library. And some pieces remain here. In
particular, much of the UTF-8 and Unicode handling is derived from the
html5lib project.
## License
This software is released under the MIT license. The original html5lib
library was also released under the MIT license.
See LICENSE.txt
Certain files contain copyright assertions by specific individuals
involved with html5lib. Those have been retained where appropriate.

153
vendor/masterminds/html5/RELEASE.md vendored Normal file
View File

@@ -0,0 +1,153 @@
# Release Notes
2.7.5 (2021-07-01)
- #204: Travis: Enable tests on PHP 8.0
- #207: Fix PHP 8.1 deprecations
2.7.4 (2020-10-01)
- #191: Fix travisci build
- #195: Add .gitattributes file with export-ignore rules
- #194: Fix query parameter parsed as character entity
2.7.3 (2020-07-05)
- #190: mitigate cyclic reference between output rules and the traverser objects
2.7.2 (2020-07-01)
- #187: Fixed memory leak in HTML5::saveHTML()
- #186: Add special case for end tag </br>
2.7.1 (2020-06-14)
- #171: add PHP 7.4 job
- #178: Prevent infinite loop on un-terminated entity declaration at EOF
2.7.0 (2019-07-25)
- #164: Drop HHVM support
- #168: Set default encoding in the DOMDocument object
2.6.0 (2019-03-10)
- #163: Allow to pass a charset to the Scanner
2.5.0 (2018-12-27)
- #162, #161, #155, #154, #153, #151: big performance improvements
- #156: fixed typos
- #160: adopt and enforce code style
- #159: remove deprecated php unit base test case
- #150: backport changes from old master branch
2.4.0 (2018-11-17)
- #148: Improve performance by moving sequence matching
- #147: Improve the Tokenizer performance
- #146: Improve performance by relying on a native string instead of InputStream
- #144: Add DOM extension in composer.json
- #145: Add more extensions on composer.json, improve phpdocs and remove dead code
- #143: Remove experimental comment
2.3.1 (2018-10-18)
- #121: Audio is not a block tag (fixed by #141)
- #136: Handle illegal self-closing according to spec (fixed by #137)
- #141: Minor fixes in the README
2.3.0 (2017-09-04)
- #129: image within inline svg breaks system (fixed by #133)
- #131: &sup2; does not work (fixed by #132)
- #134: Improve tokenizer performance by 20% (alternative version of #130 thanks to @MichaelHeerklotz)
- #135: Raw & in attributes
2.2.2 (2016-09-22)
- #116: In XML mode, tags are case sensitive
- #115: Fix PHP Notice in OutputRules
- #112: fix parsing of options of an optgroup
- #111: Adding test for the address tag
2.2.1 (2016-05-10)
- #109: Fixed issue where address tag could be written without closing tag (thanks sylus)
2.2.0 (2016-04-11)
- #105: Enable composer cache (for CI/CD)
- #100: Use mb_substitute_character inset of ini_set for environments where ini_set is disable (e.g., shared hosting)
- #98: Allow link, meta, style tags in noscript tags
- #96: Fixed xml:href on svgs that use the "use" breaking
- #94: Counting UTF8 characters performance improvement
- #93: Use newer version of coveralls package
- #90: Remove duplicate test
- #87: Allow multiple root nodes
2.1.2 (2015-06-07)
- #82: Support for PHP7
- #84: Improved boolean attribute handling
2.1.1 (2015-03-23)
- #78: Fixes bug where unmatched entity like string drops everything after &.
2.1.0 (2015-02-01)
- #74: Added `disable_html_ns` and `target_doc` dom parsing options
- Unified option names
- #73: Fixed alphabet, &szlig; now can be detected
- #75 and #76: Allow whitespace in RCDATA tags
- #77: Fixed parsing blunder for json embeds
- #72: Add options to HTML methods
2.0.2 (2014-12-17)
- #50: empty document handling
- #63: tags with strange capitalization
- #65: dashes and underscores as allowed characters in tag names
- #68: Fixed issue with non-inline elements inside inline containers
2.0.1 (2014-09-23)
- #59: Fixed issue parsing some fragments.
- #56: Incorrectly saw 0 as empty string
- Sami as new documentation generator
2.0.0 (2014-07-28)
- #53: Improved boolean attributes handling
- #52: Facebook HHVM compatibility
- #48: Adopted PSR-2 as coding standard
- #47: Moved everything to Masterminds namespace
- #45: Added custom namespaces
- #44: Added support to XML-style namespaces
- #37: Refactored HTML5 class removing static methods
1.0.5 (2014-06-10)
- #38: Set the dev-master branch as the 1.0.x branch for composer (goetas)
- #34: Tests use PSR-4 for autoloading. (goetas)
- #40, #41: Fix entity handling in RCDATA sections. (KitaitiMakoto)
- #32: Fixed issue where wharacter references were being incorrectly encoded in style tags.
1.0.4 (2014-04-29)
- #30/#31 Don't throw an exception for invalid tag names.
1.0.3 (2014-02-28)
- #23 and #29: Ignore attributes with illegal chars in name for the PHP DOM.
1.0.2 (2014-02-12)
- #23: Handle missing tag close in attribute list.
- #25: Fixed text escaping in the serializer (HTML% 8.3).
- #27: Fixed tests on Windows: changed "\n" -> PHP_EOL.
- #28: Fixed infinite loop for char "&" in unquoted attribute in parser.
- #26: Updated tag name case handling to deal with uppercase usage.
- #24: Newlines and tabs are allowed inside quoted attributes (HTML5 8.2.4).
- Fixed Travis CI testing.
1.0.1 (2013-11-07)
- CDATA encoding is improved. (Non-standard; Issue #19)
- Some parser rules were not returning the new current element. (Issue #20)
- Added, to the README, details on code test coverage and to packagist version.
- Fixed processor instructions.
- Improved test coverage and documentation coverage.
1.0.0 (2013-10-02)
- Initial release.

21
vendor/masterminds/html5/UPGRADING.md vendored Normal file
View File

@@ -0,0 +1,21 @@
From 1.x to 2.x
=================
- All classes uses `Masterminds` namespace.
- All public static methods has been removed from `HTML5` class and the general API to access the HTML5 functionalities has changed.
Before:
$dom = \HTML5::loadHTML('<html>....');
\HTML5::saveHTML($dom);
After:
use Masterminds\HTML5;
$html5 = new HTML5();
$dom = $html5->loadHTML('<html>....');
echo $html5->saveHTML($dom);

26
vendor/masterminds/html5/bin/entities.php vendored Normal file
View File

@@ -0,0 +1,26 @@
<?php
/**
* Fetch the entities.json file and convert to PHP datastructure.
*/
// The URL to the official entities JSON file.
$ENTITIES_URL = 'http://www.w3.org/TR/2012/CR-html5-20121217/entities.json';
$payload = file_get_contents($ENTITIES_URL);
$json = json_decode($payload);
$table = array();
foreach ($json as $name => $obj) {
$sname = substr($name, 1, -1);
$table[$sname] = $obj->characters;
}
echo '<?php
namespace Masterminds\\HTML5;
/** Entity lookup tables. This class is automatically generated. */
class Entities {
public static $byName = ';
var_export($table);
echo ';
}' . PHP_EOL;
//print serialize($table);

42
vendor/masterminds/html5/composer.json vendored Normal file
View File

@@ -0,0 +1,42 @@
{
"name": "masterminds/html5",
"description": "An HTML5 parser and serializer.",
"type": "library",
"homepage": "http://masterminds.github.io/html5-php",
"license": "MIT",
"keywords": ["xml", "html", "html5", "dom", "parser", "serializer", "querypath"],
"authors": [
{
"name": "Matt Butcher",
"email": "technosophos@gmail.com"
},
{
"name": "Matt Farina",
"email": "matt@mattfarina.com"
},
{
"name": "Asmir Mustafic",
"email": "goetas@gmail.com"
}
],
"require" : {
"ext-ctype": "*",
"ext-dom": "*",
"ext-libxml" : "*",
"php" : ">=5.3.0"
},
"require-dev": {
"phpunit/phpunit" : "^4.8.35 || ^5.7.21 || ^6 || ^7"
},
"autoload": {
"psr-4": {"Masterminds\\": "src"}
},
"autoload-dev": {
"psr-4": {"Masterminds\\HTML5\\Tests\\": "test/HTML5"}
},
"extra": {
"branch-alias": {
"dev-master": "2.7-dev"
}
}
}

32
vendor/masterminds/html5/example.php vendored Normal file
View File

@@ -0,0 +1,32 @@
<?php
require 'vendor/autoload.php';
use Masterminds\HTML5;
$html = <<< 'HERE'
<html>
<head>
<title>TEST</title>
<script language="javascript">
if (2 > 1) { alert("Math wins."); }
</script>
</head>
<body id='foo'>
<!-- This space intentionally left blank. -->
<section class="section-a pretty" id="bar1">
<h1>Hello World</h1><p>This is a test of the HTML5 parser.</p>
<hr>
&amp; Nobody nowhere.
</section>
<test xmlns:foo="http://example.com/foo">TEST</test>
<![CDATA[Because we can.]]>
&copy;
</body></html>
HERE;
$html5 = new HTML5();
$dom = $html5->loadHTML($html);
echo "Converting to HTML 5\n";
$html5->save($dom, fopen('php://stdin', 'w'));

8
vendor/masterminds/html5/phpunit.xml.dist vendored Normal file
View File

@@ -0,0 +1,8 @@
<?xml version="1.0" encoding="UTF-8"?>
<phpunit colors="true" bootstrap="vendor/autoload.php">
<testsuites>
<testsuite name="PHPUnit">
<directory>test/HTML5/</directory>
</testsuite>
</testsuites>
</phpunit>

246
vendor/masterminds/html5/src/HTML5.php vendored Normal file
View File

@@ -0,0 +1,246 @@
<?php
namespace Masterminds;
use Masterminds\HTML5\Parser\DOMTreeBuilder;
use Masterminds\HTML5\Parser\Scanner;
use Masterminds\HTML5\Parser\Tokenizer;
use Masterminds\HTML5\Serializer\OutputRules;
use Masterminds\HTML5\Serializer\Traverser;
/**
* This class offers convenience methods for parsing and serializing HTML5.
* It is roughly designed to mirror the \DOMDocument native class.
*/
class HTML5
{
/**
* Global options for the parser and serializer.
*
* @var array
*/
private $defaultOptions = array(
// Whether the serializer should aggressively encode all characters as entities.
'encode_entities' => false,
// Prevents the parser from automatically assigning the HTML5 namespace to the DOM document.
'disable_html_ns' => false,
);
protected $errors = array();
public function __construct(array $defaultOptions = array())
{
$this->defaultOptions = array_merge($this->defaultOptions, $defaultOptions);
}
/**
* Get the current default options.
*
* @return array
*/
public function getOptions()
{
return $this->defaultOptions;
}
/**
* Load and parse an HTML file.
*
* This will apply the HTML5 parser, which is tolerant of many
* varieties of HTML, including XHTML 1, HTML 4, and well-formed HTML
* 3. Note that in these cases, not all of the old data will be
* preserved. For example, XHTML's XML declaration will be removed.
*
* The rules governing parsing are set out in the HTML 5 spec.
*
* @param string|resource $file The path to the file to parse. If this is a resource, it is
* assumed to be an open stream whose pointer is set to the first
* byte of input.
* @param array $options Configuration options when parsing the HTML.
*
* @return \DOMDocument A DOM document. These object type is defined by the libxml
* library, and should have been included with your version of PHP.
*/
public function load($file, array $options = array())
{
// Handle the case where file is a resource.
if (is_resource($file)) {
return $this->parse(stream_get_contents($file), $options);
}
return $this->parse(file_get_contents($file), $options);
}
/**
* Parse a HTML Document from a string.
*
* Take a string of HTML 5 (or earlier) and parse it into a
* DOMDocument.
*
* @param string $string A html5 document as a string.
* @param array $options Configuration options when parsing the HTML.
*
* @return \DOMDocument A DOM document. DOM is part of libxml, which is included with
* almost all distribtions of PHP.
*/
public function loadHTML($string, array $options = array())
{
return $this->parse($string, $options);
}
/**
* Convenience function to load an HTML file.
*
* This is here to provide backwards compatibility with the
* PHP DOM implementation. It simply calls load().
*
* @param string $file The path to the file to parse. If this is a resource, it is
* assumed to be an open stream whose pointer is set to the first
* byte of input.
* @param array $options Configuration options when parsing the HTML.
*
* @return \DOMDocument A DOM document. These object type is defined by the libxml
* library, and should have been included with your version of PHP.
*/
public function loadHTMLFile($file, array $options = array())
{
return $this->load($file, $options);
}
/**
* Parse a HTML fragment from a string.
*
* @param string $string the HTML5 fragment as a string
* @param array $options Configuration options when parsing the HTML
*
* @return \DOMDocumentFragment A DOM fragment. The DOM is part of libxml, which is included with
* almost all distributions of PHP.
*/
public function loadHTMLFragment($string, array $options = array())
{
return $this->parseFragment($string, $options);
}
/**
* Return all errors encountered into parsing phase.
*
* @return array
*/
public function getErrors()
{
return $this->errors;
}
/**
* Return true it some errors were encountered into parsing phase.
*
* @return bool
*/
public function hasErrors()
{
return count($this->errors) > 0;
}
/**
* Parse an input string.
*
* @param string $input
* @param array $options
*
* @return \DOMDocument
*/
public function parse($input, array $options = array())
{
$this->errors = array();
$options = array_merge($this->defaultOptions, $options);
$events = new DOMTreeBuilder(false, $options);
$scanner = new Scanner($input, !empty($options['encoding']) ? $options['encoding'] : 'UTF-8');
$parser = new Tokenizer($scanner, $events, !empty($options['xmlNamespaces']) ? Tokenizer::CONFORMANT_XML : Tokenizer::CONFORMANT_HTML);
$parser->parse();
$this->errors = $events->getErrors();
return $events->document();
}
/**
* Parse an input stream where the stream is a fragment.
*
* Lower-level loading function. This requires an input stream instead
* of a string, file, or resource.
*
* @param string $input The input data to parse in the form of a string.
* @param array $options An array of options.
*
* @return \DOMDocumentFragment
*/
public function parseFragment($input, array $options = array())
{
$options = array_merge($this->defaultOptions, $options);
$events = new DOMTreeBuilder(true, $options);
$scanner = new Scanner($input, !empty($options['encoding']) ? $options['encoding'] : 'UTF-8');
$parser = new Tokenizer($scanner, $events, !empty($options['xmlNamespaces']) ? Tokenizer::CONFORMANT_XML : Tokenizer::CONFORMANT_HTML);
$parser->parse();
$this->errors = $events->getErrors();
return $events->fragment();
}
/**
* Save a DOM into a given file as HTML5.
*
* @param mixed $dom The DOM to be serialized.
* @param string|resource $file The filename to be written or resource to write to.
* @param array $options Configuration options when serializing the DOM. These include:
* - encode_entities: Text written to the output is escaped by default and not all
* entities are encoded. If this is set to true all entities will be encoded.
* Defaults to false.
*/
public function save($dom, $file, $options = array())
{
$close = true;
if (is_resource($file)) {
$stream = $file;
$close = false;
} else {
$stream = fopen($file, 'wb');
}
$options = array_merge($this->defaultOptions, $options);
$rules = new OutputRules($stream, $options);
$trav = new Traverser($dom, $stream, $rules, $options);
$trav->walk();
/*
* release the traverser to avoid cyclic references and allow PHP to free memory without waiting for gc_collect_cycles
*/
$rules->unsetTraverser();
if ($close) {
fclose($stream);
}
}
/**
* Convert a DOM into an HTML5 string.
*
* @param mixed $dom The DOM to be serialized.
* @param array $options Configuration options when serializing the DOM. These include:
* - encode_entities: Text written to the output is escaped by default and not all
* entities are encoded. If this is set to true all entities will be encoded.
* Defaults to false.
*
* @return string A HTML5 documented generated from the DOM.
*/
public function saveHTML($dom, $options = array())
{
$stream = fopen('php://temp', 'wb');
$this->save($dom, $stream, array_merge($this->defaultOptions, $options));
$html = stream_get_contents($stream, -1, 0);
fclose($stream);
return $html;
}
}

619
vendor/masterminds/html5/src/HTML5/Elements.php vendored Normal file
View File

@@ -0,0 +1,619 @@
<?php
/**
* Provide general element functions.
*/
namespace Masterminds\HTML5;
/**
* This class provides general information about HTML5 elements,
* including syntactic and semantic issues.
* Parsers and serializers can
* use this class as a reference point for information about the rules
* of various HTML5 elements.
*
* @todo consider using a bitmask table lookup. There is enough overlap in
* naming that this could significantly shrink the size and maybe make it
* faster. See the Go teams implementation at https://code.google.com/p/go/source/browse/html/atom.
*/
class Elements
{
/**
* Indicates an element is described in the specification.
*/
const KNOWN_ELEMENT = 1;
// From section 8.1.2: "script", "style"
// From 8.2.5.4.7 ("in body" insertion mode): "noembed"
// From 8.4 "style", "xmp", "iframe", "noembed", "noframes"
/**
* Indicates the contained text should be processed as raw text.
*/
const TEXT_RAW = 2;
// From section 8.1.2: "textarea", "title"
/**
* Indicates the contained text should be processed as RCDATA.
*/
const TEXT_RCDATA = 4;
/**
* Indicates the tag cannot have content.
*/
const VOID_TAG = 8;
// "address", "article", "aside", "blockquote", "center", "details", "dialog", "dir", "div", "dl",
// "fieldset", "figcaption", "figure", "footer", "header", "hgroup", "menu",
// "nav", "ol", "p", "section", "summary", "ul"
// "h1", "h2", "h3", "h4", "h5", "h6"
// "pre", "listing"
// "form"
// "plaintext"
/**
* Indicates that if a previous event is for a P tag, that element
* should be considered closed.
*/
const AUTOCLOSE_P = 16;
/**
* Indicates that the text inside is plaintext (pre).
*/
const TEXT_PLAINTEXT = 32;
// See https://developer.mozilla.org/en-US/docs/HTML/Block-level_elements
/**
* Indicates that the tag is a block.
*/
const BLOCK_TAG = 64;
/**
* Indicates that the tag allows only inline elements as child nodes.
*/
const BLOCK_ONLY_INLINE = 128;
/**
* The HTML5 elements as defined in http://dev.w3.org/html5/markup/elements.html.
*
* @var array
*/
public static $html5 = array(
'a' => 1,
'abbr' => 1,
'address' => 65, // NORMAL | BLOCK_TAG
'area' => 9, // NORMAL | VOID_TAG
'article' => 81, // NORMAL | AUTOCLOSE_P | BLOCK_TAG
'aside' => 81, // NORMAL | AUTOCLOSE_P | BLOCK_TAG
'audio' => 1, // NORMAL
'b' => 1,
'base' => 9, // NORMAL | VOID_TAG
'bdi' => 1,
'bdo' => 1,
'blockquote' => 81, // NORMAL | AUTOCLOSE_P | BLOCK_TAG
'body' => 1,
'br' => 9, // NORMAL | VOID_TAG
'button' => 1,
'canvas' => 65, // NORMAL | BLOCK_TAG
'caption' => 1,
'cite' => 1,
'code' => 1,
'col' => 9, // NORMAL | VOID_TAG
'colgroup' => 1,
'command' => 9, // NORMAL | VOID_TAG
// "data" => 1, // This is highly experimental and only part of the whatwg spec (not w3c). See https://developer.mozilla.org/en-US/docs/HTML/Element/data
'datalist' => 1,
'dd' => 65, // NORMAL | BLOCK_TAG
'del' => 1,
'details' => 17, // NORMAL | AUTOCLOSE_P,
'dfn' => 1,
'dialog' => 17, // NORMAL | AUTOCLOSE_P,
'div' => 81, // NORMAL | AUTOCLOSE_P | BLOCK_TAG
'dl' => 81, // NORMAL | AUTOCLOSE_P | BLOCK_TAG
'dt' => 1,
'em' => 1,
'embed' => 9, // NORMAL | VOID_TAG
'fieldset' => 81, // NORMAL | AUTOCLOSE_P | BLOCK_TAG
'figcaption' => 81, // NORMAL | AUTOCLOSE_P | BLOCK_TAG
'figure' => 81, // NORMAL | AUTOCLOSE_P | BLOCK_TAG
'footer' => 81, // NORMAL | AUTOCLOSE_P | BLOCK_TAG
'form' => 81, // NORMAL | AUTOCLOSE_P | BLOCK_TAG
'h1' => 81, // NORMAL | AUTOCLOSE_P | BLOCK_TAG
'h2' => 81, // NORMAL | AUTOCLOSE_P | BLOCK_TAG
'h3' => 81, // NORMAL | AUTOCLOSE_P | BLOCK_TAG
'h4' => 81, // NORMAL | AUTOCLOSE_P | BLOCK_TAG
'h5' => 81, // NORMAL | AUTOCLOSE_P | BLOCK_TAG
'h6' => 81, // NORMAL | AUTOCLOSE_P | BLOCK_TAG
'head' => 1,
'header' => 81, // NORMAL | AUTOCLOSE_P | BLOCK_TAG
'hgroup' => 81, // NORMAL | AUTOCLOSE_P | BLOCK_TAG
'hr' => 73, // NORMAL | VOID_TAG
'html' => 1,
'i' => 1,
'iframe' => 3, // NORMAL | TEXT_RAW
'img' => 9, // NORMAL | VOID_TAG
'input' => 9, // NORMAL | VOID_TAG
'kbd' => 1,
'ins' => 1,
'keygen' => 9, // NORMAL | VOID_TAG
'label' => 1,
'legend' => 1,
'li' => 1,
'link' => 9, // NORMAL | VOID_TAG
'map' => 1,
'mark' => 1,
'menu' => 17, // NORMAL | AUTOCLOSE_P,
'meta' => 9, // NORMAL | VOID_TAG
'meter' => 1,
'nav' => 17, // NORMAL | AUTOCLOSE_P,
'noscript' => 65, // NORMAL | BLOCK_TAG
'object' => 1,
'ol' => 81, // NORMAL | AUTOCLOSE_P | BLOCK_TAG
'optgroup' => 1,
'option' => 1,
'output' => 65, // NORMAL | BLOCK_TAG
'p' => 209, // NORMAL | AUTOCLOSE_P | BLOCK_TAG | BLOCK_ONLY_INLINE
'param' => 9, // NORMAL | VOID_TAG
'pre' => 81, // NORMAL | AUTOCLOSE_P | BLOCK_TAG
'progress' => 1,
'q' => 1,
'rp' => 1,
'rt' => 1,
'ruby' => 1,
's' => 1,
'samp' => 1,
'script' => 3, // NORMAL | TEXT_RAW
'section' => 81, // NORMAL | AUTOCLOSE_P | BLOCK_TAG
'select' => 1,
'small' => 1,
'source' => 9, // NORMAL | VOID_TAG
'span' => 1,
'strong' => 1,
'style' => 3, // NORMAL | TEXT_RAW
'sub' => 1,
'summary' => 17, // NORMAL | AUTOCLOSE_P,
'sup' => 1,
'table' => 65, // NORMAL | BLOCK_TAG
'tbody' => 1,
'td' => 1,
'textarea' => 5, // NORMAL | TEXT_RCDATA
'tfoot' => 65, // NORMAL | BLOCK_TAG
'th' => 1,
'thead' => 1,
'time' => 1,
'title' => 5, // NORMAL | TEXT_RCDATA
'tr' => 1,
'track' => 9, // NORMAL | VOID_TAG
'u' => 1,
'ul' => 81, // NORMAL | AUTOCLOSE_P | BLOCK_TAG
'var' => 1,
'video' => 65, // NORMAL | BLOCK_TAG
'wbr' => 9, // NORMAL | VOID_TAG
// Legacy?
'basefont' => 8, // VOID_TAG
'bgsound' => 8, // VOID_TAG
'noframes' => 2, // RAW_TEXT
'frame' => 9, // NORMAL | VOID_TAG
'frameset' => 1,
'center' => 16,
'dir' => 16,
'listing' => 16, // AUTOCLOSE_P
'plaintext' => 48, // AUTOCLOSE_P | TEXT_PLAINTEXT
'applet' => 0,
'marquee' => 0,
'isindex' => 8, // VOID_TAG
'xmp' => 20, // AUTOCLOSE_P | VOID_TAG | RAW_TEXT
'noembed' => 2, // RAW_TEXT
);
/**
* The MathML elements.
* See http://www.w3.org/wiki/MathML/Elements.
*
* In our case we are only concerned with presentation MathML and not content
* MathML. There is a nice list of this subset at https://developer.mozilla.org/en-US/docs/MathML/Element.
*
* @var array
*/
public static $mathml = array(
'maction' => 1,
'maligngroup' => 1,
'malignmark' => 1,
'math' => 1,
'menclose' => 1,
'merror' => 1,
'mfenced' => 1,
'mfrac' => 1,
'mglyph' => 1,
'mi' => 1,
'mlabeledtr' => 1,
'mlongdiv' => 1,
'mmultiscripts' => 1,
'mn' => 1,
'mo' => 1,
'mover' => 1,
'mpadded' => 1,
'mphantom' => 1,
'mroot' => 1,
'mrow' => 1,
'ms' => 1,
'mscarries' => 1,
'mscarry' => 1,
'msgroup' => 1,
'msline' => 1,
'mspace' => 1,
'msqrt' => 1,
'msrow' => 1,
'mstack' => 1,
'mstyle' => 1,
'msub' => 1,
'msup' => 1,
'msubsup' => 1,
'mtable' => 1,
'mtd' => 1,
'mtext' => 1,
'mtr' => 1,
'munder' => 1,
'munderover' => 1,
);
/**
* The svg elements.
*
* The Mozilla documentation has a good list at https://developer.mozilla.org/en-US/docs/SVG/Element.
* The w3c list appears to be lacking in some areas like filter effect elements.
* That list can be found at http://www.w3.org/wiki/SVG/Elements.
*
* Note, FireFox appears to do a better job rendering filter effects than chrome.
* While they are in the spec I'm not sure how widely implemented they are.
*
* @var array
*/
public static $svg = array(
'a' => 1,
'altGlyph' => 1,
'altGlyphDef' => 1,
'altGlyphItem' => 1,
'animate' => 1,
'animateColor' => 1,
'animateMotion' => 1,
'animateTransform' => 1,
'circle' => 1,
'clipPath' => 1,
'color-profile' => 1,
'cursor' => 1,
'defs' => 1,
'desc' => 1,
'ellipse' => 1,
'feBlend' => 1,
'feColorMatrix' => 1,
'feComponentTransfer' => 1,
'feComposite' => 1,
'feConvolveMatrix' => 1,
'feDiffuseLighting' => 1,
'feDisplacementMap' => 1,
'feDistantLight' => 1,
'feFlood' => 1,
'feFuncA' => 1,
'feFuncB' => 1,
'feFuncG' => 1,
'feFuncR' => 1,
'feGaussianBlur' => 1,
'feImage' => 1,
'feMerge' => 1,
'feMergeNode' => 1,
'feMorphology' => 1,
'feOffset' => 1,
'fePointLight' => 1,
'feSpecularLighting' => 1,
'feSpotLight' => 1,
'feTile' => 1,
'feTurbulence' => 1,
'filter' => 1,
'font' => 1,
'font-face' => 1,
'font-face-format' => 1,
'font-face-name' => 1,
'font-face-src' => 1,
'font-face-uri' => 1,
'foreignObject' => 1,
'g' => 1,
'glyph' => 1,
'glyphRef' => 1,
'hkern' => 1,
'image' => 1,
'line' => 1,
'linearGradient' => 1,
'marker' => 1,
'mask' => 1,
'metadata' => 1,
'missing-glyph' => 1,
'mpath' => 1,
'path' => 1,
'pattern' => 1,
'polygon' => 1,
'polyline' => 1,
'radialGradient' => 1,
'rect' => 1,
'script' => 3, // NORMAL | RAW_TEXT
'set' => 1,
'stop' => 1,
'style' => 3, // NORMAL | RAW_TEXT
'svg' => 1,
'switch' => 1,
'symbol' => 1,
'text' => 1,
'textPath' => 1,
'title' => 1,
'tref' => 1,
'tspan' => 1,
'use' => 1,
'view' => 1,
'vkern' => 1,
);
/**
* Some attributes in SVG are case sensitive.
*
* This map contains key/value pairs with the key as the lowercase attribute
* name and the value with the correct casing.
*/
public static $svgCaseSensitiveAttributeMap = array(
'attributename' => 'attributeName',
'attributetype' => 'attributeType',
'basefrequency' => 'baseFrequency',
'baseprofile' => 'baseProfile',
'calcmode' => 'calcMode',
'clippathunits' => 'clipPathUnits',
'contentscripttype' => 'contentScriptType',
'contentstyletype' => 'contentStyleType',
'diffuseconstant' => 'diffuseConstant',
'edgemode' => 'edgeMode',
'externalresourcesrequired' => 'externalResourcesRequired',
'filterres' => 'filterRes',
'filterunits' => 'filterUnits',
'glyphref' => 'glyphRef',
'gradienttransform' => 'gradientTransform',
'gradientunits' => 'gradientUnits',
'kernelmatrix' => 'kernelMatrix',
'kernelunitlength' => 'kernelUnitLength',
'keypoints' => 'keyPoints',
'keysplines' => 'keySplines',
'keytimes' => 'keyTimes',
'lengthadjust' => 'lengthAdjust',
'limitingconeangle' => 'limitingConeAngle',
'markerheight' => 'markerHeight',
'markerunits' => 'markerUnits',
'markerwidth' => 'markerWidth',
'maskcontentunits' => 'maskContentUnits',
'maskunits' => 'maskUnits',
'numoctaves' => 'numOctaves',
'pathlength' => 'pathLength',
'patterncontentunits' => 'patternContentUnits',
'patterntransform' => 'patternTransform',
'patternunits' => 'patternUnits',
'pointsatx' => 'pointsAtX',
'pointsaty' => 'pointsAtY',
'pointsatz' => 'pointsAtZ',
'preservealpha' => 'preserveAlpha',
'preserveaspectratio' => 'preserveAspectRatio',
'primitiveunits' => 'primitiveUnits',
'refx' => 'refX',
'refy' => 'refY',
'repeatcount' => 'repeatCount',
'repeatdur' => 'repeatDur',
'requiredextensions' => 'requiredExtensions',
'requiredfeatures' => 'requiredFeatures',
'specularconstant' => 'specularConstant',
'specularexponent' => 'specularExponent',
'spreadmethod' => 'spreadMethod',
'startoffset' => 'startOffset',
'stddeviation' => 'stdDeviation',
'stitchtiles' => 'stitchTiles',
'surfacescale' => 'surfaceScale',
'systemlanguage' => 'systemLanguage',
'tablevalues' => 'tableValues',
'targetx' => 'targetX',
'targety' => 'targetY',
'textlength' => 'textLength',
'viewbox' => 'viewBox',
'viewtarget' => 'viewTarget',
'xchannelselector' => 'xChannelSelector',
'ychannelselector' => 'yChannelSelector',
'zoomandpan' => 'zoomAndPan',
);
/**
* Some SVG elements are case sensitive.
* This map contains these.
*
* The map contains key/value store of the name is lowercase as the keys and
* the correct casing as the value.
*/
public static $svgCaseSensitiveElementMap = array(
'altglyph' => 'altGlyph',
'altglyphdef' => 'altGlyphDef',
'altglyphitem' => 'altGlyphItem',
'animatecolor' => 'animateColor',
'animatemotion' => 'animateMotion',
'animatetransform' => 'animateTransform',
'clippath' => 'clipPath',
'feblend' => 'feBlend',
'fecolormatrix' => 'feColorMatrix',
'fecomponenttransfer' => 'feComponentTransfer',
'fecomposite' => 'feComposite',
'feconvolvematrix' => 'feConvolveMatrix',
'fediffuselighting' => 'feDiffuseLighting',
'fedisplacementmap' => 'feDisplacementMap',
'fedistantlight' => 'feDistantLight',
'feflood' => 'feFlood',
'fefunca' => 'feFuncA',
'fefuncb' => 'feFuncB',
'fefuncg' => 'feFuncG',
'fefuncr' => 'feFuncR',
'fegaussianblur' => 'feGaussianBlur',
'feimage' => 'feImage',
'femerge' => 'feMerge',
'femergenode' => 'feMergeNode',
'femorphology' => 'feMorphology',
'feoffset' => 'feOffset',
'fepointlight' => 'fePointLight',
'fespecularlighting' => 'feSpecularLighting',
'fespotlight' => 'feSpotLight',
'fetile' => 'feTile',
'feturbulence' => 'feTurbulence',
'foreignobject' => 'foreignObject',
'glyphref' => 'glyphRef',
'lineargradient' => 'linearGradient',
'radialgradient' => 'radialGradient',
'textpath' => 'textPath',
);
/**
* Check whether the given element meets the given criterion.
*
* Example:
*
* Elements::isA('script', Elements::TEXT_RAW); // Returns true.
*
* Elements::isA('script', Elements::TEXT_RCDATA); // Returns false.
*
* @param string $name The element name.
* @param int $mask One of the constants on this class.
*
* @return bool true if the element matches the mask, false otherwise.
*/
public static function isA($name, $mask)
{
return (static::element($name) & $mask) === $mask;
}
/**
* Test if an element is a valid html5 element.
*
* @param string $name The name of the element.
*
* @return bool true if a html5 element and false otherwise.
*/
public static function isHtml5Element($name)
{
// html5 element names are case insensitive. Forcing lowercase for the check.
// Do we need this check or will all data passed here already be lowercase?
return isset(static::$html5[strtolower($name)]);
}
/**
* Test if an element name is a valid MathML presentation element.
*
* @param string $name The name of the element.
*
* @return bool true if a MathML name and false otherwise.
*/
public static function isMathMLElement($name)
{
// MathML is case-sensitive unlike html5 elements.
return isset(static::$mathml[$name]);
}
/**
* Test if an element is a valid SVG element.
*
* @param string $name The name of the element.
*
* @return bool true if a SVG element and false otherise.
*/
public static function isSvgElement($name)
{
// SVG is case-sensitive unlike html5 elements.
return isset(static::$svg[$name]);
}
/**
* Is an element name valid in an html5 document.
* This includes html5 elements along with other allowed embedded content
* such as svg and mathml.
*
* @param string $name The name of the element.
*
* @return bool true if valid and false otherwise.
*/
public static function isElement($name)
{
return static::isHtml5Element($name) || static::isMathMLElement($name) || static::isSvgElement($name);
}
/**
* Get the element mask for the given element name.
*
* @param string $name The name of the element.
*
* @return int the element mask.
*/
public static function element($name)
{
if (isset(static::$html5[$name])) {
return static::$html5[$name];
}
if (isset(static::$svg[$name])) {
return static::$svg[$name];
}
if (isset(static::$mathml[$name])) {
return static::$mathml[$name];
}
return 0;
}
/**
* Normalize a SVG element name to its proper case and form.
*
* @param string $name The name of the element.
*
* @return string the normalized form of the element name.
*/
public static function normalizeSvgElement($name)
{
$name = strtolower($name);
if (isset(static::$svgCaseSensitiveElementMap[$name])) {
$name = static::$svgCaseSensitiveElementMap[$name];
}
return $name;
}
/**
* Normalize a SVG attribute name to its proper case and form.
*
* @param string $name The name of the attribute.
*
* @return string The normalized form of the attribute name.
*/
public static function normalizeSvgAttribute($name)
{
$name = strtolower($name);
if (isset(static::$svgCaseSensitiveAttributeMap[$name])) {
$name = static::$svgCaseSensitiveAttributeMap[$name];
}
return $name;
}
/**
* Normalize a MathML attribute name to its proper case and form.
* Note, all MathML element names are lowercase.
*
* @param string $name The name of the attribute.
*
* @return string The normalized form of the attribute name.
*/
public static function normalizeMathMlAttribute($name)
{
$name = strtolower($name);
// Only one attribute has a mixed case form for MathML.
if ('definitionurl' === $name) {
$name = 'definitionURL';
}
return $name;
}
}

2236
vendor/masterminds/html5/src/HTML5/Entities.php vendored Normal file
View File

File diff suppressed because it is too large Load Diff

10
vendor/masterminds/html5/src/HTML5/Exception.php vendored Normal file
View File

@@ -0,0 +1,10 @@
<?php
namespace Masterminds\HTML5;
/**
* The base exception for the HTML5 project.
*/
class Exception extends \Exception
{
}

41
vendor/masterminds/html5/src/HTML5/InstructionProcessor.php vendored Normal file
View File

@@ -0,0 +1,41 @@
<?php
/**
* A handler for processor instructions.
*/
namespace Masterminds\HTML5;
/**
* Provide an processor to handle embedded instructions.
*
* XML defines a mechanism for inserting instructions (like PHP) into a
* document. These are called "Processor Instructions." The HTML5 parser
* provides an opportunity to handle these processor instructions during
* the tree-building phase (before the DOM is constructed), which makes
* it possible to alter the document as it is being created.
*
* One could, for example, use this mechanism to execute well-formed PHP
* code embedded inside of an HTML5 document.
*/
interface InstructionProcessor
{
/**
* Process an individual processing instruction.
*
* The process() function is responsible for doing the following:
* - Determining whether $name is an instruction type it can handle.
* - Determining what to do with the data passed in.
* - Making any subsequent modifications to the DOM by modifying the
* DOMElement or its attached DOM tree.
*
* @param \DOMElement $element The parent element for the current processing instruction.
* @param string $name The instruction's name. E.g. `&lt;?php` has the name `php`.
* @param string $data All of the data between the opening and closing PI marks.
*
* @return \DOMElement The element that should be considered "Current". This may just be
* the element passed in, but if the processor added more elements,
* it may choose to reset the current element to one of the elements
* it created. (When in doubt, return the element passed in.)
*/
public function process(\DOMElement $element, $name, $data);
}

61
vendor/masterminds/html5/src/HTML5/Parser/CharacterReference.php vendored Normal file
View File

@@ -0,0 +1,61 @@
<?php
namespace Masterminds\HTML5\Parser;
use Masterminds\HTML5\Entities;
/**
* Manage entity references.
*
* This is a simple resolver for HTML5 character reference entitites. See Entities for the list of supported entities.
*/
class CharacterReference
{
protected static $numeric_mask = array(
0x0,
0x2FFFF,
0,
0xFFFF,
);
/**
* Given a name (e.g. 'amp'), lookup the UTF-8 character ('&').
*
* @param string $name The name to look up.
*
* @return string The character sequence. In UTF-8 this may be more than one byte.
*/
public static function lookupName($name)
{
// Do we really want to return NULL here? or FFFD
return isset(Entities::$byName[$name]) ? Entities::$byName[$name] : null;
}
/**
* Given a decimal number, return the UTF-8 character.
*
* @param $int
*
* @return false|string|string[]|null
*/
public static function lookupDecimal($int)
{
$entity = '&#' . $int . ';';
// UNTESTED: This may fail on some planes. Couldn't find full documentation
// on the value of the mask array.
return mb_decode_numericentity($entity, static::$numeric_mask, 'utf-8');
}
/**
* Given a hexidecimal number, return the UTF-8 character.
*
* @param $hexdec
*
* @return false|string|string[]|null
*/
public static function lookupHex($hexdec)
{
return static::lookupDecimal(hexdec($hexdec));
}
}

705
vendor/masterminds/html5/src/HTML5/Parser/DOMTreeBuilder.php vendored Normal file
View File

@@ -0,0 +1,705 @@
<?php
namespace Masterminds\HTML5\Parser;
use Masterminds\HTML5\Elements;
use Masterminds\HTML5\InstructionProcessor;
/**
* Create an HTML5 DOM tree from events.
*
* This attempts to create a DOM from events emitted by a parser. This
* attempts (but does not guarantee) to up-convert older HTML documents
* to HTML5. It does this by applying HTML5's rules, but it will not
* change the architecture of the document itself.
*
* Many of the error correction and quirks features suggested in the specification
* are implemented herein; however, not all of them are. Since we do not
* assume a graphical user agent, no presentation-specific logic is conducted
* during tree building.
*
* FIXME: The present tree builder does not exactly follow the state machine rules
* for insert modes as outlined in the HTML5 spec. The processor needs to be
* re-written to accomodate this. See, for example, the Go language HTML5
* parser.
*/
class DOMTreeBuilder implements EventHandler
{
/**
* Defined in http://www.w3.org/TR/html51/infrastructure.html#html-namespace-0.
*/
const NAMESPACE_HTML = 'http://www.w3.org/1999/xhtml';
const NAMESPACE_MATHML = 'http://www.w3.org/1998/Math/MathML';
const NAMESPACE_SVG = 'http://www.w3.org/2000/svg';
const NAMESPACE_XLINK = 'http://www.w3.org/1999/xlink';
const NAMESPACE_XML = 'http://www.w3.org/XML/1998/namespace';
const NAMESPACE_XMLNS = 'http://www.w3.org/2000/xmlns/';
const OPT_DISABLE_HTML_NS = 'disable_html_ns';
const OPT_TARGET_DOC = 'target_document';
const OPT_IMPLICIT_NS = 'implicit_namespaces';
/**
* Holds the HTML5 element names that causes a namespace switch.
*
* @var array
*/
protected $nsRoots = array(
'html' => self::NAMESPACE_HTML,
'svg' => self::NAMESPACE_SVG,
'math' => self::NAMESPACE_MATHML,
);
/**
* Holds the always available namespaces (which does not require the XMLNS declaration).
*
* @var array
*/
protected $implicitNamespaces = array(
'xml' => self::NAMESPACE_XML,
'xmlns' => self::NAMESPACE_XMLNS,
'xlink' => self::NAMESPACE_XLINK,
);
/**
* Holds a stack of currently active namespaces.
*
* @var array
*/
protected $nsStack = array();
/**
* Holds the number of namespaces declared by a node.
*
* @var array
*/
protected $pushes = array();
/**
* Defined in 8.2.5.
*/
const IM_INITIAL = 0;
const IM_BEFORE_HTML = 1;
const IM_BEFORE_HEAD = 2;
const IM_IN_HEAD = 3;
const IM_IN_HEAD_NOSCRIPT = 4;
const IM_AFTER_HEAD = 5;
const IM_IN_BODY = 6;
const IM_TEXT = 7;
const IM_IN_TABLE = 8;
const IM_IN_TABLE_TEXT = 9;
const IM_IN_CAPTION = 10;
const IM_IN_COLUMN_GROUP = 11;
const IM_IN_TABLE_BODY = 12;
const IM_IN_ROW = 13;
const IM_IN_CELL = 14;
const IM_IN_SELECT = 15;
const IM_IN_SELECT_IN_TABLE = 16;
const IM_AFTER_BODY = 17;
const IM_IN_FRAMESET = 18;
const IM_AFTER_FRAMESET = 19;
const IM_AFTER_AFTER_BODY = 20;
const IM_AFTER_AFTER_FRAMESET = 21;
const IM_IN_SVG = 22;
const IM_IN_MATHML = 23;
protected $options = array();
protected $stack = array();
protected $current; // Pointer in the tag hierarchy.
protected $rules;
protected $doc;
protected $frag;
protected $processor;
protected $insertMode = 0;
/**
* Track if we are in an element that allows only inline child nodes.
*
* @var string|null
*/
protected $onlyInline;
/**
* Quirks mode is enabled by default.
* Any document that is missing the DT will be considered to be in quirks mode.
*/
protected $quirks = true;
protected $errors = array();
public function __construct($isFragment = false, array $options = array())
{
$this->options = $options;
if (isset($options[self::OPT_TARGET_DOC])) {
$this->doc = $options[self::OPT_TARGET_DOC];
} else {
$impl = new \DOMImplementation();
// XXX:
// Create the doctype. For now, we are always creating HTML5
// documents, and attempting to up-convert any older DTDs to HTML5.
$dt = $impl->createDocumentType('html');
// $this->doc = \DOMImplementation::createDocument(NULL, 'html', $dt);
$this->doc = $impl->createDocument(null, '', $dt);
$this->doc->encoding = !empty($options['encoding']) ? $options['encoding'] : 'UTF-8';
}
$this->errors = array();
$this->current = $this->doc; // ->documentElement;
// Create a rules engine for tags.
$this->rules = new TreeBuildingRules();
$implicitNS = array();
if (isset($this->options[self::OPT_IMPLICIT_NS])) {
$implicitNS = $this->options[self::OPT_IMPLICIT_NS];
} elseif (isset($this->options['implicitNamespaces'])) {
$implicitNS = $this->options['implicitNamespaces'];
}
// Fill $nsStack with the defalut HTML5 namespaces, plus the "implicitNamespaces" array taken form $options
array_unshift($this->nsStack, $implicitNS + array('' => self::NAMESPACE_HTML) + $this->implicitNamespaces);
if ($isFragment) {
$this->insertMode = static::IM_IN_BODY;
$this->frag = $this->doc->createDocumentFragment();
$this->current = $this->frag;
}
}
/**
* Get the document.
*/
public function document()
{
return $this->doc;
}
/**
* Get the DOM fragment for the body.
*
* This returns a DOMNodeList because a fragment may have zero or more
* DOMNodes at its root.
*
* @see http://www.w3.org/TR/2012/CR-html5-20121217/syntax.html#concept-frag-parse-context
*
* @return \DOMDocumentFragment
*/
public function fragment()
{
return $this->frag;
}
/**
* Provide an instruction processor.
*
* This is used for handling Processor Instructions as they are
* inserted. If omitted, PI's are inserted directly into the DOM tree.
*
* @param InstructionProcessor $proc
*/
public function setInstructionProcessor(InstructionProcessor $proc)
{
$this->processor = $proc;
}
public function doctype($name, $idType = 0, $id = null, $quirks = false)
{
// This is used solely for setting quirks mode. Currently we don't
// try to preserve the inbound DT. We convert it to HTML5.
$this->quirks = $quirks;
if ($this->insertMode > static::IM_INITIAL) {
$this->parseError('Illegal placement of DOCTYPE tag. Ignoring: ' . $name);
return;
}
$this->insertMode = static::IM_BEFORE_HTML;
}
/**
* Process the start tag.
*
* @todo - XMLNS namespace handling (we need to parse, even if it's not valid)
* - XLink, MathML and SVG namespace handling
* - Omission rules: 8.1.2.4 Optional tags
*
* @param string $name
* @param array $attributes
* @param bool $selfClosing
*
* @return int
*/
public function startTag($name, $attributes = array(), $selfClosing = false)
{
$lname = $this->normalizeTagName($name);
// Make sure we have an html element.
if (!$this->doc->documentElement && 'html' !== $name && !$this->frag) {
$this->startTag('html');
}
// Set quirks mode if we're at IM_INITIAL with no doctype.
if ($this->insertMode === static::IM_INITIAL) {
$this->quirks = true;
$this->parseError('No DOCTYPE specified.');
}
// SPECIAL TAG HANDLING:
// Spec says do this, and "don't ask."
// find the spec where this is defined... looks problematic
if ('image' === $name && !($this->insertMode === static::IM_IN_SVG || $this->insertMode === static::IM_IN_MATHML)) {
$name = 'img';
}
// Autoclose p tags where appropriate.
if ($this->insertMode >= static::IM_IN_BODY && Elements::isA($name, Elements::AUTOCLOSE_P)) {
$this->autoclose('p');
}
// Set insert mode:
switch ($name) {
case 'html':
$this->insertMode = static::IM_BEFORE_HEAD;
break;
case 'head':
if ($this->insertMode > static::IM_BEFORE_HEAD) {
$this->parseError('Unexpected head tag outside of head context.');
} else {
$this->insertMode = static::IM_IN_HEAD;
}
break;
case 'body':
$this->insertMode = static::IM_IN_BODY;
break;
case 'svg':
$this->insertMode = static::IM_IN_SVG;
break;
case 'math':
$this->insertMode = static::IM_IN_MATHML;
break;
case 'noscript':
if ($this->insertMode === static::IM_IN_HEAD) {
$this->insertMode = static::IM_IN_HEAD_NOSCRIPT;
}
break;
}
// Special case handling for SVG.
if ($this->insertMode === static::IM_IN_SVG) {
$lname = Elements::normalizeSvgElement($lname);
}
$pushes = 0;
// when we found a tag thats appears inside $nsRoots, we have to switch the defalut namespace
if (isset($this->nsRoots[$lname]) && $this->nsStack[0][''] !== $this->nsRoots[$lname]) {
array_unshift($this->nsStack, array(
'' => $this->nsRoots[$lname],
) + $this->nsStack[0]);
++$pushes;
}
$needsWorkaround = false;
if (isset($this->options['xmlNamespaces']) && $this->options['xmlNamespaces']) {
// when xmlNamespaces is true a and we found a 'xmlns' or 'xmlns:*' attribute, we should add a new item to the $nsStack
foreach ($attributes as $aName => $aVal) {
if ('xmlns' === $aName) {
$needsWorkaround = $aVal;
array_unshift($this->nsStack, array(
'' => $aVal,
) + $this->nsStack[0]);
++$pushes;
} elseif ('xmlns' === (($pos = strpos($aName, ':')) ? substr($aName, 0, $pos) : '')) {
array_unshift($this->nsStack, array(
substr($aName, $pos + 1) => $aVal,
) + $this->nsStack[0]);
++$pushes;
}
}
}
if ($this->onlyInline && Elements::isA($lname, Elements::BLOCK_TAG)) {
$this->autoclose($this->onlyInline);
$this->onlyInline = null;
}
try {
$prefix = ($pos = strpos($lname, ':')) ? substr($lname, 0, $pos) : '';
if (false !== $needsWorkaround) {
$xml = "<$lname xmlns=\"$needsWorkaround\" " . (strlen($prefix) && isset($this->nsStack[0][$prefix]) ? ("xmlns:$prefix=\"" . $this->nsStack[0][$prefix] . '"') : '') . '/>';
$frag = new \DOMDocument('1.0', 'UTF-8');
$frag->loadXML($xml);
$ele = $this->doc->importNode($frag->documentElement, true);
} else {
if (!isset($this->nsStack[0][$prefix]) || ('' === $prefix && isset($this->options[self::OPT_DISABLE_HTML_NS]) && $this->options[self::OPT_DISABLE_HTML_NS])) {
$ele = $this->doc->createElement($lname);
} else {
$ele = $this->doc->createElementNS($this->nsStack[0][$prefix], $lname);
}
}
} catch (\DOMException $e) {
$this->parseError("Illegal tag name: <$lname>. Replaced with <invalid>.");
$ele = $this->doc->createElement('invalid');
}
if (Elements::isA($lname, Elements::BLOCK_ONLY_INLINE)) {
$this->onlyInline = $lname;
}
// When we add some namespacess, we have to track them. Later, when "endElement" is invoked, we have to remove them.
// When we are on a void tag, we do not need to care about namesapce nesting.
if ($pushes > 0 && !Elements::isA($name, Elements::VOID_TAG)) {
// PHP tends to free the memory used by DOM,
// to avoid spl_object_hash collisions whe have to avoid garbage collection of $ele storing it into $pushes
// see https://bugs.php.net/bug.php?id=67459
$this->pushes[spl_object_hash($ele)] = array($pushes, $ele);
}
foreach ($attributes as $aName => $aVal) {
// xmlns attributes can't be set
if ('xmlns' === $aName) {
continue;
}
if ($this->insertMode === static::IM_IN_SVG) {
$aName = Elements::normalizeSvgAttribute($aName);
} elseif ($this->insertMode === static::IM_IN_MATHML) {
$aName = Elements::normalizeMathMlAttribute($aName);
}
$aVal = (string) $aVal;
try {
$prefix = ($pos = strpos($aName, ':')) ? substr($aName, 0, $pos) : false;
if ('xmlns' === $prefix) {
$ele->setAttributeNS(self::NAMESPACE_XMLNS, $aName, $aVal);
} elseif (false !== $prefix && isset($this->nsStack[0][$prefix])) {
$ele->setAttributeNS($this->nsStack[0][$prefix], $aName, $aVal);
} else {
$ele->setAttribute($aName, $aVal);
}
} catch (\DOMException $e) {
$this->parseError("Illegal attribute name for tag $name. Ignoring: $aName");
continue;
}
// This is necessary on a non-DTD schema, like HTML5.
if ('id' === $aName) {
$ele->setIdAttribute('id', true);
}
}
if ($this->frag !== $this->current && $this->rules->hasRules($name)) {
// Some elements have special processing rules. Handle those separately.
$this->current = $this->rules->evaluate($ele, $this->current);
} else {
// Otherwise, it's a standard element.
$this->current->appendChild($ele);
if (!Elements::isA($name, Elements::VOID_TAG)) {
$this->current = $ele;
}
// Self-closing tags should only be respected on foreign elements
// (and are implied on void elements)
// See: https://www.w3.org/TR/html5/syntax.html#start-tags
if (Elements::isHtml5Element($name)) {
$selfClosing = false;
}
}
// This is sort of a last-ditch attempt to correct for cases where no head/body
// elements are provided.
if ($this->insertMode <= static::IM_BEFORE_HEAD && 'head' !== $name && 'html' !== $name) {
$this->insertMode = static::IM_IN_BODY;
}
// When we are on a void tag, we do not need to care about namesapce nesting,
// but we have to remove the namespaces pushed to $nsStack.
if ($pushes > 0 && Elements::isA($name, Elements::VOID_TAG)) {
// remove the namespaced definded by current node
for ($i = 0; $i < $pushes; ++$i) {
array_shift($this->nsStack);
}
}
if ($selfClosing) {
$this->endTag($name);
}
// Return the element mask, which the tokenizer can then use to set
// various processing rules.
return Elements::element($name);
}
public function endTag($name)
{
$lname = $this->normalizeTagName($name);
// Special case within 12.2.6.4.7: An end tag whose tag name is "br" should be treated as an opening tag
if ('br' === $name) {
$this->parseError('Closing tag encountered for void element br.');
$this->startTag('br');
}
// Ignore closing tags for other unary elements.
elseif (Elements::isA($name, Elements::VOID_TAG)) {
return;
}
if ($this->insertMode <= static::IM_BEFORE_HTML) {
// 8.2.5.4.2
if (in_array($name, array(
'html',
'br',
'head',
'title',
))) {
$this->startTag('html');
$this->endTag($name);
$this->insertMode = static::IM_BEFORE_HEAD;
return;
}
// Ignore the tag.
$this->parseError('Illegal closing tag at global scope.');
return;
}
// Special case handling for SVG.
if ($this->insertMode === static::IM_IN_SVG) {
$lname = Elements::normalizeSvgElement($lname);
}
$cid = spl_object_hash($this->current);
// XXX: HTML has no parent. What do we do, though,
// if this element appears in the wrong place?
if ('html' === $lname) {
return;
}
// remove the namespaced definded by current node
if (isset($this->pushes[$cid])) {
for ($i = 0; $i < $this->pushes[$cid][0]; ++$i) {
array_shift($this->nsStack);
}
unset($this->pushes[$cid]);
}
if (!$this->autoclose($lname)) {
$this->parseError('Could not find closing tag for ' . $lname);
}
switch ($lname) {
case 'head':
$this->insertMode = static::IM_AFTER_HEAD;
break;
case 'body':
$this->insertMode = static::IM_AFTER_BODY;
break;
case 'svg':
case 'mathml':
$this->insertMode = static::IM_IN_BODY;
break;
}
}
public function comment($cdata)
{
// TODO: Need to handle case where comment appears outside of the HTML tag.
$node = $this->doc->createComment($cdata);
$this->current->appendChild($node);
}
public function text($data)
{
// XXX: Hmmm.... should we really be this strict?
if ($this->insertMode < static::IM_IN_HEAD) {
// Per '8.2.5.4.3 The "before head" insertion mode' the characters
// " \t\n\r\f" should be ignored but no mention of a parse error. This is
// practical as most documents contain these characters. Other text is not
// expected here so recording a parse error is necessary.
$dataTmp = trim($data, " \t\n\r\f");
if (!empty($dataTmp)) {
// fprintf(STDOUT, "Unexpected insert mode: %d", $this->insertMode);
$this->parseError('Unexpected text. Ignoring: ' . $dataTmp);
}
return;
}
// fprintf(STDOUT, "Appending text %s.", $data);
$node = $this->doc->createTextNode($data);
$this->current->appendChild($node);
}
public function eof()
{
// If the $current isn't the $root, do we need to do anything?
}
public function parseError($msg, $line = 0, $col = 0)
{
$this->errors[] = sprintf('Line %d, Col %d: %s', $line, $col, $msg);
}
public function getErrors()
{
return $this->errors;
}
public function cdata($data)
{
$node = $this->doc->createCDATASection($data);
$this->current->appendChild($node);
}
public function processingInstruction($name, $data = null)
{
// XXX: Ignore initial XML declaration, per the spec.
if ($this->insertMode === static::IM_INITIAL && 'xml' === strtolower($name)) {
return;
}
// Important: The processor may modify the current DOM tree however it sees fit.
if ($this->processor instanceof InstructionProcessor) {
$res = $this->processor->process($this->current, $name, $data);
if (!empty($res)) {
$this->current = $res;
}
return;
}
// Otherwise, this is just a dumb PI element.
$node = $this->doc->createProcessingInstruction($name, $data);
$this->current->appendChild($node);
}
// ==========================================================================
// UTILITIES
// ==========================================================================
/**
* Apply normalization rules to a tag name.
* See sections 2.9 and 8.1.2.
*
* @param string $tagName
*
* @return string The normalized tag name.
*/
protected function normalizeTagName($tagName)
{
/*
* Section 2.9 suggests that we should not do this. if (strpos($name, ':') !== false) { // We know from the grammar that there must be at least one other // char besides :, since : is not a legal tag start. $parts = explode(':', $name); return array_pop($parts); }
*/
return $tagName;
}
protected function quirksTreeResolver($name)
{
throw new \Exception('Not implemented.');
}
/**
* Automatically climb the tree and close the closest node with the matching $tag.
*
* @param string $tagName
*
* @return bool
*/
protected function autoclose($tagName)
{
$working = $this->current;
do {
if (XML_ELEMENT_NODE !== $working->nodeType) {
return false;
}
if ($working->tagName === $tagName) {
$this->current = $working->parentNode;
return true;
}
} while ($working = $working->parentNode);
return false;
}
/**
* Checks if the given tagname is an ancestor of the present candidate.
*
* If $this->current or anything above $this->current matches the given tag
* name, this returns true.
*
* @param string $tagName
*
* @return bool
*/
protected function isAncestor($tagName)
{
$candidate = $this->current;
while (XML_ELEMENT_NODE === $candidate->nodeType) {
if ($candidate->tagName === $tagName) {
return true;
}
$candidate = $candidate->parentNode;
}
return false;
}
/**
* Returns true if the immediate parent element is of the given tagname.
*
* @param string $tagName
*
* @return bool
*/
protected function isParent($tagName)
{
return $this->current->tagName === $tagName;
}
}

114
vendor/masterminds/html5/src/HTML5/Parser/EventHandler.php vendored Normal file
View File

@@ -0,0 +1,114 @@
<?php
namespace Masterminds\HTML5\Parser;
/**
* Standard events for HTML5.
*
* This is roughly analogous to a SAX2 or expat-style interface.
* However, it is tuned specifically for HTML5, according to section 8
* of the HTML5 specification.
*
* An event handler receives parser events. For a concrete
* implementation, see DOMTreeBuilder.
*
* Quirks support in the parser is limited to close-in syntax (malformed
* tags or attributes). Higher order syntax and semantic issues with a
* document (e.g. mismatched tags, illegal nesting, etc.) are the
* responsibility of the event handler implementation.
*
* See HTML5 spec section 8.2.4
*/
interface EventHandler
{
const DOCTYPE_NONE = 0;
const DOCTYPE_PUBLIC = 1;
const DOCTYPE_SYSTEM = 2;
/**
* A doctype declaration.
*
* @param string $name The name of the root element.
* @param int $idType One of DOCTYPE_NONE, DOCTYPE_PUBLIC, or DOCTYPE_SYSTEM
* @param string $id The identifier. For DOCTYPE_PUBLIC, this is the public ID. If DOCTYPE_SYSTEM,
* then this is a system ID.
* @param bool $quirks Indicates whether the builder should enter quirks mode.
*/
public function doctype($name, $idType = 0, $id = null, $quirks = false);
/**
* A start tag.
*
* IMPORTANT: The parser watches the return value of this event. If this returns
* an integer, the parser will switch TEXTMODE patters according to the int.
*
* This is how the Tree Builder can tell the Tokenizer when a certain tag should
* cause the parser to go into RAW text mode.
*
* The HTML5 standard requires that the builder is the one that initiates this
* step, and this is the only way short of a circular reference that we can
* do that.
*
* Example: if a startTag even for a `script` name is fired, and the startTag()
* implementation returns Tokenizer::TEXTMODE_RAW, then the tokenizer will
* switch into RAW text mode and consume data until it reaches a closing
* `script` tag.
*
* The textmode is automatically reset to Tokenizer::TEXTMODE_NORMAL when the
* closing tag is encounter. **This behavior may change.**
*
* @param string $name The tag name.
* @param array $attributes An array with all of the tag's attributes.
* @param bool $selfClosing An indicator of whether or not this tag is self-closing (<foo/>).
*
* @return int one of the Tokenizer::TEXTMODE_* constants
*/
public function startTag($name, $attributes = array(), $selfClosing = false);
/**
* An end-tag.
*/
public function endTag($name);
/**
* A comment section (unparsed character data).
*/
public function comment($cdata);
/**
* A unit of parsed character data.
*
* Entities in this text are *already decoded*.
*/
public function text($cdata);
/**
* Indicates that the document has been entirely processed.
*/
public function eof();
/**
* Emitted when the parser encounters an error condition.
*/
public function parseError($msg, $line, $col);
/**
* A CDATA section.
*
* @param string $data
* The unparsed character data
*/
public function cdata($data);
/**
* This is a holdover from the XML spec.
*
* While user agents don't get PIs, server-side does.
*
* @param string $name The name of the processor (e.g. 'php').
* @param string $data The unparsed data.
*/
public function processingInstruction($name, $data = null);
}

33
vendor/masterminds/html5/src/HTML5/Parser/FileInputStream.php vendored Normal file
View File

@@ -0,0 +1,33 @@
<?php
namespace Masterminds\HTML5\Parser;
/**
* The FileInputStream loads a file to be parsed.
*
* So right now we read files into strings and then process the
* string. We chose to do this largely for the sake of expediency of
* development, and also because we could optimize toward processing
* arbitrarily large chunks of the input. But in the future, we'd
* really like to rewrite this class to efficiently handle lower level
* stream reads (and thus efficiently handle large documents).
*
* @deprecated since 2.4, to remove in 3.0. Use a string in the scanner instead.
*/
class FileInputStream extends StringInputStream implements InputStream
{
/**
* Load a file input stream.
*
* @param string $data The file or url path to load.
* @param string $encoding The encoding to use for the data.
* @param string $debug A fprintf format to use to echo the data on stdout.
*/
public function __construct($data, $encoding = 'UTF-8', $debug = '')
{
// Get the contents of the file.
$content = file_get_contents($data);
parent::__construct($content, $encoding, $debug);
}
}

Some files were not shown because too many files have changed in this diff Show More