Common Lisp universal feed parser
CL-FEEDPARSER is a Common Lisp port of a subset of Python’s feedparser. It does not do fetching; it only does parsing.
The API of cl-feedparser is very close to the API of feedparser. As in Feedparser, the parsed feed is returned as a set of nested dictionaries (hash tables). Unsurprisingly, keys that are strings in Python are keywords in cl-feedparser, with dashes instead of underscores.
feed['author'] -> (feedparser:feed-ref feed :author) feed['author_detail'] -> (feedparser:feed-ref feed :author-detail)
feed-ref? It is just like
gethash, but if the provided
key is constant, then it does a compile-time check that the key
provided is valid. You can always just use
gethash, of course.)
There are a few important differences:
- Parsed times are returned as universal times, rather than Python time tuples.
- The sanitizer strips all inline CSS.
- Text is always sanitized, regardless of declared media type. (Python
feedparser does not sanitize
bozo-exceptionis a list of exceptions.
In regard to sanitizing, cl-feedparser offers a guarantee Python’s
feedparser does not: values that are returned as strings are always
sanitized. It is possible to turn sanitizing off, but with sanitizing
off, instances of
unsanitized-string are returned in place of
A warning: cl-feedparser is not nearly as sophisticated as Python feedparser about handling timestamps. On the other hand, timestamps are one area in which real-world feeds have actually improved since feedparser was written.
CL-FEEDPARSER is a classic case of a strangler application. Originally it was just a wrapper around the Python library; then parsing well-formed feeds was moved into Lisp, with a fallback to Python; finally, Lisp took over.
(ql:quickload '(:cl-feedparser :drakma)) (defparameter *feed* (feedparser:parse-feed (drakma:http-request "http://planet.lisp.org/rss20.xml"))) (feedparser:feed-ref *feed* :title) => "Planet Lisp"
There is an extra key,
:proxy, which is set to
the feed is from Feedburner (or uses elements in the Feedburner
namespace, as Feedblitz does).
It is also the case that the Feedburner link is always overridden with the feedburner:origLink. This is in the interest of future-proofing: Feedburner may not be around forever, so for archival purposes you want the real links instead of the Feedburner redirects.
In the wild, you will sometimes come across enormous feeds that have simply never been truncated: they contain every post ever posted. And even for ordinary feeds, you may only be interested in the latest entry or two out of 10 or 20.
CL-FEEDPARSER lets you skip entries with two options:
If you know how many entries you want, pass
(length (feed-ref (parse-feed giant-feed :max-entries 10) :entries)) => 10
If you know which entries you don’t want (because you already have them), pass
Each entry of the mask can be either an id or an (id . timestamp) pair. Any entry whose guid matches the supplied id is ignored. If the timestamp is supplied, then the matching entry will not be ignored if it has a timestamp that is newer than the timestamp on record. (That is, it will still be ignored if it does not have a timestamp.)
It’s important to understand how the limit and the mask interact. The limit takes effect first, and only then are the entries filtered with the mask. This might seem strange, but think about a feed with thousands of entries. If the mask were applied before the limit, you would get another set of older entries every time you tried to parse the feed.
By default, everything that might possibly be HTML is sanitized. This
includes cases where the media type is ambiguous; as such, information
might be lost. If that information is important, you may want to ask
cl-feedparser not to sanitize feeds.
There are actually two knobs: one turns off sanitizing for entry contents, and the other turns it off for everything else (titles and other metadata).
;; Don't sanitize entry contents. (parse-feed feed :sanitize-content nil) ;; Don’t sanitize titles or other metadata. (parse-feed feed :sanitize-title nil)
The idea here is that if you are going to be re-parsing the entry contents anyway, you may want to skip the round-trip from string to DOM and back to string again, and do the sanitizing more cheaply as part of your own processing.
Turning off sanitizing has consequences. Any time
returns a string, you may rely on it being sanitized. So, if you turn
off sanitizing, what you get are not strings, but instances of
unsanitized-string; to get the underlying string, you must call
unsanitized-string-string on them. (There is a type available,
feed-string, defined as
(or string unsanitized string), if you
want to do exhaustiveness checking.)
Since cl-feedparser builds on FXML and Plump, it can can
understand and recover from many common XML errors. Sometimes,
however, you may not want automatic recovery. You might, for example,
want to validate a feed. In this case, it is enough to pass
:safe nil to
(parse-feed buggy-feed :safe nil) => <ERROR>
Otherwise, errors during parsing are stored under the
:bozo-exception key of the feed.