TLDR: see the tutorial !
Download for GNU/Linux 64 bits: see Releases (78Mo, self-contained executable).
This is a little tool for the lisp REPL or the command line (also with a readline interactive prompt) to search for torrents and get magnet links.
update feb, 2019: a little GUI is in the works.
See the download link above. It is a self-contained executable for GNU/Linux 64-bits. You do not need to install a Lisp implementation. It's a 78Mo binary.
The Lisp lib is not distributed with Quicklisp (or Roswell) (yet?). Clone this repo where Quicklisp can find it:
git clone firstname.lastname@example.org:vindarel/cl-torrents.git ~/quicklisp/local-projects/
Clone its dependency, replic, a library that helps build the readline interface:
git clone https://github.com/vindarel/replic.git ~/quicklisp/local-projects/
and install it:
To build the executable:
If a search doesn't work with a
CRYPTO_num_locks error message:
> ./torrents matrix searching '(matrix)' on 1337... no results. error: The alien function "CRYPTOnumlocks" is undefined. searching '(matrix)' on DOWNLOADSME... no results. error: The alien function "CRYPTOnumlocks" is undefined.
You might need to upgrade your OpenSSL version. See this issue.
We can use this little app both in a Lisp REPL (Slime) and in a terminal, as a command line tool or from its embedded interactive prompt.
Results are cached in
Search for torrents:
(use-package :torrents) (search-torrents "matrix") [...] 6: The Matrix Revolutions (2003) BRRip [Dual Audio] [Hindi+Eng] 5: Matrix (1999)Blu-Ray 720p Dublado PT-BR - mo93438 4: The Matrix Trilogy (1999-2003) BDRip 1080p Dual Audio [ Hind 3: The Matrix Trilogy (1999-2003) BluRay BDRip 1080p AC3 2: The Matrix Trilogy (1999-2003) + Extras 1080p BluRay x264 Du 1: The Matrix Reloaded (2003) FullHD, Dual Audio: English + Spa 0: Matrix FRENCH DVDRIP 1999 COOL
To help finding the most relevant results (the website sometimes is a bit relaxed with our search terms), each keyword is printed in a different color:
(you need to enable ansi colors in Slime, see slime-repl-ansi-color. Disable this feature with
(setf cl-ansi-text:*enabled* nil))
We get a magnet link with:
(magnet 0) ;; "magnet:?xt=urn:btih:40eca43690cf1b99b0a4d485ebf4855d20b0bac5"
We can open it with a torrent client (transmission-gtk by default):
And voil? :)
We may add more features, eventually. Our goal is rather to write a tutorial to show you diverse Common Lisp topics.
*cache-p*: if nil, don't use the cache system.
Use the options given below,
$ torrents -h CL-torrents. Usage: Available options: -d, --details print more details (like the torrent's url) -h, --help print this help text -i, --interactive enter an interactive repl -m, --magnet ARG get the magnet link of the given search result. -n, --nb ARG set the maximum number of results to print. -o, --open INDEX download with a torrent client (transmission-gtk by default) -v, --version print the version
Example: below we search for "matrix?", we display only 1 result and we get the magnet link of the first result.
$ torrents -n 1 matrix french 1999 cool -m 0
Start it with
help and the help of each command.
The available commands are (use the auto-completion):
search: search for torrents, print the results sorted by number of seeders.
magnet <i>, where i is a result index,
firefox <i>: open the given torrent page in a web browser
download <i>: open a magnet link with a torrent client (transmission-gtk by default)
unhighlight): highlight the given words in yellow, for this session.
nto set the number of results to print to n
details: toggle the display of details
We are in a regular readline interface, so the usual keybindings and options are available.
Configuration files (in development)
cl-torrents will read two configuration files. An ini-style one, and a lispy one.
First, it will search for a
~.torrents.conf file from
~/.torrents.conf. The last one takes precedence.
For example, those are the default values:
[default] scrapers = 1337 DOWNLOADSME browser = firefox nb-results = 20 cache-p = true # use the cache system.
Moreover, you can set parameters related to the repl:
[default] # again, default values: confirm-exit = true # ask with a yes/no prompt before quiting the app. verbose = false history = true # use the repl history. write-history = true # read the history, but don't write to it.
You can set them for all replic-based apps in
~/.replic.conf, or override them in
Note: we only use a "default" profile for now.
Lisp init file
If the file
~/.torrents.lisp exists, it will be loaded after the
.conf one and before the command line arguments.
--no-userinit prevents it from loading.
You can write whatever you want there, it is just a lisp file that will be
load'ed at startup.
Note: what we will eventually do then is to expose cl-torrents' mechanism via hooks.
One thing you can do is define additional commands to be found at the repl. We do so by following replic's mechanism:
- define functions inside the
- define the completion choices for some functions
exportthe functions you want to see turned into commands.
You can copy the example below in
~/.torrents.lisp and re-run
torrents -i to try that out.
(in-package :torrents.user) (defparameter *names* '() "List of names (string) given to `hello`. Will be autocompleted by `goodbye`.") (defun hello (name) "Takes only one argument. Adds the given name to the global `*names*` global variable, used to complete arguments of `goodbye`. " (format t "hello ~a~&" name) (push name *names*)) (defun goodbye (name) "Says goodbye to name, where `name` should be completed from what was given to `hello`." (format t "goodbye ~a~&" name)) ;; Custom completion for goodbye: (replic.completion:add-completion "goodbye" (lambda () *names*)) ;; and export the two functions to find them as commands. (export '(hello goodbye))
Writing a little web scraper like this one is not difficult. However, I had to spend some time to find out the right libraries and resources. It is also not trivial at first to start a Lisp project. So this tutorial is a mix of hopefully useful stuff:
- web scraping,
- async web scraping,
- trying out things at the REPL,
- where to find documentation,
- creating and loading a project,
- basic data structures and gotchas,
- some useful libraries,
- unit tests, with mocks, running tests from the shell, continuous integration,
- parsing command line arguments, creating executables, continuous delivery,
- basics of error handling,
It will eventually tackle more topics and features (more settings, working with a local copy of TPB?) but in the meanwhile, read
the tutorial !
It was built with https://github.com/fniessen/org-html-themes.
- [X] cache (on files) v0.3
- [X] CI
- local copy of TPB
- more command line options (specially verbosity)
- [X] build with ECL. Used the lisp-devel docker image. Result: a 52Mo executable with a runtime error.
- [X] open an url
- download the torrent file
- [X] ini config file, to parse with py-configparser
- [X] add torrent to a local torrent client
- add the torrent to a remote transmission client, with cl-transmission.
- self-contained web app (doc),...
- [X] interactive prompt, with completion of commands.
dev (upcoming v0.11)
- added: load
~/.torrents.lisp, create new commands from the
- added the ability to read an ini-like config file,
~/.torrents.conf. All exported variables from the
:torrentpackage can be overriden. They are written without earmuffs:
:*nb-results* :*browser* :*torrent-client* :*cache-p*
See more on replic. (all parameters need more tests)
# ~/.torrents.conf # those are the defaults. [default] nb-results = 20 browser = firefox torrent-client = firefox cache-p = true
- added missing
-ooption to download with a torrent client (transmission-gtk).
- choose the list of scrapers.
downloadcommand (open magnet link with transmission by default).
- use the replic library to build the repl, the commands and their completions.
- -> new builtin
- -> better error handling (don't exit the repl on an error).
- -> new builtin
- fix cli usage without a keyword search, print help.
- added torrentdownloads.me scraper.
- new parameter,
*cache-p*: if nil, don't use the cache.
- added 1337x.to scraper.
- removed piratebay.to, katcr.co, torrent.cd, they're down :/
- added a readline-based repl with some commands:
open, with their completion. See
help. Built with cl-readline, but might use replic anytime soon.
- fix cache not created by binary
- create cache in
We use our replic library to automatically build the readline repl:
git clone https://github.com/vindarel/replic/ ~/quicklisp/local-projects/
Clone also cl-torrents in
now you can
End-to-end tests (checking that the websites respond and our scrapers still work):
Don't miss these good resources:
- the Common Lisp Cookbook
- (my) lisp-journey: introductory posts, lists of resources, selection of screencasts.
Setting up a Common Lisp environment
The easiest solution of all is to try Portacle, a portable and multiplatform CL environment shipping: Emacs25 and Slime (slightly customized), SBCL, Quicklisp and Git.
Emacs isn't the only editor though, see https://lispcookbook.github.io/cl-cookbook/editor-support.html for Vim plugins, to discover Lem (ncurses and Electron frontends), Atom-Slime, notebooks and the ipython-like cl-repl.
apt install sbcl
and install Quicklisp (two commands) and Slime for Emacs (in MELPA, with package.el). See https://lispcookbook.github.io/cl-cookbook/getting-started.html.