cl-torrents

2019-05-21

This is a little app to search for torrents on popular trackers.

It comes as:

  • a lisp library,
  • a command line interface,
  • a terminal readline interface,
  • a simple GUI.
  • The Web UI works too.

    We currently scrape 1337x.to (since v0.9) and torrentdownloads.me (v0.10) and present the results sorted by seeders.

    Download for GNU/Linux 64 bits: see Releases (78Mo, self-contained executable).

Installation

The library is on Quicklisp (february 2019) and Ultralisp.

(ql:quickload "torrents")
(torrents:search-torrents "tears of steel")

See the download link of the binary above. It is a self-contained executable for GNU/Linux 64-bits. You do not need to install a Lisp implementation. It's a 78Mo binary.

To build the readline app, do:

make build

To build the Tk GUI:

make build-gui

then run

./torrents --help
./torrents-tk &

Troubleshooting

The Tk GUI needs nodgui newer than march, 2019.

If a search doesn't work with a CRYPTO_num_locks error message:

> ./torrents matrix searching '(matrix)' on 1337... no results. error: The alien function "CRYPTOnumlocks" is undefined. searching '(matrix)' on DOWNLOADSME... no results. error: The alien function "CRYPTOnumlocks" is undefined.

You might need to upgrade your OpenSSL version. See this issue.

Usage

We can use this little app both in a Lisp REPL (Slime) and in a terminal, as a command line tool or from its embedded interactive prompt.

Results are cached in ~/.cl-torrents/cache/. Delete this directory if you want new results.

Lisp REPL

Search for torrents:

(use-package :torrents)
(search-torrents "matrix")
  [...]
  6: The Matrix Revolutions (2003) BRRip [Dual Audio] [Hindi+Eng]
  5: Matrix (1999)Blu-Ray 720p Dublado PT-BR - mo93438
  4: The Matrix Trilogy (1999-2003) BDRip 1080p Dual Audio [ Hind
  3: The Matrix Trilogy (1999-2003) BluRay BDRip 1080p AC3
  2: The Matrix Trilogy (1999-2003) + Extras 1080p BluRay x264 Du
  1: The Matrix Reloaded (2003) FullHD, Dual Audio: English + Spa
  0: Matrix FRENCH DVDRIP 1999 COOL

To help finding the most relevant results (the website sometimes is a bit relaxed with our search terms), each keyword is printed in a different color:

(you need to enable ansi colors in Slime, see slime-repl-ansi-color. Disable this feature with (setf cl-ansi-text:*enabled* nil))

We get a magnet link with:

(magnet 0)
;; "magnet:?xt=urn:btih:40eca43690cf1b99b0a4d485ebf4855d20b0bac5"

We can open it with a torrent client (transmission-gtk by default):

(download 0)

And voil? :)

We may add more features, eventually. Our goal is rather to write a tutorial to show you diverse Common Lisp topics.

Parameters

*cache-p*: if nil, don't use the cache system.

Command line

Use the options given below,

$ torrents -h
CL-torrents. Usage:

Available options:
  -d, --details            print more details (like the torrent's url)
  -h, --help               print this help text
  -i, --interactive        enter an interactive repl
  -m, --magnet ARG         get the magnet link of the given search result.
  -n, --nb ARG             set the maximum number of results to print.
  -o, --open INDEX         download with a torrent client (transmission-gtk by default)
  -v, --version            print the version

Example: below we search for "matrix?", we display only 1 result and we get the magnet link of the first result.

$ torrents -n 1 matrix french 1999 cool -m 0

Readline interface

Start it with torrents -i.

See the help and the help of each command.

The available commands are (use the auto-completion):

  • search: search for torrents, print the results sorted by number of seeders.
  • magnet <i>, where i is a result index,
  • open or firefox <i>: open the given torrent page in a web browser
  • download <i>: open a magnet link with a torrent client (transmission-gtk by default)
  • filter <text>: show results that have this text in their title. This reduces the tab-completion of ids for all commands.
  • highlight (and unhighlight): highlight the given words in yellow, for this session.
  • url <i>
  • nb-results n to set the number of results to print to n
  • details: toggle the display of details
  • quit (or C-c)

And the available settings to change with set are (use set's autocompletion):

  • *details*: if true, print more details below each row (like the url). Defaults to false.
  • *nb-results*: how many results to print. Defaults to 20.
  • *browser*: the browser name to use to open urls. Defaults to Firefox.
  • *torrent-client*: the desktop torrent client name. Defaults to Transmission.
  • *cache-p*: if true (the default), use the cached results.

You can also set them in a configuration file.

Note that we are in a regular readline interface, so the usual keybindings and options are available.

Configuration files (in development)

cl-torrents will read two configuration files. An ini-style one, and a lispy one.

Ini-style config

First, it will search for a ~.torrents.conf file from ~/.config/.torrents.conf and ~/.torrents.conf. The last one takes precedence.

For example, those are the default values:

[default]
scrapers = 1337 DOWNLOADSME
browser = firefox
nb-results = 20
cache-p = true  # use the cache system.

Moreover, you can set parameters related to the repl:

[default]
# again, default values:
confirm-exit = true  # ask with a yes/no prompt before quiting the app.
verbose = false
history = true  # use the repl history.
write-history = true  # read the history, but don't write to it.

You can set them for all replic-based apps in ~/.replic.conf, or override them in .torrents.conf.

Note: we only use a "default" profile for now.

Lisp init file

If the file ~/.torrents.lisp exists, it will be loaded after the .conf one and before the command line arguments.

The option --no-userinit prevents it from loading.

You can write whatever you want there, it is just a lisp file that will be load'ed at startup.

Note: what we will eventually do then is to expose cl-torrents' mechanism via hooks.

One thing you can do is define additional commands to be found at the repl. We do so by following replic's mechanism:

  • define functions inside the torrents.user package
  • define the completion choices for some functions
  • export the functions you want to see turned into commands.

You can copy the example below in ~/.torrents.lisp and re-run torrents -i to try that out.

(in-package :torrents.user)

(defparameter *names* '()
  "List of names (string) given to `hello`. Will be autocompleted by `goodbye`.")

(defun hello (name)
  "Takes only one argument. Adds the given name to the global
  `*names*` global variable, used to complete arguments of `goodbye`. "
  (format t "hello ~a~&" name)
  (push name *names*))

(defun goodbye (name)
  "Says goodbye to name, where `name` should be completed from what was given to `hello`."
  (format t "goodbye ~a~&" name))

;; Custom completion for goodbye:
(replic.completion:add-completion "goodbye" (lambda () *names*))

;; and export the two functions to find them as commands.
(export '(hello
          goodbye))

Ideas, todos

  • [X] cache (on files) v0.3
  • [X] CI
  • local copy of TPB
  • more command line options (specially verbosity)
    • [X] -i print url
  • [X] build with ECL. Used the lisp-devel docker image. Result: a 52Mo executable with a runtime error.
  • [X] open an url
  • download the torrent file
  • [X] ini config file, to parse with py-configparser
  • [X] add torrent to a local torrent client
  • add the torrent to a remote transmission client, with cl-transmission.
  • self-contained web app (doc),...
  • [X] interactive prompt, with completion of commands.

Release notes

https://gitlab.com/vindarel/cl-torrents/tags

dev (upcoming v0.11)

  • added: a simple Tk GUI (search, open in browser, download with torrent desktop client).
  • added: a filter command, to only display results whose title contains a given string. It reduces the TAB-completion of ids (but doesn't constrain it).
  • added: all functions can now TAB-complete the list of ids.
  • added: scrape and display torrents' size.
  • fixed -d cli arg.
  • added: load ~/.torrents.lisp, create new commands from the torrents.user package.
  • added the ability to read an ini-like config file, ~/.torrents.conf. All exported variables from the :torrent package can be overriden. They are written without earmuffs:
           :*nb-results*
           :*browser*
           :*torrent-client*
           :*cache-p*

gives

# ~/.torrents.conf
# those are the defaults.
[default]
nb-results = 20
browser = firefox
torrent-client = firefox
cache-p = true
See more on replic. (all parameters need more tests)
  • added missing -o option to download with a torrent client (transmission-gtk).
  • cl-torrents.conf configuration file:
    • choose the list of scrapers.
  • download command (open magnet link with transmission by default).
  • use the replic library to build the repl, the commands and their completions.
    • -> new builtin highlight and unhighlight commands,
    • -> better error handling (don't exit the repl on an error).
  • fix cli usage without a keyword search, print help.

v0.10

v0.9

  • added 1337x.to scraper.
  • removed piratebay.to, katcr.co, torrent.cd, they're down :/

v0.8

  • added a readline-based repl with some commands: search, magnet, open, with their completion. See help. Built with cl-readline, but might use replic anytime soon.

v0.7

  • fix cache not created by binary
  • create cache in ~/.cl-torrents/cache/

Dev

We use our replic library to automatically build the readline repl.

Clone cl-torrents in ~/quicklisp/local-projects,

now you can (ql:quickload :torrents).

Unit tests:

make test

End-to-end tests (checking that the websites respond and our scrapers still work):

make test-end2end

Resources

Don't miss these good resources:

Tutorial

Writing a little web scraper like this one is not difficult. However, I had to spend some time to find out the right libraries and resources. It is also not trivial at first to start a Lisp project. So the first thing we did is write a tutorial. It is a mix of hopefully useful stuff:

  • web scraping,
  • async web scraping,
  • trying out things at the REPL,
  • where to find documentation,
  • creating and loading a project,
  • basic data structures and gotchas,
  • some useful libraries,
  • unit tests, with mocks, running tests from the shell, continuous integration,
  • parsing command line arguments, creating executables, continuous delivery,
  • basics of error handling,
  • ...

    It will eventually tackle more topics and features (more settings, working with a local copy of TPB?) but in the meanwhile, read

    the tutorial !

    It was built with https://github.com/fniessen/org-html-themes.

Licence

MIT.