file:https://gitlab.com/vindarel/cl-torrents/badges/master/pipeline.svg

This is a little app to search for torrents on popular trackers and to open them with a local or a remote client.

It comes as:

• a lisp library,
• a command line interface,
• a terminal readline interface,
• a simple GUI.
• The [[https://github.com/vindarel/cl-torrents-web][Web UI]] works too.

  We currently scrape 1337x.to (since v0.9), torrentdownloads.me (v0.10) and torrents-paradise (v0.11) and present the results sorted by seeders.

  Download for GNU/Linux 64 bits: see Releases (23MB, self-contained executable).

NOTE: torrents-paradise has a rich collection, a simple website as in KISS, it works on IPFS and has it offers an API… frankly, I want to use its web service and click on its adds.

With the new default Yaru theme (and there are more):

0.1Installation

The library is on Quicklisp (february 2019) and Ultralisp.

(ql:quickload "torrents")
(torrents:search-torrents "tears of steel")

Note: the master branch needs cl-transmission which was added to the Quicklisp distribution of december, 2019.

See the download link of the binary above. It is a self-contained executable for GNU/Linux 64-bits. You do not need to install a Lisp implementation. It's a 23MB binary.

To build the readline app, do:

make build

To build the Tk GUI:

make build-gui

then run

./torrents --help
./torrents-tk &

Troubleshooting

The Tk GUI needs nodgui newer than march, 2019.

If a search doesn't work with a CRYPTO_num_locks error message:

> ./torrents matrix searching '(matrix)' on 1337... no results. error: The alien function "CRYPTO_num_locks" is undefined. searching '(matrix)' on DOWNLOADSME... no results. error: The alien function "CRYPTO_num_locks" is undefined.

You might need to upgrade your OpenSSL version. See this issue.

0.2Usage

We can use this little app both in a Lisp REPL (Slime) and in a terminal, as a command line tool or from its embedded interactive prompt.

Results are cached in ~/.cl-torrents/cache/. Delete this directory if you want new results.

0.2.1Lisp REPL

Search for torrents:

(use-package :torrents)
(search-torrents "matrix")
  [...]
  6: The Matrix Revolutions (2003) BRRip [Dual Audio] [Hindi+Eng]
  5: Matrix (1999)Blu-Ray 720p Dublado PT-BR - mo93438
  4: The Matrix Trilogy (1999-2003) BDRip 1080p Dual Audio [ Hind
  3: The Matrix Trilogy (1999-2003) BluRay BDRip 1080p AC3
  2: The Matrix Trilogy (1999-2003) + Extras 1080p BluRay x264 Du
  1: The Matrix Reloaded (2003) FullHD, Dual Audio: English + Spa
  0: Matrix FRENCH DVDRIP 1999 COOL

To help finding the most relevant results (the website sometimes is a bit relaxed with our search terms), each keyword is printed in a different color:

(you need to enable ansi colors in Slime, see slime-repl-ansi-color. Disable this feature with (setf cl-ansi-text:*enabled* nil))

We get a magnet link with:

(magnet 0)
;; "magnet:?xt=urn:btih:40eca43690cf1b99b0a4d485ebf4855d20b0bac5"

We can open it with a torrent client (transmission-gtk by default):

(download 0)

Download it with a remote client (only transmission-remote so far):

you need settings, see below. TODO

;; in ~/.config/torrents.lisp
(setf *remote-client-url* "my.url.without.port")
(setf *remote-client-username* "transmission")
(setf *remote-client-password* "password")
;; port is 9091 by default.

(remote-download 0)

And voilà :)

We may add more features, eventually. Our goal is rather to write a tutorial to show you diverse Common Lisp topics.

0.2.1.1Parameters

*cache-p*: if nil, don't use the cache system.

0.2.2Command line

Use the options given below,

$ torrents -h
CL-torrents. Usage:

Available options:
  -d, --details            print more details (like the torrent's url)
  -h, --help               print this help text
  -i, --interactive        enter an interactive repl
  -m, --magnet ARG         get the magnet link of the given search result.
  -n, --nb ARG             set the maximum number of results to print.
  -o, --open INDEX         download with a torrent client (transmission-gtk by default)
  -v, --version            print the version

Example: below we search for "matrix…", we display only 1 result and we get the magnet link of the first result.

$ torrents -n 1 matrix french 1999 cool -m 0

0.2.3Readline interface

Start it with torrents -i.

See the help and the help of each command.

The available commands are (use the auto-completion):

• search: search for torrents, print the results sorted by number of seeders.
• magnet <i>, where i is a result index,
• open or firefox <i>: open the given torrent page in a web browser
• download <i>: open a magnet link with a torrent client(transmission-gtk by default)
• filter <text>: show results that have this text in theirtitle. This reduces the tab-completion of ids for all commands.
• highlight (and unhighlight): highlight the given words inyellow, for this session.
• url <i>
• nb-results =n= to set the number of results to print to n
• details: toggle the display of details
• quit (or C-c)

And the available settings to change with set are (use set's autocompletion):

• *details*: if true, print more details below each row (like theurl). Defaults to false.
• *nb-results*: how many results to print. Defaults to 20.
• *browser*: the browser name to use to open urls. Defaults to Firefox.
• *torrent-client*: the desktop torrent client name. Defaults to Transmission.
• *cache-p*: if true (the default), use the cached results.

You can also set them in a configuration file.

Note that we are in a regular readline interface, so the usual keybindings and options are available. For example, Readline obeys the ~/.inputrc startup file. You can change the way the TAB key does completion:

TAB: menu-complete

if you add this, the first press on TAB will insert the first completion candidate (VS listing all the choices, and requiring a second keypress). For cl-torrents, it is convenient because we can enter the first result quickly: I typically do search foo then download TAB.

Note: I found out that C-x C-r re-reads the inputrc file, so you can try without quitting cl-torrents.

0.2.4Configuration files (in development)

cl-torrents will read two configuration files. An ini-style one, and a lispy one.

0.2.4.1Ini-style config

First, it will search for a ~.torrents.conf file from ~/.config/.torrents.conf and ~/.torrents.conf. The last one takes precedence.

For example, those are the default values:

[default]
scrapers = 1337 DOWNLOADSME
browser = firefox
nb-results = 20
cache-p = true  # use the cache system.

Moreover, you can set parameters related to the repl:

[default]
# again, default values:
confirm-exit = true  # ask with a yes/no prompt before quiting the app.
verbose = false
history = true  # use the repl history.
write-history = true  # read the history, but don't write to it.

You can set them for all replic-based apps in ~/.replic.conf, or override them in .torrents.conf.

Note: we only use a "default" profile for now.

0.2.4.2Lisp init file

If the file ~/.torrents.lisp exists, it will be loaded after the .conf one and before the command line arguments.

The option --no-userinit prevents it from loading.

You can write whatever you want there, it is just a lisp file that will be load'ed at startup.

Note: what we will eventually do then is to expose cl-torrents' mechanism via hooks.

0.2.4.3Extending the app, creating commands

One thing you can do is define additional commands to be found at the repl. We do so by following replic's mechanism:

• define functions inside the torrents.user package
• define the completion choices for some functions
• export the functions you want to see turned intocommands. cl-torrents, with the help of the replic library, willautomatically turn them into commands available at the applicationprompt.

You can copy the example below in ~/.torrents.lisp and re-run torrents -i to try this out.

(in-package :torrents.user)

(defparameter *names* '()
  "List of names (string) given to `hello`. Will be autocompleted by `goodbye`.")

(defun hello (name)
  "Takes only one argument. Adds the given name to the global
  `*names*` global variable, used to complete arguments of `goodbye`. "
  (format t "hello ~a~&" name)
  (push name *names*))

(defun goodbye (name)
  "Says goodbye to name, where `name` should be completed from what was given to `hello`."
  (format t "goodbye ~a~&" name))

;; Custom completion for goodbye:
(replic.completion:add-completion "goodbye" (lambda () *names*))

;; and export the two functions to find them as commands.
(export '(hello
          goodbye))

0.3Ideas, todos

• cache (on files) v0.3
• CI
• use a local copy of TPB
• build with ECL. Used the lisp-devel dockerimage. Result: a 52Mo executable with a runtime error.
• open an url
• download the torrent file
• ini config file, to parse with py-configparser
• add torrent to a local torrent client
• add the torrent to a remote transmission client, withcl-transmission. v0.12
• self-contained web app (doc). POC: cl-torrents-web.
• interactive prompt, with completion of commands.

0.4Release notes

https://gitlab.com/vindarel/cl-torrents/tags

0.4.1WIP

• added: a simple Tk GUI (search, open in browser, download withtorrent desktop client).

0.4.2v0.12

• june, 2021: added torrents-paradise API.

0.4.3v0.11

• november, 2019: added connection to transmission-remote
• added rarbg scraper (needs new tests)
• june, 2019: changed the location of the lisp init file from~/.torrents.lisp to ~/.config/torrents.lisp.
• added: a filter command, to only display results whose titlecontains a given string. It reduces the TAB-completion of ids (butdoesn't constrain it).
• added: all functions can now TAB-complete the list of ids.
• added: scrape and display torrents' size.
• fixed -d cli arg.
• added: load ~/.torrents.lisp, create new commands from thetorrents.user package.
• added the ability to read an ini-like config file,~/.torrents.conf. All exported variables from the :torrent packagecan be overriden. They are written without earmuffs:

           :*nb-results*
           :*browser*
           :*torrent-client*
           :*cache-p*

gives

# ~/.torrents.conf
# those are the defaults.
[default]
nb-results = 20
browser = firefox
torrent-client = firefox
cache-p = true

See more on replic.(all parameters need more tests)

• added missing -o option to download with a torrent client (transmission-gtk).
• cl-torrents.conf configuration file:
  • choose the list of scrapers.
• download command (open magnet link with transmission by default).
• use the replic library to build the repl, the commands and their completions.
  • -> new builtin highlight and unhighlight commands,
  • -> better error handling (don't exit the repl on an error).
• fix cli usage without a keyword search, print help.

0.4.4v0.10

• added torrentdownloads.me scraper.
• new parameter, *cache-p*: if nil, don't use the cache.

0.4.5v0.9

• added 1337x.to scraper.
• removed piratebay.to, katcr.co, torrent.cd, they're down :/

0.4.6v0.8

• added a readline-based repl with some commands: search, magnet,open, with their completion. See help. Built with cl-readline,but might use replic anytime soon.

0.4.7v0.7

• fix cache not created by binary
• create cache in ~/.cl-torrents/cache/

1Dev

Clone cl-torrents in ~/quicklisp/local-projects,

now you can (ql:quickload :torrents).

We use our replic library to automatically build the readline repl.

Unit tests:

make test

End-to-end tests (checking that the websites respond and our scrapers still work):

make test-end2end

Try the Ltk GUI: load gui-tk.lisp and run (main).

2Resources

Don't miss these good resources:

• http://lisp-lang.org/
• Awesome-cl
• the Common Lisp Cookbook
• http://quickdocs.org/
• (my) lisp-journey: introductory posts, lists of resources, selectionof screencasts.

2.1Tutorial

update, 2021: this tutorial is one of the first things I wrote when discovering CL and it probably bitrot a bit. I'd recommend the Cookbook now as I ported most of its useful content there.

Writing a little web scraper like this one is not difficult. However, I had to spend some time to find out the right libraries and resources. It is also not trivial at first to start a Lisp project. So the first thing we did is write a tutorial. It is a mix of hopefully useful stuff:

• web scraping,
• async web scraping,
• trying out things at the REPL,
• where to find documentation,
• creating and loading a project,
• basic data structures and gotchas,
• some useful libraries,
• unit tests, with mocks, running tests from the shell, continuous integration,
• parsing command line arguments, creating executables, continuous delivery,
• basics of error handling,
• ...

  It will eventually tackle more topics and features (more settings, working with a local copy of TPB…) but in the meanwhile, read

  the tutorial !

  It was built with https://github.com/fniessen/org-html-themes.

2.2Bypassing ISP blocking

Some Internet Service Providers block access to torrent sites, as it's the case in France. They currently don't block the HTTP version of torrents-paradise, but that one works on IPFS anyways. You can bypass them by using other DNS servers than your ISP's.

Here's how to do it on Ubuntu: https://doc.ubuntu-fr.org/dns (french)

You can use OpenNIC addresses or again the ones of the French Data Network association.

3Licence

MIT.