Search for torrents on popular trackers. Lisp library, CLI interface, terminal application, Tk GUI.
This is a little app to search for torrents on popular trackers and to open them with a local or a remote client.
It comes as:
- a lisp library,
- a command line interface,
- a terminal readline interface,
- a simple GUI.
- The [[https://github.com/vindarel/cl-torrents-web][Web UI]] works too.
Download for GNU/Linux 64 bits: see Releases (23MB, self-contained executable).
NOTE: torrents-paradise has a rich collection, a simple website as in KISS, it works on IPFS and has it offers an API… frankly, I want to use its web service and click on its adds.
The library is on Quicklisp (february 2019) and Ultralisp.
(ql:quickload "torrents") (torrents:search-torrents "tears of steel")
Note: the master branch needs cl-transmission which was added to the Quicklisp distribution of december, 2019.
See the download link of the binary above. It is a self-contained executable for GNU/Linux 64-bits. You do not need to install a Lisp implementation. It's a 23MB binary.
To build the readline app, do:
To build the Tk GUI:
./torrents --help ./torrents-tk &
The Tk GUI needs nodgui newer than march, 2019.
If a search doesn't work with a
CRYPTO_num_locks error message:
> ./torrents matrix searching '(matrix)' on 1337... no results. error: The alien function "CRYPTO_num_locks" is undefined. searching '(matrix)' on DOWNLOADSME... no results. error: The alien function "CRYPTO_num_locks" is undefined.
You might need to upgrade your OpenSSL version. See this issue.
We can use this little app both in a Lisp REPL (Slime) and in a terminal, as a command line tool or from its embedded interactive prompt.
Results are cached in
~/.cl-torrents/cache/. Delete this
directory if you want new results.
Search for torrents:
(use-package :torrents) (search-torrents "matrix") [...] 6: The Matrix Revolutions (2003) BRRip [Dual Audio] [Hindi+Eng] 5: Matrix (1999)Blu-Ray 720p Dublado PT-BR - mo93438 4: The Matrix Trilogy (1999-2003) BDRip 1080p Dual Audio [ Hind 3: The Matrix Trilogy (1999-2003) BluRay BDRip 1080p AC3 2: The Matrix Trilogy (1999-2003) + Extras 1080p BluRay x264 Du 1: The Matrix Reloaded (2003) FullHD, Dual Audio: English + Spa 0: Matrix FRENCH DVDRIP 1999 COOL
To help finding the most relevant results (the website sometimes is a bit relaxed with our search terms), each keyword is printed in a different color:
(you need to enable ansi colors in Slime, see slime-repl-ansi-color.
Disable this feature with
(setf cl-ansi-text:*enabled* nil))
We get a magnet link with:
(magnet 0) ;; "magnet:?xt=urn:btih:40eca43690cf1b99b0a4d485ebf4855d20b0bac5"
We can open it with a torrent client (transmission-gtk by default):
Download it with a remote client (only transmission-remote so far):
you need settings, see below. TODO
;; in ~/.config/torrents.lisp (setf *remote-client-url* "my.url.without.port") (setf *remote-client-username* "transmission") (setf *remote-client-password* "password") ;; port is 9091 by default.
And voilà :)
We may add more features, eventually. Our goal is rather to write a tutorial to show you diverse Common Lisp topics.
*cache-p*: if nil, don't use the cache system.
Use the options given below,
$ torrents -h CL-torrents. Usage: Available options: -d, --details print more details (like the torrent's url) -h, --help print this help text -i, --interactive enter an interactive repl -m, --magnet ARG get the magnet link of the given search result. -n, --nb ARG set the maximum number of results to print. -o, --open INDEX download with a torrent client (transmission-gtk by default) -v, --version print the version
Example: below we search for "matrix…", we display only 1 result and we get the magnet link of the first result.
$ torrents -n 1 matrix french 1999 cool -m 0
Start it with
help and the help of each command.
The available commands are (use the auto-completion):
search: search for torrents, print the results sorted by number of seeders.
magnet <i>, where i is a result index,
firefox <i>: open the given torrent page in a web browser
download <i>: open a magnet link with a torrent client(transmission-gtk by default)
filter <text>: show results that have this text in theirtitle. This reduces the tab-completion of ids for all commands.
unhighlight): highlight the given words inyellow, for this session.
nb-results=n= to set the number of results to print to n
details: toggle the display of details
And the available settings to change with
set are (use
*details*: if true, print more details below each row (like theurl). Defaults to false.
*nb-results*: how many results to print. Defaults to 20.
*browser*: the browser name to use to open urls. Defaults to Firefox.
*torrent-client*: the desktop torrent client name. Defaults to Transmission.
*cache-p*: if true (the default), use the cached results.
You can also set them in a configuration file.
Note that we are in a regular readline interface, so the usual keybindings and options are available.
0.2.4Configuration files (in development)
cl-torrents will read two configuration files. An ini-style one,
and a lispy one.
First, it will search for a
~/.torrents.conf. The last
one takes precedence.
For example, those are the default values:
[default] scrapers = 1337 DOWNLOADSME browser = firefox nb-results = 20 cache-p = true # use the cache system.
Moreover, you can set parameters related to the repl:
[default] # again, default values: confirm-exit = true # ask with a yes/no prompt before quiting the app. verbose = false history = true # use the repl history. write-history = true # read the history, but don't write to it.
You can set them for all replic-based apps in
override them in
Note: we only use a "default" profile for now.
0.2.4.2Lisp init file
If the file
~/.torrents.lisp exists, it will be loaded after the
.conf one and before the command line arguments.
--no-userinit prevents it from loading.
You can write whatever you want there, it is just a lisp file that
load'ed at startup.
Note: what we will eventually do then is to expose cl-torrents' mechanism via hooks.
0.2.4.3Extending the app, creating commands
One thing you can do is define additional commands to be found at the repl. We do so by following replic's mechanism:
- define functions inside the
- define the completion choices for some functions
exportthe functions you want to see turned intocommands.
cl-torrents, with the help of the
repliclibrary, willautomatically turn them into commands available at the applicationprompt.
You can copy the example below in
~/.torrents.lisp and re-run
torrents -i to try this out.
(in-package :torrents.user) (defparameter *names* '() "List of names (string) given to `hello`. Will be autocompleted by `goodbye`.") (defun hello (name) "Takes only one argument. Adds the given name to the global `*names*` global variable, used to complete arguments of `goodbye`. " (format t "hello ~a~&" name) (push name *names*)) (defun goodbye (name) "Says goodbye to name, where `name` should be completed from what was given to `hello`." (format t "goodbye ~a~&" name)) ;; Custom completion for goodbye: (replic.completion:add-completion "goodbye" (lambda () *names*)) ;; and export the two functions to find them as commands. (export '(hello goodbye))
- cache (on files) v0.3
- use a local copy of TPB
- build with ECL. Used the lisp-devel dockerimage. Result: a 52Mo executable with a runtime error.
- open an url
- download the torrent file
- ini config file, to parse with py-configparser
- add torrent to a local torrent client
- add the torrent to a remote transmission client, withcl-transmission. v0.12
- self-contained web app (doc). POC: cl-torrents-web.
- interactive prompt, with completion of commands.
- added: a simple Tk GUI (search, open in browser, download withtorrent desktop client).
- june, 2021: added torrents-paradise API.
- november, 2019: added connection to transmission-remote
- added rarbg scraper (needs new tests)
- june, 2019: changed the location of the lisp init file from
- added: a
filtercommand, to only display results whose titlecontains a given string. It reduces the TAB-completion of ids (butdoesn't constrain it).
- added: all functions can now TAB-complete the list of ids.
- added: scrape and display torrents' size.
- added: load
~/.torrents.lisp, create new commands from the
- added the ability to read an ini-like config file,
~/.torrents.conf. All exported variables from the
:torrentpackagecan be overriden. They are written without earmuffs:
:*nb-results* :*browser* :*torrent-client* :*cache-p*
See more on replic.(all parameters need more tests)
# ~/.torrents.conf # those are the defaults. [default] nb-results = 20 browser = firefox torrent-client = firefox cache-p = true
- added missing
-ooption to download with a torrent client (transmission-gtk).
- choose the list of scrapers.
downloadcommand (open magnet link with transmission by default).
- use the replic library to build the repl, the commands and their completions.
- -> new builtin
- -> better error handling (don't exit the repl on an error).
- -> new builtin
- fix cli usage without a keyword search, print help.
- added torrentdownloads.me scraper.
- new parameter,
*cache-p*: if nil, don't use the cache.
- added 1337x.to scraper.
- removed piratebay.to, katcr.co, torrent.cd, they're down :/
- added a readline-based repl with some commands:
open, with their completion. See
help. Built with cl-readline,but might use replic anytime soon.
- fix cache not created by binary
- create cache in
Clone cl-torrents in
now you can
We use our replic library to automatically build the readline repl.
End-to-end tests (checking that the websites respond and our scrapers still work):
Try the Ltk GUI: load
gui-tk.lisp and run
Don't miss these good resources:
- the Common Lisp Cookbook
- (my) lisp-journey: introductory posts, lists of resources, selectionof screencasts.
update, 2021: this tutorial is one of the first things I wrote when discovering CL and it probably bitrot a bit. I'd recommend the Cookbook now as I ported most of its useful content there.
Writing a little web scraper like this one is not difficult. However, I had to spend some time to find out the right libraries and resources. It is also not trivial at first to start a Lisp project. So the first thing we did is write a tutorial. It is a mix of hopefully useful stuff:
- web scraping,
- async web scraping,
- trying out things at the REPL,
- where to find documentation,
- creating and loading a project,
- basic data structures and gotchas,
- some useful libraries,
- unit tests, with mocks, running tests from the shell, continuous integration,
- parsing command line arguments, creating executables, continuous delivery,
- basics of error handling,
It will eventually tackle more topics and features (more settings, working with a local copy of TPB…) but in the meanwhile, read
the tutorial !
It was built with https://github.com/fniessen/org-html-themes.
2.2Bypassing ISP blocking
Some Internet Service Providers block access to torrent sites, as it's the case in France. They currently don't block the HTTP version of torrents-paradise, but that one works on IPFS anyways. You can bypass them by using other DNS servers than your ISP's.
Here's how to do it on Ubuntu: https://doc.ubuntu-fr.org/dns (french)