- -

Configure system logs on Arch to avoid filled up disk

- -

It’s been a while since I’ve been running a minimal server on a VPS, and it is a pretty humble VPS with just 32 GB of storage which works for me as I’m only hosting a handful of services. At some point I started noticing that the disk keept filling up on each time I checked.

Turns out that out of the box, Arch has a default config for systemd‘s journald that keeps a persistent journal log, but doesn’t have a limit on how much logging is kept. This means that depending on how many services, and how aggresive they log, it can be filled up pretty quickly. For me I had around 15 GB of logs, from the normal journal directory, nginx directory and my now unused prosody instance.

For prosody it was just a matter of deleting the directory as I’m not using it anymore, which freed around 4 GB of disk space. -For journal I did a combination of configuring SystemMaxUse and creating a Namespace for all “email” related services as mentioned in the Arch wiki: systemd/Journal; basically just configuring /etc/systemd/journald.conf (and /etc/systemd/journald@email.conf with the comment change) with:

[Journal]
-Storage=persistent
-SystemMaxUse=100MB # 50MB for the "email" Namespace
-

And then for each service that I want to use this “email” Namespace I add:

[Service]
-LogNamespace=email
-

Which can be changed manually or by executing systemctl edit service_name.service and it will create an override file which will be read on top of the normal service configuration. Once configured restart by running systemctl daemon-reload and systemctl restart service_name.service (probably also restart systemd-journald).

I also disabled the logging for ufw by running ufw logging off as it logs everything to the kernel “unit”, and I didn’t find a way to pipe its logs to a separate directory. It really isn’t that useful as most of the logs are the normal [UFW BLOCK] log, which is normal. If I need debugging then I’ll just enable that again. Note that you can change the logging level, if you still want some kind of logging.

Finally to clean up the nginx logs, you need to install logrotate (pacman -S logrotate) as that is what is used to clean up the nginx log directory. nginx already “installs” a config file for logrotate which is located at /etc/logrotate.d/, I just added a few lines:

/var/log/nginx/*log {
-    rotate 7
-    weekly
-    dateext
-    dateformat -%Y-%m-%d
-    missingok
-    notifempty
-    create 640 http log
-    sharedscripts
-    compress
-    postrotate
-        test ! -r /run/nginx.pid || kill -USR1 `cat /run/nginx.pid`
-    endscript
-}
-

Once you’re ok with your config, it’s just a matter of running logrotate -v -f /etc/logrotate.d/nginx which forces the run of the rule for nginx. After this, logrotate will be run daily if you enable the logrotate timer: systemctl enable logrotate.timer.

- -

- - - - Next - - - - - - - Index - - - - - - - Previous - - -

- - -

By David Luévano

Created: Thu, Jun 15, 2023 @ 10:22 UTC

Modified: Thu, Jun 15, 2023 @ 10:24 UTC

Tags: -code, english, server, short, tools, tutorial

- -

Los devs de Android/MIUI me trozaron

- -

Llevo dos semanas posponiendo esta entrada porque andaba bien enojado (todavía, pero ya se anda pasando) y me daba zzz. Pero bueno, antes que nada este pex ocupa un poco de contexto sobre dos cositas:

Tachiyomi: Una aplicación de android que uso para descargar y leer manga. Lo importante aquí es que por default se guardan los mangas con cada página siendo una sola imagen, por lo que al mover el manga de un lado a otro tarda mucho tiempo.
Adoptable storage: Un feature de android que básicamente te deja usar una micro SD (mSD) externa como si fuera interna, encriptando y dejando la mSD inutilizable en cualquier otro dispositivo. La memoria interna se pierde o algo por el estilo (bajo mi experiencia), por lo que parece es bastante útil cuando la capacidad de la memoria interna es baja.

Ahora sí vamonos por partes, primero que nada lo que sucedió fue que ordené una mSD con más capacidad que la que ya tenía (64 GB -> 512 GB, poggies), porque últimamente he estado bajando y leyendo mucho manga entonces me estaba quedando sin espacio. Ésta llegó el día de mi cumpleaños lo cuál estuvo chingón, me puse a hacer backup de la mSD que ya tenía y preparando todo, muy bonito, muy bonito.

Empecé a tener problemas, porque al estar moviendo tanto archivo pequeño (porque recordemos que el tachiyomi trata a cada página como una sola imagen), la conexión entre el celular y mi computadora se estaba corte y corte por alguna razón; en general muchos pedos. Por lo que mejor le saqué la nueva mSD y la metí directo a mi computadora por medio de un adaptador para batallar menos y que fuera más rápido.

Hacer este pedo de mover archivos directamente en la mSD puede llevar a corromper la memoria, no se los detalles pero pasa (o quizá estoy meco e hice algo mal). Por lo que al terminar de mover todo a la nueva mSD y ponerla en el celular, éste se emputó que porque no la detectaba y que quería tirar un formateo a la mSD. A este punto no me importaba mucho, sólo era questión de volvera mover archivos y ser más cuidadoso; “no issues from my end” diría en mis standups.

Todo valió vergota porque en cierto punto al elegir sí formatear la mSD mi celular me daba la opción de “usar la micro SD para el celular” o “usar la micro SD como memoria portátil” (o algo entre esas líneas), y yo, estúpidamente, elegí la primera, porque me daba sentido: “no, pues simón, voy a usar esta memoria para este celular”.

Pues mamé, resulta que esa primera opción lo que realmente quería decir es que se iba a usar la micro SD como interna usando el pex este de adoptable storage. Entonces básicamente perdí mi capacidad de memoria interna (128 GB aprox.), y toda la mSD nueva se usó como memoria interna. Todo se juntó, si intentaba sacar la mSD todo se iba a la mierda y no podía usar muchas aplicaciones. “No hay pedo”, pensé, “nada más es cuestión de desactivar esta mamada de adoptable storage”.

Ni madres dijeron los devs de Android, este pedo nada más es un one-way: puedes activar adoptable storage pero para desactivarlo ocupas, a huevo, formatear tu celular a estado de fábrica. Chingué a mi madre, comí mierda, perdí.

Pues eso fue lo que hice, ni modo. Hice backup de todo lo que se me ocurrió (también me di cuenta que G**gl* authenticator es cagada ya que no te deja hacer backup, entre otras cosas, mejor usen Aegis authenticator), desactivé todo lo que se tenía que desactivar y tocó hacer factory reset, ni modo. Pero como siempre las cosas salen mal y tocó comer mierda del banco porque me bloquearon la tarjeta, perdí credenciales necesarias para el trabajo (se resolvió rápido), etc., etc.. Ya no importa, ya casi todo está resuelto, sólo queda ir al banco a resolver lo de la tarjeta bloqueada (esto es para otro rant, pinches apps de bancos piteras, ocupan hacer una sola cosa y la hacen mal).

Al final del día, la causa del problema fueron los malditos mangas (por andar queriendo backupearlos), que terminé bajando de nuevo manualmente y resultó mejor porque aparentemente tachiyomi agregó la opción de “zippear” los mangas en formato CBZ, por lo que ya son más fácil de mover de un lado para otro, el fono no se queda pendejo, etc., etc..

Por último, quiero decir que los devs de Android son unos pendejos por no hacer reversible la opción de adoptable storage, y los de MIUI son todavía más por no dar detalles de lo que significan sus opciones de formateo, especialmente si una opción es tan chingadora que para revertirla necesitas formatear a estado de fábrica tu celular; más que nada es culpa de los de MIUI, todavía que ponen un chingo de A(i)DS en todas sus apps, no pueden poner una buena descripción en sus opciones. REEEE.

- -

- - - - Next - - - - - - - Index - - - - - - - Previous - - -

- - -

By David Luévano

Created: Sun, May 15, 2022 @ 09:51 UTC

Modified: Sun, May 15, 2022 @ 10:03 UTC

Tags: -rant, spanish, update

- -

Set up a Git server and cgit front-end

- -

My git server is all I need to setup to actually kill my other server (I’ve been moving from servers on these last 2-3 blog entries), that’s why I’m already doing this entry. I’m basically following git’s guide on setting up a server plus some specific stuff for btw i use Arch Linux (Arch Linux Wiki: Git server and Step by step guide on setting up git server in arch linux (pushable)).

Note that this is mostly for personal use, so there’s no user/authentication control other than that of normal ssh. And as with the other entries, most if not all commands here are run as root unless stated otherwise.

Table of contents¶

Table of contents
Prerequisites
Git
Cgit
- Cgit’s file rendering
-

Prerequisites¶

I might get tired of saying this (it’s just copy paste, basically)… but you will need the same prerequisites as before (check my website and mail entries), with the extras:

(Optional, if you want a “front-end”) A CNAME for “git” and (optionally) “www.git”, or some other name for your sub-domains.
An SSL certificate, if you’re following the other entries, add a git.conf and run certbot --nginx to extend the certificate.

Git¶

Git is a version control system.

If not installed already, install the git package:

pacman -S git
-

On Arch Linux, when you install the git package, a git user is automatically created, so all you have to do is decide where you want to store the repositories, for me, I like them to be on /home/git like if git was a “normal” user. So, create the git folder (with corresponding permissions) under /home and set the git user’s home to /home/git:

mkdir /home/git
-chown git:git /home/git
-usermod -d /home/git git
-

Also, the git user is “expired” by default and will be locked (needs a password), change that with:

chage -E -1 git
-passwd git
-

Give it a strong one and remember to use PasswordAuthentication no for ssh (as you should). Create the .ssh/authorized_keys for the git user and set the permissions accordingly:

mkdir /home/git/.ssh
-chmod 700 /home/git/.ssh
-touch /home/git/.ssh/authorized_keys
-chmod 600 /home/git/.ssh/authorized_keys
-chown -R git:git /home/git
-

Now is a good idea to copy over your local SSH public keys to this file, to be able to push/pull to the repositories. Do it by either manually copying it or using ssh‘s built in ssh-copy-id (for that you may want to check your ssh configuration in case you don’t let people access your server with user/password).

Next, and almost finally, we need to edit the git-daemon service, located at /usr/lib/systemd/system/ (called git-daemon@.service):

...
-ExecStart=-/usr/lib/git-core/git-daemon --inetd --export-all --base-path=/home/git --enable=receive-pack
-...
-

I just appended --enable=receive-pack and note that I also changed the --base-path to reflect where I want to serve my repositories from (has to match what you set when changing git user’s home).

Now, go ahead and start and enable the git-daemon socket:

systemctl start git-daemon.socket
-systemctl enable git-daemon.socket
-

You’re basically done. Now you should be able to push/pull repositories to your server… except, you haven’t created any repository in your server, that’s right, they’re not created automatically when trying to push. To do so, you have to run (while inside /home/git):

git init --bare {repo_name}.git
-chown -R git:git {repo_name}.git
-

Those two lines above will need to be run each time you want to add a new repository to your server. There are options to “automate” this but I like it this way.

After that you can already push/pull to your repository. I have my repositories (locally) set up so I can push to more than one remote at the same time (my server, GitHub, GitLab, etc.); to do so, check this gist.

Cgit¶

Cgit is a fast web interface for git.

This is optionally since it’s only for the web application.

Install the cgit and fcgiwrap packages:

pacman -S cgit fcgiwrap
-

Now, just start and enable the fcgiwrap socket:

systemctl start fcgiwrap.socket
-systemctl enable fcgiwrap.socket
-

Next, create the git.conf as stated in my nginx setup entry. Add the following lines to your git.conf file:

server {
-    listen 80;
-    listen [::]:80;
-    root /usr/share/webapps/cgit;
-    server_name {yoursubdomain}.{yourdomain};
-    try_files $uri @cgit;
-
-    location @cgit {
-        include fastcgi_params;
-        fastcgi_param SCRIPT_FILENAME $document_root/cgit.cgi;
-        fastcgi_param PATH_INFO $uri;
-        fastcgi_param QUERY_STRING $args;
-        fastcgi_param HTTP_HOST $server_name;
-        fastcgi_pass unix:/run/fcgiwrap.sock;
-    }
-}
-

Where the server_name line depends on you, I have mine setup to git.luevano.xyz and www.git.luevano.xyz. Optionally run certbot --nginx to get a certificate for those domains if you don’t have already.

Now, all that’s left is to configure cgit. Create the configuration file /etc/cgitrc with the following content (my personal options, pretty much the default):

css=/cgit.css
-logo=/cgit.png
-
-enable-http-clone=1
-# robots=noindex, nofollow
-virtual-root=/
-
-repo.url={url}
-repo.path={dir_path}
-repo.owner={owner}
-repo.desc={short_description}
-...
-

Where you can uncomment the robots line to not let web crawlers (like Google’s) to index your git web app. And at the end keep all your repositories (the ones you want to make public), for example for my dotfiles I have:

...
-repo.url=.dots
-repo.path=/home/git/.dots.git
-repo.owner=luevano
-repo.desc=These are my personal dotfiles.
-...
-

Otherwise you could let cgit to automatically detect your repositories (you have to be careful if you want to keep “private” repos) using the option scan-path and setup .git/description for each repository. For more, you can check cgitrc(5).

Cgit’s file rendering¶

By default you can’t see the files on the site, you need a highlighter to render the files, I use highlight. Install the highlight package:

pacman -S highlight
-

Copy the syntax-highlighting.sh script to the corresponding location (basically adding -edited to the file):

cp /usr/lib/cgit/filters/syntax-highlighting.sh /usr/lib/cgit/filters/syntax-highlighting-edited.sh
-

And edit it to use the version 3 and add --inline-css for more options without editing cgit‘s CSS file:

...
-# This is for version 2
-# exec highlight --force -f -I -X -S "$EXTENSION" 2>/dev/null
-
-# This is for version 3
-exec highlight --force --inline-css -f -I -O xhtml -S "$EXTENSION" 2>/dev/null
-...
-

Finally, enable the filter in /etc/cgitrc configuration:

source-filter=/usr/lib/cgit/filters/syntax-highlighting-edited.sh
-

That would be everything. If you need support for more stuff like compressed snapshots or support for markdown, check the optional dependencies for cgit.

- -

- - - - Next - - - - - - - Index - - - - - - - Previous - - -

- - -

By David Luévano

Created: Sun, Mar 21, 2021 @ 19:00 UTC

Modified: Sat, Jun 03, 2023 @ 03:52 UTC

Tags: -code, english, server, tools, tutorial

- -

Hoy me tocó desarrollo de personaje

- -

Sabía que hoy no iba a ser un día tan bueno, pero no sabía que iba a estar tan horrible; me tocó desarrollo de personaje y saqué el bad ending.

Básicamente tenía que cumplir dos misiones hoy: ir al banco a un trámite y vacunarme contra el Covid-19. Muy sencillas tareas.

Primero que nada me levanté de una pesadilla horrible en la que se puede decir que se me subió el muerto al querer despertar, esperé a que fuera casi la hora de salida de mi horario de trabajo, me bañé y fui directo al banco primero. Todo bien hasta aquí.

En el camino al banco, durante la plática con el conductor del Uber salió el tema del horario del banco. Yo muy tranquilo dije “pues voy algo tarde, pero sí alcanzo, cierran a las 5, ¿no?” a lo que me respondió el conductor “nel jefe, a las 4, y se van media hora antes”; quedé. Chequé y efectivamente cerraban a las 4. Entonces le dije que le iba a cambiar la ruta directo a donde me iba a vacunar, pero ya era muy tarde y quedaba para la dirección opuesta.”Ni pedo, ahí déjame y pido otro viaje, no te apures”, le dije y como siempre pues me deseó que se compusiera mi día; afortunadamente el banco sí estaba abierto para lo que tenía que hacer, así que fue un buen giro. Me puse muy feliz y asumí que sería un buen día, como me lo dijo mi conductor; literalmente NO SABÍA.

Salí feliz de poder haber completado esa misión y poder irme a vacunar. Pedí otro Uber a donde tenía que ir y todo bien. Me tocó caminar mucho porque la entrada estaba en punta de la chingada de donde me dejó el conductor, pero no había rollo, era lo de menos. Me desanimé cuando vi que había una cantidad estúpida de gente, era una fila que abarcaba todo el estacionamiento y daba demasiadas vueltas; “ni pedo”, dije, “si mucho me estaré aquí una hora, hora y media”… otra vez, literalmente NO SABÍA.

Pasó media hora y había avanzado lo que parecía ser un cuarto de la fila, entonces todo iba bien. Pues nel, había avanzado el equivalente a un octavo de la fila, este pedo no iba a salir en una hora-hora y media. Para acabarla de chingar era todo bajo el tan amado sol de Chiwawa. “No hay pedo, me entretengo tirando chal con alguien en el wasap”, pues no, aparentemente no cargué el celular y ya tenía 15-20% de batería… volví a quedar.

Se me acabó la pila, ya había pasado una hora y parecía que la fila era infinita, simplemente avanzábamos demasiado lento, a pesar de que los que venían atrás de mí repetían una y otra vez “mira, avanza bien rápido, ya mero llegamos”, ilusos. Duré aproximadamente 3 horas formado, aguantando conversaciones estúpidas a mi alrededor, gente quejándose por estar parada (yo también me estaba quejando pero dentro de mi cabeza), y por alguna razón iban familias completas de las cuales al final del día sólo uno o dos integrantes de la familia entraban a vacunarse.

En fin que se acabó la tortura y ya tocaba irse al cantón, todo bien. “No hay pedo, no me tocó irme en Uber, aquí agarro un camíon” pensé. Pero no, ningún camión pasó durante la hora que estuve esperando y de los 5 taxis que intenté parar NINGUNO se detuvo. Decidí irme caminado, ya qué más daba, en ese punto ya nada más era hacer corajes dioquis.

En el camino vi un Oxxo y decidí desviarme para comprar algo de tomar porque andaba bien deshidratado. En el mismo segundo que volteé para ir hacia el Oxxo pasó un camión volando y lo único que pensaba era que el conductor me decía “Jeje ni pedo:)”. Exploté, me acabé, simplemente perdí, saqué el bad ending.

Ya estaba harto y hasta iba a comprar un cargador para ya irme rápido, estaba cansado del día, simplemente ahí terminó la quest, había sacado el peor final. Lo bueno es que se me ocurrió pedirle al cajero un cargador y que me tirara paro. Todo bien, pedí mi Uber y llegué a mi casa sano y a salvo, pero con la peor rabia que me había dado en mucho tiempo. Simplemente ¿mi culo? explotado. Este día me tocó un desarrollo de personaje muy cabrón, se mamó el D*****o.

Lo único rescatable fue que había una (más bien como 5) chica muy guapa en la fila, lástima que los stats de mi personaje me tienen bloqueadas las conversaciones con desconocidos.

Y pues ya, este pex ya me sirvió para desahogarme, una disculpa por la redacción tan pitera. Sobres.

- -

- - - - Next - - - - - - - Index - - - - - - - Previous - - -

- - -

By David Luévano

Created: Wed, Jul 28, 2021 @ 06:10 UTC

Modified: Sat, May 06, 2023 @ 12:56 UTC

Tags: -rant, spanish, update

- -

Set up a media server with Jellyfin, Sonarr and Radarr

- -

Second part of my self hosted media server. This is a direct continuation of Set up qBitTorrent with Jackett for use with Starr apps, which will be mentioned as “first part” going forward. Sonarr, Radarr, Bazarr (Starr apps) and Jellyfin setups will be described in this part. Same introduction applies to this entry, regarding the use of documentation and configuration.

Everything here is performed in arch btw and all commands should be run as root unless stated otherwise.

Kindly note that I do not condone the use of BitTorrent for illegal activities. I take no responsibility for what you do when setting up anything shown here. It is for you to check your local laws before using automated downloaders such as Sonarr and Radarr.

Table of contents¶

Table of contents
Prerequisites
Radarr
- Reverse proxy
- Start using Radarr
  - Configuration
    - Media Management
    - Quality
    - Custom Formats
    - Profiles
    - Download clients
    - Indexers
    -
  - Download content
  -
-
Sonarr
- Reverse proxy
- Start using Sonarr
  - Configuration
    - Media Management
    - Quality
    - Profiles
    - Download clients
    - Indexers
    -
  - Download content
  -
-
Jellyfin
- Reverse proxy
  - SSL certificate
  -
- Start using Jellyfin
  - Plugins
  - Transcoding
    - NVIDIA drivers
    - Enable hardware acceleration
    -
  -
-
Bazarr
- Reverse proxy
- Start using Bazarr
  - Configuration
    - Providers
    -
  -
-

Prerequisites¶

Same prerequisites as with the First part: Prerequisites plus:

An A (and/or AAAA) or a CNAME for jellyfin. Only if you want to expose Jellyfin to a subdomain.

The First part: Directory structure is the same here. The servarr user and group should be available, too.

It is assumed that the first part was followed.

Radarr¶

Radarr is a movie collection manager that can be used to download movies via torrents. This is actually a fork of Sonarr, so they’re pretty similar, I just wanted to set up movies first.

Install from the AUR with yay:

yay -S radarr
-

Add the radarr user to the servarr group:

gpasswd -a radarr servarr
-

The default port that Radarr uses is 7878 for http (the one you need for the reverse proxy).

Reverse proxy¶

Add the following location blocks into the isos.conf with whatever subdirectory name you want, I’ll leave it as radarr:

location /radarr/ {
-    proxy_pass http://127.0.0.1:7878/radarr/; # change port if needed
-    proxy_http_version 1.1;
-
-    proxy_set_header Host $host;
-    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-    proxy_set_header X-Forwarded-Host $host;
-    proxy_set_header X-Forwarded-Proto $scheme;
-    proxy_set_header Upgrade $http_upgrade;
-    proxy_set_header Connection $http_connection;
-
-    proxy_redirect off;
-}
-# Allow the API External Access via NGINX
-location /radarr/api {
-    auth_basic off;
-    proxy_pass http://127.0.0.1:7878/radarr/api; # change port if needed
-}
-

This is taken from Radarr Nginx reverse proxy configuration. Restart the nginx service for the changes to take effect:

systemctl restart nginx.service
-

Start using Radarr¶

You can now start/enable the radarr.service:

systemctl enable radarr.service
-systemctl start radarr.service
-

This will start the service and create the default configs under /var/lib/radarr. You need to change the URLBase as the reverse proxy is under a subdirectory (/radarr). Edit /var/lib/radarr/config.xml:

...
-<UrlBase>/radarr</UrlBase>
-...
-

Then restart the radarr service:

systemctl restart radarr.service
-

Now https://isos.yourdomain.com/radarr is accessible. Secure the instance right away by adding authentication under Settings -> General -> Security. I added the “Forms” option, just fill in the username and password then click on save changes on the top left of the page. You can restart the service again and check that it asks for login credentials.

Note that if you want to have an anime movies library, it is recommended to run a second instance of Radarr for this as shown in Radarr: Linux multiple instances and follow TRaSH: How to setup quality profiles anime if an anime instance is what you want.

Configuration¶

Will be following the official Radarr: Quick start guide as well as the recommendations by TRaSH: Radarr.

Anything that is not mentioned in either guide or that is specific to how I’m setting up stuff will be stated below.

Media Management¶

File Management:
- Propers and Repacks: set it to “Do Not Prefer” and instead you’ll use the Repack/Proper custom format by TRaSH.
-

Quality¶

This is personal preference and it dictates your preferred file sizes. You can follow TRaSH: Quality settings to maximize the quality of the downloaded content and restrict low quality stuff.

Personally, I think TRaSH’s quality settings are a bit elitist and first world-y. I’m fine with whatever and the tracker I’m using has the quality I want anyways. I did, however, set it to a minimum of 0 and maximum of 400 for the qualities shown in TRaSH’s guide. Configuring anything below 720p shouldn’t be necessary anyways.

Custom Formats¶

Again, this is also completely a personal preference selection and depends on the quality and filters you want. My custom format selections are mostly based on TRaSH: HD Bluray + WEB quality profile.

The only Unwanted format that I’m not going to use is the Low Quality (LQ) as it blocks one of the sources I’m using to download a bunch of movies. The reasoning behind the LQ custom format is that these release groups don’t care much about quality (they keep low file sizes) and name tagging, which I understand but I’m fine with this as I can upgrade movies individually whenever I want (I want a big catalog of content that I can quickly watch).

Profiles¶

As mentioned in Custom Formats and Quality this is completly a personal preference. I’m going to go for “Low Quality” downloads by still following some of the conventions from TRaSH. I’m using the TRaSH: HD Bluray + WEB quality profile with the exclusion of the LQ profile.

I set the name to “HD Bluray + WEB”. I’m also not upgrading the torrents for now. Language set to “Original”.

Download clients¶

Pretty straight forward, just click on the giant “+” button and click on the qBitTorrent option. Then configure:

Name: can be anything, just an identifier.
Enable: enable it.
Host: use 127.0.0.1. For some reason I can’t make it work with the reverse proxied qBitTorrent.
Port: the port number you chose, 30000 in my case.
Url Base: leave blank as even though you have it exposed under /qbt, the service itself is not.
Username: the Web UI username, admin by default.
Password: the Web UI username, adminadmin by default (you should’ve changed it if you have the service exposed).
Category: movies.

Everything else can be left as default, but maybe change Completed Download Handling if you’d like. Same goes for the general Failed Download Handling download clients’ option.

Indexers¶

Also easy to set up, also just click on the giant “+” button and click on the custom Torznab option (you can also use the preset -> Jackett Torznab option). Then configure:

Name: can be anything, just an identifier. I like to do “Jackett - INDEXER”, where “INDEXER” is just an identifier.
URL: http://127.0.0.1:9117/jack/api/v2.0/indexers/YOURINDEXER/results/torznab/, where YOURINDEXER is specific to each indexer (yts, nyaasi, etc.). Can be directly copied from the indexer’s “Copy Torznab Feed” button on the Jackett Web UI.
API Path: /api, leave as is.
API Key: this can be found at the top right corner in Jackett’s Web UI.
Categories: which categories to use when searching, these are generic categories until you test/add the indexer. After you add the indexer you can come back and select your prefered categories (like just toggling the movies categories).
Tags: I like to add a tag for the indexer name like yts or nyaa. This is useful to control which indexers to use when adding new movies.

Everything else on default. Download Client can also be set, which can be useful to keep different categories per indexer or something similar. Seed Ratio and Seed Time can also be set and are used to manage when to stop the torrent, this can also be set globally on the qBitTorrent Web UI, this is a personal setting.

Download content¶

You can now start to download content by going to Movies -> Add New. Basically just follow the Radarr: How to add a movie guide. The screenshots from the guide are a bit outdated but it contains everything you need to know.

I personally use:

Monitor: Movie Only.
Minimum Availability: Released.
Quiality Profile: “HD Bluray + WEB”, the one configured in this entry.
Tags: the indexer name I want to use to download the movie, usually just yts for me (remember this is a “LQ” release group, so if you have that custom format disable it) as mentioned in Indexers. If you don’t specify a tag it will only use indexers that don’t have a tag set.
Start search for missing movie: toggled on. Immediatly start searching for the movie and start the download.

Once you click on “Add Movie” it will add it to the Movies section and start searching and selecting the best torrent it finds, according to the “filters” (quality settings, profile and indexer(s)).

When it selects a torrent it sends it to qBitTorrent and you can even go ahead and monitor it over there. Else you can also monitor at Activity -> Queue.

After the movie is downloaded and processed by Radarr, it will create the appropriate hardlinks to the media/movies directory, as set in First part: Directory structure.

Optionally, you can add subtitles using Bazarr.

Sonarr¶

Sonarr is a TV series collection manager that can be used to download series via torrents. Most of the install process, configuration and whatnot is going to be basically the same as with Radarr.

Install from the AUR with yay:

yay -S sonarr
-

Add the sonarr user to the servarr group:

gpasswd -a sonarr servarr
-

The default port that Radarr uses is 8989 for http (the one you need for the reverse proxy).

Reverse proxy¶

Basically the same as with Radarr: Reverse proxy, except that the proxy_set_header changes from $proxy_host to $host.

Add the following location blocks into the isos.conf, I’ll leave it as sonarr:

location /sonarr/ {
-    proxy_pass http://127.0.0.1:8989/sonarr/; # change port if needed
-    proxy_http_version 1.1;
-
-    proxy_set_header Host $proxy_host; # this differs from the radarr reverse proxy
-    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-    proxy_set_header X-Forwarded-Host $host;
-    proxy_set_header X-Forwarded-Proto $scheme;
-    proxy_set_header Upgrade $http_upgrade;
-    proxy_set_header Connection $http_connection;
-
-    proxy_redirect off;
-}
-# Allow the API External Access via NGINX
-location /sonarr/api {
-    auth_basic off;
-    proxy_pass http://127.0.0.1:8989/sonarr/api; # change port if needed
-}
-

This is taken from Sonarr: Nginx reverse proxy configuration. Restart the nginx service for the changes to take effect:

systemctl restart nginx.service
-

Start using Sonarr¶

You can now start/enable the sonarr.service:

systemctl enable sonarr.service
-systemctl start sonarr.service
-

This will start the service and create the default configs under /var/lib/sonarr. You need to change the URLBase as the reverse proxy is under a subdirectory (/sonarr). Edit /var/lib/sonarr/config.xml:

...
-<UrlBase>/sonarr</UrlBase>
-...
-

Then restart the sonarr service:

systemctl restart sonarr.service
-

Now https://isos.yourdomain.com/sonarr is accessible. Secure the instance right away by adding authentication under Settings -> General -> Security. I added the “Forms” option, just fill in the username and password then click on save changes on the top left of the page. You can restart the service again and check that it asks for login credentials.

Similar to Radarr if you want to have an anime library, it is recommended to run a second instance of Sonarr for this as shown in Sonarr: Linux multiple instances and follow TRaSH: Release profile regex (anime) and the TRaSH: Anime recommended naming scheme if an anime instance is what you want.

Configuration¶

Will be following the official Sonarr: Quick start guide as well as the recommendations by TRaSH: Sonarr.

Anything that is not mentioned in either guide or that is specific to how I’m setting up stuff will be stated below.

Media Management¶

File Management:
- Propers and Repacks: set it to “Do Not Prefer” and instead you’ll use the Propers and Repacks release profile and fill with P2P Groups + Repack/Proper.
-

Quality¶

Similar to Radarr: Quality this is personal preference and it dictates your preferred file sizes. You can follow TRaSH: Quality settings to maximize the quality of the downloaded content and restrict low quality stuff.

Will basically do the same as in Radarr: Quality: set minimum of 0 and maximum of 400 for everything 720p and above.

Profiles¶

This is a bit different than with Radarr, the way it is configured is by setting “Release profiles”. I took the profiles from TRaSH: WEB-DL Release profile regex. The only possible change I’ll do is disable the Low Quality Groups and/or the “Golden rule” filter (for x265 encoded video).

For me it ended up looking like this:

But yours can differ as its mostly personal preference. For the “Quality profile” I’ll be using the default “HD-1080p” most of the time, but I also created a “HD + WEB (720/1080)” which works best for some.

Download clients¶

Exactly the same as with Radarr: Download clients only change is the category from movies to tv (or whatever you want), click on the giant “+” button and click on the qBitTorrent option. Then configure:

Name: can be anything, just an identifier.
Enable: enable it.
Host: use 127.0.0.1.
Port: the port number you chose, 30000 in my case.
Url Base: leave blank as even though you have it exposed under /qbt, the service itself is not.
Username: the Web UI username, admin by default.
Password: the Web UI username, adminadmin by default (you should’ve changed it if you have the service exposed).
Category: tv.

Everything else can be left as default, but maybe change Completed Download Handling if you’d like. Same goes for the general Failed Download Handling download clients’ option.

Indexers¶

Also exactly the same as with Radarr: Indexers, click on the giant “+” button and click on the custom Torznab option (this doesn’t have the Jackett preset). Then configure:

Name: can be anything, just an identifier. I like to do “Jackett - INDEXER”, where “INDEXER” is just an identifier.
URL: http://127.0.0.1:9117/jack/api/v2.0/indexers/YOURINDEXER/results/torznab/, where YOURINDEXER is specific to each indexer (eztv, nyaasi, etc.). Can be directly copied from the indexer’s “Copy Torznab Feed” button on the Jackett Web UI.
API Path: /api, leave as is.
API Key: this can be found at the top right corner in Jackett’s Web UI.
Categories: which categories to use when searching, these are generic categories until you test/add the indexer. After you add the indexer you can come back and select your prefered categories (like just toggling the TV categories).
Tags: I like to add a tag for the indexer name like eztv or nyaa. This is useful to control which indexers to use when adding new series.

Download content¶

Almost the same as with Radarr: Download content, but I’ve been personally selecting the torrents I want to download for each season/episode so far, as the indexers I’m using are all over the place and I like consistencies. Will update if I find a (near) 100% automation process, but I’m fine with this anyways as I always monitor that everything is going fine.

Add by going to Series -> Add New. Basically just follow the Sonarr: Library add new guide. Adding series needs a bit more options that movies in Radarr, but it’s straight forward.

I personally use:

Monitor: All Episodes.
Quiality Profile: “HD + WEB (720/1080)”. This depends on what I want for that how, lately I’ve been experimenting with this one.
Series Type: Standard. For now I’m just downloading shows, but it has an Anime option.
Tags: the “indexer_name” I want to use to download the movie, I’ve been using all indexers so I just use all tags as I’m experimenting and trying multiple options.
Season Folder: enabled. I like as much organization as possible.
Start search for missing episodes: disabled. Depends on you, due to my indexers, I prefer to check manually the season packs, for example.
Start search for cutoff unmet episodes: disabled. Honestly don’t really know what this is.

Once you click on “Add X” it will add it to the Series section and will start as monitored. So far I haven’t noticed that it immediately starts downloading (because of the “Start search for missing episodes” setting) but I always click on unmonitor the series, so I can manually check (again, due to the low quality of my indexers).

When it automatically starts to download an episode/season it will send it to qBitTorrent and you can monitor it over there. Else you can also monitor at Activity -> Queue. Same thing goes if you download manually each episode/season via the interactive search.

To interactively search episodes/seasons go to Series and then click on any series, then click either on the interactive search button for the episode or the season, it is an icon of a person as shown below:

Then it will bring a window with the search results, where it shows the indexer it got the result from, the size of the torrent, peers, language, quality, the score it received from the configured release profiles an alert in case that the torrent is “bad” and the download button to manually download the torrent you want. An example shown below:

After the movie is downloaded and processed by Sonarr, it will create the appropriate hardlinks to the media/tv directory, as set in Directory structure.

Optionally, you can add subtitles using Bazarr.

Jellyfin¶

Jellyfin is a media server “manager”, usually used to manage and organize video content (movies, TV series, etc.) which could be compared with Plex or Emby for example (take them as possible alternatives).

Install from the AUR with yay:

yay -S jellyfin-bin
-

I’m installing the pre-built binary instead of building it as I was getting a lot of errors and the server was even crashing. You can try installing jellyfin instead.

Add the jellyfin user to the servarr group:

gpasswd -a jellyfin servarr
-

You can already start/enable the jellyfin.service which will start at http://127.0.0.1:8096/ by default where you need to complete the initial set up. But let’s create the reverse proxy first then start everything and finish the set up.

Reverse proxy¶

I’m going to have my jellyfin instance under a subdomain with an nginx reverse proxy as shown in the Arch wiki. For that, create a jellyfin.conf at the usual sites-<available/enabled> path for nginx:

server {
-    listen 80;
-    server_name jellyfin.yourdomain.com; # change accordingly to your wanted subdomain and domain name
-    set $jellyfin 127.0.0.1; # jellyfin is running at localhost (127.0.0.1)
-
-    # Security / XSS Mitigation Headers
-    add_header X-Frame-Options "SAMEORIGIN";
-    add_header X-XSS-Protection "1; mode=block";
-    add_header X-Content-Type-Options "nosniff";
-
-    # Content Security Policy
-    # See: https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP
-    # Enforces https content and restricts JS/CSS to origin
-    # External Javascript (such as cast_sender.js for Chromecast) must be whitelisted.
-    add_header Content-Security-Policy "default-src https: data: blob: http://image.tmdb.org; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' https://www.gstatic.com/cv/js/sender/v1/cast_sender.js https://www.youtube.com blob:; worker-src 'self' blob:; connect-src 'self'; object-src 'none'; frame-ancestors 'self'";
-
-        location = / {
-        return 302 https://$host/web/;
-    }
-
-    location / {
-        # Proxy main Jellyfin traffic
-        proxy_pass http://$jellyfin:8096;
-        proxy_set_header Host $host;
-        proxy_set_header X-Real-IP $remote_addr;
-        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-        proxy_set_header X-Forwarded-Proto $scheme;
-        proxy_set_header X-Forwarded-Protocol $scheme;
-        proxy_set_header X-Forwarded-Host $http_host;
-
-        # Disable buffering when the nginx proxy gets very resource heavy upon streaming
-        proxy_buffering off;
-    }
-
-    # location block for /web - This is purely for aesthetics so /web/#!/ works instead of having to go to /web/index.html/#!/
-    location = /web/ {
-        # Proxy main Jellyfin traffic
-        proxy_pass http://$jellyfin:8096/web/index.html;
-        proxy_set_header Host $host;
-        proxy_set_header X-Real-IP $remote_addr;
-        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-        proxy_set_header X-Forwarded-Proto $scheme;
-        proxy_set_header X-Forwarded-Protocol $scheme;
-        proxy_set_header X-Forwarded-Host $http_host;
-    }
-
-    location /socket {
-        # Proxy Jellyfin Websockets traffic
-        proxy_pass http://$jellyfin:8096;
-        proxy_http_version 1.1;
-        proxy_set_header Upgrade $http_upgrade;
-        proxy_set_header Connection "upgrade";
-        proxy_set_header Host $host;
-        proxy_set_header X-Real-IP $remote_addr;
-        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-        proxy_set_header X-Forwarded-Proto $scheme;
-        proxy_set_header X-Forwarded-Protocol $scheme;
-        proxy_set_header X-Forwarded-Host $http_host;
-    }
-}
-

SSL certificate¶

Create/extend the certificate by running:

certbot --nginx
-

Similarly to the isos subdomain, that will autodetect the new subdomain and extend the existing certificate(s). Restart the nginx service for changes to take effect:

systemctl restart nginx.service
-

Start using Jellyfin¶

You can now start/enable the jellyfin.service if you haven’t already:

systemctl enable jellyfin.service
-systemctl start jellyfin.service
-

Then navigate to https://jellyfin.yourdomain.com and either continue with the set up wizard if you didn’t already or continue with the next steps to configure your libraries.

The initial setup wizard makes you create an user (will be the admin for now) and at least one library, though these can be done later. For more check Jellyfin: Quick start.

Remember to use the configured directory as mentioned in Directory structure. Any other configuration (like adding users or libraries) can be done at the dashboard: click on the 3 horizontal lines on the top left of the Web UI then navigate to Administration -> Dashboard. I didn’t configure much other than adding a couple of users for me and friends, I wouldn’t recommend using the admin account to watch (personal preference).

Once there is at least one library it will show at Home along with the latest movies (if any) similar to the following (don’t judge, these are just the latest I added due to friend’s requests):

And inside the “Movies” library you can see the whole catalog where you can filter or just scroll as well as seeing Suggestions (I think this starts getting populated after a while) and Genres:

Plugins¶

You can also install/activate plugins to get extra features. Most of the plugins you might want to use are already available in the official repositories and can be found in the “Catalog”. There are a lot of plugins that are focused around anime and TV metadata, as well as an Open Subtitles plugin to automatically download missing subtitles (though this is managed with Bazarr).

To activate plugins click on the 3 horizontal lines on the top left of the Web UI then navigate to Administration -> Dashboard -> Advanced -> Plugins and click on the Catalog tab (top of the Web UI). Here you can select the plugins you want to install. By default only the official ones are shown, for more you can add more repositories.

The only plugin I’m using is the “Playback Reporting”, to get a summary of what is being used in the instance. But I might experiment with some anime-focused plugins when the time comes.

Transcoding¶

Although not recommended and explicitly set to not download any x265/HEVC content (by using the Golden rule) there might be cases where the only option you have is to download such content. If that is the case and you happen to have a way to do Hardware Acceleration, for example by having an NVIDIA graphics card (in my case) then you should enable it to avoid using lots of resources on your system.

Using hardware acceleration will leverage your GPU to do the transcoding and save resources on your CPU. I tried streaming x265 content and it basically used 70-80% on all cores of my CPU, while on the other hand using my GPU it used the normal amount on the CPU (70-80% on a single core).

This will be the steps to install on an NVIDIA graphics card, specifically a GTX 1660 Ti. But more info and guides can be found at Jellyfin: Hardware Acceleration for other acceleration methods.

NVIDIA drivers¶

Ensure you have the NVIDIA drivers and utils installed. I’ve you’ve done this in the past then you can skip this part, else you might be using the default nouveau drivers. Follow the next steps to set up the NVIDIA drivers, which basically is a summary of NVIDIA: Installation for my setup, so double check the wiki in case you have an older NVIDIA graphics card.

Install the nvidia and nvidia-utils packages:

pacman -S nvidia nvidia-utils
-

Modify /etc/mkinitcpio.conf to remove kms from the HOOKS array. It should look like this (commented line is how it was for me before the change):

...
-# HOOKS=(base udev autodetect modconf kms keyboard keymap consolefont block filesystems fsck)
-HOOKS=(base udev autodetect modconf keyboard keymap consolefont block filesystems fsck)
-...
-

Regenerate the initramfs by executing:

mkinitcpio -P
-

Finally, reboot the system. After the reboot you should be able to check your GPU info and processes being run with the GPU by executing nvidia-smi.

Enable hardware acceleration¶

Install from the AUR with yay:

yay -S jellyfin-ffmpeg6-bin
-

This provides the jellyfin-ffmpeg executable, which is necessary for Jellyfin to do hardware acceleration, it needs to be this specific one.

Then in the Jellyfin go to the transcoding settings by clicking on the 3 horizontal lines on the top left of the Web UI and navigating to Administration -> Dashboard -> Playback -> Transcoding and:

Change the Hardware acceleration from “None” to “Nvidia NVENC”.
Some other options will pop up, just make sure you enable “HEVC” (which is x265) in the list of Enable hardware encoding for:.
Scroll down and specify the ffmpeg path, which is /usr/lib/jellyfin-ffmpeg/ffmpeg.

Don’t forget to click “Save” at the bottom of the Web UI, it will ask if you want to enable hardware acceleration.

Bazarr¶

Bazarr is a companion for Sonarr and Radarr that manages and downloads subtitles.

Install from the AUR with yay:

yay -S bazarr
-

Add the bazarr user to the servarr group:

gpasswd -a bazarr servarr
-

The default port that Bazarr uses is 6767 for http (the one you need for the reverse proxy), and it has pre-configured the default ports for Radarr and Sonarr.

Reverse proxy¶

Basically the same as with Radarr: Reverse proxy and Sonarr: Reverse proxy.

Add the following setting in the server block of the isos.conf:

server {
-    # server_name and other directives
-    ...
-
-    # Increase http2 max sizes
-    large_client_header_buffers 4 16k;
-
-    # some other blocks like location blocks
-    ...
-}
-

Then add the following location blocks in the isos.conf, where I’ll keep it as /bazarr/:

location /bazarr/ {
-    proxy_pass http://127.0.0.1:6767/bazarr/; # change port if needed
-    proxy_http_version 1.1;
-
-    proxy_set_header X-Real-IP $remote_addr;
-    proxy_set_header Host $http_host;
-    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-    proxy_set_header X-Forwarded-Proto $scheme;
-    proxy_set_header Upgrade $http_upgrade;
-    proxy_set_header Connection "Upgrade";
-
-    proxy_redirect off;
-}
-# Allow the Bazarr API through if you enable Auth on the block above
-location /bazarr/api {
-    auth_request off;
-    proxy_pass http://127.0.0.1:6767/bazarr/api;
-}
-

This is taken from Bazarr: Reverse proxy help. Restart the nginx service for the changes to take effect:

systemctl restart nginx.service
-

Start using Bazarr¶

You can now start/enable the bazarr.service if you haven’t already:

systemctl start bazarr.service
-systemctl enable bazarr.service
-

This will start the service and create the default configs under /var/lib/bazarr. You need to change the base_url for the necessary services as they’re running under a reverse proxy and under subdirectories. Edit /var/lib/bazarr/config/config.ini:

[general]
-port = 6767
-base_url = /bazarr
-
-[sonarr]
-port = 8989
-base_url = /sonarr
-
-[radarr]
-port = 7878
-base_url = /radarr
-

Then restart the bazarr service:

systemctl restart bazarr.service
-

Now https://isos.yourdomain.com/bazarr is accessible. Secure the instance right away by adding authentication under Settings -> General -> Security. I added the “Forms” option, just fill in the username and password then click on save changes on the top left of the page. You can restart the service again and check that it asks for login credentials. I also disabled Settings -> General -> Updates -> Automatic.

Configuration¶

Will be following the official Bazarr: Setup guide as well as the recommendations by TRaSH: Bazarr.

Anything that is not mentioned in either guide or that is specific to how I’m setting up stuff will be stated below.

Providers¶

This doesn’t require much thinking and its up to personal preference, but I’ll list the ones I added:

OpenSubtitles.com: requires an account (the .org option is deprecated).
- For a free account it only lets you download around 20 subtitles per day, and they contain ads. You could pay for a VIP account ($3 per month) and that will give you 1000 subtitles per day and no ads. But if you’re fine with 20 ads per day you can try to get rid of the ads by running an automated script. Such option can be found at brianspilner01/media-server-scripts: sub-clean.sh.
-
YIFY Subtitles
Subscenter
Supersubtitles
TVSubtitles
Subtitulamos.tv: Spanish subtitles provider.
Argenteam: LATAM Spanish subtitles provider.
Subdivx: LATAM Spanish / Spanish subtitles provider.

I’ve tested this setup for the following languages (with all default settings as stated in the guides):

English
Spanish

I tried with “Latin American Spanish” but they’re hard to find, those two work pretty good.

None of these require an Anti-Captcha account (which is a paid service), but I created one anyways in case I need it. Though you need to add credits to it (pretty cheap though) if you ever use it.

- -

- - - - Next - - - - - - - Index - - - - - - - Previous - - -

- - -

By David Luévano

Created: Mon, Jul 24, 2023 @ 04:30 UTC

Modified: Mon, Jul 24, 2023 @ 04:32 UTC

Tags: -code, english, server, tools, tutorial

- -

Set up a Mail server with Postfix, Dovecot, SpamAssassin and OpenDKIM

- -

The entry is going to be long because it’s a tedious process. This is also based on Luke Smith’s script, but adapted to Arch Linux (his script works on debian-based distributions). This entry is mostly so I can record all the notes required while I’m in the process of installing/configuring the mail server on a new VPS of mine; ~~also I’m going to be writing a script that does everything in one go (for Arch Linux), that will be hosted here.~~ I haven’t had time to do the script so nevermind this, if I ever do it I’ll make a new entry regarding it.

This configuration works for local users (users that appear in /etc/passwd), and does not use any type of SQL database. Do note that I’m not running Postfix in a chroot, which can be a problem if you’re following my steps as noted by Bojan; in the case that you want to run in chroot then add the steps chown in the Arch wiki: Postfix in a chroot jail; the issue faced if following my steps and using a chroot is that there will be issues resolving the hostname due to /etc/hosts or /etc/hostname not being available in the chroot.

All commands executed here are run with root privileges, unless stated otherwise.

Prerequisites¶

Basically the same as with the website with Nginx and Certbot, with the extras:

You will need a CNAME for “mail” and (optionally) “www.mail”, or whatever you want to call the sub-domains (although the RFC 2181 states that it NEEDS to be an A record, fuck the police).
An SSL certificate. You can use the SSL certificate obtained following my last post using certbot (just create a mail.conf and run certbot --nginx again).
Ports 25, 587 (SMTP), 465 (SMTPS), 143 (IMAP) and 993 (IMAPS) open on the firewall (I use ufw).

Postfix¶

Postfix is a “mail transfer agent” which is the component of the mail server that receives and sends emails via SMTP.

Install the postfix package:

pacman -S postfix
-

We have two main files to configure (inside /etc/postfix): master.cf (master(5)) and main.cf (postconf(5)). We’re going to edit main.cf first either by using the command postconf -e 'setting' or by editing the file itself (I prefer to edit the file).

Note that the default file itself has a lot of comments with description on what each thing does (or you can look up the manual, linked above), I used what Luke’s script did plus some other settings that worked for me.

Now, first locate where your website cert is, mine is at the default location /etc/letsencrypt/live/, so my certdir is /etc/letsencrypt/live/luevano.xyz. Given this information, change {yourcertdir} on the corresponding lines. The configuration described below has to be appended in the main.cf configuration file.

Certificates and ciphers to use for authentication and security:

smtpd_tls_key_file = {yourcertdir}/privkey.pem
-smtpd_tls_cert_file = {yourcertdir}/fullchain.pem
-smtpd_use_tls = yes
-smtpd_tls_auth_only = yes
-smtp_tls_security_level = may
-smtp_tls_loglevel = 1
-smtp_tls_CAfile = {yourcertdir}/cert.pem
-smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
-smtp_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
-smtpd_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
-smtp_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
-tls_preempt_cipherlist = yes
-smtpd_tls_exclude_ciphers = aNULL, LOW, EXP, MEDIUM, ADH, AECDH, MD5,
-                DSS, ECDSA, CAMELLIA128, 3DES, CAMELLIA256,
-                RSA+AES, eNULL
-
-smtp_tls_CApath = /etc/ssl/certs
-smtpd_tls_CApath = /etc/ssl/certs
-
-smtpd_relay_restrictions = permit_sasl_authenticated, permit_mynetworks, defer_unauth_destination
-

Also, for the connection with dovecot, append the next few lines (telling postfix that dovecot will use user/password for authentication):

smtpd_sasl_auth_enable = yes
-smtpd_sasl_type = dovecot
-smtpd_sasl_path = private/auth
-smtpd_sasl_security_options = noanonymous, noplaintext
-smtpd_sasl_tls_security_options = noanonymous
-

Specify the mailbox home, this is going to be a directory inside your user’s home containing the actual mail files, for example it will end up being/home/david/Mail/Inbox:

home_mailbox = Mail/Inbox/
-

Pre-configuration to work seamlessly with dovecot and opendkim:

myhostname = {yourdomainname}
-mydomain = localdomain
-mydestination = $myhostname, localhost.$mydomain, localhost
-
-milter_default_action = accept
-milter_protocol = 6
-smtpd_milters = inet:127.0.0.1:8891
-non_smtpd_milters = inet:127.0.0.1:8891
-mailbox_command = /usr/lib/dovecot/deliver
-

Where {yourdomainname} is luevano.xyz in my case. Lastly, if you don’t want the sender’s IP and user agent (application used to send the mail), add the following line:

smtp_header_checks = regexp:/etc/postfix/smtp_header_checks
-

And create the /etc/postfix/smtp_header_checks file with the following content:

/^Received: .*/     IGNORE
-/^User-Agent: .*/   IGNORE
-

That’s it for main.cf, now we have to configure master.cf. This one is a bit more tricky.

First look up lines (they’re uncommented) smtp inet n - n - - smtpd, smtp unix - - n - - smtp and -o syslog_name=postfix/$service_name and either delete or uncomment them… or just run sed -i "/^\s*-o/d;/^\s*submission/d;/\s*smtp/d" /etc/postfix/master.cf as stated in Luke’s script.

Lastly, append the following lines to complete postfix setup and pre-configure for spamassassin.

smtp unix - - n - - smtp
-smtp inet n - y - - smtpd
-    -o content_filter=spamassassin
-submission inet n - y - - smtpd
-    -o syslog_name=postfix/submission
-    -o smtpd_tls_security_level=encrypt
-    -o smtpd_sasl_auth_enable=yes
-    -o smtpd_tls_auth_only=yes
-smtps inet n - y - - smtpd
-    -o syslog_name=postfix/smtps
-    -o smtpd_tls_wrappermode=yes
-    -o smtpd_sasl_auth_enable=yes
-spamassassin unix - n n - - pipe
-    user=spamd argv=/usr/bin/vendor_perl/spamc -f -e /usr/sbin/sendmail -oi -f \${sender} \${recipient}
-

Now, I ran into some problems with postfix, one being smtps: Servname not supported for ai_socktype, to fix it, as Till posted in that site, edit /etc/services and add:

smtps 465/tcp
-smtps 465/udp
-

Before starting the postfix service, you need to run newaliases first, but you can do a bit of configuration beforehand editing the file /etc/postfix/aliases. I only change the root: you line (where you is the account that will be receiving “root” mail). After you’re done, run:

postalias /etc/postfix/aliases
-newaliases
-

At this point you’re done configuring postfix and you can already start/enable the postfix service:

systemctl start postfix.service
-systemctl enable postfix.service
-

Dovecot¶

Dovecot is an IMAP and POP3 server, which is what lets an email application retrieve the mail.

Install the dovecot and pigeonhole (sieve for dovecot) packages:

pacman -S dovecot pigeonhole
-

On arch, by default, there is no /etc/dovecot directory with default configurations set in place, but the package does provide the example configuration files. Create the dovecot directory under /etc and, optionally, copy the dovecot.conf file and conf.d directory under the just created dovecot directory:

mkdir /etc/dovecot
-cp /usr/share/doc/dovecot/example-config/dovecot.conf /etc/dovecot/dovecot.conf
-cp -r /usr/share/doc/dovecot/example-config/conf.d /etc/dovecot
-

As Luke stated, dovecot comes with a lot of “modules” (under /etc/dovecot/conf.d/ if you copied that folder) for all sorts of configurations that you can include, but I do as he does and just edit/create the whole dovecot.conf file; although, I would like to check each of the separate configuration files dovecot provides I think the options Luke provides are more than good enough.

I’m working with an empty dovecot.conf file. Add the following lines for SSL and login configuration (also replace {yourcertdir} with the same certificate directory described in the Postfix section above, note that the < is required):

ssl = required
-ssl_cert = <{yourcertdir}/fullchain.pem
-ssl_key = <{yourcertdir}/privkey.pem
-ssl_min_protocol = TLSv1.2
-ssl_cipher_list = ALL:!RSA:!CAMELLIA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4:!SHA1:!SHA256:!SHA384:!LOW@STRENGTH
-ssl_prefer_server_ciphers = yes
-ssl_dh = </etc/dovecot/dh.pem
-
-auth_mechanisms = plain login
-auth_username_format = %n
-protocols = $protocols imap
-

You may notice we specify a file we don’t have under /etc/dovecot: dh.pem. We need to create it with openssl (you should already have it installed if you’ve been following this entry and the one for nginx). Just run (might take a few minutes):

openssl dhparam -out /etc/dovecot/dh.pem 4096
-

After that, the next lines define what a “valid user is” (really just sets the database for users and passwords to be the local users with their password):

userdb {
-    driver = passwd
-}
-
-passdb {
-    driver = pam
-}
-

Next, comes the mail directory structure (has to match the one described in the Postfix section). Here, the LAYOUT option is important so the boxes are .Sent instead of Sent. Add the next lines (plus any you like):

mail_location = maildir:~/Mail:INBOX=~/Mail/Inbox:LAYOUT=fs
-namespace inbox {
-    inbox = yes
-
-    mailbox Drafts {
-        special_use = \Drafts
-        auto = subscribe
-        }
-
-    mailbox Junk {
-        special_use = \Junk
-        auto = subscribe
-        autoexpunge = 30d
-        }
-
-    mailbox Sent {
-        special_use = \Sent
-        auto = subscribe
-        }
-
-    mailbox Trash {
-        special_use = \Trash
-        }
-
-    mailbox Archive {
-        special_use = \Archive
-        }
-}
-

Also include this so Postfix can use Dovecot’s authentication system:

service auth {
-    unix_listener /var/spool/postfix/private/auth {
-        mode = 0660
-        user = postfix
-        group = postfix
-        }
-}
-

Lastly (for Dovecot at least), the plugin configuration for sieve (pigeonhole):

protocol lda {
-    mail_plugins = $mail_plugins sieve
-}
-
-protocol lmtp {
-    mail_plugins = $mail_plugins sieve
-}
-
-plugin {
-    sieve = ~/.dovecot.sieve
-    sieve_default = /var/lib/dovecot/sieve/default.sieve
-    sieve_dir = ~/.sieve
-    sieve_global_dir = /var/lib/dovecot/sieve/
-

Where /var/lib/dovecot/sieve/default.sieve doesn’t exist yet. Create the folders:

mkdir -p /var/lib/dovecot/sieve
-

And create the file default.sieve inside that just created folder with the content:

require ["fileinto", "mailbox"];
-if header :contains "X-Spam-Flag" "YES" {
-    fileinto "Junk";
-}
-

Now, if you don’t have a vmail (virtual mail) user, create one and change the ownership of the /var/lib/dovecot directory to this user:

grep -q "^vmail:" /etc/passwd || useradd -m vmail -s /usr/bin/nologin
-chown -R vmail:vmail /var/lib/dovecot
-

Note that I also changed the shell for vmail to be /usr/bin/nologin. After that, to compile the configuration file run:

sievec /var/lib/dovecot/sieve/default.sieve
-

A default.svbin file will be created next to default.sieve.

Next, add the following lines to /etc/pam.d/dovecot if not already present (shouldn’t be there if you’ve been following these notes):

auth required pam_unix.so nullok
-account required pam_unix.so
-

That’s it for Dovecot, at this point you can start/enable the dovecot service:

systemctl start dovecot.service
-systemctl enable dovecot.service
-

OpenDKIM¶

OpenDKIM is needed so services like G**gle don’t throw the mail to the trash. DKIM stands for “DomainKeys Identified Mail”.

Install the opendkim package:

pacman -S opendkim
-

Generate the keys for your domain:

opendkim-genkey -D /etc/opendkim -d {yourdomain} -s {yoursubdomain} -r -b 2048
-

Where you need to change {yourdomain} and {yoursubdomain} (doesn’t really need to be the sub-domain, could be anything that describes your key) accordingly, for me it’s luevano.xyz and mail, respectively. After that, we need to create some files inside the /etc/opendkim directory. First, create the file KeyTable with the content:

{yoursubdomain}._domainkey.{yourdomain} {yourdomain}:{yoursubdomain}:/etc/opendkim/{yoursubdomain}.private
-

So, for me it would be:

mail._domainkey.luevano.xyz luevano.xyz:mail:/etc/opendkim/mail.private
-

Next, create the file SigningTable with the content:

*@{yourdomain} {yoursubdomain}._domainkey.{yourdomain}
-

Again, for me it would be:

*@luevano.xyz mail._domainkey.luevano.xyz
-

And, lastly create the file TrustedHosts with the content:

127.0.0.1
-::1
-10.1.0.0/16
-1.2.3.4/24
-localhost
-{yourserverip}
-...
-

And more, make sure to include your server IP and something like subdomain.domainname.

Next, edit /etc/opendkim/opendkim.conf to reflect the changes (or rather, addition) of these files, as well as some other configuration. You can look up the example configuration file located at /usr/share/doc/opendkim/opendkim.conf.sample, but I’m creating a blank one with the contents:

Domain {yourdomain}
-Selector {yoursubdomain}
-
-Syslog Yes
-UserID opendkim
-
-KeyFile /etc/opendkim/{yoursubdomain}.private
-Socket inet:8891@localhost
-

Now, change the permissions for all the files inside /etc/opendkim:

chown -R root:opendkim /etc/opendkim
-chmod g+r /etc/postfix/dkim/*
-

I’m using root:opendkim so opendkim doesn’t complain about the {yoursubdomani}.private being insecure (you can change that by using the option RequireSafeKeys False in the opendkim.conf file, as stated here).

That’s it for the general configuration, but you could go more in depth and be more secure with some extra configuration.

Now, just start/enable the opendkim service:

systemctl start opendkim.service
-systemctl enable opendkim.service
-

OpenDKIM DNS TXT records¶

Add the following TXT records on your domain registrar (these examples are for Epik):

DKIM entry: look up your {yoursubdomain}.txt file, it should look something like:

{yoursubdomain}._domainkey IN TXT ( "v=DKIM1; k=rsa; s=email; "
-    "p=..."
-    "..." )  ; ----- DKIM key mail for {yourdomain}
-

In the TXT record you will place {yoursubdomain}._domainkey as the “Host” and "v=DKIM1; k=rsa; s=email; " "p=..." "..." in the “TXT Value” (replace the dots with the actual value you see in your file).

-
DMARC entry: just _dmarc.{yourdomain} as the “Host” and "v=DMARC1; p=reject; rua=mailto:dmarc@{yourdomain}; fo=1" as the “TXT Value”.
-
-
SPF entry: just @ as the “Host” and "v=spf1 mx a:{yoursubdomain}.{yourdomain} - all" as the “TXT Value”.
-

And at this point you could test your mail for spoofing and more.

SpamAssassin¶

SpamAssassin is just a mail filter to identify spam.

Install the spamassassin package (which will install a bunch of ugly perl packages…):

pacman -S spamassassin
-

For some reason, the permissions on all spamassassin stuff are all over the place. First, change owner of the executables, and directories:

chown spamd:spamd /usr/bin/vendor_perl/sa-*
-chown spamd:spamd /usr/bin/vendor_perl/spam*
-chwown -R spamd:spamd /etc/mail/spamassassin
-

Then, you can edit local.cf (located in /etc/mail/spamassassin) to fit your needs (I only uncommented the rewrite_header Subject ... line). And then you can run the following command to update the patterns and compile them:

sudo -u spamd sa-update
-sudo -u spamd sa-compile
-

And since this should be run periodically, create the service spamassassin-update.service under /etc/systemd/system with the following content:

[Unit]
-Description=SpamAssassin housekeeping
-After=network.target
-
-[Service]
-User=spamd
-Group=spamd
-Type=oneshot
-
-ExecStart=/usr/bin/vendor_perl/sa-update --allowplugins
-SuccessExitStatus=1
-ExecStart=/usr/bin/vendor_perl/sa-compile
-ExecStart=/usr/bin/systemctl -q --no-block try-restart spamassassin.service
-

And you could also execute sa-learn to train spamassassin‘s bayes filter, but this works for me. Then create the timer spamassassin-update.timer under the same directory, with the content:

[Unit]
-Description=SpamAssassin housekeeping
-
-[Timer]
-OnCalendar=daily
-Persistent=true
-
-[Install]
-WantedBy=timers.target
-

You can now start/enable the spamassassin-update timer:

systemctl start spamassassin-update.timer
-systemctl enable spamassassin-update.timer
-

Next, you may want to edit the spamassassin service before starting and enabling it, because by default, it could spawn a lot of “childs” eating a lot of resources and you really only need one child. Append --max-children=1 to the line ExecStart=... in /usr/bin/systemd/system/spamassassin.service:

...
-ExecStart=/usr/bin/vendor_perl/spamd -x -u spamd -g spamd --listen=/run/spamd/spamd.sock --listen=localhost --max-children=1
-...
-

Finally, start and enable the spamassassin service:

systemctl start spamassassin.service
-systemctl enable spamassassin.service
-

Wrapping up¶

We should have a working mail server by now. Before continuing check your journal logs (journalctl -xe --unit={unit}, where {unit} could be spamassassin.service for example) to see if there was any error whatsoever and try to debug it, it should be a typo somewhere because all the settings and steps detailed here just worked; I literally just finished doing everything on a new server as of the writing of this text, it just werks on my machine.

Now, to actually use the mail service: first of all, you need a normal account (don’t use root) that belongs to the mail group (gpasswd -a user group to add a user user to group group) and that has a password.

Next, to actually login into a mail app/program, you will use the following settings, at least for thunderdbird(I tested in windows default mail app and you don’t need a lot of settings):

* server: subdomain.domain (mail.luevano.xyz in my case)
SMTP port: 587
SMTPS port: 465 (I use this one)
IMAP port: 143
IMAPS port: 993 (again, I use this one)
Connection/security: SSL/TLS
Authentication method: Normal password
Username: just your user, not the whole email (david in my case)
Password: your user password (as in the password you use to login to the server with that user)

All that’s left to do is test your mail server for spoofing, and to see if everything is setup correctly. Go to DKIM Test and follow the instructions (basically click next, and send an email with whatever content to the email that they provide). After you send the email, you should see something like:

- -

- - - - Next - - - - - - - Index - - - - - - - Previous - - -

- - -

By David Luévano

Created: Sun, Mar 21, 2021 @ 04:05 UTC

Modified: Wed, Jun 21, 2023 @ 00:17 UTC

Tags: -code, english, server, tools, tutorial

- -

Set up a manga server with Komga and mangal

- -

I’ve been wanting to set up a manga media server to hoard some mangas/comics and access them via Tachiyomi, but I didn’t have enough space in my vultr VPS. Now that I have symmetric fiber optic at home and my spare PC to use as a server I decided to go ahead and create one. As always, i use arch btw so these instructions are specifically for it, I’m not sure how easier/harder it is for other distros, I’m just too comfortable with arch honestly.

I’m going to run it as an exposed service using a subdomain of my own, so the steps are taking that into account, if you want to run it locally (or on a LAN/VPN) then it is going to be easier/with less steps (you’re on your own). Also, as you might notice I don’t like to use D*ck*r images or anything (ew).

At the time of editing this entry (06-28-2023) Komga has already upgraded to v.1.0.0 and it introduces some breaking changes if you already had your instance set up. Read more here. The only change I did here was changing the port to the new default.

As always, all commands are run as root unless stated otherwise.

Table of contents¶

Table of contents
Prerequisites
yay
- Install
- Usage
-
mangal
- Install from source
- Configuration
- Usage
  - Headless browser
  - TUI
  - Inline
  - Automation
  -
-
Komga
- Reverse proxy
  - SSL certificate
  -
- Start using Komga
- Library creation
  - Set default directory permissions
  - Populate manga library
  -
-
Alternative downloaders

Prerequisites¶

Similar to my early tutorial entries, if you want it as a subdomain:

An A (and/or AAAA) or a CNAME for komga (or whatever you want).
An SSL certificate, if you’re following the other entries (specially the website entry), add a komga.conf and run certbot --nginx (or similar) to extend/create the certificate. More details below: Reverse proxy and SSL certificate.

yay¶

This is the first time I mention the AUR (and yay) in my entries, so I might as well just write a bit about it.

The AUR is the Arch Linux User Repository and it’s basically like an extension of the official one which is supported by the community, the only thing is that it requires a different package manager. The one I use (and I think everyone does, too) is yay, which as far as I know is like a wrapper of pacman.

Install¶

To install and use yay we need a normal account with sudo access, all the commands related to yay are run as normal user and then it asks for sudo password. Installation its straight forward: clone yay repo and install. Only dependencies are git and base-devel:

Install dependencies:

sudo pacman -S git base-devel
-

Clone yay and install it (I also like to delete the cloned git repo):

git clone git@github.com:Jguer/yay.git
-cd yay
-makepkg -si
-cd ..
-sudo rm -r yay
-

Usage¶

yay is used basically the same as pacman with the difference that it is run as normal user (then later requiring sudo password) and that it asks extra input when installing something, such as if we want to build the package from source or if we want to show package diffs.

To install a package (for example Komga in this blog entry), run:

yay -S komga
-

mangal¶

mangal is a CLI/TUI manga downloader with anilist integration and custom Lua scrapers.

You could install it from the AUR with yay:

yay -S mangal-bin
-

But I’ll use my fork as it contains some fixes and extra stuff.

Install from source¶

As I mentioned in my past entry I had to fork mangal and related repositories to fix/change a few things. Currently the major fix I did in mangal is for the built in MangaDex scraper which had really annoying bug in the chunking of the manga chapter listing.

So instad of installing with yay we’ll build it from source. We need to have go installed:

pacman -S go
-

Then clone my fork of mangal and install it:

git clone https://github.com/luevano/mangal.git # not sure if you can use SSH to clone
-cd mangal
-make install # or just `make build` and then move the binary to somewhere in your $PATH
-

This will use go install so it will install to a path specified by the go environment variables, for more run go help install. It was installed to $HOME/.local/bin/go/mangal for me because my env vars, then just make sure this is included in your PATH.

Check it was correctly installed by running mangal version, which should print something like:

▇▇▇ mangal
-
-  Version         ...
-  Git Commit      ...
-  Build Date      ...
-  Built By        ...
-  Platform        ...
-

Configuration¶

I’m going to do everything with a normal user (manga-dl) which I created just to download manga. So all of the commands will be run without sudo/root privileges.

Change some of the configuration options:

mangal config set -k downloader.path -v "/mnt/d/mangal" # downloads to current dir by default
-mangal config set -k formats.use -v "cbz" # downloads as pdf by default
-mangal config set -k installer.user -v "luevano" # points to my scrapers repository which contains a few extra scrapers and fixes, defaults to metafates' one; this is important if you're using my fork, don't use otherwise as it uses extra stuff I added
-mangal config set -k logs.write -v true # I like to get logs for what happens
-

Note: For testing purposes (if you want to explore mangal) set downloader.path once you’re ready to start to populate the Komga library directory (at Komga: populate manga library).

For more configs and to read what they’re for:

mangal config info
-

Also install the custom Lua scrapers by running:

mangal sources install
-

And install whatever you want, it picks up the sources/scrapers from the configured repository (installer.<key> config), if you followed, it will show my scrapers.

Usage¶

Two main ways of using mangal:

TUI: for initial browsing/downloading and testing things out. If the manga finished publishing, this should be enough.
inline: for automation on manga that is still publishing and I need to check/download every once in a while.

Headless browser¶

Before continuing, I gotta say I went through some bullshit while trying to use the custom Lua scrapers that use the headless browser (actually just a wrapper of go-rod/rod, and honestly it is not really a “headless” browser, mangal “documentation” is just wrong). For more on my rant check out my last entry.

There is no concrete documentation on the “headless” browser, only that it is automatically set up and ready to use… but it doesn’t install any library/dependency needed. I discovered the following libraries that were missing on my Arch minimal install:

library -> arch package containing it
libnss3.so -> nss
libatk-1.0.so.0 -> at-spi2-core
libcups.so.2 -> libcups
libdrm.so.2 -> libdrm
libXcomposite.so.1 -> libxcomposite
libXdamage.so.1 -> libxdamage
libXrandr.so.2 -> libxrandr
libgbm.so.1 -> mesa
libxkbcommon.so.0 -> libxkbcommon
libpango-1.0.so.0 -> pango
libasound.so.2 -> alsa-lib

To install them::

pacman -S nss at-spi2-core libcups libdrm libxcomposite libxdamage libxrandr mesa libxkbcommon pango alsa-lib
-

I can’t guarantee that those are all the packages needed, those are the ones I happen to discover (had to fork the lua libs and add some logging because the error message was too fucking generic).

These dependencies are probably met by installing either chromedriver or google-chrome from the AUR (for what I could see on the package dependencies).

TUI¶

Use the TUI by running

mangal
-

Download manga using the TUI by selecting the source/scrapper, search the manga/comic you want and then you can select each chapter to download (use tab to select all). This is what I use when downloading manga that already finished publishing, or when I’m just searching and testing out how it downloads the manga (directory name, and manga information).

Note that some scrapters will contain duplicated chapters, as they have multiple uploaded chapters from the community, usually for different scanlation groups. This happens a lot with MangaDex.

Inline¶

The inline mode is a single terminal command meant to be used to automate stuff or for more advanced options. You can peek a bit into the “documentation” which honestly it’s ass because it doesn’t explain much. The minimal command for inline according to the mangal help is:

mangal inline --manga <option> --query <manga-title>
-

But this will not produce anything because it also needs --source (or set the default using the config key downloader.default_sources) and either --json which basically just does the search and returns the result in json format or --download to actually download whatever is found; I recommend to do --json first to check that the correct manga will be downloaded then do --download.

Something not mentioned anywhere is the --manga flag options (found it at the source code), it has 3 available options:

first: first manga entry found for the search.
last: last manga entry found for the search.
exact: exact manga title match. This is the one I use.

Similar to --chapters, there are a few options not explained (that I found at the source code, too). I usually just use all but other options:

all: all chapters found in the chapter list.
first: first chapter found in the chapter list.
last: last chapter found in the chapter list
[from]-[to]: selector for the chapters found in the chapter list, index starts at 0.
- If the selectors (from or to) exceed the amount of chapters in the chapterlist it just adjusts to he maximum available.
- I had to fix this at the source code because if you wanted to to be the last chapter, it did to + 1 and it failed due to index out of range.
-
@[sub]@: not sure how this works exactly, my understanding is that it’s for “named” chapters.

That said, I’ll do an example by using Mangapill as source, and will search for Demon Slayer: Kimetsu no Yaiba:

Search first and make sure my command will pull the manga I want:

mangal inline --source "Mangapill" --manga "exact" --query "Kimetsu no Yaiba" --json | jq # I use jq to pretty format the output
-

I make sure the json output contains the correct manga information: name, url, etc..

You can also include the flag --include-anilist-manga to include anilist information (if any) so you can check that the correct anilist id is attached. If the correct one is not attached (and it exists) then you can bind the --query (search term) to a specific anilist id by running:

mangal inline anilist set --name "Kimetsu no Yaiba" --id 101922
-

If I’m okay with the outputs, then I change --json for --download to actually download:

mangal inline --source "Mangapill" --manga "exact" --query "Kimetsu no Yaiba" --download
-

Check if the manga is downloaded correctly. I do this by going to my download directory and checking the directory name (I’m picky with this stuff), that all chapters where downloaded, that it includes a correct series.json file and it contains a cover.<img-ext>; this usually means it correctly pulled information from anilist and that it will contain metadata Komga will be able to use.

Automation¶

The straight forward approach for automation is just to bundle a bunch of mangal inline commands in a shell script and schedule it’s execution either via cron or systemd/Timers. But, as always, I overcomplicated/overengineered my approach, which is the following:

Group manga names per source.
Configure anything that should always be set before executing mangal, this includes anilist bindings.
Have a way to track the changes/updates on each run.
Use that tracker to know where to start downloading chapters from.
- This is optional, as you can just do --chapters "all" and it will work but I do it mostly to keep the logs/output cleaner/shorter.
-
Download/update each manga using mangal inline.
Wrap everything in a systemd service and timer.

Manga list example:

mangapill="Berserk|Chainsaw Man|Dandadan|Jujutsu Kaisen|etc..."
-

Function that handles the download per manga in the list:

mangal_src_dl () {
-    source_name=$1
-    manga_list=$(echo "$2" | tr '|' '\n')
-
-    while IFS= read -r line; do
-        # By default download all chapters
-        chapters="all"
-        last_chapter_n=$(grep -e "$line" "$TRACKER_FILE" | cut -d'|' -f2 | grep -v -e '^$' | tail -n 1)
-        if [ -n "${last_chapter_n}" ]; then
-            chapters="$last_chapter_n-9999"
-            echo "Downloading [${last_chapter_n}-] chapters for $line from $source_name..."
-        else
-            echo "Downloading all chapters for $line from $source_name..."
-        fi
-        dl_output=$(mangal inline -S "$source_name" -q "$line" -m "exact" -F "$DOWNLOAD_FORMAT" -c "$chapters" -d)
-
-        if [ $? -ne 0 ]; then
-            echo "Failed to download chapters for $line."
-            continue
-        fi
-
-        line_count=$(echo "$dl_output" | grep -v -e '^$' | wc -l)
-        if [ $line_count -gt 0 ]; then
-            echo "Downloaded $line_count chapters for $line:"
-            echo "$dl_output"
-            new_last_chapter_n=$(echo "$dl_output" | tail -n 1 | cut -d'[' -f2 | cut -d']' -f1)
-            # manga_name|last_chapter_number|downloaded_chapters_on_this_update|manga_source
-            echo "$line|$new_last_chapter_n|$line_count|$source_name" >> $TRACKER_FILE
-        else
-            echo "No new chapters for $line."
-        fi
-    done <<< "$manga_list"
-}
-

Where $TRACKER_FILE is just a variable holding a path to some file where you can store the tracking and $DOWNLOAD_FORMAT the format for the mangas, for me it’s cbz. Then the usage would be something like mangal_src_dl "Mangapill" "$mangapill", meaning that it is a function call per source.

A simpler function without “tracking” would be:

mangal_src_dl () {
-    source_name=$1
-    manga_list=$(echo "$2" | tr '|' '\n')
-
-    while IFS= read -r line; do
-        echo "Downloading all chapters for $line from $source_name..."
-        mangal inline -S "$source_name" -q "$line" -m "exact" -F "$DOWNLOAD_FORMAT" -c "all" -d
-        if [ $? -ne 0 ]; then
-            echo "Failed to download chapters for $line."
-            continue
-        fi
-        echo "Finished downloading chapters for $line."
-    done <<< "$manga_list"
-}
-

The tracker file would have a format like follows:

# Updated: 06/10/23 10:53:15 AM CST
-Berserk|0392|392|Mangapill
-Dandadan|0110|110|Mangapill
-...
-

And note that if you already had manga downloaded and you run the script for the first time, then it will show as if it downloaded everything from the first chapter, but that’s just how mangal works, it will actually just discover downloaded chapters and only download anything missing.

Any configuration the downloader/updater might need needs to be done before the mangal_src_dl calls. I like to configure mangal for download path, format, etc.. I found that it is needed to clear the mangal and rod browser cache (headless browser used in some custom sources) from personal experience and from others: mangal#170 and kaizoku#89.

Also you should set any anilist binding necessary for the downloading (as the cache was cleared). An example of an anilist binding I had to do is for Mushoku Tensei, as it has both a light novel and manga version, which for me it’s the following binding:

mangal inline anilist set --name "Mushoku Tensei - Isekai Ittara Honki Dasu" --id 85564
-

Finally is just a matter of using your prefered way of scheduling, I’ll use systemd/Timers but anything is fine. You could make the downloader script more sophisticated and only running every week on which each manga gets (usually) released but that’s too much work; I’ll just run it once daily probably.

A feature I want to add and probably will is sending notifications (probably through email) on a summary for manga downloaded or failed to download so I’m on top of the updates. For now this is good enough and it’s been working so far.

Komga¶

Komga is a comics/mangas media server.

Install from the AUR:

yay -S komga
-

This komga package creates a komga (service) user and group which is tied to the also included komga.service.

Configure it by editing /etc/komga.conf:

SERVER_PORT=25600
-SERVER_SERVLET_CONTEXT_PATH=/ # this depends a lot of how it's going to be served (domain, subdomain, ip, etc)
-
-KOMGA_LIBRARIES_SCAN_CRON="0 0 * * * ?"
-KOMGA_LIBRARIES_SCAN_STARTUP=false
-KOMGA_LIBRARIES_SCAN_DIRECTORY_EXCLUSIONS='#recycle,@eaDir,@Recycle'
-KOMGA_FILESYSTEM_SCANNER_FORCE_DIRECTORY_MODIFIED_TIME=false
-KOMGA_REMEMBERME_KEY=USE-WHATEVER-YOU-WANT-HERE
-KOMGA_REMEMBERME_VALIDITY=2419200
-
-KOMGA_DATABASE_BACKUP_ENABLED=true
-KOMGA_DATABASE_BACKUP_STARTUP=true
-KOMGA_DATABASE_BACKUP_SCHEDULE="0 0 */8 * * ?"
-

My changes (shown above):

cron schedules.
- It’s not actually cron but rather a cron-like syntax used by Spring as stated in the Komga config.
-
Added the remember me key.
For more check out Komga: Configuration options.

If you’re going to run it locally (or LAN/VPN) you can start the komga.service and access it via IP at http://<your-server-ip>:<port>(/base_url) as stated at Komga: Accessing the web interface, then you can continue with the mangal section, else continue with the next steps for the reverse proxy and certificate.

Reverse proxy¶

Create the reverse proxy configuration (this is for nginx). In my case I’ll use a subdomain, so this is a new config called komga.conf at the usual sites-available/enabled path:

server {
-    listen 80;
-    server_name komga.yourdomain.com; # change accordingly to your wanted subdomain and domain name
-
-    location / {
-        proxy_pass http://localhost:25600; # change port if needed
-        proxy_http_version 1.1;
-
-        proxy_set_header Host $host;
-        proxy_set_header X-Real-IP $remote_addr;
-        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-        proxy_set_header X-Forwarded-Proto $scheme;
-
-        proxy_read_timeout 600s;
-        proxy_send_timeout 600s;
-    }
-}
-

If it’s going to be used as a subdir on another domain then just change the location with /subdir instead of /; be careful with the proxy_pass directive, it has to match what you configured at /etc/komga.conf for the SERVER_SERVLET_CONTEXT_PATH regardless of the /subdir you selected at location.

SSL certificate¶

If using a subdir then the same certificate for the subdomain/domain should work fine and no extra stuff is needed, else if following along me then we can create/extend the certificate by running:

certbot --nginx
-

That will automatically detect the new subdomain config and create/extend your existing certificate(s). In my case I manage each certificate’s subdomain:

certbot --nginx -d domainname.com -d subdomain.domainname.com -d komga.domainname.com
-

Start using Komga¶

We can now start/enable the komga.service:

systemctl enable komga.service
-systemctl start komga.service
-

And access the web interface at https://komga.domainname.com which should show the login page for Komga. The first time it will ask to create an account as shown in Komga: Create user account, this will be an admin account. Fill in the email and password (can be changed later). The email doesn’t have to be an actual email, for now it’s just for management purposes.

Next thing would be to add any extra account (for read-only/download manga permissions), add/import libraries, etc.. For now I’ll leave it here until we start downloading manga on the next steps.

Library creation¶

Creating a library is as simple as creating a directory somewhere and point to it in Komga. The following examples are for my use case, change accordingly. I’ll be using /mnt/d/mangal for my library (as stated in the mangal: configuration section):

mkdir /mnt/d/mangal
-

Where I chose the name mangal as its the name of the downloader/scrapper, it could be anything, this is just how I like to organize stuff.

For the most part, the permissions don’t matter much (as long as it’s readable by the komga user) unless you want to delete some manga, then komga user also needs write permissions.

Then just create the library in Komga web interface (the + sign next to Libraries), choose a name “Mangal” and point to the root folder /mnt/d/mangal, then just click Next, Next and Add for the defaults (that’s how I’ve been using it so far). This is well explained at Komga: Libraries.

The real important part (for me) is the permissions of the /mnt/d/mangal directory, as I want to have write access for komga so I can manage from the web interface itself. It looks like it’s just a matter of giving ownership to the komga user either for owner or for group (or to all for that matter), but since I’m going to use a separate user to download manga then I need to choose carefully.

Set default directory permissions¶

The desired behaviour is: set komga as group ownership, set write access to group and whenever a new directory/file is created, inherit these permission settings. I found out via this stack exchange answer how to do it. So, for me:

chown manga-dl:komga /mnt/d/mangal # required for group ownership for komga
-chmod g+s /mnt/d/mangal # required for group permission inheritance
-setfacl -d -m g::rwx /mnt/d/mangal # default permissions for group
-setfacl -d -m o::rx /mnt/d/mangal # default permissions for other (as normal, I think this command can be excluded)
-

Where manga-dl is the user I created to download manga with. Optionally add -R flag to those 4 commands in case it already has subdirectories/files (this might mess file permissions, but it’s not an issue as far as I konw).

Checking that the permissions are set correctly (getfacl /mnt/d/mangal):

getfacl: Removing leading '/' from absolute path names
-# file: mnt/d/mangal
-# owner: manga-dl
-# group: komga
-# flags: -s-
-user::rwx
-group::rwx
-other::r-x
-default:user::rwx
-default:group::rwx
-default:other::r-x
-

You can then check by creating a new subdirectory (in /mnt/d/mangal) and it should have the same group permissions.

Populate manga library¶

You can now start downloading manga using mangal either manually or by running the cron/systemd/Timers and it will be detected by Komga automatically when it scans the library (once every hour according to my config). You can manually scan the library, though, by clicking on the 3 dots to the right of the library name (in Komga) and click on “Scan library files”.

Then you can check that the metadata is correct (once the manga is fully indexed and metadata finished building), such as title, summary, chapter count, language, tags, genre, etc., which honestly it never works fine as mangal creates the series.json with the comicId field with an upper case I and Komga expects it to be a lower case i (comicid) so it falls back to using the info from the first chapter. I’ll probably will fix this on mangal side, and see how it goes.

So, what I do is manually edit the metadata for the manga, by changing whatever it’s wrong or add what’s missing (I like adding anilist and MyAnimeList links) and then leave it as is. This is up to you.

Alternative downloaders¶

Just for the record, here is a list of downloaders/scrapers I considered before starting to use mangal:

kaizoku: NodeJS web server that uses mangal for its “backend” and honestly since I liked mangal so much I should use it, the only reason I don’t is because I’m a bitch and I don’t want to use a D*ck*r image and NodeJS (ew) (in general is pretty bloated in my opinion). If I get tired of my solution with pure mangal I might as well just migrate to it as It’s a more automatic solution.
manga-py: Python CLI application that’s a really good option as far as I’ve explored it, I’m just not using it yet as mangal has been really smooth and has everything I need, but will definitely explore it in the future if I need to. The cool thing out of the box is the amount of sources it can scrape from (somethign lacking from mangal).
mylar3: Python web server that should be the easier way to download manga with once correctly set up, but I guess I’m too dumb and don’t know how to configure it. Looks like you need to have access to specific private torrent trackers or whatever the other ways to download are, I just couldn’t figure out how to set it up and for public torrent stuff everything will be all over the place, so this was no option for me at the end.

Others:

HakuNeku: It looks pretty easy to use and future rich, only thing is that it’s not designed for headless servers, just a normal app. So this is also not an option for me. You could use it on your computer and rsync to your server or use some other means to upload to your server (a nono for me).
FMD: No fucking idea on how to use it and it’s not been updated since 2019, just listing it here as an option if it interests you.

- -

- - - - Next - - - - - - - Index - - - - - - - Previous - - -

- - -

By David Luévano

Created: Sat, Jun 10, 2023 @ 19:36 UTC

Modified: Wed, Jun 28, 2023 @ 18:35 UTC

Tags: -code, english, server, tools, tutorial

- -

Set up a pastebin alternative with PrivateBin and YOURLS

- -

I learned about PrivateBin a few weeks back and ever since I’ve been looking into installing it, along with a URL shortener (a service I wanted to self host since forever). It took me a while as I ran into some problems while experimenting and documenting all the necessary bits in here.

My setup is exposed to the public, and as always is heavily based on previous entries as described in Prerequisites. Descriptions on setting up MariaDB (preferred MySQL replacement for Arch) and PHP are written in this entry as this is the first time I’ve needed them.

Everything here is performed in arch btw and all commands should be run as root unless stated otherwise.

Table of contents¶

Table of contents
Prerequisites
MariaDB
- Create users/databases
-
PHP
- Configuration
- Nginx
-
YOURLS
- Configuration
- Nginx
  - SSL certificate
  -
- Usage
-
PrivateBin
- Configuration
  - YOURLS integration
  -
- Nginx
  - SSL certificate
  -
-

Prerequisites¶

If you want to expose to a (sub)domain, then similar to my early tutorial entries (specially the website for the reverse proxy plus certificates):

nginx for the reverse proxy.
certbot for the SSL certificates.
yay to install AUR packages.
- I briefly mention how to install and use it on Manga server with Komga: yay.
-
An A (and/or AAAA) or a CNAME for privatebin and yourls (or whatever you want to call them).

MariaDB¶

MariaDB is a drop-in replacement of MySQL.

Install the mariadb package:

pacman -S mariadb
-

Before starting/enabling the systemd service run:

mariadb-install-db --user=mysql --basedir=/usr --datadir=/var/lib/mysql
-

start/enable the mariadb.service:

systemctl start mariadb.service
-systemctl enable mariadb.service
-

Run and follow the secure installation script before proceding any further:

mariadb-secure-installation
-

Change the binding address so the service listens on localhost only by modifying /etc/my.cnf.d/server.cnf:

[mariadb]
-bind-address = localhost
-

Create users/databases¶

To use mariadb simply run the command and it will try to login with the corresponding linux user running it. The general login command is:

mariadb -u <username> -p <database_name>
-

The database_name is optional. It will prompt a password input field.

Using mariadb as root, create users with their respective database if needed with the following queries:

MariaDB> CREATE USER '<username>'@'localhost' IDENTIFIED BY '<password>';
-MariaDB> CREATE DATABASE <database_name>;
-MariaDB> GRANT ALL PRIVILEGES ON <database_name>.* TO '<username>'@'localhost';
-MariaDB> quit
-

The database_name will depend on how YOURLS and PrivateBin are configured, that is if the services use a separate database and/or table prefixes are used.

PHP¶

PHP is a general-purpose scripting language that is usually used for web development, which was supposed to be ass for a long time but it seems to be a misconseption from the old times.

Install the php, php-fpm, php-gd packages:

pacman -S php php-fpm php-gd
-

start/enable the php-fpm.service:

systemctl start php-fpm.service
-systemctl enable php-fpm.service
-

Configuration¶

Only showing changes needed, main config file is located at /etc/php/php.ini, or drop-in files can be placed at /etc/php/conf.d/ instead.

Set timezone (list of timezones):

date.timezone = Europe/Berlin
-

Enable the gd and mysql extensions:

extension=gd
-extension=pdo_mysql
-extension=mysqli
-

Nginx¶

Create a PHP specific config that can be reusable at /etc/nginx/php_fastcgi.conf:

location ~ \.php$ {
-    # required for yourls
-    add_header Access-Control-Allow-Origin $http_origin;
-
-    # 404
-    try_files $fastcgi_script_name =404;
-
-    # default fastcgi_params
-    include fastcgi_params;
-
-    # fastcgi settings
-    fastcgi_pass                        unix:/run/php-fpm/php-fpm.sock;
-    fastcgi_index                       index.php;
-    fastcgi_buffers                     8 16k;
-    fastcgi_buffer_size         32k;
-
-    # fastcgi params
-    fastcgi_param DOCUMENT_ROOT $realpath_root;
-    fastcgi_param SCRIPT_FILENAME       $realpath_root$fastcgi_script_name;
-    #fastcgi_param PHP_ADMIN_VALUE      "open_basedir=$base/:/usr/lib/php/:/tmp/";
-}
-

This then can be imported by any server directive that needs it.

YOURLS¶

YOURLS is a self-hosted URL shortener that is supported by PrivateBin.

Install from the AUR with yay:

yay -S yourls
-

Create a new user and database as described in MariaDB: Create users/databases.

Configuration¶

The default configuration file is self explanatory, it is located at /etc/webapps/yourls/config.php. Make sure to correctly set the user/database YOURLS will use and either create a cookie or get one from URL provided.

It is important to change the $yours_user_passwords variable, YOURLS will hash the passwords on login so it is not stored in plaintext. Password hashing can be disabled with:

define( 'YOURLS_NO_HASH_PASSWORD', true );
-

I also changed the “shortening method” to 62 to include more characters:

define( 'YOURLS_URL_CONVERT', 62 );
-

The $yourls_reserved_URL variable will need more blacklisted words depending on the use-case. Make sure the YOURLS_PRIVATE variable is set to true (default) if the service will be exposed to the public.

Nginx¶

Create a yourls.conf at the usual sites-<available/enabled> path for nginx:

server {
-    listen 80;
-    root /usr/share/webapps/yourls/;
-    server_name short.example.com;
-    index index.php;
-
-    location / {
-        try_files $uri $uri/ /yourls-loader.php$is_args$args;
-    }
-
-    include /etc/nginx/php_fastcgi.conf;
-}
-

Make sure the following header is included in the php‘s nginx location block described in YOURLS: Nginx:

add_header Access-Control-Allow-Origin $http_origin;
-

SSL certificate¶

Create/extend the certificate by running:

certbot --nginx
-

Restart the nginx service for changes to take effect:

systemctl restart nginx.service
-

Usage¶

The admin area is located at https://short.example.com/admin/, login with any of the configured users set with the $yours_user_passwords in the config. Activate plugins by going to the “Manage Plugins” page (located at the top left) and clicking in the respective “Activate” button by hovering the “Actin” column, as shown below:

I personally activated the “Random ShortURLs” and “Allow Hyphens in Short URLs”. Once the “Random ShortURLs” plugin is activated it can be configured by going to the “Random ShortURLs Settings” page (located at the top left, right below “Manage Plugins”), only config available is “Random Keyword Length”.

The main admin area can be used to manually shorten any link provided, by using the automatic shortening or by providing a custom short URL.

Finally, the “Tools” page (located at the top left) conains the signature token, used for YOURLS: Passwordless API as well as useful bookmarklets for URL shortening while browsing.

PrivateBin¶

PrivateBin is a minimalist self-hosted alternative to pastebin.

Install from the AUR with yay:

yay -S privatebin
-

Create a new user and database as described in MariaDB: Create users/databases.

Configuration¶

This heavily depends on personal preference, all defaults are fine. Make a copy of the sample config template:

cp /etc/webapps/privatebin/conf.sample.php /etc/webapps/privatebin/conf.php
-

The most important changes needed are basepath according to the privatebin URL and the [model] and [model_options] to use MySQL instead of plain filesystem files:

[model]
-; example of DB configuration for MySQL
-class = Database
-[model_options]
-dsn = "mysql:host=localhost;dbname=privatebin;charset=UTF8"
-tbl = "privatebin_"     ; table prefix
-usr = "privatebin"
-pwd = "<password>"
-opt[12] = true    ; PDO::ATTR_PERSISTENT
-

Any other [model] or [model_options] needs to be commented out (for example, the default filesystem setting).

YOURLS integration¶

I recommend creating a separate user for privatebin in yourls by modifying the $yours_user_passwords variable in yourls config file. Then login with this user and get the signature from the “Tools” section in the admin page, for more: YOURLS: Passwordless API.

For a “private” yourls installation (that needs username/pasword), set urlshortener:

urlshortener = "https://short.example.com/yourls-api.php?signature=xxxxxxxxxx&action=shorturl&format=json&url="
-

Note that this will expose the signature in the HTTP requests and anybody with the signature can use it to shorten external URLs.

Nginx¶

To deny access to some bots/crawlers, PrivateBin provides a sample .htaccess, which is used in Apache. We need an Nginx version, which I found here.

Add the following at the beginning of the http block of the /etc/nginx/nginx.conf file:

http {
-    map $http_user_agent $pastebin_badagent {
-        ~*bot 1;
-        ~*spider 1;
-        ~*crawl 1;
-        ~https?:// 1;
-        WhatsApp 1;
-        SkypeUriPreview 1;
-        facebookexternalhit 1;
-    }
-
-    #...
-}
-

Create a privatebin.conf at the usual sites-<available/enabled> path for nginx:

server {
-    listen 80;
-    root //usr/share/webapps/privatebin/;
-    server_name bin.example.com;
-    index index.php;
-
-    if ($pastebin_badagent) {
-       return 403;
-    }
-
-    location / {
-        try_files $uri $uri/ /index.php$is_args$args;
-    }
-
-    include /etc/nginx/php_fastcgi.conf;
-}
-

SSL certificate¶

Create/extend the certificate by running:

certbot --nginx
-

Restart the nginx service for changes to take effect:

systemctl restart nginx.service
-

- -

- - - - Next - - - - - - - Index - - - - - - - Previous - - -

- - -

By David Luévano

Created: Sun, Aug 20, 2023 @ 09:46 UTC

Modified: Sun, Aug 20, 2023 @ 10:28 UTC

Tags: -code, english, server, tools, tutorial

- -

Set up qBitTorrent with Jackett for use with Starr apps

- -

Riding on my excitement of having a good internet connection and having setup my home server now it’s time to self host a media server for movies, series and anime. I’ll setup qBitTorrent as the downloader, Jackett for the trackers, the Starr apps for the automatic downloading and Jellyfin as the media server manager/media viewer. This was going to be a single entry but it ended up being a really long one so I’m splitting it, this being the first part.

I’ll be exposing my stuff on a subdomain only so I can access it while out of home and for SSL certificates (not required), but shouldn’t be necessary and instead you can use a VPN (how to set up). For your reference, whenever I say “Starr apps” (*arr apps) I mean the family of apps such as Sonarr, Radarr, Bazarr, Readarr, Lidarr, etc..

Most of my config is based on the TRaSH-Guides (will be mentioned as “TRaSH” going forward). Specially get familiar with the TRaSH: Native folder structure and with the TRaSH: Hardlinks and instant moves. Will also use the default configurations based on the respective documentation for each Starr app and service, except when stated otherwise.

Everything here is performed in arch btw and all commands should be run as root unless stated otherwise.

Kindly note that I do not condone the use of torrenting for illegal activities. I take no responsibility for what you do when setting up anything shown here. It is for you to check your local laws before using automated downloaders such as Sonarr and Radarr.

Table of contents¶

Table of contents
Prerequisites
- Directory structure
-
Jackett
- Reverse proxy
  - SSL certificate
  -
- Start using Jackett
  - Indexers
  -
- FlareSolverr
-
qBitTorrent
- Reverse proxy
- Start using qBitTorrent
  - Configuration
  - Trackers
  -
-

Prerequisites¶

The specific programs are mostly recommendations, if you’re familiar with something else or want to change things around, feel free to do so but everything will be written with them in mind.

If you want to expose to a (sub)domain, then similar to my early tutorial entries (specially the website for the reverse proxy plus certificates):

nginx for the reverse proxy.
certbot for the SSL certificates.
ufw for the firewall.
yay to install AUR packages.
- I mentioned how to install and use it on my previous entry: Manga server with Komga: yay.
-
An A (and/or AAAA) or a CNAME for isos (or whatever you want to call it).
- For automation software (qBitTorrent, Jackett, Starr apps, etc.). One subdomain per service can be used instead.
-

Note: I’m using the explicit 127.0.0.1 ip instead of localhost in the reverse proxies/services config as localhost resolves to ipv6 sometimes which is not configured on my server correctly. If you have it configured you can use localhost without any issue.

Directory structure¶

Basically following TRaSH: Native folder structure except for the directory permissions part, I’ll do the same as with my Komga setup guide to stablish default group permissions.

The desired behaviour is: set servarr as group ownership, set write access to group and whenever a new directory/file is created, inherit these permission settings. servarr is going to be a service user and I’ll use the root of a mounted drive at /mnt/a.

Create a service user called servarr (it could just be a group, too):

useradd -r -s /usr/bin/nologin -M -c "Servarr applications" servarr
-

Create the torrents directory and set default permissions:

cd /mnt/a # change this according to your setup
-mkdir torrents
-chown servarr:servarr torrents
-chmod g+w torrents
-chmod g+s torrents
-setfacl -d -m g::rwx torrents
-setfacl -d -m o::rx torrents
-

Check that the permissions are set correctly (getfacl torrents)

# file: torrents/
-# owner: servarr
-# group: servarr
-# flags: -s-
-user::rwx
-group::rwx
-other::r-x
-default:user::rwx
-default:group::rwx
-default:other::r-x
-

Create the subdirectories you want with any user (I’ll be using servarr personally):

mkdir torrents/{tv,movies,anime}
-chown -R servarr: torrents
-

Finally repeat steps 2 - 4 for the media directory.

The final directory structure should be the following:

root_dir
-├── torrents
-│   ├── movies
-│   ├── music
-│   └── tv
-└── media
-    ├── movies
-    ├── music
-    └── tv
-

Where root_dir is /mnt/a in my case. This is going to be the reference for the following applications set up.

Later, add the necessary users to the servarr group if they need write access, by executing:

gpasswd -a <USER> servarr
-

Jackett¶

Jackett is a “proxy server” (or “middle-ware”) that translates queries from apps (such as the Starr apps in this case) into tracker-specific http queries. Note that there is an alternative called Prowlarr that is better integrated with most if not all Starr apps, requiring less maintenance; I’ll still be sticking with Jackett, though.

Install from the AUR with yay:

yay -S jackett
-

I’ll be using the default 9117 port, but change accordingly if you decide on another one.

Reverse proxy¶

I’m going to have most of the services under the same subdomain, with different subdirectories. Create the config file isos.conf at the usual sites-available/enabled path for nginx:

server {
-    listen 80;
-    server_name isos.yourdomain.com;
-
-    location /jack { # you can change this to jackett or anything you'd like, but it has to match the jackett config on the next steps
-        proxy_pass http://127.0.0.1:9117; # change the port according to what you want
-
-        proxy_set_header X-Real-IP $remote_addr;
-        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-        proxy_set_header X-Forwarded-Proto $scheme;
-        proxy_set_header X-Forwarded-Host $http_host;
-        proxy_redirect off;
-    }
-}
-

This is the basic reverse proxy config as shown in Jackett: Running Jackett behind a reverse proxy. The rest of the services will be added under different location block on their respective steps.

SSL certificate¶

Create/extend the certificate by running:

certbot --nginx
-

That will automatically detect the new subdomain config and create/extend your existing certificate(s). Restart the nginx service for changes to take effect:

systemctl restart nginx.service
-

Start using Jackett¶

You can now start/enable the jackett.service:

systemctl enable jackett.service
-systemctl start jackett.service
-

It will autocreate the default configuration under /var/lib/jackett/ServerConfig.json, which you need to edit at least to change the BasePathOverride to match what you used in the nginx config:

{
-    "Port": 9117,
-    "SomeOtherConfigs": "some_other_values",
-    "BasePathOverride": "/jack",
-    "MoreConfigs": "more_values",
-}
-

Also modify the Port if you changed it. Restart the jackett service:

systemctl restart jackett.service
-

It should now be available at https://isos.yourdomain.com/jack. Add an admin password right away by scroll down and until the first config setting; don’t forget to click on “Set Password”. Change any other config you want from the Web UI, too (you’ll need to click on the blue “Apply server settings” button).

Note that you need to set the “Base URL override” to http://127.0.0.1:9117 (or whatever port you used) so that the “Copy Torznab Feed” button works for each indexer.

Indexers¶

For Jackett, an indexer is just a configured tracker for some of the commonly known torrent sites. Jackett comes with a lot of pre-configured public and private indexers which usually have multiple URLs (mirrors) per indexer, useful when the main torrent site is down. Some indexers come with extra features/configuration depending on what the site specializes on.

To add an indexer click on the “+ Add Indexer” at the top of the Web UI and look for indexers you want, then click on the “+” icon on the far-most right for each indexer or select the ones you want (clicking on the checkbox on the far-most left of the indexer) and scroll all the way to the bottom to click on “Add Selected”. They then will show as a list with some available actions such as “Copy RSS Feed”, “Copy Torznab Feed”, “Copy Potato Feed”, a button to search, configure, delete and test the indexer, as shown below:

Jacket: configured indexers — Jackett: configured indexers

You can manually test the indexers by doing a basic search to see if they return anything, either by searching on each individual indexer by clicking on the magnifying glass icon on the right of the indexer or clicking on “Manual Search” button which is next to the “+ Add Indexer” button at the top right.

Explore each indexer’s configuration in case there is stuff you might want to change.

FlareSolverr¶

FlareSolverr is used to bypass certain protection that some torrent sites have. This is not 100% necessary and only needed for some trackers sometimes, it also doesn’t work 100%.

You could install from the AUR with yay:

yay -S flaresolverr-bin
-

At the time of writing, the flaresolverr package didn’t work for me because of python-selenium. flaresolverr-bin was updated around the time I was writing this, so that is what I’m using and what’s working fine so far, it contains almost everything needed (it has self contained libraries) except for xfvb.

Install dependencies via pacman:

pacman -S xorg-server-xvfb
-

You can now start/enable the flaresolverr.service:

systemctl enable flaresolverr.service
-systemctl start flaresolverr.service
-

Verify that the service started correctly by checking the logs:

journalctl -fxeu flaresolverr
-

It should show “Test successful” and “Serving on http://0.0.0.0:8191” (which is the default). Jackett now needs to be configured by adding http://127.0.0.1:8191 almost at the end in the “FlareSolverr API URL” field, then click on the blue “Apply server settings” button at the beginning of the config section. This doesn’t need to be exposed or anything, it’s just an internal API that Jackett (or anything you want) will use.

qBitTorrent¶

qBitTorrent is a fast, stable and light BitTorrent client that comes with many features and in my opinion it’s a really user friendly client and my personal choice for years now. But you can choose whatever client you want, there are more lightweight alternatives such as Transmission.

Install the qbittorrent-nox package (“nox” as in “no X server”):

pacman -S qbittorrent-nox
-

By default the package doesn’t create any (service) user, but it is recommended to have one so you can run the service under it. Create the user, I’ll call it qbittorrent and leave it with the default homedir (/home):

useradd -r -m qbittorrent
-

Add the qbittorrent user to the servarr group:

gpasswd -a qbittorrent servarr
-

Decide a port number you’re going to run the service on for the next steps, the default is 8080 but I’ll use 30000; it doesn’t matter much, as long as it matches for all the config. This is the qbittorrent service port, it is used to connect to the instance itself through the Web UI or via API, you also need to open a port for listening to peer connections. Choose any port you want, for example 50000 and open it with your firewall, ufw in my case:

ufw allow 50000/tcp comment "qBitTorrent - Listening port"
-

Reverse proxy¶

Add the following location block into the isos.conf with whatever subdirectory name you want, I’ll call it qbt:

location /qbt/ {
-    proxy_pass http://localhost:30000/; # change port to whatever number you want
-    proxy_http_version 1.1;
-
-    proxy_set_header Host $host;
-    proxy_set_header X-Forwarded-Host $http_host;
-    proxy_set_header X-Forwarded-For $remote_addr;
-
-    proxy_cookie_path / "/; Secure";
-    proxy_read_timeout 600s;
-    proxy_send_timeout 600s;
-}
-

This is taken from qBitTorrent: Nginx reverse proxy for Web UI. Restart the nginx service for the changes to take effect:

systemctl restart nginx.service
-

Start using qBitTorrent¶

You can now start/enable the qbittorrent-nox@.service using the service account created (qbittorrent):

systemctl enable `qbittorrent-nox@qbittorrent.service
-systemctl start `qbittorrent-nox@qbittorrent.service
-

This will start qbittorrent using default config. You need to change the port (in my case to 30000) and set qbittorrent to restart on exit (the Web UI has a close button). I guess this can be done before enabling/starting the service, but either way let’s create a “drop-in” file by “editing” the service:

systemctl edit `qbittorrent-nox@qbittorrent.service
-

Which will bring up a file editing mode containing the service unit and a space where you can add/override anything, add:

[Service]
-Environment="QBT_WEBUI_PORT=30000" # or whatever port number you want
-Restart=on-success
-RestartSec=5s
-

When exiting from the file (if you wrote anything) it will create the override config. Restart the service for changes to take effect (you might be asked to reload the systemd daemon):

systemctl restart `qbittorrent-nox@qbittorrent.service
-

You can now head to https://isos.yourdomain.com/qbt/ and login with user admin and password adminadmin (by default). Change the default password right away by going to Tools -> Options -> Web UI -> Authentication. The Web UI is basically the same as the normal desktop UI so if you’ve used it it will feel familiar. From here you can threat it as a normal torrent client and even start using for other stuff other than the specified here.

Configuration¶

It should be usable already but you can go further and fine tune it, specially to some kind of “convention” as shown in TRaSH: qBitTorrent basic setup and subsequent qbittorrent guides.

I use all the suggested settings by TRaSH, where the only “changes” are for personal paths, ports, and in general connection settings that depend on my setup. The only super important setting I noticed that can affect our setup (meaning what is described in this entry) is the Web UI -> Authentication for the “Bypass authentication for clients on localhost”. This will be an issue because the reverse proxy is accessing qbittorrent via localhost, so this will make the service open to the world, experiment at your own risk.

Finally, add categories by following TRaSH: qBitTorrent how to add categories, basically right clicking on Categories -> All (x) (located at the left of the Web UI) and then on “Add category”; I use the same “Category” and “Save Path” (tv and tv, for example), where the “Save Path” will be a subdirectory of the configured global directory for torrents (TRaSH: qBitTorent paths and categories breakdown). I added 3: tv, movies and anime.

Trackers¶

Often some of the trackers that come with torrents are dead or just don’t work. You have the option to add extra trackers to torrents either by:

Automatically add a predefined list on new torrents: configure at Tools -> Options -> BitTorrent, enable the last option “Automatically add these trackers to new downloads” then add the list of trackers. This should only be done on public torrents as private ones might ban you or something.
Manually add a list of trackers to individual torrents: configure by selecting a torrent, clicking on Trackers on the bottom of the Web UI, right clicking on an empty space and selecting “Add trackers…” then add the list of trackers.

On both options, the list of trackers need to have at least one new line in between each new tracker. You can find trackers from the following sources:

List of stable trackers
ngosang/trackerslist
- It also mentions Third party tools to automate this process.
-

Both sources maintain an updated list of trackers. You also might need to enable an advanced option so all the new trackers are contacted (Only first tracker contacted): configure at Tools -> Options -> Advanced -> libtorrent Section and enable both “Always announce to all tiers” and “Always announce to all trackers in a tier”.

- -

- - - - Next - - - - - - - Index - - - - - - - Previous - - -

- - -

By David Luévano

Created: Mon, Jul 24, 2023 @ 02:06 UTC

Modified: Mon, Jul 24, 2023 @ 03:51 UTC

Tags: -code, english, server, tools, tutorial

- -

Set up a VPN server with OpenVPN

- -

I’ve been wanting to do this entry, but had no time to do it since I also have to set up the VPN service as well to make sure what I’m writing makes sense, today is the day.

Like with any other of my entries I based my setup on the Arch Wiki, this install script and this profile generator script.

This will be installed and working alongside the other stuff I’ve wrote about on other posts (see the server tag). All commands here are executes as root unless specified otherwise. Also, this is intended only for IPv4 (it’s not that hard to include IPv6, but meh). As always, all commands are executed as root unless stated otherwise.

Table of contents¶

Table of contents
Prerequisites
Create PKI from scratch
OpenVPN
- Enable forwarding
- Create client configurations
-

Prerequisites¶

Pretty simple:

Working server with root access, and with ufw as the firewall.
Open port 1194 (default), or as a fallback on 443 (click here for more). I will do mine on port 1194 but it’s just a matter of changing 2 lines of configuration and one ufw rule.

Create PKI from scratch¶

PKI stands for Public Key Infrastructure and basically it’s required for certificates, private keys and more. This is supposed to work between two servers and one client: a server in charge of creating, signing and verifying the certificates, a server with the OpenVPN service running and the client making the request.

In a nutshel, this is supposed to work something like: 1) a client wants to use the VPN service, so it creates a requests and sends it to the signing server, 2) this server checks the requests and signs the request, returning the certificates to both the VPN service and the client and 3) the client can now connect to the VPN service using the signed certificate which the OpenVPN server knows about.

That’s how the it should be st up… but, to be honest, all of this is a hassle and (in my case) I want something simple to use and manage. So I’m gonna do all on one server and then just give away the configuration file for the clients, effectively generating files that anyone can run and will work, meaning that you need to be careful who you give this files (it also comes with a revoking mechanism, so no worries).

This is done with Easy-RSA.

Install the easy-rsa package:

pacman -S easy-rsa
-

Initialize the PKI and generate the CA keypair:

cd /etc/easy-rsa
-easyrsa init-pki
-easyrsa build-ca nopass
-

Create the server certificate and private key (while in the same directory):

EASYRSA_CERT_EXPIRE=3650 easyrsa build-server-full server nopass
-

Where server is just a name to identify your server certificate keypair, I just use server but could be anything (like luevano.xyz in my case).

Create the client revocation list AKA CRL (will be used later, but might as well have it now):

EASYRSA_CRL_DAYS=3650 easyrsa gen-crl
-

After this we should have 6 new files:

/etc/easy-rsa/pki/ca.crt
-/etc/easy-rsa/pki/private/ca.key
-/etc/easy-rsa/pki/issued/server.crt
-/etc/easy-rsa/pki/reqs/server.req
-/etc/easy-rsa/pki/private/server.key
-/etc/easy-rsa/pki/crl.pem
-

It is recommended to copy some of these files over to the openvpn directory, but I prefer to keep them here and just change some of the permissions:

chmod o+rx pki
-chmod o+rx pki/ca.crt
-chmod o+rx pki/issued
-chmod o+rx pki/issued/server.crt
-chmod o+rx pki/private
-chmod o+rx pki/private/server.key
-chown nobody:nobody pki/crl.pem
-chmod o+r pki/crl.pem
-

Finally, go to the openvpn directory and create the required files there:

cd /etc/openvpn/server
-openssl dhparam -out dh.pem 2048
-openvpn --genkey secret ta.key
-

OpenVPN¶

OpenVPN is a robust and highly flexible VPN daemon, that’s pretty complete feature-wise.

Install the openvpn package:

pacman -S openvpn
-

Now, most of the stuff is going to be handled by (each, if you have more than one) server configuration. This might be the hardest thing to configure, but I’ve used a basic configuration file that worked a lot to me, which is a compilation of stuff that I found on the internet while configuring the file a while back.

# Server ip addres (ipv4).
-local 1.2.3.4 # your server public ip
-
-# Port.
-port 1194 # Might want to change it to 443
-
-# TCP or UDP.
-;proto tcp
-proto udp # If ip changes to 443, you should change this to tcp, too
-
-# "dev tun" will create a routed IP tunnel,
-# "dev tap" will create an ethernet tunnel.
-;dev tap
-dev tun
-
-# Server specific certificates and more.
-ca /etc/easy-rsa/pki/ca.crt
-cert /etc/easy-rsa/pki/issued/server.crt
-key /etc/easy-rsa/pki/private/server.key  # This file should be kept secret.
-dh /etc/openvpn/server/dh.pem
-auth SHA512
-tls-crypt /etc/openvpn/server/ta.key 0 # This file is secret.
-crl-verify /etc/easy-rsa/pki/crl.pem
-
-# Network topology.
-topology subnet
-
-# Configure server mode and supply a VPN subnet
-# for OpenVPN to draw client addresses from.
-server 10.8.0.0 255.255.255.0
-
-# Maintain a record of client <-> virtual IP address
-# associations in this file.
-ifconfig-pool-persist ipp.txt
-
-# Push routes to the client to allow it
-# to reach other private subnets behind
-# the server.
-;push "route 192.168.10.0 255.255.255.0"
-;push "route 192.168.20.0 255.255.255.0"
-
-# If enabled, this directive will configure
-# all clients to redirect their default
-# network gateway through the VPN, causing
-# all IP traffic such as web browsing and
-# and DNS lookups to go through the VPN
-push "redirect-gateway def1 bypass-dhcp"
-
-# Certain Windows-specific network settings
-# can be pushed to clients, such as DNS
-# or WINS server addresses.
-# Google DNS.
-;push "dhcp-option DNS 8.8.8.8"
-;push "dhcp-option DNS 8.8.4.4"
-
-# The keepalive directive causes ping-like
-# messages to be sent back and forth over
-# the link so that each side knows when
-# the other side has gone down.
-keepalive 10 120
-
-# The maximum number of concurrently connected
-# clients we want to allow.
-max-clients 5
-
-# It's a good idea to reduce the OpenVPN
-# daemon's privileges after initialization.
-user nobody
-group nobody
-
-# The persist options will try to avoid
-# accessing certain resources on restart
-# that may no longer be accessible because
-# of the privilege downgrade.
-persist-key
-persist-tun
-
-# Output a short status file showing
-# current connections, truncated
-# and rewritten every minute.
-status openvpn-status.log
-
-# Set the appropriate level of log
-# file verbosity.
-#
-# 0 is silent, except for fatal errors
-# 4 is reasonable for general usage
-# 5 and 6 can help to debug connection problems
-# 9 is extremely verbose
-verb 3
-
-# Notify the client that when the server restarts so it
-# can automatically reconnect.
-# Only usable with udp.
-explicit-exit-notify 1
-

# and ; are comments. Read each and every line, you might want to change some stuff (like the logging), specially the first line which is your server public IP.

Enable forwarding¶

Now, we need to enable packet forwarding (so we can access the web while connected to the VPN), which can be enabled on the interface level or globally (you can check the different options with sysctl -a | grep forward). I’ll do it globally, run:

sysctl net.ipv4.ip_forward=1
-

And create/edit the file /etc/sysctl.d/30-ipforward.conf:

net.ipv4.ip_forward=1
-

Now we need to configure ufw to forward traffic through the VPN. Append the following to /etc/default/ufw (or edit the existing line):

...
-DEFAULT_FORWARD_POLICY="ACCEPT"
-...
-

And change the /etc/ufw/before.rules, appending the following lines after the header but before the *filter line:

...
-# NAT (Network Address Translation) table rules
-*nat
-:POSTROUTING ACCEPT [0:0]
-
-# Allow traffic from clients to the interface
--A POSTROUTING -s 10.8.0.0/24 -o interface -j MASQUERADE
-
-# do not delete the "COMMIT" line or the NAT table rules above will not be processed
-COMMIT
-
-# Don't delete these required lines, otherwise there will be errors
-*filter
-...
-

Where interface must be changed depending on your system (in my case it’s ens3, another common one is eth0); I always check this by running ip addr which gives you a list of interfaces (the one containing your server public IP is the one you want, or whatever interface your server uses to connect to the internet):

...
-2: ens3: <SOMETHING,SOMETHING> bla bla
-    link/ether bla:bla
-    altname enp0s3
-    inet my.public.ip.addr bla bla
-...
-

And also make sure the 10.8.0.0/24 matches the subnet mask specified in the server.conf file (in this example it matches). You should check this very carefully, because I just spent a good 2 hours debugging why my configuration wasn’t working, and this was te reason (I could connect to the VPN, but had no external connection to the web).

Finally, allow the OpenVPN port you specified (in this example its 1194/udp) and reload ufw:

ufw allow 1194/udp comment "OpenVPN"
-ufw reload
-

At this point, the server-side configuration is done and you can start and enable the service:

systemctl start openvpn-server@server.service
-systemctl enable openvpn-server@server.service
-

Where the server after @ is the name of your configuration, server.conf without the .conf in my case.

Create client configurations¶

You might notice that I didn’t specify how to actually connect the VPN. For that we need a configuration file similar to the server.conf file that we created.

The real way of doing this would be to run similar steps as the ones with easy-rsa locally, send them to the server, sign them, and retrieve them. Fuck all that, we’ll just create all configuration files on the server as I was mentioning earlier.

Also, the client configuration file has to match the server one (to some degree), to make this easier you can create a client-common file in /etc/openvpn/server with the following content:

client
-dev tun
-remote 1.2.3.4 1194 udp # change this to match your ip and port
-resolv-retry infinite
-nobind
-persist-key
-persist-tun
-remote-cert-tls server
-auth SHA512
-verb 3
-

Where you should make any changes necessary, depending on your configuration.

Now, we need a way to create and revoke new configuration files. For this I created a script, heavily based on one of the links I mentioned at the beginning. You can place these scripts anywhere you like, and you should take a look before running them because you’ll be running them with elevated privileges (sudo).

In a nutshell, what it does is: generate a new client certificate keypair, update the CRL and create a new .ovpn configuration file that consists on the client-common data and all of the required certificates; or, revoke an existing client and refresh the CRL. The file is placed under ~/ovpn.

Create a new file with the following content (name it whatever you like) and don’t forget to make it executable (chmod +x vpn_script):

#!/bin/sh
-# Client ovpn configuration creation and revoking.
-MODE=$1
-if [ ! "$MODE" = "new" -a ! "$MODE" = "rev" ]; then
-    echo "$1 is not a valid mode, using default 'new'"
-    MODE=new
-fi
-
-CLIENT=${2:-guest}
-if [ -z $2 ];then
-    echo "there was no client name passed as second argument, using 'guest' as default"
-fi
-
-# Expiration config.
-EASYRSA_CERT_EXPIRE=3650
-EASYRSA_CRL_DAYS=3650
-
-# Current PWD.
-CPWD=$PWD
-cd /etc/easy-rsa/
-
-if [ "$MODE" = "rev" ]; then
-    easyrsa --batch revoke $CLIENT
-
-    echo "$CLIENT revoked."
-elif [ "$MODE" = "new" ]; then
-    easyrsa build-client-full $CLIENT nopass
-
-    # This is what actually generates the config file.
-    {
-    cat /etc/openvpn/server/client-common
-    echo "<ca>"
-    cat /etc/easy-rsa/pki/ca.crt
-    echo "</ca>"
-    echo "<cert>"
-    sed -ne '/BEGIN CERTIFICATE/,$ p' /etc/easy-rsa/pki/issued/$CLIENT.crt
-    echo "</cert>"
-    echo "<key>"
-    cat /etc/easy-rsa/pki/private/$CLIENT.key
-    echo "</key>"
-    echo "<tls-crypt>"
-    sed -ne '/BEGIN OpenVPN Static key/,$ p' /etc/openvpn/server/ta.key
-    echo "</tls-crypt>"
-    } > "$(eval echo ~${SUDO_USER:-$USER}/ovpn/$CLIENT.ovpn)"
-
-    eval echo "~${SUDO_USER:-$USER}/ovpn/$CLIENT.ovpn file generated."
-fi
-
-# Finish up, re-generates the crl
-easyrsa gen-crl
-chown nobody:nobody pki/crl.pem
-chmod o+r pki/crl.pem
-cd $CPWD
-

And the way to use is to run bash vpn_script <mode> <client_name> where mode is new or rev (revoke) as sudo (when revoking, it doesn’t actually delete the .ovpn file in ~/ovpn). Again, this is a little script that I put together, so you should check it out, it may need tweaks (specially depending on your directory structure for easy-rsa).

Now, just get the .ovpn file generated, import it to OpenVPN in your client of preference and you should have a working VPN service.

- -

- - - - Next - - - - - - - Index - - - - - - - Previous - - -

- - -

By David Luévano

Created: Sun, Aug 01, 2021 @ 09:27 UTC

Modified: Tue, Jun 13, 2023 @ 09:50 UTC

Tags: -code, english, server, tools, tutorial

- -

Set up a website with Nginx and Certbot

- -

These are general notes on how to setup a Nginx web server plus Certbot for SSL certificates, initially learned from Luke’s video and after some use and research I added more stuff to the mix. And, actually at the time of writing this entry, I’m configuring the web server again on a new VPS instance, so this is going to be fresh.

As a side note, i use arch btw so everything here es aimed at an Arch Linux distro, and I’m doing everything on a VPS. Also note that most if not all commands here are executed with root privileges.

Table of contents¶

Table of contents
Prerequisites
Nginx
Certbot

Prerequisites¶

You will need two things:

A domain name (duh!). I got mine on Epik (affiliate link, btw).
- With the corresponding A and AAA records pointing to the VPS’ IPs. I have three records for each type: empty string, “www” and “*” for a wildcard, that way “domain.name”, “www.domain.name”, “anythingelse.domain.name” point to the same VPS (meaning that you can have several VPS for different sub-domains). These depend on the VPS provider.
-
A VPS or somewhere else to host it. I’m using Vultr (also an affiliate link, btw).
- With ssh already configured both on the local machine and on the remote machine.
- Firewall already configured to allow ports 80 (HTTP) and 443 (HTTPS). I use ufw so it’s just a matter of doing ufw allow 80,443/tcp (for example) as root and you’re golden.
- cron installed if you follow along (you could use systemd timers, or some other method you prefer to automate running commands every certain time).
-

Nginx¶

Nginx is a web (HTTP) server and reverse proxy server.

You have two options: nginx and nginx-mainline. I prefer nginx-mainline because it’s the “up to date” package even though nginx is labeled to be the “stable” version. Install the package and enable/start the service:

pacman -S nginx-mainline
-systemctl enable nginx.service
-systemctl start nginx.service
-

And that’s it, at this point you can already look at the default initial page of Nginx if you enter the IP of your server in a web browser. You should see something like this:

As stated in the welcome page, configuration is needed, head to the directory of Nginx:

cd /etc/nginx
-

Here you have several files, the important one is nginx.conf, which as its name implies, contains general configuration of the web server. If you peek into the file, you will see that it contains around 120 lines, most of which are commented out and contains the welcome page server block. While you can configure a website in this file, it’s common practice to do it on a separate file (so you can scale really easily if needed for mor websites or sub-domains).

Inside the nginx.conf file, delete the server blocks and add the lines include sites-enabled/*; (to look into individual server configuration files) and types_hash_max_size 4096; (to get rid of an ugly warning that will keep appearing) somewhere inside the http block. The final nginx.conf file would look something like (ignoring the comments just for clarity, but you can keep them as side notes):

worker_processes 1;
-
-events {
-    worker_connections 1024;
-}
-
-http {
-    include sites-enabled/*;
-    include mime.types;
-    default_type application/octet-stream;
-
-    sendfile on;
-
-    keepalive_timeout 65;
-
-    types_hash_max_size 4096;
-}
-

Next, inside the directory /etc/nginx/ create the sites-available and sites-enabled directories, and go into the sites-available one:

mkdir sites-available
-mkdir sites-enabled
-cd sites-available
-

Here, create a new .conf file for your website and add the following lines (this is just the sample content more or less):

server {
-    listen 80;
-    listen [::]:80;
-
-    root /path/to/root/directory;
-    server_name domain.name another.domain.name;
-    index index.html anotherindex.otherextension;
-
-    location /{
-        try_files $uri $uri/ =404;
-    }
-}
-

That could serve as a template if you intend to add more domains.

Note some things:

listen: we’re telling Nginx which port to listen to (IPv4 and IPv6, respectively).
root: the root directory of where the website files (.html, .css, .js, etc. files) are located. I followed Luke’s directory path /var/www/some_folder.
server_name: the actual domain to “listen” to (for my website it is: server_name luevano.xyz www.luevano.xyz; and for this blog is: server_name blog.luevano.xyz www.blog.luevano.xyz;).
index: what file to serve as the index (could be any .html, .htm, .php, etc. file) when just entering the website.
location: what goes after domain.name, used in case of different configurations depending on the URL paths (deny access on /private, make a proxy on /proxy, etc).
- try_files: tells what files to look for.
-

Then, make a symbolic link from this configuration file to the sites-enabled directory:

ln -s /etc/nginx/sites-available/your_config_file.conf /etc/nginx/sites-enabled
-

This is so the nginx.conf file can look up the newly created server configuration. With this method of having each server configuration file separate you can easily “deactivate” any website by just deleting the symbolic link in sites-enabled and you’re good, or just add new configuration files and keep everything nice and tidy.

All you have to do now is restart (or enable and start if you haven’t already) the Nginx service (and optionally test the configuration):

nginx -t
-systemctl restart nginx
-

If everything goes correctly, you can now go to your website by typing domain.name on a web browser. But you will see a “404 Not Found” page like the following (maybe with different Nginx version):

That’s no problem, because it means that the web server it’s actually working. Just add an index.html file with something simple to see it in action (in the /var/www/some_folder that you decided upon). If you keep seeing the 404 page make sure your root line is correct and that the directory/index file exists.

I like to remove the .html and trailing / on the URLs of my website, for that you need to add the following rewrite lines and modify the try_files line (for more: Sean C. Davis: Remove HTML Extension And Trailing Slash In Nginx Config):

server {
-    ...
-    rewrite ^(/.*)\.html(\?.*)?$ $1$2 permanent;
-    rewrite ^/(.*)/$ /$1 permanent;
-    ...
-    try_files $uri/index.html $uri.html $uri/ $uri =404;
-    ...
-

Certbot¶

Certbot is what provides the SSL certificates via Let’s Encrypt.

The only “bad” (bloated) thing about Certbot, is that it uses python, but for me it doesn’t matter too much. You may want to look up another alternative if you prefer. Install the packages certbot and certbot-nginx:

pacman -S certbot certbot-nginx
-

After that, all you have to do now is run certbot and follow the instructions given by the tool:

certbot --nginx
-

It will ask you for some information, for you to accept some agreements and the names to activate HTTPS for. Also, you will want to “say yes” to the redirection from HTTP to HTTPS. And that’s it, you can now go to your website and see that you have HTTPS active.

Now, the certificate given by certbot expires every 3 months or something like that, so you want to renew this certificate every once in a while. I did this before using cron or manually creating a systemd timer and service, but now it’s just a matter of enabling the certbot-renew.timer:

systemctl start certbot-renew.timer
-

The deploy-hook is not needed anymore, only for plugins. For more, visit the Arch Linux Wiki.

- -

- - - - Next - - - - - - - Index - - - - - - - Previous - - -

- - -

By David Luévano

Created: Fri, Mar 19, 2021 @ 02:58 UTC

Modified: Wed, Jun 21, 2023 @ 00:18 UTC

Tags: -code, english, server, tools, tutorial

- -

Set up an XMPP server with Prosody compatible with Conversations and Movim

- -

Update: I no longer host this XMPP server as it consumed a lot of resources and I wasn’t using it that much. I’ll probably re-create it in the future, though.

Recently I set up an XMPP server (and a Matrix one, too) for my personal use and for friends if they want one; made one for EL ELE EME for example. So, here are the notes on how I set up the server that is compatible with the Conversations app and the Movim social network. You can see my addresses at contact and the XMPP compliance/score of the server.

One of the best resources I found that helped me a lot was Installing and Configuring Prosody XMPP Server on Debian 9, the Arch Wiki and the oficial documentation.

As with my other entries, this is under a server running Arch Linux, with the Nginx web server and Certbot certificates. And all commands here are executed as root, unless specified otherwise.

Table of contents¶

Table of contents
Prerequisites
Prosody
Nginx configuration file
Coturn
Wrapping up

Prerequisites¶

Same as with my other entries (website, mail and git) plus:

A and (optionally) AAA DNS records for:
- xmpp: the actual XMPP server and the file upload service.
- muc (or conference): for multi-user chats.
- pubsub: the publish-subscribe service.
- proxy: a proxy in case one of the users needs it.
- vjud: user directory.
-
(Optionally, but recommended) the following SRV DNS records; make sure it is pointing to an A or AAA record (matching the records from the last point, for example):
- _xmpp-client._tcp.{your.domain}. for port 5222 pointing to xmpp.{your.domain}.
- _xmpp-server._tcp.{your.domain}. for port 5269 pointing to xmpp.{your.domain}.
- _xmpp-server._tcp.muc.{your.domain}. for port 5269 pointing to xmpp.{your.domain}.
-
SSL certificates for the previous subdomains; similar that with my other entries just create the appropriate prosody.conf (where server_name will be all the subdomains defined above) file and run certbot --nginx. You can find the example configuration file almost at the end of this entry.
Email addresses for admin, abuse, contact, security, etc. Or use your own email for all of them, doesn’t really matter much as long as you define them in the configuration and are valid, I have aliases so those emails are forwarded to me.
Allow ports 5000, 5222, 5269, 5280 and 5281 for Prosody and, 3478 and 5349 for Turnserver which are the defaults for coturn.

Prosody¶

Prosody is an implementation of the XMPP protocol that is flexible and extensible.

Install the prosody package (with optional dependencies) and the mercurial package:

pacman -S prosody, mercurial, lua52-sec, lua52-dbi, lua52-zlib
-

We need mercurial to be able to download and update the extra modules needed to make the server compliant with conversations.im and mov.im. Go to /var/lib/prosody, clone the latest Prosody modules repository and prepare the directories:

cd /var/lib/prosody
-hg clone https://hg.prosody.im/prosody-modules modules-available
-mkdir modules-enabled
-

You can see that I follow a similar approach that I used with Nginx and the server configuration, where I have all the modules available in a directory, and make a symlink to another to keep track of what is being used. You can update the repository by running hg pull --update while inside the modules-available directory (similar to Git).

Make symbolic links to the following modules:

ln -s /var/lib/prosody/modules-available/{module_name} /var/lib/prosody/modules-enabled/
-...
-

Modules ({module_name}):
- mod_bookmarks
- mod_cache_c2s_caps
- mod_checkcerts
- mod_cloud_notify
- mod_csi_battery_saver
- mod_default_bookmarks
- mod_external_services
- mod_http_avatar
- mod_http_pep_avatar
- mod_http_upload
- mod_http_upload_external
- mod_idlecompat
- mod_muc_limits
- mod_muc_mam_hints
- mod_muc_mention_notifications
- mod_presence_cache
- mod_pubsub_feeds
- mod_pubsub_text_interface
- mod_smacks
- mod_strict_https
- mod_vcard_muc
- mod_vjud
- mod_watchuntrusted
-

And add other modules if needed, but these work for the apps that I mentioned. You should also change the permissions for these files:

chown -R prosody:prosody /var/lib/prosody
-

Now, configure the server by editing the /etc/prosody/prosody.cfg.lua file. It’s a bit tricky to configure, so here is my configuration file (lines starting with -- are comments). Make sure to change according to your domain, and maybe preferences. Read each line and each comment to know what’s going on, It’s easier to explain it with comments in the file itself than strip it in a lot of pieces.

And also, note that the configuration file has a “global” section and a per “virtual server”/”component” section, basically everything above all the VirtualServer/Component sections are global, and bellow each VirtualServer/Component, corresponds to that section.

-- important for systemd
-daemonize = true
-pidfile = "/run/prosody/prosody.pid"
-
--- or your account, not that this is an xmpp jid, not email
-admins = { "admin@your.domain" }
-
-contact_info = {
-    abuse = { "mailto:abuse@your.domain", "xmpp:abuse@your.domain" };
-    admin = { "mailto:admin@your.domain", "xmpp:admin@your.domain" };
-    admin = { "mailto:feedback@your.domain", "xmpp:feedback@your.domain" };
-    security = { "mailto:security@your.domain" };
-    support = { "mailto:support@your.domain", "xmpp:support@muc.your.domain" };
-}
-
--- so prosody look up the plugins we added
-plugin_paths = { "/var/lib/prosody/modules-enabled" }
-
-modules_enabled = {
-    -- Generally required
-        "roster"; -- Allow users to have a roster. Recommended ;)
-        "saslauth"; -- Authentication for clients and servers. Recommended if you want to log in.
-        "tls"; -- Add support for secure TLS on c2s/s2s connections
-        "dialback"; -- s2s dialback support
-        "disco"; -- Service discovery
-    -- Not essential, but recommended
-        "carbons"; -- Keep multiple clients in sync
-        "pep"; -- Enables users to publish their avatar, mood, activity, playing music and more
-        "private"; -- Private XML storage (for room bookmarks, etc.)
-        "blocklist"; -- Allow users to block communications with other users
-        "vcard4"; -- User profiles (stored in PEP)
-        "vcard_legacy"; -- Conversion between legacy vCard and PEP Avatar, vcard
-        "limits"; -- Enable bandwidth limiting for XMPP connections
-    -- Nice to have
-        "version"; -- Replies to server version requests
-        "uptime"; -- Report how long server has been running
-        "time"; -- Let others know the time here on this server
-        "ping"; -- Replies to XMPP pings with pongs
-        "register"; -- Allow users to register on this server using a client and change passwords
-        "mam"; -- Store messages in an archive and allow users to access it
-        "csi_simple"; -- Simple Mobile optimizations
-    -- Admin interfaces
-        "admin_adhoc"; -- Allows administration via an XMPP client that supports ad-hoc commands
-        --"admin_telnet"; -- Opens telnet console interface on localhost port 5582
-    -- HTTP modules
-        "http"; -- Explicitly enable http server.
-        "bosh"; -- Enable BOSH clients, aka "Jabber over HTTP"
-        "websocket"; -- XMPP over WebSockets
-        "http_files"; -- Serve static files from a directory over HTTP
-    -- Other specific functionality
-        "groups"; -- Shared roster support
-        "server_contact_info"; -- Publish contact information for this service
-        "announce"; -- Send announcement to all online users
-        "welcome"; -- Welcome users who register accounts
-        "watchregistrations"; -- Alert admins of registrations
-        "motd"; -- Send a message to users when they log in
-        --"legacyauth"; -- Legacy authentication. Only used by some old clients and bots.
-        --"s2s_bidi"; -- not yet implemented, have to wait for v0.12
-        "bookmarks";
-        "checkcerts";
-        "cloud_notify";
-        "csi_battery_saver";
-        "default_bookmarks";
-        "http_avatar";
-        "idlecompat";
-        "presence_cache";
-        "smacks";
-        "strict_https";
-        --"pep_vcard_avatar"; -- not compatible with this version of pep, wait for v0.12
-        "watchuntrusted";
-        "webpresence";
-        "external_services";
-    }
-
--- only if you want to disable some modules
-modules_disabled = {
-    -- "offline"; -- Store offline messages
-    -- "c2s"; -- Handle client connections
-    -- "s2s"; -- Handle server-to-server connections
-    -- "posix"; -- POSIX functionality, sends server to background, enables syslog, etc.
-}
-
-external_services = {
-    {
-        type = "stun",
-        transport = "udp",
-        host = "proxy.your.domain",
-        port = 3478
-    }, {
-        type = "turn",
-        transport = "udp",
-        host = "proxy.your.domain",
-        port = 3478,
-        -- you could decide this now or come back later when you install coturn
-        secret = "YOUR SUPER SECRET TURN PASSWORD"
-    }
-}
-
---- general global configuration
-http_ports = { 5280 }
-http_interfaces = { "*", "::" }
-
-https_ports = { 5281 }
-https_interfaces = { "*", "::" }
-
-proxy65_ports = { 5000 }
-proxy65_interfaces = { "*", "::" }
-
-http_default_host = "xmpp.your.domain"
-http_external_url = "https://xmpp.your.domain/"
--- or if you want to have it somewhere else, change this
-https_certificate = "/etc/prosody/certs/xmpp.your.domain.crt"
-
-hsts_header = "max-age=31556952"
-
-cross_domain_bosh = true
---consider_bosh_secure = true
-cross_domain_websocket = true
---consider_websocket_secure = true
-
-trusted_proxies = { "127.0.0.1", "::1", "192.169.1.1" }
-
-pep_max_items = 10000
-
--- this is disabled by default, and I keep it like this, depends on you
---allow_registration = true
-
--- you might want this options as they are
-c2s_require_encryption = true
-s2s_require_encryption = true
-s2s_secure_auth = false
---s2s_insecure_domains = { "insecure.example" }
---s2s_secure_domains = { "jabber.org" }
-
--- where the certificates are stored (/etc/prosody/certs by default)
-certificates = "certs"
-checkcerts_notify = 7 -- ( in days )
-
--- rate limits on connections to the server, these are my personal settings, because by default they were limited to something like 30kb/s
-limits = {
-    c2s = {
-        rate = "2000kb/s";
-    };
-    s2sin = {
-        rate = "5000kb/s";
-    };
-    s2sout = {
-        rate = "5000kb/s";
-    };
-}
-
--- again, this could be yourself, it is a jid
-unlimited_jids = { "admin@your.domain" }
-
-authentication = "internal_hashed"
-
--- if you don't want to use sql, change it to internal and comment the second line
--- since this is optional, i won't describe how to setup mysql or setup the user/database, that would be out of the scope for this entry
-storage = "sql"
-sql = { driver = "MySQL", database = "prosody", username = "prosody", password = "PROSODY USER SECRET PASSWORD", host = "localhost" }
-
-archive_expires_after = "4w" -- configure message archive
-max_archive_query_results = 20;
-mam_smart_enable = true
-default_archive_policy = "roster" -- archive only messages from users who are in your roster
-
--- normally you would like at least one log file of certain level, but I keep all of them, the default is only the info = "*syslog" one
-log = {
-    info = "*syslog";
-    warn = "prosody.warn";
-    error = "prosody.err";
-    debug = "prosody.debug";
-    -- "*console"; -- Needs daemonize=false
-}
-
--- cloud_notify
-push_notification_with_body = false -- Whether or not to send the message body to remote pubsub node
-push_notification_with_sender = false -- Whether or not to send the message sender to remote pubsub node
-push_max_errors = 5 -- persistent push errors are tolerated before notifications for the identifier in question are disabled
-push_max_devices = 5 -- number of allowed devices per user
-
--- by default every user on this server will join these muc rooms
-default_bookmarks = {
-    { jid = "room@muc.your.domain", name = "The Room" };
-    { jid = "support@muc.your.domain", name = "Support Room" };
-}
-
--- could be your jid
-untrusted_fail_watchers = { "admin@your.domain" }
-untrusted_fail_notification = "Establishing a secure connection from $from_host to $to_host failed. Certificate hash: $sha1. $errors"
-
------------ Virtual hosts -----------
-VirtualHost "your.domain"
-    name = "Prosody"
-    http_host = "xmpp.your.domain"
-
-disco_items = {
-    { "your.domain", "Prosody" };
-    { "muc.your.domain", "MUC Service" };
-    { "pubsub.your.domain", "Pubsub Service" };
-    { "proxy.your.domain", "SOCKS5 Bytestreams Service" };
-    { "vjud.your.domain", "User Directory" };
-}
-
-
--- Multi-user chat
-Component "muc.your.domain" "muc"
-    name = "MUC Service"
-    modules_enabled = {
-        --"bob"; -- not compatible with this version of Prosody
-        "muc_limits";
-        "muc_mam"; -- message archive in muc, again, a placeholder
-        "muc_mam_hints";
-        "muc_mention_notifications";
-        "vcard_muc";
-    }
-
-    restrict_room_creation = false
-
-    muc_log_by_default = true
-    muc_log_presences = false
-    log_all_rooms = false
-    muc_log_expires_after = "1w"
-    muc_log_cleanup_interval = 4 * 60 * 60
-
-
--- Upload
-Component "xmpp.your.domain" "http_upload"
-    name = "Upload Service"
-    http_host= "xmpp.your.domain"
-    -- you might want to change this, these are numbers in bytes, so 10MB and 100MB respectively
-    http_upload_file_size_limit = 1024*1024*10
-    http_upload_quota = 1024*1024*100
-
-
--- Pubsub
-Component "pubsub.your.domain" "pubsub"
-    name = "Pubsub Service"
-    pubsub_max_items = 10000
-    modules_enabled = {
-        "pubsub_feeds";
-        "pubsub_text_interface";
-    }
-
-    -- personally i don't have any feeds configured
-    feeds = {
-        -- The part before = is used as PubSub node
-        --planet_jabber = "http://planet.jabber.org/atom.xml";
-        --prosody_blog = "http://blog.prosody.im/feed/atom.xml";
-    }
-
-
--- Proxy
-Component "proxy.your.domain" "proxy65"
-    name = "SOCKS5 Bytestreams Service"
-    proxy65_address = "proxy.your.domain"
-
-
--- Vjud, user directory
-Component "vjud.your.domain" "vjud"
-    name = "User Directory"
-    vjud_mode = "opt-in"
-

You HAVE to read all of the configuration file, because there are a lot of things that you need to change to make it work with your server/domain. Test the configuration file with:

luac5.2 -p /etc/prosody/prosody.cfg.lua
-

Notice that by default prosody will look up certificates that look like sub.your.domain, but if you get the certificates like I do, you’ll have a single certificate for all subdomains, and by default it is in /etc/letsencrypt/live, which has some strict permissions. So, to import it you can run:

prosodyctl --root cert import /etc/letsencrypt/live
-

Ignore the complaining about not finding the subdomain certificates and note that you will have to run that command on each certificate renewal, to automate this, add the --deploy-hook flag to your automated Certbot renewal system; for me it’s a systemd timer with the following certbot.service:

[Unit]
-Description=Let's Encrypt renewal
-
-[Service]
-Type=oneshot
-ExecStart=/usr/bin/certbot renew --quiet --agree-tos --deploy-hook "systemctl reload nginx.service && prosodyctl --root cert import /etc/letsencrypt/live"
-

And if you don’t have it already, the certbot.timer:

[Unit]
-Description=Twice daily renewal of Let's Encrypt's certificates
-
-[Timer]
-OnCalendar=0/12:00:00
-RandomizedDelaySec=1h
-Persistent=true
-
-[Install]
-WantedBy=timers.target
-

Also, go to the certs directory and make the appropriate symbolic links:

cd /etc/prosody/certs
-ln -s your.domain.crt SUBDOMAIN.your.domain.crt
-ln -s your.domain.key SUBDOMAIN.your.domain.key
-...
-

That’s basically all the configuration that needs Prosody itself, but we still have to configure Nginx and Coturn before starting/enabling the prosody service.

Nginx configuration file¶

Since this is not an ordinary configuration file I’m going to describe this, too. Your prosody.conf file should have the following location blocks under the main server block (the one that listens to HTTPS):

# HTTPS server block
-server {
-    root /var/www/prosody/;
-    server_name xmpp.luevano.xyz muc.luevano.xyz pubsub.luevano.xyz vjud.luevano.xyz proxy.luevano.xyz;
-    index index.html;
-
-    # for extra https discovery (XEP-0256)
-    location /.well-known/acme-challenge {
-        allow all;
-    }
-
-    # bosh specific
-    location /http-bind {
-        proxy_pass  https://localhost:5281/http-bind;
-
-        proxy_set_header Host $host;
-        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-        proxy_set_header X-Forwarded-Proto $scheme;
-        proxy_buffering off;
-        tcp_nodelay on;
-    }
-
-    # websocket specific
-    location /xmpp-websocket {
-        proxy_pass https://localhost:5281/xmpp-websocket;
-
-        proxy_http_version 1.1;
-        proxy_set_header Connection "Upgrade";
-        proxy_set_header Upgrade $http_upgrade;
-
-        proxy_set_header Host $host;
-        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-        proxy_set_header X-Forwarded-Proto $scheme;
-        proxy_read_timeout 900s;
-    }
-
-    # general proxy
-    location / {
-        proxy_pass https://localhost:5281;
-
-        proxy_set_header Host $host;
-        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-        proxy_set_header X-Forwarded-Proto $scheme;
-        proxy_set_header X-Real-IP $remote_addr;
-    }
-    ...
-    # Certbot stuff
-}
-# HTTP server block (the one that certbot creates)
-server {
-    ...
-}
-

Also, you need to add the following to your actual your.domain (this cannot be a subdomain) configuration file:

server {
-    ...
-    location /.well-known/host-meta {
-        default_type 'application/xrd+xml';
-        add_header Access-Control-Allow-Origin '*' always;
-    }
-
-    location /.well-known/host-meta.json {
-        default_type 'application/jrd+json';
-        add_header Access-Control-Allow-Origin '*' always;
-    }
-    ...
-}
-

And you will need the following host-meta and host-meta.json files inside the .well-known/acme-challenge directory for your.domain (following my nomenclature: /var/www/yourdomaindir/.well-known/acme-challenge/).

For host-meta file:

<?xml version='1.0' encoding='utf-8'?>
-<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0'>
-    <Link rel="urn:xmpp:alt-connections:xbosh"
-        href="https://xmpp.your.domain:5281/http-bind" />
-    <Link rel="urn:xmpp:alt-connections:websocket"
-        href="wss://xmpp.your.domain:5281/xmpp-websocket" />
-</XRD>
-

And host-meta.json file:

{
-    "links": [
-        {
-            "rel": "urn:xmpp:alt-connections:xbosh",
-                "href": "https://xmpp.your.domain:5281/http-bind"
-        },
-        {
-            "rel": "urn:xmpp:alt-connections:websocket",
-                "href": "wss://xmpp.your.domain:5281/xmpp-websocket"
-        }
-    ]
-}
-

Remember to have your prosody.conf file symlinked (or discoverable by Nginx) to the sites-enabled directory. You can now test and restart your nginx service (and test the configuration, optionally):

nginx -t
-systemctl restart nginx.service
-

Coturn¶

Coturn is the implementation of TURN and STUN server, which in general is for (at least in the XMPP world) voice support and external service discovery.

Install the coturn package:

pacman -S coturn
-

You can modify the configuration file (located at /etc/turnserver/turnserver.conf) as desired, but at least you need to make the following changes (uncomment or edit):

use-auth-secret
-realm=proxy.your.domain
-static-auth-secret=YOUR SUPER SECRET TURN PASSWORD
-

I’m sure there is more configuration to be made, like using SQL to store data and whatnot, but for now this is enough for me. Note that you may not have some functionality that’s needed to create dynamic users to use the TURN server, and to be honest I haven’t tested this since I don’t use this feature in my XMPP clients, but if it doesn’t work, or you know of an error or missing configuration don’t hesitate to contact me.

Start/enable the turnserver service:

systemctl start turnserver.service
-systemctl enable turnserver.service
-

You can test if your TURN server works at Trickle ICE. You may need to add a user in the turnserver.conf to test this.

Wrapping up¶

At this point you should have a working XMPP server, start/enable the prosody service now:

systemctl start prosody.service
-systemctl enable prosody.service
-

And you can add your first user with the prosodyctl command (it will prompt you to add a password):

prosodyctl adduser user@your.domain
-

You may want to add a compliance user, so you can check if your server is set up correctly. To do so, go to XMPP Compliance Tester and enter the compliance user credentials. It should have similar compliance score to mine:

Additionally, you can test the security of your server in IM Observatory, here you only need to specify your domain.name (not xmpp.domain.name, if you set up the SRV DNS records correctly). Again, it should have a similar score to mine:

You can now log in into your XMPP client of choice, if it asks for the server it should be xmpp.your.domain (or your.domain for some clients) and your login credentials you@your.domain and the password you chose (which you can change in most clients).

That’s it, send me a message at david@luevano.xyz if you were able to set up the server successfully.

- -

- - - - Next - - - - - - - Index - - - - - - - Previous - - -

- - -

By David Luévano

Created: Wed, Jun 09, 2021 @ 05:24 UTC

Modified: Sat, Jun 03, 2023 @ 03:54 UTC

Tags: -code, english, server, tools, tutorial

- -

Al fin ya me acomodé la página pa' los dibujos

Al fin tengo fibra ópticona

Configure system logs on Arch to avoid filled up disk

Así nomás está quedando el página

Los devs de Android/MIUI me trozaron

Así es raza, el blog ya tiene timestamps

This is the first blog post, just for testing purposes

Set up a Git server and cgit front-end

Table of contents¶

Prerequisites¶

Git¶

Cgit¶

Cgit’s file rendering¶

Hoy me tocó desarrollo de personaje

Set up a media server with Jellyfin, Sonarr and Radarr

Table of contents¶

Prerequisites¶

Radarr¶

Reverse proxy¶

Start using Radarr¶

Configuration¶

Media Management¶

Quality¶

Custom Formats¶

Profiles¶

Download clients¶

Indexers¶

Download content¶

Sonarr¶

Reverse proxy¶

Start using Sonarr¶

Configuration¶

Media Management¶

Quality¶

Profiles¶

Download clients¶

Indexers¶

Download content¶

Jellyfin¶

Reverse proxy¶

SSL certificate¶

Start using Jellyfin¶

Plugins¶

Transcoding¶

NVIDIA drivers¶

Enable hardware acceleration¶

Bazarr¶

Reverse proxy¶

Start using Bazarr¶

Configuration¶

Providers¶

I had to learn Go and Lua the hard way

Set up a Mail server with Postfix, Dovecot, SpamAssassin and OpenDKIM

Table of contents¶

Prerequisites¶

Postfix¶

Dovecot¶

OpenDKIM¶

OpenDKIM DNS TXT records¶

SpamAssassin¶

Wrapping up¶

Set up a manga server with Komga and mangal

Table of contents¶

Prerequisites¶

yay¶

Install¶

Usage¶

mangal¶

Install from source¶

Configuration¶

Usage¶

Headless browser¶

TUI¶

Inline¶

Automation¶

Komga¶

Reverse proxy¶

SSL certificate¶

Start using Komga¶

Library creation¶