diff options
Diffstat (limited to 'live/blog')
| -rw-r--r-- | live/blog/a/manga_server_with_komga.html | 85 | ||||
| -rw-r--r-- | live/blog/rss.xml | 84 | ||||
| -rw-r--r-- | live/blog/sitemap.xml | 2 |
3 files changed, 118 insertions, 53 deletions
diff --git a/live/blog/a/manga_server_with_komga.html b/live/blog/a/manga_server_with_komga.html index e264374..182956d 100644 --- a/live/blog/a/manga_server_with_komga.html +++ b/live/blog/a/manga_server_with_komga.html @@ -159,7 +159,7 @@ sudo rm -r yay </code></pre> <p>This <code>komga</code> package creates a <code>komga</code> (service) user and group which is tied to the also included <code>komga.service</code>.</p> <p>Configure it by editing <code>/etc/komga.conf</code>:</p> -<pre><code class="language-conf">SERVER_PORT=8989 +<pre><code class="language-sh">SERVER_PORT=8989 SERVER_SERVLET_CONTEXT_PATH=/ # this depends a lot of how it's going to be served (domain, subdomain, ip, etc) KOMGA_LIBRARIES_SCAN_CRON="0 0 * * * ?" @@ -176,14 +176,14 @@ KOMGA_DATABASE_BACKUP_SCHEDULE="0 0 */8 * * ?" <p>My changes (shown above):</p> <ul> <li>Port on <code>8989</code> because <code>8080</code> its too generic.</li> -<li><code>cron</code> schedules<ul> +<li><code>cron</code> schedules.<ul> <li>It’s not actually <code>cron</code> but rather a <code>cron</code>-like syntax used by <a href="https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/scheduling/support/CronSequenceGenerator.html">Spring</a> as stated in the <a href="https://komga.org/installation/configuration.html#optional-configuration">Komga config</a>.</li> </ul> </li> <li>Added the remember me key.</li> <li>For more check out <a href="https://komga.org/installation/configuration.html">Komga: Configuration options</a>.</li> </ul> -<p>If you’re going to run it locally (or LAN/VPN) you can start the <code>komga.service</code> and access it via IP at <code>http://<your-server-ip>:<port>(/base_url)</code> as stated at <a href="https://komga.org/installation/webui.html">Komga: Accessing the web interface</a>, else continue with the next steps for the reverse proxy and certificate.</p> +<p>If you’re going to run it locally (or LAN/VPN) you can start the <code>komga.service</code> and access it via IP at <code>http://<your-server-ip>:<port>(/base_url)</code> as stated at <a href="https://komga.org/installation/webui.html">Komga: Accessing the web interface</a>, then you can continue with the <a href="#mangal">mangal</a> section, else continue with the next steps for the reverse proxy and certificate.</p> <h3 id="reverse-proxy">Reverse proxy<a class="headerlink" href="#reverse-proxy" title="Permanent link">¶</a></h3> <p>Create the reverse proxy configuration (this is for <code>nginx</code>). In my case I’ll use a subdomain, so this is a new config called <code>komga.conf</code> at the usual <code>sites-available/enabled</code> path:</p> <pre><code class="language-nginx">server { @@ -203,7 +203,7 @@ KOMGA_DATABASE_BACKUP_SCHEDULE="0 0 */8 * * ?" } } </code></pre> -<p>If it’s going to be used as a subdir on another domain then just change the <code>location</code> (with <code>/subdir</code> instead of <code>/</code>) directive to the corresponding <code>.conf</code> file; be careful with the <code>proxy_pass</code> directive, it has to match what you configured at <code>/etc/komga.conf</code> for the <code>SERVER_SERVLET_CONTEXT_PATH</code> regardless of the <code>/subdir</code> you selected at <code>location</code>.</p> +<p>If it’s going to be used as a subdir on another domain then just change the <code>location</code> with <code>/subdir</code> instead of <code>/</code>; be careful with the <code>proxy_pass</code> directive, it has to match what you configured at <code>/etc/komga.conf</code> for the <code>SERVER_SERVLET_CONTEXT_PATH</code> regardless of the <code>/subdir</code> you selected at <code>location</code>.</p> <h3 id="ssl-certificate">SSL certificate<a class="headerlink" href="#ssl-certificate" title="Permanent link">¶</a></h3> <p>If using a subdir then the same certificate for the subdomain/domain should work fine and no extra stuff is needed, else if following along me then we can create/extend the certificate by running:</p> <pre><code class="language-sh">certbot --nginx @@ -259,12 +259,12 @@ default:other::r-x <p>So instad of installing with <code>yay</code> we’ll build it from source. We need to have <code>go</code> installed:</p> <pre><code class="language-sh">pacman -S go </code></pre> -<p>Then clone my fork of <code>mangal</code> and build/install it:</p> +<p>Then clone my fork of <code>mangal</code> and <code>build</code>/<code>install</code> it:</p> <pre><code class="language-sh">git clone https://github.com/luevano/mangal.git # not sure if you can use SSH to clone cd mangal make install # or just `make build` and then move the binary to somewhere in your $PATH </code></pre> -<p>This will use <code>go install</code> so it will install to a path specified by your environment variables, for more run <code>go help install</code>. It was installed to <code>$HOME/.local/bin/go/mangal</code> for me, then just make sure this is included in your PATH.</p> +<p>This will use <code>go install</code> so it will install to a path specified by the <code>go</code> environment variables, for more run <code>go help install</code>. It was installed to <code>$HOME/.local/bin/go/mangal</code> for me because my env vars, then just make sure this is included in your <code>PATH</code>.</p> <p>Check it was correctly installed by running <code>mangal version</code>, which should print something like:</p> <pre><code>▇▇▇ mangal @@ -292,11 +292,11 @@ mangal config set -k logs.write -v true # I like to get logs for what happens <h3 id="usage_1">Usage<a class="headerlink" href="#usage_1" title="Permanent link">¶</a></h3> <p>Two main ways of using <code>mangal</code>: </p> <ul> -<li>TUI: for initial browsing/downloading and testing things out. If the manga finished publishing, this should be enough.</li> -<li>inline: for automation on manga that is still publishing and I need to check/download every once in a while.</li> +<li><strong>TUI</strong>: for initial browsing/downloading and testing things out. If the manga finished publishing, this should be enough.</li> +<li><strong>inline</strong>: for automation on manga that is still publishing and I need to check/download every once in a while.</li> </ul> <h4 id="headless-browser">Headless browser<a class="headerlink" href="#headless-browser" title="Permanent link">¶</a></h4> -<p>Before continuing, I gotta say I went through some bullshit while trying to use the custom Lua scrapers that use the <em>headless</em> browser (actually just a wrapper of <a href="https://github.com/go-rod/rod">go-rod/rod</a>, and honestly it is not really a “headless” browser, <code>mangal</code> “documentation” is just wrong). For mor on my rant check out my last <a href="https://blog.luevano.xyz/a/learned_go_and_lua_hard_way.html">entry</a>.</p> +<p>Before continuing, I gotta say I went through some bullshit while trying to use the custom Lua scrapers that use the <em>headless</em> browser (actually just a wrapper of <a href="https://github.com/go-rod/rod">go-rod/rod</a>, and honestly it is not really a “headless” browser, <code>mangal</code> “documentation” is just wrong). For more on my rant check out my last <a href="https://blog.luevano.xyz/a/learned_go_and_lua_hard_way.html">entry</a>.</p> <p>There is no concrete documentation on the “headless” browser, only that it is automatically set up and ready to use… but it doesn’t install any library/dependency needed. I discovered the following libraries that were missing on my Arch minimal install:</p> <ul> <li>library -> arch package containing it</li> @@ -322,18 +322,30 @@ mangal config set -k logs.write -v true # I like to get logs for what happens <pre><code class="language-sh">mangal </code></pre> <p>Download manga using the TUI by selecting the source/scrapper, search the manga/comic you want and then you can select each chapter to download (use <code>tab</code> to select all). This is what I use when downloading manga that already finished publishing, or when I’m just searching and testing out how it downloads the manga (directory name, and manga information).</p> -<p>Note that some scrapters will contain duplicated chapters, as they have uploaded chapters from the community. This happens a lot with <a href="https://mangadex.org/">MangaDex</a>.</p> +<p>Note that some scrapters will contain duplicated chapters, as they have multiple uploaded chapters from the community, usually for different <em>scanlation groups</em>. This happens a lot with <a href="https://mangadex.org/">MangaDex</a>.</p> <h4 id="inline">Inline<a class="headerlink" href="#inline" title="Permanent link">¶</a></h4> <p>The inline mode is a single terminal command meant to be used to automate stuff or for more advanced options. You can peek a bit into the “<a href="https://github.com/metafates/mangal/wiki/Inline-mode#command-examples">documentation</a>” which honestly its ass because it doesn’t explain much. The minimal command for inline according to the help is:</p> <pre><code class="language-sh">mangal inline --manga <option> --query <manga-title> </code></pre> -<p>But this will not produce anything because it also needs <code>--source</code> (or set the default using the config key <code>downloader.default_sources</code>) and either <code>--json</code> (for the search result) or <code>--download</code> to actually download whatever was found but it could download something you don’t want so do the <code>--json</code> first.</p> +<p>But this will not produce anything because it also needs <code>--source</code> (or set the default using the config key <code>downloader.default_sources</code>) and either <code>--json</code> which basically just does the search and returns the result in <code>json</code> format or <code>--download</code> to actually download whatever is found; I recommend to do <code>--json</code> first to check that the correct manga will be downloaded then do <code>--download</code>.</p> <p>Something not mentioned anywhere is the <code>--manga</code> flag options (found it at the source code), it has 3 available options:</p> <ul> <li><code>first</code>: first manga entry found for the search.</li> <li><code>last</code>: last manga entry found for the search.</li> <li><code>exact</code>: exact manga title match. This is the one I use.</li> </ul> +<p>Similar to <code>--chapters</code>, there are a few options not explained (that I found at the source code, too). I usually just use <code>all</code> but other options:</p> +<ul> +<li><code>all</code>: all chapters found in the chapter list.</li> +<li><code>first</code>: first chapter found in the chapter list.</li> +<li><code>last</code>: last chapter found in the chapter list</li> +<li><code>[from]-[to]</code>: selector for the chapters found in the chapter list, index starts at 0.<ul> +<li>If the selectors (<code>from</code> or <code>to</code>) exceed the amount of chapters in the chapterlist it just adjusts to he maximum available.</li> +<li>I had to fix this at the source code because if you wanted <code>to</code> to be the last chapter, it did <code>to + 1</code> and it failed due to index out of range.</li> +</ul> +</li> +<li><code>@[sub]@</code>: not sure how this works exactly, my understanding is that it’s for “named” chapters.</li> +</ul> <p>That said, I’ll do an example by using <a href="https://mangapill.com">Mangapill</a> as source, and will search for <a href="https://mangapill.com/manga/2285/kimetsu-no-yaiba">Demon Slayer: Kimetsu no Yaiba</a>:</p> <ol> <li>Search first and make sure my command will pull the manga I want:</li> @@ -341,16 +353,16 @@ mangal config set -k logs.write -v true # I like to get logs for what happens <pre><code class="language-sh">mangal inline --source "Mangapill" --manga "exact" --query "Kimetsu no Yaiba" --json | jq # I use jq to pretty format the output </code></pre> <ol start="2"> -<li> -<p>I make sure the json output contains the correct manga information: name, url, etc..</p> +<li>I make sure the json output contains the correct manga information: name, url, etc..</li> +</ol> <ul> <li>You can also include the flag <code>--include-anilist-manga</code> to include anilist information (if any) so you can check that the correct anilist id is attached. If the correct one is not attached (and it exists) then you can run the command:</li> </ul> -<p><code>sh -mangal inline anilist set --name "Kimetsu no Yaiba" --id 101922</code></p> -<p>Which means that all “searches” for that <code>--name</code> flag will be attached to that specific anilist ID. -3. If I’m okay with the outputs, then I change <code>--json</code> for <code>--download</code> to actually download:</p> -</li> +<pre><code class="language-sh">mangal inline anilist set --name "Kimetsu no Yaiba" --id 101922 +</code></pre> +<p>Which means that all “searches” for that <code>--name</code> flag will be attached to that specific anilist ID.</p> +<ol start="3"> +<li>If I’m okay with the outputs, then I change <code>--json</code> for <code>--download</code> to actually download:</li> </ol> <pre><code class="language-sh">mangal inline --source "Mangapill" --manga "exact" --query "Kimetsu no Yaiba" --download </code></pre> @@ -358,26 +370,26 @@ mangal inline anilist set --name "Kimetsu no Yaiba" --id 101922</code></p> <li>Check if the manga is downloaded correctly. I do this by going to my download directory and checking the directory name (I’m picky with this stuff), that all chapters where downloaded, that it includes a correct <code>series.json</code> file and it contains a <code>cover.<img-ext></code>; this usually means it correctly pulled information from anilist and that it will contain metadata Komga will be able to use.</li> </ol> <h4 id="komga-library">Komga library<a class="headerlink" href="#komga-library" title="Permanent link">¶</a></h4> -<p>Now I just check that it is correctly added to Komga by clicking on the 3 dots to the right of the library name and click on “Scan library files” to refresh if the cron timer hasn’t pass by yet.</p> +<p>Now I just check that it is correctly added to Komga by clicking on the 3 dots to the right of the library name and click on “Scan library files” to refresh if the cron timer hasn’t activated this yet.</p> <p>Then I check that the metadata is correct (once the manga is fully indexed), such as title, summary, chapter count, language, tags, genre, etc., which honestly it never works fine as <code>mangal</code> creates the <code>series.json</code> with the <code>comicId</code> field with an upper case <code>I</code> and Komga expects it to be a lower case <code>i</code> (<code>comicid</code>) so it falls back to using the info from the first chapter. I’ll probably will fix this on <code>mangal</code> side, and see how it goes.</p> <p>So, what I do is manually edit the metadata for the manga, by changing whatever it’s wrong or add what’s missing (I like adding anilist and MyAnimeList links) and then leave it as is.</p> <h4 id="automation">Automation<a class="headerlink" href="#automation" title="Permanent link">¶</a></h4> -<p>The straight forward approach for automation is just to bundle a bunch of <code>mangal inline</code> commands in a shell script and automate either via <a href="https://wiki.archlinux.org/title/cron">cron</a> or <a href="https://wiki.archlinux.org/title/systemd/Timers">systemd/Timers</a>. But, as always, I overcomplicated/overengineered my approach, which is the following:</p> +<p>The straight forward approach for automation is just to bundle a bunch of <code>mangal inline</code> commands in a shell script and schedule it’s execution either via <a href="https://wiki.archlinux.org/title/cron">cron</a> or <a href="https://wiki.archlinux.org/title/systemd/Timers">systemd/Timers</a>. But, as always, I overcomplicated/overengineered my approach, which is the following:</p> <ol> <li>Group manga names per source.</li> +<li>Configure anything that should always be set before executing <code>mangal</code>, this includes anilist bindings.</li> <li>Have a way to track the changes/updates on each run.</li> <li>Use that tracker to know where to start downloading chapters from.<ul> -<li>This is optional, as you can just do <code>--chapters "all"</code> and it will work. This is mostly to keep the logs/output cleaner/shorter.</li> +<li>This is optional, as you can just do <code>--chapters "all"</code> and it will work but I do it mostly to keep the logs/output cleaner/shorter.</li> </ul> </li> -<li>Do any configuration needed beforehand.</li> <li>Download/update each manga using <code>mangal inline</code>.</li> -<li>Wrap everything in a <code>systemd</code> service and timer.</li> +<li>Wrap everything in a <code>systemd</code> <code>service</code> and <code>timer</code>.</li> </ol> <p>Manga list example:</p> <pre><code class="language-sh">mangapill="Berserk|Chainsaw Man|Dandadan|Jujutsu Kaisen|etc..." </code></pre> -<p>Bash function that handles the download per manga in the list:</p> +<p>Function that handles the download per manga in the list:</p> <pre><code class="language-sh">mangal_src_dl () { source_name=$1 manga_list=$(echo "$2" | tr '|' '\n') @@ -413,6 +425,22 @@ mangal inline anilist set --name "Kimetsu no Yaiba" --id 101922</code></p> } </code></pre> <p>Where <code>$TRACKER_FILE</code> is just a variable holding a path to some file where you can store the tracking and <code>$DOWNLOAD_FORMAT</code> the format for the mangas, for me it’s <code>cbz</code>. Then the usage would be something like <code>mangal_src_dl "Mangapill" "$mangapill"</code>, meaning that it is a function call per source.</p> +<p>A simpler function without “tracking” would be:</p> +<pre><code class="language-sh">mangal_src_dl () { + source_name=$1 + manga_list=$(echo "$2" | tr '|' '\n') + + while IFS= read -r line; do + echo "Downloading all chapters for $line from $source_name..." + mangal inline -S "$source_name" -q "$line" -m "exact" -F "$DOWNLOAD_FORMAT" -c "all" -d + if [ $? -ne 0 ]; then + echo "Failed to download chapters for $line." + continue + fi + echo "Finished downloading chapters for $line." + done <<< "$manga_list" +} +</code></pre> <p>The tracker file would have a format like follows:</p> <pre><code># Updated: 06/10/23 10:53:15 AM CST Berserk|0392|392|Mangapill @@ -420,8 +448,12 @@ Dandadan|0110|110|Mangapill ... </code></pre> <p>And note that if you already had manga downloaded and you run the script for the first time, then it will show as if it downloaded everything from the first chapter, but that’s just how <code>mangal</code> works, it will actually just discover downloaded chapters and only download anything missing.</p> -<p>Any configuration the downloader/updater might need needs to be done before the <code>mangal_src_dl</code> calls. I like to configure mangal for download path, format, etc.. To clear the <code>mangal</code> cache and <code>rod</code> browser (headless browser used in some custom sources) as well as set up any anilist bindings. An example of an anilist binding I had to do is for Mushoku Tensei, as it has both a light novel and manga version, both having different information, for me it was <code>mangal inline anilist set --name "Mushoku Tensei - Isekai Ittara Honki Dasu" --id 85564</code>.</p> -<p>Finally is just a matter of using your prefered way of scheduling, I’ll use <code>systemd/Timers</code> but anything is fine. You could make the downloader script more sophisticated and only running every week on which each manga gets released usually, but that’s too much work, so I’ll just run it once daily probably, or 2-3 times daily.</p> +<p>Any configuration the downloader/updater might need needs to be done before the <code>mangal_src_dl</code> calls. I like to configure mangal for download path, format, etc.. I found that it is needed to clear the <code>mangal</code> and <code>rod</code> browser cache (headless browser used in some custom sources) from personal experience and from others: <a href="https://github.com/metafates/mangal/issues/170">mangal#170</a> and <a href="https://github.com/oae/kaizoku/issues/89">kaizoku#89</a>.</p> +<p>Also you should set any anilist binding necessary for the downloading (as the cache was cleared). An example of an anilist binding I had to do is for Mushoku Tensei, as it has both a <a href="https://anilist.co/manga/85470/Mushoku-Tensei-Jobless-Reincarnation/">light novel</a> and <a href="https://anilist.co/manga/85564/Mushoku-Tensei-Jobless-Reincarnation/">manga</a> version, which for me it’s the following binding:</p> +<pre><code class="language-sh">mangal inline anilist set --name "Mushoku Tensei - Isekai Ittara Honki Dasu" --id 85564 +</code></pre> +<p>Finally is just a matter of using your prefered way of scheduling, I’ll use <code>systemd/Timers</code> but anything is fine. You could make the downloader script more sophisticated and only running every week on which each manga gets (usually) released but that’s too much work; I’ll just run it once daily probably.</p> +<p>A feature I want to add and probably will is sending notifications (probably through email) on a summary for manga downloaded or failed to download so I’m on top of the updates. For now this is good enough and it’s been working so far.</p> <h3 id="alternative-downloaders">Alternative downloaders<a class="headerlink" href="#alternative-downloaders" title="Permanent link">¶</a></h3> <p>Just for the record, here is a list of downloaders/scrapers I considered before starting to use <code>mangal</code>:</p> <ul> @@ -457,6 +489,7 @@ Dandadan|0110|110|Mangapill <div class="article-info"> <p>By David Luévano</p> <p>Created: Sat, Jun 10, 2023 @ 19:36 UTC</p> + <p>Modified: Sun, Jun 11, 2023 @ 10:33 UTC</p> <div class="article-tags"> <p>Tags: <a href="https://blog.luevano.xyz/tag/@code.html">code</a>, <a href="https://blog.luevano.xyz/tag/@english.html">english</a>, <a href="https://blog.luevano.xyz/tag/@server.html">server</a>, <a href="https://blog.luevano.xyz/tag/@tools.html">tools</a>, <a href="https://blog.luevano.xyz/tag/@tutorial.html">tutorial</a> </p> diff --git a/live/blog/rss.xml b/live/blog/rss.xml index 7997277..303b255 100644 --- a/live/blog/rss.xml +++ b/live/blog/rss.xml @@ -105,7 +105,7 @@ sudo rm -r yay </code></pre> <p>This <code>komga</code> package creates a <code>komga</code> (service) user and group which is tied to the also included <code>komga.service</code>.</p> <p>Configure it by editing <code>/etc/komga.conf</code>:</p> -<pre><code class="language-conf">SERVER_PORT=8989 +<pre><code class="language-sh">SERVER_PORT=8989 SERVER_SERVLET_CONTEXT_PATH=/ # this depends a lot of how it's going to be served (domain, subdomain, ip, etc) KOMGA_LIBRARIES_SCAN_CRON="0 0 * * * ?" @@ -122,14 +122,14 @@ KOMGA_DATABASE_BACKUP_SCHEDULE="0 0 */8 * * ?" <p>My changes (shown above):</p> <ul> <li>Port on <code>8989</code> because <code>8080</code> its too generic.</li> -<li><code>cron</code> schedules<ul> +<li><code>cron</code> schedules.<ul> <li>It’s not actually <code>cron</code> but rather a <code>cron</code>-like syntax used by <a href="https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/scheduling/support/CronSequenceGenerator.html">Spring</a> as stated in the <a href="https://komga.org/installation/configuration.html#optional-configuration">Komga config</a>.</li> </ul> </li> <li>Added the remember me key.</li> <li>For more check out <a href="https://komga.org/installation/configuration.html">Komga: Configuration options</a>.</li> </ul> -<p>If you’re going to run it locally (or LAN/VPN) you can start the <code>komga.service</code> and access it via IP at <code>http://<your-server-ip>:<port>(/base_url)</code> as stated at <a href="https://komga.org/installation/webui.html">Komga: Accessing the web interface</a>, else continue with the next steps for the reverse proxy and certificate.</p> +<p>If you’re going to run it locally (or LAN/VPN) you can start the <code>komga.service</code> and access it via IP at <code>http://<your-server-ip>:<port>(/base_url)</code> as stated at <a href="https://komga.org/installation/webui.html">Komga: Accessing the web interface</a>, then you can continue with the <a href="#mangal">mangal</a> section, else continue with the next steps for the reverse proxy and certificate.</p> <h3 id="reverse-proxy">Reverse proxy<a class="headerlink" href="#reverse-proxy" title="Permanent link">¶</a></h3> <p>Create the reverse proxy configuration (this is for <code>nginx</code>). In my case I’ll use a subdomain, so this is a new config called <code>komga.conf</code> at the usual <code>sites-available/enabled</code> path:</p> <pre><code class="language-nginx">server { @@ -149,7 +149,7 @@ KOMGA_DATABASE_BACKUP_SCHEDULE="0 0 */8 * * ?" } } </code></pre> -<p>If it’s going to be used as a subdir on another domain then just change the <code>location</code> (with <code>/subdir</code> instead of <code>/</code>) directive to the corresponding <code>.conf</code> file; be careful with the <code>proxy_pass</code> directive, it has to match what you configured at <code>/etc/komga.conf</code> for the <code>SERVER_SERVLET_CONTEXT_PATH</code> regardless of the <code>/subdir</code> you selected at <code>location</code>.</p> +<p>If it’s going to be used as a subdir on another domain then just change the <code>location</code> with <code>/subdir</code> instead of <code>/</code>; be careful with the <code>proxy_pass</code> directive, it has to match what you configured at <code>/etc/komga.conf</code> for the <code>SERVER_SERVLET_CONTEXT_PATH</code> regardless of the <code>/subdir</code> you selected at <code>location</code>.</p> <h3 id="ssl-certificate">SSL certificate<a class="headerlink" href="#ssl-certificate" title="Permanent link">¶</a></h3> <p>If using a subdir then the same certificate for the subdomain/domain should work fine and no extra stuff is needed, else if following along me then we can create/extend the certificate by running:</p> <pre><code class="language-sh">certbot --nginx @@ -205,12 +205,12 @@ default:other::r-x <p>So instad of installing with <code>yay</code> we’ll build it from source. We need to have <code>go</code> installed:</p> <pre><code class="language-sh">pacman -S go </code></pre> -<p>Then clone my fork of <code>mangal</code> and build/install it:</p> +<p>Then clone my fork of <code>mangal</code> and <code>build</code>/<code>install</code> it:</p> <pre><code class="language-sh">git clone https://github.com/luevano/mangal.git # not sure if you can use SSH to clone cd mangal make install # or just `make build` and then move the binary to somewhere in your $PATH </code></pre> -<p>This will use <code>go install</code> so it will install to a path specified by your environment variables, for more run <code>go help install</code>. It was installed to <code>$HOME/.local/bin/go/mangal</code> for me, then just make sure this is included in your PATH.</p> +<p>This will use <code>go install</code> so it will install to a path specified by the <code>go</code> environment variables, for more run <code>go help install</code>. It was installed to <code>$HOME/.local/bin/go/mangal</code> for me because my env vars, then just make sure this is included in your <code>PATH</code>.</p> <p>Check it was correctly installed by running <code>mangal version</code>, which should print something like:</p> <pre><code>▇▇▇ mangal @@ -238,11 +238,11 @@ mangal config set -k logs.write -v true # I like to get logs for what happens <h3 id="usage_1">Usage<a class="headerlink" href="#usage_1" title="Permanent link">¶</a></h3> <p>Two main ways of using <code>mangal</code>: </p> <ul> -<li>TUI: for initial browsing/downloading and testing things out. If the manga finished publishing, this should be enough.</li> -<li>inline: for automation on manga that is still publishing and I need to check/download every once in a while.</li> +<li><strong>TUI</strong>: for initial browsing/downloading and testing things out. If the manga finished publishing, this should be enough.</li> +<li><strong>inline</strong>: for automation on manga that is still publishing and I need to check/download every once in a while.</li> </ul> <h4 id="headless-browser">Headless browser<a class="headerlink" href="#headless-browser" title="Permanent link">¶</a></h4> -<p>Before continuing, I gotta say I went through some bullshit while trying to use the custom Lua scrapers that use the <em>headless</em> browser (actually just a wrapper of <a href="https://github.com/go-rod/rod">go-rod/rod</a>, and honestly it is not really a “headless” browser, <code>mangal</code> “documentation” is just wrong). For mor on my rant check out my last <a href="https://blog.luevano.xyz/a/learned_go_and_lua_hard_way.html">entry</a>.</p> +<p>Before continuing, I gotta say I went through some bullshit while trying to use the custom Lua scrapers that use the <em>headless</em> browser (actually just a wrapper of <a href="https://github.com/go-rod/rod">go-rod/rod</a>, and honestly it is not really a “headless” browser, <code>mangal</code> “documentation” is just wrong). For more on my rant check out my last <a href="https://blog.luevano.xyz/a/learned_go_and_lua_hard_way.html">entry</a>.</p> <p>There is no concrete documentation on the “headless” browser, only that it is automatically set up and ready to use… but it doesn’t install any library/dependency needed. I discovered the following libraries that were missing on my Arch minimal install:</p> <ul> <li>library -> arch package containing it</li> @@ -268,18 +268,30 @@ mangal config set -k logs.write -v true # I like to get logs for what happens <pre><code class="language-sh">mangal </code></pre> <p>Download manga using the TUI by selecting the source/scrapper, search the manga/comic you want and then you can select each chapter to download (use <code>tab</code> to select all). This is what I use when downloading manga that already finished publishing, or when I’m just searching and testing out how it downloads the manga (directory name, and manga information).</p> -<p>Note that some scrapters will contain duplicated chapters, as they have uploaded chapters from the community. This happens a lot with <a href="https://mangadex.org/">MangaDex</a>.</p> +<p>Note that some scrapters will contain duplicated chapters, as they have multiple uploaded chapters from the community, usually for different <em>scanlation groups</em>. This happens a lot with <a href="https://mangadex.org/">MangaDex</a>.</p> <h4 id="inline">Inline<a class="headerlink" href="#inline" title="Permanent link">¶</a></h4> <p>The inline mode is a single terminal command meant to be used to automate stuff or for more advanced options. You can peek a bit into the “<a href="https://github.com/metafates/mangal/wiki/Inline-mode#command-examples">documentation</a>” which honestly its ass because it doesn’t explain much. The minimal command for inline according to the help is:</p> <pre><code class="language-sh">mangal inline --manga <option> --query <manga-title> </code></pre> -<p>But this will not produce anything because it also needs <code>--source</code> (or set the default using the config key <code>downloader.default_sources</code>) and either <code>--json</code> (for the search result) or <code>--download</code> to actually download whatever was found but it could download something you don’t want so do the <code>--json</code> first.</p> +<p>But this will not produce anything because it also needs <code>--source</code> (or set the default using the config key <code>downloader.default_sources</code>) and either <code>--json</code> which basically just does the search and returns the result in <code>json</code> format or <code>--download</code> to actually download whatever is found; I recommend to do <code>--json</code> first to check that the correct manga will be downloaded then do <code>--download</code>.</p> <p>Something not mentioned anywhere is the <code>--manga</code> flag options (found it at the source code), it has 3 available options:</p> <ul> <li><code>first</code>: first manga entry found for the search.</li> <li><code>last</code>: last manga entry found for the search.</li> <li><code>exact</code>: exact manga title match. This is the one I use.</li> </ul> +<p>Similar to <code>--chapters</code>, there are a few options not explained (that I found at the source code, too). I usually just use <code>all</code> but other options:</p> +<ul> +<li><code>all</code>: all chapters found in the chapter list.</li> +<li><code>first</code>: first chapter found in the chapter list.</li> +<li><code>last</code>: last chapter found in the chapter list</li> +<li><code>[from]-[to]</code>: selector for the chapters found in the chapter list, index starts at 0.<ul> +<li>If the selectors (<code>from</code> or <code>to</code>) exceed the amount of chapters in the chapterlist it just adjusts to he maximum available.</li> +<li>I had to fix this at the source code because if you wanted <code>to</code> to be the last chapter, it did <code>to + 1</code> and it failed due to index out of range.</li> +</ul> +</li> +<li><code>@[sub]@</code>: not sure how this works exactly, my understanding is that it’s for “named” chapters.</li> +</ul> <p>That said, I’ll do an example by using <a href="https://mangapill.com">Mangapill</a> as source, and will search for <a href="https://mangapill.com/manga/2285/kimetsu-no-yaiba">Demon Slayer: Kimetsu no Yaiba</a>:</p> <ol> <li>Search first and make sure my command will pull the manga I want:</li> @@ -287,16 +299,16 @@ mangal config set -k logs.write -v true # I like to get logs for what happens <pre><code class="language-sh">mangal inline --source "Mangapill" --manga "exact" --query "Kimetsu no Yaiba" --json | jq # I use jq to pretty format the output </code></pre> <ol start="2"> -<li> -<p>I make sure the json output contains the correct manga information: name, url, etc..</p> +<li>I make sure the json output contains the correct manga information: name, url, etc..</li> +</ol> <ul> <li>You can also include the flag <code>--include-anilist-manga</code> to include anilist information (if any) so you can check that the correct anilist id is attached. If the correct one is not attached (and it exists) then you can run the command:</li> </ul> -<p><code>sh -mangal inline anilist set --name "Kimetsu no Yaiba" --id 101922</code></p> -<p>Which means that all “searches” for that <code>--name</code> flag will be attached to that specific anilist ID. -3. If I’m okay with the outputs, then I change <code>--json</code> for <code>--download</code> to actually download:</p> -</li> +<pre><code class="language-sh">mangal inline anilist set --name "Kimetsu no Yaiba" --id 101922 +</code></pre> +<p>Which means that all “searches” for that <code>--name</code> flag will be attached to that specific anilist ID.</p> +<ol start="3"> +<li>If I’m okay with the outputs, then I change <code>--json</code> for <code>--download</code> to actually download:</li> </ol> <pre><code class="language-sh">mangal inline --source "Mangapill" --manga "exact" --query "Kimetsu no Yaiba" --download </code></pre> @@ -304,26 +316,26 @@ mangal inline anilist set --name "Kimetsu no Yaiba" --id 101922</code></p> <li>Check if the manga is downloaded correctly. I do this by going to my download directory and checking the directory name (I’m picky with this stuff), that all chapters where downloaded, that it includes a correct <code>series.json</code> file and it contains a <code>cover.<img-ext></code>; this usually means it correctly pulled information from anilist and that it will contain metadata Komga will be able to use.</li> </ol> <h4 id="komga-library">Komga library<a class="headerlink" href="#komga-library" title="Permanent link">¶</a></h4> -<p>Now I just check that it is correctly added to Komga by clicking on the 3 dots to the right of the library name and click on “Scan library files” to refresh if the cron timer hasn’t pass by yet.</p> +<p>Now I just check that it is correctly added to Komga by clicking on the 3 dots to the right of the library name and click on “Scan library files” to refresh if the cron timer hasn’t activated this yet.</p> <p>Then I check that the metadata is correct (once the manga is fully indexed), such as title, summary, chapter count, language, tags, genre, etc., which honestly it never works fine as <code>mangal</code> creates the <code>series.json</code> with the <code>comicId</code> field with an upper case <code>I</code> and Komga expects it to be a lower case <code>i</code> (<code>comicid</code>) so it falls back to using the info from the first chapter. I’ll probably will fix this on <code>mangal</code> side, and see how it goes.</p> <p>So, what I do is manually edit the metadata for the manga, by changing whatever it’s wrong or add what’s missing (I like adding anilist and MyAnimeList links) and then leave it as is.</p> <h4 id="automation">Automation<a class="headerlink" href="#automation" title="Permanent link">¶</a></h4> -<p>The straight forward approach for automation is just to bundle a bunch of <code>mangal inline</code> commands in a shell script and automate either via <a href="https://wiki.archlinux.org/title/cron">cron</a> or <a href="https://wiki.archlinux.org/title/systemd/Timers">systemd/Timers</a>. But, as always, I overcomplicated/overengineered my approach, which is the following:</p> +<p>The straight forward approach for automation is just to bundle a bunch of <code>mangal inline</code> commands in a shell script and schedule it’s execution either via <a href="https://wiki.archlinux.org/title/cron">cron</a> or <a href="https://wiki.archlinux.org/title/systemd/Timers">systemd/Timers</a>. But, as always, I overcomplicated/overengineered my approach, which is the following:</p> <ol> <li>Group manga names per source.</li> +<li>Configure anything that should always be set before executing <code>mangal</code>, this includes anilist bindings.</li> <li>Have a way to track the changes/updates on each run.</li> <li>Use that tracker to know where to start downloading chapters from.<ul> -<li>This is optional, as you can just do <code>--chapters "all"</code> and it will work. This is mostly to keep the logs/output cleaner/shorter.</li> +<li>This is optional, as you can just do <code>--chapters "all"</code> and it will work but I do it mostly to keep the logs/output cleaner/shorter.</li> </ul> </li> -<li>Do any configuration needed beforehand.</li> <li>Download/update each manga using <code>mangal inline</code>.</li> -<li>Wrap everything in a <code>systemd</code> service and timer.</li> +<li>Wrap everything in a <code>systemd</code> <code>service</code> and <code>timer</code>.</li> </ol> <p>Manga list example:</p> <pre><code class="language-sh">mangapill="Berserk|Chainsaw Man|Dandadan|Jujutsu Kaisen|etc..." </code></pre> -<p>Bash function that handles the download per manga in the list:</p> +<p>Function that handles the download per manga in the list:</p> <pre><code class="language-sh">mangal_src_dl () { source_name=$1 manga_list=$(echo "$2" | tr '|' '\n') @@ -359,6 +371,22 @@ mangal inline anilist set --name "Kimetsu no Yaiba" --id 101922</code></p> } </code></pre> <p>Where <code>$TRACKER_FILE</code> is just a variable holding a path to some file where you can store the tracking and <code>$DOWNLOAD_FORMAT</code> the format for the mangas, for me it’s <code>cbz</code>. Then the usage would be something like <code>mangal_src_dl "Mangapill" "$mangapill"</code>, meaning that it is a function call per source.</p> +<p>A simpler function without “tracking” would be:</p> +<pre><code class="language-sh">mangal_src_dl () { + source_name=$1 + manga_list=$(echo "$2" | tr '|' '\n') + + while IFS= read -r line; do + echo "Downloading all chapters for $line from $source_name..." + mangal inline -S "$source_name" -q "$line" -m "exact" -F "$DOWNLOAD_FORMAT" -c "all" -d + if [ $? -ne 0 ]; then + echo "Failed to download chapters for $line." + continue + fi + echo "Finished downloading chapters for $line." + done <<< "$manga_list" +} +</code></pre> <p>The tracker file would have a format like follows:</p> <pre><code># Updated: 06/10/23 10:53:15 AM CST Berserk|0392|392|Mangapill @@ -366,8 +394,12 @@ Dandadan|0110|110|Mangapill ... </code></pre> <p>And note that if you already had manga downloaded and you run the script for the first time, then it will show as if it downloaded everything from the first chapter, but that’s just how <code>mangal</code> works, it will actually just discover downloaded chapters and only download anything missing.</p> -<p>Any configuration the downloader/updater might need needs to be done before the <code>mangal_src_dl</code> calls. I like to configure mangal for download path, format, etc.. To clear the <code>mangal</code> cache and <code>rod</code> browser (headless browser used in some custom sources) as well as set up any anilist bindings. An example of an anilist binding I had to do is for Mushoku Tensei, as it has both a light novel and manga version, both having different information, for me it was <code>mangal inline anilist set --name "Mushoku Tensei - Isekai Ittara Honki Dasu" --id 85564</code>.</p> -<p>Finally is just a matter of using your prefered way of scheduling, I’ll use <code>systemd/Timers</code> but anything is fine. You could make the downloader script more sophisticated and only running every week on which each manga gets released usually, but that’s too much work, so I’ll just run it once daily probably, or 2-3 times daily.</p> +<p>Any configuration the downloader/updater might need needs to be done before the <code>mangal_src_dl</code> calls. I like to configure mangal for download path, format, etc.. I found that it is needed to clear the <code>mangal</code> and <code>rod</code> browser cache (headless browser used in some custom sources) from personal experience and from others: <a href="https://github.com/metafates/mangal/issues/170">mangal#170</a> and <a href="https://github.com/oae/kaizoku/issues/89">kaizoku#89</a>.</p> +<p>Also you should set any anilist binding necessary for the downloading (as the cache was cleared). An example of an anilist binding I had to do is for Mushoku Tensei, as it has both a <a href="https://anilist.co/manga/85470/Mushoku-Tensei-Jobless-Reincarnation/">light novel</a> and <a href="https://anilist.co/manga/85564/Mushoku-Tensei-Jobless-Reincarnation/">manga</a> version, which for me it’s the following binding:</p> +<pre><code class="language-sh">mangal inline anilist set --name "Mushoku Tensei - Isekai Ittara Honki Dasu" --id 85564 +</code></pre> +<p>Finally is just a matter of using your prefered way of scheduling, I’ll use <code>systemd/Timers</code> but anything is fine. You could make the downloader script more sophisticated and only running every week on which each manga gets (usually) released but that’s too much work; I’ll just run it once daily probably.</p> +<p>A feature I want to add and probably will is sending notifications (probably through email) on a summary for manga downloaded or failed to download so I’m on top of the updates. For now this is good enough and it’s been working so far.</p> <h3 id="alternative-downloaders">Alternative downloaders<a class="headerlink" href="#alternative-downloaders" title="Permanent link">¶</a></h3> <p>Just for the record, here is a list of downloaders/scrapers I considered before starting to use <code>mangal</code>:</p> <ul> diff --git a/live/blog/sitemap.xml b/live/blog/sitemap.xml index 1904029..a30b52f 100644 --- a/live/blog/sitemap.xml +++ b/live/blog/sitemap.xml @@ -47,7 +47,7 @@ <url> <loc>https://blog.luevano.xyz/a/manga_server_with_komga.html</loc> - <lastmod>2023-06-10</lastmod> + <lastmod>2023-06-11</lastmod> <changefreq>weekly</changefreq> <priority>1.0</priority> </url> |