From 4299f2c21ff4e13a911f8b41d99d1fcef7823ec6 Mon Sep 17 00:00:00 2001 From: David Luevano Alvarado Date: Thu, 15 Jun 2023 04:23:12 -0600 Subject: add arch logs entry --- db/db_blog.psv | 1 + live/blog/a/arch_logs_flooding_disk.html | 182 +++++++++++++++++++++++++++++++ live/blog/a/manga_server_with_komga.html | 6 + live/blog/index.html | 1 + live/blog/rss.xml | 44 ++++++++ live/blog/sitemap.xml | 6 + live/blog/tag/@code.html | 1 + live/blog/tag/@english.html | 1 + live/blog/tag/@server.html | 1 + live/blog/tag/@short.html | 1 + live/blog/tag/@tools.html | 1 + live/blog/tag/@tutorial.html | 1 + src/blog/a/arch_logs_flooding_disk.md | 55 ++++++++++ 13 files changed, 301 insertions(+) create mode 100644 live/blog/a/arch_logs_flooding_disk.html create mode 100644 src/blog/a/arch_logs_flooding_disk.md diff --git a/db/db_blog.psv b/db/db_blog.psv index f6f0a70..fb22107 100644 --- a/db/db_blog.psv +++ b/db/db_blog.psv @@ -23,3 +23,4 @@ a/al_fin_tengo_fibra_opticona.md|1683622740.1853704|0.0|949b8fd2aca929d74d821747 a/learned_go_and_lua_hard_way.md|1685763137.7581403|0.0|493f86ce317f7b182b62c3438e5f7a8a|english,rant,short,tools a/updating_creating_entries_titles_to_setup.md|1685764004.1478639|0.0|2bcf247ed5c2aa9fd5f4b542043123fe|english,short,update a/manga_server_with_komga.md|1686425767.8936074|1686653315.6535006|7c33a7604565de1300710ce0add53944|code,english,server,tools,tutorial +a/arch_logs_flooding_disk.md|1686824540.5338242|0.0|fcd1990de048be002380aea9a92333fe|code,english,server,short,tools,tutorial diff --git a/live/blog/a/arch_logs_flooding_disk.html b/live/blog/a/arch_logs_flooding_disk.html new file mode 100644 index 0000000..4b8bd19 --- /dev/null +++ b/live/blog/a/arch_logs_flooding_disk.html @@ -0,0 +1,182 @@ + + + + + + +Configure system logs on Arch to avoid filled up disk -- Luévano's Blog + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + + +
+ +
+
+ +
+

Configure system logs on Arch to avoid filled up disk

+ +

It’s been a while since I’ve been running a minimal server on a VPS, and it is a pretty humble VPS with just 32 GB of storage which works for me as I’m only hosting a handful of services. At some point I started noticing that the disk keept filling up on each time I checked.

+

Turns out that out of the box, Arch has a default config for systemd‘s journald that keeps a persistent journal log, but doesn’t have a limit on how much logging is kept. This means that depending on how many services, and how aggresive they log, it can be filled up pretty quickly. For me I had around 15 GB of logs, from the normal journal directory, nginx directory and my now unused prosody instance.

+

For prosody it was just a matter of deleting the directory as I’m not using it anymore, which freed around 4 GB of disk space. +For journal I did a combination of configuring SystemMaxUse and creating a Namespace for all “email” related services as mentioned in the Arch wiki: systemd/Journal; basically just configuring /etc/systemd/journald.conf (and /etc/systemd/journald@email.con with the comment change) with:

+
[Journal]
+Storage=persistent
+SystemMaxUse=100MB # 50MB for the "email" Namespace
+
+

And then for each service that I want to use this “email” Namespace I add:

+
[Service]
+LogNamespace=email
+
+

Which can be changed manually or by executing systemctl edit service_name.service and it will create an override file which will be read on top of the normal service configuration. Once configured restart by running systemctl daemon-reload and systemctl restart service_name.service (probably also restart systemd-journald).

+

I also disabled the logging for ufw by running ufw logging off as it logs everything to the kernel “unit”, and I didn’t find a way to pipe its logs to a separate directory. It really isn’t that useful as most of the logs are the normal [UFW BLOCK] log, which is normal. If I need debugging then I’ll just enable that again. Note that you can change the logging level, if you still want some kind of logging.

+

Finally to clean up the nginx logs, you need to install logrotate (pacman -S logrotate) as that is what is used to clean up the nginx log directory. nginx already “installs” a config file for logrotate which is located at /etc/logrotate.d/, I just added a few lines:

+
/var/log/nginx/*log {
+    rotate 7
+    weekly
+    dateext
+    dateformat -%Y-%m-%d
+    missingok
+    notifempty
+    create 640 http log
+    sharedscripts
+    compress
+    postrotate
+        test ! -r /run/nginx.pid || kill -USR1 `cat /run/nginx.pid`
+    endscript
+}
+
+

Once you’re ok with your config, it’s just a matter of running logrotate -v -f /etc/logrotate.d/nginx which forces the run of the rule for nginx. After this, logrotate will be run daily if you enable the logrotate timer: systemctl enable logrotate.timer.

+ +
+ + + + + Index + + + + + + + Previous + + +
+ + +
+
+

By David Luévano

+

Created: Thu, Jun 15, 2023 @ 10:22 UTC

+
+

Tags: +code, english, server, short, tools, tutorial

+
+ +
+
+ + + + \ No newline at end of file diff --git a/live/blog/a/manga_server_with_komga.html b/live/blog/a/manga_server_with_komga.html index 70095c3..21c5305 100644 --- a/live/blog/a/manga_server_with_komga.html +++ b/live/blog/a/manga_server_with_komga.html @@ -468,6 +468,12 @@ default:other::r-x
+ + + + Next + + diff --git a/live/blog/index.html b/live/blog/index.html index efd84fe..21f71a1 100644 --- a/live/blog/index.html +++ b/live/blog/index.html @@ -88,6 +88,7 @@

Articles

    2023

    +
  • Jun 15 - Configure system logs on Arch to avoid filled up disk
  • Jun 10 - Set up a manga server with Komga and mangal
  • Jun 03 - Updated the how-to entries titles
  • Jun 03 - I had to learn Go and Lua the hard way
    • diff --git a/live/blog/rss.xml b/live/blog/rss.xml index 6a21a21..9c7de3e 100644 --- a/live/blog/rss.xml +++ b/live/blog/rss.xml @@ -22,6 +22,50 @@ Luévano's Blog https://blog.luevano.xyz + + Configure system logs on Arch to avoid filled up disk + https://blog.luevano.xyz/a/arch_logs_flooding_disk.html + https://blog.luevano.xyz/a/arch_logs_flooding_disk.html + Thu, 15 Jun 2023 10:22:20 GMT + Code + English + Server + Short + Tools + Tutorial + How to configure the system logs, mostly journald, from filling up the disk, on Arch. + It’s been a while since I’ve been running a minimal server on a VPS, and it is a pretty humble VPS with just 32 GB of storage which works for me as I’m only hosting a handful of services. At some point I started noticing that the disk keept filling up on each time I checked.

    +

    Turns out that out of the box, Arch has a default config for systemd‘s journald that keeps a persistent journal log, but doesn’t have a limit on how much logging is kept. This means that depending on how many services, and how aggresive they log, it can be filled up pretty quickly. For me I had around 15 GB of logs, from the normal journal directory, nginx directory and my now unused prosody instance.

    +

    For prosody it was just a matter of deleting the directory as I’m not using it anymore, which freed around 4 GB of disk space. +For journal I did a combination of configuring SystemMaxUse and creating a Namespace for all “email” related services as mentioned in the Arch wiki: systemd/Journal; basically just configuring /etc/systemd/journald.conf (and /etc/systemd/journald@email.con with the comment change) with:

    +
    [Journal]
+Storage=persistent
+SystemMaxUse=100MB # 50MB for the "email" Namespace
+
    +

    And then for each service that I want to use this “email” Namespace I add:

    +
    [Service]
+LogNamespace=email
+
    +

    Which can be changed manually or by executing systemctl edit service_name.service and it will create an override file which will be read on top of the normal service configuration. Once configured restart by running systemctl daemon-reload and systemctl restart service_name.service (probably also restart systemd-journald).

    +

    I also disabled the logging for ufw by running ufw logging off as it logs everything to the kernel “unit”, and I didn’t find a way to pipe its logs to a separate directory. It really isn’t that useful as most of the logs are the normal [UFW BLOCK] log, which is normal. If I need debugging then I’ll just enable that again. Note that you can change the logging level, if you still want some kind of logging.

    +

    Finally to clean up the nginx logs, you need to install logrotate (pacman -S logrotate) as that is what is used to clean up the nginx log directory. nginx already “installs” a config file for logrotate which is located at /etc/logrotate.d/, I just added a few lines:

    +
    /var/log/nginx/*log {
+    rotate 7
+    weekly
+    dateext
+    dateformat -%Y-%m-%d
+    missingok
+    notifempty
+    create 640 http log
+    sharedscripts
+    compress
+    postrotate
+        test ! -r /run/nginx.pid || kill -USR1 `cat /run/nginx.pid`
+    endscript
+}
+
    +

    Once you’re ok with your config, it’s just a matter of running logrotate -v -f /etc/logrotate.d/nginx which forces the run of the rule for nginx. After this, logrotate will be run daily if you enable the logrotate timer: systemctl enable logrotate.timer.

    ]]>     +     Set up a manga server with Komga and mangal https://blog.luevano.xyz/a/manga_server_with_komga.html diff --git a/live/blog/sitemap.xml b/live/blog/sitemap.xml index 976623d..0efd230 100644 --- a/live/blog/sitemap.xml +++ b/live/blog/sitemap.xml @@ -45,6 +45,12 @@ 1.0 + + https://blog.luevano.xyz/a/arch_logs_flooding_disk.html + 2023-06-15 + weekly + 1.0 + https://blog.luevano.xyz/a/manga_server_with_komga.html 2023-06-13 diff --git a/live/blog/tag/@code.html b/live/blog/tag/@code.html index 56f95fd..bcb9349 100644 --- a/live/blog/tag/@code.html +++ b/live/blog/tag/@code.html @@ -78,6 +78,7 @@

    Articles

      2023

      +
    • Jun 15 - Configure system logs on Arch to avoid filled up disk
    • Jun 10 - Set up a manga server with Komga and mangal

      • 2021

    • Aug 01 - Set up a VPN server with OpenVPN
      • diff --git a/live/blog/tag/@english.html b/live/blog/tag/@english.html index 3b7f7c4..e0dad45 100644 --- a/live/blog/tag/@english.html +++ b/live/blog/tag/@english.html @@ -78,6 +78,7 @@

      Articles

        2023

        +
      • Jun 15 - Configure system logs on Arch to avoid filled up disk
      • Jun 10 - Set up a manga server with Komga and mangal
      • Jun 03 - Updated the how-to entries titles
      • Jun 03 - I had to learn Go and Lua the hard way
        • diff --git a/live/blog/tag/@server.html b/live/blog/tag/@server.html index c5f41c4..d1292e8 100644 --- a/live/blog/tag/@server.html +++ b/live/blog/tag/@server.html @@ -78,6 +78,7 @@

        Articles

          2023

          +
        • Jun 15 - Configure system logs on Arch to avoid filled up disk
        • Jun 10 - Set up a manga server with Komga and mangal

          • 2021

        • Aug 01 - Set up a VPN server with OpenVPN
          • diff --git a/live/blog/tag/@short.html b/live/blog/tag/@short.html index de3d874..a9b5c7d 100644 --- a/live/blog/tag/@short.html +++ b/live/blog/tag/@short.html @@ -78,6 +78,7 @@

          Articles

            2023

            +
          • Jun 15 - Configure system logs on Arch to avoid filled up disk
          • Jun 03 - Updated the how-to entries titles
          • Jun 03 - I had to learn Go and Lua the hard way
          • May 09 - Al fin tengo fibra ópticona
            • diff --git a/live/blog/tag/@tools.html b/live/blog/tag/@tools.html index 26e5b05..8f9f380 100644 --- a/live/blog/tag/@tools.html +++ b/live/blog/tag/@tools.html @@ -78,6 +78,7 @@

            Articles

              2023

              +
            • Jun 15 - Configure system logs on Arch to avoid filled up disk
            • Jun 10 - Set up a manga server with Komga and mangal
            • Jun 03 - I had to learn Go and Lua the hard way
            • May 06 - Updated pyssg to include pymdvar and the website
              • diff --git a/live/blog/tag/@tutorial.html b/live/blog/tag/@tutorial.html index 7d8d4c7..fa88ac2 100644 --- a/live/blog/tag/@tutorial.html +++ b/live/blog/tag/@tutorial.html @@ -78,6 +78,7 @@

              Articles

                2023

                +
              • Jun 15 - Configure system logs on Arch to avoid filled up disk
              • Jun 10 - Set up a manga server with Komga and mangal

                • 2021

              • Aug 01 - Set up a VPN server with OpenVPN
                • diff --git a/src/blog/a/arch_logs_flooding_disk.md b/src/blog/a/arch_logs_flooding_disk.md new file mode 100644 index 0000000..6a83e29 --- /dev/null +++ b/src/blog/a/arch_logs_flooding_disk.md @@ -0,0 +1,55 @@ +title: Configure system logs on Arch to avoid filled up disk +author: David Luévano +lang: en +summary: How to configure the system logs, mostly journald, from filling up the disk, on Arch. +tags: short + server + tools + code + tutorial + english + +It's been a while since I've been running a minimal server on a VPS, and it is a pretty humble VPS with just 32 GB of storage which works for me as I'm only hosting a handful of services. At some point I started noticing that the disk keept filling up on each time I checked. + +Turns out that out of the box, Arch has a default config for `systemd`'s `journald` that keeps a persistent `journal` log, but doesn't have a limit on how much logging is kept. This means that depending on how many services, and how aggresive they log, it can be filled up pretty quickly. For me I had around 15 GB of logs, from the normal `journal` directory, `nginx` directory and my now unused `prosody` instance. + +For `prosody` it was just a matter of deleting the directory as I'm not using it anymore, which freed around 4 GB of disk space. +For `journal` I did a combination of configuring `SystemMaxUse` and creating a *Namespace* for all "email" related services as mentioned in the [Arch wiki: systemd/Journal](https://wiki.archlinux.org/title/Systemd/Journal#Per_unit_size_limit_by_a_journal_namespace); basically just configuring `/etc/systemd/journald.conf` (and `/etc/systemd/journald@email.con` with the comment change) with: + +```conf +[Journal] +Storage=persistent +SystemMaxUse=100MB # 50MB for the "email" Namespace +``` + +And then for each service that I want to use this "email" *Namespace* I add: + +```conf +[Service] +LogNamespace=email +``` + +Which can be changed manually or by executing `systemctl edit service_name.service` and it will create an override file which will be read on top of the normal service configuration. Once configured restart by running `systemctl daemon-reload` and `systemctl restart service_name.service` (probably also restart `systemd-journald`). + +I also disabled the logging for `ufw` by running `ufw logging off` as it logs everything to the `kernel` "unit", and I didn't find a way to pipe its logs to a separate directory. It really isn't that useful as most of the logs are the normal `[UFW BLOCK]` log, which is normal. If I need debugging then I'll just enable that again. Note that you can change the logging level, if you still want some kind of logging. + +Finally to clean up the `nginx` logs, you need to install `logrotate` (`pacman -S logrotate`) as that is what is used to clean up the `nginx` log directory. `nginx` already "installs" a config file for `logrotate` which is located at `/etc/logrotate.d/`, I just added a few lines: + +``` +/var/log/nginx/*log { + rotate 7 + weekly + dateext + dateformat -%Y-%m-%d + missingok + notifempty + create 640 http log + sharedscripts + compress + postrotate + test ! -r /run/nginx.pid || kill -USR1 `cat /run/nginx.pid` + endscript +} +``` + +Once you're ok with your config, it's just a matter of running `logrotate -v -f /etc/logrotate.d/nginx` which forces the run of the rule for `nginx`. After this, `logrotate` will be run daily if you `enable` the `logrotate` timer: `systemctl enable logrotate.timer`. \ No newline at end of file -- cgit v1.2.3-70-g09d2