diff options
| author | David Luevano Alvarado <david@luevano.xyz> | 2023-06-15 04:23:12 -0600 |
|---|---|---|
| committer | David Luevano Alvarado <david@luevano.xyz> | 2023-06-15 04:23:12 -0600 |
| commit | 4299f2c21ff4e13a911f8b41d99d1fcef7823ec6 (patch) | |
| tree | e601cb3989257d885c893a176caf129e22e68f34 | |
| parent | 0c8910a53a1d730b025975ca0bc651c9992750a9 (diff) | |
add arch logs entry
| -rw-r--r-- | db/db_blog.psv | 1 | ||||
| -rw-r--r-- | live/blog/a/arch_logs_flooding_disk.html | 182 | ||||
| -rw-r--r-- | live/blog/a/manga_server_with_komga.html | 6 | ||||
| -rw-r--r-- | live/blog/index.html | 1 | ||||
| -rw-r--r-- | live/blog/rss.xml | 44 | ||||
| -rw-r--r-- | live/blog/sitemap.xml | 6 | ||||
| -rw-r--r-- | live/blog/tag/@code.html | 1 | ||||
| -rw-r--r-- | live/blog/tag/@english.html | 1 | ||||
| -rw-r--r-- | live/blog/tag/@server.html | 1 | ||||
| -rw-r--r-- | live/blog/tag/@short.html | 1 | ||||
| -rw-r--r-- | live/blog/tag/@tools.html | 1 | ||||
| -rw-r--r-- | live/blog/tag/@tutorial.html | 1 | ||||
| -rw-r--r-- | src/blog/a/arch_logs_flooding_disk.md | 55 |
13 files changed, 301 insertions, 0 deletions
diff --git a/db/db_blog.psv b/db/db_blog.psv index f6f0a70..fb22107 100644 --- a/db/db_blog.psv +++ b/db/db_blog.psv @@ -23,3 +23,4 @@ a/al_fin_tengo_fibra_opticona.md|1683622740.1853704|0.0|949b8fd2aca929d74d821747 a/learned_go_and_lua_hard_way.md|1685763137.7581403|0.0|493f86ce317f7b182b62c3438e5f7a8a|english,rant,short,tools
a/updating_creating_entries_titles_to_setup.md|1685764004.1478639|0.0|2bcf247ed5c2aa9fd5f4b542043123fe|english,short,update
a/manga_server_with_komga.md|1686425767.8936074|1686653315.6535006|7c33a7604565de1300710ce0add53944|code,english,server,tools,tutorial
+a/arch_logs_flooding_disk.md|1686824540.5338242|0.0|fcd1990de048be002380aea9a92333fe|code,english,server,short,tools,tutorial
diff --git a/live/blog/a/arch_logs_flooding_disk.html b/live/blog/a/arch_logs_flooding_disk.html new file mode 100644 index 0000000..4b8bd19 --- /dev/null +++ b/live/blog/a/arch_logs_flooding_disk.html @@ -0,0 +1,182 @@ +<!DOCTYPE html> +<html class="theme-dark" lang="en +" + prefix="og: https://ogp.me/ns#"> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <link rel="icon" href="https://static.luevano.xyz/images/icons/favicon.ico"> +<title>Configure system logs on Arch to avoid filled up disk -- Luévano's Blog</title> + <meta name="description" content="How to configure the system logs, mostly journald, from filling up the disk, on Arch."/> +<link rel="alternate" type="application/rss+xml" href="https://blog.luevano.xyz/rss.xml" title="Luévano's Blog RSS"> + <!-- general style --> + <link rel="stylesheet" type="text/css" href="https://static.luevano.xyz/css/style.css"> + <link rel="stylesheet" type="text/css" href="https://static.luevano.xyz/fork-awesome/css/fork-awesome.min.css"> + <link rel="stylesheet" type="text/css" href="https://static.luevano.xyz/font-awesome/css/all.min.css"> + <!-- theme related --> + <script type="text/javascript" src="https://static.luevano.xyz/scripts/theme.js"></script> + <link id="theme-css" rel="stylesheet" type="text/css" href="https://static.luevano.xyz/css/theme.css"> + <!-- misc functions--> + <script type="text/javascript" src="https://static.luevano.xyz/scripts/return_top.js"></script> + <!-- extra --> + <!-- highlight support for code blocks --> +<script type="text/javascript" src="https://static.luevano.xyz/hl/highlight.min.js"></script> +<script type="text/javascript"> + hljs.initHighlightingOnLoad(); +</script> +<link id="code-theme-css" rel="stylesheet" type="text/css" href="https://static.luevano.xyz/hl/styles/nord.min.css"> + + + + + + + + <!-- og meta --> + <meta property="og:title" content="Configure system logs on Arch to avoid filled up disk -- Luévano's Blog"/> + <meta property="og:type" content="article"/> + <meta property="og:url" content="https://blog.luevano.xyz/a/arch_logs_flooding_disk.md"/> + <meta property="og:image" content="https://static.luevano.xyz/images/b/default.png"/> + <meta property="og:description" content="How to configure the system logs, mostly journald, from filling up the disk, on Arch."/> + <meta property="og:locale" content="en"/> + <meta property="og:site_name" content="Luévano's Blog"/> + </head> + + <body> + <header> +<nav> + <ul> + <li> + <a href="https://luevano.xyz/"><i class="fas fa-home" alt="Home"></i><span>Home</span></a> + </li> + + <li> + <a href="https://blog.luevano.xyz/"><i class="fas fa-book-open" alt="Blog"></i><span>Blog</span></a> + </li> + + <li> + <a href="https://art.luevano.xyz/"><i class="fas fa-paint-brush" alt="Art"></i><span>Art</span></a> + </li> + + <li><i class="fab fa-git" alt="Git"></i><span>Git</span> + <ul> + <li><a href="https://git.luevano.xyz/" target="_blank"><i class="fab fa-git-alt" alt="Git-alt"></i></a></li> + + <li><a href="https://github.com/luevano" target="_blank"><i class="fab fa-github" alt="Github"></i></a></li> + + <li><a href="https://gitlab.com/dluevano" target="_blank"><i class="fab fa-gitlab" alt="Gitlab"></i></a></li> + </ul> + </li> + + <li><i class="fas fa-box-open" alt="Stuff"></i><span>Stuff</span> + <ul> + <li><a href="https://gb.luevano.xyz/"><i class="fas fa-gamepad" alt="Gameboy"></i><span>Gameboy</span></a></li> + </ul> + </li> + </ul> +</nav> + +<button class="theme-switcher" onclick="toggleTheme()"><i class="fas fa-moon"></i><i class="fas fa-sun"></i></button> + + </header> + + <main> + <div class="return-top"> + <button class="return-top" onclick="returnTop()" id="returnTopButton"> + <i class="fas fa-arrow-up" alt="Return to top"></i> + </button> + </div> + <h1>Configure system logs on Arch to avoid filled up disk</h1> + + <p>It’s been a while since I’ve been running a minimal server on a VPS, and it is a pretty humble VPS with just 32 GB of storage which works for me as I’m only hosting a handful of services. At some point I started noticing that the disk keept filling up on each time I checked.</p> +<p>Turns out that out of the box, Arch has a default config for <code>systemd</code>‘s <code>journald</code> that keeps a persistent <code>journal</code> log, but doesn’t have a limit on how much logging is kept. This means that depending on how many services, and how aggresive they log, it can be filled up pretty quickly. For me I had around 15 GB of logs, from the normal <code>journal</code> directory, <code>nginx</code> directory and my now unused <code>prosody</code> instance.</p> +<p>For <code>prosody</code> it was just a matter of deleting the directory as I’m not using it anymore, which freed around 4 GB of disk space. +For <code>journal</code> I did a combination of configuring <code>SystemMaxUse</code> and creating a <em>Namespace</em> for all “email” related services as mentioned in the <a href="https://wiki.archlinux.org/title/Systemd/Journal#Per_unit_size_limit_by_a_journal_namespace">Arch wiki: systemd/Journal</a>; basically just configuring <code>/etc/systemd/journald.conf</code> (and <code>/etc/systemd/journald@email.con</code> with the comment change) with:</p> +<pre><code class="language-conf">[Journal] +Storage=persistent +SystemMaxUse=100MB # 50MB for the "email" Namespace +</code></pre> +<p>And then for each service that I want to use this “email” <em>Namespace</em> I add:</p> +<pre><code class="language-conf">[Service] +LogNamespace=email +</code></pre> +<p>Which can be changed manually or by executing <code>systemctl edit service_name.service</code> and it will create an override file which will be read on top of the normal service configuration. Once configured restart by running <code>systemctl daemon-reload</code> and <code>systemctl restart service_name.service</code> (probably also restart <code>systemd-journald</code>).</p> +<p>I also disabled the logging for <code>ufw</code> by running <code>ufw logging off</code> as it logs everything to the <code>kernel</code> “unit”, and I didn’t find a way to pipe its logs to a separate directory. It really isn’t that useful as most of the logs are the normal <code>[UFW BLOCK]</code> log, which is normal. If I need debugging then I’ll just enable that again. Note that you can change the logging level, if you still want some kind of logging.</p> +<p>Finally to clean up the <code>nginx</code> logs, you need to install <code>logrotate</code> (<code>pacman -S logrotate</code>) as that is what is used to clean up the <code>nginx</code> log directory. <code>nginx</code> already “installs” a config file for <code>logrotate</code> which is located at <code>/etc/logrotate.d/</code>, I just added a few lines:</p> +<pre><code>/var/log/nginx/*log { + rotate 7 + weekly + dateext + dateformat -%Y-%m-%d + missingok + notifempty + create 640 http log + sharedscripts + compress + postrotate + test ! -r /run/nginx.pid || kill -USR1 `cat /run/nginx.pid` + endscript +} +</code></pre> +<p>Once you’re ok with your config, it’s just a matter of running <code>logrotate -v -f /etc/logrotate.d/nginx</code> which forces the run of the rule for <code>nginx</code>. After this, <code>logrotate</code> will be run daily if you <code>enable</code> the <code>logrotate</code> timer: <code>systemctl enable logrotate.timer</code>.</p> + + <div class="page-nav"> + + <span class="index"> + <a href="https://blog.luevano.xyz" alt="Index"> + <i class="fas fa-home" alt="Home"></i> + <span>Index</span> + </a> + </span> + + <span class="previous"> + <a href="https://blog.luevano.xyz/a/manga_server_with_komga.html" alt="Previous"> + <i class="fas fa-arrow-right" alt="Arrow right"></i> + <span>Previous</span> + </a> + </span> +</div> + + + <hr> + <div class="article-info"> + <p>By David Luévano</p> + <p>Created: Thu, Jun 15, 2023 @ 10:22 UTC</p> + <div class="article-tags"> + <p>Tags: +<a href="https://blog.luevano.xyz/tag/@code.html">code</a>, <a href="https://blog.luevano.xyz/tag/@english.html">english</a>, <a href="https://blog.luevano.xyz/tag/@server.html">server</a>, <a href="https://blog.luevano.xyz/tag/@short.html">short</a>, <a href="https://blog.luevano.xyz/tag/@tools.html">tools</a>, <a href="https://blog.luevano.xyz/tag/@tutorial.html">tutorial</a> </p> +</div> + + </div> + </main> + + <footer> +<span> + <i class="fas fa-address-card" alt="Contact"></i> + <a href="https://blog.luevano.xyz/contact.html">Contact</a> +</span> + +<span> + <i class="fas fa-donate" alt="Donate"></i> + <a href="https://blog.luevano.xyz/donate.html">Donate</a> +</span> + +<span> + <i class="fas fa-rss" alt="RSS"></i> + <a target="_blank" href="https://blog.luevano.xyz/rss.xml">RSS</a> +</span> + +<br> +<span class="created-with"> + <i class="fas fa-hammer" alt="Hammer"></i> + Created with <a href="https://github.com/luevano/pyssg">pyssg</a> +</span> + +<br> +<span class="copyright"> + Copyright <i class="far fa-copyright" alt="Copyright"></i> 2023 David Luévano Alvarado +</span> + + </footer> + </body> +</html>
\ No newline at end of file diff --git a/live/blog/a/manga_server_with_komga.html b/live/blog/a/manga_server_with_komga.html index 70095c3..21c5305 100644 --- a/live/blog/a/manga_server_with_komga.html +++ b/live/blog/a/manga_server_with_komga.html @@ -468,6 +468,12 @@ default:other::r-x </ul> <div class="page-nav"> + <span class="next"> + <a href="https://blog.luevano.xyz/a/arch_logs_flooding_disk.html" alt="Next"> + <i class="fas fa-arrow-left" alt="Arrow left"></i> + <span>Next</span> + </a> + </span> <span class="index"> <a href="https://blog.luevano.xyz" alt="Index"> diff --git a/live/blog/index.html b/live/blog/index.html index efd84fe..21f71a1 100644 --- a/live/blog/index.html +++ b/live/blog/index.html @@ -88,6 +88,7 @@ <h2>Articles</h2> <ul class="page-list"> <h3>2023</h3> + <li><span class="page-list-item">Jun 15</span> - <a href="https://blog.luevano.xyz/a/arch_logs_flooding_disk.html">Configure system logs on Arch to avoid filled up disk</a></li> <li><span class="page-list-item">Jun 10</span> - <a href="https://blog.luevano.xyz/a/manga_server_with_komga.html">Set up a manga server with Komga and mangal</a></li> <li><span class="page-list-item">Jun 03</span> - <a href="https://blog.luevano.xyz/a/updating_creating_entries_titles_to_setup.html">Updated the how-to entries titles</a></li> <li><span class="page-list-item">Jun 03</span> - <a href="https://blog.luevano.xyz/a/learned_go_and_lua_hard_way.html">I had to learn Go and Lua the hard way</a></li> diff --git a/live/blog/rss.xml b/live/blog/rss.xml index 6a21a21..9c7de3e 100644 --- a/live/blog/rss.xml +++ b/live/blog/rss.xml @@ -23,6 +23,50 @@ <link>https://blog.luevano.xyz</link> </image> <item> + <title>Configure system logs on Arch to avoid filled up disk</title> + <link>https://blog.luevano.xyz/a/arch_logs_flooding_disk.html</link> + <guid isPermaLink="true">https://blog.luevano.xyz/a/arch_logs_flooding_disk.html</guid> + <pubDate>Thu, 15 Jun 2023 10:22:20 GMT</pubDate> + <category>Code</category> + <category>English</category> + <category>Server</category> + <category>Short</category> + <category>Tools</category> + <category>Tutorial</category> + <description>How to configure the system logs, mostly journald, from filling up the disk, on Arch.</description> + <content:encoded><![CDATA[<p>It’s been a while since I’ve been running a minimal server on a VPS, and it is a pretty humble VPS with just 32 GB of storage which works for me as I’m only hosting a handful of services. At some point I started noticing that the disk keept filling up on each time I checked.</p> +<p>Turns out that out of the box, Arch has a default config for <code>systemd</code>‘s <code>journald</code> that keeps a persistent <code>journal</code> log, but doesn’t have a limit on how much logging is kept. This means that depending on how many services, and how aggresive they log, it can be filled up pretty quickly. For me I had around 15 GB of logs, from the normal <code>journal</code> directory, <code>nginx</code> directory and my now unused <code>prosody</code> instance.</p> +<p>For <code>prosody</code> it was just a matter of deleting the directory as I’m not using it anymore, which freed around 4 GB of disk space. +For <code>journal</code> I did a combination of configuring <code>SystemMaxUse</code> and creating a <em>Namespace</em> for all “email” related services as mentioned in the <a href="https://wiki.archlinux.org/title/Systemd/Journal#Per_unit_size_limit_by_a_journal_namespace">Arch wiki: systemd/Journal</a>; basically just configuring <code>/etc/systemd/journald.conf</code> (and <code>/etc/systemd/journald@email.con</code> with the comment change) with:</p> +<pre><code class="language-conf">[Journal] +Storage=persistent +SystemMaxUse=100MB # 50MB for the "email" Namespace +</code></pre> +<p>And then for each service that I want to use this “email” <em>Namespace</em> I add:</p> +<pre><code class="language-conf">[Service] +LogNamespace=email +</code></pre> +<p>Which can be changed manually or by executing <code>systemctl edit service_name.service</code> and it will create an override file which will be read on top of the normal service configuration. Once configured restart by running <code>systemctl daemon-reload</code> and <code>systemctl restart service_name.service</code> (probably also restart <code>systemd-journald</code>).</p> +<p>I also disabled the logging for <code>ufw</code> by running <code>ufw logging off</code> as it logs everything to the <code>kernel</code> “unit”, and I didn’t find a way to pipe its logs to a separate directory. It really isn’t that useful as most of the logs are the normal <code>[UFW BLOCK]</code> log, which is normal. If I need debugging then I’ll just enable that again. Note that you can change the logging level, if you still want some kind of logging.</p> +<p>Finally to clean up the <code>nginx</code> logs, you need to install <code>logrotate</code> (<code>pacman -S logrotate</code>) as that is what is used to clean up the <code>nginx</code> log directory. <code>nginx</code> already “installs” a config file for <code>logrotate</code> which is located at <code>/etc/logrotate.d/</code>, I just added a few lines:</p> +<pre><code>/var/log/nginx/*log { + rotate 7 + weekly + dateext + dateformat -%Y-%m-%d + missingok + notifempty + create 640 http log + sharedscripts + compress + postrotate + test ! -r /run/nginx.pid || kill -USR1 `cat /run/nginx.pid` + endscript +} +</code></pre> +<p>Once you’re ok with your config, it’s just a matter of running <code>logrotate -v -f /etc/logrotate.d/nginx</code> which forces the run of the rule for <code>nginx</code>. After this, <code>logrotate</code> will be run daily if you <code>enable</code> the <code>logrotate</code> timer: <code>systemctl enable logrotate.timer</code>.</p>]]></content:encoded> + </item> + <item> <title>Set up a manga server with Komga and mangal</title> <link>https://blog.luevano.xyz/a/manga_server_with_komga.html</link> <guid isPermaLink="true">https://blog.luevano.xyz/a/manga_server_with_komga.html</guid> diff --git a/live/blog/sitemap.xml b/live/blog/sitemap.xml index 976623d..0efd230 100644 --- a/live/blog/sitemap.xml +++ b/live/blog/sitemap.xml @@ -46,6 +46,12 @@ </url> <url> + <loc>https://blog.luevano.xyz/a/arch_logs_flooding_disk.html</loc> + <lastmod>2023-06-15</lastmod> + <changefreq>weekly</changefreq> + <priority>1.0</priority> + </url> + <url> <loc>https://blog.luevano.xyz/a/manga_server_with_komga.html</loc> <lastmod>2023-06-13</lastmod> <changefreq>weekly</changefreq> diff --git a/live/blog/tag/@code.html b/live/blog/tag/@code.html index 56f95fd..bcb9349 100644 --- a/live/blog/tag/@code.html +++ b/live/blog/tag/@code.html @@ -78,6 +78,7 @@ <h2>Articles</h2> <ul class="page-list"> <h3>2023</h3> + <li><span class="page-list-item">Jun 15</span> - <a href="https://blog.luevano.xyz/a/arch_logs_flooding_disk.html">Configure system logs on Arch to avoid filled up disk</a></li> <li><span class="page-list-item">Jun 10</span> - <a href="https://blog.luevano.xyz/a/manga_server_with_komga.html">Set up a manga server with Komga and mangal</a></li> <h3>2021</h3> <li><span class="page-list-item">Aug 01</span> - <a href="https://blog.luevano.xyz/a/vpn_server_with_openvpn.html">Set up a VPN server with OpenVPN</a></li> diff --git a/live/blog/tag/@english.html b/live/blog/tag/@english.html index 3b7f7c4..e0dad45 100644 --- a/live/blog/tag/@english.html +++ b/live/blog/tag/@english.html @@ -78,6 +78,7 @@ <h2>Articles</h2> <ul class="page-list"> <h3>2023</h3> + <li><span class="page-list-item">Jun 15</span> - <a href="https://blog.luevano.xyz/a/arch_logs_flooding_disk.html">Configure system logs on Arch to avoid filled up disk</a></li> <li><span class="page-list-item">Jun 10</span> - <a href="https://blog.luevano.xyz/a/manga_server_with_komga.html">Set up a manga server with Komga and mangal</a></li> <li><span class="page-list-item">Jun 03</span> - <a href="https://blog.luevano.xyz/a/updating_creating_entries_titles_to_setup.html">Updated the how-to entries titles</a></li> <li><span class="page-list-item">Jun 03</span> - <a href="https://blog.luevano.xyz/a/learned_go_and_lua_hard_way.html">I had to learn Go and Lua the hard way</a></li> diff --git a/live/blog/tag/@server.html b/live/blog/tag/@server.html index c5f41c4..d1292e8 100644 --- a/live/blog/tag/@server.html +++ b/live/blog/tag/@server.html @@ -78,6 +78,7 @@ <h2>Articles</h2> <ul class="page-list"> <h3>2023</h3> + <li><span class="page-list-item">Jun 15</span> - <a href="https://blog.luevano.xyz/a/arch_logs_flooding_disk.html">Configure system logs on Arch to avoid filled up disk</a></li> <li><span class="page-list-item">Jun 10</span> - <a href="https://blog.luevano.xyz/a/manga_server_with_komga.html">Set up a manga server with Komga and mangal</a></li> <h3>2021</h3> <li><span class="page-list-item">Aug 01</span> - <a href="https://blog.luevano.xyz/a/vpn_server_with_openvpn.html">Set up a VPN server with OpenVPN</a></li> diff --git a/live/blog/tag/@short.html b/live/blog/tag/@short.html index de3d874..a9b5c7d 100644 --- a/live/blog/tag/@short.html +++ b/live/blog/tag/@short.html @@ -78,6 +78,7 @@ <h2>Articles</h2> <ul class="page-list"> <h3>2023</h3> + <li><span class="page-list-item">Jun 15</span> - <a href="https://blog.luevano.xyz/a/arch_logs_flooding_disk.html">Configure system logs on Arch to avoid filled up disk</a></li> <li><span class="page-list-item">Jun 03</span> - <a href="https://blog.luevano.xyz/a/updating_creating_entries_titles_to_setup.html">Updated the how-to entries titles</a></li> <li><span class="page-list-item">Jun 03</span> - <a href="https://blog.luevano.xyz/a/learned_go_and_lua_hard_way.html">I had to learn Go and Lua the hard way</a></li> <li><span class="page-list-item">May 09</span> - <a href="https://blog.luevano.xyz/a/al_fin_tengo_fibra_opticona.html">Al fin tengo fibra ópticona</a></li> diff --git a/live/blog/tag/@tools.html b/live/blog/tag/@tools.html index 26e5b05..8f9f380 100644 --- a/live/blog/tag/@tools.html +++ b/live/blog/tag/@tools.html @@ -78,6 +78,7 @@ <h2>Articles</h2> <ul class="page-list"> <h3>2023</h3> + <li><span class="page-list-item">Jun 15</span> - <a href="https://blog.luevano.xyz/a/arch_logs_flooding_disk.html">Configure system logs on Arch to avoid filled up disk</a></li> <li><span class="page-list-item">Jun 10</span> - <a href="https://blog.luevano.xyz/a/manga_server_with_komga.html">Set up a manga server with Komga and mangal</a></li> <li><span class="page-list-item">Jun 03</span> - <a href="https://blog.luevano.xyz/a/learned_go_and_lua_hard_way.html">I had to learn Go and Lua the hard way</a></li> <li><span class="page-list-item">May 06</span> - <a href="https://blog.luevano.xyz/a/updated_pyssg_pymdvar_and_website.html">Updated pyssg to include pymdvar and the website</a></li> diff --git a/live/blog/tag/@tutorial.html b/live/blog/tag/@tutorial.html index 7d8d4c7..fa88ac2 100644 --- a/live/blog/tag/@tutorial.html +++ b/live/blog/tag/@tutorial.html @@ -78,6 +78,7 @@ <h2>Articles</h2> <ul class="page-list"> <h3>2023</h3> + <li><span class="page-list-item">Jun 15</span> - <a href="https://blog.luevano.xyz/a/arch_logs_flooding_disk.html">Configure system logs on Arch to avoid filled up disk</a></li> <li><span class="page-list-item">Jun 10</span> - <a href="https://blog.luevano.xyz/a/manga_server_with_komga.html">Set up a manga server with Komga and mangal</a></li> <h3>2021</h3> <li><span class="page-list-item">Aug 01</span> - <a href="https://blog.luevano.xyz/a/vpn_server_with_openvpn.html">Set up a VPN server with OpenVPN</a></li> diff --git a/src/blog/a/arch_logs_flooding_disk.md b/src/blog/a/arch_logs_flooding_disk.md new file mode 100644 index 0000000..6a83e29 --- /dev/null +++ b/src/blog/a/arch_logs_flooding_disk.md @@ -0,0 +1,55 @@ +title: Configure system logs on Arch to avoid filled up disk +author: David Luévano +lang: en +summary: How to configure the system logs, mostly journald, from filling up the disk, on Arch. +tags: short + server + tools + code + tutorial + english + +It's been a while since I've been running a minimal server on a VPS, and it is a pretty humble VPS with just 32 GB of storage which works for me as I'm only hosting a handful of services. At some point I started noticing that the disk keept filling up on each time I checked. + +Turns out that out of the box, Arch has a default config for `systemd`'s `journald` that keeps a persistent `journal` log, but doesn't have a limit on how much logging is kept. This means that depending on how many services, and how aggresive they log, it can be filled up pretty quickly. For me I had around 15 GB of logs, from the normal `journal` directory, `nginx` directory and my now unused `prosody` instance. + +For `prosody` it was just a matter of deleting the directory as I'm not using it anymore, which freed around 4 GB of disk space. +For `journal` I did a combination of configuring `SystemMaxUse` and creating a *Namespace* for all "email" related services as mentioned in the [Arch wiki: systemd/Journal](https://wiki.archlinux.org/title/Systemd/Journal#Per_unit_size_limit_by_a_journal_namespace); basically just configuring `/etc/systemd/journald.conf` (and `/etc/systemd/journald@email.con` with the comment change) with: + +```conf +[Journal] +Storage=persistent +SystemMaxUse=100MB # 50MB for the "email" Namespace +``` + +And then for each service that I want to use this "email" *Namespace* I add: + +```conf +[Service] +LogNamespace=email +``` + +Which can be changed manually or by executing `systemctl edit service_name.service` and it will create an override file which will be read on top of the normal service configuration. Once configured restart by running `systemctl daemon-reload` and `systemctl restart service_name.service` (probably also restart `systemd-journald`). + +I also disabled the logging for `ufw` by running `ufw logging off` as it logs everything to the `kernel` "unit", and I didn't find a way to pipe its logs to a separate directory. It really isn't that useful as most of the logs are the normal `[UFW BLOCK]` log, which is normal. If I need debugging then I'll just enable that again. Note that you can change the logging level, if you still want some kind of logging. + +Finally to clean up the `nginx` logs, you need to install `logrotate` (`pacman -S logrotate`) as that is what is used to clean up the `nginx` log directory. `nginx` already "installs" a config file for `logrotate` which is located at `/etc/logrotate.d/`, I just added a few lines: + +``` +/var/log/nginx/*log { + rotate 7 + weekly + dateext + dateformat -%Y-%m-%d + missingok + notifempty + create 640 http log + sharedscripts + compress + postrotate + test ! -r /run/nginx.pid || kill -USR1 `cat /run/nginx.pid` + endscript +} +``` + +Once you're ok with your config, it's just a matter of running `logrotate -v -f /etc/logrotate.d/nginx` which forces the run of the rule for `nginx`. After this, `logrotate` will be run daily if you `enable` the `logrotate` timer: `systemctl enable logrotate.timer`.
\ No newline at end of file |