1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
|
<!DOCTYPE html>
<html class="theme-dark" lang="en
"
prefix="og: https://ogp.me/ns#">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="icon" href="https://static.luevano.xyz/images/icons/favicon.ico">
<title>Configure system logs on Arch to avoid filled up disk -- Luévano's Blog</title>
<meta name="description" content="How to configure the system logs, mostly journald, from filling up the disk, on Arch."/>
<link rel="alternate" type="application/rss+xml" href="https://blog.luevano.xyz/rss.xml" title="Luévano's Blog RSS">
<!-- general style -->
<link rel="stylesheet" type="text/css" href="https://static.luevano.xyz/css/style.css">
<link rel="stylesheet" type="text/css" href="https://static.luevano.xyz/fork-awesome/css/fork-awesome.min.css">
<link rel="stylesheet" type="text/css" href="https://static.luevano.xyz/font-awesome/css/all.min.css">
<!-- theme related -->
<script type="text/javascript" src="https://static.luevano.xyz/scripts/theme.js"></script>
<link id="theme-css" rel="stylesheet" type="text/css" href="https://static.luevano.xyz/css/theme.css">
<!-- misc functions-->
<script type="text/javascript" src="https://static.luevano.xyz/scripts/return_top.js"></script>
<!-- extra -->
<!-- highlight support for code blocks -->
<script type="text/javascript" src="https://static.luevano.xyz/hl/highlight.min.js"></script>
<script type="text/javascript">
hljs.initHighlightingOnLoad();
</script>
<link id="code-theme-css" rel="stylesheet" type="text/css" href="https://static.luevano.xyz/hl/styles/nord.min.css">
<!-- og meta -->
<meta property="og:title" content="Configure system logs on Arch to avoid filled up disk -- Luévano's Blog"/>
<meta property="og:type" content="article"/>
<meta property="og:url" content="https://blog.luevano.xyz/a/arch_logs_flooding_disk.md"/>
<meta property="og:image" content="https://static.luevano.xyz/images/b/default.png"/>
<meta property="og:description" content="How to configure the system logs, mostly journald, from filling up the disk, on Arch."/>
<meta property="og:locale" content="en"/>
<meta property="og:site_name" content="Luévano's Blog"/>
</head>
<body>
<header>
<nav>
<ul>
<li>
<a href="https://luevano.xyz/"><i class="fas fa-home" alt="Home"></i><span>Home</span></a>
</li>
<li>
<a href="https://blog.luevano.xyz/"><i class="fas fa-book-open" alt="Blog"></i><span>Blog</span></a>
</li>
<li>
<a href="https://art.luevano.xyz/"><i class="fas fa-paint-brush" alt="Art"></i><span>Art</span></a>
</li>
<li><i class="fab fa-git" alt="Git"></i><span>Git</span>
<ul>
<li><a href="https://git.luevano.xyz/" target="_blank"><i class="fab fa-git-alt" alt="Git-alt"></i></a></li>
<li><a href="https://github.com/luevano" target="_blank"><i class="fab fa-github" alt="Github"></i></a></li>
<li><a href="https://gitlab.com/dluevano" target="_blank"><i class="fab fa-gitlab" alt="Gitlab"></i></a></li>
</ul>
</li>
<li><i class="fas fa-box-open" alt="Stuff"></i><span>Stuff</span>
<ul>
<li><a href="https://gb.luevano.xyz/"><i class="fas fa-gamepad" alt="Gameboy"></i><span>Gameboy</span></a></li>
</ul>
</li>
</ul>
</nav>
<button class="theme-switcher" onclick="toggleTheme()"><i class="fas fa-moon"></i><i class="fas fa-sun"></i></button>
</header>
<main>
<div class="return-top">
<button class="return-top" onclick="returnTop()" id="returnTopButton">
<i class="fas fa-arrow-up" alt="Return to top"></i>
</button>
</div>
<h1>Configure system logs on Arch to avoid filled up disk</h1>
<p>It’s been a while since I’ve been running a minimal server on a VPS, and it is a pretty humble VPS with just 32 GB of storage which works for me as I’m only hosting a handful of services. At some point I started noticing that the disk keept filling up on each time I checked.</p>
<p>Turns out that out of the box, Arch has a default config for <code>systemd</code>‘s <code>journald</code> that keeps a persistent <code>journal</code> log, but doesn’t have a limit on how much logging is kept. This means that depending on how many services, and how aggresive they log, it can be filled up pretty quickly. For me I had around 15 GB of logs, from the normal <code>journal</code> directory, <code>nginx</code> directory and my now unused <code>prosody</code> instance.</p>
<p>For <code>prosody</code> it was just a matter of deleting the directory as I’m not using it anymore, which freed around 4 GB of disk space.
For <code>journal</code> I did a combination of configuring <code>SystemMaxUse</code> and creating a <em>Namespace</em> for all “email” related services as mentioned in the <a href="https://wiki.archlinux.org/title/Systemd/Journal#Per_unit_size_limit_by_a_journal_namespace">Arch wiki: systemd/Journal</a>; basically just configuring <code>/etc/systemd/journald.conf</code> (and <code>/etc/systemd/journald@email.conf</code> with the comment change) with:</p>
<pre><code class="language-ini">[Journal]
Storage=persistent
SystemMaxUse=100MB # 50MB for the "email" Namespace
</code></pre>
<p>And then for each service that I want to use this “email” <em>Namespace</em> I add:</p>
<pre><code class="language-ini">[Service]
LogNamespace=email
</code></pre>
<p>Which can be changed manually or by executing <code>systemctl edit service_name.service</code> and it will create an override file which will be read on top of the normal service configuration. Once configured restart by running <code>systemctl daemon-reload</code> and <code>systemctl restart service_name.service</code> (probably also restart <code>systemd-journald</code>).</p>
<p>I also disabled the logging for <code>ufw</code> by running <code>ufw logging off</code> as it logs everything to the <code>kernel</code> “unit”, and I didn’t find a way to pipe its logs to a separate directory. It really isn’t that useful as most of the logs are the normal <code>[UFW BLOCK]</code> log, which is normal. If I need debugging then I’ll just enable that again. Note that you can change the logging level, if you still want some kind of logging.</p>
<p>Finally to clean up the <code>nginx</code> logs, you need to install <code>logrotate</code> (<code>pacman -S logrotate</code>) as that is what is used to clean up the <code>nginx</code> log directory. <code>nginx</code> already “installs” a config file for <code>logrotate</code> which is located at <code>/etc/logrotate.d/</code>, I just added a few lines:</p>
<pre><code>/var/log/nginx/*log {
rotate 7
weekly
dateext
dateformat -%Y-%m-%d
missingok
notifempty
create 640 http log
sharedscripts
compress
postrotate
test ! -r /run/nginx.pid || kill -USR1 `cat /run/nginx.pid`
endscript
}
</code></pre>
<p>Once you’re ok with your config, it’s just a matter of running <code>logrotate -v -f /etc/logrotate.d/nginx</code> which forces the run of the rule for <code>nginx</code>. After this, <code>logrotate</code> will be run daily if you <code>enable</code> the <code>logrotate</code> timer: <code>systemctl enable logrotate.timer</code>.</p>
<div class="page-nav">
<span class="index">
<a href="https://blog.luevano.xyz" alt="Index">
<i class="fas fa-home" alt="Home"></i>
<span>Index</span>
</a>
</span>
<span class="previous">
<a href="https://blog.luevano.xyz/a/manga_server_with_komga.html" alt="Previous">
<i class="fas fa-arrow-right" alt="Arrow right"></i>
<span>Previous</span>
</a>
</span>
</div>
<hr>
<div class="article-info">
<p>By David Luévano</p>
<p>Created: Thu, Jun 15, 2023 @ 10:22 UTC</p>
<p>Modified: Thu, Jun 15, 2023 @ 10:24 UTC</p>
<div class="article-tags">
<p>Tags:
<a href="https://blog.luevano.xyz/tag/@code.html">code</a>, <a href="https://blog.luevano.xyz/tag/@english.html">english</a>, <a href="https://blog.luevano.xyz/tag/@server.html">server</a>, <a href="https://blog.luevano.xyz/tag/@short.html">short</a>, <a href="https://blog.luevano.xyz/tag/@tools.html">tools</a>, <a href="https://blog.luevano.xyz/tag/@tutorial.html">tutorial</a> </p>
</div>
</div>
</main>
<footer>
<span>
<i class="fas fa-address-card" alt="Contact"></i>
<a href="https://blog.luevano.xyz/contact.html">Contact</a>
</span>
<span>
<i class="fas fa-donate" alt="Donate"></i>
<a href="https://blog.luevano.xyz/donate.html">Donate</a>
</span>
<span>
<i class="fas fa-rss" alt="RSS"></i>
<a target="_blank" href="https://blog.luevano.xyz/rss.xml">RSS</a>
</span>
<br>
<span class="created-with">
<i class="fas fa-hammer" alt="Hammer"></i>
Created with <a href="https://github.com/luevano/pyssg">pyssg</a>
</span>
<br>
<span class="copyright">
Copyright <i class="far fa-copyright" alt="Copyright"></i> 2023 David Luévano Alvarado
</span>
</footer>
</body>
</html>
|
