Blog

01/10/2026

openvpn dco / mtu / mss clamping / packet fragmentation

Since we switched OpenVPN to the in-kernel data channel (DCO, ovpn module), traffic through the tunnel produced oversized outer UDP datagrams that got IP-fragmented. Fixed in two steps: MSS clamping in mangle on the vpn-server interface (TCP: MSS 1460 → 1400). tun-mtu 1440 in the OpenVPN config, for everything MSS clamping can't touch (QUIC, VPN-in-VPN). Verified with a bpftrace script that counts IPv4 fragments and oversized GRO segments. Both counters stay at zero.

containerd / kubernetes / open file limits

After upgrading a customer's Kubernetes cluster running on Ubuntu/Jammy, we ran into a snag. The vernemq instances wouldn't start completely, instead they gave us EMFILE errors: Too many open files. This came as a surprise. After all, we hadn't touched any limits. But, it turned out containerd v1.8 changed the LimitNOFILE setting from a very permissive infinity to the systemd default. The result? Processes inside Kubernetes would get a measly max 1024 file descriptors by default.

linux / local root exploit / module vetting

Recently, we were greeted with the Copy Fail Linux kernel vulnerability. Mitigating this was a matter of denylisting a module. But, only eight days later, there was another exploit, also (ab)using AF_ALG and kernel module autoloading. I'm betting this is not the last, now that the kernel is scrutinized using AI models that keep getting more advanced. Luckily, we had our machine inventory up to date. So when CVE-2026-31431 ("Copy Fail") came along, deploying a mitigation was a matter of:

macos tahoe / ecn / slow downloads

Recently, a customer reported an intermittent but frustrating issue: since upgrading to macOS 26 (Tahoe), downloads from our servers were occasionally crawling. Not always, and not for everyone. The culprit turned out to be a combination of Explicit Congestion Notification (ECN), NIC offloading limitations, and the way classic congestion control algorithms react to "well-intentioned" signals. The macOS Tahoe ECN lottery The first mystery was the intermittency. We noticed that some connections were fast, while others to the same server appeared throttled.

recap 2025 - updates at OSSO

2025 – Finding the Turbo Button A year of momentum. After the “certification heavy” focus of 2024, 2025 felt like shifting into a higher gear. We’ve seen our infrastructure move to 100G, our hardware embrace a new generation of speed, and our team grow into a tighter, faster unit. Looking back at the last twelve months, the theme hasn’t just been “maintenance” — it’s been about refinement. We’ve been busy cleaning out the old (goodbye spinning rust and legacy loadbalancers) to make room for the new (hello EPYC processors and Nokia core routers).

Kubernetes: the new Gateway API

Transitioning K8S towards the Gateway API We’ve heard the chatter about the retirement of the ingress-nginx ingress-controller. It seems to be a hot topic! We’ve had quite a few people ask: “Hey OSSO, have you seen this already?" Yes! We have! Actually, we’re right on top of it. We plan to smoothly transition and adapt, and today we want to clarify OSSO’s approach going forward. This post outlines the transition from the established Kubernetes Ingress resource to its successor Gateway API, detailing the reasons for the transition, our proposed migration path, and the key architectural concepts.

ubuntu / gnome config / issues / 2025

I had flushed my desktop and was ready to start fresh. Still using Ubuntu for now. It does take a few hacks and tricks to get my system right. Disable and purge snap For one, I'm removing snap. The Ubuntu Snap takeover has cost me so many man hours. As mentioned in the linked post, removing it and going with the Linux Mint packages was a great decision. Not once have I regretted not having Snap installed.

proxmox vm / no arp replies / mellanox nic

Recently, when upgrading a Proxmox host from 7.2 to 8.4, we ran into a strange issue. The hypervisor host had network connectivity, but the VM guests did not. The upgrade was continued to Proxmox 9.0, but that did not fix the problem. The host was fine. We could ssh in, ping the gateway, do updates, whatever — but the guests had trouble communicating. In particular: the guests appeared unable to find the gateway at layer 2.

zabbix / wrong nvme temperature readings

After installing some new nodes Zabbix began reporting NVMe drive temperatures that were... alarming. Think "above 60 °C" alarming. But when I checked the same drives manually, the values were sane. So, either the drives were gaslighting me, or Zabbix was. The search Decomposing what our monitoring scripts do was really just calling nvme-cli: # nvme smart-log /dev/nvme0n1 | grep temper temperature : 33 °C (306 K) Nothing wrong there.

loopback through leaf / single nic iperf test

When bringing up new servers, especially with new or untested network interface cards (NICs) or cables, you might want to test their raw network throughput. While standard iperf tests between two different machines are common, sometimes you want to isolate the test to a single NIC, pushing traffic through it and back, effectively looping it at a nearby switch. This verifies the physical path, the SFP, and the NIC's transmit/receive capabilities without involving another server's NIC or network stack.

invalid elf header magic / rook-ceph / k8s

Yesterday, we noticed some previously unseen messages in the logs: Invalid ELF header magic: != \x7fELF. Of course an ELF binary should start with a valid header. Who/what is trying to run something else? These messages appeared on Kubernetes nodes that were recently installed and not yet fully in use for production workloads. Coincidentally, these particular nodes were running Ubuntu 24.04 (Noble Numbat), while the rest of this cluster was still on older versions.

nginx / no realip in logs / bug

Recently we were doing a vulnerability scan on an external endpoint of a website. During this time, we noticed simultaneous suspicious activity coming from inside our internal network. What's the deal? To make a long story short: we use a proxy (like HAProxy) in front of our webservers, and it speaks the Proxy Protocol to pass along the original client's IP address. That way, when a request hits our internal nginx server, the logs show the real client's IP, not the IP of the reverse proxy that actually made the connection.

supermicro / ikvm / expired certificate

Supermicro computers generally work very well. At our company we've been using them for ages. One thing that does keep giving us the occasional trouble however is the BMC. Today, we'll be looking at expired certificates of the iKVM interface. Introduction First, some definitions, which we generally use interchangeably: The Baseboard Management Controller (BMC) is the chipset core of the Intelligent Platform Management Interface (IPMI) implementation, which we mostly use to remotely attach a Keyboard, Video and Mouse (KVM, over IP).

pveproxy / systemd journald / perl buffering

Because I was debugging — what later turned out to be — an application firewall that was messing with my connections to Proxmox pveproxy on port 8006, I wanted more debug logs. Telling pveproxy to give me more info gave me a bit of a hard time. Connections from a remote peer would get disconnected, and it initially looked like pveproxy was to blame. Step one: get more info from pveproxy.

sed / regular expressions / optimizing

Last week, we were looking at using sed(1) to remove line feeds from CSV files. The regular expressions in that post could use some further inspection. I'm sorry that this is a rather dry post. I wanted to get the numbers out there because the differences are significant. But without the example runs you're just left with numbers. Feel free to skip right to the conclusions. To recap: we were working on multiline CSV files.

sed / remove line feeds / csv

Can you use sed(1) to remove line feeds? Yes you can. Even though sed, the stream editor, is line based, you can use it to remove line feeds from output. In this post we'll be exploring how that is done and look into how a slight refactoring step can provide a big speed gain. The task at hand is removing line feeds from CSV file with multiline strings. Why? Because I want to use grep and awk on it, and multiline strings would complicate that a lot.

recap 2024 - updates at OSSO

2024 – a story in four acts The end of the year. A good time to reflect on what happened the past twelve months. The ups, the downs. What did we achieve? And what can we look forward to? Some of you have asked to be updated on what we’re working on at OSSO. We’ll gladly share the gist here in this recap, which is in English for once (after four Dutch editions in 2020, 2021, 2022 and 2023).

mysql binlog replay / max_allowed_packet

Trying to replay MySQL binlogs? And running into max_allowed_packet errors? Fear not. Likely this is not corruption, but packets that really are that big. In 2021 I wrote about mariabackup and selective table restore. That blog entry shows how one might restore a mariabackup-saved snapshot of a MariaDB (MySQL) database. Specifically, it focuses on recovering only one or two tables selectively and it details optional GPG decryption and decompression. Here's a recap of the mandatory steps for full recovery and additionally how to replay binlogs to a specific point in time.

mariadb check table / galera locking

After upgrading some database nodes from MariaDB 10.3 to 10.6 we encountered some issues with tables not being fully correct. We'd like to CHECK (and maybe REPAIR) TABLE the entire database. But the database must not block queries on other nodes of the cluster. The corruption we saw might have been a small corruption that had crept in during even earlier upgrades, we don't know. But we do know that we'd like to get this sorted before the corruption gets worse after further upgrades.

systemd-networkd-wait-online / stalling and failing

systemd-networkd-wait-online.service failing? Maybe it's IPv6. Do you have a list of failed systemd units that looks like this? # systemctl list-units --failed UNIT LOAD ACTIVE SUB DESCRIPTION ● systemd-networkd-wait-online.service loaded failed failed Wait for Network to be Configured Then that could be due to IPv6 networking not being set up properly. Check the interfaces list: # ip -br a lo UNKNOWN 127.0.0.1/8 ::1/128 ens18 UP 10.20.30.41/31 fe80::a124:092f:fa18:109e/64 ens19 UP 192.168.1.3/31 fe80::a124:092f:fa15:01d5/64 Proper IPv4 IPs, and only link-local IPs for IPv6?

nmap ssl-enum-ciphers / haproxy / tls / no results

When doing an nmap ssl-enum-ciphers scan on a Haproxy machine, I got zero working ciphers. But connecting with TLS worked just fine. What is up? While updating TLS ciphers to disable certain unsafe ciphers, we wanted to test the current offers. nmap with the ssl-enum-ciphers script should work fine for this, and it is reasonably fast: $ nmap -Pn --script=ssl-enum-ciphers -p 443 google.com ... 443/tcp open https | ssl-enum-ciphers: | TLSv1.

gpg-agent / ssh / ed25519 / agent refused

After putting in all the work to get ED25519 OpenPGP keys in my Yubikey smart card, I was slightly frustrated that the SSH support "sometimes" didn't work. I thought I had tested that it worked, but today it didn't. $ ssh my_server sign_and_send_pubkey: signing failed for ED25519 "cardno:000612345678" from agent: agent refused operation walter@my_server: Permission denied (publickey). That's odd. Adding some debug to the gpg-agent — debug 1024 in gpg-agent.conf — got me this:

recap 2023 - updates van OSSO

2023 – betere omgevingen, betere security Waar we vorig jaar druk bezig zijn geweest legacy op te ruimen, was dit jaar meer een jaar van de uitbreidingen: ArgoCD, NetworkPolicies, Cilium, Loki/Mimir… ContainerDay Security Met het gehele OSSO team zijn we afgelopen maart naar Hamburg geweest, naar ContainerDay Security 2023. Daar hebben we bijgepraat over de huidige stand van zaken op security gebied binnen Kubernetes. Hier kwam het laatste zetje wat we nodig hadden om Cilium te gaan uitproberen voor Kubernetes networking.

mplayer / screen saver / wayland

I don't usually watch movies from my laptop, but when I do, I don't want the screen saver to kick in. Occasionally, I notice that my old time movie player mplayer does not inhibit the screen saver or screen blanking. That means that 5 minutes into a movie, the screen turns black. I don't think this used to be the case. Maybe it's because I'm running the Wayland display server in GNOME now.

bash / postfix health check / dev tcp

Using /dev/tcp in bash for a health check? Here's an example. I had a script that used netcat to connect to a Postfix email daemon to check its health status. To avoid pipelining errors I had it sleep between each write. The core looked somewhat like this: messages=$(for x in \ 'EHLO localhost' \ 'MAIL FROM:<healthz@localhost>' \ 'RCPT TO:<postmaster@example.com>' \ RSET \ QUIT do sleep 0.