How Promtail ships container logs to Loki across an IPsec split tunnel

Short answer: Do not open a remote Docker socket to read container stdout. Promtail on the remote host tails the local socket and pushes to Loki. If that host’s IPsec policy can reach only one virtual address, and Loki is bound to a different virtual address on the same machine, push to the address the tunnel allows and DNAT port 3100 onto Loki. Loki uses the host network so it can bind that address. The viewer then reads Loki. That is the design, not a temporary bypass.

What Promtail sends

Promtail sends stdout and stderr. Docker stores those lines on the container host. Promtail reads new lines through the local Docker API and pushes them to Loki. Other machines read that copy.

/var/run/docker.sock is opened only on that host. It is the engine’s control API: the same socket that reads logs can start, stop, and create containers, and run commands inside them. Opening that socket on another host, or publishing it as a TCP port across the VPN, is a security risk. The far side then controls the engine. Promtail opens the socket locally, read-only, and sends only the log lines.

A viewer already on the Docker host reads the socket itself. Every other host runs Promtail.

What a split tunnel will forward

Picture the Loki host with interface br0, and two virtual addresses on it. The numbers are examples.

10.11.0.1/32 is tunnel A. The policy subnet is 10.11.0.0/24. Loki binds 10.11.0.1:3100.
10.22.0.1/32 is tunnel B. The policy subnet is 10.22.0.0/24.

A packet enters a tunnel only when its destination is inside that tunnel’s subnet. A peer of tunnel B may send to 10.22.0.0/24. A connection to 10.11.0.1:3100 does not match that policy, so it times out. Both addresses are on br0. Only one of them is inside this peer’s policy.

Promtail with network_mode: host uses that routing table. It can open 10.22.0.1:3100. It cannot open 10.11.0.1:3100.

The working path

Promtail, host network, Docker socket read-only, docker_sd_configs on unix:///var/run/docker.sock. Relabel the container name and set a host label.
Push URL http://10.22.0.1:3100/loki/api/v1/push — the virtual address the peer policy already allows.
On the Loki host, nat PREROUTING DNATs TCP 10.22.0.1:3100 to 10.11.0.1:3100, the address in Loki’s http_listen_address.
Loki uses network_mode: host and binds 10.11.0.1, so the rewritten packet is delivered locally.
Conntrack restores the reply source to 10.22.0.1, which is what the IPsec policy matches on the way back.
On the Loki host, the firewall accepts TCP 3100 from the Promtail peer.
The viewer calls Loki query_range with {host="…",container="…"}, built by pasting the requested container name into the query. Accept only a plain container name (letters, digits, ., _, -) before that paste. A quote or a brace in the name would change which streams Loki returns.

Stamp a Promtail position for each running container id at “now” before the first start, and leave an existing cursor alone. The docker target’s key is cursor-<container id>. A missing cursor reads the json log from the start (Since=0). Stop Promtail before writing the positions file, or its flush replaces the keys. Afterward a container appears once it writes a new line.

Daemons that log only to files are a different pipeline. This path is stdout and stderr.

Approaches that look simpler and fail

Approach	Why it fails
Publish the Docker socket, or Docker’s TCP API, across the VPN	The socket is the engine control API. The far side can start, stop, and create containers.
Bind Loki to 0.0.0.0 while authentication is disabled	Loki then answers on every address of the host, including the LAN address. A client that can reach port 3100 can query and push.
Add `10.11.0.0/24` to the Promtail peer’s IPsec policy	The peer can then reach that whole subnet, not only the one port forwarded to Loki.
Point Promtail at `10.11.0.1:3100`	That address is outside `10.22.0.0/24`. The packet never enters the tunnel and times out.
Start Promtail with an empty positions file	With no cursor, the docker target reads each container’s json log from the start and ships the backlog.

A login proxy in front of Loki protects browsers that query it. It does not give Promtail a route to 10.11.0.1.

When you can drop the DNAT

Drop the rewrite when one of these is true:

The peer policy already includes the subnet Loki binds.
You move http_listen_address onto the VIP the peer can already reach, and that address is an acceptable ingest endpoint.
You stop shipping remote stdout and only read the host that holds the socket.

Those are policy or bind-address changes, not a hidden Promtail switch. Keep the DNAT while Loki has to stay on the other virtual address.

Why this stays published

We hit this on a homelab where one Loki serves several IPsec peers and each peer policy is a single virtual subnet. Promtail pushes to the VIP the tunnel allows; DNAT on the Loki host hands that port to the address Loki binds, and the host network is what makes the bind possible.

本文由 HoHo 與 AI 協作整理，最後更新於 2026 年 10 月 3 日。