{"id":32,"date":"2026-10-03T11:01:44","date_gmt":"2026-10-03T11:01:44","guid":{"rendered":"https:\/\/hoho.live\/luda\/index.php\/2026\/10\/03\/how-promtail-ships-container-logs-to-loki-across-an-ipsec-split-tunnel\/"},"modified":"2026-10-04T09:48:23","modified_gmt":"2026-10-04T01:48:23","slug":"how-promtail-ships-container-logs-to-loki-across-an-ipsec-split-tunnel","status":"publish","type":"post","link":"https:\/\/hoho.live\/luda\/index.php\/2026\/10\/03\/how-promtail-ships-container-logs-to-loki-across-an-ipsec-split-tunnel\/","title":{"rendered":"How Promtail ships container logs to Loki across an IPsec split tunnel"},"content":{"rendered":"\n

Short answer:<\/strong> Do not open a remote Docker socket to read container stdout. Promtail<\/strong> on the remote host tails the local socket and pushes<\/strong> to Loki<\/strong>. If that host’s IPsec policy can reach only one virtual address, and Loki is bound to a different virtual address on the same machine, push to the address the tunnel allows and DNAT<\/strong> port 3100 onto Loki. Loki uses the host network<\/strong> so it can bind that address. The viewer then reads Loki. That is the design, not a temporary bypass.<\/p>\n\n\n

\n\n

What Promtail sends<\/strong><\/h4>\n\n

Promtail sends stdout and stderr. Docker stores those lines on the container host. Promtail reads new lines through the local Docker API and pushes them to Loki. Other machines read that copy.<\/p>\n\n\n

\/var\/run\/docker.sock<\/code> is opened only on that host. It is the engine’s control API: the same socket that reads logs can start, stop, and create containers, and run commands inside them. Opening that socket on another host, or publishing it as a TCP port across the VPN, is a security risk. The far side then controls the engine. Promtail opens the socket locally, read-only, and sends only the log lines.<\/p>\n\n\n

A viewer already on the Docker host reads the socket itself. Every other host runs Promtail.<\/p>\n\n\n

\n\n

What a split tunnel will forward<\/strong><\/h4>\n\n

Picture the Loki host with interface br0<\/code>, and two virtual addresses on it. The numbers are examples.<\/p>\n\n\n

    \n
  • 10.11.0.1\/32<\/code> is tunnel A. The policy subnet is 10.11.0.0\/24<\/code>. Loki binds 10.11.0.1:3100<\/code>.<\/li>\n
  • 10.22.0.1\/32<\/code> is tunnel B. The policy subnet is 10.22.0.0\/24<\/code>.<\/li>\n<\/ul>\n\n\n

    A packet enters a tunnel only when its destination is inside that tunnel’s subnet. A peer of tunnel B may send to 10.22.0.0\/24<\/code>. A connection to 10.11.0.1:3100<\/code> does not match that policy, so it times out. Both addresses are on br0<\/code>. Only one of them is inside this peer’s policy.<\/p>\n\n\n

    Promtail with network_mode: host<\/code> uses that routing table. It can open 10.22.0.1:3100<\/code>. It cannot open 10.11.0.1:3100<\/code>.<\/p>\n\n\n

    \n\n

    The working path<\/strong><\/h4>\n\n
      \n
    1. Promtail, host network, Docker socket read-only, docker_sd_configs<\/code> on unix:\/\/\/var\/run\/docker.sock<\/code>. Relabel the container name and set a host<\/code> label.<\/li>\n
    2. Push URL http:\/\/10.22.0.1:3100\/loki\/api\/v1\/push<\/code> \u2014 the virtual address the peer policy already allows.<\/li>\n
    3. On the Loki host, nat PREROUTING DNATs TCP 10.22.0.1:3100<\/code> to 10.11.0.1:3100<\/code>, the address in Loki’s http_listen_address<\/code>.<\/li>\n
    4. Loki uses network_mode: host<\/code> and binds 10.11.0.1<\/code>, so the rewritten packet is delivered locally.<\/li>\n
    5. Conntrack restores the reply source to 10.22.0.1<\/code>, which is what the IPsec policy matches on the way back.<\/li>\n
    6. On the Loki host, the firewall accepts TCP 3100 from the Promtail peer.<\/li>\n
    7. The viewer calls Loki query_range<\/code> with {host=\"\u2026\",container=\"\u2026\"}<\/code>, built by pasting the requested container name into the query. Accept only a plain container name (letters, digits, .<\/code>, _<\/code>, -<\/code>) before that paste. A quote or a brace in the name would change which streams Loki returns.<\/li>\n<\/ol>\n\n\n

      Stamp a Promtail position for each running container id at “now” before the first start, and leave an existing cursor alone. The docker target’s key is cursor-<container id><\/code>. A missing cursor reads the json log from the start (Since=0<\/code>). Stop Promtail before writing the positions file, or its flush replaces the keys. Afterward a container appears once it writes a new line.<\/p>\n\n\n

      Daemons that log only to files are a different pipeline. This path is stdout and stderr.<\/p>\n\n\n

      \n\n

      Approaches that look simpler and fail<\/strong><\/h4>\n\n
      \n\n\n\n\n\n\n\n\n\n
      Approach<\/th>\nWhy it fails<\/th>\n<\/tr>\n<\/thead>\n
      Publish the Docker socket, or Docker’s TCP API, across the VPN<\/td>\nThe socket is the engine control API. The far side can start, stop, and create containers.<\/td>\n<\/tr>\n
      Bind Loki to 0.0.0.0 while authentication is disabled<\/td>\nLoki then answers on every address of the host, including the LAN address. A client that can reach port 3100 can query and push.<\/td>\n<\/tr>\n
      Add 10.11.0.0\/24<\/code> to the Promtail peer’s IPsec policy<\/td>\nThe peer can then reach that whole subnet, not only the one port forwarded to Loki.<\/td>\n<\/tr>\n
      Point Promtail at 10.11.0.1:3100<\/code><\/td>\nThat address is outside 10.22.0.0\/24<\/code>. The packet never enters the tunnel and times out.<\/td>\n<\/tr>\n
      Start Promtail with an empty positions file<\/td>\nWith no cursor, the docker target reads each container’s json log from the start and ships the backlog.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n\n\n

      A login proxy in front of Loki protects browsers that query it. It does not give Promtail a route to 10.11.0.1<\/code>.<\/p>\n\n\n

      \n\n

      When you can drop the DNAT<\/strong><\/h4>\n\n

      Drop the rewrite when one of these is true:<\/p>\n\n\n

        \n
      • The peer policy already includes the subnet Loki binds.<\/li>\n
      • You move http_listen_address<\/code> onto the VIP the peer can already reach, and that address is an acceptable ingest endpoint.<\/li>\n
      • You stop shipping remote stdout and only read the host that holds the socket.<\/li>\n<\/ul>\n\n\n

        Those are policy or bind-address changes, not a hidden Promtail switch. Keep the DNAT while Loki has to stay on the other virtual address.<\/p>\n\n\n

        \n\n

        Why this stays published<\/strong><\/h4>\n\n

        We hit this on a homelab where one Loki serves several IPsec peers and each peer policy is a single virtual subnet. Promtail pushes to the VIP the tunnel allows; DNAT on the Loki host hands that port to the address Loki binds, and the host network is what makes the bind possible.<\/strong><\/p>\n\n\n

        \u672c\u6587\u7531 HoHo \u8207 AI \u5354\u4f5c\u6574\u7406\uff0c\u6700\u5f8c\u66f4\u65b0\u65bc 2026 \u5e74 10 \u6708 3 \u65e5\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"

        Short answer: Do not open a remote Docker socket to read container stdout. Promtail on the remote host tails the local socket and pushes to Loki. If that host’s IPsec policy can reach only one virtual address, and Loki is bound to a different virtual address on the same…<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7,8,17],"tags":[15,16,14,13],"class_list":["post-32","post","type-post","status-publish","format-standard","hentry","category-homelab","category-ipsec","category-logging","tag-docker","tag-ipsec","tag-loki","tag-promtail"],"_links":{"self":[{"href":"https:\/\/hoho.live\/luda\/index.php\/wp-json\/wp\/v2\/posts\/32","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hoho.live\/luda\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hoho.live\/luda\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hoho.live\/luda\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/hoho.live\/luda\/index.php\/wp-json\/wp\/v2\/comments?post=32"}],"version-history":[{"count":7,"href":"https:\/\/hoho.live\/luda\/index.php\/wp-json\/wp\/v2\/posts\/32\/revisions"}],"predecessor-version":[{"id":50,"href":"https:\/\/hoho.live\/luda\/index.php\/wp-json\/wp\/v2\/posts\/32\/revisions\/50"}],"wp:attachment":[{"href":"https:\/\/hoho.live\/luda\/index.php\/wp-json\/wp\/v2\/media?parent=32"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hoho.live\/luda\/index.php\/wp-json\/wp\/v2\/categories?post=32"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hoho.live\/luda\/index.php\/wp-json\/wp\/v2\/tags?post=32"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}