V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
watara
V2EX  ›  Docker

奇怪的问题: docker 容器和宿主机物理网卡 IP 不通,求教

  •  
  •   watara · 2023-02-11 18:37:54 +08:00 · 1100 次点击
    这是一个创建于 686 天前的主题,其中的信息可能已经有所发展或是发生改变。

    前置说明

    宿主机系统:CentOS 7.2009
    Docker 版本:20.10.19
    防火墙: 停止并禁用了 firewalld ,并启用了 iptables ,
    物理网卡 IP: 公网 IP (下面用 1.1.1.1 替代)
    docker 网络:172.17.0.0/16 ,宿主机 IP 172.17.0.1 ,容器 IP:172.17.0.2
    其他补充说明:由于架构设计,哪怕在本机容器中,依然需要访问物理网卡的公网 IP 访问才行。

    现象

    1 、在容器中 ping 不通宿主机公网 IP1.1.1.1 ,相应的,业务端口也不通。
    2 、但在容器中可以 ping 通其他任意公网 IP ,包括和宿主机通网段的 1.1.1.2 等,端口访问也正常。
    3 、用 docker 网卡的 IP 进行在容器以及宿主机间互 ping ,都是通的,端口访问也正常。

    测试机复现以及一些想法

    说实话我没能成功复现,在测试环境中发现如果仅仅是容器和真实网卡的 IP 之间的互通而言,和 iptables 的规则关系不大(当然,刻意写规则去禁 ping 啥的那肯定是有影响的),清空了 iptables 规则,甚至停止了 iptables ,网络依然是通的,所以我想问题应该是出在 docker0 这个网卡上,但照着这个思路查了半天也查不出一个所以然,所以特发帖求教。

    一些配置或结果

    brctl show

    brctl show
    bridge name	bridge id		STP enabled	interfaces
    docker0		8000.02423f9b3058	no		veth29942ea
    							veth5726f6f
    							veth7eca3df
    

    docker ntwork ls

    root# docker network ls
    NETWORK ID     NAME      DRIVER    SCOPE
    df8dc4223152   bridge    bridge    local
    dc4beaa42e37   host      host      local
    b9230ceb5861   none      null      local
    

    容器详情(截取了网络相关部分)

    docker inspect 7c395050d09b
    [
    	#省略了一些我认为无用的部分,不然太长了
            "NetworkSettings": {
                "Bridge": "",
                "SandboxID": "801c42feac51b082e9947d86f38175d1c3b5bc6295385b6d06ce6f100be95ddf",
                "HairpinMode": false,
                "LinkLocalIPv6Address": "",
                "LinkLocalIPv6PrefixLen": 0,
                "Ports": {},
                "SandboxKey": "/var/run/docker/netns/801c42feac51",
                "SecondaryIPAddresses": null,
                "SecondaryIPv6Addresses": null,
                "EndpointID": "8318ab5e93f6c4a02e788d300d4c31164c84a9eddc3a870db7bfb788d8e187c9",
                "Gateway": "172.17.0.1",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "IPAddress": "172.17.0.4",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "MacAddress": "02:42:ac:11:00:04",
                "Networks": {
                    "bridge": {
                        "IPAMConfig": null,
                        "Links": null,
                        "Aliases": null,
                        "NetworkID": "df8dc4223152b8fbc3676a4a86aa3c4c0a9f5528a2c3fabc9e74dd0bec5e0b06",
                        "EndpointID": "8318ab5e93f6c4a02e788d300d4c31164c84a9eddc3a870db7bfb788d8e187c9",
                        "Gateway": "172.17.0.1",
                        "IPAddress": "172.17.0.4",
                        "IPPrefixLen": 16,
                        "IPv6Gateway": "",
                        "GlobalIPv6Address": "",
                        "GlobalIPv6PrefixLen": 0,
                        "MacAddress": "02:42:ac:11:00:04",
                        "DriverOpts": null
                    }
                }
            }
        }
    ]
    

    iptables 配置

    # Generated by iptables-save v1.4.21 on Fri Feb 10 18:13:01 2023
    *filter
    :INPUT ACCEPT [8077:11154340]
    :FORWARD ACCEPT [0:0]
    :OUTPUT ACCEPT [20891:12086812]
    :DOCKER - [0:0]
    :DOCKER-ISOLATION-STAGE-1 - [0:0]
    :DOCKER-ISOLATION-STAGE-2 - [0:0]
    :DOCKER-USER - [0:0]
    iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
    iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
    iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
    iptables -A FORWARD -j DOCKER-USER
    iptables -A FORWARD -j DOCKER-ISOLATION-STAGE-1
    iptables -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
    iptables -A FORWARD -o docker0 -j DOCKER
    iptables -A FORWARD -i docker0 ! -o docker0 -j ACCEPT
    iptables -A FORWARD -i docker0 -o docker0 -j ACCEPT
    iptables -A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 9090 -j ACCEPT
    iptables -A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 9000 -j ACCEPT
    iptables -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
    iptables -A DOCKER-ISOLATION-STAGE-1 -j RETURN
    iptables -A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
    iptables -A DOCKER-ISOLATION-STAGE-2 -j RETURN
    iptables -A DOCKER-USER -j RETURN
    COMMIT
    # Completed on Fri Feb 10 18:13:01 2023
    # Generated by iptables-save v1.4.21 on Fri Feb 10 18:13:01 2023
    *nat
    :PREROUTING ACCEPT [6168:356557]
    :INPUT ACCEPT [4862:272405]
    :OUTPUT ACCEPT [4222:245620]
    :POSTROUTING ACCEPT [4223:245672]
    :DOCKER - [0:0]
    iptables -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
    iptables -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
    iptables -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
    iptables -A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 9090 -j MASQUERADE
    iptables -A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 9000 -j MASQUERADE
    iptables -A DOCKER -i docker0 -j RETURN
    iptables -A DOCKER -d 127.0.0.1/32 ! -i docker0 -p tcp -m tcp --dport 9090 -j DNAT --to-destination 172.17.0.2:9090
    iptables -A DOCKER -d 127.0.0.1/32 ! -i docker0 -p tcp -m tcp --dport 9000 -j DNAT --to-destination 172.17.0.2:9000
    COMMIT
    # Completed on Fri Feb 10 18:13:01 2023
    
    4 条回复    2023-02-14 15:46:03 +08:00
    julyclyde
        1
    julyclyde  
       2023-02-13 09:53:54 +08:00
    docker inspect 容器显示其所属 bridge 为
    建议你运行一下
    julyclyde
        2
    julyclyde  
       2023-02-13 09:54:05 +08:00
    docker network inspect df8dc4223152
    看看情况?
    watara
        3
    watara  
    OP
       2023-02-13 11:48:32 +08:00
    @julyclyde #2 多谢帮忙出谋划策,这个我对比看了,好像没啥异常,由于后续重启了 docker ,所以资源 ID 变了,下面是结果:

    ```bash
    docker network inspect dc1dc15bd744
    [
    {
    "Name": "bridge",
    "Id": "dc1dc15bd7448b829dbde584f5b3d6aedbf8b6e14bb4b8b6fcc2dbb80b81ea3f",
    "Created": "2023-02-13T10:01:31.771962878+08:00",
    "Scope": "local",
    "Driver": "bridge",
    "EnableIPv6": false,
    "IPAM": {
    "Driver": "default",
    "Options": null,
    "Config": [
    {
    "Subnet": "172.17.0.0/16",
    "Gateway": "172.17.0.1"
    }
    ]
    },
    "Internal": false,
    "Attachable": false,
    "Ingress": false,
    "ConfigFrom": {
    "Network": ""
    },
    "ConfigOnly": false,
    "Containers": {
    "066344f76f69bee353fe2751383fe9fa7a8ae0815aeec89c837a4795724f48ab": {
    "Name": "apffd_dghg",
    "EndpointID": "8e30360569e438bb9943a8ad542909e8de6e78e66e8cf6f77ac9ea946d265867",
    "MacAddress": "02:42:ac:11:00:02",
    "IPv4Address": "172.17.0.2/16",
    "IPv6Address": ""
    },
    "bddf8b8719d44ff77fe95b33a264852a34795134050fb188913db04883407203": {
    "Name": "sleepy_neumann",
    "EndpointID": "f7ef3cfcb2afdbdea998d64eb750ed0f399e54f5e759869a40318641cf1ae793",
    "MacAddress": "02:42:ac:11:00:03",
    "IPv4Address": "172.17.0.3/16",
    "IPv6Address": ""
    }
    },
    "Options": {
    "com.docker.network.bridge.default_bridge": "true",
    "com.docker.network.bridge.enable_icc": "true",
    "com.docker.network.bridge.enable_ip_masquerade": "true",
    "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
    "com.docker.network.bridge.name": "docker0",
    "com.docker.network.driver.mtu": "1500"
    },
    "Labels": {}
    }
    ]
    ```
    dengkj
        4
    dengkj  
       2023-02-14 15:46:03 +08:00
    应该是系统内核的网桥模块加载失败,升级内核可解决
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   2965 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 23ms · UTC 00:15 · PVG 08:15 · LAX 16:15 · JFK 19:15
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.