两个虚拟机，IP分别是10.10.61.110和10.10.61.109，通过Linux网桥br连接。在110上面重放报文，只有第一个报文能够到达109，其它报文过不去，但是在网桥上能看到。
这里两个抓包文件是同时抓的，tap.pcap抓的是109的网卡，br.pcap抓的是网桥。第一个报文的内容，完全相同。。。
[root@controller ~]# tcpdump -r tap.pcap -nn
reading from file tap.pcap, link-type EN10MB (Ethernet)
20:41:21.854789 IP 10.10.61.110.50780 > 10.10.61.109.9999: Flags [S], seq 2992711880, win 26880, options [mss 8960,sackOK,TS val 60008757 ecr 0,nop,wscale 7], length 0
[root@controller ~]# tcpdump -r br.pcap -nn
reading from file br.pcap, link-type EN10MB (Ethernet)
20:41:21.854737 IP 10.10.61.110.50780 > 10.10.61.109.9999: Flags [S], seq 2992711880, win 26880, options [mss 8960,sackOK,TS val 60008757 ecr 0,nop,wscale 7], length 0
20:41:21.911194 IP 10.10.61.109.9999 > 10.10.61.110.50780: Flags [S.], seq 702486902, ack 2992711881, win 28960, options [mss 1460,sackOK,TS val 63718971 ecr 60008757,nop,wscale 7], length 0
20:41:21.961547 IP 10.10.61.110.50780 > 10.10.61.109.9999: Flags [.], ack 1, win 210, options [nop,nop,TS val 60008757 ecr 63718971], length 0
20:41:22.073722 IP 10.10.61.110.50780 > 10.10.61.109.9999: Flags [P.], seq 1:82, ack 1, win 210, options [nop,nop,TS val 60008757 ecr 63718971], length 81
20:41:22.124028 IP 10.10.61.109.9999 > 10.10.61.110.50780: Flags [.], ack 82, win 227, options [nop,nop,TS val 63718971 ecr 60008757], length 0
20:41:25.407070 IP 10.10.61.109.9999 > 10.10.61.110.50780: Flags [P.], seq 1:4238, ack 82, win 227, options [nop,nop,TS val 63718974 ecr 60008757], length 4237
还有个现象，就是开始重放后，虽然流量非常非常小(10kbps)，但是109竟然网络不通了。包括110和PC，都无法ping通109，但是arping是正常的。停止重放，再等待一段时间，就会恢复。。。
应该不是iptables的问题。我已经把这些报文设置为NOTRACK，并且加了很多日志。报文走到mangle PREROUTING之后就消失了。
被mangle丢了，用iptables-save看看mangle表prerouting hook里有啥。。
|Tcpreplay is a suite of BSD licensed tools written by Aaron Turner for UNIX
|(and Win32 under Cygwin) operating systems which gives you the ability to use
|previously captured traffic in libpcap format to test a variety of network
|devices. It allows you to classify traffic as client or server, rewrite Layer
|2, 3 and 4 headers and finally replay the traffic back onto the network and
|through other devices such as switches, routers, firewalls, NIDS and IPS's.
|Tcpreplay supports both single and dual NIC modes for testing both sniffing
|and inline devices.
*mangle
:PREROUTING ACCEPT [6740335:3576288284]
:INPUT ACCEPT [5985792:3427993224]
:FORWARD ACCEPT [1842759:149468838]
:OUTPUT ACCEPT [6016151:3543145104]
:POSTROUTING ACCEPT [6894064:3654020102]
:neutron-linuxbri-FORWARD - [0:0]
:neutron-linuxbri-INPUT - [0:0]
:neutron-linuxbri-OUTPUT - [0:0]
:neutron-linuxbri-POSTROUTING - [0:0]
:neutron-linuxbri-PREROUTING - [0:0]
:neutron-linuxbri-mark - [0:0]
-A PREROUTING -j neutron-linuxbri-PREROUTING
-A PREROUTING -m mark --mark 0x270f -j LOG --log-prefix "IPT-mangle-PREROUTING "
-A INPUT -j neutron-linuxbri-INPUT
-A INPUT -m mark --mark 0x270f -j LOG --log-prefix "IPT-mangle-INPUT "
-A FORWARD -j neutron-linuxbri-FORWARD
-A FORWARD -m mark --mark 0x270f -j LOG --log-prefix "IPT-mangle-FORWARD "
-A OUTPUT -j neutron-linuxbri-OUTPUT
-A OUTPUT -m mark --mark 0x270f -j LOG --log-prefix "IPT-mangle-OUTPUT "
-A POSTROUTING -j neutron-linuxbri-POSTROUTING
-A POSTROUTING -m mark --mark 0x270f -j LOG --log-prefix "IPT-mangle-POSTROUTING "
-A neutron-linuxbri-PREROUTING -j neutron-linuxbri-mark
COMMIT
[2134533.682832] EBT-broute-BROUTING IN=tap8607eae3-19 OUT= MAC source = fa:16:3e:9e:1a:0e MAC dest = fa:16:3e:8f:bb:d7 proto = 0x0800 IP SRC=10.10.61.110 IP DST=10.10.61.109, IP tos=0x00, IP proto=6 SPT=50820 DPT=9999
[2134533.682838] EBT-nat-PREROUTING IN=tap8607eae3-19 OUT= MAC source = fa:16:3e:9e:1a:0e MAC dest = fa:16:3e:8f:bb:d7 proto = 0x0800 IP SRC=10.10.61.110 IP DST=10.10.61.109, IP tos=0x00, IP proto=6 SPT=50820 DPT=9999
[2134533.682855] IPT-mangle-PREROUTING IN=brqb193c910-f5 OUT= PHYSIN=tap8607eae3-19 MAC=fa:16:3e:8f:bb:d7:fa:16:3e:9e:1a:0e:08:00 SRC=10.10.61.110 DST=10.10.61.109 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=43626 DF PROTO=TCP SPT=50820 DPT=9999 WINDOW=243 RES=0x00 ACK URGP=0 MARK=0x270f
[2134534.266386] IN=brqb193c910-f5 OUT=brqb193c910-f5 PHYSIN=enp3s0f0 PHYSOUT=tap5d155823-40 MAC=01:00:5e:00:00:12:00:00:5e:00:01:6f:08:00 SRC=10.10.61.251 DST=224.0.0.18 LEN=40 TOS=0x00 PREC=0x00 TTL=255 ID=6415 PROTO=112
mangle表里面大多是LOG，打印出来的包，只到了PREROUTING，后面就不见了。看了iptables的计数，没有找到是被哪个规则丢包的。
iptables计数里有的话，要么是包不完整(例如tcpreplay工具有问题，只有第一个包完整)，要么是iptable bug，第一种可能性较大。