万兆网卡 RX DROP 率近 33% 求帮忙分析:

2016-05-26 17:22:33 +08:00
 yuedingwangji

昨天发了个主题 http://v2ex.com/t/281204 ,谢谢 v 友的帮助,按照提示,重新更新了驱动,调整了队列,今天 google 了很多, 还把 rx ring buffer 的缓冲区调大到最大了! 但是今天看了好像丢包还是很严重,算了一下, 丢包率有 33 ,感觉没有什么变化,求大神指点,帮忙分析分析,下面贴出网卡的信息! ####目前最新的驱动: [root@CAIJI ~]# ethtool -i p2p1 driver: ixgbe version: 4.3.15 firmware-version: 0x2b2c0001, 255.65535.255 bus-info: 0000:43:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: no ####网卡信息 root@CAIJI ~]# ifconfig p2p1 p2p1 Link encap:Ethernet HWaddr 00:16:31:F6:00:F4
inet6 addr: fe80::216:31ff:fef6:f4/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1 RX packets:17199319458 errors:0 dropped:5375941950 overruns:0 frame:0 TX packets:83 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:3000 RX bytes:5909371626285 (5.3 TiB) TX bytes:20466 (19.9 KiB)

最后一个很长, 看不懂

##网卡统计信息 root@CAIJI ~]# ethtool -S p2p1 NIC statistics: rx_packets: 17029710081 tx_packets: 83 rx_bytes: 5852566332178 tx_bytes: 20466 rx_errors: 0 tx_errors: 0 rx_dropped: 0 tx_dropped: 0 multicast: 0 collisions: 0 rx_over_errors: 0 rx_crc_errors: 0 rx_frame_errors: 0 rx_fifo_errors: 0 rx_missed_errors: 5319065929 tx_aborted_errors: 0 tx_carrier_errors: 0 tx_fifo_errors: 0 tx_heartbeat_errors: 0 rx_pkts_nic: 17027492035 tx_pkts_nic: 83 rx_bytes_nic: 7894510228645 tx_bytes_nic: 20798 lsc_int: 12 tx_busy: 0 non_eop_descs: 0 broadcast: 0 rx_no_buffer_count: 0 tx_timeout_count: 0 tx_restart_queue: 0 rx_long_length_errors: 0 rx_short_length_errors: 0 tx_flow_control_xon: 553080 rx_flow_control_xon: 0 tx_flow_control_xoff: 8405544 rx_flow_control_xoff: 0 rx_csum_offload_errors: 42680 alloc_rx_page_failed: 0 alloc_rx_buff_failed: 0 lro_aggregated: 0 lro_flushed: 0 rx_no_dma_resources: 0 hw_rsc_aggregated: 0 hw_rsc_flushed: 0 fdir_match: 0 fdir_miss: 17404483615 fdir_overflow: 0 fcoe_bad_fccrc: 0 fcoe_last_errors: 0 rx_fcoe_dropped: 0 rx_fcoe_packets: 0 rx_fcoe_dwords: 0 fcoe_noddp: 0 fcoe_noddp_ext_buff: 0 tx_fcoe_packets: 0 tx_fcoe_dwords: 0 os2bmc_rx_by_bmc: 0 os2bmc_tx_by_bmc: 0 os2bmc_tx_by_host: 0 os2bmc_rx_by_host: 0 tx_hwtstamp_timeouts: 0 rx_hwtstamp_cleared: 0 tx_queue_0_packets: 6 tx_queue_0_bytes: 2052 tx_queue_1_packets: 0 tx_queue_1_bytes: 0 tx_queue_2_packets: 0 tx_queue_2_bytes: 0 tx_queue_3_packets: 0 tx_queue_3_bytes: 0 tx_queue_4_packets: 15 tx_queue_4_bytes: 5130 tx_queue_5_packets: 0 tx_queue_5_bytes: 0 tx_queue_6_packets: 6 tx_queue_6_bytes: 2052 tx_queue_7_packets: 7 tx_queue_7_bytes: 2394 tx_queue_8_packets: 0 tx_queue_8_bytes: 0 tx_queue_9_packets: 0 tx_queue_9_bytes: 0 tx_queue_10_packets: 0 tx_queue_10_bytes: 0 tx_queue_11_packets: 0 tx_queue_11_bytes: 0 tx_queue_12_packets: 5 tx_queue_12_bytes: 1710 tx_queue_13_packets: 0 tx_queue_13_bytes: 0 tx_queue_14_packets: 6 tx_queue_14_bytes: 2052 tx_queue_15_packets: 4 tx_queue_15_bytes: 1368 tx_queue_16_packets: 0 tx_queue_16_bytes: 0 tx_queue_17_packets: 0 tx_queue_17_bytes: 0 tx_queue_18_packets: 0 tx_queue_18_bytes: 0 tx_queue_19_packets: 0 tx_queue_19_bytes: 0 tx_queue_20_packets: 0 tx_queue_20_bytes: 0 tx_queue_21_packets: 0 tx_queue_21_bytes: 0 tx_queue_22_packets: 0 tx_queue_22_bytes: 0 tx_queue_23_packets: 0 tx_queue_23_bytes: 0 tx_queue_24_packets: 0 tx_queue_24_bytes: 0 tx_queue_25_packets: 0 tx_queue_25_bytes: 0 tx_queue_26_packets: 0 tx_queue_26_bytes: 0 tx_queue_27_packets: 0 tx_queue_27_bytes: 0 tx_queue_28_packets: 0 tx_queue_28_bytes: 0 tx_queue_29_packets: 0 tx_queue_29_bytes: 0 tx_queue_30_packets: 0 tx_queue_30_bytes: 0 tx_queue_31_packets: 0 tx_queue_31_bytes: 0 tx_queue_32_packets: 0 tx_queue_32_bytes: 0 tx_queue_33_packets: 0 tx_queue_33_bytes: 0 tx_queue_34_packets: 0 tx_queue_34_bytes: 0 tx_queue_35_packets: 0 tx_queue_35_bytes: 0 tx_queue_36_packets: 12 tx_queue_36_bytes: 936 tx_queue_37_packets: 0 tx_queue_37_bytes: 0 tx_queue_38_packets: 6 tx_queue_38_bytes: 468 tx_queue_39_packets: 10 tx_queue_39_bytes: 1836 tx_queue_40_packets: 0 tx_queue_40_bytes: 0 tx_queue_41_packets: 0 tx_queue_41_bytes: 0 tx_queue_42_packets: 0 tx_queue_42_bytes: 0 tx_queue_43_packets: 0 tx_queue_43_bytes: 0 tx_queue_44_packets: 0 tx_queue_44_bytes: 0 tx_queue_45_packets: 6 tx_queue_45_bytes: 468 tx_queue_46_packets: 0 tx_queue_46_bytes: 0 tx_queue_47_packets: 0 tx_queue_47_bytes: 0 tx_queue_48_packets: 0 tx_queue_48_bytes: 0 tx_queue_49_packets: 0 tx_queue_49_bytes: 0 tx_queue_50_packets: 0 tx_queue_50_bytes: 0 tx_queue_51_packets: 0 tx_queue_51_bytes: 0 tx_queue_52_packets: 0 tx_queue_52_bytes: 0 tx_queue_53_packets: 0 tx_queue_53_bytes: 0 tx_queue_54_packets: 0 tx_queue_54_bytes: 0 tx_queue_55_packets: 0 tx_queue_55_bytes: 0 tx_queue_56_packets: 0 tx_queue_56_bytes: 0 tx_queue_57_packets: 0 tx_queue_57_bytes: 0 tx_queue_58_packets: 0 tx_queue_58_bytes: 0 tx_queue_59_packets: 0 tx_queue_59_bytes: 0 tx_queue_60_packets: 0 tx_queue_60_bytes: 0 tx_queue_61_packets: 0 tx_queue_61_bytes: 0 tx_queue_62_packets: 0 tx_queue_62_bytes: 0 tx_queue_63_packets: 0 tx_queue_63_bytes: 0 tx_queue_64_packets: 0 tx_queue_64_bytes: 0 tx_queue_65_packets: 0 tx_queue_65_bytes: 0 tx_queue_66_packets: 0 tx_queue_66_bytes: 0 tx_queue_67_packets: 0 tx_queue_67_bytes: 0 tx_queue_68_packets: 0 tx_queue_68_bytes: 0 tx_queue_69_packets: 0 tx_queue_69_bytes: 0 tx_queue_70_packets: 0 tx_queue_70_bytes: 0 rx_queue_0_packets: 1067946600 rx_queue_0_bytes: 371301786109 rx_queue_1_packets: 1061034462 rx_queue_1_bytes: 358712868901 rx_queue_2_packets: 1062751875 rx_queue_2_bytes: 365449065361 rx_queue_3_packets: 1055073987 rx_queue_3_bytes: 363927816936 rx_queue_4_packets: 1074608685 rx_queue_4_bytes: 366905950875 rx_queue_5_packets: 1066423484 rx_queue_5_bytes: 363329617576 rx_queue_6_packets: 1060550322 rx_queue_6_bytes: 365891956546 rx_queue_7_packets: 1068063494 rx_queue_7_bytes: 363372252208 rx_queue_8_packets: 1067095208 rx_queue_8_bytes: 367530097930 rx_queue_9_packets: 1061952300 rx_queue_9_bytes: 361989443609 rx_queue_10_packets: 1059287377 rx_queue_10_bytes: 361914107474 rx_queue_11_packets: 1062221260 rx_queue_11_bytes: 370078112692 rx_queue_12_packets: 1061248369 rx_queue_12_bytes: 367898254977 rx_queue_13_packets: 1062558031 rx_queue_13_bytes: 365364492171 rx_queue_14_packets: 1072149893 rx_queue_14_bytes: 374050936073 rx_queue_15_packets: 1066744879 rx_queue_15_bytes: 364849660661 rx_queue_16_packets: 0 rx_queue_16_bytes: 0 rx_queue_17_packets: 0 rx_queue_17_bytes: 0 rx_queue_18_packets: 0 rx_queue_18_bytes: 0 rx_queue_19_packets: 0 rx_queue_19_bytes: 0 rx_queue_20_packets: 0 rx_queue_20_bytes: 0 rx_queue_21_packets: 0 rx_queue_21_bytes: 0 rx_queue_22_packets: 0 rx_queue_22_bytes: 0 rx_queue_23_packets: 0 rx_queue_23_bytes: 0 rx_queue_24_packets: 0 rx_queue_24_bytes: 0 rx_queue_25_packets: 0 rx_queue_25_bytes: 0 rx_queue_26_packets: 0 rx_queue_26_bytes: 0 rx_queue_27_packets: 0 rx_queue_27_bytes: 0 rx_queue_28_packets: 0 rx_queue_28_bytes: 0 rx_queue_29_packets: 0 rx_queue_29_bytes: 0 rx_queue_30_packets: 0 rx_queue_30_bytes: 0 rx_queue_31_packets: 0 rx_queue_31_bytes: 0 rx_queue_32_packets: 0 rx_queue_32_bytes: 0 rx_queue_33_packets: 0 rx_queue_33_bytes: 0 rx_queue_34_packets: 0 rx_queue_34_bytes: 0 rx_queue_35_packets: 0 rx_queue_35_bytes: 0 rx_queue_36_packets: 0 rx_queue_36_bytes: 0 rx_queue_37_packets: 0 rx_queue_37_bytes: 0 rx_queue_38_packets: 0 rx_queue_38_bytes: 0 rx_queue_39_packets: 0 rx_queue_39_bytes: 0 rx_queue_40_packets: 0 rx_queue_40_bytes: 0 rx_queue_41_packets: 0 rx_queue_41_bytes: 0 rx_queue_42_packets: 0 rx_queue_42_bytes: 0 rx_queue_43_packets: 0 rx_queue_43_bytes: 0 rx_queue_44_packets: 0 rx_queue_44_bytes: 0 rx_queue_45_packets: 0 rx_queue_45_bytes: 0 rx_queue_46_packets: 0 rx_queue_46_bytes: 0 rx_queue_47_packets: 0 rx_queue_47_bytes: 0 rx_queue_48_packets: 0 rx_queue_48_bytes: 0 rx_queue_49_packets: 0 rx_queue_49_bytes: 0 rx_queue_50_packets: 0 rx_queue_50_bytes: 0 rx_queue_51_packets: 0 rx_queue_51_bytes: 0 rx_queue_52_packets: 0 rx_queue_52_bytes: 0 rx_queue_53_packets: 0 rx_queue_53_bytes: 0 rx_queue_54_packets: 0 rx_queue_54_bytes: 0 rx_queue_55_packets: 0 rx_queue_55_bytes: 0 rx_queue_56_packets: 0 rx_queue_56_bytes: 0 rx_queue_57_packets: 0 rx_queue_57_bytes: 0 rx_queue_58_packets: 0 rx_queue_58_bytes: 0 rx_queue_59_packets: 0 rx_queue_59_bytes: 0 rx_queue_60_packets: 0 rx_queue_60_bytes: 0 rx_queue_61_packets: 0 rx_queue_61_bytes: 0 rx_queue_62_packets: 0 rx_queue_62_bytes: 0 rx_queue_63_packets: 0 rx_queue_63_bytes: 0 rx_queue_64_packets: 0 rx_queue_64_bytes: 0 rx_queue_65_packets: 0 rx_queue_65_bytes: 0 rx_queue_66_packets: 0 rx_queue_66_bytes: 0 rx_queue_67_packets: 0 rx_queue_67_bytes: 0 rx_queue_68_packets: 0 rx_queue_68_bytes: 0 rx_queue_69_packets: 0 rx_queue_69_bytes: 0 rx_queue_70_packets: 0 rx_queue_70_bytes: 0 tx_pb_0_pxon: 0 tx_pb_0_pxoff: 0 tx_pb_1_pxon: 0 tx_pb_1_pxoff: 0 tx_pb_2_pxon: 0 tx_pb_2_pxoff: 0 tx_pb_3_pxon: 0 tx_pb_3_pxoff: 0 tx_pb_4_pxon: 0 tx_pb_4_pxoff: 0 tx_pb_5_pxon: 0 tx_pb_5_pxoff: 0 tx_pb_6_pxon: 0 tx_pb_6_pxoff: 0 tx_pb_7_pxon: 0 tx_pb_7_pxoff: 0 rx_pb_0_pxon: 0 rx_pb_0_pxoff: 0 rx_pb_1_pxon: 0 rx_pb_1_pxoff: 0 rx_pb_2_pxon: 0 rx_pb_2_pxoff: 0 rx_pb_3_pxon: 0 rx_pb_3_pxoff: 0 rx_pb_4_pxon: 0 rx_pb_4_pxoff: 0 rx_pb_5_pxon: 0 rx_pb_5_pxoff: 0 rx_pb_6_pxon: 0 rx_pb_6_pxoff: 0 rx_pb_7_pxon: 0 rx_pb_7_pxoff: 0

15172 次点击
所在节点    DevOps
33 条回复
redsonic
2016-05-26 19:20:38 +08:00
根据 ethtool 看,调 ring buffer 没用,因为在进入 ring buffer 前就丢了,这样原因就复杂了,有可能是硬件中断问题或 PCIE 带宽不够。 LZ 的机器是品牌服务器还是 DIY 的? 首先确保用的不是寨卡,如果是品牌服务器一定要用厂商指定的卡。然后用 lscpi -s 43:00.0 -vvv 检查 pcie 带宽,万兆至少要用 pcie 4x 。除此之外 ixgbe 还有几个中断相关的参数(内核自带的没有),其中 InterruptThrottleRate 是中断频度,如果太少网卡内部 buffer 会丢包,太多的话 cpu 负载会升高,默认好像自适应,你可以适当调高看看丢包是否有改善。
yuedingwangji
2016-05-26 23:29:49 +08:00
@redsonic 43:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) 这是网卡的信息 因特尔的 万兆网卡,不是山寨卡, 不过我今天听同事说这个网卡的光模块是另外采购的,不是原厂的,不知道会不会跟这个有关系,另外我用 “ lspci -s 43:00.0 -vvv ” 这个命令,输出的结果中没有找到 peie 的, 输出一大堆,前面都是 ISIS ,后面如下
iSiSiSiSiSi
Unknown small resource type 0a, will not decode more.
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] Device Serial Number 00-1b-21-ff-ff-ba-98-e0
Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 64, Function Dependency Link: 00
VF offset: 128, stride: 2, Device ID: 10ed
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 00000000d4080000 (64-bit, non-prefetchable)
Region 3: Memory at 00000000d4180000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Kernel driver in use: ixgbe

另外,我想请问如何查看网卡的中断频段 ( InterruptThrottleRate ),还有一个问题请教,就是为什么我在 /proc/interrupts 中看到的队列是 64 ,而在 ethtols -S 看到的却是 70 ?
yuedingwangji
2016-05-26 23:30:32 +08:00
@redsonic 对了 ,不知道放不方便留个 QQ 交流一下
menory
2016-05-27 08:07:48 +08:00
万兆模块还是有差别的。贵的几千,便宜的几百,几百的和几千的能一样用?
redsonic
2016-05-27 08:48:57 +08:00
@yuedingwangji @menory 82599 好像最大队列数就是 70 ,如果 cpu 核心数超过 70 也就只有 70 个队列,不开 RPS 的话其他 cpu 会闲置,少于 70 个核心队列数应该会自动调整缩减。你那个实际只有 14 个队列有收到包,这个是网卡根据源 ip hash 的结果。
光模块不用原装的等于找死啊,遇到过用国内牌子 sfp+插上去导致网卡随机 down 的情况,造成的损失够买一车原装光模块了。所以光模块不靠谱什么奇怪问题都可能遇到,找个原装换上去看看先,另外可以把万兆卡换一个 pcie 插槽试试。
yuedingwangji
2016-05-27 09:23:18 +08:00
@redsonic 服务器现在放在移动机房, 老板要我搞定这个丢包的问题,我怀疑是光模块的问题,但又找不出其他可靠的证据
yuedingwangji
2016-05-27 09:26:32 +08:00
@redsonic RPS 是什么 ,怎么看有没有开?
redsonic
2016-05-27 10:07:05 +08:00
@yuedingwangji RPS 是针对多 cpu 但队列少的情况,用来对付软中断负载不均的情况和你现在的问题没有关系。 随便找了个: http://blog.chinaunix.net/uid-20788636-id-4838269.html
另外你参考这里的 http://blog.csdn.net/cd520yy/article/details/8507633 ,把 RSS 关掉,也就是只跑一个队列试试看。
最后有个说法可以回应给老板,裸跑原来千兆不丢包, cpu 不差也不要求跑线速,现在丢包能想到的恐怕只有网卡和 sfp 了。

不过我也觉得很奇怪,除掉 sfp 的疑点,如果服务器是 dell 、 hp 那些大厂的加上官方认证的网卡,裸跑丢包 ethtool -S 看到有 rx_missed_errors 不是应该找经销商维保吗?
xencdn
2016-05-27 11:05:35 +08:00
tcpdump 把包抓下来分析下 掉包 33% 还有是怎么测试出来的? 如果生产环境的服务器这么高的掉包应果断下线服务器
innoink
2016-05-27 11:15:14 +08:00
rx_missed ,一般是因为数据在网卡缓冲区停留过久导致溢出, MPC 寄存器数值增加。通常是上层数据处理能力不够,或者网卡 pci-e 插槽速率不够(比如 x8 插到 x4 上)

以前在搞 dpdk 的时候遇到过, tcpdump 我没用过
innoink
2016-05-27 11:17:03 +08:00
万兆网卡如果没有专门软件支持一般不能发挥到全部性能,内核协议栈性能很弱的,应对大流量一般会用 PF_RING/ZC , DPDK 等
yuedingwangji
2016-05-27 14:25:30 +08:00
@redsonic 谢谢 , CPU 是 inter I7 我 64 核 120G 内存 服务器是 dell 的 R820 ,
yuedingwangji
2016-05-27 14:27:18 +08:00
@innoink 你说上层数据处理能力不够? 上次是接的是核心路由器了, 至于网卡插槽我没见过,我也不知道他是怎么插的,服务器是 dell 的 R820 ,买的时候就已经佩戴网卡了
yuedingwangji
2016-05-27 14:42:15 +08:00
@redsonic 可以远程帮我看下么?
redsonic
2016-05-27 15:12:57 +08:00
服务器用 I7 ? 应该是志强 e3 e5 吧。
不是上层处理能力不够,否则的话 rx_fifo_errors , rx_no_buffer_count 不可能都为 0 。只有小包才对协议栈有压力,如果上面是核心路由器肯定有 vlan tag , vpn 包头之类的,实际收到的包应该有 200 字节了吧,裸跑抓包禁止输出肯定不会有压力了。 设备是谁上的,如果让对方知道裸跑抓包出现这样的问题对方应该是推不掉责任的。我印象中像这类问题都是谁上架谁测试有问题也是运维和设备商一块搞定,哪能推到后面去。
yuedingwangji
2016-05-27 16:13:59 +08:00
@redsonic 是 E5 的 CPU , 刚才重启一下机器, 禁用了一下网卡,禁用了 RSS 队列 结果丢包率 200% 多 ,快哭了
yuedingwangji
2016-05-27 16:15:20 +08:00
@redsonic p2p1 Link encap:Ethernet HWaddr 00:16:31:F6:00:F4
inet6 addr: fe80::216:31ff:fef6:f4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:447267366 errors:0 dropped:1115123945 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:156112249713 (145.3 GiB) TX bytes:468 (468.0 b)
这丢的也太严重了吧
redsonic
2016-05-27 16:27:57 +08:00
看 rx_fifo_errors , rx_no_buffer_count , rx_missed_errors ,只看 ifconfig 那个没用。禁用 RSS 后 rx_fifo_errors , rx_no_buffer_count 增加是正常的,因为只有一个核心处理软中断了,关键看 rx_missed_errors 频度是否有相应的大增。
redsonic
2016-05-27 16:31:44 +08:00
如果方便可以发邮件到匿名邮箱 v2ex@guerrillamail.de 我登上去看看。
jsfaint
2016-05-27 16:54:14 +08:00
不知道楼主的多大的包,发送速率多少,但是通常 10G 的卡,用内核协议栈处理不过来的。需要上 DPDK

这是一个专为移动设备优化的页面(即为了让你能够在 Google 搜索结果里秒开这个页面),如果你希望参与 V2EX 社区的讨论,你可以继续到 V2EX 上打开本讨论主题的完整版本。

https://www.v2ex.com/t/281482

V2EX 是创意工作者们的社区,是一个分享自己正在做的有趣事物、交流想法,可以遇见新朋友甚至新机会的地方。

V2EX is a community of developers, designers and creative people.

© 2021 V2EX