xdptap - Fault Injection: Dropping Packets

May 25, 2025

xdptap, IXDP, AF_XDP, libxdp, RPCL

Motivation #

Nobody working at network and data center operations usually likes packet loss, even just a small percentage can have serious effects on overall quality, bandwidth and end user experience.

However, there are cases where it’s necessary to establish networking faults by purpose, to name a few:

Testing protocol robustness
Testing complete application and data center connectivity robustness
Evaluating the possible impact of a faulty connection
Tearing down bandwidth selectively for specific Layer 2 connections
Testing network monitoring and automatic countermeasures

The unique xdptap feature of fault injection allows to establish several faults by purpose and at wire speed up to 100GbE. This article covers probably the most well known one: Packet loss.

Useful Preliminary Information #

These links may be useful to have a look at before:

Dropping Packets: IXDP and RPCL control #

The fault injection in general is located at step 6 in the IXDP packet processing flowchart. See also IXDP Packet Flow explained for more information about the other steps.

The decision where the packet has to be sent out has already been made in step 4 (for xdptap by the IXDP callback function representing a bridge).

flowchart TD a("1. packet from interface received") --> b("2. update MAC and Ethertype statistics") --> c("3. insert packet into TAP queue (if qualified)") --> d("4. user callback") d --> |drop by callback or unknown interface| d1("10. discard packet") --> d2[/"done"/] d --> |valid output interface| d3("5. check drop machash") d3 --> |drop by drop machash| d1("10. discard packet") d3 --> d4("6. fault injection (if qualified)") d4 --> |drop by fault injection| d1("10. discard packet") d4 --> d5("7. schedule packet for TX (with TAP queue insert, if qualified)") --> d2(["done (continue busy-polling from top)"])

For all settings that need to be active on xdptap startup it’s convenient to include them in /etc/xdptap.rpcl.

Show Summary Statistics and Counters #

if.stats #

show summary statistics of all interfaces ( - )

if.stats generally displays interface statistics of all configured interfaces. In the example below, it shows a freshly started xdptap in bridging mode, where each interface of two is driven by the ixgbe 10GbE driver which implements XDP zero-copy. queues:1 indicates that packet processing is handled by one single thread and one combined queue, which is sufficient for 10GbE even on this small testing machine.

The counters and settings related to packet dropping are highlighted in blue.

Example:

ok if.stats
0 enp1s0f0 ixgbe queues:1 ZEROCOPY
  rx                    0
  tx                    1
  drop                  0
  noif                  0
  err                   0
  RX tap                disabled
  TX tap                disabled
  fault injection       disabled
    drop probability    0 (0.000%)
    bitflip probability 0 (0.000%)
    dup probability     0 (0.000%)
    dropped by FI       0
    bitflipped by FI    0
    duplicated by FI    0
1 enp1s0f1 ixgbe queues:1 ZEROCOPY
  rx                    2
  tx                    0
  drop                  1
  noif                  0
  err                   0
  RX tap                disabled
  TX tap                disabled
  fault injection       disabled
    drop probability    0 (0.000%)
    bitflip probability 0 (0.000%)
    dup probability     0 (0.000%)
    dropped by FI       0
    bitflipped by FI    0
    duplicated by FI    0
ok ▂

Enabling and Disabling Fault Injection #

For xdptap, enabling fault injection determines the direction of packet flow where fault injection applies: When fault injection is enabled for both interfaces in bridging mode, the fault injection is active for both directions (with the propabilities as set).

fault.enable #

enable fault injection for interface ( interface - )

Example:

ok 0 fault.enable 
ok ▂

fault.disable #

disable fault injection for interface ( interface - )

Example:

ok 0 fault.disable 
ok ▂

Setting the Packet Drop Probability #

fault.drop.probability #

set FI drop probability in 1/100000 units ( probability interface - )

The fault.drop.probability is interface specific and can be set in 1/100000 units. The default value is 0 (0.000%).

Example:

ok 1000 0 fault.drop.probability # setting 1% for interface 0 
ok if.stats
0 enp1s0f0 ixgbe queues:1 ZEROCOPY
  rx                    48
  tx                    2959
  drop                  0
  noif                  0
  err                   0
  RX tap                disabled
  TX tap                disabled
  fault injection       disabled
    drop probability    1000 (1.000%)
    bitflip probability 0 (0.000%) 
    dup probability     0 (0.000%) 
    dropped by FI       0
    bitflipped by FI    0
    duplicated by FI    0
1 enp1s0f1 ixgbe queues:1 ZEROCOPY
  rx                    5917
  tx                    48
  drop                  2958
  noif                  0
  err                   0
  RX tap                disabled
  TX tap                disabled
  fault injection       disabled
    drop probability    0 (0.000%) 
    bitflip probability 0 (0.000%) 
    dup probability     0 (0.000%) 
    dropped by FI       0
    bitflipped by FI    0
    duplicated by FI    0
ok ▂

Restricting Fault Injection #

There are two atomic/lockless sets of MAC addresses that allow to tailor and restrict fault injection to specific MAC addresses and SRC/DST combinations:

fault.hash.dst: contains 0 or more MAC destination addresses
fault.hash.src: contains 0 or more MAC source addresses

A packet qualifies for fault injection if the following pseudo code expression is true:

(the incoming interface is enabled for FI) &&
((fault.hash.src is empty) || (the packets SRC MAC address is a member of fault.hash.src)) &&
((fault.hash.dst is empty) || (the packets DST MAC address is a member of fault.hash.dst))

The following RPCL words control both sets (fault.hash.dst.show and fault.hash.src.show additionally show vendor/OUI information when known):

fault.hash.dst.insert #

add specific MAC address to fault injection DST machash ( mac - )

fault.hash.dst.reset #

reset fault injection DST machash to 0 elements ( - )

fault.hash.dst.show #

show fault injection DST machash ( - )

fault.hash.src.insert #

add specific MAC address to fault injection SRC machash ( mac - )

fault.hash.src.reset #

reset fault injection SRC machash to 0 elements ( - )

fault.hash.src.show #

show fault injection SRC machash ( - )

Example:

ok fault.hash.dst.show 
ok fault.hash.src.show 
ok fault.hash.src.show 
ok 11:11:11:11:11:11 fault.hash.dst.insert
ok 22:22:22:22:22:22 fault.hash.dst.insert 
ok fault.hash.dst.show
11:11:11:11:11:11 Private
22:22:22:22:22:22 Locally Administered Address (LAA)
ok 33:33:33:33:33:33 fault.hash.src.insert 
ok 44:44:44:44:44:44 fault.hash.src.insert 
ok fault.hash.src.show
44:44:44:44:44:44
33:33:33:33:33:33
ok fault.hash.dst.reset 
ok fault.hash.src.reset
ok fault.hash.dst.show
ok fault.hash.src.show
ok ▂