Contents

Background

When developing the port knocking project, I came across a few eBPF verifier errors that were quite overwhelming at first. Here’s how I managed to debug them.

Technical Overview

The eBPF verifier performs static analysis on bytecode to ensure memory safety and prevent kernel crashes. It tracks pointer types, value ranges, and memory bounds without executing the program. Understanding verifier semantics is crucial for writing production eBPF code.

Problem 1: Dereferencing null pointers

Take a look at the following code which is a simplified snippet from my port knocking project:

static __always_inline int handle_udp_knock(
    u32 source_ip, u16 port, const struct port_sequence* seq)
{
    struct ip_state* state = bpf_map_lookup_elem(&ip_tracking_map, &source_ip);

    if (state->sequence_step >= seq->length) {
        log_error("sequence step > length");
        return XDP_PASS;
    }

    if (port == seq->ports[state->sequence_step]) {
        log_info("doing something useful here...");
    }

    return XDP_PASS;
}

Attempting to the load this BPF program results in the following error:

libbpf: prog 'knock': BPF program load failed: -EACCES
libbpf: prog 'knock': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int knock(struct xdp_md* ctx)
0: (bf) r8 = r1                       ; R1=ctx() R8_w=ctx()
1: (b7) r1 = 0                        ; R1_w=0
...
; struct ip_state* state = bpf_map_lookup_elem(&ip_tracking_map, &source_ip);
69: (18) r1 = 0xffff0000c8b1f400      ; R1_w=map_ptr(map=ip_tracking_map,
                                      ; ks=4,vs=24)
71: (85) call bpf_map_lookup_elem#1   ; R0=map_value_or_null(id=2,
                                      ; map=ip_tracking_map,ks=4,vs=24)
; if (state->sequence_step >= seq->length) {
72: (71) r1 = *(u8 *)(r0 +0)
R0 invalid mem access 'map_value_or_null'
processed 70 insns (limit 1000000) max_states_per_insn 1
    total_states 7 peak_states 7 mark_read 4
...

Let’s analyze the failure further.

The verifier’s error message

The key line is:

72: (71) r1 = *(u8 *)(r0 +0)
R0 invalid mem access 'map_value_or_null'

map_value_or_null represents the verifier’s type system tracking that bpf_map_lookup_elem() returns either:

  • A valid pointer to the map value, or
  • NULL if the key doesn’t exist

This means that the pointer r0 is possibly null and we’re dereferencing it. Let’s break down the error further. We can see that r0 is the pointer to the map value as returned by bpf_map_lookup_elem:

71: (85) call bpf_map_lookup_elem#1   ; R0=map_value_or_null(id=2,
                                      ; map=ip_tracking_map,ks=4,vs=24)

Here, map_value_or_null is a pointer type annotation which is used for the value returned by bpf_map_lookup_elem. This annotation is used to indicate that the pointer may be null.

In line 72, we can see that we’re dereferencing the pointer.

Solution

We need to add a check to ensure the pointer is not null before dereferencing it:

static __always_inline int handle_udp_knock(
    u32 source_ip, u16 port, const struct port_sequence* seq)
{
    struct ip_state* state = bpf_map_lookup_elem(&ip_tracking_map, &source_ip);
    if (state == NULL) {
        log_error("state is null");
        return XDP_PASS;
    }
...

Now the program loads successfully.

You tell ‘em Bret!

Problem 2: Omitting bounds checking

Back to our previous example:

#define MAX_SEQUENCE_LENGTH 10

/* config */
struct knock_config {
    __u16 target_port;
    struct port_sequence seq;
    __u64 session_timeout;
};

/* config->seq */
struct port_sequence {
    __u16 ports[MAX_SEQUENCE_LENGTH];
    __u8 length;
};

/* state */
struct ip_state {
    __u8 sequence_step;
};

static __always_inline int handle_udp_knock(
    u32 source_ip, u16 port, const struct port_sequence* seq)
{
    struct ip_state* state = bpf_map_lookup_elem(&ip_tracking_map, &source_ip);
    if (state == NULL) {
        log_error("state is null");
        return XDP_PASS;
    }

    if (state->sequence_step >= seq->length) {
        log_error("sequence step > length");
        return XDP_PASS;
    }

    if (port == seq->ports[state->sequence_step]) {
        log_info("doing something useful here...");
    }

    return XDP_PASS;
}
...

Attempting to load this BPF program results in yet another error:

libbpf: prog 'knock': BPF program load failed: -EACCES
libbpf: prog 'knock': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int knock(struct xdp_md* ctx)
0: (bf) r8 = r1                       ; R1=ctx() R8_w=ctx()

...

; if (!state) {
72: (55) if r0 != 0x0 goto pc+37 110: R0=map_value(map=ip_tracking_map,ks=4,vs=24) R6=2
                                      R7=map_value(map=config_map,ks=4,vs=48) R8=ctx()
                                      R9=8 R10=fp0 fp-8=????mmmm fp-16=mmmm????
                                      fp-24=mmmmmmmm
; if (state->sequence_step >= seq->length) {
110: (71) r1 = *(u8 *)(r0 +0)         ; R0=map_value(map=ip_tracking_map,ks=4,vs=24)
                                      ; R1_w=scalar(smin=smin32=0,
                                      ; smax=umax=smax32=umax32=255,
                                      ; var_off=(0x0; 0xff))
; if (state->sequence_step >= seq->length) {
111: (71) r2 = *(u8 *)(r7 +28)        ; R2=scalar(smin=smin32=0,
                                      ; smax=umax=smax32=umax32=255,
                                      ; var_off=(0x0; 0xff))
                                      ; R7=map_value(map=config_map,ks=4,vs=48)
; if (state->sequence_step >= seq->length) {
112: (2d) if r2 > r1 goto pc+5 118:     R0=map_value(map=ip_tracking_map,ks=4,vs=24)
                                        R1=scalar(smin=smin32=0,
                                        smax=umax=smax32=umax32=254,
                                        var_off=(0x0; 0xff))
                                        R2=scalar(smin=umin=smin32=umin32=1,
                                        smax=umax=smax32=umax32=255,var_off=(0x0; 0xff))
                                        R6=2 R7=map_value(map=config_map,ks=4,vs=48)
                                        R8=ctx() R9=8
                                        R10=fp0 fp-8=????mmmm fp-16=mmmm???? fp-24=mmmmmmmm
; return handle_udp_knock(source_ip, port, &config->seq);
118: (07) r7 += 8                     ; R7_w=map_value(map=config_map,ks=4,vs=48,off=8)
; if (port == seq->ports[state->sequence_step]) {
119: (67) r1 <<= 1                    ; R1_w=scalar(smin=smin32=0,
                                      ; smax=umax=smax32=umax32=508,
                                      ; var_off=(0x0; 0x1fe))
120: (0f) r7 += r1                    ; R1_w=scalar(smin=smin32=0,
                                      ; smax=umax=smax32=umax32=508,
                                      ; var_off=(0x0; 0x1fe))
                                      ; R7_w=map_value(map=config_map,ks=4,vs=48,
                                      ; off=8,smin=smin32=0,smax=umax=smax32=umax32=508,
                                      ; var_off=(0x0; 0x1fe))
121: (69) r1 = *(u16 *)(r7 +0)
invalid access to map value, value_size=48 off=516 size=2
R7 max value is outside of the allowed memory range
processed 89 insns (limit 1000000) max_states_per_insn 1 total_states 8
peak_states 8 mark_read 5
...

Now this one is a bit more tricky. The verifier is complaining that we’re accessing memory outside of the allowed range. The key lines are:

121: (69) r1 = *(u16 *)(r7 +0)
invalid access to map value, value_size=48 off=516 size=2
R7 max value is outside of the allowed memory range

We can see what R7 is holding here:

R7_w=map_value(map=config_map,ks=4,vs=48,
               off=8,smin=smin32=0,smax=umax=smax32=umax32=508,
               var_off=(0x0; 0x1fe))

R7 is pointing to the config->seq struct. Breaking it down:

  • The _w suffix after R7 indicates that it is a write operation
  • vs=48 indicates the map value size is 48 bytes (the whole struct knock_config)
  • off=8 indicates the offset into the map is 8 bytes (the config->seq field is at offset 8 in the struct knock_config. You might expect this to be 2 because the target_port is a 16-bit integer, but the compiler adds padding to ensure the struct is aligned to 8 bytes).

But if we look at the error, we can see that somehow the verifier has calculated the maximum possible offset specified here to be 516 bytes, which is outside of the allowed memory range.

But where does 516 come from?

  • The verifier determined that R1 (which represents state->sequence_step) has a maximum possible value of 254
  • When accessing seq->ports[state->sequence_step], the code does r1 <<= 1 (multiply by sizeof(__u16)) to convert array index to byte offset
  • r7 += r1 adds the result to r7, which is the pointer to the config->seq struct
  • Result: 254 × 2 = 508
  • The problem is that with an 8-byte offset (from r7 += 8) plus a maximum index offset of 508, you get 8 + 508 = 516

The issue is that state->sequence_step can be up to 254, however surely we’re checking that it’s less than seq->length? The problem is that the verifier is not able to know that seq->length is less than 254 (more precisely less than MAX_SEQUENCE_LENGTH).

Solution

The solution is to therefore check either state->sequence_step or seq->length to ensure they are less than a constant, statically verifiable limit (MAX_SEQUENCE_LENGTH):

if (state->sequence_step >= MAX_SEQUENCE_LENGTH) {
    log_error("sequence step out of bounds");
    return XDP_PASS;
}

Conclusion

Both failures stem from the verifier’s conservative static analysis:

  1. Null pointer: The verifier cannot assume map lookups succeed
  2. Bounds violation: The verifier sees type ranges (__u8 = [0,255]) but not runtime invariants (sequence_step < seq->length)

The verifier operates on worst-case scenarios without runtime context.

Final thoughts

eBPF verifier errors can be quite verbose, which makes them overwhelming at first. However, their verbosity can also be a blessing as they often provide a lot of information about what’s going on.