[mini] Fail-Over Policy Based Routing

Playing with PBR recently I came across what I thought was an odd usage - two set commands in the same statement.

i.e.

route-map PBR permit 10
  match ip address to-be-matched
  set ip next-hop 192.168.0.1
  set ip default next-hop 192.168.1.1

This is a bit odd to look at until you break it down.

Turns out there's an order of operations to PBR set statements.

From the Cisco documentation:

1. set ip next-hop
2. set interface
3. set ip default next-hop
4. set default interface

This means set ip next-hop will be attempted prior to, say, set interface. If it fails, then the next statement will be evaluated.

When I saw that, the first place my brain went to was, why not create two route-map elements to fix this?

(please note it's hard to air your dirty laundry on the Internet. Yes, this seemed dumb after I tested it)

route-map PBR permit 10
  match ip address to-be-matched
  set ip next-hop 192.168.0.1
route-map PBR permit 20
  match ip address to-be-matched
  set ip default next-hop 192.168.1.1

My thought process here was that if statement 10 failed to apply the set statement, then it would move on to statement 20. This is, of course, not true. Just like an ACL, a route-map stops evaluating future statements as soon as it has a match. So in the above config, using the same ACL (or even two ACLs that both matched the same traffic in different ways), statement 10 is always matched, and if it fails, traffic is just normally routed.

So there is some reason (albeit niche cases) to put "fail-over" statements into the route-map. The CCIE lab is basically all about niche cases (less lovingly called a "stupid router trick" by most of us), so this seemed worth exploring.

Here's our topology:

It's a little complex but I wanted to show a lot of different possibilities in one route-map statement.

R1's loopback0 (1.1.1.1/32) will be our source, travelling towards R9's loopback0 (9.9.9.9). Segments are IPed as 192.168.XY.Z/24, where XY is the lower and higher router number on the segment, and Z is the local router number. Example: The serial segment between R2 and R4 is 192.168.24.0/24, with R2's interface being 192.168.24.2 and R4 being 192.168.24.4.

EIGRP is advertising every IP in the topology. However, R5, R6 and R7 are summarizing all routes behind them to a default route towards R2.
R2 has an offset list towards R4 to make paths through it less desirable:

R2:
router eigrp 100
 network 0.0.0.0

 offset-list 0 in 50 Serial4/1

The net result of this is that traffic will be sent from R2 through R3 towards R9 unless PBR is involved.

Here's R2's PBR config:

ip access-list extended match

 permit ip host 1.1.1.1 host 9.9.9.9

route-map PBR permit 10

 match ip address match

 set ip next-hop 192.168.24.4

 set interface Serial4/2

 set ip default next-hop 192.168.26.6

 set default interface Serial4/4

interface FastEthernet1/0

 ip policy route-map PBR

This will match traffic from 1.1.1.1 towards 9.9.9.9. The first step is to attempt to send the traffic towards R4.

So to be clear, non-PBR traffic will go through R3:

R2#sh ip cef 9.9.9.9

9.9.9.9/32

  nexthop 192.168.23.3 Serial4/0

R1#trace 9.9.9.9 source fa1/0  ! Not from 1.1.1.1

Type escape sequence to abort.

Tracing the route to 9.9.9.9

VRF info: (vrf in name/id, vrf out name/id)

  1 192.168.12.2 44 msec 88 msec 48 msec

  2 192.168.23.3 112 msec 40 msec 68 msec

  3 192.168.39.9 116 msec 116 msec 160 msec

Now let's try our PBR match:

R1#trace 9.9.9.9 source Loopback0

Type escape sequence to abort.

Tracing the route to 9.9.9.9

VRF info: (vrf in name/id, vrf out name/id)

  1 192.168.12.2 108 msec 48 msec 72 msec

  2 192.168.24.4 96 msec 44 msec 76 msec

  3 192.168.49.9 136 msec 108 msec 108 msec

As expected, it went R2 -> R4 -> R9.

What if the link between R2 and R4 went down?

R2(config)#int s4/0

R2(config-if)#shut

Referencing back to our route-map...

route-map PBR permit 10

 match ip address match

 set ip next-hop 192.168.24.4  (now unavailable)

 set interface Serial4/2 

 set ip default next-hop 192.168.26.6

 set default interface Serial4/4

We'd expect the PBR to send the traffic through R5 next, via Serial4/2. A very important note is these are point to point serial interfaces. Using Ethernet is not such a good choice for set interface. The problem should be obvious to any CCIE candidate: We're relying on the far side to proxy ARP for 9.9.9.9, which it will do in our design, but also because of our design, IOS will typically reject the ARP change as "wrong interface".

In short, the safe answer is to use serial (P2P) interfaces.

Also to point out again that according to the routing table, R2 should send traffic towards 9.9.9.9 via R3:

R2(config-if)#do sh ip cef 9.9.9.9

9.9.9.9/32

  nexthop 192.168.23.3 Serial4/0

From R1's traceroute, we see that traffic does go through R5:

R1#trace 9.9.9.9 source Loopback0

Type escape sequence to abort.

Tracing the route to 9.9.9.9

VRF info: (vrf in name/id, vrf out name/id)

  1 192.168.12.2 24 msec 116 msec 44 msec

  2 192.168.25.5 92 msec 48 msec 112 msec

  3 192.168.58.8 96 msec 128 msec 96 msec

  4 192.168.89.9 152 msec 108 msec 96 msec

We've successfully failed the first statement and moved to the 2nd one.

R2(config-if)#int s4/2

R2(config-if)#shut

route-map PBR permit 10

 match ip address match

 set ip next-hop 192.168.24.4  (now unavailable)

 set interface Serial4/2  (now unavailable)

 set ip default next-hop 192.168.26.6

 set default interface Serial4/4

The set [ip] default commands will only trigger if the route towards the destination is via a default.

These won't work for us yet because....

R2#sh ip route 9.9.9.9

Routing entry for 9.9.9.9/32

  Known via "eigrp 100", distance 90, metric 2300416, type internal

[output omitted]

We have a specific route.

We see our PBR is doing nothing now:

R1#trace 9.9.9.9 source Loopback0

Type escape sequence to abort.

Tracing the route to 9.9.9.9

VRF info: (vrf in name/id, vrf out name/id)

  1 192.168.12.2 76 msec 36 msec 68 msec

  2 192.168.23.3 116 msec 52 msec 60 msec (through R3)

  3 192.168.39.9 84 msec 104 msec 100 msec

I haven't got a smooth answer for this, so let's just make R3 send a default as well.  Note I've increased the delay between R5, R6, R7 and R8, so that R3 will still be preffed even with just a default being sent.

R3(config)#int s2/0

R3(config-if)#ip summary-address eigrp 100 0.0.0.0 0.0.0.0

R2#sh ip route 9.9.9.9 long

[output omitted]

Gateway of last resort is 192.168.23.3 to network 0.0.0.0

So now we match a default towards R2.  Let's see our PBR kick in again.

R1#trace 9.9.9.9 source Loopback0

Type escape sequence to abort.

Tracing the route to 9.9.9.9

VRF info: (vrf in name/id, vrf out name/id)

  1 192.168.12.2 40 msec 88 msec 40 msec

  2 192.168.26.6 88 msec 108 msec 48 msec

  3 192.168.68.8 128 msec 140 msec 124 msec

  4 192.168.89.9 92 msec 116 msec 112 msec

Through R6!

And if the link to R6 is down?

R2(config)#int s4/3

R2(config-if)#shut

route-map PBR permit 10

 match ip address match

 set ip next-hop 192.168.24.4  (now unavailable)

 set interface Serial4/2  (now unavailable)

 set ip default next-hop 192.168.26.6 (now unavailable)

 set default interface Serial4/4

R1#trace 9.9.9.9 source Loopback0

Type escape sequence to abort.

Tracing the route to 9.9.9.9

VRF info: (vrf in name/id, vrf out name/id)

  1 192.168.12.2 80 msec 60 msec 88 msec

  2 192.168.27.7 112 msec 44 msec 88 msec

  3 192.168.78.8 96 msec 92 msec 100 msec

  4 192.168.89.9 88 msec 128 msec 132 msec

Through R7.

That wraps up my main point, but while we've got this setup, let's look at recursive PBR too.

I'm no-shutting all the interfaces we turned down earlier, and re-advertising the specific route from R3.

I've also added a leak-map on R5, R6 and R7 to allow R8's Lo0 (8.8.8.8) through in addition to the default route. Additionally, I de-prefed 8.8.8.8 through R3 and R4.

So to be clear, 9.9.9.9 is now reachable via R3, and 8.8.8.8 is reachable via equal-cost load sharing on R5, R6 and R7:

R2#sh ip cef 9.9.9.9

9.9.9.9/32

  nexthop 192.168.23.3 Serial4/0

R2#sh ip cef 8.8.8.8

8.8.8.8/32

  nexthop 192.168.25.5 Serial4/2

  nexthop 192.168.26.6 Serial4/3

  nexthop 192.168.27.7 Serial4/4

Recursive PBR allows for ECMP (equal cost multipathing) and PBR to mix. In short, pre-PBR, the path to 9.9.9.9 is via R3. Post PBR, we'll target having "8.8.8.8" as the next hop - which will ECMP through R5, R6 and R7.

In our environment, however, this is a bit hard to see, because per-destination CEF ECMP won't show up on our traceroute. Let's change to per-packet:

R2(config-route-map)#int s4/2

R2(config-if)#ip load-sharing per-packet

R2(config-if)#int s4/3

R2(config-if)#ip load-sharing per-packet

R2(config-if)#int s4/4

R2(config-if)#ip load-sharing per-packet

And let's re-write our route-map for recursion:

R2(config)#no route-map PBR permit 10

R2(config)#route-map PBR permit 10

R2(config-route-map)#match ip address match

R2(config-route-map)#set ip next-hop recursive 8.8.8.8

And test:

R1#trace 9.9.9.9 source Loopback0

Type escape sequence to abort.

Tracing the route to 9.9.9.9

VRF info: (vrf in name/id, vrf out name/id)

  1 192.168.12.2 28 msec *  44 msec

  2 192.168.25.5 72 msec <-- Indicates ECMP

    192.168.26.6 76 msec <-- Indicates ECMP

    192.168.27.7 60 msec <-- Indicates ECMP

  3 192.168.58.8 132 msec

    192.168.68.8 80 msec

    192.168.78.8 64 msec

  4 192.168.89.9 76 msec 76 msec 64 msec

Hope you enjoyed!

Jeff