BGP Load Balancing with Unequal Bandwidth

Traffic unequal-cost load balancing on router never be an easy thing, especially by pure routing technique like BGP. Let’s figure out how it work in both Cisco and Juniper environments by setup following lab test.

Cisco Environment

Topology

2017-03-04 22_58_13-Clipboard.png

Basic Setup

Configuration

R1
router bgp 100 
router bgp 100 bgp router-id 192.168.1.1
router bgp 100 address-family ipv4 unicast 
router bgp 100 neighbor-group rr-client 
router bgp 100 neighbor-group rr-client remote-as 100
router bgp 100 neighbor-group rr-client update-source Loopback0
router bgp 100 neighbor-group rr-client address-family ipv4 unicast 
router bgp 100 neighbor-group rr-client address-family ipv4 unicast route-reflector-client
router bgp 100 neighbor 192.168.3.3 
router bgp 100 neighbor 192.168.3.3 use neighbor-group rr-client
router bgp 100 neighbor 192.168.3.3 address-family ipv4 unicast 
router bgp 100 neighbor 192.168.4.4 
router bgp 100 neighbor 192.168.4.4 use neighbor-group rr-client
router bgp 100 neighbor 192.168.4.4 address-family ipv4 unicast 
router bgp 100 neighbor 192.168.5.5 
router bgp 100 neighbor 192.168.5.5 use neighbor-group rr-client

There are 3 OSPF neighbors with different metric values.

RP/0/0/CPU0:R1#sh ospf neighbor 
Neighbor ID     Pri   State           Dead Time   Address         Interface
192.168.3.3     1     FULL/DR         00:00:33    192.168.13.3    GigabitEthernet0/0/0/0
    Neighbor is up for 08:17:15
192.168.4.4     1     FULL/DR         00:00:39    192.168.14.4    GigabitEthernet0/0/0/1
    Neighbor is up for 08:17:23
192.168.5.5     128   FULL/BDR        00:00:38    192.168.15.5    GigabitEthernet0/0/0/2
    Neighbor is up for 08:16:15

RP/0/0/CPU0:R1#sh ospf interface brief 
Interface          PID   Area            IP Address/Mask    Cost  State Nbrs F/C
Gi0/0/0/0          1     0               192.168.13.1/24    10    BDR   1/1
Gi0/0/0/1          1     0               192.168.14.1/24    15    BDR   1/1
Gi0/0/0/2          1     0               192.168.15.1/24    100   DR    1/1

There are 3 BGP neighbors, they are advertising same BGP route. The route from R3 is best one due to lowest IGP metric.

RP/0/0/CPU0:R1#sh bgp summary 
Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
192.168.3.3       0   100     495     500       23    0    0 08:10:47          1
192.168.4.4       0   100     495     499       23    0    0 08:10:38          1
192.168.5.5       0   100    1072     989       23    0    0 08:10:03          1

RP/0/0/CPU0:R1#sh bgp 
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network            Next Hop            Metric LocPrf Weight Path
*>i200.200.9.9/32     192.168.3.3                   100      0 200 400 i
* i                   192.168.4.4                   100      0 200 400 i
* i                   192.168.5.5                   100      0 200 400 i

RP/0/0/CPU0:R1#sh bgp 200.200.9.9/32 bestpath-compare 
Paths: (3 available, best #1)
  Advertised to update-groups (with more than one peer):
    0.3 
  Path #1: Received by speaker 0
  Advertised to update-groups (with more than one peer):
    0.3 
  200 400, (Received from a RR-client)
    192.168.3.3 (metric 11) from 192.168.3.3 (192.168.3.3)
      Origin IGP, localpref 100, valid, internal, best, group-best
      Received Path ID 0, Local Path ID 1, version 24
      best of AS 200, Overall best
  Path #2: Received by speaker 0
  Not advertised to any peer
  200 400, (Received from a RR-client)
    192.168.4.4 (metric 16) from 192.168.4.4 (192.168.4.4)
      Origin IGP, localpref 100, valid, internal
      Received Path ID 0, Local Path ID 0, version 0
      Higher IGP metric than best path (path #1)
  Path #3: Received by speaker 0
  Not advertised to any peer
  200 400, (Received from a RR-client)
    192.168.5.5 (metric 101) from 192.168.5.5 (192.168.5.5)
      Origin IGP, localpref 100, valid, internal
      Received Path ID 0, Local Path ID 0, version 0
      Higher IGP metric than best path (path #1)

Standard Multipath (same IGP metric)

Yes, load sharing on BGP is achieved by multipath normally.

Configuration

R1
router bgp 100 address-family ipv4 unicast maximum-paths ibgp 8

Verification

RP/0/0/CPU0:R1#sh route bgp 
B    200.200.9.9/32 [200/0] via 192.168.3.3, 00:18:39

RP/0/0/CPU0:R1#sh bgp               
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network            Next Hop            Metric LocPrf Weight Path
*>i200.200.9.9/32     192.168.3.3                   100      0 200 400 i
* i                   192.168.4.4                   100      0 200 400 i
* i                   192.168.5.5                   100      0 200 400 i

Nothing change! Still see single next-hop for the BGP route. It is because standard multipath work for equal-cost BGP routes only.

Unequal-Cost Multipath (different IGP metric)

Configuration

R1
router bgp 100 address-family ipv4 unicast maximum-paths ibgp 8 unequal-cost

Verification

RP/0/0/CPU0:R1#sh route bgp 
B    200.200.9.9/32 [200/0] via 192.168.5.5, 00:00:34
                    [200/0] via 192.168.4.4, 00:00:34
                    [200/0] via 192.168.3.3, 00:00:34

RP/0/0/CPU0:R1#sh bgp 200.200.9.9/32 bestpath-compare 
Paths: (3 available, best #1)
  Advertised to update-groups (with more than one peer):
    0.3 
  Path #1: Received by speaker 0
  Advertised to update-groups (with more than one peer):
    0.3 
  200 400, (Received from a RR-client)
    192.168.3.3 (metric 11) from 192.168.3.3 (192.168.3.3)
      Origin IGP, localpref 100, valid, internal, best, group-best, multipath
      Received Path ID 0, Local Path ID 1, version 25
      best of AS 200, Overall best
  Path #2: Received by speaker 0
  Not advertised to any peer
  200 400, (Received from a RR-client)
    192.168.4.4 (metric 16) from 192.168.4.4 (192.168.4.4)
      Origin IGP, localpref 100, valid, internal, multipath
      Received Path ID 0, Local Path ID 0, version 0
      iBGP multi-path
  Path #3: Received by speaker 0
  Not advertised to any peer
  200 400, (Received from a RR-client)
    192.168.5.5 (metric 101) from 192.168.5.5 (192.168.5.5)
      Origin IGP, localpref 100, valid, internal, multipath
      Received Path ID 0, Local Path ID 0, version 0
      iBGP multi-path

It is better now as we can see BGP routes from R3 and R4 are installed onto routing table. But we don’t like see the BGP route from R5 !!

Selective Unequal-Cost Multipath (per neighbor)

Configuration

R1
router bgp 100 address-family ipv4 unicast maximum-paths ibgp 8 unequal-cost selective
router bgp 100 neighbor 192.168.3.3 
router bgp 100 neighbor 192.168.3.3 use neighbor-group rr-client
router bgp 100 neighbor 192.168.3.3 address-family ipv4 unicast 
router bgp 100 neighbor 192.168.3.3 address-family ipv4 unicast multipath
router bgp 100 neighbor 192.168.4.4 
router bgp 100 neighbor 192.168.4.4 use neighbor-group rr-client
router bgp 100 neighbor 192.168.4.4 address-family ipv4 unicast 
router bgp 100 neighbor 192.168.4.4 address-family ipv4 unicast multipath
router bgp 100 neighbor 192.168.5.5 
router bgp 100 neighbor 192.168.5.5 use neighbor-group rr-client

Verification

RP/0/0/CPU0:R1#sh route bgp 
B    200.200.9.9/32 [200/0] via 192.168.4.4, 00:00:03
                    [200/0] via 192.168.3.3, 00:00:03

RP/0/0/CPU0:R1#sh bgp 200.200.9.9/32 bestpath-compare 
Paths: (3 available, best #1)
  Advertised to update-groups (with more than one peer):
    0.3 
  Path #1: Received by speaker 0
  Advertised to update-groups (with more than one peer):
    0.3 
  200 400, (Received from a RR-client)
    192.168.3.3 (metric 11) from 192.168.3.3 (192.168.3.3)
      Origin IGP, localpref 100, valid, internal, best, group-best, multipath
      Received Path ID 0, Local Path ID 1, version 28
      best of AS 200, Overall best
  Path #2: Received by speaker 0
  Not advertised to any peer
  200 400, (Received from a RR-client)
    192.168.4.4 (metric 16) from 192.168.4.4 (192.168.4.4)
      Origin IGP, localpref 100, valid, internal, multipath
      Received Path ID 0, Local Path ID 0, version 0
      iBGP multi-path
  Path #3: Received by speaker 0
  Not advertised to any peer
  200 400, (Received from a RR-client)
    192.168.5.5 (metric 101) from 192.168.5.5 (192.168.5.5)
      Origin IGP, localpref 100, valid, internal
      Received Path ID 0, Local Path ID 0, version 0
      Higher IGP metric than best path (path #1)

Cool, we can see only BGP routes from R3 and R4 in routing table. Next step, let’s check how the router perform load sharing.

RP/0/0/CPU0:R1#sh cef 200.200.9.9/32 detail 
 LDI Update time Mar  4 10:50:55.037
   via 192.168.3.3/32, 2 dependencies, recursive, bgp-multipath [flags 0x6080]
    path-idx 0 NHID 0x0 [0xa13e06f4 0x0]
    next hop 192.168.3.3/32 via 192.168.3.3/32
   via 192.168.4.4/32, 2 dependencies, recursive, bgp-multipath [flags 0x6080]
    path-idx 1 NHID 0x0 [0xa13e05f4 0x0]
    next hop 192.168.4.4/32 via 192.168.4.4/32

    Load distribution: 0 1 (refcount 1)

    Hash  OK  Interface                 Address
    0     Y   GigabitEthernet0/0/0/0    192.168.13.3   
    1     Y   GigabitEthernet0/0/0/1    192.168.14.4

The router forward the traffic in ratio of 1:1.

“Unequal multipath” just install unequal BGP metric routes into routing table, then all next-hops will share the traffic evenly. It is a normal behavior as BGP load balancing feature is IGP metric unaware.

DMZ link bandwidth (link bandwidth)

Therefore, a new BGP attribute (extended community actually) is introduced, which is “DMZ link bandwidth”. It is used to tell BGP about the link bandwidth/metric, so that BGP will count this factor on load balancing calculation.

Configuration

R1
router bgp 100 neighbor 192.168.3.3 address-family ipv4 unicast route-policy bgp_R3_in in
router bgp 100 neighbor 192.168.4.4 address-family ipv4 unicast route-policy bgp_R4_in in
!
route-policy bgp_R3_in
  set extcommunity bandwidth (100:20000)
end-policy
!
route-policy bgp_R4_in
  set extcommunity bandwidth (100:10000)
end-policy

Verification

RP/0/0/CPU0:R1#sh bgp 200.200.9.9/32
Paths: (3 available, best #1)
  Advertised to update-groups (with more than one peer):
    0.3 
  Path #1: Received by speaker 0
  Advertised to update-groups (with more than one peer):
    0.3 
  200 400, (Received from a RR-client)
    192.168.3.3 (metric 11) from 192.168.3.3 (192.168.3.3)
      Origin IGP, localpref 100, valid, internal, best, group-best, multipath
      Received Path ID 0, Local Path ID 1, version 10
      Extended community: LB:100:160 
  Path #2: Received by speaker 0
  Not advertised to any peer
  200 400, (Received from a RR-client)
    192.168.4.4 (metric 16) from 192.168.4.4 (192.168.4.4)
      Origin IGP, localpref 100, valid, internal, multipath
      Received Path ID 0, Local Path ID 0, version 0
      Extended community: LB:100:80 
  Path #3: Received by speaker 0
  Not advertised to any peer
  200 400, (Received from a RR-client)
    192.168.5.5 (metric 101) from 192.168.5.5 (192.168.5.5)
      Origin IGP, localpref 100, valid, internal
      Received Path ID 0, Local Path ID 0, version 0

RP/0/0/CPU0:R1#sh route 200.200.9.9/32
Routing entry for 200.200.9.9/32
  Known via "bgp 100", distance 200, metric 0
  Tag 200, type internal
  Installed Mar  4 14:34:20.319 for 00:01:04
  Routing Descriptor Blocks
    192.168.3.3, from 192.168.3.3, BGP multi path
      Route metric is 0, Wt is 160
    192.168.4.4, from 192.168.4.4, BGP multi path
      Route metric is 0, Wt is 80
  No advertising protos. 
RP/0/0/CPU0:R1#sh cef 200.200.9.9/32 detail  
 LDI Update time Mar  4 14:34:20.349
   via 192.168.3.3/32, 2 dependencies, recursive, bgp-multipath [flags 0x6080]
    path-idx 0 NHID 0x0 [0xa13e08f4 0x0]
    next hop 192.168.3.3/32 via 192.168.3.3/32
   via 192.168.4.4/32, 2 dependencies, recursive, bgp-multipath [flags 0x6080]
    path-idx 1 NHID 0x0 [0xa13e07f4 0x0]
    next hop 192.168.4.4/32 via 192.168.4.4/32

    Weight distribution:
    slot 0, weight 160, normalized_weight 2, class 0
    slot 1, weight 80, normalized_weight 1, class 0

    Load distribution: 0 0 1 (refcount 1)

    Hash  OK  Interface                 Address
    0     Y   GigabitEthernet0/0/0/0    192.168.13.3   
    1     Y   GigabitEthernet0/0/0/0    192.168.13.3   
    2     Y   GigabitEthernet0/0/0/1    192.168.14.4

Perfect, router forward the traffic in ratio of 2:1 now!

Final Configuration

router bgp 100
router bgp 100 bgp router-id 192.168.1.1
router bgp 100 address-family ipv4 unicast
router bgp 100 address-family ipv4 unicast maximum-paths ibgp 8 unequal-cost selective
router bgp 100 neighbor-group rr-client
router bgp 100 neighbor-group rr-client remote-as 100
router bgp 100 neighbor-group rr-client update-source Loopback0
router bgp 100 neighbor-group rr-client address-family ipv4 unicast
router bgp 100 neighbor-group rr-client address-family ipv4 unicast route-reflector-client
router bgp 100 neighbor 192.168.3.3
router bgp 100 neighbor 192.168.3.3 use neighbor-group rr-client
router bgp 100 neighbor 192.168.3.3 address-family ipv4 unicast
router bgp 100 neighbor 192.168.3.3 address-family ipv4 unicast multipath
router bgp 100 neighbor 192.168.3.3 address-family ipv4 unicast route-policy bgp_R3_in in
router bgp 100 neighbor 192.168.4.4
router bgp 100 neighbor 192.168.4.4 use neighbor-group rr-client
router bgp 100 neighbor 192.168.4.4 address-family ipv4 unicast
router bgp 100 neighbor 192.168.4.4 address-family ipv4 unicast multipath
router bgp 100 neighbor 192.168.4.4 address-family ipv4 unicast route-policy bgp_R4_in in
router bgp 100 neighbor 192.168.5.5
router bgp 100 neighbor 192.168.5.5 use neighbor-group rr-client
!
route-policy bgp_R3_in
  set extcommunity bandwidth (100:20000)
end-policy
!
route-policy bgp_R4_in
  set extcommunity bandwidth (100:10000)
end-policy

 

Juniper Environment

Similar concept and setup apply on Juniper router. But, the only difference is multipath BGP routes must with same IGP metric in Junos.

Topology

2017-03-04 22_58_53-Clipboard.png

Configuration

set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 metric 10
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 metric 10
set protocols ospf area 0.0.0.0 interface ge-0/0/2.0 metric 100
set protocols ospf area 0.0.0.0 interface lo0.0

set protocols bgp group rr-client type internal
set protocols bgp group rr-client local-address 192.168.2.2
set protocols bgp group rr-client import BGP-BW
set protocols bgp group rr-client cluster 192.168.2.2
set protocols bgp group rr-client neighbor 192.168.3.3 multipath
set protocols bgp group rr-client neighbor 192.168.4.4 multipath
set protocols bgp group rr-client neighbor 192.168.5.5

set policy-options policy-statement BGP-BW term R3 from protocol bgp
set policy-options policy-statement BGP-BW term R3 from neighbor 192.168.3.3
set policy-options policy-statement BGP-BW term R3 then community add BW-HIGH
set policy-options policy-statement BGP-BW term R3 then accept
set policy-options policy-statement BGP-BW term R4 from protocol bgp
set policy-options policy-statement BGP-BW term R4 from neighbor 192.168.4.4
set policy-options policy-statement BGP-BW term R4 then community add BW-LOW
set policy-options policy-statement BGP-BW term R4 then accept
set policy-options policy-statement load-balance then load-balance per-packet
set policy-options community BW-HIGH members bandwidth:100:20000
set policy-options community BW-LOW members bandwidth:100:10000

set routing-options forwarding-table export load-balance
set policy-options policy-statement load-balance then load-balance per-packet

Verification

root@R2> show route protocol bgp              
inet.0: 18 destinations, 20 routes (18 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
200.200.9.9/32     *[BGP/170] 01:42:57, localpref 100, from 192.168.3.3
                      AS path: 200 400 I, validation-state: unverified
                    > to 192.168.23.3 via ge-0/0/0.0
                      to 192.168.24.4 via ge-0/0/1.0
                    [BGP/170] 01:42:57, localpref 100, from 192.168.4.4
                      AS path: 200 400 I, validation-state: unverified
                    > to 192.168.24.4 via ge-0/0/1.0
                    [BGP/170] 01:42:54, localpref 100, from 192.168.5.5
                      AS path: 200 400 I, validation-state: unverified
                    > to 192.168.25.5 via ge-0/0/2.0

root@R2> show route protocol bgp extensive 
inet.0: 18 destinations, 20 routes (18 active, 0 holddown, 0 hidden)
200.200.9.9/32 (3 entries, 1 announced)
TSI:
KRT in-kernel 200.200.9.9/32 -> {indirect(1048574), indirect(1048576)}
Page 0 idx 1, (group rr-client type Internal) Type 1 val 0x95c25bc (adv_entry)
   Advertised metrics:
     Nexthop: 192.168.3.3
     Localpref: 100
     AS path: [100] 200 400 I
     Communities: bandwidth:100:20000
     Cluster ID: 192.168.2.2
     Originator ID: 192.168.3.3
    Advertise: 00000006
Path 200.200.9.9 from 192.168.3.3 Vector len 4.  Val: 1
        *BGP    Preference: 170/-101
                Next hop type: Indirect
                Address: 0x961807c
                Next-hop reference count: 3
                Source: 192.168.3.3
                Next hop type: Router, Next hop index: 574
                Next hop: 192.168.23.3 via ge-0/0/0.0, selected
                Session Id: 0x146
                Next hop type: Router, Next hop index: 573
                Next hop: 192.168.24.4 via ge-0/0/1.0
                Session Id: 0x144
                Protocol next hop: 192.168.3.3 Balance: 67%
                Indirect next hop: 0x964c110 1048574 INH Session ID: 0x141
                Protocol next hop: 192.168.4.4 Balance: 33%
                Indirect next hop: 0x964c220 1048576 INH Session ID: 0x145
                State: 
                Local AS:   100 Peer AS:   100
                Age: 1:43:28    Metric2: 11 
                Validation State: unverified 
                Task: BGP_100.192.168.3.3+179
                Announcement bits (3): 0-KRT 3-BGP_RT_Background 4-Resolve tree 1 
                AS path: 200 400 I
                Communities: bandwidth:100:20000
                Accepted Multipath
                Localpref: 100
                Router ID: 192.168.3.3
                Indirect next hops: 2
                        Protocol next hop: 192.168.3.3 Metric: 11
                        Indirect next hop: 0x964c110 1048574 INH Session ID: 0x141
                        Indirect path forwarding next hops: 1
                                Next hop type: Router
                                Next hop: 192.168.23.3 via ge-0/0/0.0
                                Session Id: 0x146
                        192.168.3.3/32 Originating RIB: inet.0
                          Metric: 11                      Node path count: 1
                          Forwarding nexthops: 1
                                Nexthop: 192.168.23.3 via ge-0/0/0.0
                        Protocol next hop: 192.168.4.4 Metric: 11
                        Indirect next hop: 0x964c220 1048576 INH Session ID: 0x145
                        Indirect path forwarding next hops: 1
                                Next hop type: Router
                                Next hop: 192.168.24.4 via ge-0/0/1.0
                                Session Id: 0x144
                        192.168.4.4/32 Originating RIB: inet.0
                          Metric: 11                      Node path count: 1
                          Forwarding nexthops: 1
                                Nexthop: 192.168.24.4 via ge-0/0/1.0
         BGP    Preference: 170/-101
                Next hop type: Indirect
                Address: 0x96949dc
                Next-hop reference count: 1
                Source: 192.168.4.4     
                Next hop type: Router, Next hop index: 573
                Next hop: 192.168.24.4 via ge-0/0/1.0, selected
                Session Id: 0x144
                Protocol next hop: 192.168.4.4
                Indirect next hop: 0x964c220 1048576 INH Session ID: 0x145
                State: 
                Inactive reason: Not Best in its group - Router ID
                Local AS:   100 Peer AS:   100
                Age: 1:43:28    Metric2: 11 
                Validation State: unverified 
                Task: BGP_100.192.168.4.4+20422
                AS path: 200 400 I
                Communities: bandwidth:100:10000
                Accepted MultipathContrib
                Localpref: 100
                Router ID: 192.168.4.4
                Indirect next hops: 1
                        Protocol next hop: 192.168.4.4 Metric: 11
                        Indirect next hop: 0x964c220 1048576 INH Session ID: 0x145
                        Indirect path forwarding next hops: 1
                                Next hop type: Router
                                Next hop: 192.168.24.4 via ge-0/0/1.0
                                Session Id: 0x144
                        192.168.4.4/32 Originating RIB: inet.0
                          Metric: 11                      Node path count: 1
                          Forwarding nexthops: 1
                                Nexthop: 192.168.24.4 via ge-0/0/1.0
         BGP    Preference: 170/-101
                Next hop type: Indirect
                Address: 0x9694ac0
                Next-hop reference count: 1
                Source: 192.168.5.5
                Next hop type: Router, Next hop index: 572
                Next hop: 192.168.25.5 via ge-0/0/2.0, selected
                Session Id: 0x142
                Protocol next hop: 192.168.5.5
                Indirect next hop: 0x964c330 1048575 INH Session ID: 0x143
                State: 
                Inactive reason: Not Best in its group - IGP metric
                Local AS:   100 Peer AS:   100
                Age: 1:43:25    Metric2: 101 
                Validation State: unverified 
                Task: BGP_100.192.168.5.5+179
                AS path: 200 400 I
                Accepted                
                Localpref: 100
                Router ID: 192.168.5.5
                Indirect next hops: 1
                        Protocol next hop: 192.168.5.5 Metric: 101
                        Indirect next hop: 0x964c330 1048575 INH Session ID: 0x143
                        Indirect path forwarding next hops: 1
                                Next hop type: Router
                                Next hop: 192.168.25.5 via ge-0/0/2.0
                                Session Id: 0x142
                        192.168.5.5/32 Originating RIB: inet.0
                          Metric: 101                     Node path count: 1
                          Forwarding nexthops: 1
                                Nexthop: 192.168.25.5 via ge-0/0/2.0

root@R2> show route forwarding-table destination 200.200.9.9/32 extensive 
Routing table: default.inet [Index 0] 
Internet:
    
Destination:  200.200.9.9/32
  Route type: user                  
  Route reference: 0                   Route interface-index: 0   
  Multicast RPF nh index: 0             
  Flags: sent to PFE 
  Next-hop type: unilist               Index: 1048578  Reference: 1    
  Next-hop type: indirect              Index: 1048574  Reference: 2    
                                    Weight: 0x0   Balance: 43690
  Nexthop: 192.168.23.3
  Next-hop type: unicast               Index: 574      Reference: 6    
  Next-hop interface: ge-0/0/0.0    Weight: 0x0  
  Next-hop type: indirect              Index: 1048576  Reference: 2    
                                    Weight: 0x0   Balance: 65535
  Nexthop: 192.168.24.4
  Next-hop type: unicast               Index: 573      Reference: 6    
  Next-hop interface: ge-0/0/1.0    Weight: 0x0  

Routing table: __master.anon__.inet [Index 4] 
Internet:
                                        
Destination:  default
  Route type: permanent             
  Route reference: 0                   Route interface-index: 0   
  Multicast RPF nh index: 0             
  Flags: sent to PFE 
  Next-hop type: reject                Index: 521      Reference: 1
Posted in IOS-XR Router, JUNOS Router | Tagged | Leave a comment

Firewall filter on Juniper router

When a Juniper router is operating over public Internet, we should setup a firewall filter to protect the router routing engine from cyber threats. Generally, the firewall filter design will look like this:

  1. Allow certain protocols, such as ospf, isis, bgp, ldp, rsvp, igmp, bfd.
  2. Set rate limit for certain traffic, such as icmp, traceroute.
  3. Allow the return traffic which is initiated from own router, such as telnet, ssh.
  4. Discard other unknown traffic.

I would like to share some tricky firewall policy design on this post.

BGP for peers only

Requirement:  Only allow BGP peers which appear on the configuration. For example:

root@R2# show protocols bgp | display set 
set protocols bgp group rr-client type internal
set protocols bgp group rr-client local-address 192.168.2.2
set protocols bgp group rr-client cluster 192.168.2.2
set protocols bgp group rr-client neighbor 192.168.3.3
set protocols bgp group rr-client neighbor 192.168.4.4
set protocols bgp group rr-client neighbor 192.168.5.5
set protocols bgp group rr-server type internal
set protocols bgp group rr-server local-address 192.168.2.2
set protocols bgp group rr-server neighbor 192.168.1.1

Then required firewall policy will be:

set firewall family inet filter PROTECT-RE term BGP from source-prefix-list BGP-PEERS
set firewall family inet filter PROTECT-RE term BGP from source-prefix-list BGP-PEERS-VRF
set firewall family inet filter PROTECT-RE term BGP from protocol tcp
set firewall family inet filter PROTECT-RE term BGP from port bgp
set firewall family inet filter PROTECT-RE term BGP then accept

set policy-options prefix-list BGP-PEERS apply-path "protocols bgp group <*> neighbor <*>"
set policy-options prefix-list BGP-PEERS-VRF apply-path "routing-instances <*> protocols bgp group <*> neighbor <*>"

root@R2# show policy-options prefix-list BGP-PEERS | display inheritance 
##
## apply-path was expanded to:
##     192.168.3.3/32; 
##     192.168.4.4/32; 
##     192.168.5.5/32; 
##     192.168.1.1/32; 
##
apply-path "protocols bgp group <*> neighbor <*>";

ICMP vs Ping Test

Requirement:

  • No limit on the ICMP traffic (ping with large packet size) from direct attached device/routers.
  • Limit other ICMP traffic from Internet to 100kbps.

Topology:

R1 (Juniper router) — R2 (peer router) ———– R3 (Internet host)

Then required firewall policy will be:

R1
set firewall family inet filter PROTECT-RE term ICMP-LINK-TEST from protocol icmp
set firewall family inet filter PROTECT-RE term ICMP-LINK-TEST from ttl 255
set firewall family inet filter PROTECT-RE term ICMP-LINK-TEST then accept

set firewall family inet filter PROTECT-RE term ICMP-OTHER from protocol icmp
set firewall family inet filter PROTECT-RE term ICMP-OTHER then policer ICMP-POLICER
set firewall family inet filter PROTECT-RE term ICMP-OTHER then accept
set firewall policer ICMP-POLICER if-exceeding bandwidth-limit 100k
set firewall policer ICMP-POLICER if-exceeding burst-size-limit 2k
set firewall policer ICMP-POLICER then discard

Verification:

R2#ping 192.168.1.1 size 1000 repeat 100 
Type escape sequence to abort.
Sending 100, 1000-byte ICMP Echos to 192.168.2.2, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 1/5/19 ms

R3#ping 192.168.2.2 size 1000 repeat 100 
Type escape sequence to abort.
Sending 100, 1000-byte ICMP Echos to 192.168.2.2, timeout is 2 seconds:
!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!
!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!
Success rate is 67 percent (67/100), round-trip min/avg/max = 9/10/39 ms

Return traffic

Requirement:

  • Router A able telnet to other routers.
  • But other routers unable telnet to Router A.

Then required firewall policy will be:

set firewall family inet filter PROTECT-RE term ESTABLISHED from protocol tcp
set firewall family inet filter PROTECT-RE term ESTABLISHED from tcp-established
set firewall family inet filter PROTECT-RE term ESTABLISHED then accept

 

Posted in Juniper, JUNOS Router | Tagged | Leave a comment

Cisco 6PE with iBGP CE

In nature IPv6 environment, IPv6 BGP route manipulation is straight forward like IPv4. But the setup will be a little bit tricky for 6PE when CE is running iBGP.

This is what we want to achieve:

Cisco 6PE with iBGP CE.png

Let’s setup a lab to study what sort of problem we will face:

2017-02-24 11_41_55-Clipboard.png

Configuration

RP/0/0/CPU0:R1#sh run formal router bgp 
router bgp 100 
router bgp 100 address-family ipv6 unicast 
router bgp 100 address-family ipv6 unicast allocate-label all
router bgp 100 neighbor 2001:14::4   <-- iBGP CE
router bgp 100 neighbor 2001:14::4 remote-as 100
router bgp 100 neighbor 2001:14::4 description RTBH Server
router bgp 100 neighbor 2001:14::4 address-family ipv6 unicast 
router bgp 100 neighbor 2001:14::4 address-family ipv6 unicast route-reflector-client
router bgp 100 neighbor 2001:14::4 address-family ipv6 unicast next-hop-self
router bgp 100 neighbor 2001:15::5   <-- eBGP CE
router bgp 100 neighbor 2001:15::5 remote-as 200
router bgp 100 neighbor 2001:15::5 address-family ipv6 unicast 
router bgp 100 neighbor 2001:15::5 address-family ipv6 unicast route-policy PASS_ALL in
router bgp 100 neighbor 2001:15::5 address-family ipv6 unicast route-policy PASS_ALL out
router bgp 100 neighbor 192.168.3.3   <-- remote 6PE
router bgp 100 neighbor 192.168.3.3 remote-as 100
router bgp 100 neighbor 192.168.3.3 description Remote PE
router bgp 100 neighbor 192.168.3.3 update-source Loopback0
router bgp 100 neighbor 192.168.3.3 address-family ipv4 unicast 
router bgp 100 neighbor 192.168.3.3 address-family ipv4 unicast next-hop-self
router bgp 100 neighbor 192.168.3.3 address-family ipv6 labeled-unicast

BGP neighbors

RP/0/0/CPU0:R1#sh bgp ipv6 labeled-unicast summary
 Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
 192.168.3.3       0   100      43      46       15    0    0 00:37:53          0   <-- R3 (remote 6PE)

RP/0/0/CPU0:R1#sh bgp ipv6 unicast summary
 Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
 2001:14::4        0   100      46      45       15    0    0 00:37:52          2   <-- R4 (iBGP CE)
 2001:15::5        0   200      47      43       15    0    0 00:37:52          2   <-- R5 (eBGP CE)

Routes received from CE

RP/0/0/CPU0:R1#sh bgp ipv6 unicast
 Status codes: s suppressed, d damped, h history, * valid, > best
 i - internal, r RIB-failure, S stale, N Nexthop-discard
 Origin codes: i - IGP, e - EGP, ? - incomplete
 Network            Next Hop            Metric LocPrf Weight Path
 *>i2001::4/128        2001:14::4               0    100      0 ?
 *> 2001::5/128        2001:15::5               0             0 200 ?
 *>i2001:14::/64       2001:14::4               0    100      0 ?
 *> 2001:15::/64       2001:15::5               0             0 200 ?

Routes advertise to remote 6PE

RP/0/0/CPU0:R1#sh bgp ipv6 labeled-unicast neighbors 192.168.3.3 advertised-routes
Network            Next Hop        From            AS Path
2001::5/128        192.168.1.1     2001:15::5      200?
2001:15::/64       192.168.1.1     2001:15::5      200?

Why R1 don’t advertise routes from R4 to R3?  By default, Cisco 6PE won’t advertise iBGP CE routes to iBGP 6PE peers even CE is RR client.

Advertise the iBGP routes

RP/0/0/CPU0:R1(config)#sh configuration 
Building configuration...
!! IOS XR Configuration 5.3.3
router bgp 100
 ibgp policy out enforce-modifications
 neighbor 192.168.3.3
  address-family ipv6 labeled-unicast
   next-hop-self   <-- affect iBGP routes only, eBGP routes "next-hop-self" implicitly.
end
RP/0/0/CPU0:R1(config)#commit 

RP/0/0/CPU0:R1#sh bgp ipv6 labeled-unicast neighbors 192.168.3.3 advertised-routes
Network            Next Hop        From            AS Path
2001::4/128        192.168.1.1     2001:14::4      ?
2001::5/128        192.168.1.1     2001:15::5      200?
2001:14::/64       192.168.1.1     2001:14::4      ?
2001:15::/64       192.168.1.1     2001:15::5      200?
Processed 4 prefixes, 4 paths

Change the next-hop of iBGP routes

For some reasons, we may need to change the next-hop of the iBGP routes. It can be achieve as below:

RP/0/0/CPU0:R3#sh bgp ipv6 unicast 
   Network            Next Hop            Metric LocPrf Weight Path
*>i2001::4/128        192.168.1.1              0    100      0 ?
*>i2001::5/128        192.168.1.1              0    100      0 200 ?
*>i2001:14::/64       192.168.1.1              0    100      0 ?
*>i2001:15::/64       192.168.1.1              0    100      0 200 ?
Processed 4 prefixes, 4 paths

RP/0/0/CPU0:R1(config)#sh configuration 
Building configuration...
!! IOS XR Configuration 5.3.3
!
route-policy IBGP_NH
  if destination in (2001::4/128) then
    set next-hop 192.168.99.99
  else
    pass   <-- other routes "next-hop-self"
  endif end-policy
!
router bgp 100
 neighbor 192.168.3.3
  address-family ipv6 labeled-unicast
   route-policy IBGP_NH out
end
RP/0/0/CPU0:R1(config)#commit

RP/0/0/CPU0:R3#sh bgp ipv6 labeled-unicast 
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network            Next Hop            Metric LocPrf Weight Path
* i2001::4/128        192.168.99.99            0    100      0 ?
*>i2001::5/128        192.168.1.1              0    100      0 200 ?
*>i2001:14::/64       192.168.1.1              0    100      0 ?
*>i2001:15::/64       192.168.1.1              0    100      0 200 ?
Processed 4 prefixes, 4 paths

 

 

Posted in IOS-XR Router | Tagged , | Leave a comment

Inter-AS mVPN Option A with Cisco & Juniper ASBRs

Setup Inter-AS mVPN (draft rosen) Option A should much easier than Option B, as the mVPN configuration on an AS is independent from another one theoretically. But we will see an interesting scenario if the network is mixed of Juniper & Cisco routers.

 

Inter-AS Multicast VPN over L3VPN Option B (pure Cisco)

Before discuss about the potential problem that face in Inter-AS mVPN Option A with mixed of Juniper & Cisco ASBRs environment, we need to understand how Inter-AS mVPN Option B operates under pure Cisco environment first.

2016-12-24 16_11_33-Clipboard.png

In order to setup PE-to-PE MDT Tunnel, we need:

  • All PE routers in an AS should know the source address of PE routers in adjacent AS.
  • ASBR should know the Source PE address in adjacent AS for RPF check.
  • SSM multicast enabled router on receiving PIM join, will look into the embedded (source address, Group) and send PIM join to upstream towards the source. So all P routers should know the source PE address in adjacent AS to send the PIM join.

Cisco Introduces BGP connector attribute, BGP MDT SAFI identifier and RPF vector to addresses these issues.

Below is very comprehensive article from INE.com describes about how Inter-AS mVPN Option B operate in pure Cisco routers environment:

Inter-AS mVPNs: MDT SAFI, BGP Connector & RPF Proxy Vector
http://blog.ine.com/2010/01/17/inter-as-mvpns-mdt-safi-bgp-connector-rpf-proxy-vector/

 

Inter-AS multicast VPN over L3VPN Option A (Cisco / Juniper)

Regarding to “connector” attribute, Cisco & Juniper behave total differently.

2016-12-24 16_19_31-Clipboard.png

Cisco ASBR
“connector” is a Cisco VPNv4 route attribute, Cisco only propagate it between VPNv4 BGP update. Therefore “connector” will propagate between different AS for Option B, but only propagate within own AS for Option A.

Configuring Multicast VPN Inter-AS Support
http://www.cisco.com/c/en/us/td/docs/ios/12_0s/feature/guide/iasmcvpn.html#wp1054007

Juniper ASBR
Juniper just simply propagate this “connector” as it is an unknown “optional transitive attribute”.

A Border Gateway Protocol 4 (BGP-4)
http://www.ietf.org/rfc/rfc4271.txt

No matter how Cisco or Juniper propagate the “connector” attribute, they don’t affect MDT tunnels establishment in Option A as the MDT tunnels are built within own AS. But it cause potential RPF check problem!

 

Topology:  RPF check problem for RP in mVPN Option A

The RPF problem happens when:

  • AS100 PE is Cisco router (generate “connector”)
  • AS100 ASBR is Juniper router (propagate “connector” to AS200)
  • AS200 PE is Cisco router (RPF fail)

 

2016-12-24 16_24_36-Clipboard.png

RP/0/0/CPU0:R6#debug pim bsr 
RP/0/0/CPU0:R6#RP/0/0/CPU0:Mar 19 15:02:03.321 : pim[1152]: [13] ABC: Received BSR message pkt len 36 from 192.168.4.4 (mdtABC) for 10.10.10.10, priority 0 hash mask length 0
RP/0/0/CPU0:Mar 19 15:02:03.321 : pim[1152]: [13] ABC: BSR lookup rib for RPF info
RP/0/0/CPU0:Mar 19 15:02:03.321 : pim[1152]: [13] ABC: Drop BSR message from 192.168.4.4 arrived on mdtABC (CT:1) for bsr addr 10.10.10.10 :- RPF failed, expected rpf nbr 192.168.9.9, expected intf mdtABC (MDT,CT:1)  expected vrf -

RP/0/0/CPU0:R6#sh pim vrf ABC rpf 10.10.10.10
Table: IPv4-Unicast-default
* 10.10.10.10/32 [200/0] 
    via mdtABC with rpf neighbor 192.168.9.9
    Connector: 100:1:192.168.9.9, Nexthop: 192.168.9.9

In Cisco implementation, R6/PE won’t expect “connector” come from peer AS in mVPN Option A environment !!

Cisco PE RPF check behavior:

  • C7200 IOS 15.2(4)S6, check connector
  • C7200 IOS 12.2(33)SRE10, check connector
  • XRv 5.3.0, check connector
  • C12406 IOS 12.0(33)S11, DO NOT check connector

Remark:  different software versions behave differently.

 

Solutions

Option 1.  Drop connector attribute in ASBR/Junos in ingress direction
Option 2.  Disable advertising extended community in PE/XR for specific VRF
Option 3.  BGP attribute filter in peer ASBR/XR in ingress direction

2016-12-24 16_36_19-Clipboard.png

Solution 1:  Drop connector attribute in ASBR/Junos in ingress direction

BGP “connector” attribute filter in Juniper ASBR. Put the command in upstream peer, then the attribute will be filtered in all downstream peer. (Support Junos 10.2 or later)

First of all, we have to figure out the BGP path attribute value for “connector”. It can be achieved by packet capturing or refer to RFC4271.

2016-12-24 16_38_17-Clipboard.png

Border Gateway Protocol (BGP) Parameters
http://www.iana.org/assignments/bgp-parameters/bgp-parameters.xhtml#bgp-parameters-2

Value     Code                 Reference
20     Connector Attribute (deprecated)     [RFC6037]

root@R3> show configuration protocols bgp group PE 
neighbor 192.168.9.9 {
    drop-path-attributes 20;
}

RP/0/0/CPU0:R4#sh bgp vpnv4 unicast vrf ABC 10.10.10.10
Fri Mar 20 21:12:20.119 UTC
BGP routing table entry for 10.10.10.10/32, Route Distinguisher: 200:1
Versions:
  Process           bRIB/RIB  SendTblVer
  Speaker                 61          61
    Local Label: 24013
Last Modified: Mar 20 19:13:03.967 for 01:59:16
Paths: (1 available, best #1)
  Advertised to update-groups (with more than one peer):
    0.2 
  Path #1: Received by speaker 0
  Advertised to update-groups (with more than one peer):
    0.2 
  100 65300
    192.168.34.3 from 192.168.34.3 (10.3.3.3)
      Origin incomplete, localpref 100, valid, external, best, group-best, import-candidate
      Received Path ID 0, Local Path ID 1, version 61   <-- Juniper ASBR no longer propagate “connector”
      Extended community: RT:200:1

Solution 2:  Disable advertising extended community in PE/XR for specific VRF

IOS  (support IOS M&T15.5(1)T or later)

router bgp 45000
 address-family ipv4 vrf ABC
  unicast-reachability source-as disable

IOS-XR  (support XR 4.3 or later)

router pim
 vrf ABC
  address-family ipv4
   mdt c-multicast-routing bgp
    unicast-reachability connector disable vrf-route-import disable source-as disable

Solution 3:  BGP attribute filter in peer ASBR/XR in ingress direction

IOS  (support IOS 15.2(4)S or later)

router bgp 200
 neighbor 192.168.4.4 path-attribute discard 20 in

IOS-XR  (support in XR 4.2.3)

router bgp 200
 attribute-filter group FILTER
  attribute CONNECTOR discard
 vrf ABC
  neighbor 192.168.34.3
   update in filtering
   attribute-filter group FILTER

 

 

Posted in Cisco, Interoperability, Juniper | Tagged , | Leave a comment

Audit CEF mis-match issue on Cisco IOS-XR routers

Background

Routing table is used to look up a specific route and resolve its next-hop and corresponding outgoing interface in control-plane level.

CEF table is used to forward packet for a specific route toward corresponding outgoing interface in data-plane level.

In fact, CEF table is derived from routing table, thus they should synchronize with each other. Which mean, when a route lookup on routing table with outgoing interface A, then same route lookup on CEF table should with outgoing interface A too.

CEF Bug

According Cisco documentation, there is a bug of “CEF route version mis-match” on multiple IOS-XR software versions which cause route lookup wrongly on CEF table. As a result, traffic will be backhole and we will receive customer complaint very soon.

Cisco Bug: CSCui93977 – FIB entry is not deleted due to version mismatch
https://quickview.cloudapps.cisco.com/quickview/bug/CSCui93977

Example:

RP/0/0/CPU0:R1#sh route 192.168.4.4/32
Sat Dec 24 02:58:47.633 UTC
Routing entry for 192.168.4.4/32
  Known via "ospf 1", distance 110, metric 4, type intra area
  Installed Dec 24 02:58:29.065 for 00:00:18
  Routing Descriptor Blocks
    192.168.12.2, from 192.168.4.4, via GigabitEthernet0/0/0/0
      Route metric is 4
  No advertising protos.

Actual outgoing interface is GigabitEthernet0/0/0/0

  
RP/0/0/CPU0:R1#sh cef 192.168.4.4/32 
Sat Dec 24 03:05:27.636 UTC
192.168.4.4/32, version 9, internal 0x1000001 0x0 (ptr 0xa1413af4) [2], 0x0 (0xa13f941c), 0x0 (0x0)
 Updated Dec 24 02:58:29.085 
 local adjacency 192.168.12.2
 Prefix Len 32, traffic index 0, precedence n/a, priority 1
   via 192.168.12.2/32, GigabitEthernet0/0/1/1, 6 dependencies, weight 0, class 0 [flags 0x0]
    path-idx 0 NHID 0x0 [0xa1051100 0x0]
    next hop 192.168.12.2/32
    local adjacency

Packet forward to incorrect outgoing interface GigabitEthernet0/0/1/1, thus traffic blackhole.

RP/0/0/CPU0:R1#show cef ipv4 summary 
Sat Dec 24 03:05:44.395 UTC
Router ID is 192.168.1.1
IP CEF with switching (Table Version 0) for node0_0_CPU0
  Load balancing: L3
  Tableid 0xe0000000 (0xa0f25468), Vrfid 0x60000000, Vrid 0x20000000, Flags 0x1019
  Vrfname default, Refcount 39
  0 pkts received from core card
  0 CEF route update drops, 0 revisions of existing leaves
  1 CEF route update drops due to version mis-match
  Resolution Timer: 15s
  0 prefixes modified in place
  0 deleted stale prefixes
  0 prefixes with label imposition, 0 prefixes with label information

Then CEF mis-match counter increase.

Bug Fix

To fix this problem, we have to install corresponding SMU (patch file) into the problem routers. Unfortunately, it is not a easy thing in the real world with true of facts below:

  • There no SMU available on certain old software versions
  • This is an interruptive SMU, and the patch will take effect only after reload the routers.
  • There are may be over hundred or thousand of routers on your network.
  • Similar CEF bug may appear again on future software version.

Workaround

If it is not possible to apply the bug fix, then the workaround of “clear CEF table” will be another option.

Whenever we observe CEF mis-match counter increase rapidly which indicate bug is triggered, we can apply below command to mitigate the problem.

Workaround (clear CEF table)
RP/0/0/CPU0:R1#process restart ipv4_rib

Optional (reset CEF counter)
RP/0/0/CPU0:R1#clear cef linecard location XXX

Purpose – Proactive Monitor

So, how can we monitor the CEF counters from hundred or thousand of routers in an effective way proactively? Regular network management tools, like SNMP & NetFlow servers are not very helpful this time. Therefore, this is time to write a script.

First, put the “router operation status backup script” that we prepared previously (previous blog) to CRON job and run everyday, then we will get some operation status (show cef ipv4 summary) archive files inside Linux box.

This Flask app is used to search a specific pattern/keywords for “cef mis-mismatch” from a set of router operation status history on TWO specific dates. Then compare the value difference and generate a report.

Thus, we will have a full picture how serious our network suffer from CEF mis-match issue. This is a quite useful proactive monitoring tool for NOC engineers to inspect CEF bug issue on global SP network.

 

File Structure

[gary@pytest Lab-01]$ tree
.
├── cefAudit.py
├── router.db
├── router.ini
├── templates
│   ├── cefAudit.html
├── status.archive
│   ├── hkr1.test.com.conf.2016.12.1
│   ├── hkr1.test.com.conf.2016.12.24
│   ├── hkr2.test.com.conf.2016.12.1
│   ├── hkr2.test.com.conf.2016.12.24
│   ├── jpr1.test.com.conf.2016.12.1
│   ├── jpr1.test.com.conf.2016.12.24
│   ├── jpr2.test.com.conf.2016.12.1
│   ├── jpr2.test.com.conf.2016.12.24
│   ├── sgr1.test.com.conf.2016.12.1
│   ├── sgr1.test.com.conf.2016.12.24
│   ├── ukr1.test.com.conf.2016.12.1
│   ├── ukr1.test.com.conf.2016.12.24
│   ├── usr1.test.com.conf.2016.12.1
│   └── usr1.test.com.conf.2016.12.24

[gary@pytest Lab-01]$ cat status.archive/hkr1.test.com.conf.2016.12.24 | grep mis-match
  645 CEF route update drops due to version mis-match

INI Configuration File

[gary@pytest Lab-01]$ cat router.ini 
[General]
dir_stat:       status.archive

Device List

[gary@pytest Lab-01]$ cat router.db 
r1.test.com:juniper:up:vmx:core
r2.test.com:juniper:up:vmx:core
r3.test.com:cisco:up:iosv:core
r4.test.com:cisco:up:iosv:core
hkr1.test.com:cisco-xr:up:xrv:core
jpr1.test.com:cisco-xr:up:xrv:core
sgr1.test.com:cisco-xr:up:xrv:core
ukr1.test.com:cisco-xr:up:xrv:core
usr1.test.com:cisco-xr:up:xrv:core

 Flask Template

[gary@pytest Lab-01]$ cat templates/cefAudit.html 
<!doctype html>
<html>
    <body>
    <h2>Audit CEF route version mis-match issue on Cisco XR router</h2>
    <table>
        <tr>
            <th align="left">From Date: (e.g. 2016.1.30)</th>
            <th align="left">To Date: (e.g. 2016.1.31)</th>
            <th align="left">Sort by:</th>
        </tr>
        <tr>
            <form action="{{ url_for('.my_search_post') }}" method="POST">
            <th align="left"><input type="text" name="from_date"  size="30" value={{ from_date }}></th>
            <th align="left"><input type="text" name="to_date" size="30" value={{ to_date }}></th>
            <th align="left">
                <select name='sort_by' style="width:150px" onchange='this.form.submit()'>
                    <option {% if sort_by==2 %}selected{% endif %} value=2>Increment</option>
                    <option {% if sort_by==1 %}selected{% endif %} value=1>Date</option>
                </select>
            </th>
            <th><input type="submit" name="my-form" value="Search"></th>
            </form>
        </tr>
    </table>
    <br>
    <pre>{{ search_result }}</pre>
    </body>
</html>

Template Layout

2016-12-24 09_04_44-10.0.14.22 - Remote Desktop Connection.png

 Main Flask Program

[gary@pytest Lab-01]$ cat cefAudit.py
import time, sys, datetime, glob, os, re, ConfigParser
from operator import itemgetter, attrgetter
from flask import Flask, Markup, render_template, Response, request, redirect, url_for, Blueprint


cefAudit = Flask(__name__)


class core(object):
    today = datetime.datetime.now()
    yesterday = today - datetime.timedelta(days=1)

    to_date = "%i.%i.%i" % (today.year, today.month, today.day)
    from_date = "%i.%i.%i" % (yesterday.year, yesterday.month, yesterday.day)
    sort_by = 2

    def __init__(self):
        self.keyword = "CEF route update drops due to version mis-match"
        self.cef_matrix = {}
        self.rtr_match = 0
        self.rtr_mismatch = 0
        self.miss_archive = 0
        self.rtr_total = 0
        self.report = ''

    def read_webform(self):
        core.from_date = request.form['from_date']
        core.to_date = request.form['to_date']
        core.sort_by = request.form['sort_by']

    def read_ini(self):
        config = ConfigParser.ConfigParser()
        config.read("router.ini")
        self.config_dir = config.get('General', 'dir_stat')

        self.addr_from = config.get('CEF Audit Email', 'addr from')
        self.addr_to_test = config.get('CEF Audit Email', 'addr to test').split()
        self.addr_to_live = config.get('CEF Audit Email', 'addr to live').split()
        self.subject = config.get('CEF Audit Email', 'subject')

    def read_device(self):
        with open("router.db", "r") as f:
            devices = f.read().splitlines()
            devices = filter(None,devices)

        for device in devices:
            device = device.split(':')
            hostname = device[0]
            software = device[1]
            status = device[2]
            network = device[4]
            if software == "cisco-xr" and status == "up" and network=="core":
                self.cef_matrix[hostname] = [-1, -1, -1]

    def read_backup_fromdate(self):
        for hostname in self.cef_matrix:
            file = self.config_dir + '/' + hostname + '.conf.' + core.from_date
            try:
                with open(file, "r") as f:
                    filedata = f.read().splitlines()
                    filedata = filter(None, filedata)

                for line in filedata:
                    if re.search(self.keyword, line):
                        cef_value = line.strip().split(' ')[0]
                        self.cef_matrix[hostname][0] = int(cef_value)
            except:
                pass


    def read_backup_todate(self):
        for hostname in self.cef_matrix:
            file = self.config_dir + '/' + hostname + '.conf.' + core.to_date
            try:
                with open(file, "r") as f:
                    filedata = f.read().splitlines()
                    filedata = filter(None, filedata)

                for line in filedata:
                    if re.search(self.keyword, line):
                        cef_value = line.strip().split(' ')[0]
                        self.cef_matrix[hostname][1] = int(cef_value)
            except:
                pass

    def calcuation(self):
        for hostname in self.cef_matrix:
            self.rtr_total += 1

            if self.cef_matrix[hostname][1] > 0:
                self.rtr_mismatch += 1
            elif self.cef_matrix[hostname][1] == 0:
                self.rtr_match += 1
            elif self.cef_matrix[hostname][1] < 0:
                self.miss_archive += 1

            if self.cef_matrix[hostname][0] != -1 and self.cef_matrix[hostname][1] != -1:
                self.cef_matrix[hostname][2] = self.cef_matrix[hostname][1] - self.cef_matrix[hostname][0]

    def render_report(self):
        cef_matrix2 = sorted(self.cef_matrix.items(), key=lambda i: i[1][int(core.sort_by)], reverse=True)

        self.report += 'SUMMARY\n'
        self.report += 'CEF health'.ljust(20) + '= ' + str(self.rtr_match) + '\n'
        self.report += 'CEF mis-match'.ljust(20) + '= ' + str(self.rtr_mismatch) + '\n'
        self.report += 'Backup missing'.ljust(20) + '= ' + str(self.miss_archive) + '\n'
        self.report += '--------------------------\n'
        self.report += 'Total XR routers'.ljust(20) + '= ' + str(self.rtr_total) + '\n\n'

        if cef_matrix2 != []:
            self.report += 'Router Name'.ljust(40) + self.from_date.ljust(15) + self.to_date.ljust(15) + 'Increment' + '\n'

            for item in cef_matrix2:
                if item[1][1] != 0:
                    item_str = item[0].ljust(40) + str(item[1][0]).ljust(15) + str(item[1][1]).ljust(15) + str(item[1][2]) + '\n'
                    self.report += item_str.encode()

            self.report += "\nRemark:\n"
            self.report += "- Those routers not on the list imply they are health with counter value of \"0\"\n"
            self.report += "- The CEF value of \"-1\" imply no data read.\n\n"


@cefAudit.route('/')
def my_search():
    today = datetime.datetime.now()
    yesterday = today - datetime.timedelta(days=1)

    to_date = "%i.%i.%i" % (today.year, today.month, today.day)
    from_date = "%i.%i.%i" % (yesterday.year, yesterday.month, yesterday.day)

    return render_template('cefAudit.html', from_date=from_date, to_date=to_date)


@cefAudit.route('/', methods=['POST'])
def my_search_post():
    search = core()
    search.read_webform()
    search.read_ini()
    search.read_device()
    search.read_backup_fromdate()
    search.read_backup_todate()
    search.calcuation()
    search.render_report()

    return render_template('cefAudit.html', from_date=search.from_date, to_date=core.to_date, sort_by=int(search.sort_by), search_result=search.report)

    
if __name__ == '__main__':
    retrieve.run(host='0.0.0.0')

Output

2017-01-09 11_34_03-http___10.0.14.22_5000_cefAudit_ - Internet Explorer.png

2017-01-09 11_34_24-http___10.0.14.22_5000_cefAudit_ - Internet Explorer.png

The report tells us should take action on ukr1.test.com immediately, otherwise we will receive fault cases report very soon !!

 

Posted in Flask, IOS-XR Router | Tagged | Leave a comment

Search a pattern from set of router history configuration over web

Put the “router configuration / operation status backup script” that we prepared previously to cron job and run everyday, then we will get some configuration / operation status archive files inside Linux box.

This Flask app is used to search some patterns/keywords from a set of router history on specific date.

Features:

  • Patterns/keywords support regular expression.
  • Host name support wildcard character.

 

File Structure

[gary@pytest Lab-01]$ tree
.
├── grepHistory.py
├── router.db
├── router.ini
├── templates
│   ├── grepHistory.html
├── config.archive
│   ├── r1.test.com.conf.2016.12.24
│   ├── r2.test.com.conf.2016.12.24
│   ├── r3.test.com.conf.2016.12.24
│   ├── r4.test.com.conf.2016.12.24
│   ├── r5.test.com.conf.2016.12.24
│   └── r6.test.com.conf.2016.12.24
├── status.archive
│   ├── r1.test.com.conf.2016.12.24
│   ├── r2.test.com.conf.2016.12.24
│   ├── r3.test.com.conf.2016.12.24
│   ├── r4.test.com.conf.2016.12.24
│   ├── r5.test.com.conf.2016.12.24
│   └── r6.test.com.conf.2016.12.24

INI Configuration File

[gary@pytest Lab-01]$ cat router.ini 
[General]
dir_conf:       config.archive
dir_stat:       status.archive

Flask Template

[gary@pytest Lab-01]$ cat templates/grepHistory.html 
<!doctype html>
<html>
    <body>
    <h2>Grep a pattern from set of router history configuration / operation status</h2>
    <table>
        <tr>
            <th align="left">Pattern: (e.g. interface.+HK.+JP)</th>
            <th align="left">History Type</th>
            <th align="left">Router Name: (e.g. hkrouter*)</th>
            <th align="left">Date: (e.g. 2016.1.31)</th>
        </tr>
        <tr>
            <form action="{{ url_for('.my_search_post') }}" method="POST">
            <th align="left"><input type="text" name="keyword"  size="50" value="{{ keyword }}"></th>
            <th align="left">
                <select name='history_type' style="width:150px" onchange='this.form.submit()'>
                    <option {% if history_type=="config" %}selected{% endif %} value='config'>Configuration</option>
                    <option {% if history_type=="status" %}selected{% endif %} value='status'>Operation Status</option>
                </select>
            </th>
            <th align="left"><input type="text" name="router_name"  size="35" value={{ router_name }}></th>
            <th align="left"><input type="text" name="date_input" size="25" value={{ date_input }}></th>
            <th><input type="submit" name="my-form" value="Search"></th>
            </form>
        </tr>
    </table>
    <br>
    <pre>{{ search_result }}</pre>
    </body>
</html>

Template Layout

2016-12-24 01_42_18-10.0.14.22 - Remote Desktop Connection.png

Main Flash Program

[gary@pytest Lab-01]$ cat grepHistory.py
import datetime, glob, os, re, ConfigParser
from flask import Flask, Markup, render_template, Response, request, redirect, url_for, Blueprint

grepHistory = Flask(__name__)

class main_class(object):
    def __init__(self):
        self.matches = []
        self.results = []
        self.match_max = 50000
        self.match_count = 0
        self.max_host_len = 0

    def read_webform(self):
        self.keyword = request.form['keyword']
        self.keyword = self.keyword.lstrip("*")
        self.history_type = request.form['history_type']
        self.router_name = request.form['router_name']
        self.date_input = request.form['date_input']

    def read_ini(self):
        config = ConfigParser.ConfigParser()
        config.read("router.ini")

        if self.history_type == "config":
            self.config_dir = config.get('General', 'dir_conf')
        elif self.history_type == "status":
            self.config_dir = config.get('General', 'dir_stat')

    def read_backup(self):
        files = glob.glob(self.config_dir + "/" + self.router_name + ".*.conf." + self.date_input)

        for file in files:
            hostname = file.replace(self.config_dir + "/", "")

            with open(file, "r") as f:
                filedata = f.read().splitlines()
                filedata = filter(None, filedata)

            for line in filedata:
                if re.search(self.keyword, line):
                    self.matches.append([hostname, line.decode('utf-8','ignore')])
                    self.match_count += 1

                    if len(hostname) > self.max_host_len:
                        self.max_host_len = len(hostname)

                    if self.match_count > self.match_max:
                        break

    def render_report(self):
        if self.match_count <= self.match_max:
            if len(self.matches) != 0:
                for match in self.matches:
                    match[0] = match[0] + ":"
                    result = match[0].ljust(self.max_host_len+3) + match[1]
                    self.results.append(result)

                self.report = "Number of match = %s\n\n" % str(self.match_count)
                self.report += "\n".join(self.results)

            else:
                self.report = "No match found"

        else:
            self.report = "Over %s matches, too much!" % str(self.match_max)

@grepHistory.route('/')
def my_search():
    now = datetime.datetime.now()
    date_today = "%i.%i.%i" % (now.year, now.month, now.day)

    return render_template('grepHistory.html', router_name="*", date_input=date_today)

@grepHistory.route('/', methods=['POST'])
def my_search_post():
    search = main_class()
    search.read_webform()
    search.read_ini()
    search.read_backup()
    search.render_report()

    return render_template('grepHistory.html', keyword=search.keyword, history_type=search.history_type, router_name=search.router_name, date_input=search.date_input, search_result=search.report)

if __name__ == '__main__':
    grepHistory.run(host='0.0.0.0')

Output

2016-12-24 01_50_46-http___10.0.14.22_5000_grepHistory_ - Internet Explorer.png

2016-12-24 01_52_43-http___10.0.14.22_5000_grepHistory_ - Internet Explorer.png

Posted in Flask | Tagged | Leave a comment

Retrieve a router history configuration / operation status over web

Put the “router configuration / operation status backup script” that we prepared previously to cron job and run everyday, then we will get some configuration / operation status archive files inside Linux box.

This Flask app is used to retrieve a specific router history on specific date via web browser .

File Structure

[gary@pytest Lab-01]$ tree
.
├── checkHistory.py
├── router.db
├── router.ini
├── templates
│   ├── checkHistory.html
├── config.archive
│   ├── r1.test.com.conf.2016.12.24
│   ├── r2.test.com.conf.2016.12.24
│   ├── r3.test.com.conf.2016.12.24
│   ├── r4.test.com.conf.2016.12.24
│   ├── r5.test.com.conf.2016.12.24
│   └── r6.test.com.conf.2016.12.24
├── status.archive
│   ├── r1.test.com.conf.2016.12.24
│   ├── r2.test.com.conf.2016.12.24
│   ├── r3.test.com.conf.2016.12.24
│   ├── r4.test.com.conf.2016.12.24
│   ├── r5.test.com.conf.2016.12.24
│   └── r6.test.com.conf.2016.12.24

INI Configuration File

[gary@pytest Lab-01]$ cat router.ini 
[General]
dir_conf:       config.archive
dir_stat:       status.archive

Flask Template

[gary@pytest Lab-01]$ cat templates/checkHistory.html 
<!doctype html>
<head>
    <meta charset="UTF-8">
</head>
<html>
    <body>
    <h2>Retrieve a history configuration / operation status</h2>
    <table>
        <tr>
            <th align="left">Router Name: (e.g. hkrouter01)</th>
            <th align="left">History Type</th>
            <th align="left">Date: (e.g. 2016.1.31)</th>
        </tr>
        <tr>
            <form action="{{ url_for('.my_search_post') }}" method="POST">
            <th align="left"><input type="text" name="router_name"  size="40" value={{ router_name }}></th>
            <th align="left">
                <select name='history_type' style="width:150px" onchange='this.form.submit()'>
                    <option {% if history_type=="config" %}selected{% endif %} value='config'>Configuration</option>
                    <option {% if history_type=="status" %}selected{% endif %} value='status'>Operation Status</option>
                </select>
            </th>
            <th align="left"><input type="text" name="date_input" size="25" value={{ date_input }}></th>
            <th><input type="submit" name="my-form" value="Search"></th>
            </form>
        </tr>
    </table>
    </br>

    <pre>{{ search_result }}</pre>

    </body>
</html>

Template Layout

2016-12-24 01_04_55-10.0.14.22 - Remote Desktop Connection.png

Main Flask Program

[gary@pytest Lab-01]$ cat checkHistory.py
import datetime, ConfigParser, glob, os
from flask import Flask, Markup, render_template, Response, request, redirect, url_for, Blueprint

checkHistory = Flask(__name__)

class main_class(object):
    def __init__(self):
        self.report = ""

    def read_webform(self):
        self.router_name = request.form['router_name']
        self.router_name = self.router_name.replace("*","")
        self.router_name = self.router_name.split(".")[0]
        self.history_type = request.form['history_type']
        self.date_input = request.form['date_input']

    def read_ini(self):
        config = ConfigParser.ConfigParser()
        config.read("router.ini")

        if self.history_type == "config":
            self.config_dir = config.get('General', 'dir_conf')
        elif self.history_type == "status":
            self.config_dir = config.get('General', 'dir_stat')

    def read_history(self):
        files = glob.glob(self.config_dir + "/" + self.router_name + ".*." + self.date_input)

        if files != []:
            for file in files:
                with open(file, "r") as f:
                    filedata = f.read().splitlines()
                    filedata = filter(None, filedata)

                # convert to byte to unicode
                for i, line in enumerate(filedata):
                    filedata[i] = line.decode('utf-8', 'ignore')

                self.report += "\n".join(filedata) + "\n"

        else:
            self.report = "File not found"

@checkHistory.route('/')
def my_search():
    now = datetime.datetime.now()
    date_today = "%i.%i.%i" % (now.year, now.month, now.day)

    return render_template('checkHistory.html', date_input=date_today)

@checkHistory.route('/', methods=['POST'])
def my_search_post():
    retrieve = main_class()
    retrieve.read_webform()
    retrieve.read_ini()
    retrieve.read_history()

    return render_template('checkHistory.html', router_name=retrieve.router_name, history_type=retrieve.history_type, date_input=retrieve.date_input, search_result=retrieve.report)

if __name__ == '__main__':
    retrieve.run(host='0.0.0.0')

 Output

2016-12-24-01_18_29-http___10-0-14-22_5000_checkhistory_-internet-explorer

2016-12-24-01_19_54-http___10-0-14-22_5000_checkhistory_-internet-explorer

Posted in Flask | Tagged | Leave a comment