http://m.getsatisfaction.com/topics/7729093 Traceroute & pings to IPs defined for use in mgmt VRF always use mgmt VRF routing table even when sourcing from non mgmt VRF interfaces 2018-05-04T14:17:44Z 2018-05-01T14:38:12Z 7729093 http://m.getsatisfaction.com/topics/7729093 7729093 Traceroute & pings to IPs defined for use in mgmt VRF always use mgmt VRF routing table even when sourcing from non mgmt VRF interfaces 2018-05-04T14:17:44Z 2018-05-01T14:38:12Z Jeff Greenfield http://m.getsatisfaction.com/people/9648269 https://www.gravatar.com/avatar/e12bffeeb76c501b8650d6c94e35f1fe?d=identicon&s=55&r=PG jeff_greenfield I've found an odd behavior I am wondering if others are experiencing. I've assigned NTP, DNS, SNMP etc.. to use eth0 via mgmt VRF.&nbsp; &nbsp;When I try and trace or ping to those IPs sourcing a non mgmt VRF interface or IP it still uses the eth0 mgmt VRF to exit;<br /><br />cumulus@switch1:~$ cat /etc/resolv.conf<br />nameserver 205.206.214.249 # vrf mgmt<br />nameserver 209.20.8.249 # vrf mgmt<br /><br /><br />cumulus@switch1:~$ net show route vrf mgmt<br /><br />show ip route vrf mgmt<br />=======================<br />Codes: K - kernel route, C - connected, S - static, R - RIP,<br />&nbsp; &nbsp; &nbsp; &nbsp;O - OSPF, I - IS-IS, B - BGP, P - PIM, E - EIGRP, N - NHRP,<br />&nbsp; &nbsp; &nbsp; &nbsp;T - Table, v - VNC, V - VNC-Direct, A - Babel,<br />&nbsp; &nbsp; &nbsp; &nbsp;&gt; - selected route, * - FIB route<br /><br />K * 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 02w0d20h<br />K&gt;* 0.0.0.0/0 [0/0] via 198.162.145.254, eth0, 02w0d20h<br />C&gt;* 198.162.145.0/24 is directly connected, eth0, 02w0d20h<br /><br />Mgmt VRF:<br />traceroute 205.206.214.249<br />traceroute to 205.206.214.249 (205.206.214.249), 30 hops max, 60 byte packets<br />&nbsp;1&nbsp; 198.162.145.254 (198.162.145.254)&nbsp; 0.570 ms&nbsp; 0.631 ms&nbsp; 0.731 ms<br />&nbsp;2&nbsp; 172.25.201.33 (172.25.201.33)&nbsp; 0.329 ms&nbsp; 0.340 ms&nbsp; 0.329 ms<br />&nbsp;3&nbsp; 172.26.31.2 (172.26.31.2)&nbsp; 16.363 ms&nbsp; 16.376 ms&nbsp; 16.365 ms<br />&nbsp;4&nbsp; 172.26.31.1 (172.26.31.1)&nbsp; 15.779 ms&nbsp; 15.768 ms&nbsp; 15.724 ms<br /><br /><br />cumulus@switch1:~$ net show route<br /><br />show ip route<br />=============<br />Codes: K - kernel route, C - connected, S - static, R - RIP,<br />&nbsp; &nbsp; &nbsp; &nbsp;O - OSPF, I - IS-IS, B - BGP, P - PIM, E - EIGRP, N - NHRP,<br />&nbsp; &nbsp; &nbsp; &nbsp;T - Table, v - VNC, V - VNC-Direct, A - Babel,<br />&nbsp; &nbsp; &nbsp; &nbsp;&gt; - selected route, * - FIB route<br /><br />B&gt;* 0.0.0.0/0 [200/0] via <b>172.26.40.66</b>, vlan3700, 02w0d20h<br />C * 172.26.39.64/27 is directly connected, vlan219-v0, 02w0d20h<br />C&gt;* 172.26.39.64/27 is directly connected, vlan219, 02w0d20h<br /><br />Global table: Wrong exit interface with IP defined for DNS using global default route<br />traceroute -s 172.26.39.67 205.206.214.249<br />traceroute to 205.206.214.249 (205.206.214.249), 30 hops max, 60 byte packets<br />&nbsp;1&nbsp; <b>198.162.145.254 (198.162.145.254)</b>&nbsp; 17.857 ms&nbsp; 17.975 ms&nbsp; 17.983 ms<br />&nbsp;2&nbsp; 172.25.201.33 (172.25.201.33)&nbsp; 17.644 ms&nbsp; 17.709 ms&nbsp; 17.793 ms<br />&nbsp;3&nbsp; 172.26.31.2 (172.26.31.2)&nbsp; 17.744 ms&nbsp; 17.780 ms&nbsp; 17.792 ms<br />&nbsp;4&nbsp; 172.26.31.1 (172.26.31.1)&nbsp; 17.861 ms&nbsp; 17.895 ms&nbsp; 17.850 ms<br /><br />Global&nbsp; table: Correct exit interface with IP in same subnet as DNS using global default route<br />&nbsp;traceroute -s 172.26.39.67 205.206.214.250<br />traceroute to 205.206.214.250 (205.206.214.250), 30 hops max, 60 byte packets<br />&nbsp;1&nbsp; <b>172.26.40.66 (172.26.40.66)</b>&nbsp; 0.783 ms&nbsp; 0.754 ms&nbsp; 0.756 ms<br />&nbsp;2&nbsp; 172.26.40.60 (172.26.40.60)&nbsp; 18.067 ms&nbsp; 18.070 ms&nbsp; 18.060 ms<br />&nbsp;3&nbsp; 172.26.40.20 (172.26.40.20)&nbsp; 16.028 ms&nbsp; 16.051 ms&nbsp; 16.051 ms<br />&nbsp;4&nbsp; 172.26.31.1 (172.26.31.1)&nbsp; 16.053 ms&nbsp; 16.274 ms&nbsp; 16.059 ms question 3 2 1 comment http://m.getsatisfaction.com/topics/7729093/replies/19500040 http://m.getsatisfaction.com/topics/7729093 19500040 Eric Pulvino responded to "Traceroute & pings to IPs defined for use in mgmt VRF always use mgmt VRF routing table even when sourcing from non mgmt VRF interfaces" 2018-05-03T03:04:35Z 2018-05-03T03:04:35Z Eric Pulvino http://m.getsatisfaction.com/people/7795369 https://d2r1vs3d9006ap.cloudfront.net/public/uploaded_images/10746583/705A1674-300_medium.jpg eric_pulvino Jeff this is due to a little documented behavior of our VRF implementation on Cumulus specifically.... check the output of the "ip rule ls" command on your system and you'll see an override for the IP addresses of your DNS servers. When the vrf interface corresponding to the mgmt VRF is brought up it creates the ip rule behaviors you see. Essentially IP rule is sort of like PBR which short circuits lookups for these IP addresses to use a specific table. This can get tedious when you want your DNS to use one VRF and your NTP to use another but they share the same IP address. Worth noting is that this behavior only exists in the control plane and won't affect traffic moving through the switch in the dataplane. In order to disable this behavior, add the following lines under your mgmt vrf interface:<br /><br /><pre>auto mgmt<br />iface mgmt<br />&nbsp; &nbsp; vrf-table auto<br />&nbsp; &nbsp; post-up ip rule del from all to&nbsp;205&#46;206&#46;214&#46;249&nbsp;lookup mgmt<br />&nbsp; &nbsp; post-up ip rule del from all to&nbsp;209&#46;20&#46;8&#46;249&nbsp;lookup mgmt</pre> 0 http://m.getsatisfaction.com/topics/7729093/replies/19503017 http://m.getsatisfaction.com/topics/7729093/replies/19500040 19503017 Jeff Greenfield responded to "Traceroute & pings to IPs defined for use in mgmt VRF always use mgmt VRF routing table even when sourcing from non mgmt VRF interfaces" 2018-05-04T13:28:28Z 2018-05-04T13:28:28Z Jeff Greenfield http://m.getsatisfaction.com/people/9648269 https://www.gravatar.com/avatar/e12bffeeb76c501b8650d6c94e35f1fe?d=identicon&s=55&r=PG jeff_greenfield Thanks Eric, I'll be sure to document this in our BCP. Upon double checking it only seems to have this behaviour for the DNS IP's in mgmt VRF. I've configured SNMP &amp; NTP to use the mgmt VRF as well but they are not added to the "ip rule ls" feature. 0 http://m.getsatisfaction.com/topics/7729093/replies/19503112 http://m.getsatisfaction.com/topics/7729093/replies/19500040 19503112 Eric Pulvino responded to "Traceroute & pings to IPs defined for use in mgmt VRF always use mgmt VRF routing table even when sourcing from non mgmt VRF interfaces" 2018-05-04T14:17:44Z 2018-05-04T14:17:44Z Eric Pulvino http://m.getsatisfaction.com/people/7795369 https://d2r1vs3d9006ap.cloudfront.net/public/uploaded_images/10746583/705A1674-300_medium.jpg eric_pulvino That is correct, when the interface:<br /><pre>auto mgmt<br />iface mgmt<br />&nbsp; &nbsp; vrf-table auto</pre>Is brought up, it looks at the configured DNS servers in /etc/resolv.conf and adds the:<br /><pre>ip rule add from all to 205&#46;206&#46;214&#46;249 lookup mgmt&nbsp;<br />ip rule add from all to 209&#46;20&#46;8&#46;249 lookup mgmt </pre>rules to the configuration. 0