How is the Management VRF special ? 2018-05-24T08:53:32Z 2017-01-25T17:13:43Z 7429955 7429955 How is the Management VRF special ? 2018-05-24T08:53:32Z 2017-01-25T17:13:43Z Sylvain Munaut sylvain_munaut Hi,<br /><br />I'm wondering what being a mgmt VRF implied exactly.<br /><br />Mostly what's different between :<br />&nbsp;* a mgmt VRF + using the default one for "public internet" routing.<br />&nbsp;* Using default one for mgmt + using a data plane VRF for "public internet".<br /><br />Currently I was more leaning toward the second solution since it seems to have less limitation (i.e. not limited to eth0 for instance), also if I ever want OSPF for my internal private network, that part has to be in the default one (not sure if VRF OSPF is planned for the future ?).<br /><br /><br />Here's a bit of explanation about my setup if needed for context.<br /><br />&nbsp;- 2 * 10G switches running cumulus with mlags to the hosts.<br /><br />&nbsp;- The switches 'eth0' are connected to an 'emergency' gateway that will only be used as a last resort. Most likely we won't administer them through there using something goes very wrong.<br /><br />&nbsp;- What we actually call "management network" that has the machine BMCs and such is just a separate VLAN but connected to the same switches (there is way too many ports so we can afford to 'waste' a few 10G port with 1G copper for that). The switches would have a L3 interface on that VLAN and that's where they would be managed from most of the time and also get their 'management' internet access (fetch packages ...)<br /><br />&nbsp;- Most machines (physical &amp; virtual) only have private RFC1918 IPs (in the management VLANs and most other VLANs for inter VM communication). And I'm trying to keep that as separate of the "public internet" side of things as possible. Machines on that private network get their internet access from routers "on a stick" on those switches (doing the NAT, firewall, ...)<br /><br />&nbsp;- Routes in those private IP vlans are redistributed using OSPF (to other sites) and there are dedicated routers that handle that. So technically the switch don't have to participate to OSPF, but maybe in the future ...<br /><br />&nbsp;- Those switches act as our border routers to the public internet (but they only receive default routes + a few more specific ones). I would put everything BGP and public IP stuff in a separate VRF. <br /><br />Maybe actually doing both options at once is worth considering:<br />&nbsp;- Use the mgmt VRF for the 'emergency access' so it has its own routes.<br />&nbsp;- Use the default VRF for our RFC1918 internal network<br />&nbsp;- Use a data plane VRF for all the BGP public internet stuff.<br /><br /><br />Cheers,<br /><br />&nbsp;&nbsp; Sylvain question 10 6 1 comment 18340588 Sean Cavanaugh responded to "How is the Management VRF special ?" 2017-01-27T19:45:17Z 2017-01-27T19:45:17Z Sean Cavanaugh sean_cavanaugh_7205989 The mgmt vrf is a special vrf for eth0 (or eth1 on some switch platforms). &nbsp;The eth ports are software only ports that are not hardware accelerated.<br /><br /><blockquote>&nbsp;- The switches 'eth0' are connected to an 'emergency' gateway that will only be used as a last resort. Most likely we won't administer them through there using something goes very wrong.</blockquote>You only want OOB (out of band) traffic to use your eth0, whether or not you have a VRF or not. &nbsp;This is b/c this port is SW switched, not HW switched.<br /><br />Where a VRF helps is when you have multiple matching routes. &nbsp;For example you could have a default route ( on eth0 you got from your DHCP server, and a default from your internet service provider for your data plane traffic. &nbsp;You now have two routes<br /><ul><li> via DHCP (Kernel Route) Admin Distance 0</li><li> via OSPF or BGP, Admin Distance &gt; 1</li></ul>What happens is your Dynamic routing protocol never gets installed.... &nbsp;VRF gives you two route tables so you can have overlapping routes.<br /><br />The rest of your questions don't seem to go along with VRF. &nbsp;What is the exactly problem? &nbsp;Did my answer help here? &nbsp;Maybe we can dive down a deeper level now. 1 18340793 Pete B responded to "How is the Management VRF special ?" 2017-01-27T20:45:14Z 2017-01-27T20:45:14Z Pete B pete_b_7033546 I like this explanation a lot, Sean. Mind if I repurpose it for the docs?&nbsp; 0 18340821 Sean Cavanaugh responded to "How is the Management VRF special ?" 2017-01-27T20:52:20Z 2017-01-27T20:52:20Z Sean Cavanaugh sean_cavanaugh_7205989 lol sure 0 18418903 Sylvain Munaut responded to "How is the Management VRF special ?" 2017-02-14T15:10:18Z 2017-02-14T15:10:18Z Sylvain Munaut sylvain_munaut Yeah, I'm aware the eth0 port should only be used for OOB administration.<br /><br />But one of the thing I was wondering was if I could use a front-panel for instead for day-to-day administration. But turns out it seems like a bad idea because (1) a bunch of rate limit rules are applied to traffic from front panel ports to the linux host, so this needs quite a bit of default config override&nbsp; (2) Traffic using those ports goes through switchd which tends to use CPU time.<br /><br />So if I want the switch to be administrable using two different path (eth0 itself is not redudant, I want to have two path in case one goes down ...), I'm better off using eth0 for "day-to-day" administration and access and then use a front-panel-port (SVI) for "emergency / recovery" in case my eth0 link is screwed up.<br /><br />The other think I was wondering is how / where the special name 'mgmt' is matched and what kind of different behavior does it trigger vs naming it 'admin' (just an examples). That name is apparently special and triggers some different part of the code, I'm wondering which ones.<br /><br />As for my reasons for using VRFs, it is mostly security / isolation. To make sure some misconfig couldn't lead to packets from the internet leaking into my internal network (since the switch is directly connected to upstreams providers and I can't trust anything coming from there). It also allows me to make sure all the "apps" running on the switch ( ntp / smtp / ... ) would go through our internal router/firewall rather than directly to the upstream providers. 0 18418973 Eric Pulvino responded to "How is the Management VRF special ?" 2017-02-14T15:32:22Z 2017-02-14T15:32:22Z Eric Pulvino eric_pulvino The mgmt name triggers special handling of DNS traffic specifically if you're using the MGMT vrf and run the "ip rule ls" command, you'll see that when applied the mgmt vrf builds a special rule to send traffic from your currently configured DNS server out the eth0 port (instead of out your front-panel ports). From talking to David Ahern, I understand that will be a configurable setting soon but that is the only piece of specialness I'm aware of. I'll send this thread over to David Ahern in case I've missed something. (I'm sure I have). 0 18418989 Sylvain Munaut responded to "How is the Management VRF special ?" 2017-02-14T15:38:02Z 2017-02-14T15:38:02Z Sylvain Munaut sylvain_munaut Ok, thanks really good to know.<br /><br />Is that rule inserted in the HW ? (i.e. if I have packets being routed from one front-panel port to another toward the DNS server IP, will they be affected) 0 18419139 David Ahern responded to "How is the Management VRF special ?" 2017-02-14T16:19:13Z 2017-02-14T16:19:13Z David Ahern david_ahern The "mgmt" name is special cased to identify the Management VRF from a data plane VRF. As Eric mentioned, FIB rules are installed for DNS servers since that is the usual deployment case. In addition, the user shell is set to the Management VRF context at login. This allows admin tools like ansible, chef, apt-get to Just Work over the management plane with no change in how the command is run.&nbsp;&nbsp;It really comes down to making Management VRF transparent and easy, especially for new users doing a typical deployment.&nbsp; 0 19541830 Richard Pilsbury responded to "How is the Management VRF special ?" 2018-05-24T05:31:56Z 2018-05-24T05:31:56Z Richard Pilsbury richard_pilsbury Hi David,<br /><br />Can you explain&nbsp;<i>how&nbsp;</i>the user shell is set to the management VRF? i.e. what config changes are made? I am trying to use a management VRF on a non-cumulus (still Debian-based) platform. I fixed the DNS issue with an ip rule, but am now facing the problem that apt etc. still tries to use the default table. 0 19541839 David Ahern responded to "How is the Management VRF special ?" 2018-05-24T05:42:13Z 2018-05-24T05:42:13Z David Ahern david_ahern Hi Richard:<br /><br />libpam-script is used to check if mgmt VRF is enabled. If so, it sets login shells to the mgmt VRF context.<br /><br />For non-CL platforms I suggest taking a look at my OSS slides:<br /><a href="" rel="nofollow" target="_blank" title="Link:"></a><br /><br />And in particular the vrf + mgmt-vrf packages in&nbsp;<br /><a href="" rel="nofollow"></a><br /><br />(building a deb or rpm and installing that way is best - it handles the libpam-script dependencies). 0 19542005 Richard Pilsbury responded to "How is the Management VRF special ?" 2018-05-24T08:53:32Z 2018-05-24T08:53:32Z Richard Pilsbury richard_pilsbury Fantastic - that's sorted it. Thanks for your help (and the really quick reply on a year old thread!). 0