In this post, we’ll show you exactly how to deploy a fully connected 5-router topology in less than 15 minutes. Transforming your Juniper SRX 100 from an old forgotten box into a high-speed engine for learning and validation.

Before we begin, here is what you’ll need to execute this fast deployment:

• Core Hardware: Your Juniper SRX 100 device.
• Connectivity: Patch cables.
• Time: A clear 15-minute window for the configuration process.

Let’s dive in!

To deploy our 5-router lab in minutes, we will follow these three quick steps:

1. Loop two physical ports on SRX.
2. Configure Packet Mode and Disable Security Flow on Juniper SRX.
3. Create the 5-Router Topology with Virtual Router Instance type.

The target lab design topology is shown in Figure 1. below.

Figure 1. Lab Design Topology.

1. Loop two physical ports on SRX.

To create a 5-router lab on a single SRX, we need to create a closed loop that allows traffic to pass between our different virtual instances. Since the SRX 100 has limited interfaces, we’ll use a two physical ports with vlan tag sub-interfaces.

The Physical Connection

Use a standard patch cable to connect two adjacent physical ports on your SRX 100. For this lab, we’ll use:

• fe-0/0/6
• fe-0/0/7

Figure 2: The custom UTP CAT 5 patch cord used for the physical loop.

set interfaces fe-0/0/6 flexible-vlan-tagging
set interfaces fe-0/0/6 fastether-options auto-negotiation  
set interfaces fe-0/0/7 flexible-vlan-tagging
set interfaces fe-0/0/7 fastether-options auto-negotiation 
commit

This physical connection will act as our high-speed, internal backbone fabric, allowing the virtual routers what we will create later to talk to each other without needing external equipment.

Figure 3. SRX Ports Looped with UTP Cable.

2. Configure Packet Mode and Disable Security Flow Juniper SRX.
The SRX is a firewall, meaning it runs in flow mode by default, inspecting traffic and requiring security policies and zones. To turn it into a dedicated, multi-router platform that simply forwards packets, we must switch it to packet mode.

⚠️ Warning: This step removes all firewall functions and security services.

Switching the SRX to Router Mode (In Next 1 minute if we will not calculate SRX boot time)

• Delete Existing Security Configuration: We first wipe all pre-existing security settings that would conflict with packet mode.

delete security

• Enable Packet Mode: We instruct the Packet Forwarding Engine (PFE) to process traffic on a per-packet basis, like a traditional router. I like to enable for all address families.

set security forwarding-options family inet mode packet-based
set security forwarding-options family inet6 mode packet-based
set security forwarding-options family mpls mode packet-based
set security forwarding-options family iso mode packet-based

• Commit and Reboot, this change is fundamental and requires a system reboot to take effect.

commit and-quit
request system reboot

(Go grab a coffee while it reboots! You’ve already completed the hardest part.)

Now that our SRX is a pure router, we can segment it into five distinct logical routers using Virtual Router (VR) instances. A VR instance is a type of routing instance that creates completely separate routing tables and forwarding domains.

3. Create the 5-Router Topology with Virtual Router Instance type.

Topology Goal We will create five isolated routing domains (AS1 through AS4 and PE) and then connect them using logical interfaces to form a virtual network.

The Virtual Router Configuration (Next 5 minutes)

set routing-instances AS1 instance-type virtual-router
set routing-instances AS2 instance-type virtual-router
set routing-instances AS3 instance-type virtual-router
set routing-instances AS4 instance-type virtual-router
Global routing table on Juniper SRX will represent PE.

Figure 4: Representing logical connectivity between routing instances.

Next Step: Connecting the Virtual Routers

Below you can find the interface configuration for fe-0/0/6 and fe-0/0/7:

fe-0/0/6

set interfaces fe-0/0/6 unit 12 vlan-id 12
set interfaces fe-0/0/6 unit 12 family inet address 10.90.12.2/30
set interfaces fe-0/0/6 unit 13 vlan-id 13
set interfaces fe-0/0/6 unit 13 family inet address 10.90.13.2/30
set interfaces fe-0/0/6 unit 23 vlan-id 23
set interfaces fe-0/0/6 unit 23 family inet address 10.90.23.2/30
set interfaces fe-0/0/6 unit 24 vlan-id 24
set interfaces fe-0/0/6 unit 24 family inet address 10.90.24.2/30
set interfaces fe-0/0/6 unit 113 vlan-id 113
set interfaces fe-0/0/6 unit 113 family inet address 10.90.113.2/30
set interfaces fe-0/0/6 unit 114 vlan-id 114
set interfaces fe-0/0/6 unit 114 family inet address 10.90.114.2/30

fe-0/0/7

set interfaces fe-0/0/7 unit 12 vlan-id 12
set interfaces fe-0/0/7 unit 12 family inet address 10.90.12.1/30
set interfaces fe-0/0/7 unit 13 vlan-id 13
set interfaces fe-0/0/7 unit 13 family inet address 10.90.13.1/30
set interfaces fe-0/0/7 unit 23 vlan-id 23
set interfaces fe-0/0/7 unit 23 family inet address 10.90.23.1/30
set interfaces fe-0/0/7 unit 24 vlan-id 24
set interfaces fe-0/0/7 unit 24 family inet address 10.90.24.1/30
set interfaces fe-0/0/7 unit 113 vlan-id 113
set interfaces fe-0/0/7 unit 113 family inet address 10.90.113.1/30
set interfaces fe-0/0/7 unit 114 vlan-id 114
set interfaces fe-0/0/7 unit 114 family inet address 10.90.114.1/30

Assigning interfaces to virtual router instance AS1

set routing-instances AS1 interface fe-0/0/7.12
set routing-instances AS1 interface fe-0/0/7.13
set routing-instances AS1 interface lo0.1

Assigning interfaces to virtual router instance AS2

set routing-instances AS2 interface fe-0/0/6.12
set routing-instances AS2 interface fe-0/0/6.23
set routing-instances AS2 interface fe-0/0/7.24
set routing-instances AS2 interface lo0.2

Assigning interfaces to virtual router instance AS3

set routing-instances AS3 interface fe-0/0/6.13
set routing-instances AS3 interface fe-0/0/7.23
set routing-instances AS3 interface fe-0/0/7.113
set routing-instances AS3 interface lo0.3

Assigning interfaces to virtual router instance AS4

set routing-instances AS4 interface fe-0/0/6.24
set routing-instances AS4 interface fe-0/0/7.114
set routing-instances AS4 interface lo0.4

The final result of the configuration is shown in Figure 5.

For routing and inter-instance communication, your newly created virtual routers support a wide range of protocols, including:

• Static Routing
• OSPF (Open Shortest Path First)
• ISIS (Intermediate System to Intermediate System)
• BGP (Border Gateway Protocol)
• MPLS (Multi-Protocol Label Switching)
• With support for IPv4 and IPv6.

Figure 5. 5 Routers network topology.

The 15-Minute Network is Live!

Congratulations! You’ve successfully converted your single SRX 100 into a complex 5-node routing lab, all within the time it takes to brew a cup of tea. This setup proves that complex labs don’t require complex hardware. Use this topology to master OSPF, BGP, or MPLS.

Go build something great!

Back to Top

We recently explored the foundational elements of Cisco ACI, diving into how Tenants, VRFs, Bridge Domains, and EPGs serve as the logical building blocks for organizing your network. We saw how these components create a structured environment for applications, but that raises a critical question, how do these EPGs actually talk to each other securely?

How do you define precisely what traffic is allowed, and what is denied, between different application tiers or external networks? This is where Cisco ACI Contracts come in. They are the core enforcement mechanism for defining communication policies and building a zero-trust security model within your ACI fabric.

In this post, we’ll cover the different types of contracts and policy enforcement options:

1. Standard Contracts: Building the ACI Security Whitelist
2. Taboo Contracts: ACI’s Blacklist for Overriding Security Policies.
3. vzAny Contracts: The Shortcut to allow communication inside VRF.
4. Preferred Groups: When to Unenforce Policy in ACI.
5. Unenforced VRFs: The Full Open Communication inside VRF.

1. Standard Contracts: Building the ACI Security Whitelist.

1. Filters are the most granular rules, specifying permitted traffic based on network attributes such as EtherType, IP, ICMPv4 and v6 Source and Destination Port/Range.

2. Subjects are Logical containers for Filters that define the directionality of the traffic flow. It also defines the directionality of the communication using options like “Apply Both Directions” and “Reverse Filter Ports” these “Subject” options are responsible for automatically creating the necessary return path TCAM entry (ACL rule) for a connection.

3. Scopes Settings (Application Profile, VRF, Tenant, and Global) that establish the contract’s domain, controlling how widely it can be shared and reused across the ACI environment.
   • Application Profile scope - communication policies that are strictly internal to a single application, ensuring that the contract is not accidentally used by other applications, even in the same VRF.
   • VRF scope - The default and most common scope for defining communication policies between EPGs that share a Layer 3 forwarding domain. This scope is used for most intra-VRF contracts.
   • Tenant Scope - Primarily for Inter-VRF (route leaking) communication when both VRFs are in the same tenant. It allows EPGs in one VRF to consume a contract provided by an EPG in a different VRF, as long as they are in the same tenant.
   • Global scope - Inter-Tenant contracts, where the provider and consumer are in different tenants. For contracts defined in the Common Tenant that need to be universally available (e.g., contracts for shared services like DNS/NTP). or Inter-VRF communication where the VRFs are in different tenants.
4. Action specifies whether to permit, deny, or log the traffic (deny actions are supported in APIC Release 3.2(x) and later with EX/FX switches).

As many are still struggling with ACI contract direction (Consumer to Provider and Provider to Consumer), let me reveal one secret:

• Consumer is the SOURCE
• Provider is the DESTINATION

Apply your policies using this simple traffic flow rule, and you’ll get it right every time!

Below figures 1 through 3 illustrate the operational behavior of the “Apply Both Directions” and “Reverse Filter Ports” settings.

Figure 1. Applying contract between two EPGs.

The Consumer EPG can only initiate a new session with the Provider EPG on the specified port, which in this case is TCP 22 for SSH.
However, communication won’t be established because the contract is currently applied only in the Consumer-to-Provider direction, causing the necessary return traffic to be dropped. Therefore, we have two options:

• either create a second contract in the Provider-to-Consumer direction or
• use the “Apply Both Directions” option with “Reverse Filter Ports.”

Since contracts utilize hardware resources on the leaf switches, I recommend using “Apply Both Directions” with “Reverse Filter Ports.”

Figure 2. Feature “Apply Both Direction” Enabled.

With “Apply Both Directions” (Option ON):

• The Consumer EPG can initiate a connection to the Provider EPG.
• The Provider EPG can also initiate a new, separate connection back to the Consumer EPG on the same port(s).

Based on Figure 2. SRV1 and SRV2 can open SSH session to DB1 and DB2 and vice versa DB1 and DB2 can open SSH session to SRV1 and SRV2 but communication won’t be established.

Figure 3. Representing establishment of SSH session from Jumphost (SRV1 and SRV2) to DB servers.

Note:
Contracts can be applied only on unicast traffic. Broadcast, Unknown unicast and Multicast Protocols (BUM) traffic and below listed protocols are implicitly permitted.

• ARP reply (unicast)
• DHCP v4 (prot 0x11, sport 0x44, dport 0x43)
• DHCP v4 (prot 0x11, sport 0x43, dport 0x44)
• DHCP v6 (prot 0x11, sport 0x222, dport 0x223)
• OSPF (prot 0x59)
• EIGRP (prot 0x58)
• PIM (prot 0x67)
• IGMP (prot 0x2)
• ND-Sol ICMPv6 (prot 0x3a dport 0x0087)
• ND-Advt ICMPv6 (prot 0x3a dport 0x0088)

Moreover, in case of Multi-Site deployment, Cisco Multi-Site Orchestrator (MSO) creates one contract for each subject.

Back to Top

2. Taboo Contracts: ACI’s Blacklist for Overriding Security Policies.

Taboo contracts are specialized contracts used to explicitly deny specific traffic for a single EPG. Unlike standard contracts, they are:

• Unidirectional - Always provided by the EPG, with no consumer relationship.
• Higher Priority - Evaluated before standard contracts in the hardware’s Policy CAM.
• Limited Scope - Applied to a single EPG to block specific traffic, such as insecure protocols.
• Taboo contracts are not supported with Endpoint Security Groups (ESGs).
• They cannot be applied to vzAny.
• Cisco best practices generally advise against its regular use in production environments and suggest to use standard contracts with deny action if it is needed.

Example Use Case:

1. You want to block HTTP (port 80) traffic to Web_EPG while allowing other traffic via standard contracts.
2. Create a taboo contract Deny_HTTP_Taboo with a filter for TCP port 80 and apply it as provided by Web_EPG.

Figure 4. A Taboo-Contract blocking access to HTTP (insecure TCP port 80).

Back to Top

3. vzAny Contracts: The Shortcut to allow communication inside VRF and more.

vzAny is a powerful construct that simplifies the application of security policies across all Endpoint Groups (EPGs) and Endpoint Security Groups (ESGs) within a Virtual Routing and Forwarding (VRF) instance.

Key characteristics of vzAny:

1. VRF-Specific - vzAny is scoped to a single VRF and cannot directly apply to EPGs/ESGs in other VRFs.
2. Contract Simplification - Instead of applying a contract to each EPG/ESG individually, vzAny applies it to all EPGs/ESGs in the VRF simultaneously. Remember vzAny representing all EPGs inside VRF.
3. Provider or Consumer Role - vzAny can act as a contract provider (offering services) or consumer (accessing services), depending on the configuration.
4. Standard Contracts Only - vzAny supports standard contracts but not taboo contracts, which are limited to individual EPGs.

Figure 5. Representation of applying the contract per EPG.

Figure 6. Representing of applying the contract to vzAny.

Under the “Application Profile” and “Topology” tabs, you can view the graphical representation of the relationship between EPGs and Contracts. Below Figure 6 illustrates how DNS and NTP services are shared for all EPGs within the VRF using vzAny.

Note the checkbox on the right, “Show VRF in EPG,” which helps us identify which VRF a particular EPG belongs to.

Figure 7. Contract relationship: vzAny consumes services provided by DNS and NTP EPGs.

Please Note: External EPGs that are associated with L3Outs and are part of a VRF are also included in the vzAny logical group.

Limitations of vzAny:
1. VRF Scope - vzAny is limited to a single VRF. It cannot directly apply contracts to EPGs/ESGs in other VRFs without additional configuration (e.g., contract export or shared services).
2. No Taboo Contracts - vzAny does not support taboo contracts, which are used to deny specific traffic for individual EPGs.
3. Over-Permissiveness Risk - Applying contracts to vzAny affects all EPGs/ESGs in the VRF, which may lead to unintended traffic flows if not carefully planned.
4. Not for Granular Control - For specific EPG-to-EPG policies, individual contracts are preferred over vzAny.

Back to Top

4. Preferred Groups: When to Unenforce Policy in ACI.

A Preferred Group is a VRF-level configuration that designates a subset of EPGs or ESGs within a VRF as members of a group that can communicate with each other without contracts. When an EPG or ESG is added to the Preferred Group, it can send and receive traffic to/from other Preferred Group members within the same VRF, bypassing the need for standard contracts, provided the VRF is in enforced mode. I would like to mentioned that only one group can exist per VRF.

Limitations of Preferred Groups:

1. Preferred Groups are limited to a single VRF.
2. No Taboo Contract Support: Taboo contracts, which explicitly deny traffic for individual EPGs, cannot be overridden by Preferred Groups. If a taboo contract is applied to an EPG, it takes precedence.
3. All-or-Nothing Communication, preferred Group members can communicate freely with each other for all traffic types.
4. Not for External Connectivity: Preferred Groups do not apply to L3Out external EPGs for external connectivity; contracts are still required.
5. Policy CAM Usage: Preferred Group rules consume Policy CAM on leaf switches, which can impact scalability in large deployments.
6. If the VRF is unenforced, contracts are bypassed, and the Preferred Group has no effect (all EPGs/ESGs can communicate freely).

Back to Top

5. Unenforced VRFs: The Full Open Communication inside VRF.

An unenforced VRF is a VRF configuration in Cisco ACI where the policy enforcement is disabled.

In this mode:

1. All EPGs and ESGs within the VRF can communicate with each other without requiring contracts.
2. The unenforced setting applies only to intra-VRF traffic; inter-VRF communication still requires contracts and route leaking.
3. Traffic flows freely between endpoints within the VRF.
4. Contracts, including standard and taboo contracts, are bypassed for intra-VRF communication.
5. Features like Preferred Groups and vzAny become irrelevant in an unenforced VRF, as all traffic is permitted.

Table 1. Representing comparison Preferred Groups vs. vzAny vs. Unenforced VRF.

Back to Top

In the world of modern data centers, Cisco Application Centric Infrastructure (ACI) stands out as a powerful solution for network automation, policy enforcement, and application agility.

This post will demystify the core building blocks of Cisco ACI, guiding you through the concepts of Tenants, VRFs (Virtual Routing and Forwarding), Bridge Domains, Application Profiles, and Endpoint Groups (EPGs). By the end of this explanation, you'll have a clear understanding of how these elements work together to form a robust and flexible network fabric, enabling you to design and implement ACI solutions with confidence. Let's dive in!

Figure 1. Workflow of Tenant to EPG deployment.

1. Understanding the Tenant: The Administrative Boundary.
2. VRF (Virtual Routing and Forwarding): The Layer 3 Isolation Domain.
3. Bridge Domain (BD): The Layer 2 Forwarding Domain.
3.1. Subnet Configuration within a Bridge Domain: Defining IP Address Space & Reachability.
4. Application Profile (AP): The Application Logical Container.
5. EPG (Endpoint Group): The Granular Policy Enforcement Point.
6. Endpoint Security Groups (ESGs): Decoupling Security from Forwarding

1. Understanding the Tenant: The Administrative Boundary.

In the ACI object model, the Tenant (fvTenant) is the root of a policy tree for a given set of configurations. All subsequent networking objects (VRF, BD, EPG, etc.) are children of a Tenant.

Default Tenants: When you initialize an ACI fabric, three default tenants are automatically created:

common: This tenant is for shared resources that can be accessed by all other tenants. Common uses include shared Layer 3 Out (L3Out) connections to external networks, common services like DNS or DHCP, and shared Layer 4-7 services (e.g., firewalls, load balancers). Objects in the common tenant can be consumed by other tenants via contracts.

infra (Infrastructure): This tenant is used for internal ACI fabric communications. It contains policies related to fabric discovery, image management, VXLAN overlay, and other underlying infrastructure functions. It’s generally not directly configured by users unless specifically instructed by Cisco for advanced scenarios like Multi-Pod or Multi-Site deployments.

mgmt (Management): This tenant is dedicated to managing the ACI fabric itself. It defines policies for in-band and out-of-band management access to the APIC controllers and fabric switches. No Direct Network Construct: A Tenant itself doesn’t map directly to a traditional network construct like a VLAN, VRF, or physical interface.

Globally Unique Names: Tenant names must be globally unique across the ACI fabric.

Figure 2. Tenant deployment workflow.

Back to Top

2. VRF (Virtual Routing and Forwarding): The Layer 3 Isolation Domain.

Layer 3 Isolation: The fundamental purpose of a VRF in ACI, just like in traditional networking, is to provide Layer 3 isolation. Each VRF has its own independent routing table, IP address space, and set of routing protocols. This means that traffic within one VRF cannot directly communicate with traffic in another VRF unless explicitly configured to do so (e.g., via a firewall or inter-VRF routing policies).

Routing Instance: A VRF is essentially a virtual routing instance. It holds the routing information (routes, ARP/ND entries) for the subnets and endpoints associated with it.

Tenant Scoped: A VRF is always contained within a Tenant. It’s a child object of a Tenant (fvCtx in the ACI object model). This means that a VRF belongs to a specific administrative domain and its policies are managed under that tenant.

Policy Control Enforcement Preference:

• “Enforced” (Default and Recommended): This means that all communication between EPGs within this VRF (or between an EPG in this VRF and an external network) requires a Contract.
• “Unenforced”: This setting implies that all EPGs within this VRF can communicate freely with each other by default, without the need for explicit Contracts. While it simplifies initial setup, it bypasses ACI’s micro-segmentation capabilities and is generally discouraged for production environments where granular security is a requirement.

Preferred Group: setting, enabled at the VRF level, allows all EPGs within that VRF to communicate with each other by default if they are part of the preferred group.

Figure 3. VRF deployment workflow

Back to Top

3. Bridge Domain (BD): The Layer 2 Forwarding Domain.

In Cisco ACI, a Bridge Domain (BD) is a critical logical construct that represents a Layer 2 forwarding domain. It’s the ACI equivalent of a traditional VLAN or broadcast domain, but with significantly enhanced capabilities and policy integration.

Figure 4. First step in Bridge Domain deployment.

Figure 5. Second step in Bridge Domain deployment.

Figure 6. Third step in Bridge Domain deployment.

Figure 7. Bridge Domain L2 features.

Advertise Host Routes: By advertising a precise /32 (or /128) route for each individual endpoint, external routers learn the exact leaf switch (and pod/site) where the endpoint resides. This directs traffic directly to the optimal entry point into the ACI fabric.

L2 Unknown Unicast Handling: This setting defines how the ACI fabric handles unknown unicast traffic within the Bridge Domain:

• Hardware Proxy (Default and Recommended): This is the most efficient and scalable method. When a leaf switch receives an unknown unicast frame, it doesn’t flood it everywhere. Instead, it sends the frame to a spine proxy. The spine acts as a central lookup point, and if it knows the destination MAC, it forwards the frame directly to the correct leaf. If the MAC is unknown to the spine, only then does the spine flood it to relevant leaves. This significantly reduces flood domains.

• Flood: This behaves like traditional Ethernet. Unknown unicast frames are flooded to all leaf switches that have an endpoint associated with this Bridge Domain. This can be inefficient and generate unnecessary traffic, especially in large environments.

Optimize ARP Flooding: This option, when enabled, means that the ACI fabric will proxy ARP requests for known MAC addresses instead of flooding them. This reduces broadcast traffic.

Limit local IP learning to BD/EPG subnets: Restrict IP Learning: The ACI leaf switches connected to this Bridge Domain will only learn IP addresses for endpoints that fall within the explicitly defined IP subnets configured under that particular Bridge Domain, or the subnets defined within the EPGs that reside within that BD. It will ignore Out-of-Subnet IPs.

Figure 8. Bridge Domain L3 features.

IP Unicast Routing (Enabled/Disabled):

• If enabled, the ACI fabric will provide a distributed default gateway (Anycast Gateway) for any IP subnets defined within this Bridge Domain. This means the Layer 3 routing for endpoints in this BD is handled directly by the ACI leaf switches. This is the typical configuration for routed networks.
• If disabled, the ACI fabric will not provide a default gateway for the subnets in this BD. This is often used for: Networks where an external router or firewall provides the default gateway. In such cases, the ACI fabric simply acts as a transparent Layer 2 forwarding plane.

Back to Top

3.1 Subnet Configuration within a Bridge Domain: Defining IP Address Space & Reachability.

Each Bridge Domain can have one or more IP subnets associated with it. These subnets represent the IP address space for the endpoints connected to the Bridge Domain. Additionally, subnets can also be created directly under an EPG, but their purpose in that context is different: it facilitates Route Leaking and Shared Services between VRFs/Tenants with policy granularity.

Table 1. Comparing Subnets: Bridge Domain vs. EPG.

Figure 9. Subnet deployment within specific Bridge Domain.

Treat as Virtual IP address: Special handling for HA (Active/Standby) external device VIPs, ensuring proper learning and failover during HA events.

Make this IP address primary Designates a preferred source IP for the BD’s own control plane traffic when multiple subnets exist within this BD.

Advertised Externally: Advertises individual /32 (or /128) host routes for learned endpoints to external networks via L3Outs, optimizing traffic paths and VM mobility.

Shared Between VRFs: This option allows the subnet to be “leaked” and reachable by other VRFs within the same or different Tenants, usually in conjunction with Contracts for inter-VRF communication.

No Default SVI Gateway: Disables ACI’s pervasive gateway for this specific subnet (useful for secondary IPs or external gateways).

IP Data Plane Learning Enables dynamic learning of endpoint IP addresses by observing network traffic (typically enabled for general endpoints).

Back to Top

4. Application Profile (AP): The Application Logical Container.

In Cisco ACI, an Application Profile (AP) is a logical container that groups together all the Endpoint Groups (EPGs) and Contracts needed to deploy a specific application or service. It provides a higher-level abstraction for managing application-centric policies within the ACI fabric. The primary purpose of an Application Profile is to shift the focus from network-centric configuration to application-centric configuration. Instead of configuring network policies based on VLANs or subnets, you define policies based on the application components and their communication requirements.

Figure 10. Deployment of Application profile.

Back to Top

5. EPG (Endpoint Group): The Granular Policy Enforcement Point.

An EPG is the smallest and most critical unit to which policy (Contracts) is applied. If two endpoints are in the same EPG, they can communicate by default (unless otherwise restricted by other policies like Preferred Group or micro-segmentation). If they are in different EPGs, communication between them is blocked by default, and a Contract is required to permit specific traffic flows. This is ACI’s core “whitelist” security model. EPGs support micro-segmentation through features like intra-EPG isolation, which prevents communication between endpoints within the same EPG if required.

Type Of EPGs:

• “Classic” EPG which is representing endpoints, they can be physical servers, virtual machines, network-attached storage devices, or even clients on the internet.
• Layer 2 external outside network instance endpoint groups (l2extInstP).
• Layer 3 external outside network instance endpoint groups (l3extInstP).
• Management Endpoints groups (mgmtOoB, mgmtInB) are used for securing access to the ACI fabric’s management plane, either through a dedicated out-of-band management network or via in-band access within the fabric itself.

Figure 11. Deployment of EPG under application profile.

To provision EPG on the port/switch we have to methods:

• Static Ports: Explicitly links an EPG to a specific physical interface.
• Static Leafs (via AAEP): Links an EPG to a logical group of interfaces defined by an AAEP, effectively pushing the EPG’s VLAN to all ports associated with that AAEP.

Figure 12. EPG to Leaf Port Static Assignment.

Back to Top

6. Endpoint Security Groups (ESGs): Decoupling Security from Forwarding

Decoupling Security from Forwarding: This is the core architectural shift and primary benefit of ESGs compared to EPGs. EPGs combine both forwarding (VLAN/VXLAN binding, BD association) and security, whereas ESGs aim to provide security policy based on attributes, independent of the underlying network forwarding.

Figure 13. Defining Endpoint Security Groups (ESGs) for Policy Decoupling.

After defining your ESGs, you will create Contracts as usual. Associate the ESG as a Provider or Consumer of the contract, just like you would with EPGs.

Remember: Contracts typically apply ESG-to-ESG, ESG-to-L3Out External EPG, ESG-to-L2Out External EPG or ESG-to-vzAny.

By using ESGs, ACI provides an additional layer of flexibility and powerful tools for defining and enforcing security policies, especially beneficial in complex and evolving data center environments.

Back to Top

Understanding Cisco ACI Access Policies doesn’t have to be complicated. This practical guide simplifies the core concepts and clearly shows you what you need to know.

Secure and controlled access is fundamental to any network. This guide walks you through both the essentials and more advanced topics of Cisco ACI Access Policies, demonstrating how to apply them effectively in your environment and designs.

1. The End-to-End Workflow for Cisco ACI Access Policy Deployment.
1.1. Interface Policies Configuration Workflow.
1.2. Leaf Switches Policies Configuration Workflow.
2. Leaf Access Port configuration.
3. Leaf Port Channel (PC) configuration.
4. Leaf Virtual Port Channel (vPC) configuration.

1. The End-to-End Workflow for Cisco ACI Access Policy Deployment.

Cisco Application Centric Infrastructure (ACI) redefines network management with its policy-driven automation, moving away from traditional device-by-device configurations. At the heart of this transformative approach lies Access Policies, which dictate how physical network devices connect to the ACI fabric and how virtual and physical workloads access network services.

1.1. Interface Policies Configuration Workflow.

• VLAN Pools A VLAN Pool defines a range (or multiple ranges) of VLAN IDs that are permitted for use on leaf switch ports for EPG encapsulation. It doesn’t provision the VLANs themselves but rather reserves them for use.

• Domains and Domain Types Purpose: Domains act as a bridge, linking VLAN Pools with policies and the infrastructure. They define the scope where a particular VLAN pool can be utilized and by what type of entity.

  1. Physical Domains integrate bare-metal servers, physical appliances (e.g., firewalls, load balancers, non-virtualized servers), or other physical devices directly connected to ACI leaf ports.

  2. Fibre Channel Domains are primarily used for integrating Fibre Channel over Ethernet (FCoE) storage networks with the ACI fabric. This allows the ACI fabric to act as a converged network that carries both IP and FCoE traffic.

  3. External L3 Domains connect the ACI fabric to external Layer 3 (routed) networks or traditional routers outside the ACI fabric (Used by L3OUT).

  4. External Bridge Domains connect the ACI fabric to external Layer 2 (bridged) networks or switches outside the ACI fabric. This is used when you need to extend Layer 2 segments from the ACI fabric to external devices without routing (Used by L2OUT).

• Attachable Access Entity Profile (AAEP) The AAEP effectively defines which VLANs are allowed on an interface or group of interfaces. It does this by referencing Domains, which in turn reference VLAN Pools. If we compare with old fashion switch configuration it should be

   switchport trunk allowed vlan
  

  Common Scenarios for AAEP Scope/Design:

  1. One-to-one with Domains: Often, you’ll create an AAEP for each type of domain (Physical AAEP, 1.VMM AAEP, L3Out AAEP, L2Out AAEP, FCoE AAEP) to clearly separate their respective VLAN pools and interface associations.

  2. Functional Grouping: You might create AAEPs based on functional groups of ports, e.g., “ServerFarm_AAEP” that encompasses all necessary domains (physical and VMM) and VLANs for server connectivity.

• Interface Policy Group (IPG) acts as a collection point for individual “interface policies.” These individual policies define specific Layer 1 or Layer 2 parameters for a port.

Key Interface Policy Group Policies are:

• Link Level Policy (speed, duplex, auto-negotiation).
• CDP (Cisco Discovery Protocol) Policy (enable/disable).
• LLDP (Link Layer Discovery Protocol) Policy (enable/disable).
• LACP Policy (for Port Channels: active, passive, on, port priority).
• Storm Control Policy control Broadcast, Unknown Unicast, and Multicast (BUM).
• MCP (Mis-Cabling Protocol) Policy.
• Spanning Tree related policies (BPDU Guard/Filter).

• L2 Interface Polices (VLAN scope Global or Port Local).

• Interface Profile

Figure 1. Representing Interface Policies Configuration Flow.

Figure 2. Interface Policy Configuration within the Cisco APIC UI.

Figure 2.1. Interface Policy Configuration within the Cisco APIC UI.

Back to Top

1.2. Leaf Switches Policies Configuration Workflow.

• Leaf Switches Policy Groups they control the behavior of the entire leaf switch. This allows for consistent application of switch-level configurations to multiple leaf nodes selected by a Leaf Switch Profile.

Key Leaf Switches Policy Groups are:

• Spanning Tree
• Monitoring polices
• CoPP Leaf Policy
• NetFlow Node Policy
• BFD IPv4 and IPv6 Policy

Figure 3. Representing Leaf Switch Policies Configuration Flow.

Figure 4. Leaf Switch Policies Configuration within the Cisco APIC UI.

Figure 5. Full configuration flow for interface and leaf switch polices.

Back to Top

2. Leaf Access Port configuration.

Figure 6. Implementation of an Interface Policy Group within the Cisco APIC user interface (UI).

After creating the Leaf Interface profile and attaching it to the Leaf Switch profile, we add specific ports under the Leaf Interface profile and select the appropriate Interface Policy Group. This approach provides reusability and simplifies automation. For Particular case we created Leaf Access Port Policy Group like below and attached to port eth1/1 inside specific Leaf Interface Profile.

Figure 7. Representing creation of Access port policy group and attaching to specific Leaf Interface profile.

Figure 8. End to end workflow to setup access port up and running on leaf switch.

Back to Top

3. Leaf Port Channel (PC) configuration.

The foundational workflow for configuring a Port Channel in Cisco ACI closely mirrors that of a standard Access Port; however, a key distinction lies in the explicit inclusion and application of an LACP Policy within the associated Interface Policy Group, which is then specifically leveraged for the two designated physical ports on a single leaf switch to form the logical Port Channel.

Figure 9. Representing creation of Port Channel policy group.

Figure 10. End to end workflow to setup Port Channel (PC) up and running on leaf switch.

Back to Top

4. Leaf Virtual Port Channel (vPC) configuration.

Virtual Port Channel (vPC) significantly enhances network resilience and operational uptime by logically combining two physical ports, each residing on a separate Cisco Nexus leaf switch, into a single, aggregated link. This innovative design provides superior high availability, as it creates active-active forwarding paths, ensuring that traffic can seamlessly continue to flow even if one of the leaf switches or an individual link experiences a failure.

Figure 11. Representing creation of Virtual Port Channel (vPC) policy group.

Figure 12.End to end workflow to setup Virtual Port Channel (vPC) up and running on leaf switches.

Back to Top

Welcome to “Designing with Cisco ACI,” where we will dive into the Cisco ACI principals of creating robust, scalable, and efficient network architectures using Cisco’s Application Centric Infrastructure (ACI).

In today’s rapidly evolving IT landscape, the ability to design flexible and resilient networks is more critical than ever. This blog will provide you with ideas on what you can do with Cisco ACI fabric, covering various use cases and key design points. It will be especially valuable for those preparing to deploy Cisco ACI and who are still unsure which design to choose. The following Cisco ACI design types will be the topics of discussion:

1. Cisco ACI Traditional Single Pod.
2. Cisco ACI Remote leaf.
3. Cisco ACI Mini.
4. Cisco ACI Multi-Pod Interconnect via IPN.
5. Cisco ACI Multi-Pod Interconnect Spine back to back.
6. Cisco ACI Multi-Site.
7. Multi-Pod - Multi-Site hybrid deployment.

So let’s start!

1. Cisco ACI Traditional Single Pod.

Figure 1. Cisco ACI Single Pod Fabric.

Main Characteristics:

• Architecture: Consists of a single APIC (Application Policy Infrastructure Controller) cluster managing one ACI fabric.
• Management: Centralized management with a single point of control.
• Scalability: Limited to the capacity of one fabric.
• Latency: Lower latency within the single fabric.
• Failure Domain: Single failure domain, meaning any issue can affect the entire fabric.

Key Differences between Cisco ACI Single and Multi Pod Fabric:

• Scalability: Multi-Pod offers greater scalability by allowing multiple Pods to be managed under one APIC cluster.
• Resiliency: Multi-Pod improves resiliency by isolating failure domains.
• Latency: Single Pod has lower latency within the fabric, while Multi-Pod may have slightly higher latency due to inter-Pod communication

Back to Top

2. Cisco ACI Remote leaf.

Figure 2. Cisco ACI Extending fabric with remote leaf to remote location.

Main Characteristics:

• VxLAN Tunnels: Unicast traffic is forwarded through VXLAN tunnels over Layer 3. This ensures that traffic between the remote leaf and the main fabric is encapsulated and securely transmitted.
• Head End Replication (HER): For Layer 2 broadcast, unknown unicast, and multicast (BUM) traffic, Head End Replication (HER) tunnels are used. This method avoids the need for multicast in the WAN.
• Policy Consistency: All policies defined in the main data center are consistently applied to the remote leaf switches, ensuring uniform policy enforcement across the entire fabric.
• Local Switching: Traffic between endpoints connected to the remote leaf is switched locally, while traffic requiring spine proxy services is forwarded to the main fabric.

Inter-Pod Network Requirements for Remote Leaf deployment:

• L3 network should support 50 Bytes overhead for VxLAN if we have default MTU 1500 Bytes.
• The latency between the ACI main DC and the remote Leaf should be less than 300 msec Round Trip Time (RTT)
• A minimum of 100 Mbps bandwidth in the IP Network. Starting from Cisco ACI Release 4.2.4, the requirement has been lowered to 10 Mbps of bandwidth.
• The remote leaf solution requires the /32 Tunnel End Point (TEP) IP addresses of the remote leaf switches and main data center leaf/spine switches to be advertised across the main data center and remote leaf switches without summarization.

Use Cases:
Satellite/Co-Location Data Centers:
Enterprises or service providers often have a main data center and several smaller satellite data centers distributed across multiple locations.
Disaster Recovery Sites:
Organizations need to maintain disaster recovery sites to ensure business continuity in case of a failure at the primary site.
Branch Offices:
Companies with multiple branch offices require consistent network policies and centralized management.
Edge Computing:
With the rise of edge computing, data processing is moved closer to the source of data generation example telco 5G.

Back to Top

3. Cisco ACI Mini.

Figure 3. Cisco ACI mini - connecting virtual APICs to fabric.

Main Characteristics

• APIC Cluster: Consists of one physical APIC and two virtual APICs (vAPICs) running in virtual machines. Reduces the physical footprint and cost of the APIC cluster, making it suitable for environments with limited rack space or budget.
• Deployment Scenarios: Ideal for colocation facilities, single-room data centers, or any scenario where full-scale ACI installations are impractical.
• Management and Policy: Provides centralized management and policy enforcement similar to full-scale ACI deployments. Does not support features like Multi-Pod, Virtual Pod, or Remote Leaf switches.
• Integration: Multi-Site Support: Can be integrated with Cisco ACI Multi-Site for broader deployment scenarios.
• Time Synchronization: Requires the physical APIC and the ESXi server hosting the virtual APICs to be time-synced with the same NTP server to avoid synchronization issues.

Use Cases:

• Colocation Facilities: Businesses that use colocation facilities to host their IT infrastructure.
• Single-Room Data Centers: Small businesses or branch offices with a single-room data center.
• Development and Test Environments: Organizations that need a dedicated environment for development and testing.
• Edge Computing: Deployments at the edge of the network, closer to data sources.

Back to Top

4. Cisco ACI Multi-Pod Interconnect via IPN.

Figure 4. Cisco ACI Multi-Pod Interconnected via Inter-Pod Network.

Main Characteristics

• Centralized Management: Managed by a single APIC cluster, providing a unified management and policy domain across all Pods1.
• Fault Isolation: Control plane protocols (e.g., IS-IS, COOP, MP-BGP) operate independently within each Pod, enhancing fault isolation.
• Scalability: Supports the interconnection of multiple Pods, allowing for greater scalability and flexibility in data center design.
• Inter-Pod Network (IPN): Pods are connected via a Layer 3 Inter-Pod Network (IPN), which uses VXLAN encapsulation for data plane traffic and MP-BGP EVPN for control plane communication.
• End-to-End Policy Enforcement: Ensures consistent policy enforcement across all Pods, maintaining uniform security and network policies.
• Resiliency: Isolated failure domains within each Pod reduce the impact of failures, enhancing overall resiliency.
• Geographical Flexibility: Pods can be located in different physical data centers, including geographically dispersed locations, with up to 50 ms RTT latency.

Inter-Pod Network (IPN) Requirements:

• OSPF: Required for routing between Pods.
• PIM Bi-Dir: Used for handling unknown unicast, multicast, and broadcast traffic.
• MTU Size: The IPN must support jumbo frames with an MTU of 9150 to accommodate VXLAN encapsulated traffic. In newer APIC version there is possibility to perform tunning of the MTU size of all the control plane packets generated by ACI nodes (leaf and spines), including inter-Pod MP-BGP control plane traffic.
• DHCP Relay Functionality: Needed to allow Spine and Leaf nodes to be discovered across the IPN.
• Prioritization: Ensures that APIC-to-APIC communication is prioritized over the IPN.
• Interface Support: 10/40/100G
• VLAN 4: Used for 802.1Q encapsulation between the spine and IPN devices.
• VRF Context: Recommended to isolate IPN traffic from other services, ensuring stable IPN connectivity.

Use Cases:

• Disaster Recovery and Business Continuity: Organizations need to ensure business continuity and disaster recovery across multiple data centers.
• Geographically Dispersed Data Centers: Enterprises with data centers in different geographical locations require a unified network fabric.
• Active/Active Data Centers: Businesses need to deploy applications across multiple active data centers for load balancing and high availability.
• Scalability and Flexibility: Large enterprises or service providers need to scale their data center infrastructure.

Back to Top

5. Cisco ACI Multi-Pod Interconnect Spine back to back.

Figure 5. Cisco ACI Multi-Pod, pods connected directly via Spines.

The characteristics and use cases for Cisco ACI Multi-Pod with spine back-to-back interconnect are essentially the same as those for Cisco ACI Multi-Pod interconnected via an Inter-Pod Network (IPN). The primary distinction is the absence of an IPN. However, it is crucial to ensure direct connectivity between remote spines.

Advantages:

• Simplified Network Design: Eliminates the need for an Inter-Pod Network (IPN), simplifying the overall network design and reducing complexity.
• Reduced Latency:
  Provides direct connectivity between remote spines, which can reduce latency compared to traversing an IPN.
• Cost Efficiency:
  Reduces the need for additional IPN devices and infrastructure, leading to cost savings.
• Improved Performance:
  Direct spine-to-spine connections can optimize traffic flow and improve overall network performance.
• Enhanced Resiliency:
  Offers redundant paths between Pods, enhancing network resiliency and fault tolerance.

Back to Top

6. Cisco ACI Multi-Site.

Figure 6. Cisco ACI Multi-Site managed by Nexus Dashboard Orchestrator.

Key Differences between Multi Site and Multi Pod deployment:

• Management: Multi-Pod uses a single APIC cluster, while Multi-Site uses multiple APIC clusters managed by NDO.
• Latency: Multi-Pod typically has lower latency between pods, whereas Multi-Site can handle higher latency across sites.
• Scalability: Multi-Site is better suited for large-scale, geographically dispersed deployments.

Main Characteristics:

• Geographical Flexibility: Supports the interconnection of geographically dispersed data centers, extending Layer 2 and Layer 3 connectivity between sites.
• Unified Policy Management: Utilizes the Nexus Dashboard Orchestrator (formerly Multi-Site Orchestrator) to manage and enforce policies across multiple sites.
• Fault Isolation: Each site operates with its own APIC cluster, ensuring fault isolation and independent control planes.
• Scalability: Allows for the integration of multiple ACI fabrics, providing scalability for large and distributed environments.
• Consistent Policy Enforcement: Ensures consistent policy enforcement across all interconnected sites, maintaining uniform security and network policies.
• Data Plane Communication: VXLAN and MP-BGP EVPN: Uses VXLAN for data-plane communication and MP-BGP EVPN for control-plane communication between sites.

Inter-Site Network (ISN) Requirements: ACI fabrics are connected via ISN routed network which will be used for VxLAN communications between endpoints which they belong to different fabrics. I would like to highlight that there is no latency limit between the different ACI fabrics part of the same Multi-Site domain.
The only latency considerations are:

• Up to 150 ms, RTT is the latency supported between the Nexus Dashboard cluster nodes where the Orchestrator service is enabled.
• Up to 500 msec RTT is the latency between each Nexus Dashboard Orchestrator node and the APIC controller nodes that are added to the Multi-Site domain. This means that the Multi-Site architecture has been designed from the ground up to be able to manage ACI fabrics that can be geographically dispersed around the world.
• No Need for Multicast.
• MTU between ISN routers and spines should be increases to minimum 1600 Bytes, in this case should be changed default MTU size 9000 Bytes in global configuration.
• IP connectivity can be achieved using OSPF, BGP or static routing.

Use Cases:
Centralized Data Center with Separate Availability Zones:
Allows for the creation of distinct availability zones within a centralized data center, enhancing fault isolation and resource management.
Disaster Recovery:
Facilitates disaster recovery by enabling data replication and failover between sites, ensuring business continuity in case of site failures.
Geographically Distributed Data Centers:
Supports the management of data centers spread across different cities, countries, or continents from a single pane of glass, simplifying provisioning, monitoring, and management.
Stretched Bridge Domain with Layer 2 Broadcast Extension:
Extends Layer 2 domains across sites, allowing for live VM migration and active/active high availability.
Stretched VRF with Inter-Site Contracts:
Enables consistent policy enforcement across sites by stretching VRFs and applying inter-site contracts.
Shared Services with Stretched Provider EPGs:
Allows shared services to be accessed across multiple sites using stretched Endpoint Groups (EPGs).

Back to Top

7. Multi-Pod - Multi-Site hybrid deployment.

Figure 7. Extending Cisco ACI Multi-Pod to Multi-Site.

Using Cisco ACI Multi-Site and Multi-Pod as a single fabric offers several benefits, enhancing the overall network architecture. Here are some key advantages:

• Centralized Management:
  Both Multi-Site and Multi-Pod architectures allow for centralized management through a single APIC cluster, simplifying policy enforcement and configuration.
• Scalability:
  Multi-Pod enables the expansion of a single ACI fabric across multiple locations without the need for separate fabrics, making it easier to scale the network.
  Multi-Site allows for the interconnection of multiple ACI fabrics, providing scalability across geographically dispersed data centers.
• Resiliency and Fault Isolation:
  Multi-Pod architecture offers full resiliency at the network level across pods while maintaining a single fabric, ensuring minimal impact from failures.
  Multi-Site architecture isolates failure domains between sites, enhancing overall network resiliency.
• Disaster Recovery and Business Continuity:
  Both architectures support disaster recovery by enabling data replication and failover between sites or pods, ensuring business continuity in case of site failures.
• Consistent Policy Enforcement: Policies and configurations can be consistently applied across all sites and pods, ensuring uniform security and compliance.
• Operational Efficiency:
  Centralized management and automation reduce administrative overhead and operational complexity, making it easier to manage large-scale networks.

Back to Top

The Cisco ACI 6.0 release has been available for a couple of months now. Let’s check out what it brings to network engineers and IT professionals. Experience a new era of enhanced performance, security, and automation capabilities. In this blog, we’ll dive into the key enhancements of Cisco ACI 6.0 and explore how they can empower your network infrastructure.

Main three features and enhancements in Cisco ACI 6.0.2 and 6.0.3

1. Virtual APIC on VMware ESXi and Cloud
2. VMM Integration with Nutanix AHV
3. Nexus Dashboard - Connectivity analysis

1. Virtual APIC 6.0(2)F

• No physical APICs.
• No configuration limitations you can administer it like a regular APICs.
• Scale is pretty much same as for physical APIC.
• Limitations not able to run applications such as ELAM, policy explorer.
• APICs can be connected to leaf switches directly.
• Or they can be connected remotely via L3 network. During the initial setup, the wizard will ask you to choose how the APIC is connected to the network. Figure 1 below is an example.
• Virtual APIC Cluster Fully functional APIC that can manage up to 200 switches.
• 6.0(2)F Cloud hosted APIC Cluster can only be deployed within AWS Cloud.

Figure 1 ACI setup wizard

2. Integration with Nutanix AHV hypervisor 6.0(3)F

• APICs connect and authenticate to Prism Manager.
• Integration of Nutanix Acropolis Hypervisor (AHV) with Cisco Application Centric Infrastructure (ACI).
• This integration provides virtual and physical network automation and VM endpoints visibility in Cisco ACI.
• End to end network automation and end to end visibility.

3. Nexus Dashboard - Connectivity Analysis 6.0(2)F

• Troubleshooting between two End points, src and dst IP addresses or src and dst MAC addresses.
• Visibility of what happening between control and data plane.
• Verification of control and data plane.
• Traces all possible forwarding paths between source and destinations.
• Identifies if there are problems in the path and what is causing issues.
• Helps to highlight real root cause, perform forwarding checks.

4.Interface policy - simplify configuration

• This type of configuration is optional there is no need to change if you already master polices configuration. On that way you will keep everything under your control.
• This is more for new guys which is starting to use ACI Fabric.
• Guys with experience will not have so many benefits from this feature.
• It is compatible with previous manual configuration of polices.
• Not recommended for brown field deployments, I would like to recommend only for green field deployment.

5.Weight-Based Symmetric PBR

In 6.0 it was introduced Weight-based PBR it means that we can send more traffic to specific node. Use case:
• On specific node we have connected Load Balancer.
• On specific node we have more powerful firewall in the group.
• Introducing new firewall, we can redirect small amount of traffic to new firewall for test purposes.

6.Stretched L3out SVI across remote leaves.

• Supported for stretching a non-vPC L3Out SVI across remote leaf switches Use cases:
• Mobile packet core.
• Containerized workloads special at the Edge.
• Cloud native network functions CNF.

7. Multi-Site support for vzAny PBR and L3Out-to-L3Out PBR use cases. From ACI 6.0(4)F and NDO 4.2(3) following uses cases are supported:

• vzAny-to-vzAny (Intra VRF).
• vzAny-to-EPG (Intra VRF).
• vzAny-to-L3out (Intra VRF).
• L3out-to-L3out ((Intra VRF) and inter-VRF).
• Standard EPG-to-EPG subnet under consumer still is required. If you have plan to implement vzAny inside multi-site design, please check all details on link below: Cisco Nexus Dashboard Orchestrator Configuration Guide for ACI Fabrics.

8. Main Routing improvements Multi-Site support for vzAny PBR and L3Out-to-L3Out PBR use cases.

• BGP AS - Remove private AS from AS_PATH.
• Support for BFD on Secondary subnet IPv4/IPv6.
• BGP Additional Paths - Receive multiple paths for the same prefix without the new paths replacing any previous paths.

9. Operating and Administering improvements.

ACI ver.6.0(1)F

• Simplifying interface configuration.
• Secure Erase (RMA).

ACI ver.6.0(2)F

• Auto firmware update for Cisco APIC on discovery - APIC will be automatically upgrade on version same as fabric.
• Installing switch SMU without reload.
• Troubleshooting of QoS Polices.

ACI ver.6.0(3)F

• Reduce of time of APIC upgrade.
• Transaction-based Logging.
• Memory-based Automatic Switch Image Selection 32/64 bit.

ACI ver.6.0(4)F

• Streamlined navigation and editing of the stat collection threshold from a TCA fault.

10. Main Security improvements

• Cisco Nexus 9K switch secure erase.
• Support for a user group map rule for SAML and OAuth2.
• Support for TLS ver 1.3
• First hop security FHS support for VMM.
• TACACS external logging for switches.
• RSA Key 307 and 4096 support on X.509 certificates.

11. Integrations.

• Enhanced LLDP TLVs for Azure Stack HCI.
• Policy mode for Cisco ACI and VMware NSX-T integration.
• VMM Domain support for VMware vSphere 8.0.

12. Scale improvements.

References:
Cisco Application Policy Infrastructure Controller
Cisco ACI Releases Changes in Behavior

Let's dive into the world of Cisco ACI physical design. After all, we all know that without hardware, there is no cloud or networking.

Dive into the world of Clos Topology, a concept born from a 1952 paper by Charles Clos, a researcher at Bell Laboratories. His revolutionary ideas on multistage telephone switching systems have transcended time, now serving as the foundation for building highly scalable data centers with cost-effective switches. The Cisco ACI fabric is based on either a two-tier (spine and leaf switch) or three-tier (spine switch, tier-1 leaf switch, and tier-2 leaf switch) architecture. In a Cisco Application Centric Infrastructure (ACI), each of the leaf, Fabric Extenders and spine switches has different functions.

Leaf Switches

• Leaf switches are the ingress/egress points for traffic into and out of an ACI fabric.
• They connect directly to servers and other devices, facilitating rapid data transfer.
• Each leaf switch is linked to every spine switch, creating a mesh of high-bandwidth connections.
• This setup ensures that any data packet is only a couple of hops away from its destination, significantly reducing latency.

Fabric Extenders (FEX)

Fabric Extenders (FEX) in Cisco’s Application Centric Infrastructure (ACI) have several key functions:

• Port Extension - FEX extends the ports of a Nexus switch, acting as a remote line card. This allows for the use of more cost-effective devices with a reduced set of features that are connected to the Leaf Switches.
• Agile Connectivity - FEX offers agile connectivity for rack and blade servers and for converged fabric deployments. It delivers innovation to the data center, reduces total cost of ownership, and gains architectural flexibility.
• Configuration and Management - All management is done on the parent switch; there is no console/vty access on the FEX. This simplifies the management of the network infrastructure.
• Cisco ACI leaf switch models with -G or -Tin the product name, such as Cisco Nexus N9K-C9348GC-FXP, N9K-C93108TC-FX, N9K-C93108TC-FX-24, N9K-C93108TC-EX, N9K-C93108TC-EX-24, N9K-C93216TC-FX2, and N9K-93108TC-FX3P.

Discover the steps to configure Fabric Extenders (FEX) in Cisco ACI in Cisco comprehensive guide available here.

Spine Switches

Spine switches are Clos intermediary switches that have a number of key functions:

• They are responsible for high-capacity data transmission across the network.
• They ensure that data packets travel efficiently from one leaf switch to another.
• The spine layer is crucial for maintaining the performance of the network as it scales, providing a robust framework that can handle increasing data demands without bottlenecks.
• Spine switches interconnect all leaf switches in a full-mesh topology.

Together, they create an architecture that is highly standardized across deployments. This design allows for a highly scalable and efficient network infrastructure.

Two-Tier or Multi-Tier Design?

Below Figure 1 illustrates the necessary components and cabling for an two-tier ACI fabric. It’s important to note that no cables should be connected between ACI leaf switches, as this is characteristic of Clos network design. Similarly, cross-cabling ACI spines will lead to the disabling of the cross-connected ports. The topology depicts a full mesh of cabling between the spine and leaf layers. However, it’s important to note that while the Spine-Leaf architecture can be implemented without ACI, with the switches working in NX-OS mode. I would like to highlight that ASICs are different for Leaf and Spine switches, so a switch like the Nexus N9K-93180YC-EX cannot run as Spine switches.

Figure 1 Two-Tier Topology Spine-Leaf classic interconnect between leaf and spine switches

In summary, I would like to recommend to check switch compatibility on the links below:

• Cisco Nexus 9300 1/10GBaseT Switches
• Cisco Nexus 9300 1/10/25/40/50/100GE Fiber Switches
• Cisco Nexus 9300 40/100 GE Switches
• Cisco Nexus 9300 400 GE Switches
• Cisco Nexus 9300 ACI Fixed Spine Switches Data Sheet

If you plan to use Cisco Nexus modular switches for Cisco ACI fabric technical details you can find on below links:

• Cisco Nexus 9500 Series Switches Data Sheet
• Nexus 9500 Chassis Comparison
• Nexus 9800 Compare Models
• Nexus 9800 Line Cards comparison

Starting from Cisco ACI 4.1, the fabric also supports a multi-tier (three-tiers) topology. This includes two tiers of leaf switches, allowing for vertical expansion of the Cisco ACI fabric. This is particularly useful for migrating a traditional three-tier architecture (core-aggregation-access), a common design model for many enterprise networks. The ACI Multi-Tier is a beneficial solution for data centers aiming to reduce inter-row cabling and have low-bandwidth needs for top-of-rack switches. In this setup, Tier 1 leaf switches are placed at the end of each row, while Tier 2 leaf switches are positioned at the top of each rack. This strategy is particularly effective in managing cables and meeting bandwidth requirements.

Figure 2 Multi -Tier Topology Spine- Tier 1 Leaf and Tier 2 Leaf

Figure 3 Multi -Tier Topology Spine- Tier 1 Leaf and Tier 2 Leaf option between two data center buildings

The topology requirements may include various models of spine and leaf switches, please check below:

• Spine: EX/FX/C/GX/GX2 spines (For example, Cisco Nexus 9332C, 9316D-GX, 9348D-GX2A, 9408 and 9500 with EX/FX/GX linecards).

• Tier-1 Leaf: EX/FX/FX2/FX3/GX/GX2 except Cisco Nexus 93180LC-EX.

• Tier-2 All EX/FX/FX2/FX3/GX/GX2.

In the scenario where server access is at 1G and there is not need for high bandwidth, it’s a viable choice to use the Cisco Nexus 93180FX3 or Nexus 9364C-GX as the tier-1 leaf and the Cisco Nexus 9348FXP as the tier-2 leaf.

Before you start to build and design a multi-tier topology in a Cisco ACI environment, please take into account the following:

1.Fabric Port Configuration: All inter-switch connections must be configured as fabric ports. This includes the connections between Tier-2 leaf switch fabric ports and Tier-1 leaf switch fabric ports.

2.Tier-2 Leaf Switch Connectivity: A Tier-2 leaf switch has the capability to connect to more than two Tier-1 leaf switches. This is in contrast to a traditional double-sided vPC design, which only supports two upstream switches. The maximum number of ECMP links supported by a Tier-2 leaf switch to a Tier-1 leaf switch is 18.

3.Endpoint Group (EPG) and Layer 3 Outside (L3Out) Connections: EPGs, L3Outs, Cisco APICs, or Fabric Extenders (FEX) can be connected to either Tier-1 or Tier-2 leaf switches.

4.Tier-1 Leaf Switch Connections: Tier-1 leaf switches can have both hosts and Tier-2 leaf switches connected to them.

5.Switch Tier Transition: Transitioning from a Tier-1 to a Tier-2 leaf switch (and vice versa) requires decommissioning and recommissioning the switch.

6.Compatibility with Cisco ACI Multi-Pod and Multi-Site: Multi-tier architectures are compatible with both Cisco ACI Multi-Pod and Cisco ACI Multi-Site.

7.Tier-2 Leaf Switch Connectivity Limitations: Tier-2 leaf switches cannot be connected to remote leaf switches (Tier-1 leaf switches).

8.Scale Considerations: The combined number of Tier-1 and Tier-2 leaf switches must not exceed the maximum number of leaf switches validated for a given release (400 per pod; 500 per Cisco ACI Multi-Pod as of Cisco ACI release 6.0(1)).

Reference for Cisco ACI Multi-tier Architecture.

Fabric Port

In Cisco's Application Centric Infrastructure (ACI), a fabric port is a type of port that is used for uplink connectivity within the fabric. These ports are typically used to connect different switches within the ACI fabric. For example, in a multi-tier topology, all switch-to-switch links must be configured as fabric ports. This includes the connections between Tier-2 leaf switch fabric ports and Tier-1 leaf switch fabric ports.

Fiber optics modules

Don't forget that based on specific switch model and uplinks design you will need to choose proper optical modules. All details about compatibility matrix for Cisco Optics and devices you can find on below links:

• Cisco Optics-to-Device Compatibility Matrix
• Cisco Optics-to-Optics Interoperability Matrix

If you struggling or you have some doubts about compatibility between switches and optical modules SFP SFP+ QSFPs I would like to recommend:

Taxonomy for Cisco Nexus 9000 Series Part Numbers

For a comprehensive understanding and decoding of all Nexus Part Numbers names, kindly refer to Figure 4.

Figure 4 Cheet-sheet for decoding Cisco Nexus 9000 Series Part Numbers

This time I will go through configuration steps for provisioning switches on Cisco ACI Fabric, NTP, DNS and BGP Route Reflectors. Steps described in this post are applicable for Cisco ACI Fabric and Cisco ACI simulator.

0. In your favorite browser address bar add https://oob.ip.address. Use IP address what you set during APIC initial configuration.

Cisco ACI Fabric APIC ver 5.2 supported browsers:

• Chrome version 59 (at minimum)
• Firefox version 54 (at minimum)
• Internet Explorer version 11 (at minimum)
• Safari version 10 (at minimum)

1. After what you will login it will appear Cisco APIC welcome screen, looks nice. Just press Let’s go as this white button suggesting.

2. One more screen from APIC this one is important as it will provided you information about:

• New features for installed APIC version of APIC.
• Updated scalability numbers.
• Very good resources and knowledge base for ACI fabric and APIC clusters.
  After what you reviewed all interesting things pres Start ACI Fabric Setup button.

3. Below we can see the most important sections which should be configured before any service configuration on ACI.

I would like to suggest following order:

1. Add leaf and spine to Cisco ACI Fabric.
2. NTP server configuration.
3. DNS server configuration.
4. BGP RR configuration for Spine.

4. Add first leaf to Cisco ACI Fabric, for serial number we will use TEP-1-102, 101 and 103 as this is Cisco ACI simulator on real Cisco ACI Fabric you will have serial numbers in following format F1234A768. For node ID choose unique number, once when you tied node ID to serial number you will not be able to change on the fly. If you would like to change you will need to do decommissioning and commissioning with new node ID. Switch name you can change anytime if you will not like assigned during initial switch provisioning.

5. Here you should pay attention and check is it switch in Registering phase, if you notice some other messages please stop and perform tshoot.

6. Once when is done we should see that switch is successfully Register as we can see below for leaf with node ID 101.

7. Same you should repeat for all Spine switches too, below you can see Registering of spine with node ID 103.

8. And Spine with node ID 103 has been successfully Register.

9. Based on workflow from point #3 I will proceed with NTP configuration. You can choose time format from local or utc. Use plus sign for adding one or more NTP servers, one of them you can set like preferable.

10. In the lab for NTP server I use public Google NTP server just for test.

11. Here I will add Google DNS server just for test. Usually you will use internal DNS. Don’t forget to add DNS records for leaf, spines and APICs.

12. Here we can check status about what we done already. Below you can notice that still we have to configure BGP route reflectors. To configure BGP just press button Begin which will redirect you to configuration page.

13. Press plus sign and add minimu two spines per Cisco ACI fabric site.

14. From drop down choose node id of your spine. As I am using Cisco ACI Simulator I have only one spine with node ID 103. For test I am using default AS number.

15. Below you can see that spine with node ID 103 was successful registered like RR in Pod 1.

16. Short overview all looks good as Proxy won’t be used.

17. Summary page, last step for configuration setup.

18. I would like to recommend you to check Dashboard sections like below on print screen

• Switches should be Healthy.
• Controllers should be In service and FullyFit.

Note
This is only zero day configuration for Cisco APIC. For MultiPod, Multisite, 
Remote Leaf and other Cisco ACI Fabric designs should be configured other parameters
and network objects.

Bonus steps for those how like to have clean ACI fabric without lot of system notifications.
Next few steps will describe how we can disable messages from Cisco Smart Software Manage, this is mostly applicable for lab deployments and ACI Simulator as after 90 days should be reconfigured.

19. From APIC notification center please choose Cisco Smart Software Manager (CSSM).

20. Press on fault notification F3057.

21. As below on print screen click on Configure Network Settings.

22. From drop down menu choose Offline click on OK and you are ready.

I have walked through most important parameters for Cisco ACI Fabric. Now you can start to play with Cisco ACI Fabric network objects or you can proceed with fine tunning for the production environment. 😎

If you already read my last post How to run Cisco ACI Simulator on KVM host you know that now we need to do first step and go through Cisco APIC initial configuration setup. If you need help this post is right place for you. So let' go and see what we need to do.

Note:
Cisco ACI simulator doesn't support data plane and you won't be able to connect any  
other VM which can represent End Point. Simple you will not be able to issue ping  
command and send ICMP request and see any ICMP reply or use any other protocol for testing purpose.

1.VM is booting

2.If you have recommended HW resource use large topology otherwise your choice is “N”.

3.After few seconds it will appear this nice logo.

4.For ACISIM I would like to recommend to accept all default parameters. All parameters you can change but in that case there is no guaranteed that Cisco ACI simulator will work properly.

5.One of parameters what actually you should change is default OOB subnet and gateway. Choose IP address/Subnet mask from your OOB lab network and set in this step. Later you will use this IP address for access APIC UI via HTTPS.

6.Please set admin password and don’t forget, later you will need for login to APIC UI 😉

7.Here you have chance to change any configuration parameter what you set during initial configuration. If you are self confident and you put password on the sticky note on your desk you just press “ENTER”

8.In your favorite browser type https://oob.ip.address what you set during initial configuration. When APIC login page appear for User ID use admin and for password use note from your sticky note from your the desk.

Final Part 2 coming soon! Are you enjoyed reading? If yes please subscribe Here

As officially ACI Simulator is not supported on KVM host and I didn't have ESXi in my lab I have decided to do experiment. Moreover I have played with amount of hardware resource necessary for this experiment. If you would like to know result of this experiment, you can proceed and read rest of the post.

At the beginning lets say couple of words about Cisco ACI Simulator or ACISIM. So this is Cisco virtual machine, inside which we can find simulation of two leaf switches, one spine and one Cisco Application Policy Infrastructure Controller (APIC). Actually this is simulation of Data Center leaf spine topology with SDN controller.

Official Cisco ACI Simulator Virtual Machine HW requirements:

To make things more interesting I have reduced HW resources just from the beginning of experiment.😉

And I created setup as below:

So, are you ready to spin ACI Simulator with less than minimum HW resource for release 5.2(4d)? If yes, let's go and see what will happen in the end.😎

Step 1. Download ACI Simulator

Go to Cisco Download Page, please don’t forget that you should have valid Cisco account, otherwise no luck.

Download all eight pieces:

• acisim-5.2-4d_part1.ova
• acisim-5.2-4d_part2.ova
• acisim-5.2-4d_part3.ova
• acisim-5.2-4d_part4.ova
• acisim-5.2-4d_part5.ova
• acisim-5.2-4d_part6.ova
• acisim-5.2-4d_part7.ova
• acisim-5.2-4d_part8.ova

Step 2. Concatenate downloaded parts and create QCOW2 image from VMDK

Use Linux cat command to concatenate all 8 parts.

cat acisim-5.2-4d_part1.ova \ 
    acisim-5.2-4d_part2.ova \
    acisim-5.2-4d_part3.ova \
    acisim-5.2-4d_part4.ova \
    acisim-5.2-4d_part5.ova \
    acisim-5.2-4d_part6.ova \
    acisim-5.2-4d_part7.ova \
    acisim-5.2-4d_part8.ova > acisim.ova

Optional check MD5 of acisim.ova VM once when concatenations is done.

md5sum -c aci.md5

Now we have virtual machine with vmdk vHD but we need qocow2 image. So what’s next, we need to untar acisim.ova and we have to convert acisim-5.2-4d-disk1.vmdk to acisim-5.2-4d-disk1.qcow2.

tar -xvf aci.ova

voila finally we have acisim-5.2-4d-disk1.vmdk let’s run one more command and we will have acisim-5.2-4d-disk1.qcow2

qemu-img convert -O qcow2 acisim-5.2-4d-disk1.vmdk path/to/images_folder/acisim-5.2-4d-disk1.qcow2

Step 3. Run ACISIM VM on Ubuntu with KVM hypervisor

Run virt-manger application from desktop UI or If you are using headless server first ssh to server (don't forget to enable x11 forwarding).

• Once when you run virt-manager command it will appear creation VM setup dialog

• Please choose like below on print screens.

• Here you will need to point to folder where you saved acisim.qcow2 image.

• Let use only 8 vCPUs and 32G of RAM, this is far from recommended by Cisco.

• Mandatory, please select network adapter type e1000 otherwise later you will not be able to connect to your ACISIM via HTTPS or SSH.
• To start VM creation press "Begin installation".

• Between all VMs choose acisim and run.

• Press OPEN button to access acisim VM console.

• After 2-3 min (depend about power of your host) it will appear APIC initial screen.

• Below we can see that we are running Cisco ACI simulator on KVM host and successful start of initial configuration on Cisco APIC.

Short Outcome:

• If we have insufficient HW resource already in the lab we can reduce number of vCPUs and amount of memory for ACISIM VM.
• In this Blog post I have used 8x vCPS and 32 GB of RAM below this it will not work.
• We can use Cisco ACI simulator on KVM host, even if it is not officially supported by vendor.

Enjoy your ACI simulator and happy testing!!!