If you spend enough time designing data center networks, eventually you encounter the same question:

How do we scale Layer 2 without inheriting all the problems that Layer 2 brings with it?

That question is one of the primary reasons EVPN exists.

Many engineers first encounter EVPN through VXLAN deployments and mistakenly assume EVPN is simply a mechanism for transporting VLANs across a routed fabric. While technically true, that view dramatically understates what EVPN actually provides.

EVPN is fundamentally a control-plane solution to decades-old Ethernet scaling problems.

To understand EVPN properly, you must first understand the problems it was created to solve.

The Evolution of Data Center Networking

Traditional Ethernet Networks

Classic Ethernet networks were simple.

Switches learned MAC addresses by observing traffic.

When a frame arrived:

1. Source MAC was learned.
2. Destination MAC was looked up.
3. If unknown, traffic was flooded.

This model worked remarkably well for small environments.

The challenge appeared when networks became larger.

Suddenly we had:

• Thousands of endpoints
• Multiple switches
• Multiple VLANs
• Redundant paths

Ethernet had no native loop prevention.

A loop meant a broadcast storm.

The solution was:

Spanning Tree Protocol (STP)

STP prevented loops by intentionally blocking links.

This solved one problem while creating several new ones.

Engineers accepted:

• Idle links
• Slow convergence
• Suboptimal forwarding paths
• Large failure domains

For years this was considered normal.

The MLAG Era

As data centers grew, engineers became increasingly frustrated by blocked links.

Technologies emerged such as:

• MLAG (Multi-Chassis Link Aggregation)
• Cisco vPC
• Cisco VSS
• HPE IRF (Intelligent Resilient Framework)
• Juniper MC-LAG

The goal was straightforward:

Allow servers to use multiple active uplinks while avoiding STP limitations.

This worked well.

However, MLAG introduced new challenges:

• Vendor-specific implementations
• Peer-link dependencies
• Operational complexity
• Scalability constraints
• Failure corner cases

Most importantly:

MLAG solved local redundancy. It did not solve network-wide Layer 2 scalability.

VXLAN Changes the Game

VXLAN introduced a powerful idea:

Instead of extending Layer 2 across a switching topology, transport Ethernet frames across a routed IP network.

The underlay became Layer 3.

The overlay became Layer 2.

This was revolutionary.

Suddenly:

• ECMP became available everywhere.
• Spanning Tree disappeared from the fabric core.
• Scale increased dramatically.
• Link utilization improved.

VXLAN created a much better data plane.

However, it introduced a major problem.

VXLAN Flood-and-Learn

Early VXLAN deployments relied on flood-and-learn behavior.

This closely resembled traditional Ethernet.

Unknown destinations still required flooding.

ARP requests still required flooding.

MAC discovery still relied on data-plane learning.

The result was familiar:

• Excessive BUM traffic
• Unknown unicast flooding
• Large flood domains
• Limited operational visibility

VXLAN solved transport.

It did not solve learning.

That is where EVPN enters the picture.

Understanding the Problem Before Understanding EVPN

One of the biggest mistakes engineers make is trying to learn EVPN before understanding the problems it addresses.

EVPN is much easier to understand when viewed as a solution rather than a protocol.

Unknown Unicast Flooding

Consider a host attempting to communicate with another host.

The switch has not learned the destination MAC.

What happens?

The frame is flooded.

Traditional Ethernet relies on flooding to discover destinations.

This is acceptable in small environments.

It becomes increasingly problematic when:

• Thousands of endpoints exist
• Hundreds of racks exist
• Workloads move frequently

Every unknown destination generates unnecessary traffic.

At sufficient scale this becomes operationally expensive.

Broadcast Domain Growth

Broadcast traffic never scales well.

Examples include:

• ARP
• DHCP
• Neighbor discovery
• Service discovery protocols

As broadcast domains grow:

• CPU utilization increases
• Network overhead increases
• Convergence slows
• Troubleshooting becomes harder

Large Layer 2 domains eventually become operational liabilities.

MAC Table Scalability

Traditional switches learn MAC addresses dynamically.

Every endpoint consumes table resources.

Modern environments often contain:

• Virtual machines
• Containers
• Kubernetes workloads
• Bare-metal servers

MAC scale can grow rapidly.

The challenge becomes:

How do devices learn only what they need without relying on flooding?

Mobility Challenges

Modern workloads move.

Frequently.

Virtual machines migrate.

Containers restart.

Applications relocate.

Traditional Ethernet was never designed with rapid mobility as a primary requirement.

The network must quickly determine:

• Where the endpoint moved
• Which path is active
• Which location is stale

Without efficient signaling, convergence becomes slow.

Multi-Homing Challenges

Engineers want:

• Redundancy
• Load balancing
• Fast failover

Traditional Ethernet achieves this imperfectly.

MLAG improves matters but introduces operational dependencies.

The challenge becomes:

How do multiple switches present themselves as a single attachment point while maintaining independent control planes?

EVPN introduces standards-based multi-homing using Ethernet Segments (ESI), eliminating many of the operational dependencies associated with traditional MLAG designs.

Failure Convergence

Large-scale networks fail continuously.

Links fail.

Optics fail.

Servers fail.

Switches fail.

The objective is not preventing failures.

The objective is minimizing impact.

Traditional Ethernet often discovers failures reactively.

EVPN introduces proactive signaling mechanisms that dramatically improve convergence behavior.

Why These Problems Matter More Today

Twenty years ago:

• Physical servers dominated
• Workloads rarely moved
• East-west traffic was limited

Today:

• Virtualization is everywhere
• Kubernetes is everywhere
• Cloud operating models dominate
• East-west traffic exceeds north-south traffic

The network must adapt.

Flood-and-learn architectures become increasingly inefficient.

Control-plane learning becomes necessary.

What EVPN Actually Is

Now we can define EVPN correctly.

A surprisingly large percentage of engineers cannot.

Practical Definition

EVPN (Ethernet VPN) is:

A BGP-based control plane that distributes Layer 2 and Layer 3 reachability information across an overlay network.

Notice what is absent from that definition.

VXLAN.

Because EVPN is not VXLAN.

EVPN Is a BGP Address Family

EVPN operates as an extension of BGP.

Specifically, it uses the EVPN address family.

BGP becomes responsible for advertising:

• MAC addresses
• MAC/IP bindings
• Ethernet segment information
• IP prefixes
• Multi-homing information

Rather than discovering endpoints through flooding, devices exchange endpoint information through BGP.

This is one of the most important concepts in EVPN.

EVPN Is a Control Plane

EVPN’s primary job is:

Distributing endpoint information.

Think of EVPN as a database synchronization mechanism.

Each VTEP learns:

• Which endpoints exist
• Where they reside
• Which VNI they belong to
• Which next hop reaches them

Instead of learning through data traffic.

EVPN Is Not VXLAN

This misconception causes endless confusion.

EVPN ≠ VXLAN.

EVPN = control plane.

VXLAN = data plane.

The two are often deployed together.

But they are independent concepts.

VXLAN Is a Data Plane

VXLAN provides:

• Encapsulation
• Transport
• Segmentation

VXLAN does not provide:

• Endpoint discovery
• MAC advertisement
• Mobility handling
• Multi-homing intelligence

Those functions belong to EVPN.

EVPN Can Operate Over Multiple Encapsulations

Another misconception:

Many engineers assume EVPN requires VXLAN.

It does not.

EVPN originated in service provider networks as a modern replacement for VPLS. Service providers needed a scalable way to distribute MAC reachability information without relying on excessive flooding and data-plane learning.

The same control-plane model later proved extremely valuable in modern data center fabrics, where virtualization, workload mobility, and large-scale Layer 2 extension created similar challenges.

Today, EVPN is widely deployed across both service provider and data center environments.

EVPN supports:

• VXLAN
• MPLS
• SR-MPLS
• SRv6

The control plane remains EVPN.

The forwarding mechanism changes.

This separation is critical.

Understanding the Relationship

Think of the architecture like this:

Application
     |
Endpoint
     |
 EVPN Control Plane
     |
 VXLAN Overlay
     |
 IP Underlay
     |
 Physical Network

Each layer has a different responsibility.

The Underlay

The underlay is simply IP connectivity.

Its job is straightforward:

Deliver packets between VTEPs.

The underlay should not care about:

• VLANs
• MAC addresses
• Tenants
• Workloads

It only provides reachability.

The Overlay

The overlay creates logical network services.

Examples:

• Layer 2 extension
• Tenant isolation
• VRFs
• Distributed gateways

The overlay provides the logical Layer 2 and Layer 3 services. EVPN acts as the control plane that distributes reachability information for those services.

Mental Model Every Engineer Should Remember

If you remember only one thing from this section, remember this:

The underlay transports packets. The overlay transports services. EVPN tells everyone where everything lives.

That single sentence explains most EVPN architectures.

EVPN Building Blocks

Before discussing route types, engineers must understand several foundational concepts:

• Route Distinguishers (RDs)
• Route Targets (RTs)
• VNIs
• VRFs
• MAC advertisements

These form the language EVPN uses to describe network state.

Route Distinguishers (RD)

An RD exists for one reason:

Uniqueness.

Two tenants may use identical addressing.

Example:

Tenant A:

10.1.1.0/24

Tenant B:

10.1.1.0/24

Without differentiation, BGP cannot distinguish them.

The RD creates uniqueness.

Example:

65000:100
65000:200

Now identical prefixes become unique VPN routes.

Important Reality

An RD is not a policy tool.

Many engineers confuse RDs and RTs.

RD:

• Makes routes unique

RT:

• Controls route distribution

These are fundamentally different functions.

Route Targets (RT)

Route Targets determine:

Who imports what.

Think of RTs as membership tags.

Example:

Export RT: 65000:100
Import RT: 65000:100

Any VRF importing that RT receives those routes.

This enables:

• Tenant isolation
• Shared services
• Controlled route leaking

RTs are among the most important EVPN design decisions.

VNI Mapping

VXLAN identifies segments using VNIs.

Traditional VLAN:

VLAN 100

VXLAN equivalent:

VNI 10100

VNIs provide vastly larger scale.

Layer 2 VNI

Represents a bridged domain.

Example:

VLAN 100
VNI 10100

Hosts remain in the same subnet.

Layer 3 VNI

Represents a VRF.

Example:

Tenant-A VRF
L3 VNI 50000

Used for routing between subnets.

MAC Learning Evolution

Perhaps the single most important architectural shift in EVPN is how MAC addresses are learned.

Traditional Ethernet Learning

Traditional switches learn from traffic.

Receive Frame
     |
Learn Source MAC
     |
Populate Table

Advantages:

• Simple
• Automatic

Disadvantages:

• Flooding required
• Slow discovery
• Limited visibility

EVPN Learning

EVPN uses control-plane learning.

Endpoint Appears
     |
VTEP Learns MAC
     |
Advertises Route Type 2
     |
Remote VTEPs Learn Through BGP Advertisement

EVPN moves endpoint learning from a flood-and-learn model to the control plane, significantly reducing unnecessary flooding and improving network scalability.

The endpoint location is distributed proactively.

Why This Matters

Control-plane learning provides:

• Faster convergence
• Better visibility
• Reduced flooding
• Better scaling
• Improved troubleshooting

This is arguably the single greatest benefit EVPN provides.

Key Takeaways Before Moving to Route Types

At this stage, every engineer should understand:

1. EVPN exists to solve Ethernet control-plane scaling and operational challenges
2. EVPN is a BGP control plane.
3. VXLAN is a forwarding mechanism, not a control plane.
4. EVPN distributes endpoint information proactively.
5. RDs provide uniqueness.
6. RTs provide policy.
7. VNIs map services into the overlay.
8. Control-plane learning replaces flood-and-learn behavior.

In Part 2, we will dive deeply into the EVPN route types (Type 1 through Type 5), explain exactly what each route does, why it exists, how it is created, and how experienced engineers troubleshoot them in production EVPN fabrics.

Every meaningful operation inside an EVPN fabric is represented through a route advertisement.

When engineers first encounter EVPN route types, the reaction is usually:

“Why does EVPN need so many route types?”

The answer is simple.

Traditional Ethernet relied on flooding and local learning.

EVPN replaces those mechanisms with explicit signaling.

Different operational functions require different types of information.

Instead of blindly flooding traffic and hoping switches learn something useful, EVPN advertises exactly what needs to be known.

Once you understand the purpose of each route type, EVPN becomes dramatically easier to troubleshoot.

One of the best pieces of advice I can give engineers learning EVPN is:

When troubleshooting EVPN, think in route types rather than protocols.

Ask:

• Which route should exist?
• Why should it exist?
• Who should advertise it?
• Who should receive it?

Most EVPN problems become much easier to solve.

The Five Route Types Every Engineer Must Understand

For VXLAN EVPN deployments, the most important route types are:

You may encounter additional route types in RFC documentation, but these five form the foundation of nearly every EVPN deployment.

Route Type 1 — Ethernet Auto Discovery Route

Why It Exists

Route Type 1 exists primarily to support:

• Multi-homing
• Fast convergence
• Aliasing
• Backup path discovery

Think of Type 1 routes as:

“I am attached to this Ethernet Segment.”

Without Type 1 routes, EVPN multi-homing would not function.

The Problem Being Solved

Imagine a server connected to:

Server
  |
  +---- Leaf1
  |
  +---- Leaf2

Both leaf switches must advertise:

• The server attachment
• The Ethernet segment identity
• Redundancy information

Remote VTEPs need to know:

• Which VTEPs participate
• Which paths are available
• Which backup paths exist

Type 1 routes provide this information.

Trigger for Creation

Type 1 routes are typically generated when:

• An Ethernet Segment becomes active
• A multi-homed device connects
• A VTEP participates in an ESI

Type 1 routes are primarily relevant in EVPN multi-homing scenarios and are advertised per Ethernet Segment and/or per EVI, depending on implementation and use case.

Key Components

A Type 1 route typically includes:

• ESI
• RD
• Next Hop
• Ethernet Tag ID

Most importantly:

Ethernet Segment Identifier (ESI)

The ESI uniquely identifies a multi-homed segment.

Example:

ESI:
0000:0000:0000:0001

Every participating leaf advertises the same ESI.

Why This Matters

Remote VTEPs now understand:

ESI 1
 |
 +-- Leaf1
 |
 +-- Leaf2

This creates path awareness.

Without it, EVPN would view each attachment independently.

Troubleshooting Indicators

When Type 1 routes are missing:

Common symptoms:

• Multi-homing fails
• Traffic blackholes after failures
• Aliasing fails
• Backup paths disappear

Questions to ask:

Is the ESI configured?

Is the Ethernet Segment active?

Are Type 1 routes visible?

Route Type 2 — MAC/IP Advertisement Route

If there is one EVPN route type every engineer must master, it is Type 2.

Type 2 routes are the heart of EVPN.

Why Type 2 Exists

Traditional Ethernet learns MAC addresses through traffic.

EVPN replaces this with:

Control-plane MAC learning.

Instead of flooding:

Who owns MAC X?

The VTEP already knows.

Because another VTEP advertised it.

What Type 2 Carries

Type 2 routes advertise:

• MAC addresses
• An optional IP address, when the MAC/IP binding is known

Mobility sequencing information may be attached to a Type 2 route via the MAC Mobility extended community.

Example:

MAC:
aa:bb:cc:dd:ee:ff

IP:
10.10.10.5

VNI:
10100

VTEP:
10.0.0.11

This tells the fabric:

“If you need this endpoint, send traffic here.”

Trigger for Creation

Type 2 routes appear when:

• MAC learned (endpoint discovered on local port)
• IP learned via ARP/ND (optional, enables MAC/IP binding)
• Host becomes active (e.g. VM boot, port-up)

Example:

VM boots

↓

Leaf learns MAC

↓

Leaf learns IP

↓

Leaf advertises Type 2

Remote VTEPs that import the relevant Route Target receive the information.

Why MAC/IP Binding Matters

A major EVPN advantage is:

MAC and IP information travel together.

Traditional Ethernet learns:

MAC only

EVPN learns:

MAC + IP

This enables:

• ARP suppression
• Mobility tracking
• Better visibility

Host Mobility

Type 2 routes also handle mobility.

Example:

VM moves

Leaf1 → Leaf2

Leaf2 advertises a new Type 2 route with a higher MAC Mobility sequence number.

Remote VTEPs prefer Leaf2’s route. Leaf1 withdraws its Type 2 route once it detects the MAC has moved.

The fabric converges to the new location automatically, without requiring flooding or manual relearning.

Troubleshooting Indicators

When Type 2 routes are missing:

Symptoms:

• Host unreachable
• ARP failures
• Unknown unicast flooding
• Endpoint visibility issues

Verify:

Show EVPN Type 2 routes

Show MAC table

Show VNI endpoint database

Healthy state:

MAC visible

IP visible

Correct VTEP visible

Broken state:

Missing endpoint

Wrong VTEP

Mobility conflict

Route Type 3 — Inclusive Multicast Ethernet Tag Route

Type 3 routes confuse many engineers because they seem less visible during normal operations.

However they are critical.

Why Type 3 Exists

Even EVPN fabrics must transport:

• Broadcast traffic
• Unknown unicast traffic
• Multicast traffic

Collectively known as:

BUM Traffic

Broadcast
Unknown Unicast
Multicast

Type 3 routes tell VTEPs:

“Here are the remote VTEPs that participate in this VNI and should receive BUM traffic.”

The Flood List

Every VTEP builds a flood list.

Example:

Tenant A

Leaf1
Leaf2
Leaf3
Leaf4

Type 3 routes help construct:

Replication Targets

Without Type 3 routes:

BUM traffic has nowhere to go.

Trigger for Creation

Generated when:

• VNI becomes active
• VTEP joins segment
• Tenant becomes operational

Replication Models

Two primary models exist.

Head-End Replication

Most common.

Source VTEP replicates traffic.

Example:

Leaf1

Copies packet to:

Leaf2
Leaf3
Leaf4

Simple.

Popular.

Widely deployed.

Multicast Replication

Underlay multicast handles replication.

Advantages:

• Efficient bandwidth use

Disadvantages:

• Operational complexity

Most modern VXLAN EVPN fabrics use head-end replication.

Troubleshooting Indicators

Missing Type 3 routes often cause:

• ARP failures
• DHCP failures
• Broadcast failures

Questions:

Do flood lists exist?

Are Type 3 routes present?

Are all VTEPs participating?

Route Type 4 — Ethernet Segment Route

Type 4 routes are another multi-homing route.

Many engineers initially confuse Type 1 and Type 4.

They serve different purposes.

Why Type 4 Exists

Type 4 routes advertise:

“I participate in Ethernet Segment X.”

Their primary purpose is:

• ESI discovery
• DF election support
• Multi-homing awareness

Relationship to Type 1

Think of it this way:

Type 1:

Path information

Type 4:

Segment participation information

Both work together.

Trigger for Creation

Generated when:

• Ethernet Segment configured
• ESI active
• Multi-homing enabled

Designated Forwarder (DF) Election

One major Type 4 function:

DF election.

In EVPN multi-homing:

Leaf1
Leaf2

Both connect to the same segment.

Only one should forward certain traffic types.

Type 4 routes facilitate the election process.

Troubleshooting Indicators

Missing Type 4 routes often result in:

• DF election failures
• Duplicate forwarding
• Traffic loss after failover

Verify:

ESI visibility

DF state

Type 4 advertisements

Route Type 5 — IP Prefix Route

Type 5 routes advertise IP prefixes, extending EVPN beyond individual host reachability to scalable Layer 3 tenant routing, route summarization, and external network interconnection.

Why Type 5 Exists

Type 5 routes advertise:

IP prefixes.

Example:

10.100.10.0/24

10.200.20.0/24

172.16.100.0/24

This enables:

• Inter-subnet routing
• Inter-VRF prefix leaking (via Route Target import/export policy)
• External route advertisement
• Tenant routing

The Problem Type 5 Solves

Imagine:

Tenant A

Subnet A
10.1.1.0/24

Subnet B
10.1.2.0/24

Hosts communicate through distributed gateways.

When the fabric needs to advertise summarized, external, or inter-VRF prefixes rather than individual host endpoints, Type 5 routes provide the mechanism.

This allows EVPN to distribute network reachability information at scale.

Trigger for Creation

Common triggers:

• Connected subnet redistributed into BGP EVPN
• Static route redistributed into BGP EVPN
• External route redistributed from IGP or BGP into EVPN

Typical Use Cases

Inter-Subnet Routing

Example:

10.1.1.0/24

↓

10.1.2.0/24

Traffic routed through distributed gateways.

Type 5 distributes reachability.

Tenant Route Distribution

Example:

Tenant A

10.1.0.0/16

Advertised throughout fabric.

External Connectivity

Example:

Firewall

Internet

WAN

MPLS

An external router or firewall redistributes these prefixes into BGP EVPN. Type 5 routes then distribute external reachability across the fabric.

Why Type 5 Is Frequently Misunderstood

A common misconception:

“Type 5 routes replace Type 2 routes.”

Incorrect.

Type 2:

tells the fabric where a specific endpoint lives.

Type 5:

tells the fabric where an IP prefix or network lives.

Both are required.

Troubleshooting Indicators

Problems often appear as:

• Inter-subnet routing failures
• Missing tenant routes
• Route leakage issues

Verify:

VRF routing table

Type 5 advertisements

Route Targets

VNI mapping

Understanding Route Types as a System

One of the biggest mistakes engineers make is treating route types independently.

In reality they work together.

Example:

Host attaches

Creates:

Type 2

Multi-homed via ESI?

Creates:

Type 1
Type 4

Tenant active?

Creates:

Type 3

Inter-subnet routing?

Creates:

Type 5

Everything is interconnected.

Practical Mental Model

When troubleshooting, think of EVPN route types as answering five questions:

This simple framework solves a surprising number of EVPN troubleshooting situations.

Common Exam and Interview Question

If someone asks:

“Which EVPN route type is most important?”

My answer is:

Type 2.

Because Type 2 routes carry MAC/IP bindings and endpoint reachability. The MAC Mobility extended community, attached to Type 2 routes, handles host mobility sequencing.

Most operational issues engineers troubleshoot daily involve Type 2 routes.

However, in multi-homing deployments, Type 1 and Type 4 become equally critical.

Key Takeaways

Before moving to EVPN multi-homing and VXLAN integration, every engineer should remember:

1. EVPN communicates through route advertisements.
2. Route Type 2 is the foundation of endpoint reachability.
3. Route Types 1 and 4 enable EVPN multi-homing.
4. Route Type 3 builds BUM replication lists.
5. Route Type 5 distributes IP prefixes and tenant routing information.
6. Host mobility relies on Type 2 route updates carrying the MAC Mobility extended community, which sequences endpoint location changes.
7. Missing route types often reveal the exact location of a problem.
8. Most EVPN troubleshooting begins by verifying the expected route type exists.

In Part 3, we will dive into EVPN Multi-Homing Deep Dive, ESI design, Designated Forwarder election, Aliasing, Fast Convergence, EVPN-VXLAN packet forwarding, Distributed Anycast Gateway, and ARP Suppression Explained Properly—the topics that separate engineers who can configure EVPN from engineers who truly understand how it behaves in production.

Before we begin, here is what you’ll need to execute this fast deployment:

• Core Hardware: Your Juniper SRX 100 device.
• Connectivity: Patch cables.
• Time: A clear 15-minute window for the configuration process.

Let’s dive in!

The target lab design topology is shown in Figure 1. below.

Figure 1. Lab Design Topology.

Loop two physical ports on SRX

To create a 5-router lab on a single SRX, we need to create a closed loop that allows traffic to pass between our different virtual instances. Since the SRX 100 has limited interfaces, we’ll use a two physical ports with vlan tag sub-interfaces.

The Physical Connection

Use a standard patch cable to connect two adjacent physical ports on your SRX 100. For this lab, we’ll use:

• fe-0/0/6
• fe-0/0/7

Figure 2: The custom UTP CAT 5 patch cord used for the physical loop.

set interfaces fe-0/0/6 flexible-vlan-tagging
set interfaces fe-0/0/6 fastether-options auto-negotiation  
set interfaces fe-0/0/7 flexible-vlan-tagging
set interfaces fe-0/0/7 fastether-options auto-negotiation 
commit

This physical connection will act as our high-speed, internal backbone fabric, allowing the virtual routers what we will create later to talk to each other without needing external equipment.

Figure 3. SRX Ports Looped with UTP Cable.

Configure Packet Mode and Disable Security Flow Juniper SRX

The SRX is a firewall, meaning it runs in flow mode by default, inspecting traffic and requiring security policies and zones. To turn it into a dedicated, multi-router platform that simply forwards packets, we must switch it to packet mode.

⚠️ Warning: This step removes all firewall functions and security services.

Switching the SRX to Router Mode (In Next 1 minute if we will not calculate SRX boot time)

• Delete Existing Security Configuration: We first wipe all pre-existing security settings that would conflict with packet mode.

delete security

• Enable Packet Mode: We instruct the Packet Forwarding Engine (PFE) to process traffic on a per-packet basis, like a traditional router. I like to enable for all address families.

set security forwarding-options family inet mode packet-based
set security forwarding-options family inet6 mode packet-based
set security forwarding-options family mpls mode packet-based
set security forwarding-options family iso mode packet-based

• Commit and Reboot, this change is fundamental and requires a system reboot to take effect.

commit and-quit
request system reboot

(Go grab a coffee while it reboots! You’ve already completed the hardest part.)

Now that our SRX is a pure router, we can segment it into five distinct logical routers using Virtual Router (VR) instances. A VR instance is a type of routing instance that creates completely separate routing tables and forwarding domains.

Create the 5-Router Topology with Virtual Router Instance type

Topology Goal We will create five isolated routing domains (AS1 through AS4 and PE) and then connect them using logical interfaces to form a virtual network.

The Virtual Router Configuration (Next 5 minutes)

set routing-instances AS1 instance-type virtual-router
set routing-instances AS2 instance-type virtual-router
set routing-instances AS3 instance-type virtual-router
set routing-instances AS4 instance-type virtual-router
Global routing table on Juniper SRX will represent PE.

Figure 4: Representing logical connectivity between routing instances.

Next Step: Connecting the Virtual Routers

Below you can find the interface configuration for fe-0/0/6 and fe-0/0/7:

fe-0/0/6

set interfaces fe-0/0/6 unit 12 vlan-id 12
set interfaces fe-0/0/6 unit 12 family inet address 10.90.12.2/30
set interfaces fe-0/0/6 unit 13 vlan-id 13
set interfaces fe-0/0/6 unit 13 family inet address 10.90.13.2/30
set interfaces fe-0/0/6 unit 23 vlan-id 23
set interfaces fe-0/0/6 unit 23 family inet address 10.90.23.2/30
set interfaces fe-0/0/6 unit 24 vlan-id 24
set interfaces fe-0/0/6 unit 24 family inet address 10.90.24.2/30
set interfaces fe-0/0/6 unit 113 vlan-id 113
set interfaces fe-0/0/6 unit 113 family inet address 10.90.113.2/30
set interfaces fe-0/0/6 unit 114 vlan-id 114
set interfaces fe-0/0/6 unit 114 family inet address 10.90.114.2/30

fe-0/0/7

set interfaces fe-0/0/7 unit 12 vlan-id 12
set interfaces fe-0/0/7 unit 12 family inet address 10.90.12.1/30
set interfaces fe-0/0/7 unit 13 vlan-id 13
set interfaces fe-0/0/7 unit 13 family inet address 10.90.13.1/30
set interfaces fe-0/0/7 unit 23 vlan-id 23
set interfaces fe-0/0/7 unit 23 family inet address 10.90.23.1/30
set interfaces fe-0/0/7 unit 24 vlan-id 24
set interfaces fe-0/0/7 unit 24 family inet address 10.90.24.1/30
set interfaces fe-0/0/7 unit 113 vlan-id 113
set interfaces fe-0/0/7 unit 113 family inet address 10.90.113.1/30
set interfaces fe-0/0/7 unit 114 vlan-id 114
set interfaces fe-0/0/7 unit 114 family inet address 10.90.114.1/30

Assigning interfaces to virtual router instance AS1

set routing-instances AS1 interface fe-0/0/7.12
set routing-instances AS1 interface fe-0/0/7.13
set routing-instances AS1 interface lo0.1

Assigning interfaces to virtual router instance AS2

set routing-instances AS2 interface fe-0/0/6.12
set routing-instances AS2 interface fe-0/0/6.23
set routing-instances AS2 interface fe-0/0/7.24
set routing-instances AS2 interface lo0.2

Assigning interfaces to virtual router instance AS3

set routing-instances AS3 interface fe-0/0/6.13
set routing-instances AS3 interface fe-0/0/7.23
set routing-instances AS3 interface fe-0/0/7.113
set routing-instances AS3 interface lo0.3

Assigning interfaces to virtual router instance AS4

set routing-instances AS4 interface fe-0/0/6.24
set routing-instances AS4 interface fe-0/0/7.114
set routing-instances AS4 interface lo0.4

The final result of the configuration is shown in Figure 5.

For routing and inter-instance communication, your newly created virtual routers support a wide range of protocols, including:

• Static Routing
• OSPF (Open Shortest Path First)
• ISIS (Intermediate System to Intermediate System)
• BGP (Border Gateway Protocol)
• MPLS (Multi-Protocol Label Switching)
• With support for IPv4 and IPv6.

Figure 5. 5 Routers network topology.

The 15-Minute Network is Live!

Congratulations! You’ve successfully converted your single SRX 100 into a complex 5-node routing lab, all within the time it takes to brew a cup of tea. This setup proves that complex labs don’t require complex hardware. Use this topology to master OSPF, BGP, or MPLS.

Go build something great!

Back to Top

How do you define precisely what traffic is allowed, and what is denied, between different application tiers or external networks? This is where Cisco ACI Contracts come in. They are the core enforcement mechanism for defining communication policies and building a zero-trust security model within your ACI fabric.

Standard Contracts: Building the ACI Security Whitelist

1. Filters are the most granular rules, specifying permitted traffic based on network attributes such as EtherType, IP, ICMPv4 and v6 Source and Destination Port/Range.

2. Subjects are Logical containers for Filters that define the directionality of the traffic flow. It also defines the directionality of the communication using options like “Apply Both Directions” and “Reverse Filter Ports” these “Subject” options are responsible for automatically creating the necessary return path TCAM entry (ACL rule) for a connection.

3. Scopes Settings (Application Profile, VRF, Tenant, and Global) that establish the contract’s domain, controlling how widely it can be shared and reused across the ACI environment.
   • Application Profile scope - communication policies that are strictly internal to a single application, ensuring that the contract is not accidentally used by other applications, even in the same VRF.
   • VRF scope - The default and most common scope for defining communication policies between EPGs that share a Layer 3 forwarding domain. This scope is used for most intra-VRF contracts.
   • Tenant Scope - Primarily for Inter-VRF (route leaking) communication when both VRFs are in the same tenant. It allows EPGs in one VRF to consume a contract provided by an EPG in a different VRF, as long as they are in the same tenant.
   • Global scope - Inter-Tenant contracts, where the provider and consumer are in different tenants. For contracts defined in the Common Tenant that need to be universally available (e.g., contracts for shared services like DNS/NTP). or Inter-VRF communication where the VRFs are in different tenants.
4. Action specifies whether to permit, deny, or log the traffic (deny actions are supported in APIC Release 3.2(x) and later with EX/FX switches).

As many are still struggling with ACI contract direction (Consumer to Provider and Provider to Consumer), let me reveal one secret:

• Consumer is the SOURCE
• Provider is the DESTINATION

Apply your policies using this simple traffic flow rule, and you’ll get it right every time!

Below figures 1 through 3 illustrate the operational behavior of the “Apply Both Directions” and “Reverse Filter Ports” settings.

Figure 1. Applying contract between two EPGs.

The Consumer EPG can only initiate a new session with the Provider EPG on the specified port, which in this case is TCP 22 for SSH.
However, communication won’t be established because the contract is currently applied only in the Consumer-to-Provider direction, causing the necessary return traffic to be dropped. Therefore, we have two options:

• either create a second contract in the Provider-to-Consumer direction or
• use the “Apply Both Directions” option with “Reverse Filter Ports.”

Since contracts utilize hardware resources on the leaf switches, I recommend using “Apply Both Directions” with “Reverse Filter Ports.”

Figure 2. Feature “Apply Both Direction” Enabled.

With “Apply Both Directions” (Option ON):

• The Consumer EPG can initiate a connection to the Provider EPG.
• The Provider EPG can also initiate a new, separate connection back to the Consumer EPG on the same port(s).

Based on Figure 2. SRV1 and SRV2 can open SSH session to DB1 and DB2 and vice versa DB1 and DB2 can open SSH session to SRV1 and SRV2 but communication won’t be established.

Figure 3. Representing establishment of SSH session from Jumphost (SRV1 and SRV2) to DB servers.

Note:
Contracts can be applied only on unicast traffic. Broadcast, Unknown unicast and Multicast Protocols (BUM) traffic and below listed protocols are implicitly permitted.

• ARP reply (unicast)
• DHCP v4 (prot 0x11, sport 0x44, dport 0x43)
• DHCP v4 (prot 0x11, sport 0x43, dport 0x44)
• DHCP v6 (prot 0x11, sport 0x222, dport 0x223)
• OSPF (prot 0x59)
• EIGRP (prot 0x58)
• PIM (prot 0x67)
• IGMP (prot 0x2)
• ND-Sol ICMPv6 (prot 0x3a dport 0x0087)
• ND-Advt ICMPv6 (prot 0x3a dport 0x0088)

Moreover, in case of Multi-Site deployment, Cisco Multi-Site Orchestrator (MSO) creates one contract for each subject.

Back to Top

Taboo Contracts: ACIs Blacklist for Overriding Security Policies

Taboo contracts are specialized contracts used to explicitly deny specific traffic for a single EPG. Unlike standard contracts, they are:

• Unidirectional - Always provided by the EPG, with no consumer relationship.
• Higher Priority - Evaluated before standard contracts in the hardware’s Policy CAM.
• Limited Scope - Applied to a single EPG to block specific traffic, such as insecure protocols.
• Taboo contracts are not supported with Endpoint Security Groups (ESGs).
• They cannot be applied to vzAny.
• Cisco best practices generally advise against its regular use in production environments and suggest to use standard contracts with deny action if it is needed.

Example Use Case:

1. You want to block HTTP (port 80) traffic to Web_EPG while allowing other traffic via standard contracts.
2. Create a taboo contract Deny_HTTP_Taboo with a filter for TCP port 80 and apply it as provided by Web_EPG.

Figure 4. A Taboo-Contract blocking access to HTTP (insecure TCP port 80).

Back to Top

vzAny Contracts: The Shortcut to allow communication inside VRF and more

vzAny is a powerful construct that simplifies the application of security policies across all Endpoint Groups (EPGs) and Endpoint Security Groups (ESGs) within a Virtual Routing and Forwarding (VRF) instance.

Key characteristics of vzAny:

1. VRF-Specific - vzAny is scoped to a single VRF and cannot directly apply to EPGs/ESGs in other VRFs.
2. Contract Simplification - Instead of applying a contract to each EPG/ESG individually, vzAny applies it to all EPGs/ESGs in the VRF simultaneously. Remember vzAny representing all EPGs inside VRF.
3. Provider or Consumer Role - vzAny can act as a contract provider (offering services) or consumer (accessing services), depending on the configuration.
4. Standard Contracts Only - vzAny supports standard contracts but not taboo contracts, which are limited to individual EPGs.

Figure 5. Representation of applying the contract per EPG.

Figure 6. Representing of applying the contract to vzAny.

Under the “Application Profile” and “Topology” tabs, you can view the graphical representation of the relationship between EPGs and Contracts. Below Figure 6 illustrates how DNS and NTP services are shared for all EPGs within the VRF using vzAny.

Note the checkbox on the right, “Show VRF in EPG,” which helps us identify which VRF a particular EPG belongs to.

Figure 7. Contract relationship: vzAny consumes services provided by DNS and NTP EPGs.

Please Note: External EPGs that are associated with L3Outs and are part of a VRF are also included in the vzAny logical group.

Limitations of vzAny:
1. VRF Scope - vzAny is limited to a single VRF. It cannot directly apply contracts to EPGs/ESGs in other VRFs without additional configuration (e.g., contract export or shared services).
2. No Taboo Contracts - vzAny does not support taboo contracts, which are used to deny specific traffic for individual EPGs.
3. Over-Permissiveness Risk - Applying contracts to vzAny affects all EPGs/ESGs in the VRF, which may lead to unintended traffic flows if not carefully planned.
4. Not for Granular Control - For specific EPG-to-EPG policies, individual contracts are preferred over vzAny.

Back to Top

Preferred Groups: When to Unenforce Policy in ACI

A Preferred Group is a VRF-level configuration that designates a subset of EPGs or ESGs within a VRF as members of a group that can communicate with each other without contracts. When an EPG or ESG is added to the Preferred Group, it can send and receive traffic to/from other Preferred Group members within the same VRF, bypassing the need for standard contracts, provided the VRF is in enforced mode. I would like to mentioned that only one group can exist per VRF.

Limitations of Preferred Groups:

1. Preferred Groups are limited to a single VRF.
2. No Taboo Contract Support: Taboo contracts, which explicitly deny traffic for individual EPGs, cannot be overridden by Preferred Groups. If a taboo contract is applied to an EPG, it takes precedence.
3. All-or-Nothing Communication, preferred Group members can communicate freely with each other for all traffic types.
4. Not for External Connectivity: Preferred Groups do not apply to L3Out external EPGs for external connectivity; contracts are still required.
5. Policy CAM Usage: Preferred Group rules consume Policy CAM on leaf switches, which can impact scalability in large deployments.
6. If the VRF is unenforced, contracts are bypassed, and the Preferred Group has no effect (all EPGs/ESGs can communicate freely).

Back to Top

Unenforced VRFs: The Full Open Communication inside VRF

An unenforced VRF is a VRF configuration in Cisco ACI where the policy enforcement is disabled.

In this mode:

1. All EPGs and ESGs within the VRF can communicate with each other without requiring contracts.
2. The unenforced setting applies only to intra-VRF traffic; inter-VRF communication still requires contracts and route leaking.
3. Traffic flows freely between endpoints within the VRF.
4. Contracts, including standard and taboo contracts, are bypassed for intra-VRF communication.
5. Features like Preferred Groups and vzAny become irrelevant in an unenforced VRF, as all traffic is permitted.

Table 1. Representing comparison Preferred Groups vs. vzAny vs. Unenforced VRF.

Back to Top

This post will demystify the core building blocks of Cisco ACI, guiding you through the concepts of Tenants, VRFs (Virtual Routing and Forwarding), Bridge Domains, Application Profiles, and Endpoint Groups (EPGs). By the end of this explanation, you'll have a clear understanding of how these elements work together to form a robust and flexible network fabric, enabling you to design and implement ACI solutions with confidence. Let's dive in!

Figure 1. Workflow of Tenant to EPG deployment.

Understanding the Tenant: The Administrative Boundary

In the ACI object model, the Tenant (fvTenant) is the root of a policy tree for a given set of configurations. All subsequent networking objects (VRF, BD, EPG, etc.) are children of a Tenant.

Default Tenants: When you initialize an ACI fabric, three default tenants are automatically created:

common: This tenant is for shared resources that can be accessed by all other tenants. Common uses include shared Layer 3 Out (L3Out) connections to external networks, common services like DNS or DHCP, and shared Layer 4-7 services (e.g., firewalls, load balancers). Objects in the common tenant can be consumed by other tenants via contracts.

infra (Infrastructure): This tenant is used for internal ACI fabric communications. It contains policies related to fabric discovery, image management, VXLAN overlay, and other underlying infrastructure functions. It’s generally not directly configured by users unless specifically instructed by Cisco for advanced scenarios like Multi-Pod or Multi-Site deployments.

mgmt (Management): This tenant is dedicated to managing the ACI fabric itself. It defines policies for in-band and out-of-band management access to the APIC controllers and fabric switches. No Direct Network Construct: A Tenant itself doesn’t map directly to a traditional network construct like a VLAN, VRF, or physical interface.

Globally Unique Names: Tenant names must be globally unique across the ACI fabric.

Figure 2. Tenant deployment workflow.

Back to Top

VRF (Virtual Routing and Forwarding): The Layer 3 Isolation Domain

Layer 3 Isolation: The fundamental purpose of a VRF in ACI, just like in traditional networking, is to provide Layer 3 isolation. Each VRF has its own independent routing table, IP address space, and set of routing protocols. This means that traffic within one VRF cannot directly communicate with traffic in another VRF unless explicitly configured to do so (e.g., via a firewall or inter-VRF routing policies).

Routing Instance: A VRF is essentially a virtual routing instance. It holds the routing information (routes, ARP/ND entries) for the subnets and endpoints associated with it.

Tenant Scoped: A VRF is always contained within a Tenant. It’s a child object of a Tenant (fvCtx in the ACI object model). This means that a VRF belongs to a specific administrative domain and its policies are managed under that tenant.

Policy Control Enforcement Preference:

• “Enforced” (Default and Recommended): This means that all communication between EPGs within this VRF (or between an EPG in this VRF and an external network) requires a Contract.
• “Unenforced”: This setting implies that all EPGs within this VRF can communicate freely with each other by default, without the need for explicit Contracts. While it simplifies initial setup, it bypasses ACI’s micro-segmentation capabilities and is generally discouraged for production environments where granular security is a requirement.

Preferred Group: setting, enabled at the VRF level, allows all EPGs within that VRF to communicate with each other by default if they are part of the preferred group.

Figure 3. VRF deployment workflow

Back to Top

Bridge Domain (BD): The Layer 2 Forwarding Domain

In Cisco ACI, a Bridge Domain (BD) is a critical logical construct that represents a Layer 2 forwarding domain. It’s the ACI equivalent of a traditional VLAN or broadcast domain, but with significantly enhanced capabilities and policy integration.

Figure 4. First step in Bridge Domain deployment.

Figure 5. Second step in Bridge Domain deployment.

Figure 6. Third step in Bridge Domain deployment.

Figure 7. Bridge Domain L2 features.

Advertise Host Routes: By advertising a precise /32 (or /128) route for each individual endpoint, external routers learn the exact leaf switch (and pod/site) where the endpoint resides. This directs traffic directly to the optimal entry point into the ACI fabric.

L2 Unknown Unicast Handling: This setting defines how the ACI fabric handles unknown unicast traffic within the Bridge Domain:

• Hardware Proxy (Default and Recommended): This is the most efficient and scalable method. When a leaf switch receives an unknown unicast frame, it doesn’t flood it everywhere. Instead, it sends the frame to a spine proxy. The spine acts as a central lookup point, and if it knows the destination MAC, it forwards the frame directly to the correct leaf. If the MAC is unknown to the spine, only then does the spine flood it to relevant leaves. This significantly reduces flood domains.

• Flood: This behaves like traditional Ethernet. Unknown unicast frames are flooded to all leaf switches that have an endpoint associated with this Bridge Domain. This can be inefficient and generate unnecessary traffic, especially in large environments.

Optimize ARP Flooding: This option, when enabled, means that the ACI fabric will proxy ARP requests for known MAC addresses instead of flooding them. This reduces broadcast traffic.

Limit local IP learning to BD/EPG subnets: Restrict IP Learning: The ACI leaf switches connected to this Bridge Domain will only learn IP addresses for endpoints that fall within the explicitly defined IP subnets configured under that particular Bridge Domain, or the subnets defined within the EPGs that reside within that BD. It will ignore Out-of-Subnet IPs.

Figure 8. Bridge Domain L3 features.

IP Unicast Routing (Enabled/Disabled):

• If enabled, the ACI fabric will provide a distributed default gateway (Anycast Gateway) for any IP subnets defined within this Bridge Domain. This means the Layer 3 routing for endpoints in this BD is handled directly by the ACI leaf switches. This is the typical configuration for routed networks.
• If disabled, the ACI fabric will not provide a default gateway for the subnets in this BD. This is often used for: Networks where an external router or firewall provides the default gateway. In such cases, the ACI fabric simply acts as a transparent Layer 2 forwarding plane.

Back to Top

3.1 Subnet Configuration within a Bridge Domain: Defining IP Address Space & Reachability.

Each Bridge Domain can have one or more IP subnets associated with it. These subnets represent the IP address space for the endpoints connected to the Bridge Domain. Additionally, subnets can also be created directly under an EPG, but their purpose in that context is different: it facilitates Route Leaking and Shared Services between VRFs/Tenants with policy granularity.

Table 1. Comparing Subnets: Bridge Domain vs. EPG.

Figure 9. Subnet deployment within specific Bridge Domain.

Treat as Virtual IP address: Special handling for HA (Active/Standby) external device VIPs, ensuring proper learning and failover during HA events.

Make this IP address primary Designates a preferred source IP for the BD’s own control plane traffic when multiple subnets exist within this BD.

Advertised Externally: Advertises individual /32 (or /128) host routes for learned endpoints to external networks via L3Outs, optimizing traffic paths and VM mobility.

Shared Between VRFs: This option allows the subnet to be “leaked” and reachable by other VRFs within the same or different Tenants, usually in conjunction with Contracts for inter-VRF communication.

No Default SVI Gateway: Disables ACI’s pervasive gateway for this specific subnet (useful for secondary IPs or external gateways).

IP Data Plane Learning Enables dynamic learning of endpoint IP addresses by observing network traffic (typically enabled for general endpoints).

Back to Top

Application Profile (AP): The Application Logical Container

In Cisco ACI, an Application Profile (AP) is a logical container that groups together all the Endpoint Groups (EPGs) and Contracts needed to deploy a specific application or service. It provides a higher-level abstraction for managing application-centric policies within the ACI fabric. The primary purpose of an Application Profile is to shift the focus from network-centric configuration to application-centric configuration. Instead of configuring network policies based on VLANs or subnets, you define policies based on the application components and their communication requirements.

Figure 10. Deployment of Application profile.

Back to Top

EPG (Endpoint Group): The Granular Policy Enforcement Point

An EPG is the smallest and most critical unit to which policy (Contracts) is applied. If two endpoints are in the same EPG, they can communicate by default (unless otherwise restricted by other policies like Preferred Group or micro-segmentation). If they are in different EPGs, communication between them is blocked by default, and a Contract is required to permit specific traffic flows. This is ACI’s core “whitelist” security model. EPGs support micro-segmentation through features like intra-EPG isolation, which prevents communication between endpoints within the same EPG if required.

Type Of EPGs:

• “Classic” EPG which is representing endpoints, they can be physical servers, virtual machines, network-attached storage devices, or even clients on the internet.
• Layer 2 external outside network instance endpoint groups (l2extInstP).
• Layer 3 external outside network instance endpoint groups (l3extInstP).
• Management Endpoints groups (mgmtOoB, mgmtInB) are used for securing access to the ACI fabric’s management plane, either through a dedicated out-of-band management network or via in-band access within the fabric itself.

Figure 11. Deployment of EPG under application profile.

To provision EPG on the port/switch we have to methods:

• Static Ports: Explicitly links an EPG to a specific physical interface.
• Static Leafs (via AAEP): Links an EPG to a logical group of interfaces defined by an AAEP, effectively pushing the EPG’s VLAN to all ports associated with that AAEP.

Figure 12. EPG to Leaf Port Static Assignment.

Back to Top

Endpoint Security Groups (ESGs): Decoupling Security from Forwarding

Decoupling Security from Forwarding: This is the core architectural shift and primary benefit of ESGs compared to EPGs. EPGs combine both forwarding (VLAN/VXLAN binding, BD association) and security, whereas ESGs aim to provide security policy based on attributes, independent of the underlying network forwarding.

Figure 13. Defining Endpoint Security Groups (ESGs) for Policy Decoupling.

After defining your ESGs, you will create Contracts as usual. Associate the ESG as a Provider or Consumer of the contract, just like you would with EPGs.

Remember: Contracts typically apply ESG-to-ESG, ESG-to-L3Out External EPG, ESG-to-L2Out External EPG or ESG-to-vzAny.

By using ESGs, ACI provides an additional layer of flexibility and powerful tools for defining and enforcing security policies, especially beneficial in complex and evolving data center environments.

Back to Top

Secure and controlled access is fundamental to any network. This guide walks you through both the essentials and more advanced topics of Cisco ACI Access Policies, demonstrating how to apply them effectively in your environment and designs.

The End-to-End Workflow for Cisco ACI Access Policy Deployment

Cisco Application Centric Infrastructure (ACI) redefines network management with its policy-driven automation, moving away from traditional device-by-device configurations. At the heart of this transformative approach lies Access Policies, which dictate how physical network devices connect to the ACI fabric and how virtual and physical workloads access network services.

1.1. Interface Policies Configuration Workflow.

• VLAN Pools A VLAN Pool defines a range (or multiple ranges) of VLAN IDs that are permitted for use on leaf switch ports for EPG encapsulation. It doesn’t provision the VLANs themselves but rather reserves them for use.

• Domains and Domain Types Purpose: Domains act as a bridge, linking VLAN Pools with policies and the infrastructure. They define the scope where a particular VLAN pool can be utilized and by what type of entity.

  1. Physical Domains integrate bare-metal servers, physical appliances (e.g., firewalls, load balancers, non-virtualized servers), or other physical devices directly connected to ACI leaf ports.

  2. Fibre Channel Domains are primarily used for integrating Fibre Channel over Ethernet (FCoE) storage networks with the ACI fabric. This allows the ACI fabric to act as a converged network that carries both IP and FCoE traffic.

  3. External L3 Domains connect the ACI fabric to external Layer 3 (routed) networks or traditional routers outside the ACI fabric (Used by L3OUT).

  4. External Bridge Domains connect the ACI fabric to external Layer 2 (bridged) networks or switches outside the ACI fabric. This is used when you need to extend Layer 2 segments from the ACI fabric to external devices without routing (Used by L2OUT).

• Attachable Access Entity Profile (AAEP) The AAEP effectively defines which VLANs are allowed on an interface or group of interfaces. It does this by referencing Domains, which in turn reference VLAN Pools. If we compare with old fashion switch configuration it should be

   switchport trunk allowed vlan
  

  Common Scenarios for AAEP Scope/Design:

  1. One-to-one with Domains: Often, you’ll create an AAEP for each type of domain (Physical AAEP, 1.VMM AAEP, L3Out AAEP, L2Out AAEP, FCoE AAEP) to clearly separate their respective VLAN pools and interface associations.

  2. Functional Grouping: You might create AAEPs based on functional groups of ports, e.g., “ServerFarm_AAEP” that encompasses all necessary domains (physical and VMM) and VLANs for server connectivity.

• Interface Policy Group (IPG) acts as a collection point for individual “interface policies.” These individual policies define specific Layer 1 or Layer 2 parameters for a port.

Key Interface Policy Group Policies are:

• Link Level Policy (speed, duplex, auto-negotiation).
• CDP (Cisco Discovery Protocol) Policy (enable/disable).
• LLDP (Link Layer Discovery Protocol) Policy (enable/disable).
• LACP Policy (for Port Channels: active, passive, on, port priority).
• Storm Control Policy control Broadcast, Unknown Unicast, and Multicast (BUM).
• MCP (Mis-Cabling Protocol) Policy.
• Spanning Tree related policies (BPDU Guard/Filter).

• L2 Interface Polices (VLAN scope Global or Port Local).

• Interface Profile

Figure 1. Representing Interface Policies Configuration Flow.

Figure 2. Interface Policy Configuration within the Cisco APIC UI.

Figure 2.1. Interface Policy Configuration within the Cisco APIC UI.

Back to Top

1.2. Leaf Switches Policies Configuration Workflow.

• Leaf Switches Policy Groups they control the behavior of the entire leaf switch. This allows for consistent application of switch-level configurations to multiple leaf nodes selected by a Leaf Switch Profile.

Key Leaf Switches Policy Groups are:

• Spanning Tree
• Monitoring polices
• CoPP Leaf Policy
• NetFlow Node Policy
• BFD IPv4 and IPv6 Policy

Figure 3. Representing Leaf Switch Policies Configuration Flow.

Figure 4. Leaf Switch Policies Configuration within the Cisco APIC UI.

Figure 5. Full configuration flow for interface and leaf switch polices.

Back to Top

Leaf Access Port configuration

Figure 6. Implementation of an Interface Policy Group within the Cisco APIC user interface (UI).

After creating the Leaf Interface profile and attaching it to the Leaf Switch profile, we add specific ports under the Leaf Interface profile and select the appropriate Interface Policy Group. This approach provides reusability and simplifies automation. For Particular case we created Leaf Access Port Policy Group like below and attached to port eth1/1 inside specific Leaf Interface Profile.

Figure 7. Representing creation of Access port policy group and attaching to specific Leaf Interface profile.

Figure 8. End to end workflow to setup access port up and running on leaf switch.

Back to Top

Leaf Port Channel (PC) configuration

The foundational workflow for configuring a Port Channel in Cisco ACI closely mirrors that of a standard Access Port; however, a key distinction lies in the explicit inclusion and application of an LACP Policy within the associated Interface Policy Group, which is then specifically leveraged for the two designated physical ports on a single leaf switch to form the logical Port Channel.

Figure 9. Representing creation of Port Channel policy group.

Figure 10. End to end workflow to setup Port Channel (PC) up and running on leaf switch.

Back to Top

Leaf Virtual Port Channel (vPC) configuration

Virtual Port Channel (vPC) significantly enhances network resilience and operational uptime by logically combining two physical ports, each residing on a separate Cisco Nexus leaf switch, into a single, aggregated link. This innovative design provides superior high availability, as it creates active-active forwarding paths, ensuring that traffic can seamlessly continue to flow even if one of the leaf switches or an individual link experiences a failure.

Figure 11. Representing creation of Virtual Port Channel (vPC) policy group.

Figure 12.End to end workflow to setup Virtual Port Channel (vPC) up and running on leaf switches.

Back to Top

In today’s rapidly evolving IT landscape, the ability to design flexible and resilient networks is more critical than ever. This blog will provide you with ideas on what you can do with Cisco ACI fabric, covering various use cases and key design points. It will be especially valuable for those preparing to deploy Cisco ACI and who are still unsure which design to choose. The following Cisco ACI design types will be the topics of discussion:

So let’s start!

Cisco ACI Traditional Single Pod

Figure 1. Cisco ACI Single Pod Fabric.

Main Characteristics:

• Architecture: Consists of a single APIC (Application Policy Infrastructure Controller) cluster managing one ACI fabric.
• Management: Centralized management with a single point of control.
• Scalability: Limited to the capacity of one fabric.
• Latency: Lower latency within the single fabric.
• Failure Domain: Single failure domain, meaning any issue can affect the entire fabric.

Key Differences between Cisco ACI Single and Multi Pod Fabric:

• Scalability: Multi-Pod offers greater scalability by allowing multiple Pods to be managed under one APIC cluster.
• Resiliency: Multi-Pod improves resiliency by isolating failure domains.
• Latency: Single Pod has lower latency within the fabric, while Multi-Pod may have slightly higher latency due to inter-Pod communication

Back to Top

Cisco ACI Remote leaf

Figure 2. Cisco ACI Extending fabric with remote leaf to remote location.

Main Characteristics:

• VxLAN Tunnels: Unicast traffic is forwarded through VXLAN tunnels over Layer 3. This ensures that traffic between the remote leaf and the main fabric is encapsulated and securely transmitted.
• Head End Replication (HER): For Layer 2 broadcast, unknown unicast, and multicast (BUM) traffic, Head End Replication (HER) tunnels are used. This method avoids the need for multicast in the WAN.
• Policy Consistency: All policies defined in the main data center are consistently applied to the remote leaf switches, ensuring uniform policy enforcement across the entire fabric.
• Local Switching: Traffic between endpoints connected to the remote leaf is switched locally, while traffic requiring spine proxy services is forwarded to the main fabric.

Inter-Pod Network Requirements for Remote Leaf deployment:

• L3 network should support 50 Bytes overhead for VxLAN if we have default MTU 1500 Bytes.
• The latency between the ACI main DC and the remote Leaf should be less than 300 msec Round Trip Time (RTT)
• A minimum of 100 Mbps bandwidth in the IP Network. Starting from Cisco ACI Release 4.2.4, the requirement has been lowered to 10 Mbps of bandwidth.
• The remote leaf solution requires the /32 Tunnel End Point (TEP) IP addresses of the remote leaf switches and main data center leaf/spine switches to be advertised across the main data center and remote leaf switches without summarization.

Use Cases:
Satellite/Co-Location Data Centers:
Enterprises or service providers often have a main data center and several smaller satellite data centers distributed across multiple locations.
Disaster Recovery Sites:
Organizations need to maintain disaster recovery sites to ensure business continuity in case of a failure at the primary site.
Branch Offices:
Companies with multiple branch offices require consistent network policies and centralized management.
Edge Computing:
With the rise of edge computing, data processing is moved closer to the source of data generation example telco 5G.

Back to Top

Cisco ACI Mini

Figure 3. Cisco ACI mini - connecting virtual APICs to fabric.

Main Characteristics

• APIC Cluster: Consists of one physical APIC and two virtual APICs (vAPICs) running in virtual machines. Reduces the physical footprint and cost of the APIC cluster, making it suitable for environments with limited rack space or budget.
• Deployment Scenarios: Ideal for colocation facilities, single-room data centers, or any scenario where full-scale ACI installations are impractical.
• Management and Policy: Provides centralized management and policy enforcement similar to full-scale ACI deployments. Does not support features like Multi-Pod, Virtual Pod, or Remote Leaf switches.
• Integration: Multi-Site Support: Can be integrated with Cisco ACI Multi-Site for broader deployment scenarios.
• Time Synchronization: Requires the physical APIC and the ESXi server hosting the virtual APICs to be time-synced with the same NTP server to avoid synchronization issues.

Use Cases:

• Colocation Facilities: Businesses that use colocation facilities to host their IT infrastructure.
• Single-Room Data Centers: Small businesses or branch offices with a single-room data center.
• Development and Test Environments: Organizations that need a dedicated environment for development and testing.
• Edge Computing: Deployments at the edge of the network, closer to data sources.

Back to Top

Cisco ACI Multi-Pod Interconnect via IPN

Figure 4. Cisco ACI Multi-Pod Interconnected via Inter-Pod Network.

Main Characteristics

• Centralized Management: Managed by a single APIC cluster, providing a unified management and policy domain across all Pods1.
• Fault Isolation: Control plane protocols (e.g., IS-IS, COOP, MP-BGP) operate independently within each Pod, enhancing fault isolation.
• Scalability: Supports the interconnection of multiple Pods, allowing for greater scalability and flexibility in data center design.
• Inter-Pod Network (IPN): Pods are connected via a Layer 3 Inter-Pod Network (IPN), which uses VXLAN encapsulation for data plane traffic and MP-BGP EVPN for control plane communication.
• End-to-End Policy Enforcement: Ensures consistent policy enforcement across all Pods, maintaining uniform security and network policies.
• Resiliency: Isolated failure domains within each Pod reduce the impact of failures, enhancing overall resiliency.
• Geographical Flexibility: Pods can be located in different physical data centers, including geographically dispersed locations, with up to 50 ms RTT latency.

Inter-Pod Network (IPN) Requirements:

• OSPF: Required for routing between Pods.
• PIM Bi-Dir: Used for handling unknown unicast, multicast, and broadcast traffic.
• MTU Size: The IPN must support jumbo frames with an MTU of 9150 to accommodate VXLAN encapsulated traffic. In newer APIC version there is possibility to perform tunning of the MTU size of all the control plane packets generated by ACI nodes (leaf and spines), including inter-Pod MP-BGP control plane traffic.
• DHCP Relay Functionality: Needed to allow Spine and Leaf nodes to be discovered across the IPN.
• Prioritization: Ensures that APIC-to-APIC communication is prioritized over the IPN.
• Interface Support: 10/40/100G
• VLAN 4: Used for 802.1Q encapsulation between the spine and IPN devices.
• VRF Context: Recommended to isolate IPN traffic from other services, ensuring stable IPN connectivity.

Use Cases:

• Disaster Recovery and Business Continuity: Organizations need to ensure business continuity and disaster recovery across multiple data centers.
• Geographically Dispersed Data Centers: Enterprises with data centers in different geographical locations require a unified network fabric.
• Active/Active Data Centers: Businesses need to deploy applications across multiple active data centers for load balancing and high availability.
• Scalability and Flexibility: Large enterprises or service providers need to scale their data center infrastructure.

Back to Top

Cisco ACI Multi-Pod Interconnect Spine back to back.

Figure 5. Cisco ACI Multi-Pod, pods connected directly via Spines.

The characteristics and use cases for Cisco ACI Multi-Pod with spine back-to-back interconnect are essentially the same as those for Cisco ACI Multi-Pod interconnected via an Inter-Pod Network (IPN). The primary distinction is the absence of an IPN. However, it is crucial to ensure direct connectivity between remote spines.

Advantages:

• Simplified Network Design: Eliminates the need for an Inter-Pod Network (IPN), simplifying the overall network design and reducing complexity.
• Reduced Latency:
  Provides direct connectivity between remote spines, which can reduce latency compared to traversing an IPN.
• Cost Efficiency:
  Reduces the need for additional IPN devices and infrastructure, leading to cost savings.
• Improved Performance:
  Direct spine-to-spine connections can optimize traffic flow and improve overall network performance.
• Enhanced Resiliency:
  Offers redundant paths between Pods, enhancing network resiliency and fault tolerance.

Back to Top

Cisco ACI Multi-Site

Figure 6. Cisco ACI Multi-Site managed by Nexus Dashboard Orchestrator.

Key Differences between Multi Site and Multi Pod deployment:

• Management: Multi-Pod uses a single APIC cluster, while Multi-Site uses multiple APIC clusters managed by NDO.
• Latency: Multi-Pod typically has lower latency between pods, whereas Multi-Site can handle higher latency across sites.
• Scalability: Multi-Site is better suited for large-scale, geographically dispersed deployments.

Main Characteristics:

• Geographical Flexibility: Supports the interconnection of geographically dispersed data centers, extending Layer 2 and Layer 3 connectivity between sites.
• Unified Policy Management: Utilizes the Nexus Dashboard Orchestrator (formerly Multi-Site Orchestrator) to manage and enforce policies across multiple sites.
• Fault Isolation: Each site operates with its own APIC cluster, ensuring fault isolation and independent control planes.
• Scalability: Allows for the integration of multiple ACI fabrics, providing scalability for large and distributed environments.
• Consistent Policy Enforcement: Ensures consistent policy enforcement across all interconnected sites, maintaining uniform security and network policies.
• Data Plane Communication: VXLAN and MP-BGP EVPN: Uses VXLAN for data-plane communication and MP-BGP EVPN for control-plane communication between sites.

Inter-Site Network (ISN) Requirements: ACI fabrics are connected via ISN routed network which will be used for VxLAN communications between endpoints which they belong to different fabrics. I would like to highlight that there is no latency limit between the different ACI fabrics part of the same Multi-Site domain.
The only latency considerations are:

• Up to 150 ms, RTT is the latency supported between the Nexus Dashboard cluster nodes where the Orchestrator service is enabled.
• Up to 500 msec RTT is the latency between each Nexus Dashboard Orchestrator node and the APIC controller nodes that are added to the Multi-Site domain. This means that the Multi-Site architecture has been designed from the ground up to be able to manage ACI fabrics that can be geographically dispersed around the world.
• No Need for Multicast.
• MTU between ISN routers and spines should be increases to minimum 1600 Bytes, in this case should be changed default MTU size 9000 Bytes in global configuration.
• IP connectivity can be achieved using OSPF, BGP or static routing.

Use Cases:
Centralized Data Center with Separate Availability Zones:
Allows for the creation of distinct availability zones within a centralized data center, enhancing fault isolation and resource management.
Disaster Recovery:
Facilitates disaster recovery by enabling data replication and failover between sites, ensuring business continuity in case of site failures.
Geographically Distributed Data Centers:
Supports the management of data centers spread across different cities, countries, or continents from a single pane of glass, simplifying provisioning, monitoring, and management.
Stretched Bridge Domain with Layer 2 Broadcast Extension:
Extends Layer 2 domains across sites, allowing for live VM migration and active/active high availability.
Stretched VRF with Inter-Site Contracts:
Enables consistent policy enforcement across sites by stretching VRFs and applying inter-site contracts.
Shared Services with Stretched Provider EPGs:
Allows shared services to be accessed across multiple sites using stretched Endpoint Groups (EPGs).

Back to Top

MultiPod MultiSite hybrid deployment

Figure 7. Extending Cisco ACI Multi-Pod to Multi-Site.

Using Cisco ACI Multi-Site and Multi-Pod as a single fabric offers several benefits, enhancing the overall network architecture. Here are some key advantages:

• Centralized Management:
  Both Multi-Site and Multi-Pod architectures allow for centralized management through a single APIC cluster, simplifying policy enforcement and configuration.
• Scalability:
  Multi-Pod enables the expansion of a single ACI fabric across multiple locations without the need for separate fabrics, making it easier to scale the network.
  Multi-Site allows for the interconnection of multiple ACI fabrics, providing scalability across geographically dispersed data centers.
• Resiliency and Fault Isolation:
  Multi-Pod architecture offers full resiliency at the network level across pods while maintaining a single fabric, ensuring minimal impact from failures.
  Multi-Site architecture isolates failure domains between sites, enhancing overall network resiliency.
• Disaster Recovery and Business Continuity:
  Both architectures support disaster recovery by enabling data replication and failover between sites or pods, ensuring business continuity in case of site failures.
• Consistent Policy Enforcement: Policies and configurations can be consistently applied across all sites and pods, ensuring uniform security and compliance.
• Operational Efficiency:
  Centralized management and automation reduce administrative overhead and operational complexity, making it easier to manage large-scale networks.

Back to Top

Cisco ACI 6.0.2 and 6.0.3 Comparison Features, Changes, and Enhancements

1. Virtual APIC on VMware ESXi and Cloud
2. VMM Integration with Nutanix AHV
3. Nexus Dashboard - Connectivity analysis

1. Virtual APIC 6.0(2)F

• No physical APICs.
• No configuration limitations you can administer it like a regular APICs.
• Scale is pretty much same as for physical APIC.
• Limitations not able to run applications such as ELAM, policy explorer.
• APICs can be connected to leaf switches directly.
• Or they can be connected remotely via L3 network. During the initial setup, the wizard will ask you to choose how the APIC is connected to the network. Figure 1 below is an example.
• Virtual APIC Cluster Fully functional APIC that can manage up to 200 switches.
• 6.0(2)F Cloud hosted APIC Cluster can only be deployed within AWS Cloud.

Figure 1 ACI setup wizard

2. Integration with Nutanix AHV hypervisor 6.0(3)F

• APICs connect and authenticate to Prism Manager.
• Integration of Nutanix Acropolis Hypervisor (AHV) with Cisco Application Centric Infrastructure (ACI).
• This integration provides virtual and physical network automation and VM endpoints visibility in Cisco ACI.
• End to end network automation and end to end visibility.

3. Nexus Dashboard - Connectivity Analysis 6.0(2)F

• Troubleshooting between two End points, src and dst IP addresses or src and dst MAC addresses.
• Visibility of what happening between control and data plane.
• Verification of control and data plane.
• Traces all possible forwarding paths between source and destinations.
• Identifies if there are problems in the path and what is causing issues.
• Helps to highlight real root cause, perform forwarding checks.

4.Interface policy - simplify configuration

• This type of configuration is optional there is no need to change if you already master polices configuration. On that way you will keep everything under your control.
• This is more for new guys which is starting to use ACI Fabric.
• Guys with experience will not have so many benefits from this feature.
• It is compatible with previous manual configuration of polices.
• Not recommended for brown field deployments, I would like to recommend only for green field deployment.

5.Weight-Based Symmetric PBR

In 6.0 it was introduced Weight-based PBR it means that we can send more traffic to specific node. Use case:
• On specific node we have connected Load Balancer.
• On specific node we have more powerful firewall in the group.
• Introducing new firewall, we can redirect small amount of traffic to new firewall for test purposes.

6.Stretched L3out SVI across remote leaves.

• Supported for stretching a non-vPC L3Out SVI across remote leaf switches Use cases:
• Mobile packet core.
• Containerized workloads special at the Edge.
• Cloud native network functions CNF.

7. Multi-Site support for vzAny PBR and L3Out-to-L3Out PBR use cases. From ACI 6.0(4)F and NDO 4.2(3) following uses cases are supported:

• vzAny-to-vzAny (Intra VRF).
• vzAny-to-EPG (Intra VRF).
• vzAny-to-L3out (Intra VRF).
• L3out-to-L3out ((Intra VRF) and inter-VRF).
• Standard EPG-to-EPG subnet under consumer still is required. If you have plan to implement vzAny inside multi-site design, please check all details on link below: Cisco Nexus Dashboard Orchestrator Configuration Guide for ACI Fabrics.

8. Main Routing improvements Multi-Site support for vzAny PBR and L3Out-to-L3Out PBR use cases.

• BGP AS - Remove private AS from AS_PATH.
• Support for BFD on Secondary subnet IPv4/IPv6.
• BGP Additional Paths - Receive multiple paths for the same prefix without the new paths replacing any previous paths.

9. Operating and Administering improvements.

ACI ver.6.0(1)F

• Simplifying interface configuration.
• Secure Erase (RMA).

ACI ver.6.0(2)F

• Auto firmware update for Cisco APIC on discovery - APIC will be automatically upgrade on version same as fabric.
• Installing switch SMU without reload.
• Troubleshooting of QoS Polices.

ACI ver.6.0(3)F

• Reduce of time of APIC upgrade.
• Transaction-based Logging.
• Memory-based Automatic Switch Image Selection 32/64 bit.

ACI ver.6.0(4)F

• Streamlined navigation and editing of the stat collection threshold from a TCA fault.

10. Main Security improvements

• Cisco Nexus 9K switch secure erase.
• Support for a user group map rule for SAML and OAuth2.
• Support for TLS ver 1.3
• First hop security FHS support for VMM.
• TACACS external logging for switches.
• RSA Key 307 and 4096 support on X.509 certificates.

11. Integrations.

• Enhanced LLDP TLVs for Azure Stack HCI.
• Policy mode for Cisco ACI and VMware NSX-T integration.
• VMM Domain support for VMware vSphere 8.0.

12. Scale improvements.

References:
Cisco Application Policy Infrastructure Controller
Cisco ACI Releases Changes in Behavior

Dive into the world of Clos Topology, a concept born from a 1952 paper by Charles Clos, a researcher at Bell Laboratories. His revolutionary ideas on multistage telephone switching systems have transcended time, now serving as the foundation for building highly scalable data centers with cost-effective switches. The Cisco ACI fabric is based on either a two-tier (spine and leaf switch) or three-tier (spine switch, tier-1 leaf switch, and tier-2 leaf switch) architecture. In a Cisco Application Centric Infrastructure (ACI), each of the leaf, Fabric Extenders and spine switches has different functions.

Leaf Switches

• Leaf switches are the ingress/egress points for traffic into and out of an ACI fabric.
• They connect directly to servers and other devices, facilitating rapid data transfer.
• Each leaf switch is linked to every spine switch, creating a mesh of high-bandwidth connections.
• This setup ensures that any data packet is only a couple of hops away from its destination, significantly reducing latency.

Fabric Extenders (FEX)

Fabric Extenders (FEX) in Cisco’s Application Centric Infrastructure (ACI) have several key functions:

• Port Extension - FEX extends the ports of a Nexus switch, acting as a remote line card. This allows for the use of more cost-effective devices with a reduced set of features that are connected to the Leaf Switches.
• Agile Connectivity - FEX offers agile connectivity for rack and blade servers and for converged fabric deployments. It delivers innovation to the data center, reduces total cost of ownership, and gains architectural flexibility.
• Configuration and Management - All management is done on the parent switch; there is no console/vty access on the FEX. This simplifies the management of the network infrastructure.
• Cisco ACI leaf switch models with -G or -Tin the product name, such as Cisco Nexus N9K-C9348GC-FXP, N9K-C93108TC-FX, N9K-C93108TC-FX-24, N9K-C93108TC-EX, N9K-C93108TC-EX-24, N9K-C93216TC-FX2, and N9K-93108TC-FX3P.

Discover the steps to configure Fabric Extenders (FEX) in Cisco ACI in Cisco comprehensive guide available here.

Spine Switches

Spine switches are Clos intermediary switches that have a number of key functions:

• They are responsible for high-capacity data transmission across the network.
• They ensure that data packets travel efficiently from one leaf switch to another.
• The spine layer is crucial for maintaining the performance of the network as it scales, providing a robust framework that can handle increasing data demands without bottlenecks.
• Spine switches interconnect all leaf switches in a full-mesh topology.

Together, they create an architecture that is highly standardized across deployments. This design allows for a highly scalable and efficient network infrastructure.

Two-Tier or Multi-Tier Design?

Below Figure 1 illustrates the necessary components and cabling for an two-tier ACI fabric. It’s important to note that no cables should be connected between ACI leaf switches, as this is characteristic of Clos network design. Similarly, cross-cabling ACI spines will lead to the disabling of the cross-connected ports. The topology depicts a full mesh of cabling between the spine and leaf layers. However, it’s important to note that while the Spine-Leaf architecture can be implemented without ACI, with the switches working in NX-OS mode. I would like to highlight that ASICs are different for Leaf and Spine switches, so a switch like the Nexus N9K-93180YC-EX cannot run as Spine switches.

Figure 1 Two-Tier Topology Spine-Leaf classic interconnect between leaf and spine switches

In summary, I would like to recommend to check switch compatibility on the links below:

• Cisco Nexus 9300 1/10GBaseT Switches
• Cisco Nexus 9300 1/10/25/40/50/100GE Fiber Switches
• Cisco Nexus 9300 40/100 GE Switches
• Cisco Nexus 9300 400 GE Switches
• Cisco Nexus 9300 ACI Fixed Spine Switches Data Sheet

If you plan to use Cisco Nexus modular switches for Cisco ACI fabric technical details you can find on below links:

• Cisco Nexus 9500 Series Switches Data Sheet
• Nexus 9500 Chassis Comparison
• Nexus 9800 Compare Models
• Nexus 9800 Line Cards comparison

Starting from Cisco ACI 4.1, the fabric also supports a multi-tier (three-tiers) topology. This includes two tiers of leaf switches, allowing for vertical expansion of the Cisco ACI fabric. This is particularly useful for migrating a traditional three-tier architecture (core-aggregation-access), a common design model for many enterprise networks. The ACI Multi-Tier is a beneficial solution for data centers aiming to reduce inter-row cabling and have low-bandwidth needs for top-of-rack switches. In this setup, Tier 1 leaf switches are placed at the end of each row, while Tier 2 leaf switches are positioned at the top of each rack. This strategy is particularly effective in managing cables and meeting bandwidth requirements.

Figure 2 Multi -Tier Topology Spine- Tier 1 Leaf and Tier 2 Leaf

Figure 3 Multi -Tier Topology Spine- Tier 1 Leaf and Tier 2 Leaf option between two data center buildings

The topology requirements may include various models of spine and leaf switches, please check below:

• Spine: EX/FX/C/GX/GX2 spines (For example, Cisco Nexus 9332C, 9316D-GX, 9348D-GX2A, 9408 and 9500 with EX/FX/GX linecards).

• Tier-1 Leaf: EX/FX/FX2/FX3/GX/GX2 except Cisco Nexus 93180LC-EX.

• Tier-2 All EX/FX/FX2/FX3/GX/GX2.

In the scenario where server access is at 1G and there is not need for high bandwidth, it’s a viable choice to use the Cisco Nexus 93180FX3 or Nexus 9364C-GX as the tier-1 leaf and the Cisco Nexus 9348FXP as the tier-2 leaf.

Before you start to build and design a multi-tier topology in a Cisco ACI environment, please take into account the following:

1.Fabric Port Configuration: All inter-switch connections must be configured as fabric ports. This includes the connections between Tier-2 leaf switch fabric ports and Tier-1 leaf switch fabric ports.

2.Tier-2 Leaf Switch Connectivity: A Tier-2 leaf switch has the capability to connect to more than two Tier-1 leaf switches. This is in contrast to a traditional double-sided vPC design, which only supports two upstream switches. The maximum number of ECMP links supported by a Tier-2 leaf switch to a Tier-1 leaf switch is 18.

3.Endpoint Group (EPG) and Layer 3 Outside (L3Out) Connections: EPGs, L3Outs, Cisco APICs, or Fabric Extenders (FEX) can be connected to either Tier-1 or Tier-2 leaf switches.

4.Tier-1 Leaf Switch Connections: Tier-1 leaf switches can have both hosts and Tier-2 leaf switches connected to them.

5.Switch Tier Transition: Transitioning from a Tier-1 to a Tier-2 leaf switch (and vice versa) requires decommissioning and recommissioning the switch.

6.Compatibility with Cisco ACI Multi-Pod and Multi-Site: Multi-tier architectures are compatible with both Cisco ACI Multi-Pod and Cisco ACI Multi-Site.

7.Tier-2 Leaf Switch Connectivity Limitations: Tier-2 leaf switches cannot be connected to remote leaf switches (Tier-1 leaf switches).

8.Scale Considerations: The combined number of Tier-1 and Tier-2 leaf switches must not exceed the maximum number of leaf switches validated for a given release (400 per pod; 500 per Cisco ACI Multi-Pod as of Cisco ACI release 6.0(1)).

Reference for Cisco ACI Multi-tier Architecture.

Fabric Port

In Cisco's Application Centric Infrastructure (ACI), a fabric port is a type of port that is used for uplink connectivity within the fabric. These ports are typically used to connect different switches within the ACI fabric. For example, in a multi-tier topology, all switch-to-switch links must be configured as fabric ports. This includes the connections between Tier-2 leaf switch fabric ports and Tier-1 leaf switch fabric ports.

Fiber optics modules

Don't forget that based on specific switch model and uplinks design you will need to choose proper optical modules. All details about compatibility matrix for Cisco Optics and devices you can find on below links:

• Cisco Optics-to-Device Compatibility Matrix
• Cisco Optics-to-Optics Interoperability Matrix

If you struggling or you have some doubts about compatibility between switches and optical modules SFP SFP+ QSFPs I would like to recommend:

Taxonomy for Cisco Nexus 9000 Series Part Numbers

For a comprehensive understanding and decoding of all Nexus Part Numbers names, kindly refer to Figure 4.

Figure 4 Cheet-sheet for decoding Cisco Nexus 9000 Series Part Numbers

This time I will go through configuration steps for provisioning switches on Cisco ACI Fabric, NTP, DNS and BGP Route Reflectors. Steps described in this post are applicable for Cisco ACI Fabric and Cisco ACI simulator.

0. In your favorite browser address bar add https://oob.ip.address. Use IP address what you set during APIC initial configuration.

Cisco ACI Fabric APIC ver 5.2 supported browsers:

• Chrome version 59 (at minimum)
• Firefox version 54 (at minimum)
• Internet Explorer version 11 (at minimum)
• Safari version 10 (at minimum)

1. After what you will login it will appear Cisco APIC welcome screen, looks nice. Just press Let’s go as this white button suggesting.

2. One more screen from APIC this one is important as it will provided you information about:

• New features for installed APIC version of APIC.
• Updated scalability numbers.
• Very good resources and knowledge base for ACI fabric and APIC clusters.
  After what you reviewed all interesting things pres Start ACI Fabric Setup button.

3. Below we can see the most important sections which should be configured before any service configuration on ACI.

I would like to suggest following order:

1. Add leaf and spine to Cisco ACI Fabric.
2. NTP server configuration.
3. DNS server configuration.
4. BGP RR configuration for Spine.

4. Add first leaf to Cisco ACI Fabric, for serial number we will use TEP-1-102, 101 and 103 as this is Cisco ACI simulator on real Cisco ACI Fabric you will have serial numbers in following format F1234A768. For node ID choose unique number, once when you tied node ID to serial number you will not be able to change on the fly. If you would like to change you will need to do decommissioning and commissioning with new node ID. Switch name you can change anytime if you will not like assigned during initial switch provisioning.

5. Here you should pay attention and check is it switch in Registering phase, if you notice some other messages please stop and perform tshoot.

6. Once when is done we should see that switch is successfully Register as we can see below for leaf with node ID 101.

7. Same you should repeat for all Spine switches too, below you can see Registering of spine with node ID 103.

8. And Spine with node ID 103 has been successfully Register.

9. Based on workflow from point #3 I will proceed with NTP configuration. You can choose time format from local or utc. Use plus sign for adding one or more NTP servers, one of them you can set like preferable.

10. In the lab for NTP server I use public Google NTP server just for test.

11. Here I will add Google DNS server just for test. Usually you will use internal DNS. Don’t forget to add DNS records for leaf, spines and APICs.

12. Here we can check status about what we done already. Below you can notice that still we have to configure BGP route reflectors. To configure BGP just press button Begin which will redirect you to configuration page.

13. Press plus sign and add minimu two spines per Cisco ACI fabric site.

14. From drop down choose node id of your spine. As I am using Cisco ACI Simulator I have only one spine with node ID 103. For test I am using default AS number.

15. Below you can see that spine with node ID 103 was successful registered like RR in Pod 1.

16. Short overview all looks good as Proxy won’t be used.

17. Summary page, last step for configuration setup.

18. I would like to recommend you to check Dashboard sections like below on print screen

• Switches should be Healthy.
• Controllers should be In service and FullyFit.

Note
This is only zero day configuration for Cisco APIC. For MultiPod, Multisite, 
Remote Leaf and other Cisco ACI Fabric designs should be configured other parameters
and network objects.

Bonus steps for those how like to have clean ACI fabric without lot of system notifications.
Next few steps will describe how we can disable messages from Cisco Smart Software Manage, this is mostly applicable for lab deployments and ACI Simulator as after 90 days should be reconfigured.

19. From APIC notification center please choose Cisco Smart Software Manager (CSSM).

20. Press on fault notification F3057.

21. As below on print screen click on Configure Network Settings.

22. From drop down menu choose Offline click on OK and you are ready.

I have walked through most important parameters for Cisco ACI Fabric. Now you can start to play with Cisco ACI Fabric network objects or you can proceed with fine tunning for the production environment. 😎