Module 3: Advanced Traffic Management and Security

This module explores advanced features of DestinationRules and Istio’s security capabilities. You will learn about load balancing, failover, and circuit breaking in DestinationRules, as well as how to implement request authentication and authorization policies to secure your services.

DestinationRules: Beyond the Basics

You’ve already encountered DestinationRules in previous modules. Let’s review what we’ve covered and then explore additional powerful features.

What We’ve Already Seen

In Module 1, you learned that DestinationRules can configure mTLS settings. In Module 2, you used DestinationRules to define service subsets for traffic splitting. These are fundamental uses of DestinationRules:

mTLS Configuration: Setting TLS mode (e.g., ISTIO_MUTUAL) for secure service-to-service communication
Subsets: Defining service versions or variants using labels for routing purposes

DestinationRules are versatile resources that define how traffic should be handled when it reaches a destination service. They complement VirtualServices, which define where traffic should go.

Load Balancing

One of the most important features of DestinationRules is configuring load balancing algorithms. By default, Istio uses round-robin load balancing, but you can configure various algorithms to suit your needs.

Load Balancing Algorithms

Istio supports several load balancing algorithms:

ROUND_ROBIN: Distributes requests evenly across all endpoints (default)
LEAST_CONN: Routes to the endpoint with the fewest active connections
RANDOM: Randomly selects an endpoint
PASSTHROUGH: Passes requests directly to the original destination without load balancing

Configuring Load Balancing

Here’s an example of configuring load balancing in a DestinationRule:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: my-service
spec:
  host: my-service
  trafficPolicy:
    loadBalancer:
      simple: LEAST_CONN
  subsets:
  - name: v1
    labels:
      version: v1
    trafficPolicy:
      loadBalancer:
        simple: ROUND_ROBIN
  - name: v2
    labels:
      version: v2
    trafficPolicy:
      loadBalancer:
        simple: RANDOM

This example shows:
* Global load balancing policy at the DestinationRule level
* Subset-specific load balancing policies that override the global policy

Locality-Aware Load Balancing

Istio also supports locality-aware load balancing, which prioritizes endpoints in the same zone or region:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: my-service
spec:
  host: my-service
  trafficPolicy:
    loadBalancer:
      localityLbSetting:
        enabled: true
        distribute:
        - from: "region1/zone1/*"
          to:
            "region1/zone1/*": 80
            "region1/zone2/*": 20

This configuration ensures that traffic prefers local endpoints while allowing failover to other zones when needed.

Failover

DestinationRules enable you to configure failover behavior, allowing automatic redirection to backup services when primary services fail.

Outlier Detection

Failover relies on outlier detection, which identifies unhealthy endpoints and removes them from the load balancing pool. Configure outlier detection like this:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: my-service
spec:
  host: my-service
  trafficPolicy:
    outlierDetection:
      consecutiveErrors: 5
      interval: 30s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
      minHealthPercent: 50

Key parameters:

consecutiveErrors: Number of consecutive errors before ejecting an endpoint
interval: Time window for error counting
baseEjectionTime: Minimum time an endpoint is ejected
maxEjectionPercent: Maximum percentage of endpoints that can be ejected
minHealthPercent: Minimum percentage of healthy endpoints required

Failover Configuration

You can configure failover to backup services using subsets:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: my-service
spec:
  host: my-service
  trafficPolicy:
    outlierDetection:
      consecutiveErrors: 3
      interval: 30s
  subsets:
  - name: primary
    labels:
      version: v1
  - name: backup
    labels:
      version: v2
    trafficPolicy:
      outlierDetection:
        consecutiveErrors: 5

When combined with a VirtualService, you can route traffic to the backup subset when the primary fails.

Circuit Breaking

Circuit breaking is a critical resilience pattern that prevents cascading failures by stopping requests to unhealthy services.

What is Circuit Breaking?

A circuit breaker monitors service health and, when failures exceed a threshold, "opens" the circuit to stop sending requests. This prevents overwhelming a failing service and allows it to recover.

Circuit Breaker Configuration

Configure circuit breaking in DestinationRules:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: my-service
spec:
  host: my-service
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 10
        http2MaxRequests: 100
        maxRequestsPerConnection: 2
    outlierDetection:
      consecutiveErrors: 5
      interval: 30s
      baseEjectionTime: 30s
      maxEjectionPercent: 50

Connection pool settings:

maxConnections: Maximum number of TCP connections
http1MaxPendingRequests: Maximum pending HTTP/1.1 requests
http2MaxRequests: Maximum concurrent HTTP/2 requests
maxRequestsPerConnection: Maximum requests per connection

When these limits are exceeded, the circuit breaker opens and requests are rejected, preventing overload.

Circuit Breaker States

A circuit breaker operates in three states:

Closed: Normal operation, requests flow through
Open: Circuit is open, requests are immediately rejected
Half-Open: Testing if the service has recovered, allowing limited requests

The circuit breaker automatically transitions between these states based on the configured thresholds.

Request Authentication

Istio provides powerful authentication mechanisms to verify the identity of clients making requests to your services.

What is Request Authentication?

Request Authentication (RequestAuthentication) verifies JSON Web Tokens (JWTs) attached to requests. It validates that requests come from trusted sources and extracts identity information from tokens.

Request Authentication is different from PeerAuthentication:

PeerAuthentication: Verifies service-to-service communication (mTLS)
RequestAuthentication: Verifies end-user identity (JWT tokens)

RequestAuthentication Resource

Create a RequestAuthentication resource to validate JWTs:

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: jwt-auth
  namespace: my-namespace
spec:
  selector:
    matchLabels:
      app: my-app
  jwtRules:
  - issuer: "https://auth.example.com"
    jwksUri: "https://auth.example.com/.well-known/jwks.json"

Key components:

selector: Specifies which workloads this policy applies to
jwtRules: Defines JWT validation rules
issuer: The JWT issuer that must match the token
jwksUri: URI to fetch JSON Web Key Set for token validation

JWT Validation

RequestAuthentication validates:

Token signature using keys from the JWKS endpoint
Token expiration (exp claim)
Token issuer (iss claim)
Token audience (aud claim), if specified

If validation fails, the request is rejected with a 401 Unauthorized response.

Multiple JWT Providers

You can configure multiple JWT providers for different authentication sources:

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: multi-jwt-auth
spec:
  selector:
    matchLabels:
      app: my-app
  jwtRules:
  - issuer: "https://auth1.example.com"
    jwksUri: "https://auth1.example.com/.well-known/jwks.json"
  - issuer: "https://auth2.example.com"
    jwksUri: "https://auth2.example.com/.well-known/jwks.json"

Istio will accept tokens from any of the configured issuers.

RequestAuthentication Best Practices

When implementing RequestAuthentication:

Use HTTPS for JWKS endpoints to prevent man-in-the-middle attacks
Configure appropriate token expiration times
Use specific selectors to limit the scope of authentication policies
Monitor authentication failures to detect potential attacks
Combine with AuthorizationPolicy for complete access control

Authorization

While RequestAuthentication verifies who is making a request, AuthorizationPolicy controls what they can do.

What is Authorization?

Authorization determines whether an authenticated user or service is allowed to perform a specific action. Istio’s AuthorizationPolicy provides fine-grained access control based on various request attributes.

AuthorizationPolicy Resource

Create an AuthorizationPolicy to control access:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-specific-users
  namespace: my-namespace
spec:
  selector:
    matchLabels:
      app: my-app
  action: ALLOW
  rules:
  - from:
    - source:
        requestPrincipals: ["auth.example.com/*"]
    to:
    - operation:
        methods: ["GET", "POST"]
        paths: ["/api/*"]

Key components:

selector: Specifies which workloads this policy applies to
action: ALLOW or DENY (default is ALLOW)
rules: Defines the conditions for the policy

Authorization Actions

AuthorizationPolicy supports three actions:

ALLOW: Explicitly allows matching requests
DENY: Explicitly denies matching requests
AUDIT: Logs matching requests without allowing or denying (for monitoring)

Authorization Rules

Authorization rules can match on various request attributes:

Source-Based Rules

Match requests based on their source:

rules:
- from:
  - source:
      principals: ["cluster.local/ns/default/sa/my-service"]
      namespaces: ["default"]
      requestPrincipals: ["auth.example.com/user123"]

principals: Service account identities (from mTLS)
namespaces: Source namespaces
requestPrincipals: End-user identities (from JWT)

Operation-Based Rules

Match requests based on the operation:

rules:
- to:
  - operation:
      methods: ["GET", "POST"]
      paths: ["/api/*"]
      hosts: ["api.example.com"]

methods: HTTP methods (GET, POST, PUT, DELETE, etc.)
paths: URI paths (supports exact, prefix, suffix, regex)
hosts: Host headers

Custom Attribute Rules

Match requests based on custom attributes:

rules:
- when:
  - key: request.headers[user-agent]
    values: ["Mozilla/*"]
  - key: source.ip
    values: ["192.168.1.0/24"]

Deny Rules

Create explicit deny rules to block specific requests:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-admin
spec:
  selector:
    matchLabels:
      app: my-app
  action: DENY
  rules:
  - to:
    - operation:
        paths: ["/admin/*"]

This policy denies all requests to /admin/* paths.

Combining Multiple Policies

You can create multiple AuthorizationPolicies for different scenarios:

# Policy 1: Allow authenticated users to access API
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-api
spec:
  selector:
    matchLabels:
      app: my-app
  action: ALLOW
  rules:
  - from:
    - source:
        requestPrincipals: ["*"]
    to:
    - operation:
        paths: ["/api/*"]

---
# Policy 2: Deny access to admin endpoints
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-admin
spec:
  selector:
    matchLabels:
      app: my-app
  action: DENY
  rules:
  - to:
    - operation:
        paths: ["/admin/*"]

Policies are evaluated in order, and DENY policies take precedence over ALLOW policies.

Authorization Best Practices

When implementing authorization:

Follow the principle of least privilege
Use specific selectors to limit policy scope
Combine RequestAuthentication with AuthorizationPolicy for complete security
Test authorization policies thoroughly before deploying to production
Monitor authorization denials to detect potential security issues
Document policies clearly for team understanding
Use namespaces to isolate authorization policies when appropriate

Combining Authentication and Authorization

For complete security, combine RequestAuthentication and AuthorizationPolicy:

RequestAuthentication: Verifies the identity of the requester (via JWT)
AuthorizationPolicy: Controls what the authenticated requester can do

Example workflow:

# Step 1: Require JWT authentication
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: require-jwt
spec:
  selector:
    matchLabels:
      app: my-app
  jwtRules:
  - issuer: "https://auth.example.com"
    jwksUri: "https://auth.example.com/.well-known/jwks.json"

---
# Step 2: Authorize based on identity
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: require-auth
spec:
  selector:
    matchLabels:
      app: my-app
  action: ALLOW
  rules:
  - from:
    - source:
        requestPrincipals: ["auth.example.com/*"]
    to:
    - operation:
        methods: ["GET", "POST"]

This configuration:
* Requires valid JWTs from the specified issuer
* Only allows GET and POST requests from authenticated users
* Rejects all other requests

Summary

In this module, you have learned:

DestinationRules provide advanced traffic management beyond mTLS and subsets
Load balancing can be configured with various algorithms (round-robin, least-conn, random)
Failover and outlier detection help maintain service availability
Circuit breaking prevents cascading failures by limiting connections and requests
RequestAuthentication validates JWT tokens to verify user identity
AuthorizationPolicy provides fine-grained access control based on various attributes
Authentication and authorization work together to provide complete security

These advanced features enable you to build resilient, secure microservices architectures with Istio.

Next Steps

In the next module, you will explore observability features that help you monitor, trace, and understand the behavior of your service mesh.