Introduction
Today’s data center is the home to essential computation, storage and applications that support your business. Because of their key importance to the business, data centers are designed to be robust, fast and highly reliable.
This article describes options for connecting two geographically dispersed data centers that are running the same services. Such a scenario is typical for businesses running clustered database services and distributed storage systems, both of which are foundations for reliable and stable application services.
Data Center Overview
Three types of equipment are found in data centers:
server or computing equipment
networking equipment
storage equipment
Servers perform all the computing roles; applications are run on the servers, performing operations to fulfill requests from users. Networking equipment assumes the role of interconnecting servers, with the network facing the users. Data storage equipment, usually in the form of a Storage Area Network (SAN), usually consists of a standalone network that connects servers to storage systems, and is used to carry data from massive-storage media (such as disks or tapes) to the servers for processing.
The data center is usually built using all or some of the following layers: the presentation layer, the application layer and the database layer. In this layered or tiered approach, each layer performs its individual task, relying on underlying layers to supply the requested information from the user or application.
The presentation layer (or web server layer) runs web-serving software and takes care of presenting the results of underlying back-office applications to the clients accessing the service by using a web browser.
The application layer performs the business logic necessary to fulfill the user’s request.
The database layer provides all the information needed by the applications to perform the request.
The tiered approach adds robustness to the design of the data center. These data center tiers can run in parallel on the same equipment by using virtualization, or the tiers can run on separated/dedicated equipment. Each tier should be designed with high availability in mind, so that a failure in a particular server does not affect the whole service. Server load-balancing equipment, such as the Cisco Application Control Engine (ACE), is very often used to handle load distribution and failover.
Figure 1 shows a tiered data center architecture. Shown are the web server tier, application server tier, and database server tier. Database servers are usually cluster-based.
Figure 1:
Tiered data center architecture. [Source: Cisco Systems]
Database Servers and Server Clustering
The servers running the database are one of the key components of a data center. These servers must provide a quick and reliable response, so they are usually designed and deployed in server clusters. Server clusters act as one server to the requesting side, and work in load-sharing configurations. These servers also constantly monitor each other using heartbeat signals, which serve as a basis for high availability. If one server from the cluster is down, the other servers accept and load-share the requests.
Heartbeat signals usually need direct Layer-2 reachability, which must be taken into account when designing the data center access layer, where the Layer-2 domain needs to be extended to include all relevant servers on several access switches. Clustered database servers interact with each other and with the outside world, having IP addresses in one or two distinct Virtual Local Area Networks (VLANs).
Depending on the cluster design, the database cluster can use the same VLAN for both private and public communication, or use one dedicated VLAN for public communication (to clients) and one for private communication (server to server). In the public VLAN, the IP address providing services moves between the servers. The private VLAN is required for intra-cluster communication; however, some implementations use the public VLAN for both public and private communication.
Figure 2 shows a basic high-availability cluster, with public and private networks linking the cluster nodes. The servers are also connected to a shared storage array (e.g., via SAN), which is a common scenario.
Figure 2:
High-availability cluster. [Source: Cisco Systems]
DID YOU KNOW?
The virtual IP address is the IP address that the requestor/client uses when communicating with the database server cluster. The servers have their own IP addresses, but based on the clustering implementation and/or load-sharing algorithm, they respond to requests sent to this virtual IP address. The clients use the virtual IP address to make requests to the server clusters – in the pictured case, the application servers use the virtual IP address of the server cluster to perform requests.
If you have more than one data center, and the clustered servers are located in both the primary and secondary data center, you need to establish Layer-2 visibility between data centers.
DID YOU KNOW?
Server clusters do not tolerate intelligent switches or routers, or any other equipment that would prevent the propagation of User Datagram Protocol (UDP) broadcasts; any optimization of Proxy Address Resolution Protocol (ARP) and MAC address caching, Internet Control Message Protocol (ICMP) optimizations, or transformation of broadcasts to unicasts. The segment must behave like a standard LAN, providing less than 500 ms. latency.
Primary and Backup, or Distributed Data Centers
When your organization experiences usage growth and more demanding business requirements, upgrading the data center may become necessary. Usually it makes sense to upgrade the primary and only data center to a limited extent, and to start investing in a secondary data center that will provide additional geographic resiliency in case of unpredicted natural conditions or technical trouble.
We will not consider how the primary and the secondary data centers accept client requests, or how load-balancing between resources is performed; those topics are outside the scope of this article, which is how to interconnect the two data centers. For information on geographical load-balancing, look into the Cisco Application Control Engine (ACE) and Global Site Selector (GSS) products.
The major problem in data center interconnection is how to extend direct Layer-2 connectivity to a secondary data center that is located at a distance beyond traditional local area connection limits.
In recent years, service providers have started to offer connectivity services to address such needs. The providers mainly employ tunneling mechanisms such as Layer-2 Tunneling Protocol Version 3 (L2TPv3), or MPLS-based connectivity solutions such as Ethernet over MPLS (EoMPLS) and Virtual Private LAN Service (VPLS). The best solution for the server cluster would be to use a Layer-3 (routed) connection to exchange the heartbeats, but often this is impossible due to server cluster design.
Inter-site Connectivity Using MPLS
Multi-protocol label switching has become the prevalent technology for high-speed packet switching in service provider networks, and MPLS VPN has been used since then to provide inter-site enterprise connectivity, offering high connection speed and sufficient privacy to enterprise clients.
Usually, MPLS VPN networks are Layer 3, meaning that there is a routed path between the sites of the enterprise customer. The Service Provider participates in customer routing and provides optimal path transport over the SP core.
As mentioned earlier, server clusters mostly require Layer-2 connectivity, so EoMPLS and VPLS technologies are used in such cases. EoMPLS is a point-to-point technology, whereas VPLS is multipoint – it can connect several sites by using one virtual LAN segment.
From the network perspective, two MPLS labels need to be used for such transport: a) one is the destination next-hop label, which identifies the exiting “provider edge” (PE) device within the SP network; and b) the other label is used for edge services – in the EoMPLS case, for virtual circuit identification, as shown in Figure 3. The EoMPLS ingress node adds two labels to any frame coming from a port or VLAN, and the packet uses MPLS for transport through the SP network. The benefit is that no Layer-2 loop-prevention technologies (such as the Spanning Tree Protocol) need to be extended from the customer to the SP metro network. From the client’s perspective, packets are bridged within sites; from the SP perspective, packets are label-switched through the SP core.
Figure 3:
View of an EoMPLS protocol data unit – top label for the eggress next-hop, and inner label for virtual circuit identification. [Source: NIL]
Several scenarios cover the point in the network at which the EoMPLS termination can be configured:
EoMPLS termination on the Metropolitan Area Network (MAN) access routers
EoMPLS termination on the WAN edge routers
EoMPLS termination at the DC aggregation layer
EoMPLS termination at the DC aggregation layer, using a loopback cable
We will outline the first option only, to give you a general idea of how it works; see the Cisco “Data Center High Availability Clusters Design Guide” for other options.
Figure 4 shows how the Layer-2 domain is extended through the DC aggregation switches to the MAN routers, where frames are encapsulated in MPLS packets (using the xconnect feature – see the “Cisco Catalyst 6500 Data Center Interconnect Solutions”) and then travel through the MAN/SP network.
Figure 4:
EoMPLS termination on the MAN routers. [Source: Cisco]
With this approach, the Layer-2 domain is extended from the aggregation layer and trunked to the core devices, which bridge the domain within two sites using EoMPLS.
Distributed Data Storage Systems
Each data center needs a data storage system to provide storage services for servers (disk images, etc.) and for business data. If you have two data centers in a high-availability configuration, it is necessary that data stored in one data center be stored in the other data center as well. Another possibility is used for backup purposes, when you have a tape library located only in one data center, and you need to convey the data to be stored from one DC to the other.
The Storage Area Networks use the Fibre Channel (FC) protocol. The Fibre Channel protocol can cover large distances natively, carried over SONET or DWDM networks. As a second option, FC frames can be encapsulated into IP packets by using the Fibre Channel over IP (FCIP) protocol. This way, the traffic between the two dispersed storage systems in different data centers can be treated as normal IP-routed traffic, and packets can flow over any regular routed network.
Due to SAN specifics, you may have another problem: quality of service (QoS). Generally, data storage traffic is very sensitive to any latency and data loss, so bandwidth and delay between DC sites should be provisioned and included in the Service Level Agreement (SLA) from the service provider. The packet maximum transmission unit (MTU) can also be an issue, as service providers need to support frames the size of a storage block written on disks. When looking for a solution for an inter-site data center link for storage replication, some involvement is needed from the SP side as well.
Data storage traffic also has some bandwidth-related implications, as large volumes of data are transferred over the SP network. Direct replication traffic comes in large bursts and suffers from delays, especially if synchronous replication is used. On the other hand, overnight backups require about medium bandwidth, and bandwidth usage is constant (i.e., not bursty).
DID YOU KNOW?
Synchronous replication of storage data is very sensitive to delay in the DC-to-DC interconnection. When the storage system in the primary DC issues a “data write,” it waits for confirmation that the data record has actually been written to storage in the secondary DC. In the meantime, the storage system is unable to respond to other requests, with applications suffering greatly. Latency should therefore be minimal, or “asynchronous write” should be used, along with some other form of consistency-checking.
In Figure 5, you can see how storage devices are positioned in the network to act as Fibre Channel to IP gateways, allowing the SAN traffic to be conveyed over an IP network between data centers.
Figure 5:
FCIP transport over an IP/MPLS network. [Source: Cisco]
The key piece of equipment is the FCIP gateway, which encapsulates FC frames into IP packets. FCIP can be implemented using multiservice modules for the Cisco MDS 9200 series switches and/or MDS 9500 series directors.
Point of interest: Encrypting FC traffic
Some clients face the requirement that their data must be encrypted at all times when going over the network. This rule particularly applies to state departments, ministries, etc. Though FC encryption devices exist, such solutions are often very expensive. Another solution would be to use FCIP and encrypt this IP traffic with IPSec, using hardware acceleration modules.
Transport of FCIP traffic is then treated as any other IP traffic, making it relatively simple to ensure QoS and proper transport path.
Resources
Cisco Data Center High Availability Clusters Design Guide
Cisco Data Center Infrastructure 2.5 Design Guide
Cisco Catalyst 6500 Data Center Interconnect Solutions
Keywords
Data Center Interconnection Designs
Layer 2 vs. Layer 3
L2 Technologies – L2TPv3, EoMPLS/VPLS
Storage Interconnection – FCIP
Intra-Cluster communication – Heartbeats
Public vs. Private Interface
Nessun commento:
Posta un commento