Fault Management - an overview (2023)

Performance

Fault management and performance management are the two domains that account for most of the daily activities of frontline staff and application specialists. While fault management looks at an application to determine whether it is “broken” (i.e., it has failed or has another serious problem), performance management is concerned with keeping an application running “well.” That is, performance management seeks to ensure that an application is operating within the parameters specified in the relevant service-level agreement (SLA) and operational-level agreement (OLA). This is largely a question of the speed at which the application does its work. An absolute requirement for application performance management (APM) is a set of tools to collect and analyze data about the application. Some of the collected data will be archived to allow a historical perspective to be taken when analyzing problems. SLAs are discussed in Appendix A, “Service-Level Management.”

Passive Approach

Passive fault management techniques typically depended on monitoring agents to detect and report network abnormality using alarms or symptom events. These events are then analyzed and correlated in order to reach the root faults. Various event correlation models were proposed including the rule-based analyzing system [10], model-based system [11], case-based diagnosing system, and model traversing techniques. Different techniques are also introduced to improve the performance, accuracy, and resilience of fault localization. In Appleby and Goldszmidt [5], a model-based event correlation engine is designed for multilayer fault diagnosis. In Kliger et al. [3], a coding approach is applied to the deterministic model to reduce the reasoning time and improve system resilience. A novel incremental event-driven fault reasoning technique is presented in Steinder and Sethi [1, 4] to improve the robustness of the fault localization system by analyzing lost, positive, and spurious symptoms.

The above techniques were developed based on passively received symptoms. If the evidence (symptoms) is collected correctly, the fault reasoning results can be accurate. However, in real systems, symptom loss or spurious symptoms (observation noise) are unavoidable. Even with a good strategy [1] to deal with observation noise, such techniques have limited resilience to noise because of their underlying passive approach, which might also increase the fault detection time.

Fault Management

Within the ISO framework, fault management is responsible for detecting, correlating, and providing the necessary interfaces for correcting a failure within the managed devices. A failure may be defined as an event within the network causing the system to operate outside its normal operating conditions. The failure may be defined as transient or persistent, requiring the management system to have the capacity to detect either condition under all operating environments.

Upon detection and correction of a failure condition it is critical the management system is capable of recording all events surrounding the event to permanent record. Once the system has been restored to normal operation, each failure condition should be evaluated in detail to make sure all events leading up to the failure are well understood. Any possible corrective action to prevent the conditions from occurring again should be put into place.

Events and alarms are typically displayed within the management system as a sorted table listing each of the conditions the system has detected.

The key attributes of an alarm or event include the ID of the condition, the severity of the problem as defined by the operator (critical/major/minor), the source of the condition including the device name and type, the time the condition was received by the management system and finally the number of times this condition has been reported by the network element.

In today’s environment of a distributed management system it is important for a network operator to ACKnowledge each alarm they are working to resolve. This tells the operator’s staff that a colleague has already begun taking action on the condition received by the system. Enforcing this discipline is critical in order to avoid one of two undesirable outcomes: either two people start working on the same problem and work at cross-purpose, or everyone assumes someone else has it, so no one works on it.

Alarm and event notification is critical to a fault management system in order to facilitate automatic reporting to staff. Notifications are typically set up for particular alarm/event, severity level, frequency, and device types through email, SMS text message, voice message, and system alarm. These features are critical to allow for indications of various conditions and escalated problems within a managed system to be easily communicated to the staff.

Integration with Trouble Ticketing Systems

We recommend that you only allow your fault management software to create trouble tickets if the alerts triggering the tickets are so application specific and free from false positives (like a passive check) that you can ensure the event requires attention by a real person. On the other hand, we highly recommend you take advantage of help desk systems that allow you to associate fault manager event IDs and information with your trouble tickets. For example, it would make sense to have your help desk software have the capability to acknowledge an open alert in Nagios, or to have the help desk software clear an alert as soon as the issue is resolved by the help desk person working on the event in question.

8.5.8 Client Layers

We will describe some of the performance and fault management features in the client layer protocols described in Chapter 6. The performance and fault management mechanisms of the SONET/SDH and the electronic layer of OTN have already been discussed. Since SONET/SDH and OTN provide constant bit rate service, they use bit error rate (BER) as a performance measure as well as loss of signal. Network elements are informed of error and fault events through defect indicators (see Subsection 8.5.4). They also have trace information in their overhead.

Protocols that provide packet transport services such as Ethernet or MPLS have performance measures that are packet oriented, such as packet loss rate, packet delay, and packet delay variation (jitter). To detect if a connection (link or path) is up, “hello” or continuity check messages are sent periodically through the connection between the end nodes. If these messages are not received, then it is assumed that the connection is down. Remote defect indicators and AIS signals are used by one end of a link to inform the other end that it has detected a failure or error. Management occurs at different levels. At the lowest level, individual links are managed, while at the highest level end-to-end connections are managed. In the middle level, segments of an end-to-end connection can be managed such as when a segment goes through another network operator. In addition, end-to-end management can be customer oriented or service provider oriented.

What Fault-Management Problems Do Users Have and How Complex Are They?

Positively, the team limits the scope of its tool to solving just these problems. Findings from contextual interviews show that technical specialists also manage and prevent faults for reliability purposes. For example, they tune applications and other components (performed to improve system components and software code), and they determine health-of-the-system metrics (conducted to identify baseline measures for alarm thresholds). But these situations are handled by a different group of specialists, not those in charge of assuring functionality. Because reliability specialists are not the VizAppManager team's targeted users, design team members decide that support for tuning, optimizing, and other reliability activities are outside the scope of the program.

The traits the team identifies are common across situations and work sites. Degrees of complexity, however, vary from one troubleshooting situation to the next. Sometimes, IT analysts encounter common or familiar problems, and through experience, they have assimilated standard methods and fixes for investigating and repairing them.

In other cases, however, entangled infrastructure conditions or unapparent chain reactions confound analysts. Problems elude a clear definition and are difficult to trace to a source. One example is an intermittent slowdown with no obvious cause, and another is Benkei's entangled faults. The VizAppManager team realizes that its single application-monitoring tool must support all these degrees of complexity.

From contextual inquiry findings, the team estimates that roughly 80% of tier-two and tier-three troubleshooting involves common problems. The other 20% are abnormal, often intermittent, and may take days or weeks to solve. As we see later, because of this distribution, the team devotes most of its efforts to creating support for the 80%. Unfortunately, team members fail to realize that a good deal of complexity still resides in solving these familiar problems, and this complexity is shared by the other 20%, as well.

4.3.3 Basic Network Management Concepts

When studying the survivability of communication networks, it is also useful to look at the general framework in which traffic management is implemented. Network management has become an indispensable tool of communication networks since it is responsible for ensuring continuous and secure functioning of a network. Generally, network management is divided into several functional areas: performance management, configuration management, and fault management. These key areas are implemented by control modules that operate in an integrated way to manage the network, including functions that support traffic management and restoration survivability techniques.

To implement the highest priority task, that is, restore services, fault management uses the functionality provided by configuration management and performance management. Upon detection of a network failure by the fault management system or detection of QoS degradation by the performance management system, the failed traffic connections are identified by the configuration management system, new paths are searched for if needed, the best path i.e., the one with the lowest cost, is selected for each failed connection, and the traffic is rerouted by establishing new connections along the selected paths. In parallel with the service restoration process, a repair process should begin with the network manager performing root cause identification followed by the detailed repair or replacement of the failed components. Once the failed components have been repaired, they can be put back in service and a normalization or reversion process might occur that consists of moving the traffic from their current routes to their original prefailure routes. Furthermore, all incidents in the process are monitored and reported for billing and management purposes.

II.A.2. Fault Management

This function is required to detect abnormal network behavior. Fault management follows a sequence of actions: error detection, error diagnosis, and error recovery.

Error detection monitors events such as alarm signals from network devices (when thresholds are exceeded or in the event of hardware failure), deterioration of performance, or application failures. Error detection facilities also include an error log for future analysis.

Error diagnosis involves the analysis of detected errors in an effort to determine the cause of an error and a course of action to rectify it. Recent approaches to error diagnosis include the use of artificial intelligence techniques such as deductive reasoning.

Error recovery involves a range of measures proportional to the error's magnitude. Simple errors may require the fine-tuning of a device on the network, where more serious errors may mandate the replacement of a faulty device. Persistent performance failures are usually an indicator of poor network health. Remedying such problems typically involves reconfiguration of the problematic section of the network.

1.2.3 Element management system (EMS)

The EM may be aware of virtualization and can collaborate with VNFM to perform those functions that require exchange of information regarding the NFVI resources associated with VNF. It is not a requirement to have a 1:1 map between VNF and EMS and one single EMS may manage many VNFs.

6.5 MANAGEMENT AND CONTROL

A very important feature of ATM networks is that they can make a number of management and control decisions to discriminate among connections and to provide the variety of QoS that different applications need. The decisions are divided into three groups. When a request is made for a connection with a particular QoS, the network must determine whether to accept or reject the request, depending on the resources then available. (Recall that QoS involves three sets of parameters: delay, cell loss, and source traffic rate.) If the resources are insufficient to meet the request, the network may negotiate with the user the traffic parameters in the requested service class.

Once the connection is admitted, the network must assign a route or path to the virtual channel that carries the connection. It must inform the switches and other network elements along the path that this virtual channel must be allocated certain resources so that the agreed-on QoS is met.

Lastly, the network must monitor the data transfer to make sure that the source also conforms to the QoS specification and to drop its cells as appropriate. (This is called traffic policing.) The network may also ask a source to slow down its transmissions.

In addition, the network carries a number of information flows to monitor its operations and to detect and identify the location of congested or failed devices.

The BISDN standard so far is silent about how these decisions are to be carried out. (However, recent ITU recommendations deal with OAM, performance monitoring and protection switching at the ATM layer to complement SONET protection switching.) We shall discuss potential solutions in Chapter 8. The ATM Forum specifies frame formats that the network should use to carry its monitoring information and to interact with users. We review these next.

The layers in the control plane are the functions needed to set up, supervise, and release a virtual circuit connection. These functions, implemented by signaling protocols such as PNNI, are needed only for switched virtual connections and are absent in a network that implements only permanent virtual connections. (In a permanent virtual circuit connection, the path or route assigned to a source and destination and the VCI for that route are fixed.)

The layer management plane contains management functions specific to individual layers. Layer mangement also handles the operations and maintenance flows specific to each layer. The protocols used for these functions include ILMI and SNMP.

Finally, plane management consists of the functions that supervise the operations of the whole network. Plane management has no layered structure.