[OptiX OSN 8800] RMON_ALM_INBADOCTS_OVER alarm in a TOA board Highlighted

Hi guys!!

I would like to share with you an issue about RMON_ALM_INBADOCTS_OVER (service quality alarm).

Description

Statistics of receiving bad packets is over the upper threshold. The alarm is generated when the octet number of bad packets detected by the Ethernet port of the board crosses the upper threshold.

Condition:

The scenario is as follows:

GE service is configured between SITE 1 and SITE 2:

• SITE 1 (OSN8800 T32) / TOA / Slot 5 / TX4-RX4

• SITE 2 (OSN8800 T32) / TOA / Slot 3 / TX6-RX6

The main parameters configured in the GE service in both TOA boards are:

Service Type -> GE (TTT-GMP)

Ethernet Working Mode -> Auto-negotiation (1000M Full-Duplex)

MTU -> 9600bytes (by default)

The other parameters are by default.

The Client Data Equipment main parameters are configurated as follows:

Ethernet Working Mode -> Auto-negotiation (1000M Full-Duplex)

MTU -> 1500bytes

Symptom:

Customer detects CRC error packets in the GE service. After the warning, the RMON_ALM_INBADOCTS_OVER alarm is verified in the NMS:

Transmit and receive directions of RMON performance events are checked (in both sites).

The abnormal data packets include:

• ETHOVER (Oversize Packets Received), excessively long packet whose length exceeds the configured maximum length.

• ETHFCS (FCS errors), CRC error packets.

• TXBBAD (Bad Octets Transmitted), the total number of octets of bad packets transmitted on the network (excluding framing bits but including FCS octets).  

On the other hand, the Ethernet Performance results on both client ports (SITE 1 and SITE 2) of the TOA boards are as follows:

It's obvious that the client port RX4/TX4 of the TOA board in SITE 1 receiving abnormal data packets. This abnormal data packets are transmitted by the GE service trail over DWDM network to SITE 2.

Impact:

The GE service quality is degraded.

Severity:

Minor

Root Cause:

Possible Causes:

• 1) The data equipment on the client side sends the abnormal data packets.

• 2) The abnormal link between the client side and the Ethernet port on the board leads to the error of the data    packets in transmission.

• 3) The faulty board leads to the regeneration of the signals or detection error.

Solution:

Firstly the transmit/receive optical power of the two connected GE ports client (TOA board) of both sites are cheked, to clarify if are within the permitted range. The result is OK:

A support technician goes to one of the customer's headquarters (SITE 1 side) with Ethernet Network Tester (MTS 5800). An RFC 2544 Enhanced measure is performed with physical loop in other end customer's headquarte (SITE 2 side).

The result of RFC 2544 is OK:

With the previous RFC 2544 Enhanced measure, the following possible causes are discard:

• 2) The abnormal link between the client side and the Ethernet port on the board leads to the error of the data    packets in transmission.

• 3) The faulty board leads to the regeneration of the signals or detection error.

In addition, jumbo frame traffic is injected with 9600bytes extended load frame length and 9601bytes extended load frame length.

-> 9600bytes extended load frame length: The service does not displays the RMON_ALM_INBADOCTS_OVER alarm.

-> 9601bytes extended load frame length: The service displays the RMON_ALM_INBADOCTS_OVER alarm.

With the previous tests it is verified that the TOA board reports the RMON_ALM_INBADOCTS_OVER correctly.

The results certify the correct functioning of the link at the DWDM transport layer level. The traffic generated by the measurement equipment to carry out this test, as can be seen in the previous attached image, transmitted with different types of frames with different sizes, including 9600 bytes Jumbo Frame.

Conclusion: In this way, everything indicates that the problem must be in the Client Data Equipment - SITE 1 and that their configuration should be reviewed (HW, SW, configuration parameters...):

• 1) The data equipment on the client side sends the abnormal data packets.

After the customer checks the Client Data Equipment, switch port failure is reported.

BR