HPE FlexFabric 5940 & 5930 Switch Series
Network Management and Monitoring
Configuration Guide
P
art number: 5200-4878c
Software
version: Release 2609 and later
Document version: 6W103-20200310
© Copyright 2020 Hewlett Packard Enterprise Development LP
The information contained herein is subject to change without notice. The only warranties for Hewlett Packard
Enterprise products and services are set forth in the express warranty statements accompanying such
products and services. Nothing herein should be construed as constituting an additional warranty. Hewlett
Packard Enterprise shall not be liable for technical or editorial errors or omissions contained herein.
Confidential computer software. Valid license from Hewlett Packard Enterprise required for possession, use, or
copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software
Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s
standard commercial license.
Links to third-party websites take you outside the Hewlett Packard Enterprise website. Hewlett Packard
Enterprise has no control over and is not responsible for information outside the Hewlett Packard Enterprise
website.
Acknowledgments
Intel®, Itanium®, Pentium®, Intel Inside®, and the Intel Inside logo are trademarks of Intel Corporation in the
United States and other countries.
Microsoft® and Windows® are either registered trademarks or trademarks of Microsoft Corporation in the
United States and/or other countries.
Adobe® and Acrobat® are trademarks of Adobe Systems Incorporated.
Java and Oracle are registered trademarks of Oracle and/or its affiliates.
UNIX® is a registered trademark of The Open Group.
i
Contents
Using ping, tracert, and system debugging ···················································· 1
Ping ···································································································································································· 1
Using a ping command to test network connectivity ·················································································· 1
Ping example ············································································································································· 1
Tracert ································································································································································ 3
Prerequisites ·············································································································································· 4
Using a tracert command to identify failed or all nodes in a path ······························································· 4
Tracert example ········································································································································· 4
System debugging ············································································································································· 6
Debugging information control switches ···································································································· 6
Debugging a feature module ······················································································································ 6
Configuring NQA ··························································································· 8
Overview ···························································································································································· 8
NQA operation ··········································································································································· 8
Collaboration with Track ····························································································································· 9
Threshold monitoring ······························································································································· 10
NQA configuration task list ······························································································································· 10
Configuring the NQA server ····························································································································· 10
Enabling the NQA client ··································································································································· 11
Configuring NQA operations on the NQA client ······························································································· 11
NQA operation configuration task list ······································································································· 11
Configuring the ICMP echo operation ······································································································ 12
Configuring the ICMP jitter operation ······································································································· 13
Configuring the DHCP operation·············································································································· 14
Configuring the DNS operation ················································································································ 14
Configuring the FTP operation ················································································································· 15
Configuring the HTTP operation ·············································································································· 16
Configuring the UDP jitter operation ········································································································ 17
Configuring the SNMP operation ············································································································· 18
Configuring the TCP operation ················································································································· 19
Configuring the UDP echo operation ······································································································· 20
Configuring the UDP tracert operation ····································································································· 20
Configuring the voice operation ··············································································································· 22
Configuring the DLSw operation ·············································································································· 24
Configuring the path jitter operation ········································································································· 24
Configuring optional parameters for the NQA operation ·········································································· 25
Configuring the collaboration feature ······································································································· 26
Configuring threshold monitoring ············································································································· 27
Configuring the NQA statistics collection feature ····················································································· 29
Configuring the saving of NQA history records ························································································ 30
Scheduling the NQA operation on the NQA client ··················································································· 31
Configuring NQA templates on the NQA client ································································································ 31
NQA template configuration task list ········································································································ 31
Configuring the ICMP template ················································································································ 32
Configuring the DNS template ················································································································· 33
Configuring the TCP template ·················································································································· 33
Configuring the TCP half open template ·································································································· 34
Configuring the UDP template ················································································································· 35
Configuring the HTTP template ················································································································ 36
Configuring the HTTPS template ············································································································· 37
Configuring the FTP template ·················································································································· 39
Configuring the RADIUS template ··········································································································· 40
Configuring the SSL template ·················································································································· 41
Configuring optional parameters for the NQA template ··········································································· 41
Displaying and maintaining NQA ····················································································································· 42
NQA configuration examples ··························································································································· 42
ii
ICMP echo operation configuration example ··························································································· 42
ICMP jitter operation configuration example ···························································································· 44
DHCP operation configuration example ··································································································· 46
DNS operation configuration example ····································································································· 47
FTP operation configuration example ······································································································ 49
HTTP operation configuration example ···································································································· 50
UDP jitter operation configuration example······························································································ 51
SNMP operation configuration example ··································································································· 53
TCP operation configuration example ······································································································ 55
UDP echo operation configuration example····························································································· 56
UDP tracert operation configuration example ·························································································· 57
Voice operation configuration example ···································································································· 59
DLSw operation configuration example ··································································································· 61
Path jitter operation configuration example ······························································································ 62
NQA collaboration configuration example ································································································ 64
ICMP template configuration example ····································································································· 66
DNS template configuration example ······································································································· 67
TCP template configuration example ······································································································· 68
TCP half open template configuration example ······················································································· 69
UDP template configuration example ······································································································· 69
HTTP template configuration example ····································································································· 70
HTTPS template configuration example ·································································································· 71
FTP template configuration example ······································································································· 71
RADIUS template configuration example ································································································· 72
SSL template configuration example ······································································································· 73
Configuring NTP ·························································································· 75
Overview ·························································································································································· 75
How NTP works ······································································································································· 75
NTP architecture ······································································································································ 76
Association modes ··································································································································· 77
NTP security ············································································································································· 78
NTP for MPLS L3VPN instances ············································································································· 79
Protocols and standards ·························································································································· 80
Configuration restrictions and guidelines ········································································································· 80
Configuration task list ······································································································································· 80
Enabling the NTP service································································································································· 81
Configuring NTP association mode·················································································································· 81
Configuring NTP in client/server mode ···································································································· 81
Configuring NTP in symmetric active/passive mode ················································································ 82
Configuring NTP in broadcast mode ········································································································ 82
Configuring NTP in multicast mode ·········································································································· 83
Configuring access control rights ····················································································································· 84
Configuring NTP authentication ······················································································································· 84
Configuring NTP authentication in client/server mode ············································································· 84
Configuring NTP authentication in symmetric active/passive mode ························································ 86
Configuring NTP authentication in broadcast mode ················································································· 87
Configuring NTP authentication in multicast mode ·················································································· 88
Configuring NTP optional parameters ·············································································································· 90
Specifying the source interface for NTP messages ················································································· 90
Disabling an interface from receiving NTP messages ············································································· 91
Configuring the maximum number of dynamic associations ···································································· 91
Setting a DSCP value for NTP packets···································································································· 92
Configuring the local clock as a reference source ··························································································· 92
Displaying and maintaining NTP ······················································································································ 92
NTP configuration examples ···························································································································· 93
NTP client/server mode configuration example ······················································································· 93
IPv6 NTP client/server mode configuration example ··············································································· 94
NTP symmetric active/passive mode configuration example ··································································· 95
IPv6 NTP symmetric active/passive mode configuration example ··························································· 96
NTP broadcast mode configuration example ··························································································· 98
NTP multicast mode configuration example ····························································································· 99
iii
IPv6 NTP multicast mode configuration example ·················································································· 102
Configuration example for NTP client/server mode with authentication················································· 105
Configuration example for NTP broadcast mode with authentication ···················································· 107
Configuration example for MPLS L3VPN network time synchronization in client/server mode ············· 109
Configuration example for MPLS L3VPN network time synchronization in symmetric active/passive mode
······························································································································································· 111
Configuring SNTP ······················································································ 114
Configuration restrictions and guidelines ······································································································· 114
Configuration task list ····································································································································· 114
Enabling the SNTP service ···························································································································· 114
Specifying an NTP server for the device ········································································································ 114
Configuring SNTP authentication ··················································································································· 115
Displaying and maintaining SNTP·················································································································· 116
SNTP configuration example ························································································································· 116
Network requirements ···························································································································· 116
Configuration procedure ························································································································· 116
Configuring PTP ························································································ 118
Overview ························································································································································ 118
Basic concepts ······································································································································· 118
Synchronization mechanism ·················································································································· 120
Protocols and standards ························································································································ 122
Compatibility information ································································································································ 123
Feature and hardware compatibility ······································································································· 123
Feature and software version compatibility ···························································································· 123
Configuring clock nodes ································································································································· 123
Configuration task list ····························································································································· 123
Specifying a PTP standard ····················································································································· 124
Specifying a clock node type ·················································································································· 124
Specifying a PTP domain ······················································································································· 125
Configuring an OC to operate only as a member clock ········································································· 125
Configuring a priority for a clock ············································································································· 125
Configuring the role of a PTP port ·········································································································· 125
Specifying a delay measurement mechanism for a BC or an OC ·························································· 126
Configuring the port type for a TC+OC ·································································································· 126
Setting the interval for sending announce messages and the interval timeout ······································ 127
Setting the interval for sending Pdelay_Req messages ········································································· 127
Setting the interval for sending Sync messages ···················································································· 128
Setting the minimum interval for sending Delay_Req messages ··························································· 128
Configuring the MAC address for non-pdelay messages······································································· 128
Specifying the protocol for encapsulating PTP messages as UDP (IPv4) ············································· 129
Configuring the source IP address for multicast PTP message transmission over UDP (IPv4) ············ 129
Configuring the destination IP address for unicast PTP message transmission over UDP (IPv4)········· 129
Setting the delay correction value ·········································································································· 130
Setting the cumulative offset between the UTC and TAI ······································································· 130
Setting the correction date of the UTC ··································································································· 130
Setting a DSCP value for PTP messages transmitted over UDP (IPv4) ················································ 131
Specifying a VLAN tag for PTP messages ····························································································· 131
Specifying the system time protocol as PTP ·························································································· 131
Enabling PTP on a port ·························································································································· 132
Displaying and maintaining PTP ···················································································································· 132
PTP configuration examples ·························································································································· 132
PTP configuration example (IEEE 1588 version 2, IEEE 802.3/Ethernet encapsulation) ······················ 132
PTP configuration example (IEEE 1588 version 2, multicast transmission) ·········································· 135
PTP configuration example (IEEE 802.1AS)·························································································· 137
Configuring SNMP ····················································································· 141
Overview ························································································································································ 141
SNMP framework ··································································································································· 141
MIB and view-based MIB access control ······························································································· 141
SNMP operations ··································································································································· 142
iv
Protocol versions ···································································································································· 142
Access control modes ···························································································································· 142
SNMP silence ········································································································································· 142
FIPS compliance ············································································································································ 143
Configuring SNMP basic parameters ············································································································· 143
Configuring SNMPv1 or SNMPv2c basic parameters ············································································ 143
Configuring SNMPv3 basic parameters ································································································· 145
Configuring SNMP logging ····························································································································· 148
Configuring SNMP notifications ····················································································································· 149
Enabling SNMP notifications ·················································································································· 149
Configuring parameters for sending SNMP notifications ······································································· 150
Displaying the SNMP settings ························································································································ 151
SNMPv1/SNMPv2c configuration example ···································································································· 152
Network requirements ···························································································································· 152
Configuration procedure ························································································································· 152
Verifying the configuration ······················································································································ 153
SNMPv3 configuration example ····················································································································· 153
Network requirements ···························································································································· 153
Configuration procedure ························································································································· 153
Verifying the configuration ······················································································································ 155
Configuring RMON ···················································································· 156
Overview ························································································································································ 156
RMON groups ········································································································································ 156
Sample types for the alarm group and the private alarm group ····························································· 158
Protocols and standards ························································································································ 158
Configuring the RMON statistics function ······································································································ 158
Creating an RMON Ethernet statistics entry ·························································································· 158
Creating an RMON history control entry ································································································ 158
Configuring the RMON alarm function ··········································································································· 159
Displaying and maintaining RMON settings ··································································································· 160
RMON configuration examples ······················································································································ 160
Ethernet statistics group configuration example ···················································································· 160
History group configuration example ······································································································ 161
Alarm function configuration example ···································································································· 162
Configuring the Event MIB ········································································· 164
Overview ························································································································································ 164
Monitored objects ··································································································································· 164
Object owner ·········································································································································· 164
Trigger test ············································································································································· 164
Event actions ·········································································································································· 165
Prerequisites ·················································································································································· 166
Event MIB configuration task list ···················································································································· 166
Configuring Event MIB sampling ···················································································································· 166
Configuring Event MIB object lists ················································································································· 166
Configuring an event ······································································································································ 167
Creating an event ··································································································································· 167
Configuring a set action for an event ····································································································· 167
Configuring a notification action for an event ························································································· 168
Configuring a trigger······································································································································· 168
Configuring a Boolean trigger test·········································································································· 169
Configuring an existence trigger test ······································································································ 170
Configuring a threshold trigger test ········································································································ 170
Enabling SNMP notifications for Event MIB ··································································································· 171
Displaying and maintaining the Event MIB····································································································· 172
Event MIB configuration examples················································································································· 172
Existence trigger test configuration example ························································································· 172
Boolean trigger test configuration example ···························································································· 174
Threshold trigger test configuration example ························································································· 177
v
Configuring NETCONF ·············································································· 180
Overview ························································································································································ 180
NETCONF structure ······························································································································· 180
NETCONF message format ··················································································································· 181
How to use NETCONF ··························································································································· 182
Protocols and standards ························································································································ 182
FIPS compliance ············································································································································ 182
NETCONF configuration task list ··················································································································· 183
Configuring NETCONF over SOAP ··············································································································· 183
Enabling NETCONF over SSH ······················································································································ 184
Enabling NETCONF logging ·························································································································· 184
Configuring NETCONF to use module-specific namespaces ········································································ 185
Overview ················································································································································ 185
Configuration restrictions and guidelines ······························································································· 186
Configuration procedure ························································································································· 186
Establishing a NETCONF session ················································································································· 186
Setting the NETCONF session idle timeout time ··················································································· 186
Entering XML view ································································································································· 186
Exchanging capabilities ·························································································································· 187
Subscribing to event notifications··················································································································· 187
Overview ················································································································································ 187
Subscribing to syslog events ·················································································································· 187
Subscribing to events monitored by NETCONF····················································································· 188
Subscribing to events reported by modules ··························································································· 190
Example for subscribing to event notifications ······················································································· 191
Locking/unlocking the configuration ··············································································································· 192
Locking the configuration ······················································································································· 192
Unlocking the configuration ···················································································································· 192
Example for locking the configuration ···································································································· 193
Performing service operations ······················································································································· 194
Performing the <get>/<get-bulk> operation ··························································································· 194
Performing the <get-config>/<get-bulk-config> operation ······································································ 196
Performing the <edit-config> operation ·································································································· 196
All-module configuration data retrieval example ···················································································· 197
Syslog configuration data retrieval example ·························································································· 199
Example for retrieving a data entry for the interface table ····································································· 200
Example for changing the value of a parameter ···················································································· 201
Saving, rolling back, and loading the configuration ························································································ 202
Saving the configuration ························································································································· 202
Rolling back the configuration based on a configuration file ·································································· 203
Rolling back the configuration based on a rollback point ······································································· 203
Loading the configuration ······················································································································· 207
Example for saving the configuration ····································································································· 208
Enabling preprovisioning ································································································································ 209
Filtering data ·················································································································································· 209
Table-based filtering ······························································································································· 210
Column-based filtering ··························································································································· 210
Example for filtering data with regular expression match ······································································· 213
Example for filtering data by conditional match ······················································································ 214
Performing CLI operations through NETCONF······························································································ 215
Configuration procedure ························································································································· 215
CLI operation example ··························································································································· 216
Retrieving NETCONF information ·················································································································· 217
Retrieving YANG file content ························································································································· 217
Retrieving NETCONF session information····································································································· 218
Terminating another NETCONF session ······································································································· 219
Configuration procedure ························································································································· 219
Configuration example ··························································································································· 220
Returning to the CLI ······································································································································· 220
vi
Appendix ··································································································· 221
Appendix A Supported NETCONF operations ······························································································· 221
Configuring CWMP ···················································································· 230
Overview ························································································································································ 230
CWMP network framework ···················································································································· 230
Basic CWMP functions ··························································································································· 230
How CWMP works ································································································································· 231
Configuration task list ····································································································································· 233
Enabling CWMP from the CLI ························································································································ 234
Configuring ACS attributes····························································································································· 234
Configuring the preferred ACS attributes ······························································································· 234
Configuring the default ACS attributes from the CLI ·············································································· 235
Configuring CPE attributes····························································································································· 236
Configuring ACS authentication parameters ·························································································· 236
Configuring the provision code ··············································································································· 236
Configuring the CWMP connection interface ························································································· 237
Configuring autoconnect parameters ····································································································· 237
Enabling NAT traversal for the CPE ······································································································· 238
Specifying an SSL client policy for HTTPS connection to ACS ····························································· 239
Displaying and maintaining CWMP ················································································································ 239
CWMP configuration example························································································································ 239
Network requirements ···························································································································· 239
Configuration procedure ························································································································· 240
Verifying the configuration ······················································································································ 249
Configuring EAA ························································································ 250
Overview ························································································································································ 250
EAA framework ······································································································································ 250
Elements in a monitor policy ·················································································································· 251
EAA environment variables ···················································································································· 252
Configuring a user-defined EAA environment variable ·················································································· 253
Configuring a monitor policy··························································································································· 254
Configuration restrictions and guidelines ······························································································· 254
Configuring a monitor policy from the CLI ······························································································ 254
Configuring a monitor policy by using Tcl ······························································································ 256
Suspending monitor policies ·························································································································· 257
Displaying and maintaining EAA settings······································································································· 257
EAA configuration examples ·························································································································· 257
CLI event monitor policy configuration example ···················································································· 257
Track event monitor policy configuration example ················································································· 259
CLI-defined policy with EAA environment variables configuration example ·········································· 260
Tcl-defined policy configuration example ······························································································· 262
Monitoring and maintaining processes ······················································· 264
Displaying and maintaining processes ··········································································································· 264
Displaying and maintaining user processes ··································································································· 264
Starting or stopping a third-party process ······································································································ 265
Starting a third-party process ················································································································· 265
Stopping a third-party process ··············································································································· 265
Monitoring kernel threads······························································································································· 266
Configuring kernel thread deadloop detection ······················································································· 266
Configuring kernel thread starvation detection······················································································· 266
Displaying and maintaining kernel threads ···························································································· 267
Configuring port mirroring ·········································································· 268
Overview ························································································································································ 268
Terminology ··········································································································································· 268
Port mirroring classification and implementation ···················································································· 269
Configuring local port mirroring ······················································································································ 272
Local port mirroring configuration task list······························································································ 272
vii
Creating a local mirroring group ············································································································· 272
Configuring source ports for the local mirroring group ··········································································· 272
Configuring source CPUs for the local mirroring group·········································································· 273
Configuring the monitor port for the local mirroring group ······································································ 273
Configuring Layer 2 remote port mirroring ····································································································· 274
Layer 2 remote port mirroring with reflector port configuration task list ················································· 275
Layer 2 remote port mirroring with egress port configuration task list ···················································· 275
Configuring a remote destination group on the destination device ························································ 276
Configuring a remote source group on the source device ····································································· 277
Configuring local mirroring group with multiple monitor ports ········································································ 280
Configuration restrictions and guidelines ······························································································· 280
Configuration procedure ························································································································· 281
Configuring Layer 3 remote port mirroring (in tunnel mode) ·········································································· 281
Layer 3 remote port mirroring configuration task list ·············································································· 282
Configuration restrictions and guidelines ······························································································· 282
Configuration prerequisites ···················································································································· 282
Creating local mirroring groups ·············································································································· 282
Configuring source ports for a local mirroring group ·············································································· 283
Configuring source CPUs for a local mirroring group············································································· 283
Configuring the monitor port for a local mirroring group ········································································· 284
Configuring Layer 3 remote port mirroring (in ERSPAN mode) ····································································· 285
Layer 3 remote port mirroring configuration task list ·············································································· 285
Configuration restrictions and guidelines ······························································································· 285
Creating a local mirroring group on the source device··········································································· 285
Configuring source ports for the local mirroring group ··········································································· 285
Configuring source CPUs for the local mirroring group·········································································· 286
Configuring the monitor port for the local mirroring group ······································································ 286
Displaying and maintaining port mirroring ······································································································ 287
Port mirroring configuration examples ··········································································································· 288
Local port mirroring configuration example (in source port mode) ························································· 288
Local port mirroring configuration example (in source CPU mode) ······················································· 289
Layer 2 remote port mirroring configuration example (reflector port) ····················································· 290
Layer 2 remote port mirroring configuration example (with egress port) ················································ 292
Local port mirroring with multiple monitor ports configuration example ················································· 294
Layer 3 remote port mirroring configuration example (in tunnel mode) ················································· 295
Layer 3 remote port mirroring configuration example (in ERSPAN mode) ············································ 298
Configuring flow mirroring ·········································································· 300
Flow mirroring configuration task list ·············································································································· 300
Configuring match criteria ······························································································································ 300
Configuring a traffic behavior ························································································································· 301
Configuring a QoS policy ······························································································································· 301
Applying a QoS policy ···································································································································· 302
Applying a QoS policy to an interface ···································································································· 302
Applying a QoS policy to a VLAN ··········································································································· 302
Applying a QoS policy globally ··············································································································· 302
Applying a QoS policy to the control plane ···························································································· 303
Flow mirroring configuration example ············································································································ 303
Network requirements ···························································································································· 303
Configuration procedure ························································································································· 304
Verifying the configuration ······················································································································ 304
Configuring sFlow ······················································································ 305
Protocols and standards ································································································································ 305
sFlow configuration task list ··························································································································· 305
Configuring the sFlow agent and sFlow collector information ········································································ 306
Configuring flow sampling ······························································································································ 306
Configuring counter sampling ························································································································ 307
Displaying and maintaining sFlow ·················································································································· 307
sFlow configuration example·························································································································· 307
Network requirements ···························································································································· 307
Configuration procedure ························································································································· 308
viii
Verifying the configurations ···················································································································· 308
Troubleshooting sFlow configuration ············································································································· 309
The remote sFlow collector cannot receive sFlow packets ···································································· 309
Configuring the information center ····························································· 310
Overview ························································································································································ 310
Log types ················································································································································ 310
Log levels ··············································································································································· 310
Log destinations ····································································································································· 311
Default output rules for logs ··················································································································· 311
Default output rules for diagnostic logs ·································································································· 311
Default output rules for security logs ······································································································ 311
Default output rules for hidden logs ······································································································· 312
Default output rules for trace logs ·········································································································· 312
Log formats ············································································································································ 312
FIPS compliance ············································································································································ 314
Information center configuration task list ······································································································· 315
Outputting logs to the console ······················································································································· 315
Outputting logs to the monitor terminal ·········································································································· 316
Outputting logs to log hosts···························································································································· 316
Outputting logs to the log buffer ····················································································································· 317
Saving logs to the log file ······························································································································· 317
Managing security logs ·································································································································· 318
Saving security logs to the security log file ···························································································· 318
Managing the security log file ················································································································· 319
Saving diagnostic logs to the diagnostic log file ····························································································· 319
Configuring the maximum size of the trace log file ························································································ 320
Setting the minimum storage period for logs ································································································· 320
Enabling synchronous information output ······································································································ 321
Enabling duplicate log suppression ··············································································································· 321
Configuring log suppression for a module······································································································ 321
Disabling an interface from generating link up or link down logs ··································································· 322
Enabling SNMP notifications for system logs································································································· 322
Displaying and maintaining information center ······························································································ 322
Information center configuration examples ···································································································· 323
Configuration example for outputting logs to the console ······································································ 323
Configuration example for outputting logs to a UNIX log host ······························································· 324
Configuration example for outputting logs to a Linux log host ······························································· 325
Configuring the packet capture ·································································· 327
Overview ························································································································································ 327
Packet capture modes ··························································································································· 327
Filter elements ········································································································································ 327
Building a capture filter ··························································································································· 333
Building a display filter ··························································································································· 334
Packet capture configuration task list············································································································· 334
Configuring local packet capture ···················································································································· 335
Configuring remote packet capture ················································································································ 335
Configuring feature image-based packet capture ·························································································· 336
Saving captured packets to a file ··········································································································· 336
Filtering packet data to display··············································································································· 337
Displaying the contents in a packet file ·········································································································· 337
Displaying and maintaining packet capture···································································································· 337
Packet capture configuration examples ········································································································· 337
Remote packet capture configuration example ······················································································ 337
Feature image-based packet capture configuration example ································································ 338
Configuring VCF fabric ·············································································· 342
Overview ························································································································································ 342
VCF fabric topology ································································································································ 342
Neutron concepts and components ······································································································· 343
Neutron deployment ······························································································································· 344
ix
Automated VCF fabric provisioning and deployment ····················································································· 345
Topology discovery ································································································································ 346
Automated underlay network provisioning ····························································································· 346
Automated overlay network deployment ································································································ 348
Configuration restrictions and guidelines ······································································································· 348
VCF fabric configuration task list···················································································································· 349
Enabling VCF fabric topology discovery ········································································································ 349
Configuration restrictions and guidelines ······························································································· 349
Configuration procedure ························································································································· 349
Configuring automated underlay network provisioning ·················································································· 349
Configuration restrictions and guidelines ······························································································· 349
Configuration procedure ························································································································· 350
Configuring automated overlay network deployment ····················································································· 350
Configuration restrictions and guidelines ······························································································· 350
Configuration procedure ························································································································· 351
Displaying and maintaining VCF fabric ·········································································································· 352
Configuring Puppet ···················································································· 353
Overview ························································································································································ 353
Puppet network framework ···················································································································· 353
Puppet resources ··································································································································· 354
Configuring Puppet ········································································································································ 354
Configuration prerequisites ···················································································································· 354
Starting Puppet ······································································································································ 355
Shutting down Puppet on the device······································································································ 355
Puppet configuration example ······················································································································· 356
Network requirements ···························································································································· 356
Configuration procedure ························································································································· 356
Puppet resources······················································································· 357
netdev_vlan ···················································································································································· 357
Attributes ················································································································································ 357
Resource example ································································································································· 357
netdev_device ················································································································································ 357
Attributes ················································································································································ 358
Resource example ································································································································· 358
netdev_interface············································································································································· 358
Attributes ················································································································································ 359
Resource example ································································································································· 360
netdev_l2_interface ········································································································································ 360
Attributes ················································································································································ 360
Resource example ································································································································· 361
netdev_lagg···················································································································································· 361
Attributes ················································································································································ 361
Resource example ································································································································· 362
netdev_l2vpn ·················································································································································· 362
Attributes ················································································································································ 363
Resource example ································································································································· 363
netdev_vsi ······················································································································································ 363
Attributes ················································································································································ 363
Resource example ································································································································· 363
netdev_vte······················································································································································ 364
Attributes ················································································································································ 364
Resource example ································································································································· 365
netdev_vxlan ·················································································································································· 365
Attributes ················································································································································ 365
Resource example ································································································································· 366
Configuring Chef ························································································ 367
Overview ························································································································································ 367
Chef network framework ························································································································ 367
Chef resources ······································································································································· 368
x
Chef configuration file ···························································································································· 368
Configuring Chef ············································································································································ 370
Configuration prerequisites ···················································································································· 370
Starting Chef ·········································································································································· 370
Shutting down Chef ································································································································ 371
Chef configuration example ··························································································································· 371
Network requirements ···························································································································· 371
Configuration procedure ························································································································· 371
Chef resources ·························································································· 374
netdev_vlan ···················································································································································· 374
Properties and action ····························································································································· 374
Resource example ································································································································· 374
netdev_device ················································································································································ 374
Properties and action ····························································································································· 374
Resource example ································································································································· 375
netdev_interface············································································································································· 375
Properties ··············································································································································· 375
Resource example ································································································································· 376
netdev_l2_interface ········································································································································ 377
Properties ··············································································································································· 377
Resource example ································································································································· 377
netdev_lagg···················································································································································· 378
Properties and action ····························································································································· 378
Resource example ································································································································· 378
netdev_l2vpn ·················································································································································· 379
Properties ··············································································································································· 379
Resource example ································································································································· 379
netdev_vsi ······················································································································································ 379
Properties and action ····························································································································· 379
Resource example ································································································································· 380
netdev_vte······················································································································································ 380
Properties and action ····························································································································· 380
Resource example ································································································································· 381
netdev_vxlan ·················································································································································· 381
Properties and action ····························································································································· 381
Resource example ································································································································· 382
Document conventions and icons ······························································ 383
Conventions ··················································································································································· 383
Network topology icons ·································································································································· 384
Support and other resources ····································································· 385
Accessing Hewlett Packard Enterprise Support····························································································· 385
Accessing updates ········································································································································· 385
Websites ················································································································································ 386
Customer self repair ······························································································································· 386
Remote support ······································································································································ 386
Documentation feedback ······················································································································· 386
Index ·········································································································· 388
1
Using ping, tracert, and system
debugging
This chapter covers ping, tracert, and information about debugging the system.
Ping
Use the ping utility to determine if an address is reachable.
Ping sends ICMP echo requests (ECHO-REQUEST) to the destination device. Upon receiving the
requests, the destination device responds with ICMP echo replies (ECHO-REPLY) to the source
device. The source device outputs statistics about the ping operation, including the number of
packets sent, number of echo replies received, and the round-trip time. You can measure the
network performance by analyzing these statistics.
Using a ping command to test network connectivity
Execute ping commands in any view.
Task
Command
Remarks
Determine if an address in
an IP network is
reachable.
• For IPv4 networks:
ping [ ip ] [ -a source-ip | -c count | -f | -h
ttl | -i interface-type interface-number | -m
interval | -n | -p pad | -q | -r | -s
packet-size | -t timeout | -tos tos | -v |
-vpn-instance vpn-instance-name ] *
host
• For IPv6 networks:
ping ipv6 [ -a source-ipv6 | -c count | -i
interface-type interface-number | -m
interval | -q | -s packet-size | -t timeout | -v
| -tc traffic-class | -vpn-instance
vpn-instance-name ] * host
Increase the timeout time
(indicated by the -t keyword)
on a low-speed network.
For information about the ping mpls command, see MPLS Command Reference.
For information about the ping trill command, see TRILL Command Reference.
Ping example
Network requirements
As shown in Figure 1, determine if Device A and Device C can reach each other. If they can reach
each other, get detailed information about routes from Device A to Device C.
2
Figure 1 Network diagram
Configuration procedure
# Use the ping command on Device A to test connectivity to Device C.
<DeviceA> ping 1.1.2.2
Ping 1.1.2.2 (1.1.2.2): 56 data bytes, press CTRL_C to break
56 bytes from 1.1.2.2: icmp_seq=0 ttl=254 time=2.137 ms
56 bytes from 1.1.2.2: icmp_seq=1 ttl=254 time=2.051 ms
56 bytes from 1.1.2.2: icmp_seq=2 ttl=254 time=1.996 ms
56 bytes from 1.1.2.2: icmp_seq=3 ttl=254 time=1.963 ms
56 bytes from 1.1.2.2: icmp_seq=4 ttl=254 time=1.991 ms
--- Ping statistics for 1.1.2.2 ---
5 packet(s) transmitted, 5 packet(s) received, 0.0% packet loss
round-trip min/avg/max/std-dev = 1.963/2.028/2.137/0.062 ms
The output shows the following information:
•
Device A sends five ICMP packets to Device C and Device A receives five ICMP packets.
•
No ICMP packet is lost.
•
The route is reachable.
# Get detailed information about routes from Device A to Device C.
<DeviceA> ping -r 1.1.2.2
Ping 1.1.2.2 (1.1.2.2): 56 data bytes, press CTRL_C to break
56 bytes from 1.1.2.2: icmp_seq=0 ttl=254 time=4.685 ms
RR: 1.1.2.1
1.1.2.2
1.1.1.2
1.1.1.1
56 bytes from 1.1.2.2: icmp_seq=1 ttl=254 time=4.834 ms (same route)
56 bytes from 1.1.2.2: icmp_seq=2 ttl=254 time=4.770 ms (same route)
56 bytes from 1.1.2.2: icmp_seq=3 ttl=254 time=4.812 ms (same route)
56 bytes from 1.1.2.2: icmp_seq=4 ttl=254 time=4.704 ms (same route)
--- Ping statistics for 1.1.2.2 ---
5 packet(s) transmitted, 5 packet(s) received, 0.0% packet loss
round-trip min/avg/max/std-dev = 4.685/4.761/4.834/0.058 ms
The test procedure of ping –r is as shown in Figure 1:
Device A Device B Device C
1.1.1.1/24
1.1.1.2/24
ECHO-REQUEST
(NULL) ECHO-REQUEST
1st=1.1.2.1
1.1.2.1/24
1.1.2.2/24
ECHO-REPLY
1st=1.1.2.1
2nd=1.1.2.2
ECHO-REPLY
1st=1.1.2.1
2nd=1.1.2.2
3rd＝1.1.1.2
ECHO-REPLY
1st=1.1.2.1
2nd=1.1.2.2
3rd＝1.1.1.2
4th=1.1.1.1
3
1. The source device (Device A) sends an ICMP echo request to the destination device (Device C)
with the RR option blank.
2. The intermediate device (Device B) adds the IP address of its outbound interface (1.1.2.1) to
the RR option of the ICMP echo request, and forwards the packet.
3. Upon receiving the request, the destination device copies the RR option in the request and
adds the IP address of its outbound interface (1.1.2.2) to the RR option. Then the destination
device sends an ICMP echo reply.
4. The intermediate device adds the IP address of its outbound interface (1.1.1.2) to the RR option
in the ICMP echo reply, and then forwards the reply.
5. Upon receiving the reply, the source device adds the IP address of its inbound interface (1.1.1.1)
to the RR option. The detailed information of routes from Device A to Device C is formatted as:
1.1.1.1 <-> {1.1.1.2; 1.1.2.1} <-> 1.1.2.2.
Tracert
Tracert (also called Traceroute) enables retrieval of the IP addresses of Layer 3 devices in the path
to a destination. In the event of network failure, use tracert to test network connectivity and identify
failed nodes.
Figure 2 Tracert operation
Tracert uses received ICMP error messages to get the IP addresses of devices. Tracert works as
shown in Figure 2:
1. The source device sends a UDP packet with a TTL value of 1 to the destination device. The
destination UDP port is not used by any application on the destination device.
2. The first hop (Device B, the first Layer 3 device that receives the packet) responds by sending a
TTL-expired ICMP error message to the source, with its IP address (1.1.1.2) encapsulated. This
way, the source device can get the address of the first Layer 3 device (1.1.1.2).
3. The source device sends a packet with a TTL value of 2 to the destination device.
4. The second hop (Device C) responds with a TTL-expired ICMP error message, which gives the
source device the address of the second Layer 3 device (1.1.2.2).
5. This process continues until a packet sent by the source device reaches the ultimate
destination device. Because no application uses the destination port specified in the packet, the
destination device responds with a port-unreachable ICMP message to the source device, with
its IP address encapsulated. This way, the source device gets the IP address of the destination
device (1.1.3.2).
6. The source device determines that:
The packet has reached the destination device after receiving the port-unreachable ICMP
message.
Device A Device B Device D
Device C
1.1.1.1/24
1.1.1.2/24
Hop Limit=1
TTL exceeded
1.1.2.1/24 1.1.3.1/24
1.1.2.2/24 1.1.3.2/24
Hop Limit=2
TTL exceeded
UDP port unreachable
Hop Limit=n
4
The path to the destination device is 1.1.1.2 to 1.1.2.2 to 1.1.3.2.
Prerequisites
Before you use a tracert command, perform the tasks in this section.
For an IPv4 network:
•
Enable sending of ICMP timeout packets on the intermediate devices (devices between the
source and destination devices). If the intermediate devices are HPE devices, execute the ip
ttl-expires enable command on the devices. For more information about this command, see
Layer 3—IP Services Command Reference.
•
Enable sending of ICMP destination unreachable packets on the destination device. If the
destination device is an HPE device, execute the ip unreachables enable command. For more
information about this command, see Layer 3—IP Services Command Reference.
For an IPv6 network:
•
Enable sending of ICMPv6 timeout packets on the intermediate devices (devices between the
source and destination devices). If the intermediate devices are HPE devices, execute the ipv6
hoplimit-expires enable command on the devices. For more information about this command,
see Layer 3—IP Services Command Reference.
•
Enable sending of ICMPv6 destination unreachable packets on the destination device. If the
destination device is an HPE device, execute the ipv6 unreachables enable command. For
more information about this command, see Layer 3—IP Services Command Reference.
Using a tracert command to identify failed or all nodes in a
path
Execute tracert commands in any view.
Task
Command
Remarks
Display the routes
from source to
destination.
• For IPv4 networks:
tracert [ -a source-ip | -f first-ttl | -m max-ttl | -p port | -q
packet-number | -t tos | -vpn-instance
vpn-instance-name [ -resolve-as { global | none |
vpn } ] | -w timeout ] * host
• For IPv6 networks:
tracert ipv6 [-a source-ip | -f first-hop | -m max-hops |
-p port | -q packet-number | -t traffic-class |
-vpn-instance vpn-instance-name [ -resolve-as
{ global | none | vpn } ] | -w timeout ] * host
The -a source-ip
option is available in
the tracert ipv6
command in Release
2612P06 and later.
For information about the tracert mpls command, see MPLS Command Reference.
For information about the tracert trill command, see TRILL Command Reference.
Tracert example
Network requirements
As shown in Figure 3, Device A failed to Telnet to Device C.
Test the network connectivity between Device A and Device C. If they cannot reach each other,
locate the failed nodes in the network.
5
Figure 3 Network diagram
Configuration procedure
1. Configure the IP addresses for devices as shown in Figure 3.
2. Configure a static route on Device A.
<DeviceA> system-view
[DeviceA] ip route-static 0.0.0.0 0.0.0.0 1.1.1.2
[DeviceA] quit
3. Use the ping command to test connectivity between Device A and Device C.
<DeviceA> ping 1.1.2.2
Ping 1.1.2.2(1.1.2.2): 56 -data bytes, press CTRL_C to break
Request time out
Request time out
Request time out
Request time out
Request time out
--- Ping statistics for 1.1.2.2 ---
5 packet(s) transmitted,0 packet(s) received,100.0% packet loss
The output shows that Device A and Device C cannot reach each other.
4. Use the tracert command to identify failed nodes:
# Enable sending of ICMP timeout packets on Device B.
<DeviceB> system-view
[DeviceB] ip ttl-expires enable
# Enable sending of ICMP destination unreachable packets on Device C.
<DeviceC> system-view
[DeviceC] ip unreachables enable
# Execute the tracert command on Device A.
<DeviceA> tracert 1.1.2.2
traceroute to 1.1.2.2 (1.1.2.2) 30 hops at most,40 bytes each packet, press CTRL_C
to break
1 1.1.1.2 (1.1.1.2) 1 ms 2 ms 1 ms
2 * * *
3 * * *
4 * * *
5
<DeviceA>
The output shows that Device A can reach Device B but cannot reach Device C. An error has
occurred on the connection between Device B and Device C.
5. To identify the cause of the problem, execute the following commands on Device A and Device
C:
Execute the debugging ip icmp command and verify that Device A and Device C can send
and receive the correct ICMP packets.
Execute the display ip routing-table command to verify that Device A and Device C have a
route to each other.
Device A Device B Device C
1.1.1.1/24 1.1.1.2/24 1.1.2.1/24 1.1.2.2/24
6
System debugging
The device supports debugging for the majority of protocols and features, and provides debugging
information to help users diagnose errors.
Debugging information control switches
The following switches control the display of debugging information:
•
Module debugging switch—Controls whether to generate the module-specific debugging
information.
•
Screen output switch—Controls whether to display the debugging information on a certain
screen. Use terminal monitor and terminal logging level commands to turn on the screen
output switch. For more information about these two commands, see Network Management
and Monitoring Command Reference.
As shown in Figure 4, the device can provide debugging for the three modules 1, 2, and 3. The
debugging information can be output on a terminal only when both the module debugging switch and
the screen output switch are turned on.
Debugging information is typically displayed on a console. You can also send debugging information
to other destinations. For more information, see "Configuring the information center."
Figure 4 Relationship between the module and screen output switch
Debugging a feature module
Output of debugging commands is memory intensive. To guarantee system performance, enable
debugging only for modules that are in an exceptional condition. When debugging is complete, use
the undo debugging all command to disable all the debugging functions.
In an IRF 3.1 system, PEXs do not send debugging information to the parent fabric. To debug a PEX,
use the switchto pex command to log in to the PEX and then enable the debugging functions as
needed. For more information about the switchto pex command, see Virtual Technologies
Command Reference.
To debug a feature module:
13
1 2 3
13
13
Debugging
information
Protocol
debugging
switch
Screen
output switch
Debugging
information
Protocol
debugging
switch
Screen
output switch
ON OFF ON
OFF
1 2 3
ON OFF ON
ON
7
Step
Command
Remarks
1. Enable debugging for a
module in user view. debugging module-name
[ option ] By default, all debugging
functions are disabled.
2. (Optional.) Display the
enabled debugging in any
view. display debugging
[ module-name ] N/A
8
Configuring NQA
Overview
Network quality analyzer (NQA) allows you to measure network performance, verify the service
levels for IP services and applications, and troubleshoot network problems. It provides the following
types of operations:
•
ICMP echo.
•
ICMP jitter.
•
DHCP.
•
DLSw.
•
DNS.
•
FTP.
•
HTTP.
•
Path jitter.
•
SNMP.
•
TCP.
•
UDP echo.
•
UDP jitter.
•
UDP tracert.
•
Voice.
As shown in Figure 5, the NQA source device (NQA client) sends data to the NQA destination device
by simulating IP services and applications to measure network performance. The obtained
performance metrics include the one-way latency, jitter, packet loss, voice quality, application
performance, and server response time.
All types of NQA operations require the NQA client, but only the TCP, UDP echo, UDP jitter, and
voice operations require the NQA server. The NQA operations for services that are already provided
by the destination device such as FTP do not need the NQA server.
You can configure the NQA server to listen and respond to specific IP addresses and ports to meet
various test needs.
Figure 5 Network diagram
NQA operation
The following describes how NQA performs different types of operations:
•
A TCP or DLSw operation sets up a connection.
•
An ICMP jitter, UDP jitter, or voice operation sends a number of probe packets. The number of
probe packets is set by using the probe packet-number command.
•
An FTP operation uploads or downloads a file.
IP network
NQA source device/
NQA client NQA destination device
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
/