Broadcom 88690-PG207-PUB
September 8, 2020
BCM88690
Traffic Manager
Programming Guide
BCM88690
BCM88800
BCM88480
BCM88280
Broadcom, the pulse logo, Connecting everything, Avago Technologies, Avago, the A logo, BroadSync, and StrataDNX are
among the trademarks of Broadcom and/or its affiliates in the United States, certain other countries, and/or the EU.
Copyright © 2018–2020 Broadcom. All Rights Reserved.
The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries. For more information, please visit www.broadcom.com.
Broadcom reserves the right to make changes without further notice to any products or data herein to improve reliability,
function, or design. Information furnished by Broadcom is believed to be accurate and reliable. However, Broadcom does
not assume any liability arising out of the application or use of this information, nor the application or use of any product or
circuit described herein, neither does it convey any license under its patent rights nor the rights of others.
BCM88690 Programming Guide Traffic Manager
Broadcom 88690-PG207-PUB
2
Broadcom 88690-PG207-PUB
3
BCM88690 Programming Guide Traffic Manager
Table of Contents
Chapter 1: SDK General Concepts .................................................................................................. 17
1.1 BCM APIs.................................................................................................................................................................17
1.1.1 API Return Values ..........................................................................................................................................17
1.1.2 BCM API Objects – GPORTs .........................................................................................................................17
1.2 SOC Properties........................................................................................................................................................19
1.2.1 Default Configuration File ...............................................................................................................................19
1.3 Shell Commands.....................................................................................................................................................20
1.4 Application Reference............................................................................................................................................20
1.4.1 CINT Reference ..............................................................................................................................................20
1.4.2 Default Application Reference ........................................................................................................................20
1.4.3 Tune Reference ..............................................................................................................................................21
1.5 Device-Family Differences .....................................................................................................................................21
Chapter 2: Initialization Template ...................................................................................................22
2.1 Reference.................................................................................................................................................................22
2.2 Driver Reference .....................................................................................................................................................22
2.2.1 APIs ................................................................................................................................................................22
2.2.2 SOC Properties ...............................................................................................................................................28
2.2.3 Shell Commands.............................................................................................................................................28
2.2.4 Configuration Flow ..........................................................................................................................................29
2.2.5 Application Reference.....................................................................................................................................30
2.2.5.1 Enable or Disable Applications .............................................................................................................31
2.2.5.2 Typical Traffic Manager Reference Applications ..................................................................................31
2.2.5.3 Change Application Attributes ..............................................................................................................31
2.3 Device-Family Differences .....................................................................................................................................32
Chapter 3: Hardware Interfaces .......................................................................................................33
3.1 Network Interface....................................................................................................................................................33
3.1.1 Theory of Operation ........................................................................................................................................33
3.1.1.1 Logical Port Mapping ............................................................................................................................33
3.1.1.2 Constraints............................................................................................................................................33
3.1.2 Ethernet Ports .................................................................................................................................................33
3.1.2.1 Port Indexing.........................................................................................................................................33
3.1.3 Driver Reference.............................................................................................................................................41
3.1.3.1 APIs ......................................................................................................................................................41
3.1.3.2 SOC Properties.....................................................................................................................................50
3.1.3.3 Shell Commands ..................................................................................................................................52
3.1.3.4 Configuration Flow................................................................................................................................52
3.1.3.5 Application Reference...........................................................................................................................52
Broadcom 88690-PG207-PUB
4
BCM88690 Programming Guide Traffic Manager
3.1.4 Interlaken ........................................................................................................................................................53
3.1.4.1 ILKN Rates ...........................................................................................................................................53
3.1.4.2 ILKN FEC..............................................................................................................................................53
3.1.4.3 ILKN Lane Assignment and Mapping ...................................................................................................54
3.1.4.4 Driver Reference...................................................................................................................................54
3.1.4.5 Interlaken over Fabric ...........................................................................................................................56
3.1.4.6 External Lookup Interface Ports ...........................................................................................................56
3.1.5 Statistics Counters ..........................................................................................................................................57
3.1.5.1 Driver Reference...................................................................................................................................58
3.1.6 LSS Fault Indications ......................................................................................................................................59
3.1.6.1 Driver Reference...................................................................................................................................59
3.1.7 NIF Priority ......................................................................................................................................................60
3.1.7.1 Driver Reference...................................................................................................................................60
3.1.8 RX Timestamp ................................................................................................................................................61
3.1.8.1 Driver Reference...................................................................................................................................61
3.1.9 FlexE Interface................................................................................................................................................62
3.1.9.1 FlexE Physical Interface .......................................................................................................................62
3.1.9.2 FlexE Group..........................................................................................................................................62
3.1.9.3 FlexE Client ..........................................................................................................................................62
3.1.9.4 FlexE Virtual Client ...............................................................................................................................62
3.1.9.5 FlexE System Modes............................................................................................................................63
3.1.9.6 Driver Reference...................................................................................................................................63
3.1.9.7 APIs ......................................................................................................................................................63
3.1.9.8 SOC Properties.....................................................................................................................................74
3.1.10 Device-Specific Differences ..........................................................................................................................75
3.2 Fabric Interface .......................................................................................................................................................75
3.2.1 Link Indexing...................................................................................................................................................75
3.2.1.1 Driver Reference...................................................................................................................................75
3.2.2 Fabric Rates....................................................................................................................................................77
3.2.2.1 Driver Reference...................................................................................................................................77
3.2.3 PCS Layer Line Coding ..................................................................................................................................77
3.2.3.1 Driver Reference...................................................................................................................................77
3.2.4 Power Consumption........................................................................................................................................79
3.2.4.1 Driver Reference...................................................................................................................................79
3.3 PHY...........................................................................................................................................................................80
3.3.1 PHY Constraints .............................................................................................................................................80
3.3.1.1 Blackhawk Constraints .........................................................................................................................80
3.3.1.2 Falcon Constraints................................................................................................................................80
3.3.2 Driver Reference.............................................................................................................................................80
3.3.2.1 APIs ......................................................................................................................................................80
3.3.2.2 SOC Properties.....................................................................................................................................84
Broadcom 88690-PG207-PUB
5
BCM88690 Programming Guide Traffic Manager
3.3.2.3 Shell Commands ..................................................................................................................................85
3.3.3 Lane Map ........................................................................................................................................................86
3.3.3.1 API ........................................................................................................................................................87
3.3.3.2 ILKN Lane Mapping ..............................................................................................................................89
3.3.3.3 Lane Mapping Usage Examples ...........................................................................................................91
3.3.4 Device Family-Specific Differences ................................................................................................................94
3.4 Linkscan...................................................................................................................................................................95
3.4.1 Driver Reference.............................................................................................................................................95
3.4.1.1 APIs ......................................................................................................................................................95
3.4.1.2 SOC Properties.....................................................................................................................................96
3.4.1.3 Shell Commands ..................................................................................................................................98
3.4.1.4 Application Reference...........................................................................................................................98
3.5 Synchronous Ethernet............................................................................................................................................98
3.5.1 Driver Reference.............................................................................................................................................98
3.5.1.1 APIs ......................................................................................................................................................98
3.5.1.2 SOC Properties.....................................................................................................................................99
3.6 DRAM .....................................................................................................................................................................100
3.6.1 Reference .....................................................................................................................................................100
3.6.2 Theory of Operation ......................................................................................................................................100
3.6.3 Temperature Monitor ....................................................................................................................................100
3.6.4 Driver Reference...........................................................................................................................................101
3.6.4.1 APIs ....................................................................................................................................................101
3.6.4.2 SOC Properties...................................................................................................................................102
3.6.4.3 Configuration Flow..............................................................................................................................104
3.6.4.4 Shell Commands ................................................................................................................................105
3.6.5 Tune..............................................................................................................................................................105
3.6.6 Device-Family Differences ............................................................................................................................106
3.7 Packet Reception and Transmission ..................................................................................................................107
3.7.1 BCM Packet Structure ..................................................................................................................................107
3.7.1.1 Structure bcm_pkt_blk_t .....................................................................................................................107
3.7.1.2 Metadata.............................................................................................................................................107
3.7.1.3 Dune Special Metadata ......................................................................................................................108
3.7.1.4 BCM Packet Flags ..............................................................................................................................109
3.7.2 Packet Receive and Transmit Functions ......................................................................................................110
3.7.2.1 APIs ....................................................................................................................................................110
3.7.2.2 SOC Properties...................................................................................................................................123
3.7.3 Receiving Packets ........................................................................................................................................123
3.7.4 Sending Packets ...........................................................................................................................................124
3.7.5 Device-Family Differences ............................................................................................................................124
Chapter 4: Data Interfaces and Ports ............................................................................................ 125
Broadcom 88690-PG207-PUB
6
BCM88690 Programming Guide Traffic Manager
4.1 Physical Ports .......................................................................................................................................................125
4.2 Physical Interface..................................................................................................................................................125
4.3 Logical Port and Local Port .................................................................................................................................126
4.4 TM Port...................................................................................................................................................................126
4.5 System Port ...........................................................................................................................................................126
4.6 Module Port ...........................................................................................................................................................126
4.7 Egress Replication Port........................................................................................................................................126
4.8 Multiple FAP-IDs per Core....................................................................................................................................127
4.8.1 FAP-ID Allocation..........................................................................................................................................127
4.8.2 Mapping a System Port to Module Port ........................................................................................................127
4.8.3 Mapping a VOQ Connector to VOQ..............................................................................................................127
4.9 Driver Reference ...................................................................................................................................................128
4.9.1 APIs ..............................................................................................................................................................128
4.9.1.1 Port Provisioning.................................................................................................................................128
4.9.1.2 System and Module Ports...................................................................................................................128
4.10 Device-Family Differences .................................................................................................................................128
Chapter 5: Port Provisioning ......................................................................................................... 129
5.1 Driver Reference ...................................................................................................................................................130
5.1.1 APIs ..............................................................................................................................................................130
5.1.1.1 Port Add and Remove ........................................................................................................................130
5.1.2 SOC Properties .............................................................................................................................................133
5.1.2.1 Adding Ports .......................................................................................................................................133
5.1.3 Shell Commands...........................................................................................................................................135
5.1.4 Configuration Flow ........................................................................................................................................136
5.1.5 Application Reference...................................................................................................................................137
Chapter 6: Trunk (Link Aggregation) Ports .................................................................................. 139
6.1 Theory of Operation..............................................................................................................................................139
6.1.1 Trunk LB-Key and PSC.................................................................................................................................139
6.1.2 Trunk PP Ports..............................................................................................................................................139
6.1.3 Trunks as Multicast Group Members ............................................................................................................140
6.1.3.1 Trunk as Ingress MC Members ..........................................................................................................140
6.1.3.2 Trunk as Egress MC Members ...........................................................................................................140
6.1.4 Trunk Member Flags.....................................................................................................................................140
6.1.5 Virtual Trunks................................................................................................................................................141
6.2 Driver Reference ...................................................................................................................................................141
6.2.1 APIs ..............................................................................................................................................................141
6.2.2 SOC Properties .............................................................................................................................................146
6.2.3 Application Reference...................................................................................................................................146
Chapter 7: Ingress Traffic Management ....................................................................................... 147
Broadcom 88690-PG207-PUB
7
BCM88690 Programming Guide Traffic Manager
7.1 Queue Creation .....................................................................................................................................................148
7.1.1 VOQs ............................................................................................................................................................148
7.1.2 FMQs ............................................................................................................................................................148
7.1.3 Driver Reference...........................................................................................................................................148
7.1.3.1 APIs ....................................................................................................................................................148
7.1.3.2 SOC Properties...................................................................................................................................153
7.1.3.3 Shell Commands ................................................................................................................................153
7.1.3.4 Configuration Flow..............................................................................................................................153
7.1.3.5 Application Reference.........................................................................................................................155
7.2 Queue Scheduling.................................................................................................................................................156
7.2.1 Driver Reference...........................................................................................................................................157
7.2.1.1 APIs ....................................................................................................................................................157
7.2.1.2 SOC Properties...................................................................................................................................165
7.2.1.3 Shell Commands ................................................................................................................................165
7.2.1.4 Configuration Flow..............................................................................................................................165
7.2.1.5 Application Reference.........................................................................................................................165
7.3 Queue Statistics Generation................................................................................................................................166
7.3.1 Reference .....................................................................................................................................................166
7.3.2 Driver Reference...........................................................................................................................................166
7.3.2.1 APIs ....................................................................................................................................................166
7.3.2.2 Shell Commands ................................................................................................................................167
7.4 Ingress Congestion Management........................................................................................................................167
7.4.1 Reference .....................................................................................................................................................168
7.5 VOQ Ingress Congestion Management...............................................................................................................168
7.5.1 Reference .....................................................................................................................................................168
7.5.2 Driver Reference...........................................................................................................................................168
7.5.2.1 APIs ....................................................................................................................................................169
7.5.2.2 SOC Properties...................................................................................................................................176
7.5.2.3 Shell Commands ................................................................................................................................176
7.5.2.4 Configuration Flow..............................................................................................................................176
7.5.2.5 Application Reference.........................................................................................................................177
7.6 Virtual Statistics Queues......................................................................................................................................177
7.6.1 Reference .....................................................................................................................................................177
7.6.2 Driver Reference...........................................................................................................................................177
7.6.2.1 APIs ....................................................................................................................................................177
7.6.2.2 SOC Properties...................................................................................................................................188
7.6.2.3 Shell Commands ................................................................................................................................188
7.6.2.4 Application Reference.........................................................................................................................188
7.7 System RED...........................................................................................................................................................189
7.7.1 Reference .....................................................................................................................................................189
Broadcom 88690-PG207-PUB
8
BCM88690 Programming Guide Traffic Manager
7.7.2 Driver Reference...........................................................................................................................................189
7.7.2.1 APIs ....................................................................................................................................................189
7.7.2.2 SOC Properties...................................................................................................................................192
7.7.2.3 Shell Commands ................................................................................................................................192
7.8 Device-Level Ingress Congestion Management.................................................................................................192
7.8.1 Driver Reference...........................................................................................................................................192
7.8.1.1 APIs ....................................................................................................................................................192
7.9 Drop Precedence...................................................................................................................................................194
7.9.1 Driver Reference...........................................................................................................................................194
7.9.1.1 APIs ....................................................................................................................................................194
7.10 Ingress Compensation .......................................................................................................................................195
7.10.1 Reference ...................................................................................................................................................195
7.10.2 Driver Reference.........................................................................................................................................195
7.10.2.1 APIs ..................................................................................................................................................195
7.10.2.2 Application Reference.......................................................................................................................197
7.11 Ingress Shapers ..................................................................................................................................................198
7.11.1 APIs ............................................................................................................................................................198
7.12 Priority Drop ........................................................................................................................................................199
7.12.1 Reference ...................................................................................................................................................200
7.12.2 Driver Reference.........................................................................................................................................200
7.12.2.1 APIs ..................................................................................................................................................200
7.13 Mirror-On-Drop (Dropped Packet Reports).......................................................................................................209
7.13.1 Reference ...................................................................................................................................................209
7.13.2 Driver Reference.........................................................................................................................................210
7.13.2.1 APIs ..................................................................................................................................................210
7.13.3 Configuration Sequence .............................................................................................................................211
7.13.4 Application Reference.................................................................................................................................211
7.14 Device-Family Differences .................................................................................................................................211
Chapter 8: Multicast .......................................................................................................................212
8.1 Theory of Operation..............................................................................................................................................212
8.2 Multicast Mechanisms..........................................................................................................................................212
8.3 Driver Reference ...................................................................................................................................................213
8.3.1 APIs ..............................................................................................................................................................213
8.3.2 SOC Properties .............................................................................................................................................217
8.3.3 Shell Commands...........................................................................................................................................218
8.3.4 Configuration Flow ........................................................................................................................................218
8.3.5 Application Reference...................................................................................................................................220
8.4 Device-Family Differences ...................................................................................................................................220
Chapter 9: Fabric Data Path ...........................................................................................................221
9.1 Fabric Pipes...........................................................................................................................................................221
Broadcom 88690-PG207-PUB
9
BCM88690 Programming Guide Traffic Manager
9.1.1 Driver Reference...........................................................................................................................................221
9.1.1.1 SOC Properties...................................................................................................................................221
9.2 Fabric Priority........................................................................................................................................................223
9.2.1 Driver Reference...........................................................................................................................................224
9.2.1.1 APIs ....................................................................................................................................................224
9.2.1.2 Configuration Flow..............................................................................................................................224
9.3 Mesh Configuration ..............................................................................................................................................225
9.3.1 Driver Reference...........................................................................................................................................225
9.3.1.1 APIs ....................................................................................................................................................225
9.3.1.2 SOC Properties...................................................................................................................................227
9.3.1.3 Configuration Flow..............................................................................................................................227
9.4 Fabric Routing.......................................................................................................................................................228
9.4.1 Driver Reference...........................................................................................................................................228
9.4.1.1 APIs ....................................................................................................................................................228
9.5 Fabric Congestion Management..........................................................................................................................230
9.5.1 Driver Reference...........................................................................................................................................231
9.5.1.1 APIs ....................................................................................................................................................231
9.6 Fabric Pipes Scheduling (WFQs).........................................................................................................................232
9.6.1 Driver Reference...........................................................................................................................................232
9.6.1.1 APIs ....................................................................................................................................................232
9.7 Fabric Device Isolation.........................................................................................................................................234
9.7.1 Driver Reference...........................................................................................................................................234
9.7.1.1 APIs ....................................................................................................................................................234
9.8 0: Isolate device.Device Family-Specific Differences........................................................................................234
Chapter 10: Egress Traffic Management ......................................................................................235
10.1 Egress Transmit Scheduler................................................................................................................................235
10.2 Reference.............................................................................................................................................................235
10.3 Port and Interface Shaping ................................................................................................................................236
10.3.1 Driver Reference .........................................................................................................................................236
10.3.1.1 APIs ..................................................................................................................................................236
10.3.1.2 Shell Commands ..............................................................................................................................237
10.3.1.3 Configuration Flow............................................................................................................................238
10.4 Priority Propagation (Queue Pair Priority)........................................................................................................238
10.4.1 Driver Reference .........................................................................................................................................239
10.4.1.1 APIs ..................................................................................................................................................239
10.4.1.2 Configuration Flow............................................................................................................................239
10.5 Header Compensation........................................................................................................................................240
10.5.1 Driver Reference .........................................................................................................................................240
10.5.1.1 Shell Commands ..............................................................................................................................240
10.6 Egress Port Scheduler........................................................................................................................................240
Broadcom 88690-PG207-PUB
10
BCM88690 Programming Guide Traffic Manager
10.6.1 Driver Reference .........................................................................................................................................241
10.6.1.1 APIs ..................................................................................................................................................241
10.6.1.2 SOC Properties.................................................................................................................................241
10.6.2 8P-Port Scheduler.......................................................................................................................................242
10.6.3 Driver Reference .........................................................................................................................................242
10.6.3.1 APIs ..................................................................................................................................................242
10.6.3.2 Configuration Flow............................................................................................................................245
10.6.4 4P-Port Scheduler.......................................................................................................................................245
10.6.4.1 Driver Reference...............................................................................................................................246
10.6.5 2P-Port Scheduler.......................................................................................................................................246
10.6.6 Driver Reference .........................................................................................................................................247
10.6.6.1 APIs ..................................................................................................................................................247
10.6.6.2 SOC Properties.................................................................................................................................247
10.6.6.3 Shell Commands ..............................................................................................................................247
10.6.6.4 Configuration Flow............................................................................................................................248
10.6.7 1P-Port Scheduler.......................................................................................................................................248
10.6.8 Driver Reference .........................................................................................................................................248
10.6.8.1 APIs ..................................................................................................................................................248
10.6.8.2 SOC Properties.................................................................................................................................248
10.6.8.3 Shell Commands ..............................................................................................................................248
10.6.8.4 Configuration Flow............................................................................................................................249
10.7 Queue-Pair Scheduler.........................................................................................................................................249
10.7.1 Driver Reference .........................................................................................................................................249
10.7.1.1 APIs ..................................................................................................................................................249
10.8 Advanced Egress Traffic Management.............................................................................................................250
10.8.1 Port Enable/Disable ....................................................................................................................................250
10.8.2 Driver Reference .........................................................................................................................................250
10.8.2.1 APIs ..................................................................................................................................................250
10.9 Recycling Interfaces...........................................................................................................................................252
10.9.1 Driver Reference .........................................................................................................................................252
10.9.1.1 APIs ..................................................................................................................................................252
10.10 Egress Congestion Management ....................................................................................................................254
10.10.1 Theory of Operation ..................................................................................................................................254
10.10.1.1 Global Traffic Resources ................................................................................................................254
10.10.1.2 Unicast Traffic Resources...............................................................................................................254
10.10.1.3 Multicast Resource Management ...................................................................................................255
10.10.2 Driver Reference.......................................................................................................................................255
10.10.2.1 API ..................................................................................................................................................256
10.10.2.2 SOC Properties...............................................................................................................................257
10.10.2.3 Shell Commands ............................................................................................................................258
Broadcom 88690-PG207-PUB
11
BCM88690 Programming Guide Traffic Manager
10.10.2.4 Application Reference.....................................................................................................................258
10.11 Egress Port and CoS Mapping.........................................................................................................................263
10.11.1 Driver Reference.......................................................................................................................................263
10.11.1.1 APIs ................................................................................................................................................263
10.12 Mapping Port Priority (Queue Pair) to Multicast PD Service Pools..............................................................265
10.13 Device-Family Differences ...............................................................................................................................266
Chapter 11: Egress (End-to-End) Credit Scheduler .................................................................... 267
11.1 Credit Generator..................................................................................................................................................267
11.1.1 Driver Reference .........................................................................................................................................268
11.1.1.1 APIs ..................................................................................................................................................268
11.2 Core-Port Scheduler ...........................................................................................................................................268
11.2.1 Driver Reference .........................................................................................................................................268
11.2.1.1 APIs ..................................................................................................................................................268
11.2.1.2 Configuration Flow............................................................................................................................269
11.3 Port-TC Scheduler...............................................................................................................................................269
11.3.1 Driver Reference .........................................................................................................................................270
11.3.1.1 APIs ..................................................................................................................................................270
11.3.1.2 SOC Properties.................................................................................................................................272
11.3.1.3 Configuration Flow............................................................................................................................272
11.3.1.4 Shell Commands ..............................................................................................................................272
11.4 Scheduler Flows..................................................................................................................................................273
11.4.1 Aggregate Schedulers ................................................................................................................................273
11.4.2 VOQ Connectors.........................................................................................................................................273
11.4.3 Driver Reference .........................................................................................................................................273
11.4.3.1 APIs ..................................................................................................................................................273
11.4.3.2 SOC Properties.................................................................................................................................284
11.4.3.3 Shell Commands ..............................................................................................................................285
11.4.3.4 Configuration Flow............................................................................................................................285
11.4.3.5 Application Reference.......................................................................................................................288
11.5 Port Priority Propagation ...................................................................................................................................288
11.5.1 Driver Reference .........................................................................................................................................289
11.5.1.1 APIs ..................................................................................................................................................289
11.5.1.2 SOC Properties.................................................................................................................................289
11.6 Egress Replication Port......................................................................................................................................289
11.6.1 Driver Reference .........................................................................................................................................289
11.6.1.1 SOC Properties.................................................................................................................................289
11.6.1.2 Configuration Flow............................................................................................................................290
11.7 Fabric Multicast Scheduling Hierarchy.............................................................................................................291
11.7.1 Driver Reference .........................................................................................................................................291
11.7.1.1 APIs ..................................................................................................................................................291
Broadcom 88690-PG207-PUB
12
BCM88690 Programming Guide Traffic Manager
11.7.1.2 Configuration Flow............................................................................................................................291
11.8 Scheduling Hierarchy Modification During Traffic ..........................................................................................292
11.9 Device-Family Differences .................................................................................................................................292
Chapter 12: TDM and OTN Application ......................................................................................... 293
12.1 Theory of Operation............................................................................................................................................293
12.1.1 Bypass Path................................................................................................................................................293
12.1.2 Packet Path.................................................................................................................................................294
12.1.3 Direct Fabric Routing ..................................................................................................................................294
12.1.4 TDM Mesh MC............................................................................................................................................294
12.2 Driver Reference .................................................................................................................................................294
12.2.1 APIs ............................................................................................................................................................294
12.2.1.1 TDM Cell Size...................................................................................................................................294
12.2.1.2 TDM Interface Configuration.............................................................................................................294
12.2.1.3 TDM Ingress Settings .......................................................................................................................295
12.2.1.4 TDM Egress Settings........................................................................................................................297
12.2.2 SOC Properties...........................................................................................................................................298
12.2.3 Detailed Configuration Flow........................................................................................................................299
Chapter 13: Flow-Control Configuration ...................................................................................... 301
13.1 Theory of Operation............................................................................................................................................301
13.1.1 Flow Control Path Mapping.........................................................................................................................301
13.1.2 PFC Deadlock Detection and Recovery .....................................................................................................302
13.2 Driver Reference .................................................................................................................................................303
13.2.1 APIs ............................................................................................................................................................303
13.2.2 SoC Properties............................................................................................................................................313
13.2.3 Shell Commands.........................................................................................................................................314
13.2.4 Configuration Flow ......................................................................................................................................314
13.2.4.1 FC Path Mapping..............................................................................................................................314
13.2.4.2 PFC Deadlock Detection and Recovery ...........................................................................................315
13.2.5 Application Reference.................................................................................................................................315
Chapter 14: Sniff ............................................................................................................................. 316
14.1 Theory of Operation............................................................................................................................................316
14.2 Driver Reference .................................................................................................................................................317
14.2.1 APIs ............................................................................................................................................................317
14.2.1.1 SOC Properties.................................................................................................................................321
14.2.1.2 Shell Commands ..............................................................................................................................321
14.2.1.3 Configuration Flow............................................................................................................................321
14.2.1.4 Application Reference.......................................................................................................................321
14.3 Device-Family Differences .................................................................................................................................321
Chapter 15: Counter Processor .....................................................................................................322
Broadcom 88690-PG207-PUB
13
BCM88690 Programming Guide Traffic Manager
15.1 Theory of Operation............................................................................................................................................322
15.2 Counter Processor Configuration Steps ..........................................................................................................323
15.2.1 Expansion Configuration.............................................................................................................................323
15.2.1.1 Driver Reference...............................................................................................................................323
15.2.2 Database Configuration ..............................................................................................................................325
15.2.2.1 Attaching Engines.............................................................................................................................325
15.2.2.2 Database Properties .........................................................................................................................325
15.2.3 Driver Reference .........................................................................................................................................325
15.2.3.1 SC Generation Points – References.................................................................................................325
15.2.3.2 Multicast............................................................................................................................................326
15.2.4 Engine Eviction Configuration.....................................................................................................................333
15.2.4.1 Driver Reference...............................................................................................................................333
15.2.4.2 Shell Commands ..............................................................................................................................336
15.2.4.3 Configuration Flow............................................................................................................................336
15.2.4.4 Database Destroy .............................................................................................................................337
15.2.4.5 Reduce Database Range .................................................................................................................337
15.2.4.6 Extend Database Range...................................................................................................................337
15.2.4.7 Application Reference.......................................................................................................................337
15.3 Reading Counters...............................................................................................................................................338
15.3.1 Theory of Operation ....................................................................................................................................338
15.3.2 Driver Reference .........................................................................................................................................339
15.3.2.1 APIs ..................................................................................................................................................340
15.3.2.2 SOC Properties.................................................................................................................................342
15.3.2.3 Shell Commands ..............................................................................................................................342
15.3.2.4 Application Reference.......................................................................................................................343
15.4 Counter Packet Size Compensation..................................................................................................................343
15.4.1 Reference ...................................................................................................................................................343
15.4.2 Theory of Operation ....................................................................................................................................343
15.4.3 Driver Reference .........................................................................................................................................344
15.4.3.1 APIs ..................................................................................................................................................344
15.5 Counter Drop Reason Filtering and Grouping.................................................................................................345
15.5.1 Reference ...................................................................................................................................................345
15.5.2 Theory of Operation ....................................................................................................................................345
15.5.3 Driver Reference .........................................................................................................................................346
15.5.3.1 APIs ..................................................................................................................................................346
15.6 Counter Handling During Warm Boot ...............................................................................................................348
15.6.1 Theory of Operation ....................................................................................................................................348
15.7 Device-Family Differences .................................................................................................................................348
Chapter 16: Metering ...................................................................................................................... 349
16.1 Theory of Operation............................................................................................................................................349
Broadcom 88690-PG207-PUB
14
BCM88690 Programming Guide Traffic Manager
16.1.1 Mapping Object-ID to Specific Generic Metering Index ..............................................................................350
16.2 Ingress Global Meter...........................................................................................................................................351
16.2.1 Theory of Operation ....................................................................................................................................351
16.2.2 Driver Reference .........................................................................................................................................351
16.2.2.1 APIs ..................................................................................................................................................352
16.2.2.2 SOC Properties.................................................................................................................................353
16.2.2.3 Shell Commands ..............................................................................................................................353
16.2.2.4 Configuration Flow............................................................................................................................354
16.2.2.5 Application Reference.......................................................................................................................354
16.3 Generic Meter Database (Ingress and Egress).................................................................................................354
16.3.1 Theory of Operation ....................................................................................................................................354
16.3.2 Driver Reference .........................................................................................................................................356
16.3.2.1 APIs ..................................................................................................................................................356
16.3.2.2 SOC Properties.................................................................................................................................359
16.3.2.3 Shell Commands ..............................................................................................................................359
16.3.2.4 Configuration Flow............................................................................................................................359
16.3.2.5 Application Reference.......................................................................................................................361
16.4 Meter Properties..................................................................................................................................................361
16.4.1 Theory of Operation ....................................................................................................................................361
16.4.1.1 Metering Modes ................................................................................................................................362
16.4.1.2 Rate Dependencies ..........................................................................................................................362
16.4.1.3 Meter Burst Size ...............................................................................................................................362
16.4.1.4 policer_id Settings ............................................................................................................................362
16.4.2 Driver Reference .........................................................................................................................................362
16.4.2.1 APIs ..................................................................................................................................................363
16.4.2.2 SOC Properties.................................................................................................................................365
16.4.2.3 Shell Commands ..............................................................................................................................366
16.4.2.4 Configuration Flow............................................................................................................................366
16.4.2.5 Application Reference.......................................................................................................................366
16.5 Resolution Tables ...............................................................................................................................................366
16.5.1 Theory of Operation ....................................................................................................................................366
16.5.2 Driver Reference .........................................................................................................................................368
16.5.2.1 SOC Properties.................................................................................................................................370
16.5.2.2 Shell Commands ..............................................................................................................................370
16.5.2.3 Configuration Flow............................................................................................................................370
16.5.2.4 Application Reference.......................................................................................................................371
16.6 Meter Packet Size Adjustment...........................................................................................................................371
16.6.1 Theory of Operation ....................................................................................................................................371
16.6.2 Driver Reference .........................................................................................................................................371
16.6.2.1 APIs ..................................................................................................................................................371
16.7 Meter Pointer Generation ...................................................................................................................................372
Broadcom 88690-PG207-PUB
15
BCM88690 Programming Guide Traffic Manager
16.7.1 Theory of Operation ....................................................................................................................................372
16.7.1.1 Application Reference.......................................................................................................................373
16.8 Device Actions Based on Meter.........................................................................................................................373
16.8.1 Theory of Operation ....................................................................................................................................373
16.9 Count Ingress Meters .........................................................................................................................................374
16.9.1 Theory of Operation ....................................................................................................................................374
16.9.2 Driver Reference .........................................................................................................................................374
16.10 Configure Policer Jittering...............................................................................................................................375
16.11 Policer API Structure Initialization ..................................................................................................................375
16.12 Device-Family Differences ...............................................................................................................................375
Chapter 17: Statistics Interface ..................................................................................................... 376
17.1 Theory of Operation............................................................................................................................................376
17.2 Driver Reference .................................................................................................................................................379
17.2.1 APIs ............................................................................................................................................................379
17.2.2 SOC Properties...........................................................................................................................................385
17.2.3 Shell Commands.........................................................................................................................................389
17.2.4 Configuration Flow ......................................................................................................................................389
17.2.5 Application Reference.................................................................................................................................390
17.3 Statistic Interface Compensation......................................................................................................................391
17.3.1 Theory of Operation ....................................................................................................................................391
17.3.2 Driver Reference .........................................................................................................................................392
17.3.2.1 APIs ..................................................................................................................................................392
17.4 Statistics Interface Using the BCM16K (OP2) ..................................................................................................393
17.4.1 Configuration Flow ......................................................................................................................................395
Chapter 18: Interrupts ....................................................................................................................396
18.1 Theory of Operation............................................................................................................................................396
18.1.1 Interrupt Handling Overview .......................................................................................................................396
18.1.2 Interrupt Handler Calling Sequence............................................................................................................396
18.2 Driver Reference .................................................................................................................................................397
18.2.1 APIs ............................................................................................................................................................397
18.2.2 SOC Properties...........................................................................................................................................400
18.2.3 Shell Commands.........................................................................................................................................402
18.2.4 Application Reference.................................................................................................................................402
18.2.4.1 Application Override Interrupt Handler .............................................................................................402
18.2.4.2 Application Initialization Sequence Reference Code ........................................................................403
18.2.4.3 CMIC Interrupts ................................................................................................................................403
18.2.4.4 Forcing Interrupts..............................................................................................................................403
18.2.4.5 Hard Reset Callback Registration.....................................................................................................404
Chapter 19: Shadowing .................................................................................................................. 405
Broadcom 88690-PG207-PUB
16
BCM88690 Programming Guide Traffic Manager
19.1 Theory of Operation............................................................................................................................................405
19.1.1 Caching (Shadowing) Mechanism ..............................................................................................................405
19.1.2 Memory Protection ......................................................................................................................................405
19.2 Driver Reference .................................................................................................................................................406
19.2.1 APIs ............................................................................................................................................................406
19.2.2 SOC Properties...........................................................................................................................................406
19.2.3 Shell Commands.........................................................................................................................................406
19.2.4 Application Reference.................................................................................................................................406
Appendix A: LED Interface ............................................................................................................ 408
A.1 Overview ...............................................................................................................................................................408
A.2 LED Serial Port ....................................................................................................................................................408
A.3 LED Controller .....................................................................................................................................................409
A.3.1 led_accu ......................................................................................................................................................409
A.3.2 led_ram (bank0) ..........................................................................................................................................411
A.3.3 led_send ......................................................................................................................................................411
A.3.3.1 Generation of led_clk ........................................................................................................................412
A.4 Driver Reference ..................................................................................................................................................412
A.4.1 Diagnostic Shell ...........................................................................................................................................412
Appendix B: S-channel Messaging............................................................................................... 414
B.1 Messaging Mechanism .......................................................................................................................................414
B.2 Messaging Registers ...........................................................................................................................................414
B.3 S-channel Messaging Mechanism .....................................................................................................................415
B.3.1 S-channel Message Source and Destination Blocks ...................................................................................416
B.3.2 S-channel Message Format ........................................................................................................................416
B.3.3 Driver Reference .........................................................................................................................................417
Related Documents ........................................................................................................................ 418
Glossary .......................................................................................................................................... 419
Broadcom 88690-PG207-PUB
17
BCM88690 Programming Guide Traffic Manager
Chapter 1: SDK General Concepts
1.1 BCM APIs
A configuration that can be controlled dynamically is exposed to the user application through the BCM API. The BCM
API is
the software library that configures the BCM88XXX. It offers a functional view of the device architecture,
including a use-model
and allocation services.
NOTE: In this document, the term DNX FAP devices refers to the BCM8869X (Jericho2), BCM8880X (J2C), BCM8848X
(Q2A), and BCM8828X (Q2U)devices. Older FAP devices are referred to by name or called legacy FAP devices.
NOTE: In this document and in the SDK, the BCM8828X device (Q2U) is part of the BCM8848X family. All information in
this document that is related to the BCM8848X also applies to BCM8828X unless explicitly stated otherwise.
DNX FAP devices support a subset of APIs in the BCM API library.
Only the APIs mentioned in this document are supported
by DNX FAP devices. If an API is supported by only a subset of DNX FAP devices, or if a description exists per device type,
the relevant device types are explicitly mentioned.
If a single CPU uses multithreading to control multiple devices, create one thread per device. Each thread is responsible for
calling the APIs for the device that is assigned to it. This allows parallel execution of APIs on different devices, while not
attempting to call APIs in parallel on the same device. The driver internally creates threads that perform certain
device
operations in parallel.
The BCM API is independent of the OS and BSP. This is achieved by using software and hardware abstraction
layers,
described in detail in the Core Switch Software Platform Guide Software Development Kit (SDK-PG8xx).
Every BCM API call passes through a dispatch layer that allows controlling local and remote devices, as well as
devices of
different architectures.
Unless specified otherwise, the BCM API prefix is bcm_<module>_xx, which is declared in the SDK_ROOT/include/bcm/
<module>.h header file.
1.1.1 API Return Values
All BCM APIs return success or error values defined by the bcm_error_e enum in include/bcm/error.h. The return
value of
every call to a BCM API must be checked. A return value different from BCM_E_NONE indicates an API failure. Upon
failure, an error message is dispatched according to the logging configuration as defined in the Broadcom System
Log (BSL).
1.1.2 BCM API Objects – GPORTs
The BCM API is a highly polymorphic API offering a sparse set of high-level operations on generic object types.
The most
important object type is gport (also known as generalized port).
A gport is an object that has some of the following attributes:
It is related to a destination or an intermediate point in the flow of a packet through the system, or its control
path.
It is related to a hardware object that can be attributed with bandwidth, weight, or priority.
It is related to a FIFO with configurable admit/discard decision.
It represents a resource that requires allocation from a resource pool.
Broadcom 88690-PG207-PUB
18
BCM88690 Programming Guide Traffic Manager
Almost any object of interest other than the static translation tables is a gport. A substantial part of a user
application involves
the creation and parameterization of gports, and most of the Driver Reference description
concerns the mapping of user tasks
to gport operations.
The gport object has numerous usages. One subclass of gport objects that is widely used in Traffic
Manager (TM) application
is the family of Class of Service queue (cosq) gports.
Gports are stored in variables of type bcm_gport_t. Values of these variables are gport handles. Gport variables are
usually set in one of two ways:
By an API that creates the object they refer to or just return a handle to an existing object. For example:
bcm_cosq_gport_add()
By macros such as:
BCM_GPORT_MODPORT_SET(gport_variable, module_id, port_id)
Broadcom 88690-PG207-PUB
19
BCM88690 Programming Guide Traffic Manager
1.2 SOC Properties
Some device configurations are static, which means they are set only during device initialization. These static configurations,
as well as the
dynamic attributes typically required for device initialization, are provided in a form of configuration properties,
also known as SOC properties. These properties can be provided to the driver in several ways, for example, in
the form of a
configuration file. For details, refer to the Core Switch Software Platform Guide Software Development Kit (SDK-PG8xx).
If several devices use the same configuration file, it is possible to distinguish the configuration by using a special suffix.
Distinguish by unit ID.
XXX.<unit> is read only by devices with the same unit ID.
For example, fabric_connect_mode.0=FE affects unit ID 0 only.
Distinguish by chip name.
The chip name represents a tuple of device ID and revision ID.
XXX.<chip name> is read only by devices with the same chip name.
For example, fabric_connect_mode.BCM88690_A0=FE affects Jericho2 revision A0 only.
Distinguish by device family name.
Each family of devices is distinguished by their family name.
XXX.<family name> is read only by devices with the same family name.
The device family names are as follows:
– The Jericho2 family name is BCM8869X.
– The J2C family name is BCM8880X.
– The Q2A family name is BCM8848X.
For example, fabric_connect_mode.BCM8869X=FE affects the Jericho2 family (the device ID prefix is 8869).
1.2.1 Default Configuration File
All default SOC properties are located at the following path in the SDK, per device family:
$SDK/rc/config-jr2.bcm
$SDK/rc/config-q2a.bcm
$SDK/rc/config-j2c.bcm
Device-specific configuration files are located at the following path at the SDK per device name:
$SDK/rc/dnx_sku/bcm88693.soc
Broadcom 88690-PG207-PUB
20
BCM88690 Programming Guide Traffic Manager
1.3 Shell Commands
The Broadcom shell is an interactive application running on both Broadcom SDK platforms and Broadcom
reference design
platforms. The BCM shell provides access to registers and memories on the switch devices, provides a means of higher-level
configuration, and includes many helpful diagnostic commands. For a detailed description of the BCM shell application, refer
to the
BCM Shell section in the Core Switch Software Platform Guide Software Development Kit (SDK-PG8xx).
Some of the driver reference sections include a list of shell commands relevant to the specific module, along with a short
description.
For a detailed description of a shell command, type the command in the Broadcom shell followed by the usage keyword.
1.4 Application Reference
Driver reference sections across this document might include references to a CINT script, default application, or tuning
configuration.
1.4.1 CINT Reference
For all major features, application-level code is implemented for customer reference. This code demonstrates
a correct API
calling sequence for each feature. The reference code is delivered as part of the SDK release package and verified for each
SDK release. The code is
implemented in the form of C-interpreter (CINT) code. For details on the CINT interpreter, refer to
Network Switching Software Development Kit CINT Interface for Diagnostic Shell (56XX-PG10xx-R).
CINT example code is located at:
SDK_ROOT/src/examples/sand for CINT files that are supported by both DNX FAP devices and legacy DNX FAP
devices (such as the BCM88670 and BCM88470).
SDK_ROOT/src/examples/dnx for new CINT files that are supported by the DNX FAP devices.
A README.txt file at each of these locations contains lists
of
available CINT examples per supported feature. A detailed
description of each reference scenario appears in the
header of the relevant CINT file.
1.4.2 Default Application Reference
The DNX FAP device SDK includes a sample application.
This application is a reference targeted at the Broadcom system validation kit (SVK). The SVK is a platform for validating
device capabilities and software maturity. In addition, it may also serve as a reference or a template for customer
applications.
The application reference code is located at SDK_ROOT/src/appl/reference/*.
For a more detailed overview of the default application reference, see Section 2.2.5, Application Reference.
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
/