2. Computers & electronics
  3. Software
  4. Dell
  5. Storage SC5020F
Storage SC5020F

Dell Storage SC5020F User guide

  • Hello! I am an AI chatbot trained to assist you with the Dell Storage SC5020F User guide. I’ve already reviewed the document and can help you find the information you need or explain it in simple terms. Just ask your questions, and providing more details will help me assist you more effectively!
Storage Manager 2018 R1
Administrator’s Guide
Notes, cautions, and warnings
NOTE: A NOTE indicates important information that helps you make better use of your product.
CAUTION: A CAUTION indicates either potential damage to hardware or loss of data and tells you how to avoid the problem.
WARNING: A WARNING indicates a potential for property damage, personal injury, or death.
© 2017 – 2018 Dell Inc. or its subsidiaries. All rights reserved. Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other
trademarks may be trademarks of their respective owners.
2018 – 11
Rev. C
Contents
About This Guide............................................................................................................................................ 23
How to Find Information................................................................................................................................................. 23
Contacting Dell.................................................................................................................................................................23
Revision History................................................................................................................................................................23
Audience............................................................................................................................................................................24
Related Publications.........................................................................................................................................................24
Storage Manager Documents.................................................................................................................................. 24
Storage Center Documents...................................................................................................................................... 24
FluidFS Cluster Documents...................................................................................................................................... 25
Dell TechCenter..........................................................................................................................................................25
1 Storage Manager Overview.......................................................................................................................... 27
Management Compatibility.............................................................................................................................................27
Software and Hardware Requirements......................................................................................................................... 28
Data Collector Requirements....................................................................................................................................28
Storage Manager Client Requirements................................................................................................................... 29
Server Agent Requirements..................................................................................................................................... 30
Default Ports Used by Storage Manager......................................................................................................................30
Data Collector Ports.................................................................................................................................................. 30
Client Ports..................................................................................................................................................................31
Server Agent Ports.................................................................................................................................................... 32
IPv6 Support.................................................................................................................................................................... 32
Storage Manager Features............................................................................................................................................. 33
Storage Management Features................................................................................................................................33
Disaster Recovery Features......................................................................................................................................34
Monitoring and Reporting Features.........................................................................................................................35
Storage Manager Client Overview................................................................................................................................ 36
2 Getting Started............................................................................................................................................39
Use the Client to Connect to the Data Collector.........................................................................................................39
Next Steps.........................................................................................................................................................................41
Add Storage Manager Users..................................................................................................................................... 41
Add Storage Centers to Storage Manager..............................................................................................................41
Conî‚‚gure Storage Center Volumes......................................................................................................................... 42
Add Servers to the Storage Centers....................................................................................................................... 42
Add PS Groups to Storage Manager.......................................................................................................................42
Add FluidFS Clusters to Storage Manager............................................................................................................. 42
Conî‚‚gure Email Notiî‚‚cations................................................................................................................................... 42
Set up Remote Storage Centers and Relication QoS............................................................................................42
Conî‚‚gure Replications and Live Volumes............................................................................................................... 43
Prepare for Disaster Recovery................................................................................................................................. 43
3 Storage Center Overview............................................................................................................................ 45
Contents
3
How Storage Virtualization Works................................................................................................................................. 45
Storage Center Hardware Components..................................................................................................................45
Disk Management......................................................................................................................................................46
Drive Spares................................................................................................................................................................47
Volumes....................................................................................................................................................................... 47
Storage Types.............................................................................................................................................................48
Data Progression........................................................................................................................................................49
Low Space Modes..................................................................................................................................................... 50
Storage Center Operation Modes............................................................................................................................ 51
Storage Proî‚‚les...........................................................................................................................................................51
User Interface for Storage Center Management.........................................................................................................54
Summary Tab..............................................................................................................................................................55
Storage Tab.................................................................................................................................................................56
Hardware Tab..............................................................................................................................................................57
IO Usage Tab.............................................................................................................................................................. 58
Charting Tab............................................................................................................................................................... 58
Alerts Tab.................................................................................................................................................................... 59
Logs Tab......................................................................................................................................................................59
4 Storage Center Deployment......................................................................................................................... 61
Supported Operating Systems for Storage Center Automated Setup ..................................................................... 61
Discover and Conî‚‚gure Uninitialized SCv2000 Series Storage Centers................................................................... 61
Open the Discover and Conî‚‚gure Uninitialized Storage Centers Wizard From the Storage Manager
Client Welcome Screen..............................................................................................................................................61
Open the Discover and Conî‚‚gure Uninitialized Storage Centers Wizard from the Storage Manager
Client........................................................................................................................................................................... 62
Select a Storage Center to Initialize........................................................................................................................ 62
Set System Information............................................................................................................................................ 63
Set Administrator Information.................................................................................................................................. 63
Conî‚‚gure iSCSI Fault Domains (iSCSI Front-End Only).......................................................................................63
Conî‚‚rm the Storage Center Conî‚‚guration............................................................................................................ 64
Deploy the Storage Center....................................................................................................................................... 64
Review Redundant Paths (FC or SAS Front-End Only)....................................................................................... 64
Inherit Settings...........................................................................................................................................................64
Conî‚‚gure Time Settings........................................................................................................................................... 65
Conî‚‚gure SMTP Server Settings............................................................................................................................ 65
Set Up SupportAssist ...............................................................................................................................................65
Accept the SupportAssist Collection Agreement.................................................................................................. 66
Provide Contact Information.................................................................................................................................... 66
Provide Site Contact Information............................................................................................................................ 66
Validate SupportAssist Connection......................................................................................................................... 66
Update Storage Center............................................................................................................................................. 67
Set Default Storage Proî‚‚le (SCv2000 Series Controllers Only).......................................................................... 67
Complete Conî‚‚guration and Perform Next Steps.................................................................................................68
Discover and Conî‚‚gure Uninitialized SCv3000 Series Storage Centers ................................................................. 69
4
Contents
Open the Discover and Conî‚‚gure Uninitialized Storage Centers Wizard From the Storage Manager
Client Welcome Screen.............................................................................................................................................69
Open the Discover and Conî‚‚gure Uninitialized Storage Centers Wizard from the Storage Manager
Client........................................................................................................................................................................... 69
Select a Storage Center to Initialize........................................................................................................................ 69
Welcome Page............................................................................................................................................................70
Customer Installation Authorization.........................................................................................................................70
Set System Information.............................................................................................................................................70
Set Administrator Information................................................................................................................................... 71
Conî‚‚rm the Storage Center Conî‚‚guration............................................................................................................. 71
Deploy the Storage Center........................................................................................................................................71
Enter Key Management Server Settings................................................................................................................. 71
Create a Storage Type...............................................................................................................................................72
Fault Tolerance........................................................................................................................................................... 72
Inherit Settings........................................................................................................................................................... 74
Conî‚‚gure Time Settings............................................................................................................................................74
Conî‚‚gure SMTP Server Settings.............................................................................................................................74
Set Up SupportAssist ...............................................................................................................................................75
Accept the SupportAssist Collection Agreement.................................................................................................. 75
Provide Contact Information.................................................................................................................................... 75
Provide Site Contact Information.............................................................................................................................75
Validate SupportAssist Connection..........................................................................................................................76
Update Storage Center............................................................................................................................................. 76
Complete Conî‚‚guration and Perform Next Steps.................................................................................................76
Discover and Conî‚‚gure Uninitialized SC5020 and SC7020 Storage Centers.......................................................... 77
Open the Discover and Conî‚‚gure Uninitialized Storage Centers Wizard From the Storage Manager
Client Welcome Screen..............................................................................................................................................77
Open the Discover and Conî‚‚gure Uninitialized Storage Centers Wizard from the Storage Manager
Client............................................................................................................................................................................77
Introduction.................................................................................................................................................................77
Select a Storage Center to Initialize.........................................................................................................................78
Welcome Page............................................................................................................................................................78
Customer Installation Authorization.........................................................................................................................79
Set System Information.............................................................................................................................................79
Set Administrator Information.................................................................................................................................. 79
Conî‚‚rm the Storage Center Conî‚‚guration............................................................................................................ 79
Deploy the Storage Center.......................................................................................................................................80
Enter Key Management Server Settings................................................................................................................ 80
Create a Storage Type...............................................................................................................................................80
Fault Tolerance............................................................................................................................................................81
Inherit Settings...........................................................................................................................................................83
Conî‚‚gure Time Settings........................................................................................................................................... 83
Conî‚‚gure SMTP Server Settings............................................................................................................................ 83
Set Up SupportAssist ...............................................................................................................................................83
Accept the SupportAssist Collection Agreement.................................................................................................. 84
Contents
5
Provide Contact Information.................................................................................................................................... 84
Provide Site Contact Information............................................................................................................................ 84
Validate SupportAssist Connection......................................................................................................................... 85
Complete Conî‚‚guration and Perform Next Steps.................................................................................................85
Conî‚‚guring SC9000 Storage Centers.......................................................................................................................... 85
Welcome Page............................................................................................................................................................86
Set System Information............................................................................................................................................ 86
Submit the Storage Center License........................................................................................................................ 86
Create a Disk Folder...................................................................................................................................................87
Deploy the Storage Center – Add a Controller.......................................................................................................87
Deploy the Storage Center....................................................................................................................................... 87
Enter Key Management Server Settings.................................................................................................................87
Create a Storage Type...............................................................................................................................................88
Conî‚‚gure Ports..........................................................................................................................................................88
Inherit Settings...........................................................................................................................................................90
Conî‚‚gure Time Settings........................................................................................................................................... 90
Conî‚‚gure SMTP Server Settings............................................................................................................................ 90
Set Up SupportAssist ................................................................................................................................................91
Accept the SupportAssist Collection Agreement...................................................................................................91
Provide Contact Information.....................................................................................................................................91
Provide Site Contact Information............................................................................................................................. 91
Validate SupportAssist Connection......................................................................................................................... 92
Complete Conî‚‚guration and Perform Next Steps.................................................................................................92
Conî‚‚guring SC4020 and SC8000 Storage Centers................................................................................................... 92
Welcome Page............................................................................................................................................................93
Submit the Storage Center License........................................................................................................................ 93
Create a Disk Folder...................................................................................................................................................93
Add a Controller (Conî‚‚gure Storage Center Wizard)........................................................................................... 94
Set System Information............................................................................................................................................ 94
Enter Key Management Server Settings................................................................................................................ 94
Conî‚‚gure Ports..........................................................................................................................................................95
Inherit Settings........................................................................................................................................................... 97
Conî‚‚gure Time Settings............................................................................................................................................97
Conî‚‚gure SMTP Server Settings............................................................................................................................ 97
Accept the SupportAssist Collection Agreement...................................................................................................97
Provide Contact Information.................................................................................................................................... 98
Provide Site Contact Information............................................................................................................................ 98
Validate SupportAssist Connection......................................................................................................................... 98
Complete Conî‚‚guration and Perform Next Steps.................................................................................................98
Set Up a localhost or VMware Host..............................................................................................................................99
Set Up a localhost from Initial Setup.......................................................................................................................99
Set Up a VMware Host from Initial Setup.............................................................................................................100
Set Up a VMware vCenter Host from Initial Setup...............................................................................................101
5 Storage Center Administration................................................................................................................... 103
Adding and Organizing Storage Centers..................................................................................................................... 103
6
Contents
Storage Center User Privileges and User Groups................................................................................................ 103
User Privilege Levels................................................................................................................................................103
Adding and Removing Storage Centers................................................................................................................ 104
Organizing Storage Centers....................................................................................................................................106
Managing Volumes......................................................................................................................................................... 108
Attributes That Determine Volume Behavior........................................................................................................ 108
Volume Icons.............................................................................................................................................................108
Creating Volumes..................................................................................................................................................... 109
Modifying Volumes....................................................................................................................................................113
Copying Volumes.......................................................................................................................................................118
Migrating Volumes With Live Migrate....................................................................................................................120
Creating and Managing Volume Folders................................................................................................................ 127
Creating and Managing Volume Snapshots.......................................................................................................... 128
Mapping Volumes to Servers..................................................................................................................................132
Deleting Volumes and Volume Folders...................................................................................................................135
Managing Data Reduction.............................................................................................................................................136
Data Eligible for Data Reduction............................................................................................................................. 137
Supported Hardware Platforms.............................................................................................................................. 137
Compression............................................................................................................................................................. 138
Deduplication.............................................................................................................................................................138
View Amount of Space Saved by Data Reduction...............................................................................................139
Change the Default Data Reduction Proî‚‚le..........................................................................................................139
Pause or Resume Data Reduction..........................................................................................................................140
Disable Data Reduction for a Volume......................................................................................................................141
Managing Snapshot Proî‚‚les.......................................................................................................................................... 141
Default Snapshot Proî‚‚les.........................................................................................................................................141
Consistent and Non-Consistent Snapshot Proî‚‚les...............................................................................................141
Creating and Applying Snapshot Proî‚‚les.............................................................................................................. 142
Modifying Snapshot Proî‚‚les................................................................................................................................... 144
Managing Expiration Rules for Remote Snapshots..............................................................................................145
Managing Storage Proî‚‚les............................................................................................................................................ 146
Create a Storage Proî‚‚le (Storage Center 7.2.1 and Earlier)................................................................................ 146
Create a Storage Proî‚‚le (Storage Center 7.2.10 and Later)................................................................................146
Apply a Storage Proî‚‚le to One or More Volumes................................................................................................. 147
Apply a Storage Proî‚‚le to a Server........................................................................................................................ 147
Delete a Storage Proî‚‚le........................................................................................................................................... 147
Managing QoS Proî‚‚les.................................................................................................................................................. 148
Create a QoS Proî‚‚le.................................................................................................................................................148
Edit a QoS Proî‚‚le..................................................................................................................................................... 148
Delete a QoS Volume Proî‚‚le................................................................................................................................... 149
Apply a QoS Proî‚‚le to a Volume.............................................................................................................................149
Remove a Group QoS Proî‚‚le From a Volume.......................................................................................................149
Importing Volumes from an External Storage Array...................................................................................................149
Connect to an External Storage Array (iSCSI) ....................................................................................................150
PS Series Storage Array Import Requirements.....................................................................................................150
Contents
7
Storage Center Import Requirements....................................................................................................................150
PowerVault MD Series Import Requirements........................................................................................................ 151
Supported Server Operating Systems for Online Import..................................................................................... 151
Import Data from an External Device (Oine)......................................................................................................151
Import Data from an External Device (Online)......................................................................................................152
6 Storage Center Server Administration........................................................................................................153
Server Management Options........................................................................................................................................153
Managing Storage Center Server Objects............................................................................................................153
Managing Servers Centrally Using Storage Manager..........................................................................................154
Managing Servers on a Storage Center...................................................................................................................... 154
Creating Servers.......................................................................................................................................................155
Modifying Servers.................................................................................................................................................... 160
Mapping Volumes to Servers..................................................................................................................................163
Creating and Managing Server Folders................................................................................................................. 166
Deleting Servers and Server Folders......................................................................................................................167
Managing Servers Centrally on the Servers View......................................................................................................167
Server Types That Can Be Centrally Managed.....................................................................................................167
Storage Manager Server Agent for Windows Servers........................................................................................ 168
IPMI Support for NAS Appliances..........................................................................................................................168
Registering Servers with Storage Manager.......................................................................................................... 168
Organizing and Removing Registered Servers......................................................................................................170
Updating Server Information...................................................................................................................................172
Managing Server Data Collection and Reporting Settings..................................................................................173
Creating Server Volumes and Datastores .............................................................................................................173
Assigning/Creating Virtual Servers on Storage Centers..................................................................................... 176
Manually Mapping a Windows Server to a Storage Center Server....................................................................177
Managing NAS Appliances Powered by Windows Storage Server.................................................................... 178
Installing and Using the Server Agent on a Windows Server................................................................................... 180
Download the Server Agent....................................................................................................................................180
Install and Register the Server Agent.................................................................................................................... 180
Manage the Server Agent with Server Agent Manager......................................................................................182
Uninstalling the Server Agent................................................................................................................................. 183
7 Managing Virtual Volumes With Storage Manager...................................................................................... 185
Conî‚‚guring VVols in Storage Manager........................................................................................................................185
Requirements and Recommendations for Conî‚‚guring VVols in Storage Manager..........................................185
Safeguarding VVols Data......................................................................................................................................... 186
VMware Virtual Volume Concepts............................................................................................................................... 186
Setting Up VVols Operations on Storage Manager....................................................................................................187
Virtual Volumes Restrictions................................................................................................................................... 188
VASA Provider................................................................................................................................................................ 188
VASA Provider Restrictions.....................................................................................................................................189
Register the VASA Provider....................................................................................................................................189
Unregister a VASA Provider.................................................................................................................................... 189
Using Storage Manager Certiî‚‚cates With VASA Provider..................................................................................190
8
Contents
Managing Storage Containers....................................................................................................................................... 191
How Storage Container Options Aî‚„ect vCenter Advertised Capabilities..........................................................191
Data Reduction Options for VVols..........................................................................................................................192
Create a Storage Container Using the Storage View.......................................................................................... 195
Edit a Storage Container Using the Storage View............................................................................................... 196
Delete a Storage Container.....................................................................................................................................196
View Storage Container Information......................................................................................................................196
Creating VVol Datastores...............................................................................................................................................197
Create a Datastore or Storage Container and Map it to VMware vSphere ........................................................... 197
View VVol and Datastore Information..........................................................................................................................198
Protocol Endpoint Monitoring.......................................................................................................................................198
8 PS Series Storage Array Administration..................................................................................................... 201
About Groups..................................................................................................................................................................201
Adding PS Series Groups........................................................................................................................................202
Organizing PS Series Groups................................................................................................................................. 204
Remove a PS Series Group.................................................................................................................................... 205
Launch Group Manager..........................................................................................................................................205
About Volumes...............................................................................................................................................................205
Create a Volume.......................................................................................................................................................207
Modify a Volume...................................................................................................................................................... 208
Create a Volume Folder...........................................................................................................................................209
Edit a Volume Folder............................................................................................................................................... 209
Delete a Volume Folder........................................................................................................................................... 209
Move a Volume to a Folder..................................................................................................................................... 210
Move Multiple Volumes to a Folder........................................................................................................................210
Rename a Volume.....................................................................................................................................................210
Clone a Volume..........................................................................................................................................................211
Modify Volume Access Settings..............................................................................................................................211
Set a Volume Online or Oine.................................................................................................................................211
Add Access Policy Groups to a Volume................................................................................................................. 212
Add Access Policies to a Volume............................................................................................................................212
Create a Basic Access Point................................................................................................................................... 212
Delete a Volume........................................................................................................................................................ 213
Restore a Volume from the Recycle Bin................................................................................................................ 213
Empty the Recycle Bin.............................................................................................................................................213
Permanently Delete a Volume in the Recycle Bin.................................................................................................213
About Snapshots............................................................................................................................................................ 214
Create a Snapshot....................................................................................................................................................214
Create a Snapshot Schedule...................................................................................................................................214
Modify Snapshot Properties................................................................................................................................... 215
Control Snapshot Space Borrowing ......................................................................................................................215
Set a Snapshot Online or Oine............................................................................................................................ 216
Restore a Volume from a Snapshot........................................................................................................................216
Delete a Snapshot.................................................................................................................................................... 216
Managing Replication Schedules..................................................................................................................................217
Contents
9
Create an Hourly Replication Schedule..................................................................................................................217
Create a Daily Replication Schedule....................................................................................................................... 217
Schedule a Replication to Run Once......................................................................................................................218
Edit a Replication Schedule.....................................................................................................................................218
Enable or Disable a Replication Schedule..............................................................................................................219
Delete a Replication Schedule.................................................................................................................................219
About Access Policies....................................................................................................................................................219
Create a Local CHAP Account...............................................................................................................................220
Edit a Local CHAP Account....................................................................................................................................220
Modify Target Authentication.................................................................................................................................220
Set the iSCSI Discovery Filter.................................................................................................................................221
Create an Access Policy Group...............................................................................................................................221
Edit an Access Policy Group................................................................................................................................... 221
Delete an Access Policy Group.............................................................................................................................. 222
Create an Access Policy..........................................................................................................................................222
Edit an Access Policy...............................................................................................................................................223
Delete an Access Policy.......................................................................................................................................... 225
Monitoring a PS Series Group......................................................................................................................................225
View Logs................................................................................................................................................................. 225
View Event Logs...................................................................................................................................................... 225
View Audit Logs....................................................................................................................................................... 225
View Outbound Replications.................................................................................................................................. 226
View Inbound Replications......................................................................................................................................226
View Replication History......................................................................................................................................... 226
View Alerts................................................................................................................................................................226
9 Storage Center Maintenance..................................................................................................................... 227
Managing Storage Center Settings............................................................................................................................. 227
Viewing and Modifying Storage Center Information........................................................................................... 227
Modifying Storage Center Network Settings.......................................................................................................229
Conî‚‚guring Storage Center User Preferences.................................................................................................... 230
Conî‚‚guring Storage Center Data Settings...........................................................................................................233
Conî‚‚guring Storage Center Secure Console Settings........................................................................................235
Conî‚‚guring Storage Center SMTP Settings........................................................................................................ 237
Conî‚‚guring Storage Center SNMP Settings....................................................................................................... 238
Conî‚‚guring Storage Center Time Settings.......................................................................................................... 240
Conî‚‚guring Filters to Restrict Administrative Access.........................................................................................240
Conî‚‚guring a Storage Center to Inherit Settings...................................................................................................... 243
Managing Storage Center Users and Groups............................................................................................................ 243
User Privilege Levels................................................................................................................................................243
User Groups..............................................................................................................................................................243
User Account Management and Authentication................................................................................................. 244
Managing Local Storage Center Users................................................................................................................. 244
Managing Local Storage Center User Groups......................................................................................................249
Enabling Directory Services Authentication......................................................................................................... 252
Managing Directory Service Users........................................................................................................................255
10
Contents
Managing Directory User Groups.......................................................................................................................... 259
Managing Local Storage Center User Password Requirements.........................................................................261
Managing Front-End I/O Ports....................................................................................................................................263
Front-End Connectivity Modes..............................................................................................................................263
About Fault Domains and Ports............................................................................................................................. 264
Failover Behavior..................................................................................................................................................... 266
Rebalancing Front-End Ports.................................................................................................................................266
Managing Front-End I/O Port Hardware..............................................................................................................266
Converting Front-End Ports to Virtual Port Mode...............................................................................................271
Grouping Fibre Channel I/O Ports Using Fault Domains...........................................................................................272
Create a Fibre Channel Fault Domain.................................................................................................................... 272
Rename a Fibre Channel Fault Domain................................................................................................................. 272
Delete a Fibre Channel Fault Domain.....................................................................................................................273
Grouping iSCSI I/O Ports Using Fault Domains......................................................................................................... 273
iSCSI VLAN Tagging Support................................................................................................................................. 273
Creating iSCSI Fault Domains.................................................................................................................................274
Modifying iSCSI Fault Domains.............................................................................................................................. 276
Conî‚‚guring NAT Port Forwarding for iSCSI Fault Domains...............................................................................280
Conî‚‚guring CHAP for iSCSI Fault Domains......................................................................................................... 283
Grouping SAS I/O Ports Using Fault Domains...........................................................................................................285
Create a SAS Fault Domain.................................................................................................................................... 285
Delete a SAS Fault Domain.....................................................................................................................................285
Managing Disks and Disk Folders................................................................................................................................ 285
Storage Center Disk Management........................................................................................................................ 286
Scan for New Disks................................................................................................................................................. 286
Create a Disk Folder................................................................................................................................................ 286
Delete Disk Folder.................................................................................................................................................... 287
Modify a Disk Folder................................................................................................................................................ 287
Manage Unassigned Disks......................................................................................................................................288
Enable or Disable the Disk Indicator Light............................................................................................................ 288
Release a Disk.......................................................................................................................................................... 288
Cancel Releasing a Disk.......................................................................................................................................... 289
Delete a Disk.............................................................................................................................................................289
Restore a Disk.......................................................................................................................................................... 290
Replace a Failed Disk...............................................................................................................................................290
Managing Secure Data.................................................................................................................................................. 291
How Secure Data Works......................................................................................................................................... 291
Conî‚‚gure Key Server...............................................................................................................................................291
Conî‚‚gure Rekey Interval for Disk Folder.............................................................................................................. 292
Rekey a Disk Folder................................................................................................................................................. 292
Rekey a Disk ............................................................................................................................................................ 293
Copy Volumes to Disk Folder..................................................................................................................................293
Create Secure Data Disk Folder.............................................................................................................................293
Managing Data Redundancy........................................................................................................................................ 294
Redundancy Requirements.....................................................................................................................................294
Contents
11
Managing RAID........................................................................................................................................................ 295
Managing Storage Types........................................................................................................................................ 296
Managing Disk Enclosures............................................................................................................................................ 297
Add an Enclosure..................................................................................................................................................... 297
Remove an Enclosure..............................................................................................................................................298
Replace an Enclosure.............................................................................................................................................. 299
Rename a Disk Enclosure........................................................................................................................................299
Set an Asset Tag for a Disk Enclosure.................................................................................................................. 300
Delete an Enclosure.................................................................................................................................................300
Mute an Enclosure Alarm........................................................................................................................................300
Unmute an Enclosure Alarm....................................................................................................................................301
Clear the Swap Status for an Enclosure Cooling Fan..........................................................................................301
Clear the Swap Status for an Enclosure I/O Module...........................................................................................301
Clear the Swap Status for an Enclosure Power Supply.......................................................................................301
Replace a Failed Power Supply.............................................................................................................................. 302
Clear the Under Voltage Status for a Power Supply...........................................................................................302
Clear the Swap Status for a Temperature Sensor...............................................................................................302
Clear the Minimum and Maximum Recorded Values for Temperature Sensor................................................ 303
Replace a Failed Cooling Fan Sensor.................................................................................................................... 303
Enable or Disable the Disk Indicator Light............................................................................................................ 303
Clear the Swap Status for a Disk...........................................................................................................................304
Managing Storage Center Controllers........................................................................................................................ 304
Add a Controller....................................................................................................................................................... 304
Replace a Failed Controller..................................................................................................................................... 304
Enable or Disable a Controller Indicator Light...................................................................................................... 305
Replace a Failed Cooling Fan Sensor.................................................................................................................... 305
Conî‚‚gure Back-End Ports......................................................................................................................................306
Managing I/O Card Changes................................................................................................................................. 306
Add a UPS to a Storage Center.............................................................................................................................307
Updating Storage Center..............................................................................................................................................308
Update Storage Center Software..........................................................................................................................308
Using the Storage Center Update Utility..............................................................................................................308
Shutting Down and Restarting a Storage Center......................................................................................................309
Shut Down All Controllers in Storage Center.......................................................................................................309
Restart All Controllers in a Storage Center........................................................................................................... 310
Shut Down a Controller........................................................................................................................................... 310
Restart a Controller..................................................................................................................................................310
Reset a Controller to Factory Default.....................................................................................................................311
Managing Field Replaceable Units (FRU).................................................................................................................... 311
Managing FRU Tickets............................................................................................................................................. 311
10 Viewing Storage Center Information......................................................................................................... 313
Viewing Summary Information......................................................................................................................................313
Storage Center Summary Plugins.......................................................................................................................... 313
Viewing Summary Information for a Storage Center...........................................................................................314
Viewing Summary Information for Multiple Storage Centers............................................................................. 315
12
Contents
Using the Status Plugin........................................................................................................................................... 316
Using the Storage Summary Plugin....................................................................................................................... 318
Using the Front End IO Summary Plugin..............................................................................................................320
Using the Current Alerts Plugin.............................................................................................................................. 321
Using the Replication Validation Plugin................................................................................................................. 322
Using the Top 10 Fastest Growing Volumes Plugin............................................................................................. 322
Using the Current Threshold Alerts Plugin........................................................................................................... 323
Viewing Detailed Storage Usage Information.............................................................................................................323
View Storage Usage by Tier and RAID Type.........................................................................................................323
View Storage Usage by Volumes........................................................................................................................... 324
View Historical Storage Usage...............................................................................................................................325
View a Data Progression Pressure Report............................................................................................................325
Viewing Historical IO Performance.............................................................................................................................. 327
Using the IO Usage Tab...........................................................................................................................................327
Viewing Current IO Performance.................................................................................................................................329
Using the Charting Tab........................................................................................................................................... 329
Conî‚‚guring Chart Options............................................................................................................................................ 331
Conî‚‚guring User Settings for Charts.................................................................................................................... 331
Conî‚‚guring Chart Settings.....................................................................................................................................332
Conî‚‚gure the Storage Center Data Gathering Schedule......................................................................................... 333
Exporting Usage Data................................................................................................................................................... 333
Export Storage Usage Data....................................................................................................................................333
Export I/O Usage Data........................................................................................................................................... 334
Monitoring Storage Center Hardware.........................................................................................................................336
Monitoring a Storage Center Controller................................................................................................................336
Monitoring a Storage Center Disk Enclosure....................................................................................................... 339
Monitoring SSD Endurance.....................................................................................................................................341
Viewing UPS Status................................................................................................................................................ 343
11 SMI-S....................................................................................................................................................... 345
SMI-S Provider.............................................................................................................................................................. 345
Supported Management Solutions........................................................................................................................345
Supported SMI-S 1.6 Proî‚‚les................................................................................................................................. 345
SMI-S Namespace...................................................................................................................................................346
Setting Up SMI-S.......................................................................................................................................................... 346
SMI-S Prerequisites.................................................................................................................................................346
Enable SMI-S for the Data Collector.....................................................................................................................346
Using the Dell SMI-S Provider with Microsoft SCVMM...........................................................................................347
SCVMM Prerequisites............................................................................................................................................. 347
Limitations for SCVMM 2012................................................................................................................................. 348
Modify the SCVMM 2012 Management Server Registry to Allow HTTPS......................................................348
Prepare the SCVMM 2012 Server for Indications............................................................................................... 349
Use SCVMM to Discover the SMI-S Provider.....................................................................................................349
12 FluidFS Administration.............................................................................................................................. 351
How FS8600 Scale-Out NAS Works........................................................................................................................... 351
Contents
13
FS8600 Scale-Out NAS Terminology....................................................................................................................352
Key Features of the Scale-Out NAS..................................................................................................................... 353
Overview of the FS8600 Hardware...................................................................................................................... 354
Overview of the FS8600 Architecture................................................................................................................. 355
Data Caching and Redundancy..............................................................................................................................356
File Metadata Protection........................................................................................................................................ 356
Load Balancing and High Availability......................................................................................................................357
Ports Used by the FluidFS Cluster........................................................................................................................ 358
FluidFS System Management for FS Series Appliances...........................................................................................358
NAS Access..............................................................................................................................................................358
Management Access...............................................................................................................................................358
Seamless Session Failover......................................................................................................................................358
Using the Storage Manager Client or CLI to Connect to the FluidFS Cluster.................................................358
Managing Secured Management.......................................................................................................................... 360
Managing the FluidFS Cluster Name.................................................................................................................... 363
Accept the End-User License Agreement............................................................................................................364
Managing the System Time....................................................................................................................................364
Managing the FTP Server...................................................................................................................................... 365
Managing SNMP..................................................................................................................................................... 366
Managing the Health Scan Throttling Mode........................................................................................................368
Managing the Operation Mode..............................................................................................................................368
Managing Client Connections................................................................................................................................369
Shutting Down and Restarting NAS Controllers...................................................................................................371
Managing NAS Appliance and NAS Controller Blinking.......................................................................................372
FluidFS Networking....................................................................................................................................................... 373
Managing the Default Gateway..............................................................................................................................373
Managing DNS Servers and Suî‚…xes.....................................................................................................................373
Managing Static Routes..........................................................................................................................................375
Managing the Client Networks.............................................................................................................................. 376
Viewing the Fibre Channel WWNs........................................................................................................................ 380
Managing iSCSI SAN Connectivity........................................................................................................................ 381
Add or Remove an iSCSI Port.................................................................................................................................381
Add or Remove an iSCSI Fabric..............................................................................................................................381
Change the VLAN Tag for an iSCSI Fabric...........................................................................................................382
Change the NAS Controller IP Addresses for an iSCSI Fabric...........................................................................382
FluidFS Account Management and Authentication...................................................................................................382
Account Management and Authentication...........................................................................................................382
Default Administrative Accounts............................................................................................................................383
Default Local User and Local Group Accounts.................................................................................................... 385
Managing Administrator Accounts........................................................................................................................ 385
Managing Local Users and Groups Using MMC..................................................................................................388
Managing Local Users.............................................................................................................................................389
Managing Local Groups........................................................................................................................................... 391
Managing Active Directory..................................................................................................................................... 394
Managing LDAP.......................................................................................................................................................396
14
Contents
Managing NIS...........................................................................................................................................................399
Managing User Mappings Between Windows and UNIX/Linux Users..............................................................401
FluidFS NAS Volumes, Shares, and Exports...............................................................................................................403
Managing the NAS Pool..........................................................................................................................................403
View Internal Storage Reservations...................................................................................................................... 404
View the Size of the NAS Pool.............................................................................................................................. 404
Expand the Size of the NAS Pool..........................................................................................................................404
Set the Metadata Tier.............................................................................................................................................404
Enable or Disable the NAS Pool Used Space Alert..............................................................................................405
Enable or Disable the NAS Pool Unused Space Alert......................................................................................... 405
About Multitenancy.................................................................................................................................................406
Create a New Tenant.............................................................................................................................................. 409
Moving a NAS Volume Between Tenants..............................................................................................................410
Managing NAS Volumes...........................................................................................................................................411
Managing SMB Shares............................................................................................................................................422
Managing NFS Exports...........................................................................................................................................433
Global Namespace................................................................................................................................................... 437
Using FTP................................................................................................................................................................. 438
Using Symbolic Links...............................................................................................................................................439
Managing Quota Rules............................................................................................................................................439
About Data Reduction.............................................................................................................................................440
FluidFS Data Protection................................................................................................................................................443
Managing Antivirus..................................................................................................................................................443
Managing Snapshots...............................................................................................................................................444
Managing NDMP..................................................................................................................................................... 449
Managing Replication..............................................................................................................................................458
File Access Notiî‚‚cation.................................................................................................................................................472
FluidFS Monitoring.........................................................................................................................................................472
Monitoring NAS Appliance Hardware....................................................................................................................472
FluidFS Maintenance.....................................................................................................................................................475
Connecting Multiple Data Collectors to the Same Cluster.................................................................................475
Adding and Removing FluidFS Clusters in Storage Manager.............................................................................476
Organizing FluidFS Clusters Using Folders........................................................................................................... 477
Adding a Storage Center to a FluidFS Cluster..................................................................................................... 478
Adding and Deleting NAS Appliances in a FluidFS Cluster................................................................................. 479
Detaching, Attaching, and Replacing a NAS Controller....................................................................................... 481
Managing Service Packs.........................................................................................................................................483
Managing Firmware Updates................................................................................................................................. 485
Update Firmware Using FTP.................................................................................................................................. 485
Restoring the NAS Volume Conî‚‚guration............................................................................................................ 485
Restoring Local Users............................................................................................................................................. 486
Restoring Local Groups........................................................................................................................................... 487
Reinstalling FluidFS from the Internal Storage Device........................................................................................ 488
FS Series VAAI Plugin................................................................................................................................................... 489
Enable or Disable the FS Series VAAI Plugin........................................................................................................489
Contents
15
Installation Instructions........................................................................................................................................... 489
Plugin Veriî‚‚cation....................................................................................................................................................490
Removal Instructions...............................................................................................................................................490
FluidFS Troubleshooting................................................................................................................................................ 491
Viewing the Event Log.............................................................................................................................................491
Running Diagnostics................................................................................................................................................492
Conî‚‚guring the BMC Network..............................................................................................................................494
Launching the iBMC Virtual KVM..........................................................................................................................494
Troubleshooting Common Issues...........................................................................................................................495
13 Remote Storage Centers and Replication QoS.......................................................................................... 513
Connecting to Remote Storage Centers..................................................................................................................... 513
Connecting Storage Centers Using Fibre Channel...............................................................................................513
Connecting Storage Centers Using iSCSI............................................................................................................. 513
Creating and Managing Replication Quality of Service Deî‚‚nitions.......................................................................... 515
Create a QoS Deî‚‚nition...........................................................................................................................................515
Rename a QoS Deî‚‚nition........................................................................................................................................ 516
Change the Link Speed for a QoS Deî‚‚nition........................................................................................................ 516
Enable or Disable Bandwidth Limiting for a QoS Deî‚‚nition................................................................................ 516
Modify the Bandwidth Limit Schedule for a QoS Deî‚‚nition............................................................................... 516
Delete a QoS Deî‚‚nition............................................................................................................................................517
14 Storage Center Replications and Live Volumes..........................................................................................519
Storage Center Replications......................................................................................................................................... 519
Replication Types.....................................................................................................................................................520
Replication Requirements........................................................................................................................................521
Replication Behavior When a Destination Volume Fails.......................................................................................521
Replicating a Single Volume to Multiple Destinations.......................................................................................... 521
Replication on SCv2000 Series Storage Systems...............................................................................................522
Replication Icons......................................................................................................................................................523
Simulating Replications........................................................................................................................................... 523
Replicating Volumes.................................................................................................................................................524
Migrating Volumes to Another Storage Center................................................................................................... 525
Modifying Replications............................................................................................................................................526
Monitoring Replications.......................................................................................................................................... 529
Managing Cross-Platform Replication......................................................................................................................... 531
Cross-Platform Replication Requirements............................................................................................................ 531
Managing Replications Between PS Series Groups and Storage Centers........................................................531
Managing Replication Schedules...........................................................................................................................534
Portable Volume Disks.................................................................................................................................................. 536
Portable Volume Requirements..............................................................................................................................536
Portable Volume Process........................................................................................................................................537
Types of Portable Volume Disks............................................................................................................................. 537
Requirements for USB Disks.................................................................................................................................. 537
Requirements for PowerVault RD1000 Disk Bays................................................................................................537
Portable Volume Nodes.......................................................................................................................................... 538
16
Contents
Using Portable Volume Disks to Transfer Replication Data................................................................................ 538
Managing Replication Baselines and Portable Volume Disks............................................................................. 540
Storage Center Live Volumes.......................................................................................................................................545
Behavior of Volume QoS Settings in Live Volume Operations...........................................................................545
Live Volume Requirements..................................................................................................................................... 545
Live Volume Types................................................................................................................................................... 546
Live Volume Icon......................................................................................................................................................546
Live Volume Roles....................................................................................................................................................546
Automatic Failover for Live Volumes.....................................................................................................................548
Managed Replications for Live Volumes...............................................................................................................552
Creating Live Volumes............................................................................................................................................ 553
Modifying Live Volumes..........................................................................................................................................555
Modifying Live Volumes with Automatic Failover.................................................................................................561
Live Volume ALUA................................................................................................................................................... 562
Monitoring Live Volumes........................................................................................................................................ 564
15 Storage Center DR Preparation and Activation.........................................................................................567
How Disaster Recovery Works.....................................................................................................................................567
Step 1: A Volume is Replicated to a DR Site......................................................................................................... 567
Step 2: The Source Site Goes Down.....................................................................................................................568
Step 3: An Administrator Activates Disaster Recovery.......................................................................................568
Step 4: Connectivity is Restored to the Source Site.......................................................................................... 569
Step 5: An Administrator Restores the Source Volume......................................................................................569
Disaster Recovery Administration Options..................................................................................................................571
Preparing for Disaster Recovery...................................................................................................................................571
Saving and Validating Restore Points.....................................................................................................................571
Predeî‚‚ning Disaster Recovery Settings for Replications....................................................................................572
Test Activating Disaster Recovery.........................................................................................................................573
Activating Disaster Recovery....................................................................................................................................... 575
Types of Disaster Recovery Activation for Live Volumes....................................................................................575
Disaster Recovery Activation Limitations..............................................................................................................576
Planned vs Unplanned Disaster Recovery Activation..........................................................................................576
Disaster Recovery Activation Procedures.............................................................................................................576
Activating Disaster Recovery for PS Series Group Replications..............................................................................579
Restarting Failed Replications......................................................................................................................................580
Restart Replication for Multiple Restore Points...................................................................................................580
Restart a Replication for a Single Restore Point..................................................................................................580
Restoring Replications and Live Volumes.................................................................................................................... 581
Volume Restore Options..........................................................................................................................................581
Volume Restore Limitations.....................................................................................................................................581
Restoring a Live Volume and a Managed Replication.......................................................................................... 581
Volume Restore Procedures...................................................................................................................................582
Deleting Restore Points................................................................................................................................................ 583
16 Remote Data Collector............................................................................................................................. 585
Remote Data Collector Management......................................................................................................................... 585
Contents
17
Remote Data Collector Requirements.........................................................................................................................585
Conî‚‚guration Requirements...................................................................................................................................585
Software Requirements.......................................................................................................................................... 586
Storage Manager Virtual Appliance Requirements....................................................................................................586
Installing and Conî‚‚guring a Remote Data Collector..................................................................................................586
Install a Remote Data Collector..............................................................................................................................586
Install a Virtual Appliance as a Remote Data Collector........................................................................................587
Disconnecting and Reconnecting a Remote Data Collector.................................................................................... 589
Temporarily Disconnect a Remote Data Collector...............................................................................................589
Reconnect a Remote Data Collector to a Storage Center.................................................................................589
Remove a Remote Data Collector.........................................................................................................................590
Using a Remote Data Collector to Activate Disaster Recovery...............................................................................590
Log in to the Remote Data Collector.....................................................................................................................590
Create a User........................................................................................................................................................... 592
Use a Remote Data Collector to Prepare for Disaster Recovery.......................................................................592
Use a Remote Data Collector to Test Activate Disaster Recovery....................................................................593
Use a Remote Data Collector to Activate Disaster Recovery............................................................................ 593
Enabling Email Notiî‚‚cations for the Remote Data Collector....................................................................................593
17 Storage Replication Adapter for VMware SRM.........................................................................................595
Where to Find Dell SRA Deployment Instructions.....................................................................................................595
Dell SRA Limitations...................................................................................................................................................... 595
Dell SRA Requirements for VMware SRM................................................................................................................. 595
VMware SRM and Storage Manager Prerequisites.................................................................................................. 596
Dell SRA with Stretched Storage and vMotion......................................................................................................... 596
Storage Manager SRA Conî‚‚gurations........................................................................................................................ 597
Primary Data Collector Only Conî‚‚guration...........................................................................................................597
Remote Data Collector Conî‚‚guration................................................................................................................... 597
Selecting the Snapshot Type to Use for SRM 5.x and 6.x Volume Failover...........................................................598
Limitations for Selecting the Snapshot Type for SRM Failover.........................................................................598
Change the Snapshot Type Used for SRM Volume Failover..............................................................................599
18 Storage Center Threshold Alerts...............................................................................................................601
Conî‚‚guring Threshold Deî‚‚nitions................................................................................................................................601
Setting Up Threshold Deî‚‚nitions............................................................................................................................601
Assigning Storage Objects to Threshold Deî‚‚nitions........................................................................................... 604
Assigning Threshold Deî‚‚nitions to Storage Objects........................................................................................... 605
Viewing Threshold Alerts for Threshold Deî‚‚nitions............................................................................................ 606
Viewing and Deleting Threshold Alerts........................................................................................................................607
View Current and Historical Threshold Alerts...................................................................................................... 607
Filter Threshold Alerts by Storage Center............................................................................................................ 607
Filter Threshold Alerts by Threshold Deî‚‚nition Properties................................................................................. 607
View the Threshold Deî‚‚nition that Generated an Alert......................................................................................608
Delete Historical Threshold Alerts......................................................................................................................... 608
Conî‚‚guring Volume Advisor Movement Recommendations................................................................................... 608
Threshold Deî‚‚nitions That Support Volume Advisor.......................................................................................... 608
18
Contents
General Volume Advisor Requirements.................................................................................................................609
Additional Requirements for the Volume Latency Threshold Deî‚‚nition............................................................609
Types of Volume Movement Recommendations.................................................................................................609
Creating Threshold Deî‚‚nitions to Recommend Volume Movement...................................................................611
Moving a Volume Based on a Recommendation.................................................................................................. 614
Export Threshold Alert Data to a File...........................................................................................................................616
Conî‚‚guring Email Notiî‚‚cations for Threshold Alerts.................................................................................................616
Conî‚‚gure SMTP Server Settings...........................................................................................................................616
Conî‚‚gure an Email Address for Your User Account.............................................................................................617
Conî‚‚gure Email Notiî‚‚cation Settings for Your User Account............................................................................ 617
Performing Threshold Queries...................................................................................................................................... 617
View Saved Queries................................................................................................................................................. 618
Create a Threshold Query....................................................................................................................................... 618
Run a Saved Threshold Query................................................................................................................................ 619
Export the Results of a Threshold Query..............................................................................................................619
Edit a Saved Threshold Query............................................................................................................................... 620
19 Storage Center Reports............................................................................................................................ 621
Chargeback Reports...................................................................................................................................................... 621
Storage Center Automated Reports............................................................................................................................ 621
Displaying Reports......................................................................................................................................................... 622
View a Storage Center Automated Report...........................................................................................................622
View a Chargeback Report.....................................................................................................................................623
Working with Reports....................................................................................................................................................624
Update the List of Reports.....................................................................................................................................624
Navigate Through the Pages of a Report.............................................................................................................624
Print a Report...........................................................................................................................................................624
Save a Report to the Client Computer................................................................................................................. 624
Delete a Report........................................................................................................................................................625
Conî‚‚guring Automated Report Generation............................................................................................................... 625
Set Up Automated Reports for All Storage Centers (Global Settings).............................................................625
Set Up Automated Reports for an Individual Storage Center............................................................................626
Testing Automated Reports Settings.......................................................................................................................... 626
Conî‚‚gure Storage Manager to Email Reports........................................................................................................... 627
Conî‚‚gure SMTP Server Settings.......................................................................................................................... 627
Conî‚‚gure an Email Address for Your User Account............................................................................................628
Conî‚‚gure Email Notiî‚‚cation Settings for Your User Account........................................................................... 628
20 Storage Center Chargeback.................................................................................................................... 629
Conî‚‚gure Chargeback or Modify Chargeback Settings...........................................................................................629
Assign Storage Costs for Global Disk Classes..................................................................................................... 630
Assign Storage Costs for Storage Center Disk Tiers........................................................................................... 631
Conî‚‚guring Chargeback Departments....................................................................................................................... 632
Setting Up Departments.........................................................................................................................................632
Managing Department Line Items......................................................................................................................... 634
Assigning Volumes to Chargeback Departments.................................................................................................635
Contents
19
Perform a Manual Chargeback Run............................................................................................................................ 638
Viewing Chargeback Runs............................................................................................................................................639
View a Chart of Department Costs for a Chargeback Run................................................................................639
View the Results of the Chargeback Run in Table Format.................................................................................639
View Cost and Storage Savings Realized by Dynamic Capacity for a Chargeback Run................................ 639
View Cost and Storage Savings Realized by Using Data Instant Snapshots for a Chargeback Run............ 640
View Cost Savings Realized by Using Data Progression for a Chargeback Run............................................. 640
Working with Charts..................................................................................................................................................... 640
Zoom in on an Area of the Chart...........................................................................................................................640
Return to the Normal Zoom Level of the Chart.................................................................................................. 640
Save the Chart as a PNG Image............................................................................................................................ 641
Print the Chart..........................................................................................................................................................641
Exporting Chargeback Data..........................................................................................................................................641
Export Chargeback Run Data................................................................................................................................. 641
Export Chargeback Run Data for a Single Department.......................................................................................641
21 Storage Center Monitoring.......................................................................................................................643
Storage Alerts................................................................................................................................................................ 643
Status Levels for Alerts and Indications................................................................................................................643
Viewing Storage System Alerts..............................................................................................................................643
Send Storage Center Alerts and Indications to the Data Collector Immediately............................................. 646
Events Alerts.................................................................................................................................................................. 646
Storage Manager Event Types...............................................................................................................................646
Viewing Data Collector Events...............................................................................................................................647
Conî‚‚guring Email Alerts for Storage Manager Events....................................................................................... 649
Storage Logs................................................................................................................................................................... 651
Send Storage Center Logs to Storage Manager..................................................................................................651
Viewing Storage Logs............................................................................................................................................. 654
Audit Logs.......................................................................................................................................................................656
Viewing Audit Logs..................................................................................................................................................656
Export Monitoring Data................................................................................................................................................ 658
Conî‚‚gure Data Collection Schedules..........................................................................................................................658
22 Data Collector Management.....................................................................................................................661
Access the Data Collector View .................................................................................................................................. 661
Conî‚‚guring Data Collector Settings............................................................................................................................662
Conî‚‚guring General Settings.................................................................................................................................662
Conî‚‚guring Environment Settings........................................................................................................................ 668
Conî‚‚guring Monitoring Settings.............................................................................................................................671
Conî‚‚guring Virtual Appliance Settings..................................................................................................................676
Managing Available Storage Centers...........................................................................................................................677
Delete an Available Storage Center........................................................................................................................677
Clear All Data for a Storage Center....................................................................................................................... 678
Remove a Storage Center from a Data Collector User Account....................................................................... 678
Managing Available PS Series Groups.........................................................................................................................679
Delete an Available PS Series Group..................................................................................................................... 679
20
Contents
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
/