Avid MediaCentral
®
Platform Services
Upgrading Guide Version 2.4.0
This document is a guide to upgrading to MediaCentral Platform Services (MCS) 2.4.0 from a
previous version of MCS or Interplay Central Services (ICS).
Important Information
Avid
®
recommends that you read all the information in this upgrade guide thoroughly
before installing or using the corresponding software release.
Important: Check the Avid Knowledge Base for the latest updates to this guide and all
related documentation for Avid MediaCentral Platform Services.
http://avid.force.com/pkb/articles/en_US/readme/Avid-MediaCentral-Version-2-4-x-
Documentation
Note: For a list products qualified for use with MCS 2.4, and the supported RHEL operating
system, see Software Compatibility Matrix on the Avid Knowledge Base
.
Revision History
Date Revised
Changes Made
November 10, 2015
Added steps to the cluster upgrade process for the RabbitMQ cluster.
Appendix G: Glossary has been removed. For more information on the
systems discussed in this guide, see the MediaCentral Platform Services
Concepts and Clustering Guide.
November 6, 2015
Added note regarding Dell servers to the cluster upgrade process.
October 29, 2015
Simplified Verifying Gluster Volume Permissions process
Other minor updates
October 8, 2015
Added steps to the upgrade process to ensure that Gluster volumes
remount normally after the MCS software installation.
September 24, 2015
MediaCentral 2.4.0 First Publication.
• Added “Upgrading the iNEWS Client Version”.
• Added steps for configurations with Media Index.
• Removed “Upgrading GlusterFS” procedure. This process is no
longer applicable as MCS 2.4.0 automatically updates Gluster.
MediaCentral Services 2.4 Upgrading Guide
2
Contents
Important Information ....................................................................................................................... 1
Revision History .................................................................................................................................. 1
Contents ..................................................................................................................................................... 2
Overview .................................................................................................................................................... 4
How to Use this Guide ............................................................................................................................... 4
Operating System ....................................................................................................................................... 5
Obtaining the Software Packages .............................................................................................................. 5
Update Installation versus Full Installation ................................................................................................ 5
Upgrading Multi-Zone Deployments ......................................................................................................... 5
Upgrading Media | Index Deployments .................................................................................................... 6
Upgrading the MediaCentral Distribution Service (MCDS) ........................................................................ 6
An Important Note on SSL Certificate Passwords ...................................................................................... 7
Backing Up and Restoring SSL Private Keys ............................................................................................... 8
Installing the Closed Captioning Service .................................................................................................... 8
Upgrading the iNEWS Client Version ......................................................................................................... 8
Migrating User Settings from UMS to USS During an Upgrade ................................................................. 9
Upgrade Prerequisites ............................................................................................................................. 10
Upgrade Paths .......................................................................................................................................... 10
Upgrades for systems already running RHEL 6.5 ................................................................................. 10
Upgrades for systems running RHEL 6.3 .............................................................................................. 10
Upgrades from 1.2.x or 1.3.x to MCS 2.4 ............................................................................................. 10
Upgrading a Single Server RHEL 6.5 System to MCS 2.4 .......................................................................... 11
Installing MediaCentral Platform Services 2.4 ..................................................................................... 11
Upgrading a Cluster RHEL 6.5 System to MCS 2.4 ................................................................................... 13
Identifying the Master, Slave and Load-Balancing Nodes .................................................................... 13
Verifying Gluster File Replication ......................................................................................................... 14
Backing-up the User Database and System Settings ............................................................................ 15
Updating the Media Index Database .................................................................................................... 15
Taking the Cluster Offline ..................................................................................................................... 15
Updating the Cluster ............................................................................................................................ 16
Stopping the RabbitMQ Service and Rebooting the Cluster ................................................................ 18
Bringing the Corosync Cluster Online ................................................................................................... 19
Verifying Gluster Volume Permissions ................................................................................................. 20
MediaCentral Services 2.4 Upgrading Guide
3
Removing the Gluster Metadata Cache Replication Volume ............................................................... 21
Upgrading a Single Server RHEL 6.3 System to MCS 2.4 .......................................................................... 22
Backing-up System Settings.................................................................................................................. 22
Installing RHEL 6.5 and MCS 2.4 ........................................................................................................... 23
Restoring System Settings and Migrating the User Database .............................................................. 24
Additional Upgrade Procedures ........................................................................................................... 25
Upgrading a Cluster RHEL 6.3 System to MCS 2.4 ................................................................................... 27
Identifying the Master, Slave and Load-Balancing Nodes .................................................................... 27
Backing Up the System Settings ........................................................................................................... 28
Taking the Cluster Offline ..................................................................................................................... 29
Upgrading the Software ....................................................................................................................... 29
Restoring System Settings and Migrating the User Database .............................................................. 30
Additional Upgrade Procedures ........................................................................................................... 31
Appendix A: Post Upgrade Notes ............................................................................................................. 33
Reconfiguring MCS for Upgrades from ICS 1.7 or Earlier ..................................................................... 33
Reconfiguring MCS for Use with Media Composer | Cloud ............................................................. 33
Reconfiguring MCS for Use with Interplay | MAM ........................................................................... 34
Resetting Layouts After Upgrading from V2.0 to V2.4 ......................................................................... 35
Resetting Layouts After Upgrading from ICS V1.6 or Earlier ................................................................ 35
Appendix B: Migrating the UMS Database to a Test Node ...................................................................... 36
Migrating the 1.6.x (or later) UMS Database ....................................................................................... 36
Migrating the 1.4.x/1.5.x UMS Database ............................................................................................. 37
Appendix C: Mounting and Unmounting a USB Drive ............................................................................. 38
Appendix D: Copying Software to the MCS Server .................................................................................. 40
Copying Software Using WinSCP .......................................................................................................... 40
Copying Software Using a USB Drive .................................................................................................... 41
Appendix E: Deleting the RAID 5 .............................................................................................................. 43
Deleting the RAID 5 on an HP Gen9 ..................................................................................................... 43
Deleting the RAID 5 on an HP Gen8 ..................................................................................................... 44
Deleting the RAID 5 on an Dell Server .................................................................................................. 47
Appendix F: Backing Up and Restoring System Settings and the ICS/MCS Database .............................. 51
Copyright and Disclaimer ......................................................................................................................... 55
MediaCentral Services 2.4 Upgrading Guide
4
Overview
Upgrading to MediaCentral Platform Services (MCS) 2.4 for MediaCentral UX, Media
Composer | Cloud, and Interplay | MAM, involves the migration of settings and data along
with the software upgrade. Depending on the upgrade path you are faced with, the
migration of settings and data is manual or automated.
Note: Beginning with version 2.0, the term “MediaCentral Platform Services” replaces
“Interplay Central Services.” In addition, the term “MediaCentral Playback Service” replaces
“Interplay Central Playback Service.”
There are two main upgrade paths to MCS 2.4, depending on the currently installed
operating system:
• RHEL 6.5 – If this is the currently installed OS, the upgrade is non-destructive and
largely automated. Manually backing up system settings and MCS database is not
required, but recommended.
• RHEL 6.3 and earlier – This upgrade involves a clean installation of RHEL 6.5 and
MCS. Manually backing up the system settings and MCS database before the
upgrade and restoring them after the upgrade is complete is mandatory.
When the upgrade path involves a RHEL version change, you must also manually
back up any private keys associated with SSL certificates received from a Certificate
Authority, if CA-signed certificates are in use. In the case of a cluster, you must
manually re-create the cluster and reconfigure Gluster cache replication (if used).
How to Use this Guide
This guide documents the overall upgrade process for the supported upgrade scenarios. It
must be used in conjunction with the MediaCentral Platform Services Installation and
Configuration Guide. For clustering, the MediaCentral Platform Services Concepts and
Clustering Guide provides valuable information regarding cluster operations and post-
upgrade verification.
For each step in the overall upgrade process, this document offers the following guidance:
• Includes step-by-step instructions for a procedure
- or -
• Cross-references a section in the MediaCentral Platform Services Installation and
Configuration Guide
In the second case, the heading of the relevant section in the MediaCentral Platform
Services Installation and Configuration Guide is provided. Since direct page references are
unreliable, you must scan the table of contents in that guide to find the heading and page.
MediaCentral Services 2.4 Upgrading Guide
5
Operating System
MCS 2.4 requires Red Hat Enterprise Linux (RHEL) v6.5. In addition, whether performing an
upgrade or a full installation, the MCS installer requires the RHEL 6.5 ISO to be mounted, so
you must obtain the RHEL 6.5 ISO. This is true, even when performing an upgrade where the
operating system does not change.
For convenience, the following table lists the ICS/MCS releases and corresponding RHEL
versions.
ICS/MCS Version
RHEL Version
ICS 1.2.3 – 1.5.0
RHEL 6.2
ICS 1.6.0 – 1.8.0
RHEL 6.3
ICS 1.8.1
RHEL 6.5
MCS 2.0.0 – 2.4.x
RHEL 6.5
Obtaining the Software Packages
Multiple software packages are required to properly install and configure MediaCentral
Platform Services. For more information on downloading these packages, see “Obtaining
the Software” in the MediaCentral Platform Services Installation and Configuration Guide.
Update Installation versus Full Installation
For MCS systems already operating under RHEL v6.5, the upgrade process is fairly straight
forward and requires a minimal amount of time to complete.
For older versions of ICS or MCS operating under RHEL v6.3, the upgrade to MCS 2.4 is closer
to a new installation than it is to an upgrade. The process involves using the MCS
Installation USB Drive to reimage the server and complete a fresh installation of MCS.
Because of this, extra care must be taken to ensure the MCS system settings and user
database are preserved.
Upgrading Multi-Zone Deployments
If your deployment is configured for multi-zone, note the following:
• Upgrade slave zones first. If you cannot bring all zones offline at once, upgrade the
slave zone(s) before the master zone.
An upgraded slave zone no longer has access to the centralized user management
service (UMS), which is owned by the master zone. Users in the upgraded slave zone
can continue to log in —the local, read-only copy of the UMS database is used —
but you cannot create new users.
• After the master zone has been updated as the last zone of the multi-zone setup,
the /opt/avid/cluster/bin/pgpool_remote_start script needs to be run to enable the
proper syncing of the user databases between all zones again. Please be aware that
the script will restart some services on the systems in the remote zones. After the
services are restarted, users will need to sign back into MediaCentral.
MediaCentral Services 2.4 Upgrading Guide
6
Upgrading Media | Index Deployments
If your deployment is configured for Media Index, note the following:
• Systems configured for Media Index require additional steps for certain upgrade
paths. Review and complete the following steps as appropriate for your installation:
If you are upgrading from MediaCentral v2.1 or earlier, you must delete the indexes
in all zones before the upgrade. Deleting the indexes reverts the index table to a
clean (empty) state. For instructions, see the Avid Media | Index Configuration
Guide.
If you are upgrading from MediaCentral v2.2.x or later, the indexes are compatible
with MediaCentral v2.4 and do not need to be deleted. However, a migration script
must be run to update the indexes to include new features associated with MCS
v2.4. For instructions, see the Avid Media | Index Configuration Guide.
• To configure the Production Engine Bus Connector (PEBCo) for Media Index on MCS
v2.4.x, you must use the Interplay Administrator tool for Interplay Production v3.3
or later. PEBCo supports the Interplay Production Engine v3.1 and later.
Upgrading the MediaCentral Distribution Service (MCDS)
MCDS is used in MediaCentral UX configurations which employ a Send To Playback (STP)
workflow. Consult the version compatibility chart on the Avid Knowledge Base for the
correct version of MCDS for your installation and upgrade the software if necessary.
http://avid.force.com/pkb/articles/en_US/compatibility/Avid-Video-Compatibility-Charts
To upgrade the MediaCentral Distribution Service:
1. MCDS could be in one or more location within your environment. Use the following
process to verify where MCDS is installed:
a. Log into MediaCentral UX as a user with Administrator privileges.
See “Logging into MediaCentral” in the MediaCentral Platform Services
Installation and Configuration Guide for additional information.
b. Select System Settings from the Layout selector.
c. In the Settings pane, select Interplay | Production.
d. Make note of the MCDS Service URL. This field lists the location or locations
where MCDS has been installed.
e. Log out of MediaCentral UX and close the browser.
2. On the system where MCDS has been installed, use the “Programs and Features”
Control Panel to uninstall MCDS.
Note: MCDS v3.3 is a 64 bit application. Prior 32 bit versions must be manually removed to
avoid the installation of multiple versions on the same system.
3. If upgrading from a 32 bit version of MCDS, navigate to C:\Program Files (x86)\Avid\ and
delete the residual “MediaCentral Distribution Service” folder.
4. Install the updated version of MCDS.
5. Repeat this process if you need to upgrade MCDS on a second host system.
MediaCentral Services 2.4 Upgrading Guide
7
An Important Note on SSL Certificate Passwords
Interplay Central uses the Secure Sockets Layer (SSL) for its server to browser connections.
Default passwords may have been used by the system to automatically generate a self-
signed certificate. Customized certificates can also be used, including those that have been
issued by a Certificate Authority (CA).
When customized or CA-signed certificates are used, the SSL passwords must be written to
the user-available Application Properties file in the following directory:
/opt/avid/etc/avid/avid-interplay-central/config/application.properties
This should not be confused with the system-reserved Application Properties file, in the
following location:
/opt/avid/avid-interplay-central/config/application.properties
Caution: The user-available Application Properties file is backed up and restored by the system-
backup script. The system-reserved Application Properties file is not backed up. If the SSL
passwords are in the wrong file, SSL configuration will be broken by the upgrade.
To avoid mishaps, double-check the contents of the system-reserved Application Properties
file before proceeding with the upgrade.
To double-check the system-reserved Application Properties file:
1. List the contents of the Application Properties file reserved for use by the system:
less /opt/avid/avid-interplay-
central/config/application.properties
2. Look for the following two lines (they might not be next to each other):
system.org.ops4j.pax.web.ssl.password=OBF\:1lfc1n7n1l1618qm18qo1kxw1n3v1lc6
system.org.ops4j.pax.web.ssl.keypassword=OBF\:1lfc1n7n1l1618qm18qo1kxw1n3v1l
c6
The obfuscated passwords shown above (in bold) are the default system passwords.
3. If the passwords you see match the above, nothing more needs to be done.
4. If the passwords you see are different, it indicates the file has been edited and you must
copy the two modified lines into the following file (you might need to create the file):
/opt/avid/etc/avid/avid-interplay-central/config/application.properties
Copying the modified lines to the user-available Application Properties file ensures they
are preserved during the upgrade.
MediaCentral Services 2.4 Upgrading Guide
8
Backing Up and Restoring SSL Private Keys
Upgrading can be a “true” (non-destructive) upgrade, or can involve a fresh install of RHEL.
For fresh installs, the system-backup script has been provided to back up and restore system
settings related to MCS, including SSL data (the Jetty keystore and certificates it contains).
However, if you are using a Trusted CA-signed Certificate, a private key (jettyPrivateKey.key)
file was created during the certificate creation process. The procedure suggests that you
back up this file in a safe place, such as the root user home directory (/root) directory. If you
are performing a fresh install of RHEL, be sure to back up this private key to an external
location before performing the upgrade. Backing up private key files is good practice,
whatever the type of upgrade being performed.
Note: If the private key was not backed up elsewhere, it is doubly important to back up the
keystore before performing the upgrade. The private key is used to generate Certificate Signing
Requests (CSRs).
For instructions on creating and installing SSL certificates see the following article:
http://avid.force.com/pkb/articles/en_US/how_to/SSL-Certificates-for-server-to-browser-
connections
To back up and restore the Jetty keystore:
1. Locate the separate private key file used to generate the Certificate Signing Request (if
any).
The default name is jettyPrivateKey.key. During its creation, it was suggested you store it
in the following directory:
/root
2. Copy the private key to a safe location (off the MCS server), restoring it at an
appropriate step in the upgrade.
Installing the Closed Captioning Service
MCS v2.3 introduced a Closed Captioning Service for use with MediaCentral UX. This service
is not included in the MCS software package and requires a separate installation. Follow the
steps in this guide to complete the upgrade to MCS v2.4. Once the upgrade is complete, the
Closed Captioning Service can be installed or upgraded.
See “PART VII: INSTALLING THE CLOSED CAPTIONING SERVICE” in the MediaCentral Platform
Services Installation and Configuration Guide for upgrade instructions.
Upgrading the iNEWS Client Version
If your workflow includes integration with Avid iNEWS, ensure your iNEWS system is
updated with the iNEWS client version for this release of MCS. See the Avid MediaCentral
Platform Services ReadMe for iNEWS client version information. See “Editing
SYSTEM.CLIENT.VERSIONS” in the MediaCentral Platform Services Installation and
Configuration Guide for instructions on adding the client version to the iNEWS server.
MediaCentral Services 2.4 Upgrading Guide
9
Migrating User Settings from UMS to USS During an Upgrade
MediaCentral 2.0 introduced a new user database scheme. Upgrades of systems prior to 2.0
need to complete a user database migration using the supplied avid-uss-import script.
• For upgrades from 1.8.0 (RedHat 6.3) or earlier, a backup and restore of the MCS
database is required. The user database migration must be completed after
restoring the MCS database, but before logging in via the MediaCentral | UX user
interface.
• For upgrades from 1.8.1 (RedHat 6.5), a backup and restore of the MCS database is
not required. However, the avid-uss-import script must still be run before logging in
via the MediaCentral | UX user interface.
Note: In a cluster, the user database migration script must be run once on the master node only.
See Appendix F: Backing Up and Restoring System Settings and the ICS/MCS Database for
details on the backup and restore procedure. Procedures for using the avid-uss-import
script are covered in
Upgrades for systems running RHEL 6.3.
MediaCentral Services 2.4 Upgrading Guide
10
Upgrade Prerequisites
Before starting any MCS upgrade, it is important to ensure you have a healthy environment.
Verify the following as appropriate:
(All configurations) Ensure you can log into MediaCentral UX via a web browser and
verify basic functionality.
(Cluster configurations) Verify the status of RabbitMQ and the ACS Bus.
See “Verifying the Status of RabbitMQ” in the MediaCentral Platform Services
Installation and Configuration Guide for details.
See “Verifying ACS Bus Functionality” in the MediaCentral Platform Services Installation
and Configuration Guide for details.
(Cluster configurations) Verify the status of the Corosync cluster. Ensure there are no
errors.
See “Monitoring MCS High-Availability” in the MediaCentral Platform Services
Installation and Configuration Guide for details.
(Multi-zone configurations) Verify that the “Zones” section of the MediaCentral UX
System Settings layout displays all expected zones with no errors.
See “Troubleshooting the Multi-Zone Setup” in the MediaCentral Platform Services
Installation and Configuration Guide for details.
Upgrade Paths
This section presents the upgrade paths for specific deployment options.
Upgrades for systems already running RHEL 6.5
For upgrading instructions:
• Upgrading a Single Server RHEL 6.5 System to MCS 2.4 on page 11
.
• Upgrading a Cluster RHEL 6.5 System to MCS 2.4 on page 13.
Upgrades for systems running RHEL 6.3
For upgrading instructions:
• Upgrading a Single Server RHEL 6.3 System to MCS 2.4 on page 22
.
• Upgrading a Cluster RHEL 6.3 System to MCS 2.4 on page 27.
Upgrades from 1.2.x or 1.3.x to MCS 2.4
For Interplay Central and / or ICPS 1.2.x and 1.3.x upgrade options, please consult your Avid
representative.
MediaCentral Services 2.4 Upgrading Guide
11
Upgrading a Single Server RHEL 6.5 System to MCS 2.4
Installing MCS 2.4 on a single server already operating under RHEL 6.5 is a simple automated
update of the affected packages. All MCS system settings are preserved.
Expected downtime: 30 min.
Note: Prior to upgrading, ensure you have verified the
Upgrade Prerequisites on page 10.
Installing MediaCentral Platform Services 2.4
1. Prepare the v2.4 MCS Installation USB Drive and connect it to your MCS server.
Installing MCS 2.4 from the MCS Installation USB Drive is required as it contains the
RHEL 6.5 ISO required for the upgrade process.
See “Creating the MCS Installation USB Drive” in the MediaCentral Platform Services
Installation and Configuration Guide for details.
2. Mount the USB drive and change to the USB mount point.
See “Appendix C: Mounting and Unmounting a USB Drive
” for details.
Warning: The MCS Installation USB Drive contains all software used to image a new
MediaCentral server. Do not reboot the server with this USB drive attached.
3. Although the upgrade is non-destructive, it is a good idea to back-up the MCS database.
This is an optional, but recommended step.
Back up the MCS settings and database using the backup script:
./system-backup.sh –b
See Appendix F: Backing Up and Restoring System Settings and the ICS/MCS Database
for additional details.
4. (Media Index configurations only) If Media Index has been configured on your MCS
system, you need to run a database migration script prior to upgrading the MCS
software. For instructions, see the Avid Media | Index Configuration Guide.
5. Start the installation script:
./install.sh
Note: Be sure to use the dot-slash (“./”) notation, which tells Linux to look for the installation
script in the current directory. Otherwise you will receive the following error message:
-bash: install.sh: command not found
If your system is configured for Media Index, you will receive a message indicating that
“A Media | Index configuration has been detected on this system”. Select one of the
following options:
• Type
IGNORE if you have already upgraded your Media | Index data or want to
upgrade your system without completing the migration.
• Type
QUIT if you have not yet upgraded your Media | Index data and need to exit
the installer.
MediaCentral Services 2.4 Upgrading Guide
12
During the installation process, progress is displayed. The update completes with an
indication of success:
INSTALLATION FINISHED
If any errors were encountered during the upgrade, you can obtain more information in
the installation log at:
/var/log/MediaCentral_Services_<version>_Build_<number>_Linux.log
6. Check the installation was successful using the ics_version script:
ics_version
Service version numbers are returned as follows:
UMS Version: 2.4.x.x
ICPS Version: 2.4.x.x
ICPS manager Version: 2.4.x.x
ACS Version: 2.4.x.x
System ID: "xxxxxxxxxxx"
ICS installer: 2.4 (Build xx)
Created on <installer creation date>
Note: For precise version numbers for this release, see the Avid MediaCentral Platform
Services 2.4 ReadMe.
Note: If you receive a “command not found” error, logout and log back into the server or SSH
session (PuTTY). This will refresh the login for the upgraded system and should allow the
command to run properly.
Note: The System ID is an 11-digit number used in support calls. You enter it via the
MediaCentral UX user interface. See the MCS Installation and Configuration Guide for more
information.
7. Unmount the USB drive.
See “Appendix C: Mounting and Unmounting a USB Drive
” for details.
8. (Multi-Zone configurations only) Once the master zone has been updated, run the
following script from the master zone to re-enable the database syncing between all
zones:
/opt/avid/cluster/bin/pgpool_remote_start
MediaCentral Services 2.4 Upgrading Guide
13
Upgrading a Cluster RHEL 6.5 System to MCS 2.4
Installing MCS 2.4 on a cluster already operating under RHEL 6.5 involves an automated
update of the affected packages. All MCS system settings are preserved. All nodes must be
taken offline to perform the update.
The cluster upgrade involves the following steps:
Identifying the Master, Slave and Load-Balancing Nodes
Verifying Gluster File Replication
(optional) Backing-up the User Database and System Settings
(Media Index only) Updating the Media Index Database
Taking the Cluster Offline
Updating the Cluster
Stopping the RabbitMQ Service and Rebooting the Cluster
Bringing the Corosync Cluster Online
(if applicable) Verifying Gluster Volume Permissions
(if applicable) Removing the Gluster Metadata Cache Replication Volume
Expected downtime: 1 hr. for 2 servers, + 30 min. for each additional server.
Note: Prior to upgrading, ensure you have verified the
Upgrade Prerequisites on page 10.
Identifying the Master, Slave and Load-Balancing Nodes
There are three types of nodes in a cluster: master, slave, and load-balancing. The master
“owns” multiple resources such as the cluster IP address. The slave assumes the role of
master in the event of a failover. Additional nodes play a load-balancing role, but can never
take on the role of master.
To identify the master, slave, and load-balancing nodes:
1. Verify the current role of each node in the cluster through the Cluster Resource
Monitor. Log in to any machine in the cluster as the root user and type:
crm_mon
2. To identify the master and slave nodes, look for the line containing “Master/Slave Set”.
For example:
Master/Slave Set: ms_drbd_postgres [drbd_postgres]
Masters: [ wavd-mcs01 ]
Slaves: [ wavd-mcs02 ]
In this example, the master node is wavd-mcs01 and the slave node is wavd-mcs02.
3. To identify the load-balancing nodes, look for the line containing “Clone Set”.
For example:
Clone Set: AvidAllEverywhere [AvidAll]
Started: [ wavd-mcs01 wavd-mcs02 wavd-mcs0 wavd-mcs04]
In this example, the load-balancing nodes are wavd-mcs03 and wavd-mcs04.
4. Exit crm_mon by pressing CTRL-C on the keyboard.
MediaCentral Services 2.4 Upgrading Guide
14
Verifying Gluster File Replication
The MCS v2.4 installation includes the Gluster v3.6.4 software packages to enable file
replication across the cluster nodes. If your cluster is already configured with Gluster and
you are running an earlier version of Gluster, the packages will be automatically updated
during the MCS software installation. If your system is not configured with Gluster, the
packages will still be installed, but the software will not be activated.
If your MCS servers are configured for Gluster file replication, complete the “Gluster only”
steps in MCS upgrade process below. If your system is not configured for Gluster, the
“Gluster only” steps can be skipped.
To determine if Gluster is configured:
1. Verify if the “glusterd” service is running on your MCS servers:
service glusterd status
If Gluster is active, the command should respond with the following:
glusterd (pid 1457) is running...
If the glusterd service is not running, the command should respond with:
glusterd: unrecognized service
2. You can further verify if Gluster has been configured by looking for the mounted
volumes created through the Gluster configuration process:
df -h
If Gluster has been configured, you should see the three “gl-cache” volumes:
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/lvg-lv_root 75G 12G 60G 17% /
tmpfs 1.9G 45M 1.9G 3% /dev/shm
/dev/sda1 504M 45M 434M 10% /boot
/dev/mapper/vg_ics_cache-lv_ics_cache 99G 189M 94G 1% /cache
wavd-mcs01:/gl-cache-dl 99G 189M 94G 1% /cache/download
wavd-mcs01:/gl-cache-fl 99G 189M 94G 1% /cache/fl_cache
wavd-mcs01:/gl-cache-mcam 99G 189M 94G 1% /cache/render
MediaCentral Services 2.4 Upgrading Guide
15
Backing-up the User Database and System Settings
Although the upgrade is non-destructive, it is a good idea to back-up the MCS database.
This is an optional, but recommended step.
To back up the database and settings:
1. Prepare the v2.4 MCS Installation USB Drive and connect it to the cluster master node.
See “Creating the MCS Installation USB Drive” in the MediaCentral Platform Services
Installation and Configuration Guide for details.
2. Mount the USB drive and change to the mount point.
See “Appendix C: Mounting and Unmounting a USB Drive
” for details.
Warning: The MCS Installation USB Drive contains all software used to image a new
MediaCentral server. Do not reboot the server with this USB drive attached.
3. Back up the current system settings and MCS database using the backup script.
See Appendix F: Backing Up and Restoring System Settings and the ICS/MCS Database
for additional details.
4. Repeat this process for the slave and all load balancing nodes.
Updating the Media Index Database
If Media Index has been configured on your MCS system, you need to run a database
migration script prior to upgrading the MCS software. This must be completed while the
cluster is online. For instructions, see the Avid Media | Index Configuration Guide.
Taking the Cluster Offline
Corosync and RabbitMQ maintain independent clustering mechanisms. Corosync clusters
operate in a master/slave relationship whereas RabbitMQ clusters run in an Active/Active
state where any node in the cluster could be the owner of the RabbitMQ database. This
means that the order in which the nodes join and leave the RabbitMQ cluster is particularly
important. In the following process, you will put each node of the Corosync cluster into
“standby” and you will stop the RabbitMQ resources to pause its clustering functionality.
Note: Take care to note the order in which nodes are taken offline. You will need to bring the
nodes back online in the reverse order. Failure to do so may break the RabbitMQ cluster.
Note: If Media Index has been configured on your system, complete the database migration
prior to taking the cluster offline. See Upgrading Media | Index Deployments
for details.
In this process you will put each Corosync cluster node into standby and then stop the
RabbitMQ on each server. Complete this process one node at a time, waiting 30 seconds
between each node. Avid recommends starting with the final node first.
Example: If you have four nodes, begin with node 4 (wavd-mcs04), followed by node 3
(wavd-mcs03).
MediaCentral Services 2.4 Upgrading Guide
16
To bring the cluster into standby mode:
1. Begin by putting the Corosync load-balancing nodes into standby mode and then
stopping the rabbitmq service. Complete this process one node at a time, waiting 30
seconds between each node:
crm node standby <node name>
service rabbitmq-server stop
2. Wait 30 seconds and put the Corosync slave node into standby:
crm node standby <node name>
service rabbitmq-server stop
3. Wait 30 seconds and put the Corosync master node into standby:
crm node standby <node name>
service rabbitmq-server stop
Updating the Cluster
Upgrade the nodes in the reverse order that you stopped the services. Start with the
Corosync master node, followed by the slave node and finally the load balancing nodes.
To update the cluster:
1. If you have not already done so, prepare the v2.4 MCS Installation USB Drive and
connect it to your first MCS server.
Installing MCS 2.4 from the MCS Installation USB Drive is required as it contains the
RHEL 6.5 ISO required for the upgrade process.
See “Creating the MCS Installation USB Drive” in the MediaCentral Platform Services
Installation and Configuration Guide for details.
2. Mount the USB drive and change to the mount point.
See “Appendix C: Mounting and Unmounting a USB Drive
” for details.
Warning: The MCS Installation USB Drive contains all software used to image a new
MediaCentral server. Do not reboot the server with this USB drive attached.
3. (Gluster only) Unmount the Gluster volumes:
umount /cache/download
umount /cache/fl_cache
umount /cache/render
4. Start the installation script.
./install.sh
Note: Be sure to use the dot-slash (“./”) notation, which tells Linux to look for the installation
script in the current directory. Otherwise you will receive the following error message:
-bash: install.sh: command not found
MediaCentral Services 2.4 Upgrading Guide
17
If your system is configured for Media Index, you will receive a message indicating that
“A Media | Index configuration has been detected on this system”. Select one of the
following options:
• Type
IGNORE if you have already upgraded your Media | Index data or want to
upgrade your system without completing the migration.
• Type
QUIT if you have not yet upgraded your Media | Index data and need to exit
the installer.
During the installation process, progress is displayed. The update completes with an
indication of success:
INSTALLATION FINISHED
If any errors were encountered during the upgrade, you can obtain more information in
the installation log at:
/var/log/MediaCentral_Services_<version>_Build_<number>_Linux.log
5. Check the installation was successful using the ics_version script:
ics_version
Service version numbers are returned as follows:
UMS Version: 2.4.x.x
ICPS Version: 2.4.x.x
ICPS manager Version: 2.4.x.x
ACS Version: 2.4.x.x
System ID: "xxxxxxxxxxx"
ICS installer: 2.4 (Build xx)
Created on <installer creation date>
Note: For precise version numbers for this release, see the Avid MediaCentral Platform
Services 2.4 ReadMe.
Note: If you receive a “command not found” error, logout and log back into the server or SSH
session (PuTTY). This will refresh the login for the upgraded system and should allow the
command to run properly.
6. Unmount the USB drive, proceed to the next node, and follow the same pattern:
a. Mount the MCS 2.4 Installation USB drive
b. (Gluster only) Unmount the Gluster volumes
c. Run the update script
d. Verify the ics_version output
e. Unmount the USB drive
7. (Gluster only) Complete the following steps on each node to regenerate the Gluster
volumes:
a. Stop the glusterd service:
service glusterd stop
MediaCentral Services 2.4 Upgrading Guide
18
b. Regenerate the volfile names on all bricks:
glusterd --xlator-option *.upgrade=on –N
c. Start the glusterd service:
service glusterd start
d. Remount the Gluster volumes:
mount /cache/download
mount /cache/fl_cache
mount /cache/render
Stopping the RabbitMQ Service and Rebooting the Cluster
Installing the software update restarted the RabbitMQ service. The services need to be
stopped again before rebooting the cluster.
Prior to completing this process, note the following:
• If you are upgrading from MCS v2.2.3 or earlier, this process is required.
• If you are upgrading from MCS v2.2.4 or later, this process can be skipped.
To stop RabbitMQ and reboot the cluster:
1. Stop the RabbitMQ services on the final load-balancing node:
service rabbitmq-server stop
Example: If you have four cluster nodes, number 4 is the “final” load-balancing node.
2. Waiting 30 seconds between each node, stop the RabbitMQ services on the next load-
balancing node, then the slave node and finally the master node.
3. Once the RabbitMQ services have been stopped on each node, all nodes in the cluster
must be rebooted. The order in which the nodes are rebooted is important to ensure
RabbitMQ starts successfully. Begin by rebooting the master node:
reboot <master node name>
4. Waiting 30 seconds between each node, reboot the slave node and each load-balancing
node.
reboot <node name>
5. Once all nodes have been rebooted and RabbitMQ has been started on each node,
verify the status of the RabbitMQ service.
See “Verifying the Status of RabbitMQ” in the MediaCentral Platform Services
Installation and Configuration Guide for details.
See “Verifying ACS Bus Functionality” in the MediaCentral Platform Services Installation
and Configuration Guide for details.
Note: The RabbitMQ cluster must be healthy prior to bringing the Corosync cluster back
online. If you suspect there is a problem, see the following Avid Knowledge Base article for
assistance:
http://avid.force.com/pkb/articles/en_US/troubleshooting/RabbitMQ-cluster-
troubleshooting
MediaCentral Services 2.4 Upgrading Guide
19
Bringing the Corosync Cluster Online
To complete the upgrade process, the Corosync cluster must be manually brought online.
To bring the cluster back online:
1. Reactivate the cluster by running the following command on the master node only:
sh /opt/avid/cluster/bin/reactivate-cluster
Note: If after running the reactivate-cluster script you observe that a node is the state of
“Offline - Unclean Node”, restart Pacemaker on that node.
Warning: Do not run the reactivate-cluster script in MCS v2.4.0 on Dell servers. Instead,
bring the nodes back online manually. For more information see bug number RE-1065 in the
MCS v2.4.1 ReadMe.
If the reactivate-cluster command does not bring all nodes back online, you can
complete the process manually. For example, to bring the master node back online:
crm node online <node name>
Similarly, you can bring the slave node back online:
crm node online <node name>
If applicable, bring the load-balancing nodes back online:
crm node online <node name>
2. Start the Cluster Resource Monitor to verify the cluster status:
crm_mon -f
• Verify the master, slave, and load-balancing nodes have rejoined the cluster, as
expected.
• Review the fail counts for the following resources (at a minimum): AvidIPC,
AvidUMS, AvidACS, pgsqlDB.
3. If failures are listed, run the cluster resource manager cleanup command to reset them:
crm resource cleanup <rsc> [<node>]
<rsc> is the resource name of interest: AvidIPC, AvidUMS, AvidACS, pgsqlDB (or another)
<node> (optional) is the node of interest.
Note: If you receive an “object/attribute does not exist” error message, it indicates the
resource is active on more than one node. Repeat the command using the group name for
the resource (the “everywhere” form). For example, for the AvidAll resource, use
AvidAllEverywhere. For AvidConnectivityMon, use AvidConnectivityMonEverywhere.
Note: You can address the services contained in the postgres resource group (postgres_fs,
AvidClusterIP and pgsqlDB) individually, or as a group.
For example, to reset the fail count for AvidALL resource, issue the following command:
crm resource cleanup AvidAllEverywhere
MediaCentral Services 2.4 Upgrading Guide
20
4. (Media Index configurations only) If you are upgrading a system that includes Media
Index, you will need to run a script to restart the appropriate services.
/opt/avid/cluster/bin/search-cluster rsc-start
This command only needs to be run once on any node in the cluster.
5. (Multi-Zone configurations only) Once the master zone has been updated, run the
following script from the Master node of the cluster in the Master zone to re-enable the
database syncing between all zones.
/opt/avid/cluster/bin/pgpool_remote_start
Verifying Gluster Volume Permissions
The process below was added to the v2.2.0 MediaCentral Platform Services Installation and
Configuration Guide. Depending on when your system was installed, permissions for some
Gluster volumes may or may not have been set correctly. If your system was originally
installed with MCS 1.8.1, MCS 2.0.x or MCS 2.1.x, review the following procedure.
The following directories must be owned by user maxmin and have group id set to maxmin:
cache/gluster/gluster_data_download
cache/gluster/gluster_data_fl_cache
These directories are associated with RHEL directories (/cache/download and
/cache/fl_cache) used to store files for http-based streaming, such as media converted to
FLV for file-based playback. They are also used to store media converted to Mpeg2TS for
playback on iOS devices. The download directory contains links to simplify iOS playback.
Restarting the Gluster daemon (glusterd) results in user ID and group ownership of the
Gluster volumes being changed from maxmin to root, which breaks playback on iOS devices.
Thus if a cluster node is rebooted, playback issues can arise.
To prevent issues for an existing cluster, you must configure the two Gluster cache volumes
to use the same UID and GID as the maxmin user, as described in the following procedure.
To set Gluster volume ownership so it does not change:
On each node in the cluster perform the following steps (all nodes, any order):
1. Obtain the user ID (uid) of the user maxmin (this might be different on each machine):
id -u maxmin
2. Change the user ownership of the gluster volume to maxmin using the user ID:
gluster volume set gl-cache-dl storage.owner-uid <uid>
gluster volume set gl-cache-fl storage.owner-uid <uid>
In the above commands, do not type the angle brackets. Enter the number obtained in
the previous step.
Note: Do not alter the gl-cache-mcam (multicam) volume. It uses the default root ownership.
3. Obtain the group ID (gid) of the user maxmin (this might be different on each machine):
id -g maxmin
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
/
