Avid MediaCentral Platform Services 2.3 User guide

Avid MediaCentral

Platform Services

Upgrading Guide Version 2.3.0

This document is a guide to upgrading to MediaCentral Platform Services (MCS) 2.3.0 from a

previous version of MCS or Interplay Central Services (ICS). If you are already running MCS

v2.3.0 and need to upgrade to 2.3.2, see the Avid MediaCentral Platform Services ReadMe

for specific patch installation instructions.

Note: For a list products qualified for use with MCS 2.3, and the supported RHEL operating

system, see the Avid MediaCentral Platform Services v2.3 ReadMe.

Important Information

Avid

recommends that you read all the information in this upgrade guide thoroughly

before installing or using the corresponding software release.

Important: Check the Avid Knowledge Base for the latest updates to this guide and all

related documentation for Avid MediaCentral Platform Services.

http://avid.force.com/pkb/articles/en_US/readme/Avid-MediaCentral-Version-2-3-x-

Documentation

Revision History

Date Revised

Changes Made

November 10, 2015

Added steps to the cluster upgrade process for the RabbitMQ cluster.

Appendix G: Glossary has been removed. For more information on the

systems discussed in this guide, see the MediaCentral Platform Services

Concepts and Clustering Guide.

November 6, 2015

Added note regarding Dell servers to the cluster upgrade process.

August 27, 2015

GlusterFS 3.6.4 support added for MCS 2.3.2.

August 14, 2015

Update to “Restoring System Settings and Migrating User Database”

process. Invalid “-p” switch removed from command.

A prerequisite step was added to the GlusterFS 3.4.4 upgrade procedure.

July 30, 2015

Included GlusterFS 3.4.4 upgrade procedure for MCS 2.3.1.

June 25, 2015

First publication

Updates from the 2.2 Upgrade Guide include:

• Upgrade instructions for the MediaCentral Distribution Service.

• Additional notes on Media | Index and Multi-Zone configurations

MediaCentral Services 2.3 Upgrading Guide

Contents

Important Information ....................................................................................................................... 1

Revision History .................................................................................................................................. 1

Contents ..................................................................................................................................................... 2

Overview .................................................................................................................................................... 4

How to Use this Guide ............................................................................................................................... 4

Operating System ....................................................................................................................................... 5

Obtaining the Software Packages .............................................................................................................. 5

Update Installation versus Full Installation ................................................................................................ 5

Upgrading Multi-Zone Deployments ......................................................................................................... 5

Upgrading Media | Index Deployments .................................................................................................... 6

Upgrading the MediaCentral | Distribution Service (MCDS) ..................................................................... 6

An Important Note on SSL Certificate Passwords ...................................................................................... 7

Backing Up and Restoring SSL Private Keys ............................................................................................... 8

Installing the Closed Captioning Service .................................................................................................... 8

Upgrading GlusterFS .................................................................................................................................. 9

Migrating User Settings from UMS to USS During an Upgrade ............................................................... 12

Upgrade Prerequisites ............................................................................................................................. 12

Upgrade Paths .......................................................................................................................................... 13

Upgrades from 1.2.x or 1.3.x to MCS 2.3 ............................................................................................. 13

Upgrades for systems already running RHEL 6.5 ................................................................................. 13

Upgrades for systems running RHEL 6.3 .............................................................................................. 13

Upgrading a Single Server RHEL 6.5 System to MCS 2.3 .......................................................................... 13

Installing MCS 2.3 ................................................................................................................................. 13

Upgrading a Cluster RHEL 6.5 System to MCS 2.3 ................................................................................... 15

Identifying the Master, Slave and Load-Balancing Nodes .................................................................... 15

Backing-up the User Database and System Settings ............................................................................ 16

Taking the Cluster Offline ..................................................................................................................... 16

Updating the Cluster ............................................................................................................................ 17

Stopping the RabbitMQ Service and Rebooting the Cluster ................................................................ 18

Bringing the Corosync Cluster Online ................................................................................................... 19

Verifying Gluster Volume Permissions ................................................................................................. 20

Removing the Gluster Metadata Cache Replication Volume ............................................................... 21

Upgrading a Single Server RHEL 6.3 System to MCS 2.3 .......................................................................... 22

MediaCentral Services 2.3 Upgrading Guide

Backing-up System Settings.................................................................................................................. 22

Installing RHEL 6.5 and MCS 2.3 ........................................................................................................... 23

Restoring System Settings and Migrating the User Database .............................................................. 24

Additional Upgrade Procedures ........................................................................................................... 25

Upgrading a Cluster RHEL 6.3 System to MCS 2.3 ................................................................................... 27

Identifying the Master, Slave and Load-Balancing Nodes .................................................................... 27

Backing Up the System Settings ........................................................................................................... 28

Taking the Cluster Offline ..................................................................................................................... 29

Upgrading the Software ....................................................................................................................... 29

Restoring System Settings and Migrating the User Database .............................................................. 30

Additional Upgrade Procedures ........................................................................................................... 31

Appendix A: Post Upgrade Notes ............................................................................................................. 33

Reconfiguring MCS for Media Composer | Cloud or Interplay | MAM in an Upgrade from ICS 1.7 or

Earlier ................................................................................................................................................... 33

Reconfiguring MCS for Use with Media Composer | Cloud ............................................................. 33

Reconfiguring MCS for Use with Interplay | MAM ........................................................................... 34

Resetting Layouts After Upgrading from V2.0 to V2.3 ......................................................................... 35

Resetting Layouts After Upgrading from ICS V1.6 or Earlier ................................................................ 35

Appendix B: Migrating the UMS Database to a Test Node ...................................................................... 36

Migrating the 1.6.x (or later) UMS Database ....................................................................................... 36

Migrating the 1.4.x/1.5.x UMS Database ............................................................................................. 37

Appendix C: Mounting and Unmounting the USB Key ............................................................................ 38

Appendix D: Copying Software to the MCS Server .................................................................................. 40

Copying Software Using WinSCP .......................................................................................................... 40

Copying Software Using a USB Drive .................................................................................................... 41

Appendix E: Deleting the RAID 5 .............................................................................................................. 43

Deleting the RAID 5 on an HP Gen9 ..................................................................................................... 43

Deleting the RAID 5 on an HP Gen8 ..................................................................................................... 44

Deleting the RAID 5 on an Dell Server .................................................................................................. 47

Appendix F: Backing Up and Restoring System Settings and the ICS/MCS Database .............................. 51

Copyright and Disclaimer ......................................................................................................................... 55

MediaCentral Services 2.3 Upgrading Guide

Overview

Upgrading to MediaCentral Platform Services (MCS) 2.3 for MediaCentral, Media Composer

| Cloud, and Interplay | MAM, involves the migration of settings and data along with the

software upgrade. Depending on the upgrade path you are faced with, the migration of

settings and data is manual or automated.

Note: Beginning with version 2.0, the term “MediaCentral Platform Services” replaces

“Interplay Central Services.” In addition, the term “MediaCentral Playback Service” replaces

“Interplay Central Playback Service.”

There are two main upgrade paths to MCS 2.3, depending on the currently installed

operating system:

• RHEL 6.5 – If this is the currently installed OS, the upgrade is non-destructive and

largely automated. Manually backing up system settings and MCS Database is not

required, but recommended.

• RHEL 6.3 and earlier – This upgrade involves a clean installation of RHEL 6.5 and

MCS. Manually backing up system settings and MCS database before the upgrade

and restoring them after the upgrade is complete is mandatory.

When the upgrade path involves a RHEL version change, you must also manually

back up any private keys associated with SSL certificates received from a Certificate

Authority, if CA-signed certificates are in use. In the case of a cluster, you must

manually re-create the cluster and reconfigure cache replication (if used).

How to Use this Guide

This guide documents the overall upgrade process for the supported upgrade scenarios. It

must be used in conjunction with the MCS 2.3 Installation and Configuration Guide. For

clustering, the MCS Clustering Guide provides valuable information regarding cluster

operations and post-upgrade verification.

For each step in the overall upgrade process, this document offers the following guidance:

• Includes step-by-step instructions for a procedure

- or -

• Cross-references a section in the MCS 2.3 Installation and Configuration Guide

In the second case, the heading of the relevant section in the MCS 2.3 Installation and

Configuration Guide is provided. Since direct page references are unreliable, you must scan

the table of contents in that guide to find the heading and page.

MediaCentral Services 2.3 Upgrading Guide

Operating System

MCS 2.3 requires Red Hat Enterprise Linux (RHEL) v6.5. In addition, whether performing an

upgrade or a full installation, the MCS installer requires the RHEL 6.5 ISO to be mounted, so

you must obtain the RHEL 6.5 ISO. This is true, even when performing an upgrade where the

operating system does not change.

For convenience, the following table lists the ICS/MCS releases and corresponding RHEL

versions.

ICS/MCS Version

RHEL Version

ICS 1.2.3 – 1.5.0

RHEL 6.2

ICS 1.6.0 – 1.8.0

RHEL 6.3

ICS 1.8.1

RHEL 6.5

MCS 2.0.0 – 2.3.x

RHEL 6.5

Obtaining the Software Packages

Multiple software packages are required to properly install and configure MediaCentral. For

more information on downloading these packages, see “Obtaining the Software” in the MCS

2.3 Installation and Configuration Guide.

Update Installation versus Full Installation

For MCS systems already operating under RHEL v6.5, the upgrade process is fairly straight

forward and requires a minimal amount of time to complete.

For older versions of ICS or MCS operating under RHEL v6.3, the upgrade to MCS 2.3 is closer

to a new installation than it is to an upgrade. The process involves using the MCS

Installation USB Drive to reimage the server and complete a fresh installation of MCS.

Because of this, extra care must be taken to ensure the MCS system settings and user

database are preserved.

Upgrading Multi-Zone Deployments

If your deployment is configured for multi-zone, note the following:

• Upgrade slave zones first. If you cannot bring all zones off-line at once, upgrade the

slave zone(s) before the master zone.

An upgraded slave zone no longer has access to the centralized user management

service (UMS), which is owned by the master zone. Users in the upgraded slave zone

can continue to log in —the local, read-only copy of the UMS database is used —

but you cannot create new users.

• After the master zone has been updated as last zone of the multi-zone setup, the

/opt/avid/cluster/bin/pgpool_remote_start script needs to be run to enable the

proper syncing of the user databases between all zones again. Please be aware, that

some services on the remote zone systems will be restarted which will cause a short

downtime to the users.

MediaCentral Services 2.3 Upgrading Guide

Upgrading Media | Index Deployments

If your deployment is configured for Media Index, note the following:

• Systems configured for Media Index require additional steps for certain upgrade

paths. Review and complete the following steps as appropriate for your installation:

If you are upgrading from MediaCentral v2.1 or earlier, you must delete the indexes

in all zones before the upgrade. Deleting the indexes reverts the index table to a

clean (empty) state. For instructions, see the Avid Media | Index Configuration

Guide.

If you are upgrading from MediaCentral v2.2.x, the indexes are compatible with

MediaCentral v2.3 and do not need to be deleted. However, a migration script must

be run to update the indexes to include new features associated with MCS v2.3. For

instructions, see the Avid Media | Index Configuration Guide.

• To configure the Production Engine Bus Connector (PEBCo) for Media Index on MCS

v2.3.x, you must use the Interplay Administrator tool for Interplay Production v3.3.

The PEBCo v2.3.x supports the Interplay Production Engine v3.1 and later.

Upgrading the MediaCentral | Distribution Service (MCDS)

MCDS is used in MediaCentral | UX configurations which employ a Send To Playback (STP)

workflow. In these instances MCDS must be upgraded.

To upgrade the MediaCentral Distribution Service:

1. MCDS could be in one or more location within your environment. Use the following

process to verify where MCDS is installed:

a. Log into MediaCentral | UX.

See “Logging into MediaCentral” in the MediaCentral Platform Services 2.0

Installation and Configuration Guide for additional information.

b. While logged in as the Administrator, select System Settings from the Layout

selector.

c. In the Settings pane, select Interplay | Production.

d. Make note of the MCDS Service URL. This field lists the location or locations

where MCDS has been installed.

e. Log out of MediaCentral | UX and close the browser.

2. On the system where MCDS has been installed, use the “Programs and Features”

Control Panel to uninstall MCDS.

Note: MCDS v3.3 is a 64bit application. Prior 32bit versions must be manually removed to

avoid the installation of multiple versions on the same system.

3. Navigate to C:\Program Files (x86)\Avid\ and delete the residual “MediaCentral

Distribution Service” folder.

4. Install the updated version of MCDS.

5. Repeat this process if you need to upgrade MCDS on a second host system.

MediaCentral Services 2.3 Upgrading Guide

An Important Note on SSL Certificate Passwords

Interplay Central uses the Secure Sockets Layer (SSL) for its server to browser connections.

Default passwords may have been used by the system to automatically generate a self-

signed certificate. Customized certificates can also be used, including those that have been

issued by a Certificate Authority (CA).

When customized or CA-signed certificates are used, the SSL passwords must be written to

the user-available Application Properties file in the following directory:

/opt/avid/etc/avid/avid-interplay-central/config/application.properties

This should not be confused with the system-reserved Application Properties file, in the

following location:

/opt/avid/avid-interplay-central/config/application.properties

Caution: The user-available Application Properties file is backed up and restored by the system-

backup script. The system-reserved Application Properties file is not backed up. If the SSL

passwords are in the wrong file, SSL configuration will be broken by the upgrade.

To avoid mishaps, double-check the contents of the system-reserved Application Properties

file before proceeding with the upgrade.

To double-check the system-reserved Application Properties file:

1. List the contents of the Application Properties file reserved for use by the system:

less /opt/avid/avid-interplay-

central/config/application.properties

2. Look for the following two lines (they might not be next to each other):

system.org.ops4j.pax.web.ssl.password=OBF\:1lfc1n7n1l1618qm18qo1kxw1n3v1lc6

system.org.ops4j.pax.web.ssl.keypassword=OBF\:1lfc1n7n1l1618qm18qo1kxw1n3v1l

The obfuscated passwords shown above (in bold) are the default system passwords.

3. If the passwords you see match the above, nothing more needs to be done.

4. If the passwords you see are different, it indicates the file has been edited and you must

copy the two modified lines into the following file (you might need to create the file):

/opt/avid/etc/avid/avid-interplay-central/config/application.properties

Copying the modified lines to the user-available Application Properties file ensures they

are preserved during the upgrade.

MediaCentral Services 2.3 Upgrading Guide

Backing Up and Restoring SSL Private Keys

Upgrading can be a “true” (non-destructive) upgrade, or can involve a fresh install of RHEL.

For fresh installs, the system-backup script has been provided to back up and restore system

settings related to MCS, including SSL data (the Jetty keystore and certificates it contains).

However, if you are using a Trusted CA-signed Certificate, a private key (jettyPrivateKey.key)

file was created during the certificate creation process. The procedure suggests that you

back up this file in a safe place, such as the root user home directory (/root) directory. If you

are performing a fresh install of RHEL, be sure to back up this private key to an external

location before performing the upgrade. Backing up private key files is good practice,

whatever the type of upgrade being performed.

Note: If the private key was not backed up elsewhere, it is doubly important to back up the

keystore before performing the upgrade. The private key is used to generate Certificate Signing

Requests (CSRs).

For instructions on creating and installing SSL certificates see the following article:

http://avid.force.com/pkb/articles/en_US/how_to/SSL-Certificates-for-server-to-browser-

connections

To back up and restore the Jetty keystore:

1. Locate the separate private key file used to generate the Certificate Signing Request (if

any).

The default name is jettyPrivateKey.key. During its creation, it was suggested you store it

in the following directory:

/root

2. Copy the private key to a safe location (off the MCS server), restoring it at an

appropriate step in the upgrade.

Installing the Closed Captioning Service

MCS v2.3 introduced a Closed Captioning Service for use with MediaCentral UX. This service

requires a separate installation. Follow the steps in this guide to complete the upgrade to

MCS v2.3. Once the upgrade is complete, the Closed Captioning Service can be added.

See “PART VII: INSTALLING THE CLOSED CAPTIONING SERVICE” in the MCS 2.3 Installation

and Configuration Guide for details.

MediaCentral Services 2.3 Upgrading Guide

Upgrading GlusterFS

MediaCentral Platform Services v2.3.2 introduced support for GlusterFS v3.6.4. This is the

minimum version required for MCS v2.3.2 and higher

MediaCentral Platform Services v2.3.1 introduced support for GlusterFS v3.4.4.

MediaCentral Platform Services v2.0 introduced support for GlusterFS v3.4.0. GlusterFS

v3.4.0 is the minimum version required for MCS v2.0 – v2.3.1.

Note: If you are installing GlusterFS v3.4.4 with MCS v2.3.1, a perquisite package called

“xfsprogs” is required. This software is not automatically installed during the RHEL and MCS

install process. See the MCS 2.3.1 ReadMe for details on how to obtain and install this software

GlusterFS can be upgraded on an existing cluster from v3.4.x to v3.6.4 in one of two ways:

• Upgrading GlusterFS with a reboot

• Upgrading GlusterFS without rebooting

Obtaining the updated GlusterFS files (applies to both processes):

1. Download the GlusterFS software packages according to the “Obtaining the Software”

section of the MediaCentral Platform Services Installation and Configuration Guide.

2. On each cluster node, create a “gluster364” folder:

mkdir /media/installers

mkdir /media/installers/gluster364

3. Copy the GlusterFS v3.6.4 installation files to the “Gluster364” folder.

See “Appendix D: Copying Software to the MCS Server

” for details.

4. Navigate to the folder containing the GlusterFS installation files.

cd media/installers/gluster364

Upgrading GlusterFS with a reboot:

1. Install the Gluster software:

rpm -Uvh *

This installs all Gluster packages in the correct order using a single command.

Note: Ensure there are no other .rpm files in the same directory as the GlusterFS files. If other

.rpm files exist in the same directory, they will also be installed by the above command.

2. Once the updated Gluster files are installed, reboot the servers according to the

recommendations found at:

http://avid.force.com/pkb/articles/en_US/how_to/Shutdown-or-reboot-procedure-for-

a-MediaCentral-cluster

MediaCentral Services 2.3 Upgrading Guide

Upgrading GlusterFS without rebooting:

Depending upon your workflow, the following procedure might decrease downtime

associated with a full reboot of the cluster. However, this process still involves restarting

cluster services on each node which will briefly interrupt client operations. Once the

updated Gluster files are installed, follow the instructions below to restart the appropriate

services and activate the new Gluster software:

The following process will use a four node cluster as an example, where:

• wavd-mcs01 is the master node

• wavd-mcs02 is the slave node

• wavd-mcs03 and wavd-mcs04 are load-balancing nodes

1. Open the cluster resource manager on the master node:

crm_mon –f

2. If there are fail counts listed, run the cluster resource manager cleanup command to

reset them:

crm resource cleanup <rsc> [<node>]

<rsc> is the resource name of interest: AvidIPC, AvidUMS, AvidACS, pgsqlDB (or another)

<node> (optional) is the node of interest.

Note: If you receive an “object/attribute does not exist” error message, it indicates the

resource is active on more than one node. Repeat the command using the group name for

the resource (the “everywhere” form). For example, for the AvidAll resource, use

AvidAllEverywhere. For AvidConnectivityMon, use AvidConnectivityMonEverywhere.

Note: You can address the services contained in the postgres resource group (postgres_fs,

AvidClusterIP and pgsqlDB) individually, or as a group.

For example, to reset the fail count for AvidALL resource, issue the following command:

crm resource cleanup AvidAllEverywhere

3. Stop the cluster services.

If your environment includes load-balancing nodes, stop the cluster services on the last

load-balancing node (wavd-mcs04):

service pacemaker stop && service corosync stop

4. Unmount the GlusterFS shares:

umount /cache/fl_cache

umount /cache/render

umount /cache/download

MediaCentral Services 2.3 Upgrading Guide

5. Update the GlusterFS software to v3.6.4:

rpm -Uvh *

This installs all GlusterFS packages in the correct order using a single command.

Note: Ensure there are no other .rpm files in the same directory as the GlusterFS files. If other

.rpm files exist in the same directory, they will also be installed by the above command.

6. Restart the GlusterFS service:

service glusterd restart

7. Remount the GlusterFS shares:

mount /cache/fl_cache

mount /cache/render

mount /cache/download

8. Restart the cluster services on this node:

service corosync start && service pacemaker start

9. Watch the cluster resource manager on the master node as the load-balancing node

rejoins the cluster.

10. Once the node is reconnected to the cluster and appropriate services are running,

repeat steps 3 – 9 on any additional load-balancing nodes (wavd-mcs03, etc). Perform

this process one node at a time.

11. Once GlusterFS has been upgraded on all load-balancing nodes, repeat steps 3 - 9 on the

cluster slave node (wavd-mcs02).

12. Once GlusterFS has been upgraded on the slave node, close (CTRL-C) the cluster

resource manager on the master node and open it on the slave node.

13. Repeat steps 3 - 9 on the cluster master node (wavd-mcs01).

Stopping the cluster services on the master node will initiate a failover to the slave

node. A brief interruption of service will occur and MediaCentral UX clients will need to

log back in.

14. Once GlusterFS has been upgraded on the master node, watch the crm_mon utility as

the node rejoins the cluster and takes the role of the slave node.

15. If any failures appear in the crm_mon utility, clear the fail counts before proceeding to

the MCS upgrade.

MediaCentral Services 2.3 Upgrading Guide

Migrating User Settings from UMS to USS During an Upgrade

MediaCentral 2.0 introduced a new user database scheme. Upgrades of systems prior to 2.0

need to complete a user database migration using the supplied avid-uss-import script.

• For upgrades from 1.8.0 (RedHat 6.3) or earlier, a backup and restore of the MCS

database is required. The user database migration must be completed after

restoring the MCS database, but before logging in via the MediaCentral | UX user

interface.

• For upgrades from 1.8.1 (RedHat 6.5), a backup and restore of the MCS database is

not required. However, the avid-uss-import script must still be run before logging in

via the MediaCentral | UX user interface.

Note: In a cluster, the user database migration script must be run once on the master node only.

See Appendix F: Backing Up and Restoring System Settings and the ICS/MCS Database for

details on the backup and restore procedure. Procedures for using the avid-uss-import

script are covered in

Upgrades for systems running RHEL 6.3.

Upgrade Prerequisites

Before you starting any MCS upgrade, it is important to ensure you have a healthy

environment. Verify the following as appropriate:

 (All configurations) Ensure you can log into MediaCentral via a web browser and verify

basic functionality.

 (Cluster configurations) Verify the status of RabbitMQ and the ACS Bus.

See “Verifying the Status of RabbitMQ” in the MediaCentral Platform Services

Installation and Configuration Guide for details.

See “Verifying ACS Bus Functionality” in the MediaCentral Platform Services Installation

and Configuration Guide for details.

 (Cluster configurations) Verify the status of the Corosync cluster. Ensure there are no

errors.

See “Monitoring MCS High-Availability” in the MCS 2.3 Installation and Configuration

Guide for details.

MediaCentral Services 2.3 Upgrading Guide

Upgrade Paths

This section presents the upgrade paths for specific deployment options.

Upgrades from 1.2.x or 1.3.x to MCS 2.3

For Interplay Central and/or ICPS 1.2.x and 1.3.x upgrade options, please consult your Avid

representative.

Upgrades for systems already running RHEL 6.5

For upgrading instructions:

• Upgrading a Single Server RHEL 6.5 System to MCS 2.3 on page 13

• Upgrading a Cluster RHEL 6.5 System to MCS 2.3 on page 15.

Upgrades for systems running RHEL 6.3

For upgrading instructions:

• Upgrading a Single Server RHEL 6.3 System to MCS 2.3 on page 22

• Upgrading a Cluster RHEL 6.3 System to MCS 2.3 on page 27.

Upgrading a Single Server RHEL 6.5 System to MCS 2.3

Installing MCS 2.3 on a single server already operating under RHEL 6.5 is a simple automated

update of the affected packages. All MCS system settings are preserved.

Expected downtime: 30 min.

Installing MCS 2.3

1. Prepare and insert the MCS 2.3 installation USB key.

See “Creating the MCS Installation USB Drive” in the MCS 2.3 Installation and

Configuration Guide for details.

2. Mount the USB key and change to the USB mount point.

See “Appendix C: Mounting and Unmounting the USB Key” for details.

3. Although the upgrade is non-destructive, it is a good idea to back-up the MCS database.

This is an optional, but recommended step.

Back up the MCS settings and database using the backup script:

./system-backup.sh –b

See Appendix F: Backing Up and Restoring System Settings and the ICS/MCS Database

for additional details.

MediaCentral Services 2.3 Upgrading Guide

4. Unzip the installation package.

unzip MediaCentral_Services_2.3_Build_XX_Linux.zip

5. Navigate to the directory containing the unzipped package.

cd /media/usb/MediaCentral_Services_2.3_Build_XX_Linux

6. Start the installation script.

./install.sh

Note: Be sure to use the dot-slash (“./”) notation, which tells Linux to look for the installation

script in the current directory. Otherwise you will receive the following error message:

-bash: install.sh: command not found

During the installation process, progress is displayed. The update completes with an

indication of success:

INSTALLATION FINISHED

If any errors were encountered during the upgrade, you can obtain more information in

the installation log at:

/var/log/MediaCentral_Services_<version>Build<number>_Linux.log

7. Check the installation was successful using the ics_version script:

ics_version

Service version numbers are returned as follows:

UMS Version: 2.3.x.x

ICPS Version: 2.3.x.x

ICPS manager Version: 2.3.x.x

ACS Version: 2.3.x.x

System ID: "xxxxxxxxxxx"

ICS installer: 2.3 (Build xx)

Created on <installer creation date>

Note: For precise version numbers for this release, see the MediaCentral Platform 2.3

ReadMe.

Note: If you receive a “command not found” error, logout and log back into the server or SSH

session (PuTTY). This will refresh the login for the upgraded system and should allow the

command to run properly.

Note: The System ID is an 11-digit number used in support calls. You enter it via the

MediaCentral UX user interface. See the MCS 2.3 Installation and Configuration Guide.

8. Unmount the USB key.

See “Appendix C: Mounting and Unmounting the USB Key” for details.

9. (Multi-Zone configurations only) Once the master zone has been updated, run the

following script from the master zone to re-enable the database syncing between all

zones.

/opt/avid/cluster/bin/pgpool_remote_start

MediaCentral Services 2.3 Upgrading Guide

Upgrading a Cluster RHEL 6.5 System to MCS 2.3

Installing MCS 2.3 on a cluster already operating under RHEL 6.5 involves an automated

update of the affected packages. All MCS system settings are preserved. All nodes must be

taken off-line to perform the update.

The cluster upgrade involves the following steps:

 Identifying the Master, Slave and Load-Balancing Nodes

 (optional) Backing-up the User Database and System Settings

 Taking the Cluster Offline

 Updating the Cluster

 Stopping the RabbitMQ Service and Rebooting the Cluster

 Bringing the Corosync Cluster Online

 (if applicable) Verifying Gluster Volume Permissions

 (if applicable) Removing the Gluster Metadata Cache Replication Volume

Expected downtime: 1 hr. for 2 servers, + 30 min. for each additional server.

Note: Prior to upgrading, ensure you have verified the

Upgrade Prerequisites on page 12.

Identifying the Master, Slave and Load-Balancing Nodes

There are three types of nodes in a cluster: master, slave, and load-balancing. The master

“owns” multiple resources such as the cluster IP address. The slave assumes the role of

master in the event of a failover. Additional nodes play a load-balancing role, but can never

take on the role of master

To identify the master, slave, and load-balancing nodes:

1. Verify the current role of each node in the cluster through the Cluster Resource

Monitor. Log in to any machine in the cluster as the root user and type:

crm_mon

2. To identify the master and slave nodes, look for the line containing “Master/Slave Set”.

For example:

Master/Slave Set: ms_drbd_postgres [drbd_postgres]

Masters: [ wavd-mcs01 ]

Slaves: [ wavd-mcs02 ]

In this example, the master node is wavd-mcs01 and the slave node is wavd-mcs02.

3. To identify the load-balancing nodes, look for the line containing “Clone Set”.

For example:

Clone Set: AvidAllEverywhere [AvidAll]

Started: [ wavd-mcs01 wavd-mcs02 wavd-mcs03 wavd-mcs04]

In this example, the load-balancing nodes are wavd-mcs03 and wavd-mcs04.

4. Exit crm_mon by pressing CTRL-C on the keyboard.

MediaCentral Services 2.3 Upgrading Guide

Backing-up the User Database and System Settings

Although the upgrade is non-destructive, it is a good idea to back-up the MCS database.

This is an optional, but recommended step.

To back up the database and settings:

1. Prepare the v2.4 MCS Installation USB Drive and connect it to the cluster master node.

See “Creating the MCS Installation USB Drive” in the MediaCentral Platform Services

Installation and Configuration Guide for details.

2. Mount the USB drive and change to the mount point.

See “Appendix C: Mounting and Unmounting the USB Key

” for details.

Warning: The MCS Installation USB Drive contains all software used to image a new

MediaCentral server. Do not reboot the server with this USB drive attached.

3. Back up the current system settings and MCS database using the backup script.

See Appendix F: Backing Up and Restoring System Settings and the ICS/MCS Database

for additional details.

4. Repeat this process for the slave and all load balancing nodes.

Taking the Cluster Offline

Corosync and RabbitMQ maintain independent clustering mechanisms. Corosync clusters

operate in a master/slave relationship whereas RabbitMQ clusters run in an Active/Active

state where any node in the cluster could be the owner of the RabbitMQ database. This

means that the order in which the nodes join and leave the RabbitMQ cluster is particularly

important. In the following process, you will put each node of the Corosync cluster into

“standby” and you will stop the RabbitMQ resources to pause its clustering functionality.

Note: Take care to note the order in which nodes are taken offline. You will need to bring the

nodes back online in the reverse order. Failure to do so may break the RabbitMQ cluster.

In this process you will put each Corosync cluster node into standby and then stop the

RabbitMQ on each server. Complete this process one node at a time, waiting 30 seconds

between each node. Avid recommends starting with the final node first.

Example: If you have four nodes, begin with node 4 (wavd-mcs04), followed by node 3

(wavd-mcs03).

To bring the cluster into standby mode:

1. Begin by putting the Corosync load-balancing nodes into standby mode and then

stopping the rabbitmq service. Complete this process one node at a time, waiting 30

seconds between each node:

crm node standby <node name>

service rabbitmq-server stop

2. Wait 30 seconds and put the Corosync slave node into standby:

crm node standby <node name>

service rabbitmq-server stop

MediaCentral Services 2.3 Upgrading Guide

3. Wait 30 seconds and put the Corosync master node into standby:

crm node standby <node name>

service rabbitmq-server stop

Updating the Cluster

Upgrade the nodes in the reverse order that you stopped the services. Start with the

Corosync master node, followed by the slave node and finally the load balancing nodes.

To update the cluster:

1. If you have not already done so, prepare the v2.3 MCS Installation USB Drive and

connect it to your first MCS server.

Installing MCS 2.3 from the MCS Installation USB Drive is required as it contains the

RHEL 6.5 ISO required for the upgrade process.

See “Creating the MCS Installation USB Drive” in the MediaCentral Platform Services

Installation and Configuration Guide for details.

2. Mount the USB drive and change to the mount point.

See “Appendix C: Mounting and Unmounting the USB Key

” for details.

Warning: The MCS Installation USB Drive contains all software used to image a new

MediaCentral server. Do not reboot the server with this USB drive attached.

3. Start the installation script.

./install.sh

Note: Be sure to use the dot-slash (“./”) notation, which tells Linux to look for the installation

script in the current directory. Otherwise you will receive the following error message:

-bash: install.sh: command not found

During the installation process, progress is displayed. The update completes with an

indication of success:

INSTALLATION FINISHED

If any errors were encountered during the upgrade, you can obtain more information in

the installation log at:

/var/log/MediaCentral_Services_<version>Build<number>_Linux.log

4. Check the installation was successful using the ics_version script:

ics_version

Service version numbers are returned as follows:

UMS Version: 2.3.x.x

ICPS Version: 2.3.x.x

ICPS manager Version: 2.3.x.x

ACS Version: 2.3.x.x

System ID: "xxxxxxxxxxx"

ICS installer: 2.3 (Build xx)

Created on <installer creation date>

MediaCentral Services 2.3 Upgrading Guide

Note: For precise version numbers for this release, see the Avid MediaCentral Platform

Services 2.3 ReadMe.

Note: If you receive a “command not found” error, logout and log back into the server or SSH

session (PuTTY). This will refresh the login for the upgraded system and should allow the

command to run properly.

5. Unmount the USB key, proceed to the next node, and follow the same pattern:

a. Mount the USB key (and the RHEL image, if it is not on the USB key)

b. Run the update script

c. Verify the ics_version output

d. Unmount the USB key

Stopping the RabbitMQ Service and Rebooting the Cluster

Installing the software update restarted the RabbitMQ service. The services need to be

stopped again before rebooting the cluster. MediaCentral v2.3.0 requires a reboot of the

cluster nodes to enable a system configuration change made in the MCS v2.2.4 release.

To stop RabbitMQ and reboot the cluster:

1. Stop the RabbitMQ services on the final load-balancing node:

service rabbitmq-server stop

Example: If you have four cluster nodes, number 4 is the “final” load-balancing node.

2. Waiting 30 seconds between each node, stop the RabbitMQ services on the next load-

balancing node, then the slave node and finally the master node.

3. Once the RabbitMQ services have been stopped on each node, all nodes in the cluster

must be rebooted. The order in which the nodes are rebooted is important to ensure

RabbitMQ starts successfully. Begin by rebooting the master node:

reboot <master node name>

4. Waiting 30 seconds between each node, reboot the slave node and each load-balancing

node.

reboot <node name>

5. Once all nodes have been rebooted and RabbitMQ has been started on each node,

verify the status of the RabbitMQ service.

See “Verifying the Status of RabbitMQ” in the MediaCentral Platform Services

Installation and Configuration Guide for details.

See “Verifying ACS Bus Functionality” in the MediaCentral Platform Services Installation

and Configuration Guide for details.

Note: The RabbitMQ cluster must be healthy prior to bringing the Corosync cluster back

online. If you suspect there is a problem, see the following Avid Knowledge Base article for

assistance:

http://avid.force.com/pkb/articles/en_US/troubleshooting/RabbitMQ-cluster-

troubleshooting

MediaCentral Services 2.3 Upgrading Guide

Bringing the Corosync Cluster Online

The reboot of the nodes restarted the RabbitMQ service, but the Corosync cluster remains

in standby until manually brought online.

To bring the cluster back online:

1. Once you have verified that the RabbitMQ cluster is functioning normally, reactivate the

cluster by running the following command on the master node only:

sh /opt/avid/cluster/bin/reactivate-cluster

Note: If after running the reactivate-cluster script you observe that a node is the state of

“Offline - Unclean Node”, restart Pacemaker on that node.

Warning: Do not run the reactivate-cluster script in MCS v2.3.0 on Dell servers. Instead,

bring the nodes back online manually. For more information see bug number RE-1065 in the

MCS v2.3.2 ReadMe.

If the reactivate-cluster command does not bring all nodes back online, you can

complete the process manually. For example, to bring the master node back online:

crm node online <node name>

Similarly, you can bring the slave node back online:

crm node online <node name>

If applicable, bring the load-balancing nodes back online:

crm node online <node name>

2. Start the Cluster Resource Monitor to verify the cluster status:

crm_mon -f

• Verify the master, slave, and load-balancing nodes have rejoined the cluster, as

expected.

• Review the fail counts for the following resources (at a minimum): AvidIPC,

AvidUMS, AvidACS, pgsqlDB.

3. If failures are listed, run the cluster resource manager cleanup command to reset them:

crm resource cleanup <rsc> [<node>]

<rsc> is the resource name of interest: AvidIPC, AvidUMS, AvidACS, pgsqlDB (or another)

<node> (optional) is the node of interest.

Note: If you receive an “object/attribute does not exist” error message, it indicates the

resource is active on more than one node. Repeat the command using the group name for

the resource (the “everywhere” form). For example, for the AvidAll resource, use

AvidAllEverywhere. For AvidConnectivityMon, use AvidConnectivityMonEverywhere.

Note: You can address the services contained in the postgres resource group (postgres_fs,

AvidClusterIP and pgsqlDB) individually, or as a group.

For example, to reset the fail count for AvidALL resource, issue the following command:

crm resource cleanup AvidAllEverywhere

MediaCentral Services 2.3 Upgrading Guide

4. (Media | Index configurations only) If you are upgrading a system that includes Media |

Index, you will need to run a script to restart the appropriate services.

/opt/avid/cluster/bin/search-cluster rsc-start

This command only needs to be run once on any node in the cluster.

5. (Multi-Zone configurations only) Once the master zone has been updated, run the

following script from the Master node of the cluster in the Master zone to re-enable the

database syncing between all zones.

/opt/avid/cluster/bin/pgpool_remote_start

Verifying Gluster Volume Permissions

The process below was added to the Installation Guide in version 2.2.0. Depending on when

your system was installed, permissions for some Gluster volumes may or may not have been

set correctly. If your system was originally installed with MCS 1.8.1, MCS 2.0.x or MCS 2.1.x,

review the following procedure.

The following directories must be owned by user maxmin and have group id set to maxmin:

cache/gluster/gluster_data_download

cache/gluster/gluster_data_fl_cache

These directories are associated with RHEL directories (/cache/fl_cache and

/cache/download) used to store files for http-based streaming, such as media converted to

FLV for file-based playback. They are also used to store media converted to Mpeg2TS for

playback on iOS devices. The download directory contains links to simplify iOS playback.

Restarting the Gluster daemon (glusterd) results in user ID and group ownership of the

Gluster volumes being changed from maxmin to root, which breaks playback on iOS devices.

Thus if a cluster node is rebooted, playback issues can arise.

To prevent issues for an existing cluster, you must configure the two Gluster cache volumes

to use the same UID and GID as the maxmin user, as described in the following procedure.

To set Gluster volume ownership:

On each node in the cluster perform the following steps (all nodes, any order):

1. Obtain the user ID (uid) of the user maxmin (this might be different on each machine):

id -u maxmin

2. Change the user ownership of the gluster volume to maxmin using the user ID:

gluster volume set gl-cache-dl storage.owner-uid <uid>

gluster volume set gl-cache-fl storage.owner-uid <uid>

In the above commands, do not type the angle brackets. Enter the number obtained in

the previous step.

Note: Do not alter the gl-cache-mcam (multicam) volume. It uses the default root ownership.

3. Obtain the group ID of the user maxmin (this might be different on each machine):

id -g maxmin

Page is loading ...

Avid MediaCentral Platform Services 2.3 User guide

Related papers

Other documents