Objective of this KB is to answer a few frequent requests/questions Tech support receives in related to the upgrades. Please append the generic questions in the comments so that we can keep this KB updated.
This information is generic and if you have follow-up questions, please create a ticket to us.
The customizations on your instance can affect the below behavior and this is from an OOB perspective.
Schedule/Modify/Cancel the upgrade
QPP and EOL upgrade program
Upgrade didn’t start and it went past the planned start time on the HI change?
Planned start of the change determine when the Assigned WAR version of the instance should be changed in HI instance record.
Upgrade is triggered by the "Upgrade" Job on the instance. So the above Assigned WAR will stay updated on HI instance record till the next Run time of the Scheduled Job
OOB; this job is set to run in an hourly interval.
Instance upgrade can take up to 1 hour to trigger on the instance from the planned start time of the HI Change.
Only when the upgrade is triggered on the instance, Actual Work start time is updated on the HI Change.
You can check the *upgrade jobs by going to the below URL.
Please make sure the value on System ID is blank on the "Upgrade" Job. If there is a value, please make sure that this is selected to --None--
Hardcoding the System ID is a bad approach and there are chances that the node selected can be a far node (from secondary DC in case of HA) and the upgrade can run for long.
Upgrade job will download the WAR file and upgrade the node where this job is triggered. This node is then restarted and the second job "Check Upgrade Script" will be triggered on System startup and will continue the Upgrade. Both these jobs run on worker thread.
Modifying the Next action time after the "Planned start time" of the "Upgrade" job is not a recommended approach as this can create issues.
Please plan your change "Planned start time" about 10-15 minutes before the run time of this job. Or you can adjust the Next action time (minutes), may be 2-3 hours before the "Planned start time" of the change so that it is in lieu with the start of the change and it could run few cycles successfully.
Best process to DRY RUN my upgrade?
To get an ETA/behavior of a PROD upgrade is very important.
To record the correct timelines/behavior there are 2 options:
- Option 1: Initiate a clone from PROD to SUBPROD including (Default is EXCLUDE) attachments, audit and log and then upgrade the SUB PROD instance. Including the above specified tables are very important as these are the largest tables on an instance and index creation/schema changes to this table can take a major chunk of the total upgrade time. If they are excluded, that time cannot be factored into the planned upgrade activity. From London, during a clone you have an option to select the amount of data on task tables( Default is 90 days). Please make sure you are selecting "Full" as we need the full task table to analyze the correct timelines.
- Option 2: Request customer support for a restore of your PROD instance to the SUB PROD instance so that the SUB PROD instance can be an exact replica. SUB PROD can be upgraded and this should give you a definite ETA on the upgrade run time.
Upgrades should be DRY RUN multiple times. If we do a proper testing of upgrade on SUB PROD’s the upgrade on PROD will go fine and the support is available 24X7 to look into any unforeseen issue.
How long will my upgrade run?
This is a broader question and are unable to provide an ETA for this. Every instance is different with Organizational specific customizations and the upgrade time can vary based on the extent of data/customizations on the instance. DRY RUN is the only way to get this information.
What happens on my instance during an upgrade?
During an upgrade, users can still be logged on to the instance and work as usual. There will be minimal impact and user connection will be reset once when the Node to which user is connected to is getting upgraded.
Record creation and records created will not be Impacted by the upgrade. Only the Scripts and the Schema will be modified via upgrade.
One node will be running the whole process by Upgrading the node then triggering the upgrade of the Schema/DB. In parallel, it will run the upgrade on other nodes and after the upgrade, nodes will be restarted resetting the connections.
So once the nodes are restarted, the nodes will be in Newer Version and the DB will not be in Newer version till the Database upgrade is completed. Even though record creations are not interrupted; my personal recommendation will be to restrict users during the upgrade. By this the system can use all the available resources for the upgrade activity.
Can we take ad-hoc backup prior to the upgrade?
Unfortunately, there is not an option to run an ad-hoc backup.
We take separate backups for Primary and Secondary DB (where available).
The backup cycle consists of four weekly full backups and the past 6 days of daily differential backups that provide 28 days of backups. All backups are written to disk, no tapes are used and no backups are sent off site. All the controls that apply to live customer data also apply to backups. If data is encrypted in the live database then it will also be encrypted in the backups."
Your Data is safe with us. In a most unlikely scenario of a data corruption in PROD, we have an option to do a POINT IN TIME restore of PROD instance to your SUB PROD instance by which we can bring back the data on PROD instance to that specific time for data comparison (to a specific minute) on the SUB PROD instance.
Point in time restore is done by restoring the instance to the last available backup and recreating the remaining data specified from the BIN logs/Transaction Logs. This process can take more time. But we can bring the instance to any point/date-time we need in previous 2-3 days (BINLOG retention is 3 days).
PIT restore is a manual process involving multiple teams and this will be initiated in Critical conditions and cannot be treated as a Failover/Recovery/Back-out/Rollback Method.
We do not do PIT restore directly on PROD and we will only fix forward and the PIT will be used only for Data comparison and Data recovery.
Features impacted on the instance/application during an upgrade?
How to restrict users during an upgrade?
There is no OOB option available to restrict users during an upgrade.
There are a few workarounds and customization to achieve this and this should be implemented with your Developers/Partners and out of Customer support scope.
Few options available are as follows:
- If your instance has Single sign on implemented, you can restrict the access via the portal/identity provider for SSO/SAML.
- You can also run a custom script to logout all logged in users prior to the change/upgrade window.
- You can also create a Local Admin account and only admin users can access the Instance with a Local Account of the instance skipping the SAML/SSO login
There is no OOB option available to restrict users during an upgrade.
Monitoring the upgrade from the back-end localhost node log is a non-realistic approach as there will be up-to 150 lines created in the back-end per second.
The best way to Monitor the upgrade is via the Upgrade monitor window. Customer support also use this window to monitor the upgrades because of the above said reason.
Upgrade monitor will tell the number of plugins remaining and an estimate of what is upgrading and an estimate of remaining records per plugin as and when it run.
Node Upgrade is failed/Node is Down
Nodes are just placeholder JVM for the users to access the DB. We do not have data on the nodes and if the nodes are down, there are no impact to the instance and there are automated monitoring and rules in place to sort this out automatically. All the user traffic will be diverted to the available nodes till this is sorted.
If you notice any nodes are having issues/down/failed during the course of upgrade, please wait till the Upgrade change is auto-closed. This will take upto 20-30 minutes from the Upgrade Summary report is generated.
If the issue is not resolved, please call Customer Support and we can try to manually upgrade the node or spin up a new node on the new version and disable the stale one. This will NOT have any impact on the instance.
Can I rollback the upgrade?
Unfortunately, there are NO option to run a rollback if the upgrade was to a different family.
Only Patch upgrades can be rolled back.
If customer support is initiating the Rollback on PROD via Change Management, we will have to test this on a SUB PROD to make sure we are getting the intended result-set. For this, we will restore PROD to a SUB PROD and try the Rollback and get customer confirmation and then do it on PROD.
The rollback does not record schema drops (tables or columns, index drops are ok), re-parenting / column promotion, table truncate, table/column rename, column type changes, or column width decrease. These are blacklisted from the rollback and are not configurable.
More information on the above is on the below documentation:-
From London, instance admin can trigger this :-