Resolved -
During a maintenance window last night, Five9 adjusted their configurations in their US data center to resolve the outstanding degradation of their Digital Engagement (Chat/Email) services. Since changes were implemented, CallTower has received positive reports from our customer base that all services are online and operational.
CallTower will be closing this master case and all associated cases. Within three business days, an Official Root Cause Analysis (RCA) or a Preliminary RCA will be available and provided to customers upon request. If you are still experiencing any further issues, please contact support by email support@calltower.com or by calling 8003475444.
Sep 26, 09:37 MDT
Monitoring -
Five9 has completed its process of mitigating Digital Engagement Services with its US data centers. Five9 has asked all affected users to clear their cache and cookies (All time if possible). Please close and restart all Five9 services and log back in. CallTower is monitoring for any additional reports involving Digital Engagement Services (chat and email) with Five9. If you are still experiencing any issues please reach out to CallTower by emailing support@calltower.com or calling our support line at 8003475444 and we can work with Five9 to resolve.
CallTower has opened a master case for this issue as it is affecting more than one CallTower customer. This issue may or may not directly affect your services. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com and expects the next update at 9:00am MST on 9/26/2023.
Sep 25, 17:25 MDT
Update -
The following is an update from Five9:
Five9 is currently working on restoring services for digital engagement at the moment (email and chat). Please restart your system to see if you are able to run the system. If you are still experiencing downtime due to the outage, please let us know and we shall request for your services to be migrated to the ATL data center.
CallTower has opened a master case for this issue as it is affecting more than one CallTower customer. This issue may or may not directly affect your services. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 60 minutes unless identified otherwise.
Sep 25, 15:58 MDT
Update -
The following is an update from Five9:
Five9 is in the process of restarting Digital Engagement services in the US West data center. During this time Digital Engagement (chat and email) services will be unavailable. The current estimated time for restoration is up to one hour.
CallTower has opened a master case for this issue as it is affecting more than one CallTower customer. This issue may or may not directly affect your services. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 60 minutes, unless identified otherwise.
Sep 25, 14:24 MDT
Update -
CallTower has escalated with Five9 management in an attempt to help drive a resolution for this issue. Five9 continues to work on mitigating a database issue in their US data centers. Digital Engagement Services has been restored for some customers and Five9 is continuing to restore services for all impacted chat and email customers.
CallTower has opened a master case for this issue as it is affecting more than one CallTower customer. This issue may or may not directly affect your services. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 60 minutes, unless identified otherwise.
Sep 25, 13:27 MDT
Update -
The following is an update directly from Five9:
Five9 continues to work on mitigating a database issue in their US data centers. Digital Engagement Services has been restored for some customers and we are continuing to restore services for all impacted chat and email customers. The current estimated time to resolution to regain chat and email services is up to 15 minutes.
CallTower has opened a master case for this issue as it is affecting more than one CallTower customer. This issue may or may not directly affect your services. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 60 minutes, unless identified otherwise.
Sep 25, 12:10 MDT
Identified -
Below is an update directly from Five9:
Five9 continues to work on mitigating a database issue in our US data centers that is causing general application slowness with chat and email transactions, user login timeouts, and other symptoms. Digital Engagement services (chat and email) will not be available during this time. The current estimated time to resolution to regain chat and email services is up to one hour
CallTower has opened a master case for this issue as it is affecting more than one CallTower customer. This issue may or may not directly affect your services. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 60 minutes unless identified otherwise.
Sep 25, 11:13 MDT
Update -
CallTower has escalated and continues to work urgently with Five9 on intermittent issues in an effort to come to a resolution. Customers may experience: Slow service on delivering Emails from queues, inability to send/receive chat messages, inability for some users to log into agent portal. However, Inbound and outbound calls are not affected.
CallTower has opened a master case to investigate this issue as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 60 minutes, unless identified otherwise. This issue is being investigated as a top priority.
Sep 25, 10:18 MDT
Investigating -
CallTower Support has received reports of slow or degraded service with Five9, Currently CallTower is working urgently with Five9 to come to a resolution.
CallTower has opened a master case to investigate this issue as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is under investigation. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 30 minutes unless identified otherwise. This issue is being investigated as a top priority.
Sep 25, 09:23 MDT
Completed -
The scheduled maintenance has been completed.
Sep 18, 04:00 MDT
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Sep 17, 22:01 MDT
Scheduled -
Purpose: Patching and preventative maintenance on multiple systems. Systems Affected:
CT Cloud Voice: -CT Cloud Communicator - September 2023 Update ***Possible Impact: Users will receive notification that a new version is available. CallTower recommends all users update to the latest version to receive up-to-date bug/security fixes and feature updates. -ATL Meta - September 2023 Update ***Possible Impact: Customers on ATL Meta will experience downtime while servers are rebooted. Downtime will be 15 minutes or less. -ATL Meta SBC Upgrade ***Possible Impact: Customers on ATL Meta will experience downtime while servers are rebooted. Downtime will be 15 minutes or less. -CT Cloud Voice - CommPortal UI Refresh ***Possible Impact: CommPortal for CT Cloud Voice will be unavailable while changes are made. Downtime will be 15 minutes or less. -ATL Meta SBC Upgrade ***Possible Impact: Customers on ATL Meta will experience downtime while servers are rebooted. Downtime will be 15 minutes or less.
Sep 12, 13:29 MDT
Completed -
The scheduled maintenance has been completed.
Sep 16, 04:00 MDT
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Sep 15, 22:00 MDT
Scheduled -
Purpose: Patching and preventative maintenance on multiple systems. Systems Affected:
Fax Services: -MasterFax & CT Cloud Fax - General Server Maintenance ***Possible Impact: Customers will experience interruptions to both MasterFax and CT Cloud Fax services during the Maintenance Window.
Cisco: -Renew CUCM Certs All 10.5 Clusters ***Possible Impact: CallTower will be conducting a reboot of the Cisco Unified Communications Manager your company is built on. Servers within your cluster will be rebooted in intervals so as to not disrupt business. If your phone is registered to the server that is being rebooted at the time, your phone will re-register to the next server in the list. For calls in progress, calls in progress 'should' remain up until the call has ended. Once your call has ended, your phone may re-register with the next server in the list depending on how long your call lasted. There is a possibility that once all servers have been rebooted that you would have noticed your phone register with a different server multiple times. Please do not be alarmed as this is normal behavior.
CT Cloud Voice: -Poly Trio EPP Update ***Possible Impact: Users may see their Poly Trio 8300, Trio 8500, Trio 8800 update firmware during the Maintenance Window.
-Routing Maintenance in New York ***Possible Impact: Brief interruptions in services out of the New York datacenter during the Maintenance Window.
Sep 12, 13:28 MDT
Resolved -
CallTower continues to see normal call completion. We have seen no further issues affecting call completion for our Intermedia and Teams customers since 9/11 at 11 am MT. CallTower believes the issue has been resolved but continues to work with Microsoft to identify the root cause.
CallTower will be closing this master case and all associated cases.
Sep 13, 12:37 MDT
Update -
CallTower continues to see successful calls and has not had any new related reports from our customers. CallTower continues to work with our vendors to identify the issue and ensure its resolution.
CallTower has opened a master case to investigate this issue as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltower.com.
Sep 12, 08:50 MDT
Update -
CallTower and customers continue to see normal call completion. Engineering continues to work with associated vendors to ensure the underlying issue has been addressed.
*CallTower has opened a master case to investigate this issue as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltower.com.
Sep 11, 16:16 MDT
Update -
CallTower customers have not reported any call drop issue since 11 am MT. Although customers are no longer able to reproduce the issue, CallTower Engineers continue to investigate this issue that is potentially impacting some of our Intermedia and Teams users. If you are still experiencing the issue please report your call examples to support@calltower.com.
*CallTower has opened a master case to investigate this issue as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 60 minutes, unless identified otherwise. This issue is being investigated as a top priority.
Sep 11, 14:23 MDT
Update -
CallTower continues to work with our Vendor to investigate the dropped calls. This continues to remain our highest priority. We do not yet have an ETA on resolution.
*CallTower has opened a master case to investigate this issue as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 30 minutes, unless identified otherwise. This issue is being investigated as a top priority.
Sep 11, 12:28 MDT
Update -
CallTower and our vendor continue to investigate dropped calls affecting our Teams and Intermedia customers. Please continue to provide examples of the affected calls to aid in our investigation.
*CallTower has opened a master case to investigate this issue as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 30 minutes, unless identified otherwise. This issue is being investigated as a top priority.
Sep 11, 11:47 MDT
Update -
CallTower continues to investigate the issue affecting dropped calls on Intermedia and Teams. We have opened a case with the vendor to assist in our investigation. We don't have an ETA on the resolution yet.
*CallTower has opened a master case to investigate this issue as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 30 minutes, unless identified otherwise. This issue is being investigated as a top priority.
Sep 11, 11:13 MDT
Update -
CallTower engineers are urgently investigating the issue involving dropped calls with our Intermedia and Teams customers. If you are experiencing the issue, please reach out to support@calltower.com with call examples.
*CallTower has opened a master case to investigate this issue as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 30 minutes, unless identified otherwise. This issue is being investigated as a top priority.
Sep 11, 10:39 MDT
Investigating -
CallTower is investigating reports of calls dropping for our Intermedia and Teams customers.
*CallTower has opened a master case to investigate this issue as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 30 minutes, unless identified otherwise. This issue is being investigated as a top priority.
Sep 11, 10:14 MDT