Resolved -
CallTower Engineers have successfully increased the bandwidth capacity between CallTower and Five9. CallTower continues to see normal utilization with no audio degradation and considers this issue resolved.
CallTower will be closing this master case and all associated cases. Within three business days, an Official Root Cause Analysis (RCA) or a preliminary RCA will be available and provided to customers upon request.
Nov 28, 15:09 MST
Update -
CallTower continues to monitor the circuit between CallTower and Five9. We currently see an increase in traffic utilization, hovering under 80% utilization and below critical status. CallTower Engineering continues to work urgently to implement a permanent resolution with no current ETA.
CallTower has opened a master case for this issue as it is affecting more than one CallTower customer. This issue may or may not directly affect your services. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every hour unless identified otherwise.
Nov 28, 11:40 MST
Update -
CallTower continues to monitor the circuit between CallTower and Five9. We have seen an increase in traffic utilization but are below thresholds and have not seen any recurrence of audio issues. CallTower Engineering urgently continues to work with our vender to implement a permanent resolution with no current ETA.
CallTower has opened a master case for this issue as it is affecting more than one CallTower customer. This issue may or may not directly affect your services. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every hour, unless identified otherwise.
Nov 28, 09:52 MST
Update -
CallTower continues to monitor the circuit between CallTower and Five9. We continue to see normal traffic utilization and no further call-quality audio issues. CallTower Engineering continues to work with our vendor to implement a permanent resolution.
CallTower has opened a master case for this issue as it is affecting more than one CallTower customer. This issue may or may not directly affect your services. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every hour unless identified otherwise.
Nov 28, 06:16 MST
Monitoring -
CallTower Engineering continues to engage with our vendor to implement a resolution after hours however, currently, there is no ETA on resolution. CallTower will continue to monitor the circuit between CallTower and Five9. However, traffic has normalized, and we are no longer seeing the issue on current calls. CallTower will continue to monitor the connection throughout the evening.
CallTower has opened a master case for this issue as it is affecting more than one CallTower customer. This issue may or may not directly affect your services. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updated on status.calltower.com, next update at 8:00am ET 11/28th
Nov 27, 16:22 MST
Update -
CallTower continues to urgently investigate the issue affecting audio quality for inbound and outbound calls on Five9. We believe that we have identified an issue and Engineering is engaged on a conference call with our vendor working to implement a resolution. Customers can expect calls to connect as expected, however, static may occur intermittently on some inbound/or outbound calls.
CallTower has opened a master case for this issue as it is affecting more than one CallTower customer. This issue may or may not directly affect your services. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 60 minutes, unless identified otherwise.
Nov 27, 14:35 MST
Update -
CallTower is investigating multiple reports of static audio on inbound and outbound calls with Five9. We have engaged our Engineering team to investigate but have not isolated the issue at this time.
CallTower has opened a master case to investigate this issue as it has been reported by more than one CallTower Customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every hour unless identified otherwise. This issue is being investigated as a top priority.
Nov 27, 13:04 MST
Identified -
CallTower Engineers have identified an issue on our connection to Five9 and we are currently working with our vender to quickly resolve.
CallTower has opened a master case for this issue as it is affecting more than one CallTower customer. This issue may or may not directly affect your services. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 60 minutes, unless identified otherwise.
Nov 27, 11:36 MST
Investigating -
CallTower is investigating multiple reports of static audio on inbound and outbound calls with Five9. We have engaged our Engineering team to investigate but have not isolated the issue at this time.
CallTower has opened a master case to investigate this issue as it has been reported by more than one CallTower Customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every hour, unless identified otherwise. This issue is being investigated as a top priority.
Nov 27, 10:44 MST
Resolved -
CallTower has identified the issue and applied a fix which is now allowing outbound calls to route normally. CallTower continues to monitor calls and asks that customers report any continued issues to support@calltower.com.
CallTower will be closing this master case and all associated cases. Within three business days, an Official Root Cause Analysis (RCA) or a Preliminary RCA will be available and provided to customers upon request.
Nov 22, 05:57 MST
Update -
CallTower continues to urgently work on this issue. At this time we have called each of the reporting customers to schedule a time to implement a workaround. This workaround has been tested and does resolve the issue affecting outbound calling for our GCC High customers. CallTower will continue to work directly with the affected customers and provide our next update at 8:00 AM ET.
*CallTower has opened a master case for this issue as it is affecting more than one CallTower customer. This issue may or may not directly affect your services. If you are affected, please report the issue to support@calltower.com.
Nov 21, 19:09 MST
Update -
CallTower continues to work with Microsoft to resolve the issue affecting outbound calls for our GCC High customers. CallTower is reaching out to affected customers to implement a workaround that restores outbound calling functionality. We do not yet have an ETA on resolution. This continues to remain our highest priority.
Nov 21, 18:20 MST
Update -
CallTower is urgently investigating the issue affecting outbound calls for our Microsoft Teams GCC High customers. Our previously mentioned changes did not resolve the issue. We have opened a priority case with Microsoft to assist in the investigation. We do not yet have any eta on the resolution.
*CallTower has opened a master case to investigate this issue as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 60 minutes, unless identified otherwise. This issue is being investigated as a top priority.
Nov 21, 16:18 MST
Update -
CallTower engineers continue to urgently work this issue. We believe we have identified an issue and are currently testing a potential fix in our internal testing environment. We will roll this fix out to the affected customers once confirmed. We do not yet have an ETA.
*CallTower has opened a master case to investigate this issue as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 60 minutes, unless identified otherwise. This issue is being investigated as a top priority.
Nov 21, 15:00 MST
Update -
CallTower continues to urgently investigate this issue. We do not yet have an ETA on the resolution. This continues to only affect a subset of our GCC High customers.
*CallTower has opened a master case to investigate this issue as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 30 minutes, unless identified otherwise. This issue is being investigated as a top priority.
Nov 21, 13:43 MST
Update -
CallTower continues to investigate an issue affecting outbound calling for our GCC High customers. Our engineering teams are urgently working the issue. We ask that customers continue to report failed calls to Support@calltower.com
*CallTower has opened a master case to investigate this issue as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 30 minutes, unless identified otherwise. This issue is being investigated as a top priority.
Nov 21, 12:57 MST
Investigating -
CallTower is investigating reports of GCC High outbound calls failing. This appears to only be affecting a subset of our customers on GCC High. CallTower has made a change and is asking customers to retest and report any failed outbound calls to support@calltower.com
Nov 21, 12:14 MST
Completed -
The scheduled maintenance has been completed.
Nov 20, 04:01 MST
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 19, 22:01 MST
Scheduled -
Purpose: Patching and preventative maintenance on multiple systems. Systems Affected:
CT Cloud Voice:
-CT Cloud Communicator - November 2023 Update ***Possible Impact: CT Cloud Communicator users will receive notification that a new version is available. CallTower recommends all users update to the latest version to receive up-to-date bug/security fixes and feature updates.
-ATL Meta - November 2023 Update ***Possible Impact: Customers on ATL Meta will experience downtime while servers are rebooted. Downtime will be 15 minutes or less.
Nov 10, 16:23 MST
Completed -
The scheduled maintenance has been completed.
Nov 18, 04:00 MST
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 17, 22:01 MST
Scheduled -
Purpose: Patching and preventative maintenance on multiple systems. Systems Affected:
Cisco: -Device pack and Reboot HCS Cluster ***Possible Impact: Cisco Call Manager/Phones - CallTower will be loading a new Device Pack from Cisco to HCS. This includes new Firmware on select models. A reboot of the Cisco Unified Communications Manager is required. Your phone will re-register to the next server in the Server list. Calls in progress 'should' remain up until the call has ended. Phone will most likely re-register multiple times during the MW time. This is expected behavior.
CT Cloud Voice: -CT Cloud Voice - Grandstream Ht8xx EPP Update ***Possible Impact: Users may see their Grandstream HT801, HT801m HT812, HT814, HT818 update firmware during the Maintenance Window.
All Platforms: -Routing maintenance in Los Angeles ***Possible Impact: Brief interruptions in services out of the Los Angeles datacenter during maintenance.
Nov 10, 16:21 MST
Completed -
The scheduled maintenance has been completed.
Nov 15, 00:00 MST
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 14, 23:00 MST
Scheduled -
Dear Valued Customer,
The following is a message from Intermedia, regarding their scheduled maintenance:
As part of our continuous commitment to provide the most highly reliable and secure cloud network available, we will be performing a network upgrade to our infrastructure. This upgrade is to move the backend infrastructure into Network Availability Zones, which provides redundancy and resiliency, resulting in improved availability of your services.
We will cut over to the new Availability Zones at 1 a.m. Eastern Standard Time on November 14th. The cutover will take seconds.
We will be adding IP new endpoints to our system which MAY require you to make some changes if you use the features described below.
New IP endpoints to whitelist and information you will need:
APIs: For customers using API, the IPs listed below will need to be added to your network (firewalls, routers) IE: “Whitelisted”. This can be done ahead of the migration. 64.78.55.19 and 64.78.55.20
FTP: For customers using FTP, the IPs listed below will need to be added to your network (firewalls, routers) IE: “Whitelisted”. This can be done ahead of the migration. 64.78.55.19 and 64.78.55.20
Customer Owned External Storage: For customers using their own (Azure / AWS / other) external cloud storage to store their call recordings, access must be allowed in the storage cloud (ACL/Firewall) to our public IP addresses: 64.78.55.19 and 64.78.55.20
Start time: Nov 14, 01:00 EST
Estimated duration: 1 hour
WHAT DO I NEED TO DO? Whitelist the new IP endpoints and obtain the necessary information, as explained in Intermedia’s maintenance notification section ‘New IP endpoints to whitelist and information you will need’
Please note: This is not a monitored email address. If you find an issue, post update, please open a P1 ticket at support@calltower.com.