CallTower Status
All Systems Operational

About This Site

Welcome to CallTower's home for real-time system status updates. Here you will find a current and historical data on system performance.

Cisco Unified Communications Operational
Cisco Voice Operational
Cisco Jabber ? Operational
VoiceCube Operational
Attendant Console Operational
Skype for Business Operational
Skype for Business - Voice ? Operational
Instant Messenger and Presence Operational
Attendant Console Operational
CT Cloud Operational
CT Cloud SIP ? Operational
CT Cloud Voice ? Operational
Attendant Console Operational
CT Cloud Fax Operational
Contact Center Operational
Clarity Connect Operational
Five9 Operational
Telax Operational
Microsoft 365 Operational
Microsoft Teams - Direct Routing Operational
Conferencing Operational
CT Cloud Meeting Operational
Adobe Connect Operational
iMeet Operational
ON Demand Audio Conferencing Operational
CallTower Connect Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Jun 14, 2021

No incidents reported today.

Jun 13, 2021
Resolved - Microsoft is reporting all services have been restored. Here is their latest update:

Status
Service Restored

Details
Title: Users noticing that calls through Microsoft Teams are going to voicemail

User Impact: Users may notice that calls through Microsoft Teams are going to voicemail.

More info: Affected users with access to the Microsoft Teams web app may have been able to use that platform as an alternate solution.
Some mobile device users may have not received push notifications as expected during this event.

Final status: We've deployed our solution and have confirmed via monitoring our service telemetry that impact has been remediated.

Scope of impact: This issue may have impacted your Microsoft Teams desktop client or mobile app users.

Start time: Friday, June 11, 2021, at 1:00 PM UTC

End time: Sunday, June 13, 2021, at 7:00 PM UTC

Preliminary root cause: A recent configuration change prevented the infrastructure responsible for generating call notifications from confirming an active session for end users, resulting in calls being sent directly to voicemail.

Next steps:
We're reviewing our update procedures to better identify similar issues during our development and testing cycles.

We'll publish a post-incident report within five business days.

CallTower will be closing this master case and all associated cases. CallTower will be seeking an Official Root Cause Analysis (RCA) from Microsoft. We will make a copy of the RCA available to customers after it is received upon request.
Jun 13, 15:40 MDT
Update - Microsoft has provided the following update:

Title: Users noticing that calls through Microsoft Teams are going to voicemail

User Impact: Users may notice that calls through Microsoft Teams are going to voicemail.

More info: This issue primarily impacts users using the Microsoft Teams desktop client, though a subset of mobile app users may also experience impact.

Some affected users with access to the Microsoft Teams web app may be able to use that platform as an alternate solution.

On mobile devices, some users may not receive push notifications as expected.

Current status: We've confirmed via telemetry that Teams Desktop client users should no longer be experiencing impact. For mobile app users, we have determined the change can take longer depending on your device. For Android devices, we recommend users restart their session to expedite recovery. For iOS devices, we expect the change to complete within the next 24 to 48 hours.

Scope of impact: This issue could impact any of your Microsoft Teams desktop client users and some mobile app users.

Start time: Friday, June 11, 2021, 7:00 AM (1:00 PM UTC)

Estimated time to resolve: Based on current progress we expect that iOS users should experience resolution within the next 24-48 hours.

Preliminary root cause: A recent configuration change is preventing the infrastructure responsible for generating call notifications from confirming an active session for end users, resulting in calls being sent directly to voicemail.

Next update by: Sunday, June 13, 2021, 12:00 PM (6:00 PM UTC)


*CallTower has opened a master case for this issue as it is affecting more than one CallTower customer. This issue may or may not directly affect your services. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 30 minutes, unless identified otherwise.
Jun 12, 08:42 MDT
Update - Microsoft has provided the following update:

Title: Users noticing that inbound or outbound calls through Microsoft Teams are going to voicemail

User Impact: Users may notice that inbound or outbound calls through Microsoft Teams are going to voicemail.

More info: This issue primarily impacts users using the Microsoft Teams desktop client, though a subset of mobile app users may also experience impact. Some affected users with access to the Microsoft Teams web app may be able to use that platform as an alternate solution.

Current status: Our telemetry indicates that the rollback is nearly complete and we're seeing resolution for customers. While we're closely monitoring the service to determine if additional actions are required, we recommend users restart their Teams client or app for more immediate relief.

Scope of impact: This issue could impact your Microsoft Teams desktop client users and some mobile app users.

Start time: Friday, June 11, 2021, 7:00 AM (1:00 PM UTC)

Preliminary root cause: A recent configuration change is preventing the infrastructure responsible for generating call notifications from confirming an active session for end users, resulting in calls being sent directly to voicemail.

Next update by: Friday, June 11, 2021, 8:00 PM (6/12/2021, 2:00 AM UTC)

*CallTower has opened a master case for this issue as it is affecting more than one CallTower customer. This issue may or may not directly affect your services. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 30 minutes, unless identified otherwise.
Jun 11, 19:05 MDT
Update - Microsoft has posted an additional update:

Title: Users noticing that inbound or outbound calls through the Microsoft Teams desktop client are going to voicemail

User Impact: Users may notice that inbound or outbound calls through the Microsoft Teams desktop client are going to voicemail.

More info: Some impacted users with access to the Microsoft Teams web app or mobile apps may be able to use that platform as an alternate solution. We're investigating isolated reports that some users are experiencing similar impact when using the mobile app.

Current status: Our telemetry indicates that the rollback is providing relief in some circumstances. Further investigation is underway to determine if additional action is required to fully remediate impact.

Scope of impact: This issue could impact any of your Microsoft Teams desktop client users. Start time: Friday, June 11, 2021, 7:00 AM (1:00 PM UTC)

Preliminary root cause: A recent configuration change is preventing the infrastructure responsible for generating call notifications from confirming an active session for end users, resulting in calls being sent directly to voicemail.

Next update by: Friday, June 11, 2021, 7:30 PM (6/12/2021, 1:30 AM UTC)
Jun 11, 17:53 MDT
Monitoring - Microsoft has provided the following update:

Title: Users noticing that inbound or outbound calls through the Microsoft Teams desktop client are going to voicemail

User Impact: Users may notice that inbound or outbound calls through the Microsoft Teams desktop client are going to voicemail.

More info: Impacted users with access to the Microsoft Teams web app or mobile apps may be able to use that platform as an alternate solution.

Current status: We're continuing to monitor the rollback to determine if this action has remediated impact.

Scope of impact: This issue could impact your Microsoft Teams desktop client users.

Start time: Friday, June 11, 2021, at 1:00 PM UTC

Root cause: A recent configuration change is preventing the infrastructure responsible for generating call notifications from confirming an active session for end users, resulting in calls being sent directly to voicemail.

Next update by: Friday, June 11, 2021, at 11:30 PM UTC


*CallTower has opened a master case for this issue as it is affecting more than one CallTower customer. This issue may or may not directly affect your services. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 30 minutes, unless identified otherwise.
Jun 11, 15:38 MDT
Update - Microsoft has provided the following update:

Current status: We've isolated a recent change that has caused portions of infrastructure to send some Microsoft Teams calls straight to voicemail. We're currently rolling back this change.

Scope of impact: This issue could impact any of your Microsoft Teams desktop client users.

Start time: Friday, June 11, 2021, 7:00 AM (1:00 PM UTC)

Preliminary root cause: A recent configuration change is preventing the infrastructure responsible for generating call notifications from confirming an active session for end users, resulting in calls being sent directly to voicemail.


*CallTower has opened a master case for this issue as it is affecting more than one CallTower customer. This issue may or may not directly affect your services. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 30 minutes, unless identified otherwise.
Jun 11, 14:38 MDT
Identified - Microsoft has provided the following update and potential workaround to use while they continue to troubleshoot the issue:

Title: Inbound and outbound calls through the Microsoft Teams desktop client are going to voicemail

User Impact: Affected users may have inbound or outbound calls through the Microsoft Teams desktop client sent straight to Voicemail

More info: Impacted users with access to the Microsoft Teams web app or mobile apps may be able to use that platform as an alternate solution.

Current status: After additional investigation we've determined that the infrastructure responsible for generating call notifications is unable to confirm an active session for the end user, resulting in calls being sent directly to voicemail. We're investigating further to isolate the source of this issue and determine the steps necessary to address the problem.

Scope of impact: This issue could impact any of your Microsoft Teams desktop client users.

Start time: Friday, June 11, 2021, 7:00 AM (1:00 PM UTC)

Next update by: Friday, June 11, 2021, 3:30 PM (9:30 PM UTC)

Potential Workaround: Affected users who are able can shut down their Teams desktop client to force calls to be directed to their mobile device as a primary recipient platform. They may also use the Teams web application as an alternative way to receive calls. This is a quick update designed to give the latest information on the issue.


CallTower Support is available to work with customers on placing temporary call forwards upon request. Please reach out to Support at 1-800-347-5444 to discuss potential work around options.


*CallTower has opened a master case for this issue as it is affecting more than one CallTower customer. This issue may or may not directly affect your services. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 30 minutes, unless identified otherwise.
Jun 11, 14:07 MDT
Investigating - CallTower has been notified of an issue where incoming calls are not completing for Microsoft Teams – Direct Routing customers.

Microsoft has provided the following information regarding this incident:
We're investigating a reported issue in which some incoming Microsoft Teams calls may be sent straight to voicemail. We'll be providing updates under TM261472 in the admin center.

*CallTower has opened a master case to investigate this issue as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 30 minutes, unless identified otherwise. This issue is being investigated as a top priority.
Jun 11, 13:23 MDT
Jun 12, 2021
Jun 11, 2021
Jun 10, 2021
Completed - The scheduled maintenance has been completed.
Jun 10, 22:00 MDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jun 10, 20:00 MDT
Update - We will be undergoing scheduled maintenance during this time.
Jun 7, 14:43 MDT
Scheduled - CallTower Engineers will perform out-of-cycle maintenance on Thursday, June 10, 2021, starting at 10:00 PM Mountain Time. This maintenance window could last up to 2 hours.

How this will affect your CallTower service:
No customer impact is expected during this window. CallTower will be implementing additional security requirements for CT Cloud passwords and voicemail PINs.

What you need to do:
End Users who have never changed their password and/or voicemail PIN will need to reset them. Passwords and/or PINs should be reset by 6/10/2021 at 9:00 AM Mountain Time in order to continue to have access to CT Cloud Communicator, Commportal, or voicemail.

End Users who have not previously changed their password and/or PIN, and choose not to reset prior to the 9AM deadline, will have their password and/or PIN reset to a random value during the maintenance window. These end users will need to perform password recovery to regain access to CT Cloud Communicator and Commportal.

Instructions on how to reset and recover lost passwords and voicemail PINs can be found in our solution center: https://www.uc.solutions/CT_Cloud/HPBX/CT_Cloud_Voice_User_Password_and_PIN_Management
Jun 10, 22:00 MDT
Jun 9, 2021

No incidents reported.

Jun 8, 2021

No incidents reported.

Jun 7, 2021

No incidents reported.

Jun 6, 2021

No incidents reported.

Jun 5, 2021

No incidents reported.

Jun 4, 2021

No incidents reported.

Jun 3, 2021
Resolved - Five is reporting that the issue has been mitigated.

Please contact CallTower Support by emailing support@calltower.com if your domain continues to experience a problem.
Jun 3, 10:18 MDT
Update - Five9 has provided the following update:

"Five9 engineers are still engaged with the highest degree of urgency. Additional updates will be provided once available."
Jun 3, 09:59 MDT
Update - Five9 has provided the following update:

"Five9 continues to troubleshoot the issue towards resolution. Next update will be provided as soon as more information becomes available."
Jun 3, 09:29 MDT
Update - Five9 has provided the following update:

"Some customers are still experiencing issues logging in to Five9 applications and reporting. Five9 engineers are still engaged with the highest degree of urgency."
Jun 3, 09:05 MDT
Monitoring - Five9 has provided the following update:

“Accessibility to Five9 applications has been mitigated, affected customers should be able to login. We are continuing to troubleshoot issues with reporting and will provide another update in 15 minutes.”
Jun 3, 08:05 MDT
Identified - Five9 has identified an issue affecting agents ability to log into the platform. This issue is affecting customers in the US East Datacenter. CallTower will provide status updates as they become available from Five9.

If you would like numbers forwarded during this issue, please contact CallTower Support at support@calltower.com.
Jun 3, 08:02 MDT
Jun 2, 2021

No incidents reported.

Jun 1, 2021

No incidents reported.

May 31, 2021

No incidents reported.