All Systems Operational

Updated a few seconds ago

Scheduled Maintenance

CL05 Sandbox-2 MaintenanceScheduled Maintenance

Schedule

March 4, 2020 8:00PM - March 5, 2020 12:00AM MST

Components

Workfront Non-Production

Locations

CL05 - Preview/Sandbox

Description

As previously communicated, we have been making improvements to our core infrastructure. As a part of this effort, we communicated the need to white-list new IP addresses for your firewall rules no later than January 2020. During the maintenance window on Wednesday, March 4th at 8:00 PM Mountain, we will be adding new infrastructure to our services. During this maintenance, access to the Sandbox-2 environment may be unavailable. If you experience any issues after the maintenance has concluded, please contact support for assistance. For information on IP addresses, please see the following help article: https://experience.workfront.com/s/article/Configuring-your-firewall-1140336638

Schedule

March 5, 2020 10:00PM - March 6, 2020 12:00AM MST

Components

Workfront Application Platform

Locations

CL01, CL02, CL03

Description

We will be performing database maintenance the evening of Thursday, March 5th. The Production environment may be unavailable for up to 30 minutes while maintenance takes place. Expected Time of Maintenance: Thursday, March 5th from 10:00 PM to 12:00 Midnight Mountain Time.
CL05 Sandbox-1 MaintenanceScheduled Maintenance

Schedule

March 9, 2020 8:00PM - March 10, 2020 12:00AM MDT

Components

Workfront Non-Production

Locations

CL05 - Preview/Sandbox

Description

As previously communicated, we have been making improvements to our core infrastructure. As a part of this effort, we communicated the need to white-list new IP addresses for your firewall rules no later than January 2020. During the maintenance window on Monday, March 9th at 8:00 PM Mountain, we will be adding new infrastructure to our services. During this maintenance, access to the Sandbox-1 environment may be unavailable. If you experience any issues after the maintenance has concluded, please contact support for assistance. For information on IP addresses, please see the following help article: experience.workfront.com/s/article/Configuring-your-firewall-1140336638
CL05 Preview MaintenanceScheduled Maintenance

Schedule

March 11, 2020 8:00PM - March 12, 2020 12:00AM MDT

Components

Workfront Non-Production

Locations

CL05 - Preview/Sandbox

Description

As previously communicated, we have been making improvements to our core infrastructure. As a part of this effort, we communicated the need to white-list new IP addresses for your firewall rules no later than January 2020. During the maintenance window on Wednesday, March 11th at 8:00 PM Mountain, we will be adding new infrastructure to our services. During this maintenance, access to the Preview environment may be unavailable. If you experience any issues after the maintenance has concluded, please contact support for assistance. For information on IP addresses, please see the following help article: experience.workfront.com/s/article/Configuring-your-firewall-1140336638

Workfront Application Platform




Operational

Workfront API




Operational

Workfront Event Subscription




Operational

Workfront Fusion




Operational

Workfront Library




Operational

WorkfrontDAM




Operational

Workfront Proof




Operational

Workfront Non-Production




Operational

Customer Experience Site




Operational

External Services

Workfront Uptime

99.99%




Workfront-Proof Uptime

100.00%




Avg. Round Trip

77.33ms




Total Requests

4759466.67 Hits




Workfront Uptime

100.00%




Workfront-Proof Uptime

100.00%




Avg. Round Trip

75.05ms




Total Requests

91030502.57 Hits




Workfront Uptime

100.00%




Workfront-Proof Uptime

99.95%




Avg. Round Trip

83.96ms




Total Requests

171353886.33 Hits




Maintenance Windows North America: CL01 | CL02 | CL03 | CL05 | CL06 2nd and 4th Saturday 8:00 pm to 12:00 am MT 1st and 3rd Thursday 8:00 pm to 12:00 am MT EMEA: CL04 1st and 3rd Thursday 8:00 pm to 12:00 am GMT CL01, CL02, CL03, CL05, and CL06 Preview environments refresh on Saturday CL04 Preview environment refreshes on Friday **We infrequently schedule maintenance in the windows listed above. If down time is necessary, it will be scheduled in the corresponding window and announced on this page.**

0

Active Incidents

0

Active Maintenances

4

Upcoming Maintenances

History (Last 30 days)

Delayed Event SubsComponent Disruption - SEV 3

Incident Status

Component Disruption - SEV 3


Components

Workfront Event Subscription


Locations

Workfront Event Subscription




February 27, 2020 2:27PM MST
[Investigating] We are seeing a delayed in event subs. We are currently investigating the root cause.

February 27, 2020 2:58PM MST
[Investigating] We're still working with our engineering team to resolve the issue, starting with restarting associated services to resolve the delay.

February 27, 2020 3:22PM MST
[Monitoring] Event subs have resumed normal operations. We will continue to monitor for anomalies.

February 27, 2020 3:57PM MST
[Resolved] Incident resolved.

Incident Status

Operational


Components

Workfront Application Platform, Workfront Event Subscription


Locations

CL01, CL02, CL03, CL05, Workfront Event Subscription




February 18, 2020 2:22PM MST
[Investigating] We are seeing a delay in Event subscriptions and are investigating the root cause.

February 18, 2020 4:28PM MST
[Investigating] We are still seeing the delay in event subs. We have restarted all pods in event-sub namespace and cleaned up dead logs. We are continuing our investigation on the issue.

February 18, 2020 6:23PM MST
[Monitoring] At this time engineers have restored functionality to the event subscription service. Additional investigation is ongoing, and Workfront will continue to monitor the behavior of event subscriptions. If you continue to experience any issues, please contact support.

February 20, 2020 10:48AM MST
[Resolved] Engineering has completed monitoring over a 24 hour period for this issue, and there have been no further delays in event subscriptions for production lanes. Issue has been resolved successfully.

Incident Status

Component Disruption - SEV 3


Components

Workfront Fusion


Locations

Workfront Fusion




February 18, 2020 4:37AM MST
[Investigating] At this time fusion is experiencing issues that may lead to failure or performance issues. Workfront has been engaged with engineers to resolve these issues.

February 18, 2020 5:19AM MST
[Investigating] We discovered that the Fusion is experiencing issues that may lead to partition ‘write’ transaction abort on Fusion Document cards. We are working to fix the problem.

February 18, 2020 8:27AM MST
[Investigating] Customers are still experiencing Fusion FLOs Document downloading cards issue. We have escalated this issue to engineering within the Fusion product and are continuing to investigate the underlying cause.

February 18, 2020 10:46AM MST
[Identified] The problem has been identified and we are working to alleviate the issue.

February 18, 2020 11:41AM MST
[Monitoring] The issue has been solved and we are currently monitoring affected FLOs.

February 18, 2020 12:39PM MST
[Resolved] We have received confirmation that this issue is now resolved.

Incident Status

Operational


Components

Workfront Fusion


Locations

Workfront Fusion




February 14, 2020 4:45AM MST
[Identified] We have identified that Fusion FLOs are experiencing performance issues. At this time Workfront has been engaged with engineers to resolve these issues. If you continue to experience issues with Fusion, please contact support.

February 14, 2020 4:47AM MST
[Identified] We have identified that Fusion FLOs are experiencing performance issues. At this time Workfront has been engaged with engineers to resolve these issues. If you continue to experience issues with Fusion, please contact support.

February 14, 2020 7:52AM MST
[Identified] Customers are still experiencing Fusion FLOs performance issue. Workfront is working with engineering to identify the underlying cause.

February 14, 2020 10:24AM MST
[Identified] We are working to fix Fusion FLOs performance issues. The node engines have been restart, but had no positive effect. Additional resources are being pulled in from the engineering team.

February 14, 2020 11:54AM MST
[Identified] As of 11:45am mnt Fusion is down and inaccessible. We are on call with engineering to bring Fusion back up.

February 14, 2020 12:22PM MST
[Monitoring] We have blacklisted 4 workflows and bounced engine nodes and flows. Fusion coming back up for those other than the 4 FLOs that were blacklisted.

February 14, 2020 3:47PM MST
[Resolved] Issue has been resolved as of 12:45pm MT

Incident Status

Service Disruption - SEV 1


Components

Workfront Application Platform, Workfront Proof


Locations

CL01, CL02, CL03, CL05, Workfront Proof




February 14, 2020 8:43AM MST
[Identified] Proofing is not available now and Workfront clusters have performance degradation. At this time Workfront has been engaged with engineers to resolve these issues.

February 14, 2020 9:01AM MST
[Identified] Workfront Engineers are working to address the underlying cause. While this work is underway users may experience issues with delayed performance within Workfront and Proofing may be non-responsive or unavailable.

February 14, 2020 9:25AM MST
[Identified] Performance on clusters 1, 2, and 3 for the Workfront application should be working as normal. Cluster 5 is experiencing slowness. Proofing is currently unavailable.

February 14, 2020 9:43AM MST
[Monitoring] Speeds on clusters 1,2,3, and 5 should be normal and proofing service should be restored. We are watching to ensure that the underlying issue has been corrected.

February 14, 2020 9:56AM MST
[Resolved] Services have been restored.

Incident Status

Operational


Components

Workfront Application Platform


Locations

CL01, CL02, CL03, CL06




February 12, 2020 1:14PM MST
[Investigating] Email notifications are delayed and potentially undelivered, based on internal tests. We are performing additional tests and investigating the issue.

February 12, 2020 1:27PM MST
[Investigating] Emails do not appear to have been lost as part of this incident, but should only be delayed. We're continuing to investigate the issue.

February 12, 2020 1:47PM MST
[Resolved] We have investigated and resolved the issue, the results of which have been verified by our internal tests. Please reach out to our support teams if there are unexpected delays with email notifications from this point.

Incident Status

Component Disruption - SEV 3


Components

Workfront Fusion


Locations

Workfront Fusion




February 4, 2020 3:57PM MST
[Monitoring] A small portion of Fusion FLOs have been identified as experiencing performance issues. At this time Workfront has been engaged with engineers to resolve these issues. An infrastructure change has been made and we are investigating the results of this maintenance work. If you continue to experience issues with Fusion, please contact support. We expect an update on this overall issue at approximately 10 AM Mountain on 2/5/2020.

February 5, 2020 9:45AM MST
[Monitoring] Post maintenance behavior continues to be improved. We are still monitoring customer interactions directly and seeing success in many interactions which previously did not succeed. However, we have identified some ongoing issues for a smaller section of interactions and are continuing to follow up on those items. If you have any questions at this time, please contact support.

February 5, 2020 3:54PM MST
[Monitoring] Workfront has verified that while systemic impact preventing FLO execution from functioning has been resolved, ongoing issues with visual indicators may still appear to indicate FLOs have failed or not completed. We are continuing to look into the source of this visual defect and will provide updates when available. If you have additional questions, please contact support.

February 8, 2020 12:18PM MST
[Resolved] At this time FLO execution has returned to normal. If you have any questions or concerns please contact support.

Incident Status

Operational


Components

Workfront Application Platform


Locations

CL01, CL02, CL03, CL05




February 7, 2020 12:54PM MST
[Investigating] Customers on Cl01, Cl02, Cl03 and Cl05 may experience degraded performance when interacting with documents or avatars. We are currently resolving the cause of the issue.

February 7, 2020 1:12PM MST
[Identified] Users may encounter degraded performance on Cl05. We are working on restoring service.

February 7, 2020 1:45PM MST
[Monitoring] Service should be back to normal. We are continuing to monitor this issue.

February 7, 2020 4:47PM MST
[Resolved] The system has returned to normal operations. We will continue to monitor system performance involving documents closely.