Schedules

App Xchange Platform Database API Workflows Schedules
 

We are experiencing delays in schedules that rely on flows to complete. We are also seeing delayed updates to the UI from flows finishing.

Root cause is related to a Vista service pack that released overnight to customers. We have identified the issue, but a queue remains. We have scaled up to process that queue more quickly.

We expect delayed updates for a few more hours.

 

We have been working through our backlog as fast as we can. There was a very large backlog so it took some time. Flows are still running but we have a backlog for the next 15-minutes. We continue to monitor.


 

Queue has returned to normal ranges. All schedules and flows are completing in expected timeframes. This incident has been resolved.

App Xchange Platform Workflows Schedules
 

We are experiencing delays in schedules that rely on flows to complete. We are also seeing delayed updates to the UI from flows finishing.

We have identified the issue, but a queue remains. We have scaled up to process that queue more quickly. We expect delayed updates for about another hour.

 

Queue has returned to normal ranges. All schedules and flows are completing in expected timeframes. This incident has been resolved.

App Xchange Platform Database API Workflows Schedules On-Premise Agent Connectivity Public Connectors
 

We are experiencing an issue with our Partner Portal (http://partnerapp.xchange.trimble.com). This website is used by two partners, Procore and Newforma. The website is offline at the moment and we are looking into it.

  • Procore is unable to review configurations in the Partner Portal.
  • Newforma customers are unable to view or modify project mappings.

We are actively working on the issue and will provide status updates as we have them.

NOTE: No other integrations or services are impacted.


 

We have identified and fixed the issue with Partner Portal. The site is working since 12:15pm CST.

This was an issue in the UI for that site. All other services were working normally across the platform.

App Xchange Platform Database API Workflows Schedules On-Premise Agent Connectivity Public Connectors
 

Dear Customer,


Our team is investigating an issue on the following service(s):

App Xchange Platform


We have identified and address the root cause already, this is a retroactive notification.


During the outage App Network and Direct API, specific flow steps and service runs are encountering failures.


We sincerely apologize for the service interruption and will provide further updates as soon as possible.

 

The team has resolved the primary issue and is monitoring the situation.

10:15 am to 22:15pm GMT

App Xchange Platform Schedules
 

We are experiencing processing delays in schedules that is resulting in some activity to be delayed by as much as 10 minutes. We are actively working to address the issue.

 

We expanded capacity today to address the issues with schedules not starting on time. Queues are returning to normal, metrics are back to regular thresholds, and things are trending in the right direction since 10:51am CST.

Schedule delay is down to 5-mins or less.

We continue to monitor the platform.


 

Schedules are back to normal since 11:33am CST. Processing is now within expected time ranges and this incident is resolved.

App Xchange Platform Database API Workflows Schedules On-Premise Agent Connectivity
 

We have identified an issue impacting Concur or FTP-based integrations. The sync between Concur List and some FTP connections are failing at a higher rate than normal. We are taking steps to mitigate and will deploy a hotfix as soon as possible.

We will post a status update when services for those affected integrations has returned to normal. We expect this to be later today.

 

Hotfix was released and services have returned to normal. The next scheduled Concur List refresh or FTP transfer will work as expected.

 

This incident is resolved.

App Xchange Platform Database API Workflows Schedules
 

We are experiencing elevated error levels since about 11am CST that is impacting legacy services and causing performance delays. We are actively investigating the issue.

 

We have identified the issue and determined that it is limited to the Procore <> Vista legacy integration.

This incident is now limited to the Procore <> Vista legacy integration.

The Viewpoint services for the Procore <> Vista integration are failing to complete at a higher than normal rate. It is not all customers and it is not every hourly run, but the failure rate is higher than normal. Customers would experience a delay in data transfer to Procore and a higher than normal failure rate when exporting (the error may say something generic like failed entirely).

We expect some database maintenance to complete overnight. We will continue to monitor the situation. We plan to leave this incident open and provide an update in the morning.

 

Maintenance completed around 6:51pm late yesterday. Legacy Viewpoint syncs for the Procore <> Vista integration started working normally after that. They ran overnight and through this morning as expected.

This incident has been resolved.

App Xchange Platform Schedules
 
We are investigating an issue preventing some schedules from running to completion. We will provide status updates as we have them.
 

We have identified the issue and the impact is minimal to end customers, but not resolved to our high standards. We are keeping this incident open to review in the morning. Since 5:17am CST 6/25 there have been an abnormal number of schedules that finish a service, but remain in a started status. It will then appear as "Cancelled" in the job logs after a period of time. We will continue to monitor, and provide updates tomorrow morning as we have them.

 
Tonight around 9pm CST we will be performing some maintenance to address this issue. We expect up to 5-10 minutes of delayed processing for some messages but overall should have very little end user impact.
 
Maintenance is complete and it was effective in addressing the root cause of the issue.
App Xchange Platform Schedules
 

Some integrations with file transfers, most notably the Procore <> Vista Financials integration are failing to upload to our file share. We are investigating and will provide status updates as we have them. This is impacting the hourly syncs for this integration since yesterday around 5pm CST. This would also be impacting some Concur services or others that are doing uploads from Vista.

 

We have identified the issue and put a fix in place around 10:15am CST. We are monitoring the services to ensure scheduled runs complete as expected. Customers would have experienced a delay in data transfer from Vista to other system (Procore, Concur, etc.) from 5:22pm CST yesterday to 10:15am CST today.

This incident has been resolved.


App Xchange Platform Database API Workflows Schedules On-Premise Agent Connectivity
 
We have identified an issue affecting only the Newforma <> Procore integration. No other integration is impacted. The Newforma <> Procore integration is not properly returning submittal and RFIs back to Procore from Newforma on some workspaces. We already identified the issue and are working to fix as soon as possible. Once a hotfix is released, the next scheduled run will transfer data to Procore. We will post updates as we have them.
No components were marked as affected by this incident.
 

Fixed the issue around 4:20pm CST. The next cache write for customers impacted will pick up the missing customer data. We continue to monitor.

This incident has been resolved.