LINQ District Nutrition - Service Degradation Update - Status Page

Resolved Sep 22, 2023 7:01 AM PDT

As of 10:00 AM EDT on September 22, all LINQ District Nutrition operations are performing normally, including a release last night with improvements to product and platform stability and performance.

As a follow-up to our POS status updates, LINQ typically processes 4 to 5 million transactions per day for districts’ POS systems. The increased volume from back to school caused a large spike resulting in slowness or delays to a number of districts in processing their transactions. All delayed transactions have been processed, and our support team will continue to work to resolve outstanding support cases and escalations.

We hope you found these updates useful. We will continue to work to improve our communications to our districts and will err on the side of system-wide communication when the impact is not known, even if a limited number of districts are ultimately affected. Please reach out to our support team at 888-464-5467 or email to support@titank12.com on these issues or any other LINQ District Nutrition questions. Thank you again for your partnership with LINQ.

Monitoring Sep 21, 2023 3:31 AM PDT

As of 6:30 am EDT on Sept 21, we continue to see sustained improvement in error rates and performance even as transaction processing numbers increase. We are also continuing to see fewer district reports of new related issues in our phone calls and support cases. Our support team is prioritizing previous reports of missed or duplicated transactions along with any other newly reported urgent issues.

We have seen some prior transactions being processed, so if you have not done so, we continue to advise all district administrators to centrally clear their POS terminal sessions during a non-serving period for any terminals used since August 1. This process should be done before or after serving for the day has been completed. To clear the terminal session for each device, in District Portal:

At a time when the terminal is not being used, go to Configuration > Point of Service > Devices.
Click the pencil to select each device in the list.
Click Clear Session and Save.

When the terminal is turned back on after the session has been cleared, the terminal user will need the District POS ID. This ID will be used for all district POS terminals and is found in District Portal under Configuration > District > District Management > General Tab > POS ID. IMPORTANT: do NOT click to Generate a new POS ID.

We apologize greatly for the inconvenience and disruption to your normal operations and thank all of our districts for their assistance and helpful feedback. Our teams are moving towards full resolution of these issues with the highest urgency. Our phone support response times have improved, but we ask that you please update your existing support cases or submit a new case to support@titank12.com for POS related issues.
For status updates, please have your users subscribe to our status page at https://titank12.hund.io. When you subscribe, check ALL to receive all updates by email.

Monitoring Sep 20, 2023 3:36 AM PDT

As of 6:30 am EDT on Sept 20, we continue to see sustained improvement in error rates and performance. We are also continuing to see fewer district reports of new related issues in our phone calls and support cases. Our support team is prioritizing previous reports of missed or duplicated transactions along with any other newly reported urgent issues.

As a reminder from our previous updates this week, to ensure that all available prior transactions are processed, we advise all district administrators to centrally clear their POS terminal sessions during a non-serving period for any terminals used since August 1. This process should be done before or after serving for the day has been completed. To clear the terminal session for each device, in District Portal:

At a time when the terminal is not being used, go to Configuration > Point of Service > Devices.
Click the pencil to select each device in the list.
Click Clear Session and Save.

When the terminal is turned back on after the session has been cleared, the terminal user will need the District POS ID. This ID will be used for all district POS terminals and is found in District Portal under Configuration > District > District Management > General Tab > POS ID. IMPORTANT: do NOT click to Generate a new POS ID.

We apologize greatly for the inconvenience and disruption to your normal operations and thank all of our districts for their assistance and helpful feedback. Our teams are moving towards full resolution of these issues with the highest urgency. Our phone support response times have improved, but we ask that you please update your existing support cases or submit a new case to support@titank12.com for POS related issues.
For status updates, please have your users subscribe to our status page at https://titank12.hund.io. When you subscribe, check ALL to receive all updates by email.

Investigating Sep 19, 2023 3:40 AM PDT

As of 6:30 am EDT on Sept 19, we continue to see sustained improvement in error rates and performance. We are also continuing to see fewer district reports of new related issues in our phone calls and support cases.

As a reminder from our 9/18 afternoon update, our technical team continues to see blocked transactions being processed from prior serving periods, including last week. To ensure that all available transactions are processed, we advise all district administrators to centrally clear their POS terminal sessions during a non-serving period for any terminals used since August 1. This process should be done before or after serving for the day has been completed. To clear the terminal session for each device, in District Portal:

At a time when the terminal is not being used, go to Configuration > Point of Service > Devices.
Click the pencil to select each device in the list.
Click Clear Session and Save.

When the terminal is turned back on after the session has been cleared, the terminal user will need the District POS ID. This ID will be used for all district POS terminals and is found in District Portal under Configuration > District > District Management > General Tab > POS ID. IMPORTANT: do NOT click to Generate a new POS ID.

We apologize greatly for the inconvenience and disruption to your normal operations and thank all of our districts for their assistance and helpful feedback. Our teams are moving towards full resolution of these issues with the highest urgency. Our phone support response times have improved, but we ask that you please update your existing support cases or submit a new case to support@titank12.com for POS related issues.
For status updates, please have your users subscribe to our status page at https://titank12.hund.io.

Investigating Sep 18, 2023 2:26 PM PDT

As of 5:00 pm EDT on Sept 18, our LINQ District Nutrition technical team continued to see sustained improvement in error rates throughout all serving periods today. We are also continuing to see fewer district reports of new related issues in our phone calls and support cases.

Our technical team continues to see a small number of blocked transactions being processed from prior serving periods. We are in the process of reconciling the transactions with support cases. To ensure that all available transactions are processed, we advise all district administrators to centrally clear their POS terminal sessions during a non-serving period for any terminals used since August 1. This process should be done before or after serving for the day has been completed. To clear the terminal session for each device, in District Portal:

At a time when the terminal is not being used, go to Configuration > Point of Service > Devices.
Click the pencil to select each device in the list.
Click Clear Session and Save.

When the terminal is turned back on after the session has been cleared, the terminal user will need the District POS ID. This ID will be used for all district POS terminals and is found in District Portal under Configuration > District > District Management > General Tab > POS ID. IMPORTANT: do NOT click to Generate a new POS ID.

We apologize greatly for the inconvenience and disruption to your normal operations and thank all of our districts for their assistance and helpful feedback. Our teams are moving towards full resolution of these issues with the highest urgency. Our phone support response times have improved, but we ask that you please update your existing support cases or submit a new case to support@titank12.com for POS related issues.
For status updates, please have your users subscribe to our status page at https://titank12.hund.io.

Updated Sep 18, 2023 3:39 AM PDT

As of 6:30 am EDT on Sept 18, our weekend monitoring continued to show sustained improvement in system performance. Processing the transactions that were blocked by errors continued through the weekend.

Our support team will be working today and through the week to respond to reported issues of missing transactions with information on the data processed since last Wednesday’s release. We know that this is a very high priority issue and apologize greatly for the disruption to your normal operations. Thank you to all districts for their very helpful feedback. Our teams are moving towards full resolution of these issues with the highest urgency.

Please update your existing support cases or submit a new case to support@titank12.com for related issues.
For status updates, please have your users subscribe to our status page at https://titank12.hund.io.

Investigating Sep 15, 2023 2:04 PM PDT

As of 5 pm EDT on Sept 15, our LINQ District Nutrition technical teams continue to see sustained improvement in error rates throughout all serving periods today. We are also continuing to see fewer reports of new related issues in our phone calls and support cases. We will update on status by 6:30 am EDT Monday, Sept. 18.

Our technical team will be working through the weekend to continue to identify and resolve both error and performance issues. They are also still processing the transactions that were blocked by errors. Our support team will be working through the weekend to respond to support cases and reconcile reported missing transactions with this data.

We apologize greatly for the disruption to your normal operations and thank all of our districts for their very helpful feedback. Our teams are moving towards full resolution of these issues with the highest urgency.

Please update your existing support cases or submit a new case to support@titank12.com for related issues.
For status updates, please have your users subscribe to our status page at https://titank12.hund.io.

Investigating Sep 15, 2023 3:46 AM PDT

On Friday, Sept. 15, LINQ District Nutrition may continue to experience slowness in processing during high volume periods, which may intermittently affect processes including POS. Our technical team continues to see improvements in performance with fewer errors, but we did receive district reports on errors from yesterday’s serving sessions that are being investigated and resolved as quickly as possible. We continue to proactively advise having backup manual processes like rosters in place in case of slow processing during serving.

Today and through the weekend our teams will be responding to support cases with information on newly processed transactions. Please continue to update us with any additional information on missing transaction or other issues related to the current performance issues.

We apologize for the continued inconvenience and thank you for your continued feedback and patience as we work towards resolution of all issues. Our phone volumes continue to be high, but response time has improved. Please update your existing support cases or submit a new case to support@titank12.com for related issues.
For status updates, please have your users subscribe to our status page at https://titank12.hund.io.

Monitoring Sep 14, 2023 2:11 PM PDT

As of 5 pm EDT, our LINQ District Nutrition technical teams have seen improvement in error rates throughout all serving periods today. We have also experienced lower phone call and new case volume, but some districts are still experiencing and reporting periods of slowness or errors during processing. We will continue to monitor throughout the evening and will update again early tomorrow, Friday, September 15th to provide any additional updates on system performance.

Our technical team is also still processing the transactions that were blocked by errors, some of which dated back to early August. We will have this data available by district and serving terminal and will work with all districts to reconcile their reported missing transactions with this data.

We apologize for the continued inconvenience and thank you for your continued feedback and patience as we work towards resolution of all issues. Our phone volumes continue to be high, but response time is improving. Please update your existing support cases or submit a new case to support@titank12.com for related issues.
For status updates, please have your users subscribe to our status page at https://titank12.hund.io.

Monitoring Sep 14, 2023 4:11 AM PDT

Today LINQ District Nutrition may continue to experience slowness in processing during high volume periods, which may intermittently affect processes including POS. We do expect some improvement in performance due to corrections in last night’s release, but we continue to proactively advise having backup manual processes like rosters in place in case of slow processing during serving.

Our technical team did confirm the successful correction of a sales tax bug that was producing errors and blocked transactions. In post-release monitoring, they saw both a reduction in errors for improved performance and the processing of previously blocked transactions. We are compiling more information to inform districts that have reported on POS issues.

Our technical team continues to make progress on additional work to improve performance and restore full reliability. Our other corporate teams are working together to reduce our support backlog and improve response times. We are increasing resources daily to provide the highest level of service that we know our customers need and deserve. Thank you for your incredible patience as we make these improvements.

We are fully aware of the disruption and additional work this is causing for our LINQ District Nutrition customers and commit to resolving all issues as quickly as possible. The issues are being addressed with the highest level of urgency across our company. We greatly apologize for the inconvenience and thank you for the feedback and opportunity to improve on all levels.

Our phone volumes continue to be high but response time is improving. Please update your existing support cases or submit a new case to support@titank12.com for related issues.
For status updates, please have your users subscribe to our status page at https://titank12.hund.io.

Problem Identified Sep 13, 2023 11:20 AM PDT

LINQ District Nutrition is experiencing database performance issues that may cause periods of transaction processing slowness or other errors. The technical team is working on all aspects of this issue. We apologize for
the inconvenience and assure you that we are working on resolutions as our highest priority.

During these periods of slowness, we advise that having a backup manual process of tracking transactions by roster is the safest POS method. To create a Homeroom Roster template for use as a serving roster:

Navigate to Reports > Person > Person Data.
Select the Homeroom Roster in the template drop down. The selected columns should reflect Person ID, First Name, Last Name, Homeroom, Grade & 1 Week Grid.
If needed, you may add Balance, Allergy, Religious Restriction, Dietary Restriction and Alert Message by selecting these filters from the Available Columns.
Finally, Group By Homeroom and under the Sort filter list Homeroom and Last Name.
To retain the added filter selections click Update Template towards the bottom of the screen and > Save.

We are experiencing high phone call volume during serving periods due to the technical issues, but our expanded support team resources are working diligently to respond to all calls and cases. Please update previously submitted support cases or submit new cases to support@titank12.com.

Problem Identified Sep 13, 2023 6:07 AM PDT

LINQ District Nutrition continues to experience periods of service degradation and slowness which can intermittently affect processes throughout the TITAN product. More specific information on the cause of these issues is available below. We are aware that many districts have reported issues including missing or duplicated transactions, and we are committed to bringing resolutions for all reported issues with all possible speed.

Today, we do anticipate that TITAN product modules may continue to experience database performance issues that may cause periods of processing slowness. As a result, we advise that having a backup manual process of tracking transactions by roster is the safest POS method for today. Offline POS is available, but to avoid issues with processing transactions we advise processing at 4 pm EDT or later to avoid nationwide serving times.

There are several interdependent causes of these issues, and all are being addressed with the highest level of urgency including a release tonight that will bring some improvement.

First, a subset of our districts that use our POS & G/L services are experiencing transactions that are not syncing from their POS devices to LINQ Nutrition. There is a confirmed and tested product issue for these districts that will be addressed in tonight’s limited release, which is targeted for product issue fixes. This may bring some resolution of blocked transactions and we will be working with individual districts on those until resolution can be reached.

Second, we have seen intermittent and at times sustained periods of performance degradation and transaction syncing errors experienced by many of our districts. The following architectural improvements were started earlier this year, but we have accelerated them to move more quickly to improved stability and performance:
1. Short-Term performance optimization to reduce the overhead in processing POS transactions
2. Releasing an updated transaction processing service to handle millions of transactions daily regardless of performance degradation in other services
3. Moving non-critical data to other data services to reduce the size of our larger databases in order to improve performance
4. Migrating data to new and existing databases to “load balance” our data services and improve transaction throughput by a significant margin.

These actions are in flight, and we expect incremental improvements each day.

We are fully aware of the disruption and additional work that the performance issues are causing for our LINQ District Nutrition customers and commit to resolving all issues as quickly as possible. We greatly apologize for the inconvenience and thank you for your patience. The issues are being addressed with the highest level of urgency across all departments, including our customer support area, where we are adding additional resources to manage customer inquiries and phone calls.

Please update your existing support cases or submit a new case to support@titank12.com for related issues.
For status updates, please have your users subscribe to our status page at https://titank12.hund.io.