Highland Networks Status - Incident history https://highland.instatus.com Highland Networks Tue, 15 Nov 2022 06:00:00 +0000 Emergency Telcom Reboot Type: Maintenance Duration: 2 hours Affected Components: Network Infrastructure Nov 15, 08:00:00 GMT+0 - Completed - Maintenance has completed successfully Nov 15, 06:00:01 GMT+0 - Identified - Maintenance is now in progress Nov 15, 06:00:00 GMT+0 - Identified - Our telecommunications operator needs to perform an emergency equipment reboot. This can cause up to a 30 minute outage for the reboot. Type: Maintenance

Duration: 2 hours

Affected Components:

<p><small>Nov <var data-var='date'> 15</var>, <var data-var='time'>08:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Nov <var data-var='date'> 15</var>, <var data-var='time'>06:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Nov <var data-var='date'> 15</var>, <var data-var='time'>06:00:00</var> GMT+0</small><br><strong>Identified</strong> - Our telecommunications operator needs to perform an emergency equipment reboot. This can cause up to a 30 minute outage for the reboot. .</p> ]]> Tue, 15 Nov 2022 06:00:00 +0000 https://highland.instatus.com/maintenance/clah5kvcg6697i0n0jczmq1iy https://highland.instatus.com/maintenance/clah5kvcg6697i0n0jczmq1iy Initial: Fiber Outage Type: Incident Duration: 1 hour and 13 minutes Affected Components: Network Infrastructure, Main Website, VPS Cloud Infrastructure, Billing & Ticket System, DNS Servers Oct 14, 08:00:00 GMT+0 - Resolved - We just resolved the issue! Oct 14, 06:47:19 GMT+0 - Investigating - We are currently investigating this incident. There is currently an impact to our fiber network that is impacting services. We will update once we have an ETR. Oct 14, 07:18:39 GMT+0 - Identified - Update: It appears a bad configuration has been applied to our core systems that is causing issues with connectivity to on our network. We are reverting the configuration now. Oct 14, 07:23:44 GMT+0 - Monitoring - Configuration has been rolled back and we're monitoring the performance. Type: Incident

Duration: 1 hour and 13 minutes

Affected Components: , , , ,

<p><small>Oct <var data-var='date'> 14</var>, <var data-var='time'>08:00:00</var> GMT+0</small><br><strong>Resolved</strong> - We just resolved the issue!.</p> <p><small>Oct <var data-var='date'> 14</var>, <var data-var='time'>06:47:19</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident. There is currently an impact to our fiber network that is impacting services. We will update once we have an ETR. .</p> <p><small>Oct <var data-var='date'> 14</var>, <var data-var='time'>07:18:39</var> GMT+0</small><br><strong>Identified</strong> - Update: It appears a bad configuration has been applied to our core systems that is causing issues with connectivity to on our network. We are reverting the configuration now. .</p> <p><small>Oct <var data-var='date'> 14</var>, <var data-var='time'>07:23:44</var> GMT+0</small><br><strong>Monitoring</strong> - Configuration has been rolled back and we're monitoring the performance. .</p> ]]> Fri, 14 Oct 2022 06:47:19 +0000 https://highland.instatus.com/incident/cl984nnx29656i9ofxq4z6z1l https://highland.instatus.com/incident/cl984nnx29656i9ofxq4z6z1l Network Upgrade 10/01/22 Type: Maintenance Duration: 1 hour Affected Components: Network Infrastructure Oct 2, 05:30:00 GMT+0 - Completed - Maintenance has completed successfully Oct 2, 04:30:00 GMT+0 - Identified - We are going to be upgrading some of our core network equipment during this maintenance window. Please expect a network of outage of around 20 minutes during this maintenance period so we can upgrade some of our core components. Oct 2, 04:30:01 GMT+0 - Identified - Maintenance is now in progress Type: Maintenance

Duration: 1 hour

Affected Components:

<p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>05:30:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>04:30:00</var> GMT+0</small><br><strong>Identified</strong> - We are going to be upgrading some of our core network equipment during this maintenance window. Please expect a network of outage of around 20 minutes during this maintenance period so we can upgrade some of our core components. .</p> <p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>04:30:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> ]]> Sun, 2 Oct 2022 04:30:00 +0000 https://highland.instatus.com/maintenance/cl8qshnqf2733iglntm0lsy22 https://highland.instatus.com/maintenance/cl8qshnqf2733iglntm0lsy22 Informational: Network Peering Communication Type: Incident Duration: 15 days, 14 hours and 17 minutes Affected Components: Network Infrastructure Sep 8, 18:17:27 GMT+0 - Investigating - We are currently working with our telcom operator regarding some issues with network peering. This does not impact your service's ability to connect to the internet, it only impacts the speed in which it does. We have discovered an issue with some international network peering across our network and we are working with our telcom operator to get this resolved as soon as possible. Sep 24, 08:34:33 GMT+0 - Resolved - We just resolved the issue! Type: Incident

Duration: 15 days, 14 hours and 17 minutes

Affected Components:

<p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>18:17:27</var> GMT+0</small><br><strong>Investigating</strong> - We are currently working with our telcom operator regarding some issues with network peering. This does not impact your service's ability to connect to the internet, it only impacts the speed in which it does. We have discovered an issue with some international network peering across our network and we are working with our telcom operator to get this resolved as soon as possible. .</p> <p><small>Sep <var data-var='date'> 24</var>, <var data-var='time'>08:34:33</var> GMT+0</small><br><strong>Resolved</strong> - We just resolved the issue!.</p> ]]> Thu, 8 Sep 2022 18:17:27 +0000 https://highland.instatus.com/incident/cl7tdgjx121994i1o8451dxczz https://highland.instatus.com/incident/cl7tdgjx121994i1o8451dxczz Increased Packet loss Type: Incident Duration: 5 hours and 37 minutes Affected Components: Network Infrastructure Sep 3, 13:27:27 GMT+0 - Investigating - We are currently seeing increased latency, loss of speed, and increased packet loss across many endpoints. We are currently working with our Telco operator to see what is happening. Sep 3, 15:13:22 GMT+0 - Identified - Emergency Maintenance has started! Network services will be briefly down during this maintenance, we will update this incident once resolved. Sep 3, 19:04:00 GMT+0 - Resolved - We just resolved the issue! Type: Incident

Duration: 5 hours and 37 minutes

Affected Components:

<p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>13:27:27</var> GMT+0</small><br><strong>Investigating</strong> - We are currently seeing increased latency, loss of speed, and increased packet loss across many endpoints. We are currently working with our Telco operator to see what is happening. .</p> <p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>15:13:22</var> GMT+0</small><br><strong>Identified</strong> - Emergency Maintenance has started! Network services will be briefly down during this maintenance, we will update this incident once resolved. .</p> <p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>19:04:00</var> GMT+0</small><br><strong>Resolved</strong> - We just resolved the issue!.</p> ]]> Sat, 3 Sep 2022 13:27:27 +0000 https://highland.instatus.com/incident/cl7lxwc4d14062hhlqxwwx3vph https://highland.instatus.com/incident/cl7lxwc4d14062hhlqxwwx3vph Network Optimization and Groom Type: Maintenance Duration: 45 minutes Affected Components: Network Infrastructure Aug 15, 04:00:00 GMT+0 - Identified - Date: 08/13/2022 Time: 12AM - 12:45AM Impact: Up to 45 minutes harddown Description: We are performing a network optimization to our core firewalls to improve network performance overall and prepare for further bandwidth offerings. This will occur during our final rack optimization project. During this period your network connection may be briefly interrupted. Please contact support with any questions or concerns. Aug 15, 04:45:00 GMT+0 - Completed - Maintenance has completed successfully Aug 15, 04:00:01 GMT+0 - Identified - Maintenance is now in progress Type: Maintenance

Duration: 45 minutes

Affected Components:

<p><small>Aug <var data-var='date'> 15</var>, <var data-var='time'>04:00:00</var> GMT+0</small><br><strong>Identified</strong> - Date: 08/13/2022 Time: 12AM - 12:45AM Impact: Up to 45 minutes harddown Description: We are performing a network optimization to our core firewalls to improve network performance overall and prepare for further bandwidth offerings. This will occur during our final rack optimization project. During this period your network connection may be briefly interrupted. Please contact support with any questions or concerns. .</p> <p><small>Aug <var data-var='date'> 15</var>, <var data-var='time'>04:45:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Aug <var data-var='date'> 15</var>, <var data-var='time'>04:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> ]]> Mon, 15 Aug 2022 04:00:00 +0000 https://highland.instatus.com/maintenance/cl6fqpehh44218d8lmial6x7qv https://highland.instatus.com/maintenance/cl6fqpehh44218d8lmial6x7qv Rack Optimization Maintenance Final Type: Maintenance Duration: 2 hours Affected Components: Network Infrastructure Aug 15, 06:00:00 GMT+0 - Completed - Maintenance has completed successfully Aug 15, 04:00:01 GMT+0 - Identified - Maintenance is now in progress Aug 15, 04:00:00 GMT+0 - Identified - Date: 08/13/2022 Time: 12AM EST to 2AM EST Impact: Up to 2 hour hard down Description: This is the first CRQ in our series of optimizations throughout our datacenter. You will be contacted directly if this CRQ impacts you in any particular way as part of our network optimization project. If you are contacted, you will be required to take your server down during the maintenance period or provide prior approval for us to power off the server to perform the maintenance. Your server will not be down the entire period of the maintenance window and we can provide a more exact window if required. Please contact support with any questions. Type: Maintenance

Duration: 2 hours

Affected Components:

<p><small>Aug <var data-var='date'> 15</var>, <var data-var='time'>06:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Aug <var data-var='date'> 15</var>, <var data-var='time'>04:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Aug <var data-var='date'> 15</var>, <var data-var='time'>04:00:00</var> GMT+0</small><br><strong>Identified</strong> - Date: 08/13/2022 Time: 12AM EST to 2AM EST Impact: Up to 2 hour hard down Description: This is the first CRQ in our series of optimizations throughout our datacenter. You will be contacted directly if this CRQ impacts you in any particular way as part of our network optimization project. If you are contacted, you will be required to take your server down during the maintenance period or provide prior approval for us to power off the server to perform the maintenance. Your server will not be down the entire period of the maintenance window and we can provide a more exact window if required. Please contact support with any questions. .</p> ]]> Mon, 15 Aug 2022 04:00:00 +0000 https://highland.instatus.com/maintenance/cl6fqk50q41101d8lmbf7oawit https://highland.instatus.com/maintenance/cl6fqk50q41101d8lmbf7oawit Rack Optimization Maintnenace Series 2 Type: Maintenance Duration: 2 hours Affected Components: Network Infrastructure Aug 12, 06:00:00 GMT+0 - Completed - Maintenance has completed successfully Aug 12, 04:00:01 GMT+0 - Identified - Maintenance is now in progress Aug 12, 04:00:00 GMT+0 - Identified - Date: 08/11/2022 Time: 12AM EST to 2AM EST Impact: Up to 2 hour hard down Description: This is the first CRQ in our series of optimizations throughout our datacenter. You will be contacted directly if this CRQ impacts you in any particular way as part of our network optimization project. If you are contacted, you will be required to take your server down during the maintenance period or provide prior approval for us to power off the server to perform the maintenance. Your server will not be down the entire period of the maintenance window and we can provide a more exact window if required. Please contact support with any questions. Type: Maintenance

Duration: 2 hours

Affected Components:

<p><small>Aug <var data-var='date'> 12</var>, <var data-var='time'>06:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Aug <var data-var='date'> 12</var>, <var data-var='time'>04:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Aug <var data-var='date'> 12</var>, <var data-var='time'>04:00:00</var> GMT+0</small><br><strong>Identified</strong> - Date: 08/11/2022 Time: 12AM EST to 2AM EST Impact: Up to 2 hour hard down Description: This is the first CRQ in our series of optimizations throughout our datacenter. You will be contacted directly if this CRQ impacts you in any particular way as part of our network optimization project. If you are contacted, you will be required to take your server down during the maintenance period or provide prior approval for us to power off the server to perform the maintenance. Your server will not be down the entire period of the maintenance window and we can provide a more exact window if required. Please contact support with any questions. .</p> ]]> Fri, 12 Aug 2022 04:00:00 +0000 https://highland.instatus.com/maintenance/cl6fqj5hy40159dflm3rmfywfd https://highland.instatus.com/maintenance/cl6fqj5hy40159dflm3rmfywfd Client Portal is not responding Type: Incident Duration: 3 minutes Affected Components: Billing & Ticket System Aug 6, 21:58:35 GMT+0 - Investigating - We have received reports that our client portal is not responding on our web servers. We are currently working to resolve this. Aug 6, 21:58:56 GMT+0 - Identified - We've determined the root cause to be related to our database servers. We are working to resolve this currently. Aug 6, 22:02:04 GMT+0 - Resolved - Database connectivity has been resolved and site is functioning properly. Type: Incident

Duration: 3 minutes

Affected Components:

<p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>21:58:35</var> GMT+0</small><br><strong>Investigating</strong> - We have received reports that our client portal is not responding on our web servers. We are currently working to resolve this. .</p> <p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>21:58:56</var> GMT+0</small><br><strong>Identified</strong> - We've determined the root cause to be related to our database servers. We are working to resolve this currently. .</p> <p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>22:02:04</var> GMT+0</small><br><strong>Resolved</strong> - Database connectivity has been resolved and site is functioning properly. .</p> ]]> Sat, 6 Aug 2022 21:58:35 +0000 https://highland.instatus.com/incident/cl6ifts9r26144xdmwglzujzwy https://highland.instatus.com/incident/cl6ifts9r26144xdmwglzujzwy Rack Optimization Maintenance Series 1 Type: Maintenance Duration: 1 hour Affected Components: Network Infrastructure Aug 6, 04:00:01 GMT+0 - Identified - Maintenance is now in progress Aug 6, 04:00:00 GMT+0 - Identified - Date: 08/06/2022 Time: 12AM EST to 1AM EST Impact: Up to 1 hour hard down Description: This is the first CRQ in our series of optimizations throughout our datacenter. You will be contacted directly if this CRQ impacts you in any particular way as part of our network optimization project. If you are contacted, you will be required to take your server down during the maintenance period or provide prior approval for us to power off the server to perform the maintenance. Your server will not be down the entire period of the maintenance window and we can provide a more exact window if required. Please contact support with any questions. Aug 6, 05:00:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 1 hour

Affected Components:

<p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>04:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>04:00:00</var> GMT+0</small><br><strong>Identified</strong> - Date: 08/06/2022 Time: 12AM EST to 1AM EST Impact: Up to 1 hour hard down Description: This is the first CRQ in our series of optimizations throughout our datacenter. You will be contacted directly if this CRQ impacts you in any particular way as part of our network optimization project. If you are contacted, you will be required to take your server down during the maintenance period or provide prior approval for us to power off the server to perform the maintenance. Your server will not be down the entire period of the maintenance window and we can provide a more exact window if required. Please contact support with any questions. .</p> <p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>05:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Sat, 6 Aug 2022 04:00:00 +0000 https://highland.instatus.com/maintenance/cl6fqhvgw38891d8lm5lk63ood https://highland.instatus.com/maintenance/cl6fqhvgw38891d8lm5lk63ood Router Fiber Upgrade Type: Maintenance Duration: 1 hour Affected Components: Network Infrastructure Jul 9, 04:00:00 GMT+0 - Identified - We will be performing an upgrade to the fiber optic components of our core routers to increase bandwidth across our network. Due to the nature of the physical changes that will occur, your network connection may be interrupted for up to 5-15 minutes during this outage window. Please contact our support if you have any questions or concerns regarding this. Jul 9, 04:00:01 GMT+0 - Identified - Maintenance is now in progress Jul 9, 05:00:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 1 hour

Affected Components:

<p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>04:00:00</var> GMT+0</small><br><strong>Identified</strong> - We will be performing an upgrade to the fiber optic components of our core routers to increase bandwidth across our network. Due to the nature of the physical changes that will occur, your network connection may be interrupted for up to 5-15 minutes during this outage window. Please contact our support if you have any questions or concerns regarding this. .</p> <p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>04:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>05:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Sat, 9 Jul 2022 04:00:00 +0000 https://highland.instatus.com/maintenance/cl5b7aslt475946bnd61argovn https://highland.instatus.com/maintenance/cl5b7aslt475946bnd61argovn Automated: High Latency Detected Type: Incident Duration: 3 hours and 33 minutes Affected Components: Network Infrastructure Jul 6, 17:58:56 GMT+0 - Investigating - **Automated Incident**: Our Incident Management system has 10 consecutive minutes of high latency across our network endpoints. Network Operations has been informed and is currently investigating the issue. Further information will be provided once discovered. Jul 6, 18:26:37 GMT+0 - Monitoring - Automated Update: Latency is no longer detected, traffic levels have returned to normal. We will continue to monitor this throughout the day. Jul 6, 21:32:13 GMT+0 - Resolved - We are safe to call this issue resolved after validations. Type: Incident

Duration: 3 hours and 33 minutes

Affected Components:

<p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>17:58:56</var> GMT+0</small><br><strong>Investigating</strong> - **Automated Incident**: Our Incident Management system has 10 consecutive minutes of high latency across our network endpoints. Network Operations has been informed and is currently investigating the issue. Further information will be provided once discovered. .</p> <p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>18:26:37</var> GMT+0</small><br><strong>Monitoring</strong> - Automated Update: Latency is no longer detected, traffic levels have returned to normal. We will continue to monitor this throughout the day. .</p> <p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>21:32:13</var> GMT+0</small><br><strong>Resolved</strong> - We are safe to call this issue resolved after validations. .</p> ]]> Wed, 6 Jul 2022 17:58:56 +0000 https://highland.instatus.com/incident/cl59wm6w611644lnofv2t5pnle https://highland.instatus.com/incident/cl59wm6w611644lnofv2t5pnle Router and Firewall Firmware Update Type: Maintenance Duration: 10 minutes Affected Components: Network Infrastructure Jul 3, 00:49:01 GMT+0 - Completed - Maintenance has completed successfully. Jul 3, 00:38:55 GMT+0 - Identified - Starting the maintenance earlier, services will experience a brief interruption during the failover. Jul 3, 16:00:00 GMT+0 - Identified - We will be performing an update to our firewalls and routers this evening to remediate some known issues with the platform. This maintenance will have a brief impact on the services while the failover occurs in the cluster. Type: Maintenance

Duration: 10 minutes

Affected Components:

<p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>00:49:01</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> <p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>00:38:55</var> GMT+0</small><br><strong>Identified</strong> - Starting the maintenance earlier, services will experience a brief interruption during the failover. .</p> <p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>16:00:00</var> GMT+0</small><br><strong>Identified</strong> - We will be performing an update to our firewalls and routers this evening to remediate some known issues with the platform. This maintenance will have a brief impact on the services while the failover occurs in the cluster. .</p> ]]> Sun, 3 Jul 2022 16:00:00 +0000 https://highland.instatus.com/maintenance/cl54dewo90042rnmzbdttpsct https://highland.instatus.com/maintenance/cl54dewo90042rnmzbdttpsct Network Throughput Upgrade Type: Maintenance Duration: 7 days, 11 hours and 5 minutes Affected Components: Network Infrastructure Jun 12, 06:00:00 GMT+0 - Completed - Maintenance has completed successfully Jun 12, 04:00:01 GMT+0 - Identified - Maintenance is now in progress Jun 4, 18:55:24 GMT+0 - Identified - Maintenance postponed to 6/12/22. Jun 12, 04:00:00 GMT+0 - Identified - Date: 06/05/2022 Time: 12AM - 2AM EST Description: We will be performing a network throughput upgrade on our core network equipment to increase total bandwidth capacity. Impact: 5-10 minutes of downtime due to Fiber switchover. Please contact support if you have any questions. Type: Maintenance

Duration: 7 days, 11 hours and 5 minutes

Affected Components:

<p><small>Jun <var data-var='date'> 12</var>, <var data-var='time'>06:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Jun <var data-var='date'> 12</var>, <var data-var='time'>04:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>18:55:24</var> GMT+0</small><br><strong>Identified</strong> - Maintenance postponed to 6/12/22. .</p> <p><small>Jun <var data-var='date'> 12</var>, <var data-var='time'>04:00:00</var> GMT+0</small><br><strong>Identified</strong> - Date: 06/05/2022 Time: 12AM - 2AM EST Description: We will be performing a network throughput upgrade on our core network equipment to increase total bandwidth capacity. Impact: 5-10 minutes of downtime due to Fiber switchover. Please contact support if you have any questions. .</p> ]]> Sun, 12 Jun 2022 04:00:00 +0000 https://highland.instatus.com/maintenance/cl3ylembb1955mjoraym1q302 https://highland.instatus.com/maintenance/cl3ylembb1955mjoraym1q302 Final: Migration of VPS Servers to New Platform Type: Maintenance Duration: 2 hours Affected Components: VPS Cloud Infrastructure May 13, 04:00:01 GMT+0 - Identified - Maintenance is now in progress May 13, 04:00:00 GMT+0 - Identified - Date: 5/13/22 Time: 12AM to 2AM EST Description: Final Migration of VPS Servers from our legacy system to new system. During this maintenance window, VPS services will need to be shutdown to be moved and the outage may last anywhere from 15-30 minutes per server to prevent data loss. Please contact support if you have any questions. May 13, 06:00:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 2 hours

Affected Components:

<p><small>May <var data-var='date'> 13</var>, <var data-var='time'>04:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>May <var data-var='date'> 13</var>, <var data-var='time'>04:00:00</var> GMT+0</small><br><strong>Identified</strong> - Date: 5/13/22 Time: 12AM to 2AM EST Description: Final Migration of VPS Servers from our legacy system to new system. During this maintenance window, VPS services will need to be shutdown to be moved and the outage may last anywhere from 15-30 minutes per server to prevent data loss. Please contact support if you have any questions. .</p> <p><small>May <var data-var='date'> 13</var>, <var data-var='time'>06:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Fri, 13 May 2022 04:00:00 +0000 https://highland.instatus.com/maintenance/cl2zt0xc11512737j2opjpay1o3o https://highland.instatus.com/maintenance/cl2zt0xc11512737j2opjpay1o3o Billing Panel Database Upgrade Type: Maintenance Duration: 1 hour Affected Components: Billing & Ticket System May 12, 05:00:00 GMT+0 - Completed - Maintenance has completed successfully May 12, 04:00:01 GMT+0 - Identified - Maintenance is now in progress May 12, 04:00:00 GMT+0 - Identified - Date: 05/12/22 Time: 12AM EST - 1AM EST Description: We will be upgrading our database on our billing panel at portal.highland-networks.com. During this time our billing portal will be unavailable. This outage will not impact your services and will only impact our management panel. During this time, support can be contacted by email at support@highland-networks.com Type: Maintenance

Duration: 1 hour

Affected Components:

<p><small>May <var data-var='date'> 12</var>, <var data-var='time'>05:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>May <var data-var='date'> 12</var>, <var data-var='time'>04:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>May <var data-var='date'> 12</var>, <var data-var='time'>04:00:00</var> GMT+0</small><br><strong>Identified</strong> - Date: 05/12/22 Time: 12AM EST - 1AM EST Description: We will be upgrading our database on our billing panel at portal.highland-networks.com. During this time our billing portal will be unavailable. This outage will not impact your services and will only impact our management panel. During this time, support can be contacted by email at support@highland-networks.com.</p> ]]> Thu, 12 May 2022 04:00:00 +0000 https://highland.instatus.com/maintenance/cl2yfsp6y638454j2opjsaii14t https://highland.instatus.com/maintenance/cl2yfsp6y638454j2opjsaii14t Initial Migration of VPS Servers to new platform Type: Maintenance Duration: 4 hours and 50 minutes Affected Components: VPS Cloud Infrastructure May 10, 02:00:01 GMT+0 - Identified - Maintenance is now in progress May 10, 06:49:40 GMT+0 - Completed - Initial Migration is completed. Our final migration is scheduled for 5/13/22. Details are above. May 10, 03:09:30 GMT+0 - Identified - Maintenance duration updated. We have updated the duration of our maintenance window until 4AM EST. This will still have no impact on your VPS Cloud services. May 10, 02:00:00 GMT+0 - Identified - Date: 05/09/2022 Time: 10PM EST to 12AM EST Description: We will be migrating our existing VPS Cloud system to a new platform. This will have no impact on services or management. After the maintenance is done, our management panel will experience a brief 15 minute outage as we upgrade the software on our billing panel side. IMPACT: **No Impact to services on VPS Cloud Platform** May 10, 05:18:51 GMT+0 - Identified - Update: Select VPS servers will need to be rebooted during this migration process due to disk migration incompatibilities. Your instance may be instantly rebooted back to the operating system. Type: Maintenance

Duration: 4 hours and 50 minutes

Affected Components:

<p><small>May <var data-var='date'> 10</var>, <var data-var='time'>02:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>May <var data-var='date'> 10</var>, <var data-var='time'>06:49:40</var> GMT+0</small><br><strong>Completed</strong> - Initial Migration is completed. Our final migration is scheduled for 5/13/22. Details are above. .</p> <p><small>May <var data-var='date'> 10</var>, <var data-var='time'>03:09:30</var> GMT+0</small><br><strong>Identified</strong> - Maintenance duration updated. We have updated the duration of our maintenance window until 4AM EST. This will still have no impact on your VPS Cloud services. .</p> <p><small>May <var data-var='date'> 10</var>, <var data-var='time'>02:00:00</var> GMT+0</small><br><strong>Identified</strong> - Date: 05/09/2022 Time: 10PM EST to 12AM EST Description: We will be migrating our existing VPS Cloud system to a new platform. This will have no impact on services or management. After the maintenance is done, our management panel will experience a brief 15 minute outage as we upgrade the software on our billing panel side. IMPACT: **No Impact to services on VPS Cloud Platform**.</p> <p><small>May <var data-var='date'> 10</var>, <var data-var='time'>05:18:51</var> GMT+0</small><br><strong>Identified</strong> - Update: Select VPS servers will need to be rebooted during this migration process due to disk migration incompatibilities. Your instance may be instantly rebooted back to the operating system..</p> ]]> Tue, 10 May 2022 02:00:00 +0000 https://highland.instatus.com/maintenance/cl2z693jm1284682j2op2co335nh https://highland.instatus.com/maintenance/cl2z693jm1284682j2op2co335nh Billing Panel is inaccessible due to server issue Type: Incident Duration: 1 hour Affected Components: Billing & Ticket System Mar 19, 17:13:00 GMT+0 - Investigating - Billing panel is inaccessible due to a server issue. We are currently working on getting this back up. This has no impact to your service functionality Mar 19, 18:13:00 GMT+0 - Resolved - We just resolved the issue! Type: Incident

Duration: 1 hour

Affected Components:

<p><small>Mar <var data-var='date'> 19</var>, <var data-var='time'>17:13:00</var> GMT+0</small><br><strong>Investigating</strong> - Billing panel is inaccessible due to a server issue. We are currently working on getting this back up. This has no impact to your service functionality.</p> <p><small>Mar <var data-var='date'> 19</var>, <var data-var='time'>18:13:00</var> GMT+0</small><br><strong>Resolved</strong> - We just resolved the issue!.</p> ]]> Sat, 19 Mar 2022 17:13:00 +0000 https://highland.instatus.com/incident/cl2z6mvmt1293075j2opf5noh15i https://highland.instatus.com/incident/cl2z6mvmt1293075j2opf5noh15i INFORMATIONAL: Fiber Ring Upgrade Type: Maintenance Duration: 2 hours Affected Components: Network Infrastructure Mar 11, 07:00:00 GMT+0 - Completed - Maintenance has completed successfully Mar 11, 05:00:00 GMT+0 - Identified - We are performing updates to our fiber ring and traffic will be redirected onto the other half of the fiber ring during this time. There will be no outage on your services. There may be increased latency during this period between 50-100ms. Mar 11, 07:00:00 GMT+0 - Completed - Maintenance has completed successfully Mar 11, 05:00:01 GMT+0 - Identified - Maintenance is now in progress Type: Maintenance

Duration: 2 hours

Affected Components:

<p><small>Mar <var data-var='date'> 11</var>, <var data-var='time'>07:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Mar <var data-var='date'> 11</var>, <var data-var='time'>05:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are performing updates to our fiber ring and traffic will be redirected onto the other half of the fiber ring during this time. There will be no outage on your services. There may be increased latency during this period between 50-100ms..</p> <p><small>Mar <var data-var='date'> 11</var>, <var data-var='time'>07:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Mar <var data-var='date'> 11</var>, <var data-var='time'>05:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> ]]> Fri, 11 Mar 2022 05:00:00 +0000 https://highland.instatus.com/maintenance/cl07g4oy3296602e6lhdbbq73vk https://highland.instatus.com/maintenance/cl07g4oy3296602e6lhdbbq73vk INFORMATIONAL: Network Security Maintenance Type: Maintenance Duration: 1 hour Affected Components: Network Infrastructure Mar 3, 06:00:00 GMT+0 - Completed - Maintenance has completed successfully Mar 3, 05:00:01 GMT+0 - Identified - Maintenance is now in progress Mar 3, 05:00:00 GMT+0 - Identified - We are performing security patches on some of our network hardware. There is no predicted impact to your services. Mar 3, 06:00:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 1 hour

Affected Components:

<p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>06:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>05:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>05:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are performing security patches on some of our network hardware. There is no predicted impact to your services. .</p> <p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>06:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Thu, 3 Mar 2022 05:00:00 +0000 https://highland.instatus.com/maintenance/cl07g2p36295383e6lh39chyk9s https://highland.instatus.com/maintenance/cl07g2p36295383e6lh39chyk9s Network Bandwidth Upgrade - 2/21/22 Type: Maintenance Duration: 5 days and 52 minutes Affected Components: Network Infrastructure Feb 22, 06:19:30 GMT+0 - Completed - Maintenance has completed successfully. All Services are restored. Feb 17, 05:27:10 GMT+0 - Identified - Correction: Maintenance will occur Monday going onto Tuesday at 12AM. Feb 21, 05:00:00 GMT+0 - Identified - We are planning on upgrading one of our main 10GBPS links to support more traffic. Network appliances will need to be re-configured during this time period and modules replaced throughout our network. We estimate this maintenance will last 1hr-1hr and 30 minutes. Feb 22, 05:01:40 GMT+0 - Identified - Maintenance is now in progress. Feb 22, 04:50:20 GMT+0 - Identified - ATTN: Maintenance will be starting in 10 minutes. This will last around 90 minutes and there will be a network outage during this window. We will inform you once it has been completed. Type: Maintenance

Duration: 5 days and 52 minutes

Affected Components:

<p><small>Feb <var data-var='date'> 22</var>, <var data-var='time'>06:19:30</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully. All Services are restored..</p> <p><small>Feb <var data-var='date'> 17</var>, <var data-var='time'>05:27:10</var> GMT+0</small><br><strong>Identified</strong> - Correction: Maintenance will occur Monday going onto Tuesday at 12AM. .</p> <p><small>Feb <var data-var='date'> 21</var>, <var data-var='time'>05:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning on upgrading one of our main 10GBPS links to support more traffic. Network appliances will need to be re-configured during this time period and modules replaced throughout our network. We estimate this maintenance will last 1hr-1hr and 30 minutes. .</p> <p><small>Feb <var data-var='date'> 22</var>, <var data-var='time'>05:01:40</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress..</p> <p><small>Feb <var data-var='date'> 22</var>, <var data-var='time'>04:50:20</var> GMT+0</small><br><strong>Identified</strong> - ATTN: Maintenance will be starting in 10 minutes. This will last around 90 minutes and there will be a network outage during this window. We will inform you once it has been completed. .</p> ]]> Mon, 21 Feb 2022 05:00:00 +0000 https://highland.instatus.com/maintenance/ckzoc6zfz92139ebn2jbzzj397 https://highland.instatus.com/maintenance/ckzoc6zfz92139ebn2jbzzj397 Network Wide: Fiber Cut Type: Incident Duration: 2 hours and 12 minutes Affected Components: Billing & Ticket System, DNS Servers, Main Website, Network Infrastructure, VPS Cloud Infrastructure Feb 11, 05:10:00 GMT+0 - Investigating - Segra is reporting a fiber cut on one of the main backbones that occurred at 12:10 AM. Techs are on site investigating the damage. The facility has not provided an estimated time to restore. Feb 11, 06:16:32 GMT+0 - Identified - Update: Construction crews have been dispatched to replace the cut fiber. I should have another update in 30-40 minutes. Feb 11, 07:22:03 GMT+0 - Resolved - Fiber cut has been repaired by construction teams. Datacenter connectivity has been restored. Feb 11, 07:01:26 GMT+0 - Identified - Update: Construction teams are still en-route to the source of the fiber cut. Type: Incident

Duration: 2 hours and 12 minutes

Affected Components: , , , ,

<p><small>Feb <var data-var='date'> 11</var>, <var data-var='time'>05:10:00</var> GMT+0</small><br><strong>Investigating</strong> - Segra is reporting a fiber cut on one of the main backbones that occurred at 12:10 AM. Techs are on site investigating the damage. The facility has not provided an estimated time to restore..</p> <p><small>Feb <var data-var='date'> 11</var>, <var data-var='time'>06:16:32</var> GMT+0</small><br><strong>Identified</strong> - Update: Construction crews have been dispatched to replace the cut fiber. I should have another update in 30-40 minutes. .</p> <p><small>Feb <var data-var='date'> 11</var>, <var data-var='time'>07:22:03</var> GMT+0</small><br><strong>Resolved</strong> - Fiber cut has been repaired by construction teams. Datacenter connectivity has been restored. .</p> <p><small>Feb <var data-var='date'> 11</var>, <var data-var='time'>07:01:26</var> GMT+0</small><br><strong>Identified</strong> - Update: Construction teams are still en-route to the source of the fiber cut. .</p> ]]> Fri, 11 Feb 2022 05:10:00 +0000 https://highland.instatus.com/incident/ckzhyx5vy108131dvlxwrikey18 https://highland.instatus.com/incident/ckzhyx5vy108131dvlxwrikey18 Extended Maintenance Window Tonight 12-4 Type: Maintenance Duration: 3 hours and 16 minutes Affected Components: Billing & Ticket System, DNS Servers, Network Infrastructure, VPS Cloud Infrastructure Jan 21, 06:13:06 GMT+0 - Identified - Maintenance is completed on our Billing & Ticket System. In Progress: DNS Servers VPS Infrastructure Jan 21, 07:57:25 GMT+0 - Identified - Firewall cutover is being performed now. Expect connectivity loss as we replace a primary firewall appliance. Jan 21, 05:13:46 GMT+0 - Identified - Routine Maintenance is under way. Impacted systems: Billing and Ticket System DNS Servers. Jan 21, 05:00:00 GMT+0 - Identified - We are planning for a scheduled maintenance during that time. Jan 21, 08:16:17 GMT+0 - Completed - Maintenance has been completed. Network Maintenance was successful. Type: Maintenance

Duration: 3 hours and 16 minutes

Affected Components: , , ,

<p><small>Jan <var data-var='date'> 21</var>, <var data-var='time'>06:13:06</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is completed on our Billing & Ticket System. In Progress: DNS Servers VPS Infrastructure .</p> <p><small>Jan <var data-var='date'> 21</var>, <var data-var='time'>07:57:25</var> GMT+0</small><br><strong>Identified</strong> - Firewall cutover is being performed now. Expect connectivity loss as we replace a primary firewall appliance. .</p> <p><small>Jan <var data-var='date'> 21</var>, <var data-var='time'>05:13:46</var> GMT+0</small><br><strong>Identified</strong> - Routine Maintenance is under way. Impacted systems: Billing and Ticket System DNS Servers. .</p> <p><small>Jan <var data-var='date'> 21</var>, <var data-var='time'>05:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance during that time..</p> <p><small>Jan <var data-var='date'> 21</var>, <var data-var='time'>08:16:17</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has been completed. Network Maintenance was successful. .</p> ]]> Fri, 21 Jan 2022 05:00:00 +0000 https://highland.instatus.com/maintenance/ckyny0cfd39217dlodc63v3cyy https://highland.instatus.com/maintenance/ckyny0cfd39217dlodc63v3cyy Rack 01 - Rack Switch Down Type: Incident Duration: 6 minutes Affected Components: Network Infrastructure Jan 17, 14:26:00 GMT+0 - Investigating - Rack 1's main switch has crashed due to a configuration fault. We are currently working on resolving the configuration fault and reverting the switch back to a recent config. Jan 17, 14:32:00 GMT+0 - Resolved - Services are back online Type: Incident

Duration: 6 minutes

Affected Components:

<p><small>Jan <var data-var='date'> 17</var>, <var data-var='time'>14:26:00</var> GMT+0</small><br><strong>Investigating</strong> - Rack 1's main switch has crashed due to a configuration fault. We are currently working on resolving the configuration fault and reverting the switch back to a recent config. .</p> <p><small>Jan <var data-var='date'> 17</var>, <var data-var='time'>14:32:00</var> GMT+0</small><br><strong>Resolved</strong> - Services are back online.</p> ]]> Mon, 17 Jan 2022 14:26:00 +0000 https://highland.instatus.com/incident/ckyisikqj395932dooty0im2dxg https://highland.instatus.com/incident/ckyisikqj395932dooty0im2dxg NETWORK WIDE: DEGRADED - High Latency Type: Incident Duration: 28 minutes Affected Components: Network Infrastructure Jan 16, 03:01:26 GMT+0 - Resolved - NTT NOC has corrected the MTU value on their switch and bounced the switch port. Services are fully back online and responding normally. Jan 16, 02:33:00 GMT+0 - Investigating - We are currently investigating an issue with packet loss and high latency on our NTT Carrier Link. Jan 16, 02:41:19 GMT+0 - Investigating - NTT NOC has been contacted and they are investigating the issue. Jan 16, 02:55:31 GMT+0 - Identified - NTT NOC has identified an issue with the MTU setting on their equipment and is working to revert it back to the correct settings. Type: Incident

Duration: 28 minutes

Affected Components:

<p><small>Jan <var data-var='date'> 16</var>, <var data-var='time'>03:01:26</var> GMT+0</small><br><strong>Resolved</strong> - NTT NOC has corrected the MTU value on their switch and bounced the switch port. Services are fully back online and responding normally. .</p> <p><small>Jan <var data-var='date'> 16</var>, <var data-var='time'>02:33:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating an issue with packet loss and high latency on our NTT Carrier Link. .</p> <p><small>Jan <var data-var='date'> 16</var>, <var data-var='time'>02:41:19</var> GMT+0</small><br><strong>Investigating</strong> - NTT NOC has been contacted and they are investigating the issue. .</p> <p><small>Jan <var data-var='date'> 16</var>, <var data-var='time'>02:55:31</var> GMT+0</small><br><strong>Identified</strong> - NTT NOC has identified an issue with the MTU setting on their equipment and is working to revert it back to the correct settings. .</p> ]]> Sun, 16 Jan 2022 02:33:00 +0000 https://highland.instatus.com/incident/ckygnj04v346432e1mqvnn21s1l https://highland.instatus.com/incident/ckygnj04v346432e1mqvnn21s1l Rack 01 - Rack Switch Down Type: Incident Duration: 15 minutes Affected Components: Network Infrastructure Jan 15, 15:21:10 GMT+0 - Resolved - Rack 01 Switch has recovered and connectivity is restored. Jan 15, 15:06:00 GMT+0 - Identified - Rack 1's main switch has crashed due to a configuration fault. We are currently working on resolving the configuration fault and reverting the switch back to a recent config. Type: Incident

Duration: 15 minutes

Affected Components:

<p><small>Jan <var data-var='date'> 15</var>, <var data-var='time'>15:21:10</var> GMT+0</small><br><strong>Resolved</strong> - Rack 01 Switch has recovered and connectivity is restored. .</p> <p><small>Jan <var data-var='date'> 15</var>, <var data-var='time'>15:06:00</var> GMT+0</small><br><strong>Identified</strong> - Rack 1's main switch has crashed due to a configuration fault. We are currently working on resolving the configuration fault and reverting the switch back to a recent config. .</p> ]]> Sat, 15 Jan 2022 15:06:00 +0000 https://highland.instatus.com/incident/ckyfz2p65116424e1mqn1kgd5yv https://highland.instatus.com/incident/ckyfz2p65116424e1mqn1kgd5yv Network Wide: Firewall Memory Leak Type: Incident Duration: 14 hours and 29 minutes Affected Components: Network Infrastructure Nov 30, 19:05:00 GMT+0 - Monitoring - Support teams have identified a memory leak in our primary firewall that has greatly degraded all network services. We implemented a fix and currently monitoring the result. Dec 1, 09:34:22 GMT+0 - Resolved - We are satisfied with the monitoring of the system. We will continue to monitor throughout the night. Type: Incident

Duration: 14 hours and 29 minutes

Affected Components:

<p><small>Nov <var data-var='date'> 30</var>, <var data-var='time'>19:05:00</var> GMT+0</small><br><strong>Monitoring</strong> - Support teams have identified a memory leak in our primary firewall that has greatly degraded all network services. We implemented a fix and currently monitoring the result..</p> <p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>09:34:22</var> GMT+0</small><br><strong>Resolved</strong> - We are satisfied with the monitoring of the system. We will continue to monitor throughout the night. .</p> ]]> Tue, 30 Nov 2021 19:05:00 +0000 https://highland.instatus.com/incident/ckwmjgy0d30807p6mry3f8wwv8 https://highland.instatus.com/incident/ckwmjgy0d30807p6mry3f8wwv8 Network Wide: Fiber Cut Type: Incident Affected Components: Network Infrastructure Nov 23, 19:45:00 GMT+0 - Resolved - We identified an issue with BGP not automatically re-routing traffic after a fiber cut from one of our carriers. Outage identified: 2:45 PM EST Outage Resolved: 4:03 PM EST Resolution: BGP was manually adjusted to prefer our other carriers. Type: Incident

Affected Components:

<p><small>Nov <var data-var='date'> 23</var>, <var data-var='time'>19:45:00</var> GMT+0</small><br><strong>Resolved</strong> - We identified an issue with BGP not automatically re-routing traffic after a fiber cut from one of our carriers. Outage identified: 2:45 PM EST Outage Resolved: 4:03 PM EST Resolution: BGP was manually adjusted to prefer our other carriers. .</p> ]]> Tue, 23 Nov 2021 19:45:00 +0000 https://highland.instatus.com/incident/ckwclvuwk52368fn9acolflsk https://highland.instatus.com/incident/ckwclvuwk52368fn9acolflsk