Affecting System - Kansas
We are aware of the full outage in Kansas, we are awaiting word from the DC on what has happened
As of right now we are unable to do anything until the issue is officially announced and resolved, we will work ASAP to get services back online
It is somewhat confirmed an issue with the cooling for the room we are in, while we are currently online we are monitoring and if temps continue to increase we will be forced to power down to protect the equipment
Temps are back to normal and all services are back to normal and being monitored.
The issue has occurred again, DC has contractors on site to repair the issue
Kansas is back online however we are monitoring, the network is still considered at risk!
Affecting System - Fremont PXE Server
We have been made aware of an issue reinstalling in Fremont, we are working on rebuilding our PXE server to resolve this issue.
Once fixed, I will update the status here.
Fixed:
This issue has been resolved and reinstalls are again working as expected.
Affecting System - RPKI
Hello All,
We are aware of on ongoing issue that as caused RPKI to go `not-found` please note they have not gone invalid so this should not affect your announcing, if your provider is dropping rpki not found then I urge you to instead reach out to them, if you announce via us we only drop RPKI invalid thus is the norm in networking.
This issue seems to be caused by our publication repo via RIPE however, please know there is nothing in my power I can do, our Krill is online and all records are active the issue is between our Krill and RIPE which I have raised a ticket but can do nothing but wait.
As mentioned above, the common practice in networking is to drop RPKI invalids, nothing has gone to invalid instead it has gone to not found in which case your upstream should fall back to IRR, if they have not this is not something we can resolve for you instead you should speak to your provider.
Affecting System - Kansas Core Network
Hello,
There will be a short downtime in Kansas while we upgrade our core router, we expect this to last less then 30 minutes however the network will be down for this.
This has now been completed and the main core is back online, the nest stage will be moving our SIX connection however this will not impact customers.
This has all now been completed!
Affecting System - VPS Panel
I am working on bringing the VPS panel back online, VPS services are still up and running but the VPS panel has some how corrupted which is why I am restoring from a backup but this will take time
Affecting Other - AMS-HV003
Dear Valued Clients,
We regret to inform you that one of our hypervisors, AMS-HV003, is currently experiencing a network connectivity issue. While we can still communicate with the hypervisor via IPMI, the issue is impacting hosted services. Our on-site remote hands team is en route to investigate and replace any faulty hardware as necessary.
We sincerely apologize for the inconvenience caused and the delay in providing this update. Please rest assured that we are doing everything possible to resolve the issue promptly.
In the meantime, if you would like us to restore your VM to an alternate node from our most recent backup (timestamp: 2025-01-12T02:31:01Z), please let us know, and we will initiate the process immediately.
Clients on AMS-HV003 are the only ones affected by this outage. Our remote hands team is expected to arrive within the hour, and we will provide further updates as soon as more information is available.
To compensate for this disruption, we will review all affected accounts and offer SLA credits or upgrades accordingly once the issue is resolved.
Thank you for your patience and understanding. If you have any questions or require immediate assistance, please do not hesitate to contact us.
Best regards,
Aaran Perry
Founder & CEO, HYEHOST
Resolved
The issue was a failed DAC cable connecting the node, while most nodes we have multiple connections this was a new model of server for us to use that has only a single 25G port, we are looking into adding additional connections to the server for redundancy and apologise again for the issues caused, we will reach out via email shortly to those who were affected and offer SLA for the downtime.
Affecting Other - Kansas
We are currently working on upgrades to hypervisors and general maintenance of the cluster, in this time normal funtionality of the panel may be unavailible.
Affecting Other - Amsterdam Network
We are aware of some issues in Amsterdam, currently we have enabled mitigation, we are working to bring back full connectivity.
UPDATE: Network should be back to normal, we are still monitoring the situation.
Affecting System - Network
We have been notified of an issue within our Kansas network, we are working to fix this issue and prevent future issues, we understand this is causing issues for everyone and are working as fast as possible to resolve this.
UPDATE: For now we have added some fixes, please note there is a lot more information we will provide once the issue is resolved however for security we will not publicly state what we have done.
UPDATE: The issue has spread onwards, for now it is a waiting game.
Affecting Other - Frankfurt Network
Seems Frankfurt has once again decided to fail, I am attempting to bring up a replacement however my attempts are failing currently, I am unfortunately unable to get to ipmi for this node and as such am waiting on our provider to respond
Update: While we have not found the underlaying issue I have managed to bring the router back online, for now the network is back online however will still be considered at risk as I am unsure what caused this issue and if it is failing hardware then I may need to find replacements.
For now I have added some manual scripts and monitoring so that if the network goes down again it will log the logs and dmesg and if still no connectivity it will issue a reboot which should be at least for now a patch to the situation but would still require the network to be down for a maximum of around 5-10 minutes before it sees the issue and reboots.
Affecting Other - Frankfurt Network
We are aware of some issues, I am sorry for the delay in reporting this, I am working on this asap
21:37 - I am bringing up a a separate core router for now, it should be back online shortly but will reduced capacity on IXs, I will keep you all posted
Affecting Other - Frankfurt
Frankfurt is currently experiencing a disk issue, we are working on resolving this issue and I will post updates as they come.
UPDATE: I am rebuilding the raid array, you will notice degraded disk performance while this is happing
