Earthling Bot - Notice history

Our bots have began responding again, we are monitoring the results.

Recently, Earthling went through a significant outage that lasted about 14 hours, during which commands either did not respond or took several minutes to load. This problem started soon after we updated our Giveaways module. The bot kept checking every giveaway every second due to an error in the way we calculated giveaway end times. As a result, there were far more pointless database queries than usual.

Our database reached its API limit as the number of queries increased, and it started to queue up each new request. As a result, every command - including dashboard actions, utilities, moderation, levelling, and more - had to wait behind thousands of giveaway checks. As a result, the entire bot behaved slowly or not at all. After locating the unusual query pattern, we isolated the problem by tracing it back to the new giveaway loop.

By correcting the timing calculations, halting the runaway process, emptying the database queue, and keeping an eye on things until they stabilised, we were able to resolve the incident. Stricter background task limits and better query spike monitoring are just two of the precautions we're putting in place to make sure something similar doesn't happen again. We appreciate your patience while we resolved this; everything is now completely functional.

We are continuing to investigating this incident.

We are continuing to investigate this issue, it seems most commands are failing to respond.

We believe this issue has been resolved.

A fix has been implemented and we are currently monitoring the result.

Our bot is back online, but some features may be temporarily unavailable or unstable while we continue restoring all services.

We have updated this incident as the bot is now offline. We have identified that the issue lies with our host and their services having problems.

Our bot has returned online. Our host is still investigating the issue, so there’s a chance this could happen again until a fix is provided.

We’ve identified an issue with our host, causing connection timeouts. We’re communicating with them to resolve this.

We recently have had two incidents where Earthling was put offline due to a sudden loss of connectivity with our hosts’ POP systems. These outages caused our services to become unavailable as we were not able to access our systems. We believe this was caused by one of our upstreams being attacked by many terabits and continued for 20 minutes. Our host responded by moving systems over to the failover, which failed to kick in automatically due to the way the systems went down. We apologise for the inconvenience caused and are constantly looking for ways to mitigate these outages from happening again.

Our services are now back online. Our host is monitoring and resolving the situation, and should hopefully cause minimal disruption to our bots.

One of our upstreams has gone completely offline, causing everything to fail. Our host has implemented a failover, while our upstream is working on a fix. Our bots are now returning online.

We have identified the issue with one of our upstreams, we are awaiting a fix.

Our services are gradually recovering, and we’re continuing to monitor the situation closely.

We’re currently using temporary measures, such as periodically restarting the bot, to mitigate the issue until a permanent fix is implemented. We believe we’ve identified the root cause, and a fix is scheduled for deployment tomorrow.