Gemini, the New York registered digital asset exchange has made note of the reason for the service outages experienced by users this week.
The team said that with an increased number of users, the system load on its primary production environment, which is currently hosted on Amazon Web Services (AWS) has had to scale-up to accommodate.
However, on August 16th and 17th, Gemini started hitting Amazon’s IOPS burst quota, which then cascaded to pause the exchange’s core matching engine.
When the Gemini matching engine paused, they were forced to bring down the rest of the system to diagnose the problem. After review, it was ascertained that the matching engine component of the Gemini platform was the only component affected by the infrastructure failure.
While Gemini shut down its entire system for maintenance, they temporarily suspended deposit and withdrawal processing. Gemini also confirmed that some customer trading activity during the period was affected, although no details were given, the company confirmed it will be reaching out to those affected with more detailed information.
Cameron Winklevoss, Co-Founder, and President at Gemini stated:
“At no time were customers funds or accounts at risk, and there was no security impact on our online and offline digital asset storage systems. To remedy this issue, we’ve increased our Amazon IOPS burst quota by over 100x and scaled up the machines that we have running the core matching engine.”
“This is not the first scaling challenge we’ve encountered, and it won’t be the last — but usually, we’re able to resolve these sorts of issues behind the scenes. We already have a system in place to monitor for degraded exchange performance and alert our Site Reliability team so they can remediate the problem before it affects our customers. We’ve upgraded to a larger and different instance type that supports a higher IOPS burst quota and allows us to monitor it. And we’re continuing to improve our performance and infrastructure monitoring so we can anticipate potential problems more quickly in the future.”
“We realize that our communication during the system outage was not consistent with the quality experience you have come to expect from Gemini, and we will be improving our communication plan going forward, taking into account the feedback we’ve received.”
