Addressing Claims of Deleted Balances

A response to the Reddit post: “The Synthetix "dApp" deleted my balance”.

The most important point is Onyx never had his balance deleted and he knows this, in fact he even linked to the transactions where it is clear this was not the case. Some people will say this is semantics, but it is critical to understanding what actually happened.

Front running protection
Here is a quote from my discussion with Onyx on Reddit: “I actually want you to keep running the bot so we can make these changes and see if they are effective.” So yes, we promised we wouldn’t delete his balance and we didn’t, but I absolutely said we were planning to introduce anti-frontrunning measures.

What actually happened is we implemented a slashing condition into the oracle. This was documented here and was openly discussed and debated within the community at the time. Was this an aggressive solution to the problem of front running? Yes absolutely, but the justification was simple, in the absence of a penalty a malicious actor would have no disincentive to attack the system. If the only risk was not profiting then it was clear that more people would choose to attack the system. To be clear the system we implemented in the oracle was not arbitrary and it didn’t target one wallet specifically, it targeted any wallet that met the slashing conditions. It also had a secondary mechanism to increase the certainty the wallet was front running by significantly increasing the gwei and observing whether the account immediately increased gwei to match. But this is somewhat besides the point, as there is no argument as to whether Onyx was front running, he has freely admitted it.

The oracle outage
There have been bots attempting front running attacks for almost a year now, but Onyx was the first one (that we know of) to read from the mempool. We began looking into mechanisms to prevent this once we observed his bot, unfortunately before we had the chance to implement them we had an Oracle Outage. The front running bot was active at this time, and it exploited the outage to generate 37m sETH. Many news outlets reported this as a billion dollar “hack”, but the reality is a little more complex than that.

What happened is the debt in the system hyper-inflated, so the system was undercollateralised. The attacker could not profit from this attack. But he was not aware of this until after the system had been frozen by the team. Yes, we can we freeze the system, and in this case we felt it was justified to provide time to assess and resolve the issue.

A malicious actor
Onyx has claimed his only intention was to demonstrate Synthetix is centralised, this is a blatant lie. He was actively attacking the system when the oracle outage occurred, and told me afterwards that he was trying to steal at minimum $100k from the system. These funds were coming from SNX stakers, so he was attempting to steal from the SNX holders and at no point before the oracle outage did he attempt to contact the team or anyone else to disclose the flaw he was exploiting, we even had an open bug bounty program at the time.

The reality is Onyx is an opportunist who was trying to exploit the system, and when he realised he had even more leverage due to the oracle outage he doubled down and tried to extort $100k USD to assist with restoring the system. There was no reason to trust he would not use any future leverage to attack the system and extort more money from us. We negotiated with him to restore the system, and then transferred $40k worth of ETH. This was from the outset a hostage negotiation, and we were doing everything we could to restore the system without needing to redeploy the contracts and roll back the debt.

This is critical, because we could have rolled back the system and paid him nothing, but we have repeatedly stated we would not cross this line, though we do technically have the power to do so. I was around for The DAO hard fork and I watched the Griff live stream during the attack so I know how contentious any action like that would be, potentially for years afterwards. So we paid a “bounty” to Onyx for disclosing the issue after the fact, but bounty is honestly being too kind, it was extortion.

I should point out, there is no functionality to modify balances in the contracts. In order to modify balances we would need to redeploy the contracts, almost any project team could do something similar, it is just a question of whether the community would accept the forked token or stick with the original contracts. The point is, there is nuance here, none of the issues we are discussing are black and white.

Restoring the system
The process used to restore the system was straight forward, once we agreed on the amount to be paid. We manually modified the exchange rate on the sKRW token and he traded back into that token to reduce his balance. We then restored the correct exchange rate. It is probably also worth pointing out at this stage, that we had an agreement about the amount of sETH he would be keep after this trade, he immediately broke this agreement and kept an additional 20 sETH ($5k USD). So even after being paid $40k he couldn’t help but to take another $5k, the level of trust at this stage was pretty low.

Of course Onyx then continued to attack the system, as I knew he would, whether we encouraged it or not. We accept people will attack the system and we will need to respond, this is crypto, it is an adversarial space.

We then spent almost 24 hours straight working through the solutions available, and each time we came up with a solution it was obvious that it was not going to change the calculus for an attacker. We needed some kind of slashing condition, otherwise the optimal strategy would always be to attack. So we implemented a slashing condition and his bot was caught by it and slashed.

Specifically what happened was the oracle detected a tx in the mempool trying to front run a price update. It then implemented a sandwich attack to raise the exchange fee to 99% for that transaction by sending one tx with higher gwei to raise it and another with lower gwei to drop it back down to the normal rate. Here is the transaction that slashed his funds by 99% and sent them to the feepool to be distributed back to SNX stakers.

I will leave it to the reader to determine whether this approach was justified, but suffice to say SNX holders were almost universally positive of this solution, front runners not so much... But no one can deny, we needed a mechanism to deter front runners and this was effective.

Escalating attacks
Now you would think this might be the end of the story. But unfortunately, at this point, Onyx decided we had broken an agreement to not delete his balance. During our discussion what delete meant was not ambiguous, he was worried we would redeploy the contracts to modify his balance and I unequivocally said we would not do that. It would make no sense to pay him $40k only to then do the thing we paid him to avoid having to do. But I said we were working on front running protections, which he laughed at and said there was no way we could stop him. So he decided to take “revenge” and try to destroy the project, he went into discord and told his story and was met with a fairly unsympathetic audience as you might imagine.

After being dismissed on discord he started front running again, but he implemented a check in his contract to protect against slashing with a “fuck you” message for us. It was clear we were dealing with someone who was going to keep escalating and we needed an response because the front running attack vector still existed. We had falsely assumed funds being at risk would be sufficient to dissuade an attacker but we underestimated how determined Onyx was based on this new sense of the injustice of the situation as he saw it. Again I will let the reader determine whether his position was reasonable.

We have a function in the system to allow us to purge balances from one synth into another synth. It doesn’t delete balances it just converts them to another synth. Because of the way the SIP-6/7 functionality was implemented and Onyx’s response we needed to use this functionality to allow the oracle to trap a front running bot in a specific synth and then purge them to slash their funds. The mechanism described earlier was used to determine with high probability that the wallet being targeted by the oracle was actually front running by reading the mempool, and then this function slashed his balance.

After the second slashing Onyx gave up from what we can tell, he claimed to be working on a new implementation in Discord but while we saw some test bots running none was ever implemented at scale that we could detect. I should also note that he appears to have deleted all of his messages in discord, again I will leave it up to the reader to determine why that may have been the case.

Now he is resorting to spreading FUD, claiming his balance was “deleted” which is a lie and he knows it, but he also knows most people will not review the details of the story and that an inflammatory statement like that is likely to get maximum exposure.

We have subsequently implemented more long term approaches to the front running attacks, but there are still bots soft front running, ie attempting to guess the price movement before the oracle updates by observing spot markets. Some of these have been quite effective, but none of them have been slashed because this kind of front running is far less profitable and doesn’t trigger the SIP-6 mechanism. In fact the bots frequently lose money if there is a large movement against them. This is a very different kind of attack and far less dangerous to the system, because the profit is not risk free and transaction fees can significantly eat into any actual profit. But we are working on solutions to this as well, which will be implemented soon.

Decentralisation as a process
I have written about the trade-offs with centralisation in dApps elsewhere several times, including this twitter thread. But I will briefly address the issue here since some readers may be hearing about Synthetix for the first time.

We made a decision early on to use a proxy architecture to allow us to rapidly iterate on the system. It is no exaggeration to say that had we not done so it is almost certain that the project would have failed by now. Had we not implemented this functionality, it would have taken us much longer to pivot from a stablecoin project to a synthetic asset platform. Not to say we couldn’t have done it but it would likely have taken much longer and the delay between implementation and market validation would have meant we faded to irrelevance. You need only observe the extremely long development cycles of projects like Maker and Augur to see the impact of choosing the purely decentralised path. While I applaud the fact that Joey and Rune have tried to avoid almost any centralised compromises there is a very real trade-off by going down that path, and I strongly believe you first need to create value before it makes sense to be fully decentralised. This is very much an ideological debate and one we won’t resolve in this post.

There must also be plans in place to ensure when you reach a tipping point in user adoption and value stored in the contracts you have a plan to cut the centralised bridges and move to a more decentralised governance process. We are already in the process of implementing some of these changes. We have moved to a more decentralised upgrade process using a system modelled off the EIP process, it can be found here: Synthetix Improvement Proposals. We have also implemented a community governed multi-sig model for the distribution of staking rewards from the Uniswap sETH pool. I have also proposed a model for decentralised proxy contracts. We are undergoing a system review with an external party to identify areas of the system which can be decentralised and work on a roadmap to achieve this. One additional consideration is that in any system where the oracle is controlled by the team or a small group of people any attempt to convince users that funds are not at risk is essentially theatre. So unless a project has a robust decentralised oracle solution in place assume your funds are at risk from malicious actors both internal and external. This includes most of the systems in production today with a few exceptions like Augur and Uniswap that avoid the need for external oracles.

In closing, this incident exposed a number of issues in the current implementation of Synthetix, some issues we have already addressed and others we continue to address. It is of course up to each individual to make a determination about the set of trade-offs we have chosen and the implications. We believe the vast majority of our current users understand and accept these trade-offs and have tried to be as open about them as possible. But I personally accept responsibility for ensuring we do a better job of ensuring all users are as informed as possible. We have also made a strong commitment to our community to move towards a fully decentralised DAO like model for Synthetix, something which I personally believe is critical to the long term viability of the project, but that needs to be well thought out and timed correctly.