* This article is reprinted from kimmy lin(Mar 31)

Two years after leaving Silicon Valley, my mind was blown away by Satoshi’s whitepaper. I joined a startup company AMIS primarily for its focus on Ethereum Infrastructure and the monumental work they did on EIP 650 Istanbul Byzantine Fault Tolerance. My first project was an enterprise Ethereum wallet service, which basically manages Ethereum private keys and send transactions on behalf of the customers.

I had never worked in a startup company before. I was quite impressed, and then I was not…

Wayback Machine

Let me step back a little. I joined Google in 2005, spending 4 yrs on build tools, 5 yrs on Search Infrastructure, and 1 yr on Sign-In Security. Google was notorious for its products always in beta, but one product it would never mess around with was Search. There I had to care about performance like QPS, latency under 200ms/query, real-time index update in serving clusters while maintaining 99.999 availability, … etc.

With a gigantic system to play with, I love making bold changes to feel meaningful. I tried to test my changes thoroughly and had always flag-gated the features so the SREs can turn if off with a flag flip should anything went wrong. This unique quality earned me one of two handfuls seats of production privileges amongst all infra developers. In short, I was trained to be ambitious but cautious at the same time.

Warp Speed Ahead

Back to 2018. My new coworkers were way cooooool. They were young, energetic and passionate. They moved so fast that a new feature can be released within a few hours. I was in awe. At the same time, I worry about the side effects of the fast design to deployment cycle. For one, code was not so well tested. Secondly, developers didn’t write code with a defensive mindset against attackers. Lastly, and most importantly, production system, code or data, was not revered. People did not have qualms about modifying the live system.

I told myself then: You are a dinosaur left behind by time. This is how the new world operates. Try to keep up, will you?

But hang on… maybe being a dinosaur isn’t all that wrong.

In 2018 alone, a billion USD worth of crypto was stolen worldwide. Most incidents can be attributed to careless and irresponsible key management. If crypto is considered fiat, we would’ve been banks! And our situation is way worse. Banks can stop most transactions they find fraudulent or made by mistakes. If we sign the blockchain transactions incorrectly, or, God forbid, lose the private keys, the money is forever GONE! No recourse whatsoever.

In another words, we are living in the crypto middle age, the Crypto Mordor I call it. It’s a dog-eat-dog world where evil has no boundaries. So maybe we SHOULD treat ourselves like highly-regulated banks.

So I quietly started a SecurityFirst campaign. The management was extremely supportive of the initiative. As expected, we got plenty pushbacks in the process. Luckily, our culture of open debate and discussion quickly resolved differences and allowed us to forge ahead.

Below are the aspects we focused on.

Reprogram the Programmers

Software engineers are trained to be lazy. My CS professor’s mantra:

“If you have to do the same thing more than twice, write a tool to do it.”

That’s a great quality for coding, but not so much when it comes to security. Oftentimes the questions we got were like:

It’s already difficult enough to hack into our system, do we really have to enforce access control on different components? — eyes roller

We already use https, it seems redundant to do end-to-end encryption… — heavy sigher

Why should we rotate API keys so frequently? And do we really need to use different API keys for different operations? — arms crosser

What’s the chance of hackers gaining access to our backends and know how to put the secrets together? Maybe we don’t need this extra layer of encryption? — deep groaner

If I learned anything in Google Sign-In Security, it is that security is layers and layers of obstacles piled on top of each other. We assumed there is always a hole in any given tier. If the attackers are going to break them anyway, we want them to significantly up their ante too. It doesn’t help that these kind of code is far from sexy, more like downright tedious and boring. Luckily, our engineers were very quick to grasp the importance in the context of wallet security.

Cloud Security

A system is as secure as its weakest link. A secure service needs to be deployed on secure infrastructure. We recruited devops with intense focus on cloud security. We want to be able to deploy a prod environment as air tight as possible. Apart from the usual container security, we also make sure each component has minimal authorization to do its thing, hiding the secrets with multiple layers of indirection, and ensure short TTL and frequent rotations for authentication/authorization tokens…, etc.

Separation of Knowledge and Power

We broke down our engineers to two groups, devs and SREs. Devs know only the code; SREs know only the credentials. Not one person has all the information to compromise the system by mistake or maliciously.

I would advocate further to create an environment such that devs care about fast design-to-deploy cycle; SREs want first and foremost service security and reliability (so they can sleep at night). Devs are ambitious; SREs are cautious. There should be this dynamics of check and balance, especially for a crypto wallet/custodial company like us that also needs to keep up with latest technology.

Audit

Where there is money, there is audit. We keep digital trails of all actions taken and ensure those records live as long as the law requires. This is not just a matter of conforming to financial regulations, it also allows us to detect abnormal activities and system anomalies.

Disaster Recovery

I believe when it rains, it pours. Having a DR plan in place is not enough, we make a point to regularly do DR drills with different team members. Code rots, and so do documentations. If those things are not kept up-to-date, when a natural or man-made disaster happens, we will most likely find ourselves in a worse place than we had thought.

How about performance? … Meh

Programmers’ pride dictates that we care about creating fast and scalable systems. In the course of development, we often found ourselves struggling over various security measures slowing down the system. It hurts. Period. But our SecurityFirst mindset helps us prioritize and let go of personal pride for greater good. Coincidentally, blockchains are still slow. We don’t quite yet have to worry about performance. It’s definitely our next big challenge, but it’s always going to be second to security.

Our system will never be secure enough. This is a war we don’t ever expect to gain the upper hand. Some people are frustrated by it. I find the process of adapting our system to withstand ever-evolving hacking techniques exhilarating. It’s a bit like Sisyphus, the Greek king who pushes a large rock up on a steep hill, only for it to roll back down when it nears the top. Except in my scenario, each hill is unique and challenging in its own way.

In cryptosphere, the word “decentralisation” is the buzzword, rightfully. The concept of “power to the people”, “self sovereignty”, …, etc., is important, critical even for those countries in turmoil. But for people in areas where remembering passwords counts as a difficult task in their daily lives, having them manage their crypto private keys without in-depth education is bordering on shirking social responsibility.

I Believe: Decentralizing Power in a Socially Responsible Way

Most people simply want to grow their nest eggs but can’t begin to grasp the idea of cryptocurrency. They deserve a secure path to financial freedom just as much as crypto aficionados.

Satoshi gifted the world with cryptocurrencies. My purpose is to further help the world manage those crypto assets securely. Decentralization is ideal, and we are making headway towards it. But security, IMO, should always be the first and foremost concern when it comes to people’s livelihoods. This mindset is instrumental in surviving this Crypto Mordor.

In the Fellowship of the Ring, Gandalf blocks the Balrog saying:

You cannot pass….I am a servant of the Secret Fire, wielder of the flame of Anor. You cannot pass. The dark fire will not avail you, flame of Udûn. Go back to the Shadow! You cannot pass.

Go back to the Shadow! You cannot pass.