Zero Knowledge - Not Boring by Packy McCormick

Welcome to the 1,258 newly Not Boring people who have joined us since last Monday! Join 56,891 smart, curious folks by subscribing here:

🎧 To get this essay straight in your ears: listen on Spotify or Apple Podcasts

Today’s Not Boring is brought to you by... UserLeap

A lot of you are already familiar with UserLeap. Since I started including a UserLeap microsurvey at the bottom of each email in March, you’ve given me feedback over 5,000 times! 63% of you love Not Boring essays 🤗 3% of you think they’re bad 😢

UserLeap is a continuous research platform that makes it easy to build and embed microsurveys into your product to learn about your customers in real-time. Product teams use UserLeap to gather qualitative insights as easily as they get quantitative ones, and automatically analyze the results so teams can take quick action.

I check UserLeap at least once a day, and I’m not alone. Teams at Square, Adobe, and Dropbox love UserLeap too. In fact, the UserLeap team knows that once teams start using the product, they don’t stop, so it offers a generous free plan to try it out. Sign up below to start collecting insights today.

Hi friends 👋 ,

Happy Monday!

There are some topics that I write about that make me realize how unbelievably dumb I am compared to some people out there. Today’s topic is one of those.

Luckily, I have friends who are much smarter than me, and who are down to collaborate. Today’s co-author is one of those.

Jill Carlson is someone I’ve respected from afar for a while and gotten to know over the past few months. She’s been a Goldman credit trader, Cambridge researcher, a writer, an advisor to some of the most promising crypto projects, and a venture capitalist. I’m missing some, too. Now, she’s working on a new company. If you’re interested in the bleeding edge, particularly in crypto, follow Jill.

She’s orders of magnitude smarter than me, which is good, because while today’s topic is orders of magnitude more complex than I could figure out on my own, it has the potential to be vitally important in the coming years. I don’t want my own lack of … knowledge … to keep you from learning about it.

Let’s get to it.

You’re going to be hearing a lot about Zero-Knowledge Proofs (ZKPs) over the next few years.

Every so often, a technology comes around with the right combination of promise, nebulousness, inscrutability, and abstractness to fully capture and incept the hive mind’s imagination. These technologies aren’t just going to change a few things; they’re going to change everything.

Blockchain. AI. Chatbots. The cloud. VR. Autonomous. Metaverse. mRNA.

In their infancy, these technologies become blank canvases onto which the world projects its dreams and via which hucksters peddle their snake oil.

When someone figures out how to do something with one of these technologies for the first time, but before practitioners have dealt with the hard work of actually implementing them, the field of possibilities is wide open. As you’ll remember from high school physics, potential energy is highest before that kinetic energy gets burnt off. Samesies here.

Tim Urban, the mad genius behind Wait But Why, illustrated it well in this image:

March 5th 2021

When nothing’s been done, all possibilities are wide open.

This, incidentally, is why it’s easier to raise venture capital with an idea and a deck and maybe a prototype than it is once you have a product in customers’ hands and a little bit of revenue. It’s probably why the NBA Draft gets twice as many viewers as the average nationally televised game. People fucking love potential.

This dynamic helps explain the graph we’re about to show you, which should be very familiar to long-time Not Boring readers: the Gartner Hype Cycle.

The Gartner Hype Cycle is incredible in its consistency. Every promising new innovation goes through the cycle. It’s a rite of passage. The Hype Cycle works because it’s a big mirror. It just reflects our predictable group behaviors back at us, kind of taunting us. “You’re going to get really excited about this thing, and it’s all going to come crashing down. Watch. But don’t worry, it’ll bounce back, sometimes.”

Part of the fun is figuring out the things that just experienced their Technology Trigger and getting in super early -- building with, investing in, and thinkboi’ing about that tech before others have even heard about it. That’s a lot of what Not Boring’s here for, to let you know when we’re at inflection points, but I’m nowhere near technical enough to figure out when those triggers occur by myself. I crowdsource by following smart people on Twitter and making friends with people like Jill.

Take DAOs, which are going through their own mini-Hype Cycle. When I wrote The DAO of DAOs in March, I cited Jill’s tweet as one of a few that made me realize it was time to pay attention.

Jill was right. DAOs have exploded into the hive consciousness over the past three months.

A couple weeks ago, Jill dropped another prediction:

When Jill predicts, I listen. Luckily, Mr. Fox tagged me in, and Jill and I decided to team up to explain what zero-knowledge proofs are, why they’re important, and their potential applications.

ZKPs essentially let someone prove that they know or have something without giving up any information about what they know or have.

For example, I could prove that I know the password to an account without entering the password and risking its exposure, or that I have enough money to cover rent for the next year without telling some random broker all of the details of my personal finances. The technology has implications for personal privacy, crypto, businesses, and even nuclear disarmament.

Because of all of the potential use cases, and because ZKPs are so hard to grasp, they are ripe for the Hype Cycle. As Jill tweeted:

What’s ultimately so promising about ZKPs is that they have the potential to eliminate a major trade-off inherent in living, working, and transacting online: the convenience, speed, reach, and scale of the internet in exchange for our privacy. They also allow for nuance in a privacy landscape that’s often black-and-white.

When ZKPs are at-scale, people and companies may no longer have to assume that privacy is unattainable or overly constrictive, which both has predictable direct benefits and opens up a new design space for unpredictable innovation.

Today, we’ll glimpse into that future and its implications by covering:

We should probably start with what a Zero-Knowledge Proof is in the first place.

Our data is everywhere. Names, dates of birth, email addresses, credit card numbers, the addresses we have lived at in the last five years, our mothers’ maiden names… These are just a few of the near infinite bits of information on ourselves that we all fork over every day to companies, social media sites, customer service representatives, and sometimes (unwittingly) to scammers.

The issues with this state of affairs are self-evident. Identity theft, email compromise, data breaches, and other forms of fraud cost individuals and businesses tens of billions of dollars per year and orders of magnitude more than that in spending on defense and prevention. This is not to mention the headaches of dealing with the fallout.

The proliferation of data and its associated vulnerability has come to be an accepted cost of doing business as a participant in the modern, interconnected world. We have to trust each other. We have to enter our credit card numbers into websites. We have to send our landlord our credit history. We have to give our bank our social security numbers. And it’s not just us as individuals: companies and institutions have to disclose sensitive information to each other all the time in order to run their businesses. Sharing information and accepting its insecurity is a necessary sacrifice in order for society as we know it to function.

But what if there was a way to conduct these interactions and transactions with the same levels of trust and certainty without sharing all of this data? Well, thanks to a tricky little bit of mathematics that even cryptographers refer to as “moon math,” this is possible. Enter: the zero-knowledge proof.

Using this technique, I can prove to you that I know something without disclosing to you the information that I know. Without getting into the moon math itself, it works something like the following.

My color blind friend and I are looking at balls on a table that are identical except that one is red and the other green. He is not sure that he believes me when I tell him they are two different colors. We decide to establish that they are, in fact, different colors by playing a game:

If we repeat this game enough times, and if I answer correctly every time, then I will demonstrate to him that the balls are almost certainly two different colors. Importantly, here, I have proven this to him without revealing any other information. Perhaps frustratingly for him, he still doesn’t know which ball is red and which ball is green.

There are lots of these types of explanations out there. If you want a fun look at an older one that is literally geared towards children, you can check it out here.

To quote the original paper defining zero-knowledge proofs, they “convey no additional knowledge other than the correctness of the proposition in question.” Backing up from colored bouncy balls, the implications of this are enormous.

Zero-knowledge proofs are not new: they were first designed and devised in the 1980’s by MIT researchers Shafi Goldwasser, Silvio Micali, and Charles Rackoff.

The trio were working on a problem related to a concept called an interactive proof system. In such a system, there are two parties: a prover of some information and a verifier of that information. Generally in these systems, it is assumed that the prover cannot be trusted while the verifier can be. It is the goal of the system to be designed in such a way that:

Think of it this way: I am in line outside of a bar trying to get in. There is a bouncer standing at the door who asks to check my ID. In this scenario, I am the prover (of my age) and the bouncer is the verifier. The bouncer does not know if he can trust me. If the bouncer and I use an interactive proof system, and I am over 21, I will be able to demonstrate to him that I am over 21. If the guy behind me is underage, he will not be able to convince the bouncer to let him in.

Goldwasser, Micali, and Rackoff added an additional layer of complexity to the whole interaction. They asked (and answered) the question: how do we handle it if, not only can the bouncer not trust me -- but I also cannot trust the bouncer. Maybe I suspect that the bouncer has been stealing people’s identities. Maybe I don’t want him to know that I’m a Taurus. Whatever the issue is, suddenly the prover does not want the verifier to know the underlying information.

Using what those MIT academics introduced as a zero-knowledge proof, I can convince the bouncer to let me in without ever telling him my actual birthdate.

Over the course of the 1980’s and 1990’s, zero-knowledge proofs enjoyed further research in academia but very little in the way of actual implementation or use. It was not until the new millennium that researchers and entrepreneurs started applying the theory to practical applications.

One of the first such applications, in the mid-2000’s, was the use of zero-knowledge proofs in password security.

When you go to login to a website using a password, often what happens is that you type in your password and send it to the server, which condenses your password into a string of gobbledygook and then compares it to see if it matches the gobbledygook they have stored next to your name in their system. If the values are the same, you’re in. The gobbledygook is there as a layer of password protection: your password cannot be derived from it and it keeps the company or whoever from storing your password as plaintext. However, there is still an issue here: you are still disclosing your password to the server in the first place! Protocols for using zero-knowledge proofs to solve this vulnerability were uncovered over the course of the 2000s: some of the first examples of applying ZKPs to a real-world problem.

Lots more was written and explored over that decade around how zero-knowledge systems could be applied to identity and authentication problems. But the real inflection point for zero-knowledge happened years later.

In 2013 and 2014, zero-knowledge proofs found commercial application in another system that spent many years incubating in academia before being introduced to the world: cryptocurrency. Despite the fact that cryptocurrencies tend to have a reputation for being good for use in illicit transactions, a core feature of bitcoin and most other cryptocurrencies is that everything that happens with them is publicly recorded. All you need to look at are cases like the Silk Road bust or, more recently, the Colonial Pipeline ransomware crackdown by the FBI to understand that bitcoin is in fact completely transparent.

Cue the introduction of Zcash (originally called Zerocash, as well as its predecessor Zerocoin). Zcash, introduced in 2014 and launched in 2016, uses a specific type of zero-knowledge system to create a cryptocurrency that maintains the decentralized properties of something like bitcoin while introducing privacy-preserving properties much closer to those of physical cash -- which is to say near total lack of traceability.

In Zcash, zero-knowledge proofs specifically enable the network of computers running the Zcash protocol to verify that every transaction is valid (i.e. I actually have the 10 Zcash I am sending to you) while maintaining the privacy of the transaction data.

If you want to learn more, you should give this 2017 Radiolab episode on Zcash a listen:

Zcash and other privacy-oriented cryptocurrencies have attracted a great deal of attention -- and money. The market cap of Zcash recently approached $4 billion (before retracing… this stuff is volatile). All of this money and attention has led to a dramatic acceleration of breakthroughs in zero-knowledge systems, particularly when it comes to implementing them.

A new generation of efficient zero-knowledge proofs have been developed that solve many of the issues that previously surrounded the clunky, burdensome, expensive, and controversial setups of these systems. (If you listened to that RadioLab episode, many of the problems discussed in it are no longer problems… only 5 years later!)

Optimizations have led to huge advancements in the speed and scalability of these systems. A new and larger community of builders have cropped up leading to implementation improvements. Active open source development of libraries around these systems are accelerating development and increasing performance.

The moment for zero-knowledge systems to take off is finally upon us.

Let’s take a moment here to appreciate the positive impacts of speculation and commercialization, the upside of hype.

Zero-knowledge proofs have existed in academia for decades. They pre-date Tim Berners-Lee’s invention of the World Wide Web. Had our internet forefathers baked zero-knowledge systems into the web, the world might be a very different place. To the extent that people want to get the benefits of the internet without trading their privacy to get them, research into ZKPs is Important Work.

But ZKPs remained a niche interest for quirky mathematicians and cryptographers until there was a way to marry the technical and the commercial. Zcash kicked off a new wave of research into ZKPs, and heightened interest from investors and entrepreneurs. More work will be done on ZKPs and their applications in the next five years than in the previous thirty-six.

This is a common phenomenon. Hype and speculation attract resources which help solve hard problems, fulfill the hype, and reward some speculators. Instead of dismissing hype and speculation, it’s worth understanding the purpose they serve. They make innovation possible.

Take two examples: Bitcoin and mRNA.

Satoshi’s real innovation with Bitcoin was solving the double-spending problem. When money is basically a digital file, how do you prevent someone from just making copies to pay a bunch of different people? This was a known problem, the solution to which eluded researchers for decades, that blocked digital money’s adoption and growth. Bitcoin, both the technical innovation and the speculation it set off, unlocked enormous sums of financial and human capital and unleashed a financial renaissance.

Outside of crypto, mRNA provides a pertinent example of the same phenomenon. All the way back in the 1970s, Dr. Katalin Karikó saw the promise of mRNA, which passes messages from DNA to ribosomes about which proteins to make. For nearly 50 years, she couldn’t get anyone to take its potential seriously. Then, in 2020, Karikó’s work led to the development of Pfizer’s mRNA-based COVID vaccine, which will save millions of lives and generate billions of dollars in profit.

Now, mRNA is being heralded as the potential cure for everything that ails us. “The implications of mRNA technology are staggering,” wrote Yale’s Swati Gupta in May, “Several vaccine developers are studying this technology for deployment against rabies, influenza, Zika, HIV and cancer, as well as for veterinary purposes.” mRNA’s commercial success and COVID-fueled hype opened up a whole new design space in biotech.

Zero-knowledge proofs today are where mRNA was a couple of years ago, and where the double-spending problem was a decade ago. Privacy is becoming commercially viable.

When it comes to privacy, there are two camps:

Most of us fall into that first camp. Sure, we’ll update our passwords every once in a while, and we certainly prefer that hackers don’t steal our data, but maintaining real privacy when so much of our information lives online is more work than we’re willing to undertake.

In other words, most of us are willing to trade privacy for convenience.

Zero-knowledge proofs have the potential to eliminate the need for the trade-off and bring about a paradigm shift in the way we think about privacy. Instead of a black and white decision, we can ask, “Exactly who needs exactly how much information, and under what circumstances?”

With ZKPs, entities that need to know things can know them without seeing all of the information they don’t need to know. They don’t need to store all of the data themselves. They can still verify the things they need to verify.

To be sure, ZKPs are not the only potential solution to providing more privacy with less friction. Viable solutions will provide more privacy with zero or little change in consumer behavior.

Companies like Stytch and Magic eliminate the need for passwords by giving developers a toolkit to easily authenticate users in more secure ways. This will sound obvious, but without passwords, there’s no need to worry about your password being stolen.

Evervault builds encryption infrastructure for developers that ensures that any data users enter is automatically encrypted at the field level, never lives unencrypted in a company’s database, and can only be processed in a privacy cage. In its eight-point Pragmatic Privacy Manifesto, Evervault captures the ethos of the new generation of privacy technology perfectly:

Privacy is a basic expectation and human right, but it’s something that should never create any friction or slow down the speed of technological advancement.

In other words, we no longer need to accept the privacy trade-offs we historically have.

Zero-knowledge proofs also aim to eliminate the trade-off, but take a different approach. They ask, “What if that data is never exchanged in the first place?” The answer has a wide range of implications and applications.

It’s difficult to really comprehend the implications of zero-knowledge proofs without walking through some practical applications. Let’s do that, starting with one simple one to expand on why ZKPs are valuable and how they work, before moving onto some more sophisticated examples.

A Simple Personal Example: Renting an Apartment

Packy here. After a good quarantine year in New Jersey, Puja and I are getting ready to move back to Brooklyn. The other day, we toured an apartment, where we met our broker, we’ll call her Dolores, for the first time ever, for like 10 minutes. We liked it, so we told Dolores we wanted to submit an application. “Great!” she said, “Just fill out the PDF and email me your bank statements and income verification and both of your Social Security Numbers so that we can run a credit check.”

With all due respect to Dolores, we just met, and nothing would lead me to believe that she’s the privacy tin foil hat type. I would rather not have her and her team see the ins and outs of our financial lives, nor do I necessarily trust her to ensure that hackers can’t access that data.

Using a zero-knowledge system, we wouldn’t have to trust Dolores, and she wouldn’t have to trust us. We would be able to prove that we have good credit and 40x monthly rent without sharing our Social Security Numbers or any of the actual data underlying that assumption, and without working with a third-party like a bank or CPA to verify on our behalf. We wouldn’t need to share our underlying data with anyone, even the ZKP itself.

All clear? No? Ok ok, you’re going to make us explain what’s going on in that ZKP box, huh?

This is still a relatively simple example, but it’s more complex than the red and green ball problem from earlier because in this case, we need to prove not just that two things are different, but that a number is greater than the required number. This is true for both the credit score and the income. We can use the income example to show how the ZKP works at a slightly more advanced level than green and red ball, using a modified version of Yao’s Millionaires Problem.

Let’s say the apartment is $1k per month, and that to qualify, we need to make at least 40x one month’s rent. That means we need to prove that we make at least $40k per year. Dolores doesn’t want us to game the system, so she doesn’t tell us how much the rent is. Here’s how she tests us:

There are ten boxes, marked $10k to $100K in $10k increments. Each has a key and a paper-width slot. Dolores goes into the room with the boxes first, destroys nine of the keys, and takes the one for the box marked $40k.

Then Puja and I go into the room, with ten slips of paper. We mark “+” or “-” on the papers depending on whether the income on each box is less than or more than our income. Let’s say we make $75k, so we put slips with “+” in the first 7 boxes, and slips “-” in the last three.

We leave the room, Dolores goes back in the room, and she uses her one key to unlock the $40k box. Inside, she finds a “+”. She knows that we make more than the $40k required, and we qualify for the apartment without her ever finding out how much we actually make!

In the real-world, we obviously wouldn’t be able to manually mark the papers, they would automatically be marked based on actual data; in this example, you just need to trust us. Similarly, there would be millions of potential options, and hashes instead of dollar amounts, but the logic is similar. Plus, a ZKP is probably overkill for this use case. A product like Truework’s True identity would work perfectly well here, and who knows, might include ZKPs one day. This is what happens at the beginning of a Hype Cycle: people try to apply a new technology everywhere, even when it’s overkill!

But armed with our new understanding, we can explore how ZKPs might actually apply to areas where they’re desperately needed, like crypto, finance, and the cloud.

More Sophisticated Examples

In the alternative universe of cryptocurrency, the type of interaction between Packy, Puja, and Dolores is already becoming possible. There are projects today working on using ZKPs for proving credit scores among traders. There are others that are building new, generalized platforms for ZKP-oriented assets and transactions. And others still focused on bringing ZKP-based privacy to Ethereum.

All of this innovation promises to open up cryptocurrency and its applications to users who, to date, have been excluded from it. Today, you don’t see many businesses running international payroll using stablecoins, despite their proven efficiency over wire transfers. You don’t see many financial institutions seeking alpha amidst the lucrative and high-yielding markets of decentralized finance protocols. One key reason for this is because companies like this cannot just go around spilling their treasury information and transaction history all over the internet. They have to worry about data privacy. And the vast majority of blockchains don’t even clear the lowest bar of data privacy guarantees.

Why don’t these businesses use something like Zcash then? Or a privacy-oriented stablecoin? Because, in addition to worrying about data security, these businesses also have to worry about compliance! They can’t use something that is all the way at the other end of the privacy spectrum because they will not have an audit trail to prove that they were making a payment to a known counterparty and not, say, a sanctioned state actor or a terrorist organization? (Yes, these are real issues that companies have to worry about, not just Bourne Identity-style movie plots.)

The cool thing about ZKPs as they are being developed today is that they facilitate not only privacy functionality, but also selective disclosure of knowledge! As such, you can now imagine (and build) a stablecoin product that would meet both the data and compliance needs of a company: an asset for which the issuer could have a complete audit trail and be able to verify the compliance status of all of the holders, but which, to the view of most of the world, moves around completely privately among anonymous accounts.

Zero-knowledge proofs are finding applications in the blockchain universe that extend beyond matters of privacy and selective disclosure. These systems are also, interestingly, being used for scaling.

One major issue of blockchain scaling is that, after a while, blockchains become enormous in the amount of data they store. A blockchain is a publicly verifiable ledger of everything happening with a cryptocurrency. So if you want to go back and verify the ledger, you have to sync your computer to that entire ledger all the way back to the very first transaction. The memory and bandwidth requirements to actually do this are becoming prohibitively high for a lot of normal users.

Jill here: as an aside, I once tried to sync the entire Monero blockchain (another ZKP-based privacy coin!) while I was at my parents’ house. They didn’t appreciate the amount of bandwidth I was using up so I switched over to attempting to do this while tethered to the service I had on my cell phone. At that point, it became more of a joke than anything else. I left it running for about 4 days (with my laptop making all kinds of angry noises at me) before the thing crashed and I gave up.

Novel designs of blockchains leverage ZKPs to do away with this problem. The entire history of transactions can be compressed down to a single proof. Rather than verify the whole ledger, now you can just verify the proof. That proof will never be more than the size of a few tweets, meaning it can be done by anyone -- even from a cell phone at your parents’ house.

New blockchains aside, ZKPs are also being leveraged to scale Ethereum. A construction called zk-Rollups promises to reduce fees spent by users on Ethereum transactions and increase the throughput and scalability of the Ethereum blockchain. zk-Rollups lighten the burden of what must exist directly on the blockchain ledger, replacing all of the data that was previously required to be “on-chain” with a much cheaper and lighter-weight proof of that data. Much more on the cool work being done on this front can be found here.

Okay, so this nerdy niche area of cryptography (ZKPs) is being applied to solve problems in a different nerdy niche area of cryptography (blockchains).

Why should you, a non-nerd, non-cryptographer care?

While much of the innovation around ZKPs is being pioneered in the realm of cryptocurrency (and much of the funding going towards this innovation is deriving from that crazy industry) the implications and applications are wider reaching.

Cloud infrastructure, for example, could become a whole lot more secure using these ZKPs (no cryptocurrency required). Users could leverage cloud computing, for example, without ever exposing sensitive consumer data to the cloud providers. This is particularly important for financial services, government, and other similarly risk-averse entities when it comes to data security. These industries have been notorious laggards in their adoption of hosted cloud infrastructure, much to the frustration of forward-thinking CTOs, cost-sensitive CFOs, and end-users craving a more modern experience. If zero-knowledge can enable these industries to start using cloud providers without trusting those providers with that sensitive data, then a revolution awaits.

These are just a few of the applications of zero-knowledge systems that are awaiting our exploration over the coming years. Remember: we are still just at the Technology Trigger point of the Hype Cycle. We may not have a concrete sense of how the zero-knowledge revolution manifests until we have gone through Peak, Trough, and find ourselves on the Slope of Enlightenment a few years from now.

The most exciting part about catching something so early in the Hype Cycle is the new design spaces and opportunities for experimentation brought about by the Technology Trigger. It’s a time of imagination, when all of the things that could possibly happen still might happen.

From the Technology Trigger through the Peak of Inflated Expectations, it’s all about seeing which ideas will work, which won’t, and which things emerge that the space’s early explorers hadn’t even thought up. When the market gets its hands on new infrastructure, millions of peoples’ dreams and craziest ideas do battle with technical and social limitations. Then, excitement around the space plummets as the limitations taste temporary victory; but the builders keep building, persisting in the business of applying the new technology to real use cases.

But today, we’re still in the dreaming phase, at the base of the mountain. So we can dream a little bit.

In the extreme bull case, zero-knowledge proofs move more on-prem businesses to the cloud, more business activity onto the blockchain, and allow us to transact at the right level of transparency and disclosure required for any given transaction. They grease the wheels of internet commerce.

More wildly, zero-knowledge proofs might help unlock the avatar-based economy Jill envisioned in October 2020. Anyone could remain pseudonymous and still prove that they have certain skills and qualifications, can pass a background check, and provide any number of other non-identifying pieces of information a hiring company or Liquid Super Team might need to know.

Those are just a few examples in an open design space of millions. I’m sure that you can cook up some ideas of your own. We’re certainly not the only ones who are excited…

… and if history has taught us anything, we’re certainly not the last. Consider yourself members of the elite ranks of the zero-knowledge-initiated. Be warned: once you learn about them, you start seeing potential applications everywhere.

Zero-knowledge proofs are incredibly promising and exciting, but don’t be surprised if expectations exceed reality in the short-term. There are hard technical, economic, and social challenges to solve, and let’s be real, nothing fuels hype like zero knowledge.

Thanks to Jill for collaborating with me on this, and to Charles for reviewing!

How did you like this week’s Not Boring? Your feedback helps me make this great.

Thanks for reading and see you on Thursday,

Packy