Your own personal Google: how Mem plans to reinvent note-taking apps with AI

Illustration by Kristen Radtke / The Verge

What if you had access to everything the internet knows about you, and what if you could use it to make your life better? That’s what Mem is trying to figure out.

In the summer of 2019, Kevin Moody and Dennis Xu started meeting with investors to pitch their new app. They had this big idea about reshaping the way users’ personal information moves around the internet, coalescing all their data into a single tool in a way that could actually work for them. But they quickly ran into a problem: all of their mock-ups and descriptions made it seem like they were building a note-taking app. And even in those hazy early days of product development — before they had a prototype, a design, even a name — they were crystal clear that this would not be a note-taking app.

Instead, the founders wanted to create something much bigger. It would encompass all of your notes but also your interests, your viewing history, your works-in-progress. “Imagine if you had a Google search bar but for all nonpublic information,” Xu says. “For every piece of information that was uniquely relevant to you.”

That’s what Moody and Xu were actually trying to build. So they kept tweaking the approach until it made sense. At one point, their app was going to be called NSFW, a half-joke that stood for “Notes and Search for Work,” and for a while, it was called Supernote. But after a few meetings and months, they eventually landed on the name “Mem.” Like Memex, a long-imagined device that humans could use to store their entire memory. Or like, well, memory.

Either way, it’s not a note-taking app. It’s more like a protocol for private information, a way to pipe in everything that matters to you — your email, your calendar events, your airline confirmations, your meeting notes, that idea you had on the train this morning — and then automatically organize and make sense of it all. More importantly, it’s meant to use cutting-edge AI to give all that information back to you at exactly the right time and in exactly the right place.

The most ambitious version of the product will require a huge amount of work and a lot of technology that either doesn’t exist or only barely works. It will require partnerships with huge tech firms that may be reticent to share their data, trust from users who have to compile it all together, and a user experience that is infinitely flexible without being too complicated or overwhelming. But if Mem can accomplish its goals or even come remotely close, it could become the most powerful productivity tool on the planet. It could maybe even help shift the way the internet works.

“There’s all this data in the world about us,” Xu says, “and none of it we can actually use, right? It’s all trapped inside of Google’s servers or Facebook’s servers, Netflix, YouTube, all that. What we want to do is actually put that in your hands.”

So far, Mem is… mostly a note-taking app. It’s blisteringly fast and deliberately sparse — mostly just a timeline of every mem (the company’s parlance for an individual note) you’ve ever created or viewed, with a few simple ways to categorize and organize them. It does tasks and tags, but a full-featured project manager or Second Brain system this is not.

But if you look carefully, the app already contains a few signs of where Mem is headed: a tool called Writer that can actually generate information for you, based on both its knowledge of the public internet and your personal information; AI features that summarize tweet threads for you; a sidebar that automatically displays mems related to what you’re working on.

All this still barely scratches the surface of what Mem wants to do and will need to do to be more than a note-taking app. The competition will be steep, too: Dropbox recently acquired Command E to power cross-platform document search, Neeva is building personal search right next to web search, while Evernote’s Chrome extension lets you search your notes through Google. Google’s even tried to build this itself, though it gave up on Google Desktop when it became clear cloud storage was the future. As generative AI gets more powerful and less expensive, every app you use will find a way to incorporate it.

The point is, “a search engine for your private stuff” is not a new or even particularly novel idea. But thanks to huge recent advancements in AI and a generational shift in how we manage our digital lives, the time is ripe for someone to build it.

Your own personal Google

There’s no lightbulb moment at the founding of Mem, more a series of cobbled-together ideas over the years. Moody and Xu were Stanford classmates and had talked about working together while Moody was at Google and Xu at Yelp. But Moody does remember one specific aha moment from 2018 when he was at a restaurant in Oregon. “I was like, ‘Shit, I think I’ve been here before, and I forget what I ordered,’” he says. He couldn’t remember if he should reorder what he got last time or was potentially about to make the same menu mistake again. This immediately felt like a solvable problem. “I wanted to be able to come back to the same restaurant,” he says, “and a note would just pop up and say, ‘The seared ahi, 10/10; the meatball sub, send it back before it even gets to your table.’” He wrote that idea down in Apple Notes and never quite forgot about it.

From the very beginning, Mem’s founders knew they needed a huge amount of data to get their vision off the ground. (Google doesn’t work if there’s no internet to crawl, right?) So the Mem app was built with easy capture in mind, which meant making the app fast to open and fast to type into. Before they had a mobile app, they built a way for users to text notes to Mem through SMS or WhatsApp. That proved to be one of Mem’s most popular features. Even in the early days, when Mem didn’t do much, it was still one of the easiest tools for just quickly jotting something down.

But the dream is not quick notes. The dream is for all your info and everything about you to just appear in Mem, where it can be sorted and searched and made useful on your behalf. If a tool could access your Netflix history and your Spotify playlists, just to name one example, it could learn a lot about your taste. So the team began early on to search for even bigger sources of personal information, ones that don’t require users to take their own notes.

It started with the biggest player of all: Google. After all, if you wanted a single source of a person’s important online information, their inbox is a good place to start. Mem has been working with Google for months to let users connect to their Gmail and allow Mem to index all of their emails since time immemorial. They’re still working on the implementation, but the goal, Xu says, is to “essentially turn all of that unstructured information into something that is really easy to use for you.” After that, Mem will need to do something similar for, well, every other platform on the internet.

There’s a good chance it’s impossible to bring the entire internet’s knowledge of you into a single place — even Xu and Moody acknowledge that most companies have no incentive to give Mem or anyone else access to the data they have. And even if it can pull it off, Mem would instantly become a security nightmare. Xu and Moody say they’ve invested in platform security since the beginning and that they take their security responsibility seriously. “If we succeed even close to the degree we want to, we’re going to have the most valuable data in the world,” Xu says, “and that’s going to put a huge target on our back.” In the long run, they’re interested in using decentralized platforms to avoid having a single point of failure. Both co-founders say they’ll never sell or otherwise abuse user data. Still, to do what they plan will require an unprecedented amount of trust from users at a time when trust in digital platforms seems awfully hard to come by.

Mem’s best — maybe only — chance is to make the tradeoff worth it.

Zero clicks away

A useful question to ask about any internet service is, what are they doing with all your data? Often, it’s targeted ads, which are of specious value. Other times, it’s to make better recommendations for stuff to watch or buy or listen to. In Mem’s case, it thinks that by collecting and combining all your information, it can both help you find all that data and make it much more useful.

In our very first conversation in 2020, Moody off-handedly asks a question that hinted at things to come: why doesn’t Google Docs know what’s in your docs? Docs can search through the words, sure, but it doesn’t understand what your doc is about. It can’t generate a table of contents or a one-paragraph summary; it can’t tell you which of your other docs might be relevant to the topic at hand. (Ironically, in the months since, Google has actually begun to roll out some of these features. Moody was onto something.) It’s not just Google, either, he says. Too many notes apps, document holders, and collaboration tools are just “dumb containers of information.” Mem aspires to be more than that.

Mem’s real theory of the future starts with transformers. Not the robots-in-disguise kind — the machine-learning kind. It starts with a whitepaper from 2017 called “Attention is All You Need,” written by several Google researchers, that changed the way people think about AI.

The short version of a long paper goes something like this: Using transformers, AI engines can be trained to understand full sentences, paragraphs, even entire documents at a time. That gives those engines much more context with which to infer meaning — transformers allow computers to figure out whether you wrote “bank” meaning the house of money or the side of a river or whether the tone of a story is happy or sad or somewhere in between. Transformers are changing the way the whole tech industry thinks about language processing. “This has been an incredible breakthrough,” says Zoubin Ghahramani, a distinguished researcher at Google, “that I feel like has enabled a lot of these other interesting language models to flourish over the last five years.”

Image: Google
This is how transformers work — by encoding and understanding content as a whole, not word by word.

The way Mem sees it, transformers flip the whole model of information management on its head. If the computer can deeply understand and organize your stuff, it means you don’t have to. For as long as digital note-taking and information storage tools have existed, they’ve required users to carefully tend to them. “It takes a lot of time and effort to organize everything,” Xu says, “and keep everything in the right place and make sure you have the right tags. Sometimes it’s just impossible because you don’t realize what connections are going to form and the ways you’re going to want to use a certain piece of information until you get new information.”

As he explains this to me, Xu shares his Mem screen over Google Meet and opens up a mem with a Twitter thread he’s writing about the death of folders and digital organization systems. As he types, an automated list of related mems appears in the right sidebar: a few tweets he’s collected on the subject of information storage, an article from The Verge about how young internet users don’t see the internet in terms of files and folders anymore, a few mems that looked like other drafts of the same thread, an early manifesto Moody wrote about Mem’s plans. “If a human were to read this,” Xu says as he scrolls through the list, “you’d understand what it’s about, right? And what’s surfacing here is not only things I’ve written in the past; it’s things that Kevin has shared with the team, mission statements — you can create this kind of hive mind.”

This is Mem X, Mem’s most advanced auto-organizational tool. Mem X is how Mem becomes a “self-organizing workspace,” the term Xu and Moody have come to use to describe the platform. (It’s also how Mem plans to make money: the basic app will be free, the company has said, but advanced features like Mem X and Writer will come at a price.) Through a partnership with OpenAI, Mem has begun ingesting huge amounts of user data and finding connections without any user input.

Mem It for Twitter is another example: Mem can grab a Twitter thread and generate a summary of its contents while also recommending similar threads from other users. Going forward, Mem wants to do the same for every URL you save to the app — not just understand the URL or even the words in the title but everything on the page and what it’s about. Pair that with knowledge about your history, habits, and taste, and Mem can also become your own personal guide to the internet.

The all-knowing Mem

The tech is far from perfect: Xu, at one point, searches for “transformers” to prove that Mem X understands the difference between language models and Optimus Prime, and indeed Optimus Prime crept into the search results. When he searched for my name — which should bring up all the meetings we’d had together, all the emails I’d sent, and more — it only brought up a few results because I have multiple email aliases. But Mem X drew enough unexpectedly correct connections that the value already outweighs the imperfections. Mem’s AI isn’t perfect, and neither are the fundamentals it’s based on. But they both already work.

Just last week, well over a year after our first conversation, Xu shares his screen with me on Google Meet and shows me something the team just finished — the most futuristic thing Mem has built yet. It’s called Mem Writer, and it pairs your notes with a generative AI to turn note-taking into a collaboration between you and Mem’s AI.

To demonstrate, Xu starts to write what would become a press release announcing Mem’s new $23.5 million round of funding, which OpenAI is leading after seeing what Mem can do. Then he invokes Writer with a slash command and writes, “link to Mem X demo video.” With no further instructions, Mem’s AI goes into Xu’s notes, finds the Loom video with that title, and pastes the link in place.

In another note, Xu had written that he wanted to order some books about large language models and generative AI. Then he writes, “Here’s a list of 10 books that I’m considering:” and has Writer fill out the bulleted list with books like Deep Learning and The Master Algorithm. With another prompt, Writer fetches Amazon links for all 10 books. Another prompt brings in AI-generated summaries of each one.

Image: Mem
Mem’s Writer can automatically find — and generate — information in your notes.

Then, obviously excited by his own progress, Xu continues. “I say, ‘Okay, which of the books above are most similar to books I have read before?’” Xu keeps notes on everything he reads in Mem, because of course he does — he’s the co-CEO. Writer thinks for a minute and spits back an answer: because of two books he’d read in the past, “it seems you might appreciate The Book of Why or The Master Algorithm as they both take a broad, conceptual view of their respective subjects.’”

Again, not all of Xu’s demos work. But some of them do.

If you go along with Mem’s vision for a while, you get to a world in which Mem practically becomes internet infrastructure. When the company raised $5.6 million from Andreessen Horowitz and others in 2021, a16z partner David Ulevitch compared Mem’s potential to the all-knowing Jarvis from Iron Man. And once Mem knows everything you do and see and like, it could also offer a way to share that data with others. “Imagine, let’s say, three years down the line,” Xu says, “there’s a ‘Sign in with Mem’ button, right? And you sign into this product called Newflix, and Newflix asks you, ‘Can we access your list of favorite movie genres?’ And you say, ‘Sure.’” He compares Mem to WeChat in that sense — that it’s not just an app but part of the fabric of users’ online lives.

The app is nothing close to that now, of course, but Mem’s idea of a self-organizing workspace already works surprisingly well. I’ve been using the app off and on for more than a year and find myself continually coming back to Mem because the app is so fast and so good at showing me stuff I forgot I’d even saved that I’ve learned to just dump everything in there and trust that I’ll be able to find it later. The app is hardly a triumph of design, and the mobile app is still very much a minimum viable product, but it’s fast and reliable.

Mem’s ambition seems to grow all the time — the founders occasionally talk about shifting the way the internet works, connecting platforms and reintegrating systems that have been systematically siloed off from one another, maybe one day being a threat to Google itself. But even if they can’t take over the internet and reinvent data ownership, Xu and Moody are pretty sure they can build a search engine that knows your PreCheck number, all about your favorite movies, and whether you should skip the ahi. And they’re pretty sure that’d make your life easier.