Adventures with AI-powered notetaking
Introducing AutoNotes: hierarchical tags, chat with your notes, and highlights to help uncover hidden gems in your personal notes
By Vivian Tsai, Ellen Jiang, Adam Connors, Alejandra Molina, & Aaron Donsbach
From meeting minutes to grocery lists, people have plenty of experience taking notes. So why does it often feel difficult?
As designers and engineers who are passionate about notetaking (and in the spirit of PAIR’s AI-Augmented Life project), we spent two weeks experimenting with how Gemini could help. We quickly realized that the hard part isn’t actually taking notes — it’s finding them later. In response, we created AutoNotes (view live demo and code)!
AutoNotes is an experiment — inspired by NotebookLM’s chat-based Q&A system and Google Keep’s quick note capture — that introduces new, AI-powered ways to organize and explore your personal notes:
- Hierarchical tagging. Click through top-level categories (#food) and nested subcategories (#soup) to read summaries and discover compelling details from your notes (e.g., from #food to #food/soup to “beautiful soup”).
- Chat with your notes. Use our chat interface to ask Gemini questions about your notes and receive answers that link back to their original sources.
- Fun highlights. Generate a personalized overview of your notes, including direct quotes, writing style, and overall impressions.
In this article, we discuss our journey building AutoNotes: how we decided and iterated on these features, and what we discovered at the end.
A focus on “finding”
We all take notes in different ways and for different reasons:
- As reminders — Alejandra texts herself to remember book titles, ticket numbers, and doctor’s appointments, while Vivian makes weekly checklists in a bullet journal
- As references — Ellen uses her phone to capture art inspiration, including photos, works by other artists, and one-line ideas
- For journaling — Adam types in-depth journal entries about once a month
- For brainstorming — Aaron uses a digital tablet to sketch out and iterate on project ideas
But the one thing we had in common was our struggle to find information later. In particular, we often sought:
- A way to proactively explore notes in depth — rather than relying on a high-level summary, we wanted a quick way to determine what’s interesting and an easy path to the source material
- Known but “fuzzier” items, like an article whose title we couldn’t quite remember
- Things we didn’t know we were looking for, like a forgotten but delightful journal entry, or a delicious recipe for a rainy day
As we experimented with how AI might play a role in content retrieval, each of these needs led to one of AutoNotes’s final features.
Experiment #1: Hierarchical tagging
To enable proactive note exploration, we needed a user-friendly organizational system. We started with tags and summaries: for each note created, we auto-tagged it by category; for each tag, we auto-summarized the notes underneath it.
While this created an easy-to-navigate structure, the tags and summaries didn’t feel granular enough for spontaneous exploration.
Then we thought: What if we adopted a two-layer hierarchy of tags?
With both top-level tags (#food) and subtags (#food/marmalade, #food/soup), we were able to keep the first layer of tags finite and, at the same time, generate summaries for the second layer that felt much more specific and interesting:
#food/soup: “One note mentions… a large cauldron full of soup. Another note simply states that the soup is beautiful.”
To allow users to easily navigate, we nested subtags/summaries in collapsible containers under top-level tags/summaries.
This is when the hierarchical tagging started to really feel magical, encouraging exploration and enabling users to serendipitously uncover hidden gems in their notes.
In our test Anne of Green Gables dataset, we expected categories vaguely following the book’s plot, but were instead rewarded with delightful collections such as #nature, which quoted and linked to beautiful descriptions across Anne’s notes.
We also enjoyed diving into completely new datasets. When Ellen used AutoNotes to browse Alejandra’s reminders, she used the top-level categories and summaries to immediately orient herself and — despite Alejandra’s claims that her notes were 100% boring — quickly tunneled into a fun memo about Alejandra training her cat to avoid cables.
Automatically generating a tag hierarchy is a simple mechanism, but it was surprisingly effective at helping us navigate complex, unstructured notes and discover content we didn’t know we were looking for. We’re excited to continue experimenting with this kind of affordance to understand which use cases are most enabled by it.
Experiment #2: Chat with your notes
When discussing how to find note content quickly, we imagined a version of NotebookLM’s chat that could construct answers from shorter, more chaotic notes (like the ones we often take on the go).
To create the “chat with your notes” feature, we asked Gemini to:
- Select up to 5 notes most relevant to the latest user chat (given all AutoNotes notes)
- Generate a chat response given some conversational examples, the conversation’s actual history, and the “relevant” notes for context
This almost immediately solved our use case of finding “fuzzier” notes.
More interestingly, we discovered that a conversational query to Gemini might not only locate note content, but also bring in outside context. Instead of only asking, “What are the steps to the lobster dance?” (based on a prior note), a user might request, “Compare [steps from the lobster dance] to the steps of a real quadrille.”
This mix of note-based and Web-based content was an unexpected — but useful — side effect that quickly elevated our ability to research topics and brainstorm ideas. To help accommodate this feature, we added the option to save chat responses as notes.
Experiment #3: Fun highlights
While our hierarchical tagging allowed users to find “things we didn’t know we were looking for,” we also wanted an immediate way to preview delightful excerpts.
On our last day experimenting with AutoNotes, we tried a series of prompts to extract eloquent quotes, favorite people, and other bite-sized revelations across notes, then bundled the best ones to create Highlights.
This whimsical spin on summaries was exactly what we were hoping for! Requiring Gemini to build recaps only from our own words resulted in a much more tailored and charming experience.
We only had a short amount of time to play with Highlights, but in the future, we’d love to see other fun and creative snippets added to this reel.
What’s next
Through our three experiments, we found that our simple, AI-powered features had a profound impact on how we might access, analyze, and grow from our own notes. AI opens up many new possibilities for the way we interact with the hazy, crazy, wonderful mess inside our heads, and we think there is much more to be discovered in the notetaking space.
While we don’t currently have future plans for AutoNotes, we do have plenty of ideas for new and extended features:
- Multimodal capabilities. Many of us sketch diagrams or save photos as part of our notetaking. It’d be great to include that content in the AutoNotes tagging and chat workflows — and it might even lead to some more colorful collapsible summaries!
- Insights based on location and time. Right now, we’re only sending note content to Gemini — but we’d love for note creation/edited date and location to be considered. This might not only cut out irrelevant material (e.g., a daily checklist from three years ago), but also uncover interesting trends (e.g., how your habits or writing style changed over time).
- Related notes panel. When first discussing the problem of finding notes, we learned that all of us fall into the trap of starting a new note (project idea, movie watchlist, etc.) when we could’ve continued an existing one. What if there was a way to see related notes (perhaps in a side panel) while constructing a new draft, with a way to optionally merge your draft into one of those notes? We’re not sure how this would look, but we think it could be useful.
Try AutoNotes for yourself
AutoNotes is an experiment, not a product — we built it as a temporary surface to imagine new possibilities — but we’re putting it out in the world so that you can try it out for yourself (or add your own features!).
To use AutoNotes, check out our live demo, where you can explore results via our test datasets, upload existing notes from Markdown or Google Keep, or create new notes directly in the interface (add your own Gemini API key to chat with or tag any new content). You can also view or fork the code on GitHub.
Happy note-finding!
This project was created by: Ellen Jiang and Vivian Tsai, software engineers (PAIR); Aaron Donsbach and Alejandra Molina, UX designers (AIUX); Adam Connors, software engineer and resident writer (PAIR).
