What are memories, how they are created and used in LLMs, how they will lead to more accurate Health AI and what new risks will surface.
The Problem
One of the common problems in LLMs is CONTEXT.
Suppose you ask a question like “Who are my company’s competitors?”. To answer this question the LLM would need to know the name of your company (i.e., Context).
Suppose you ask “What is the best bicycle to buy?” The LLM would need to know whether you’re a professional racer, a weekend bike junkie or just starting out.
The Current Solution
The common solution in the past has been to continue the same chat thread with the LLM so it can use the previous context. People use the same chat thread for months asking it all their diverse questions.
The problem with this approach is that it is really expensive to keep processing your past chat history (LLMs charge per token/word) but more importantly you may be asking a completely different question than you asked yesterday.
Perhaps yesterday you asked about your company’s competitors but today you’re asking about which bicycle to buy. The chat history about yesterday’s conversation will tend to confuse the LLM and it is more likely to hallucinate or give you sub-optimal answers.
Introducing Memory
The new solution is to have the LLM form memories about you from past conversations and use those memories when answering your questions. Examples of memories would be:
- Works at Microsoft
- Bikes 10 miles on every weekend
Now in new conversations LLMs can use these memories as context to answer your questions. You can now create new chat sessions when your topic changes without needing to continue a long chat session about many different topics.
Types of Memories
There are three main types of memories:
- Personal memories – these are memories unique to you. For example, “Bikes on weekends”
- Group memories – these are memories that are shared by a group e.g., “Works at Microsoft”
- Process memories – these are memories about how a certain process is done e.g., “Baking a bread requires an oven and flour”.
Personal memories are only used when YOU are interacting with the LLM.
Group memories are used when a member of that group is interacting with the LLM. You are a member of different groups; your department, your company, your neighborhood and your family.
Process memories are used when someone is asking about that particular process, “How do I bake bread?”
How Memories Are Formed
The most common technique today is to use the LLM to generate memories from past conversations and from additional documents such as company mission statements, department status updates, product data sheets and software code.
This can be as simple as using a prompt, “Extract memories from this text”. However typically your AI platform should use more sophisticated techniques.
The AI platform stores these memories about the group and about the current user. The group memories are stored in a database under the group and personal memories are stored in a database under the person.
Process memories can be formed by feeding the LLM documentation about that process such as flowcharts, user documentation and process specifications. Process memories are also stored in a database under the process identifier.
How Memories Are Used
Now when a user asks a question, we can search their group memories and their personal memories for relevant memories and insert those in the context window of the LLM as a prompt. This search is typically done by comparing the text embedding of the question with the embedding of each memory to find most closely related memory. (If you don’t understand text embeddings, read my previous post: https://healthcareconsumer.ai/part-5-embeddings-are-the-coordinates-of-the-llm/)
For example, if I ask “Who are my company’s competitors”, the text embedding of this would match the text embedding of the group memory “Works at Microsoft” because “my company” and “works at” are similar concepts. So we would insert the “Works at Microsoft” automatically in the context window. Now the LLM will answer the question with the competitors to Microsoft.
Now I start a new chat session and ask “What bicycle should I buy?”. The text embedding of this question will not match the text embedding of the memory “Works at Microsoft” but will match the personal memory of “bikes 10 miles on every weekend” since “bicycle” and “bike” are similar concepts. So we would insert the “bikes 10 miles every weekend” into the context window of the current chat session before answering “What bicycle should I buy?”
How Memory Will Change Companies
Companies need to personalize the AI experience of their employees to understand the context of the company, the department and the employee. Otherwise their employees are getting answers based on publicly available data only!
A company can feed their internal documents to create group memories and store them. Then every question of their employees used these group memories to get more accurate answers.
In addition, companies can create memories about the role of each person and what they do at the company. These personal memories can then be used to personalize answers for each employee.
How Memory Will Change Healthcare
Healthcare is ALL about context.
Two patients asking the same question “What should I do about my foot pain?” should get very different answers based on their context.
If Jim is a diabetic then he should get answers focused on foot problems associated with diabetes.
On the other hand, Alice who runs 10 miles a day should get answers focused on foot pain associated with running.
A cardiologist asking about a Lipid Panel test of a patient should get a different answer than a Primary Care physician asking about the same test or a patient asking about the same test.
I believe memories will be a key part in how Health AI will provide more accurate answers. Successful Health AI solutions will create and store memories about the group, the patient and the process. They will then use these memories to personalize the answers we give to each user.
What About Security and Privacy?
I believe that users should be in control of what memories are stored about them.
AI solutions, especially Health AI solutions, should provide transparency to users about all the memories they’ve formed about them. Users should be able to delete memories that they do not believe correctly portray them. And users should be in full control of what these memories are used for and who they can be shared with.
Memories are more valuable than social security numbers!
What The Future Holds
Today we store our credit cards and our identities (like mobile driver licenses) in the Apple Wallet or Google Wallet on our phones. In the future, we will be able to store our memories in our mobile memory wallet.
These memories can then be used when we interact with various apps and AI assistants similar to how we can use credit cards in our mobile wallet at restaurants.
These stored memories will be key to everyone getting a very personalized experience whichever app or AI assistant they interact with.
The Upcoming Risk
Today company valuations are based on how much data they have about their users. Companies like Facebook (your social media history), Google (your search history) and OpenAI (your chat history) have higher valuations because of the data they hold about their users.
In the future, company valuations will be based on how many memories they have about their users. Memories are a more compact way to understand a person than their social media history, their search history or their chat history.
Because of this, companies will try to acquire as many memories about their users and use them for increasing revenue. Think about how companies use web browser cookies in one website to sell you something in another website. The first company gets a referral fee. This is why Facebook shows you ads about the same stuff you just Googled for!
It will be important for consumers to have control over their memories and what they are used for.
Summary
Memory is a new concept in LLMs that holds great promise. LLMs can form memories based on past conversations and additional documents.
There are three types of memories: Personal, Group and Process.
Memory will lead to more accurate and better personalized answers from AI.
Company Valuations will be based on the breadth and depth of memories they hold about their users instead of the data and engagement history that determines company valuations today.
Memories will be game changing for AI accuracy for companies and healthcare.
However, they will create a new risk about who controls your memories and what they can use them for.