Jun 29, 2023
Hi Wim, here is the response from Jerry Liu, creator of LlamaIndex, so you basically have two approaches, see below, quoting Jerry:
1) by default the SimpleDirectoryReader loads in all documents in-memory but you can customize it so that it loads in one document at a time
2) You can store the parsed nodes into a document store (for instance mongodb). then you get persistent storage instead of storing chunks in-memory