Wenqi Glantz
Jun 29, 2023

--

Hi Wim, here is the response from Jerry Liu, creator of LlamaIndex, so you basically have two approaches, see below, quoting Jerry:

1) by default the SimpleDirectoryReader loads in all documents in-memory but you can customize it so that it loads in one document at a time

2) You can store the parsed nodes into a document store (for instance mongodb). then you get persistent storage instead of storing chunks in-memory

--

--