Plans for RAG

[oatmealm]

I’ll install and try asap but wanted to ask, are “instructions” text that’s sent with the prompt as is, assuming it won’t be truncated to fit the context? I’m guessing with Gemini models and others with huge context window it’s not a problem but with ollama, which I believe defaults to 2k this could be an issue (I think gptel might have it hardcoded for 4k or 8k if I remember correctly).

And in this context, what are your plans for rag? I’ve been trying elysa for a while but lately it simply hangs vectorizing the prompt, or is very slow. But the idea sounds cool.

[daedsidog]

I’ll install and try asap but wanted to ask, are “instructions” text that’s sent with the prompt as is, assuming it won’t be truncated to fit the context? I’m guessing with Gemini models and others with huge context window it’s not a problem but with ollama, which I believe defaults to 2k this could be an issue (I think gptel might have it hardcoded for 4k or 8k if I remember correctly).

You can see exactly how the the prompt is crafted by previewing it with evedel-preview-directive-prompt, that will open a new buffer with the prompt there. If by "as is" you mean that the reference text and directive text as being summarized/truncated in some way, then no, they are untouched, and are baked into an existing prompt. So far, no measures have been taken to consider model context limits. If you exceed the model context, you'll get a gptel error.

And in this context, what are your plans for rag? I’ve been trying elysa for a while but lately it simply hangs vectorizing the prompt, or is very slow. But the idea sounds cool.

I haven't decided yet. I would like to leverage the fact that you can tag your instructions and add commentary, in hopes that it will create a much more accurate vectorization. Right now, the approach I have in my head is to vectorize only references and memoize the vectorization, rebuilding the index only when the references were actually changed. We can chunk the references and make them fit the model context constraints.

I know RAG is something that will definitely be added sometime, but at the moment there are other more crucial features I would like to have that will help me work on the package, such as linked instructions. RAG will essentially try to free you from having to supply query tags, which is very lovely.

Converted to discussion.

[oatmealm]

Thanks for the explantion.
Installd and using. Works well.
Question: what's the strategy for removing generated content? Simply revert the buffer?
Hopefully there will be a standard library for emacs (like llm.el) for vectorizing content which the other package can use, the way gptel and llm are being leveraged right now. Automatically vectorzing contents on save for example (with exceptions and black lists) would be nice I think.