[FR]: Support for jinja2 template as prompt templates

[klcd]

Proposal summary

Allow definition of a prompt via a jinja2 file or string that corresponds to the file and add a formatting method to use such prompts in evaluations.

Motivation

Jinja2 and other templating languages allow to separate the data generation and presentation concern when developing prompts. This helps to create more complex prompts where e.g. the memory or another state can be represented without the explicit need for the data.

I've have looked at the code so far and it seems to me that adding a template_type parameter would suffice to be able to differentiate to what rendering method would be used to generate the final prompt. Furthermore, this would make the Prompt easily extensible to other templating languages.

I would be happy to help with the python part.

[jverre]

Hi @klcd ,
Are you using the evaluate_prompt method or just the standard evaluate method ?

It a good suggestion, don't see why we couldn't work on it this week by extending the prompt object. Let me talk to the team tomorrow about it

[klcd]

Hi @jverre

Thanks, happy to help. I believe technically it would be possible to safe it in the meta-data field but I think the property could be worth it to have an actual field in the prompt object.

Great Repo by the way, awesome code quality that makes it easy to understand.

[alexkuzmik]

Hi @klcd!
I like the suggestion 👍
Could you please share an example snippet of how you'd like to use a prompt with an alternative templating engine?
It would also be nice to see an example of prompts you might need. As we can use prompts not only on the SDK side, we will need to think about how to integrate it properly in the other Opik parts.

Great Repo by the way, awesome code quality that makes it easy to understand.

Thank you!

[klcd]

Hi @alexkuzmik

Usage Pattern and some thoughts

From the documentation we have this snipped

prompt = client.create_prompt(name="prompt-summary", prompt="Write a summary of the following text: {{text}}")

In python I would use a StrEnum to pass an additional value e.g.

prompt = client.create_prompt(name="prompt-summary", prompt="Write a summary of the following text: {{text}}", template_type=TemplateTypes.JINJA2)

or equivalently

prompt = client.create_prompt(name="prompt-summary", prompt="Write a summary of the following text: {{text}}", template_type="jinja2")

The default of the parameter should conform to your standard template e.g. TemplateTypes.OPIK.

A similar type probably had to be generated or introduce in the java backend and the typescript frontend. When using the format member of the prompt the type would then be known and no further information needs to be given from the user. That should make it backwards compatible to what you already have and minimal intervention from the user necessary.

Example Prompt Template

You are an expert on paraphrasing, when paraphrasing

1. Ensure that the paraphrases maintain the tone and context of the original sentences.
2. If the original sentence is formal, the paraphrase should also be formal.
3. If the original sentence is informal or colloquial, the paraphrase should reflect that tone.
4. If no sentences are provided, the paraphrasing section will be omitted.

{%- if paraphrases %}
## Paraphrased Sentences

| Original Sentence | Paraphrased Sentence |
| ----------------- | -------------------- |
{%- for original, paraphrased in paraphrases %}
| {{ original }} | {{ paraphrased }} |
{%- endfor %}
{%- endif %}

Hope this is helpful, and let me know if you need more information

[alexkuzmik]

Thanks @klcd, we'll look into it

[klcd]

Great, please note that jinja2 templates are also supporting include and inheriting statements. I would leave them out here on purpose as they would make this quite a bit more involved.

And in case I can help, I'd be happy to help on the python side.

[alexkuzmik]

Hey @klcd! We are going to add a prompt type field to the PromptVersion entity on the backend side soon. Once we do it, the Python SDK will be unblocked to implement this feature :)
I'll keep you posted and get back with some details as soon as we make progress on the backend side!

[alexkuzmik]

@klcd we've added a type field for a prompt version on the backend side.
You can take a look at api/v1/private/prompts/versions POST endpoint for prompt version creation.

We don't have any special processing of that type, it is just being stored for now. However, SDK is now opened for adding jinja2 support.

A tip that might be useful to figure out the changes needed if you work on that: the Prompt on the SDK side is what we call a PromptVersion on the backend side.

[klcd]

Hi @alexkuzmik

Thanks for the update. I'll try to get to it soon