[Feature] Dealing with Formatted responses

[aiadvantageuser]

What feature would you like to see?

I am creating a prompt where i am simulating a debate. The prompt for my LM is the following:
This is a multi round debate with 3 different roles:
-Affarmative debater: who tries to prove the rating is correct
-Negative debater: who tries to prove the rating is incorrect
-Moderator: who will compare the two debaters' arguments for the given requirement and evaluation criteria and draws a conclusion

The parameter which the debate is about it the following:
The need is accuretly represented in the requirement.

The ratings are from 1-5, 5 means strongly agree.

The debate goes on for three turns. The moderator has to draw a conclusion after the last round and if there is a chosen side, give the final rating and answer by the end of the third round.

And for this prompt the following json_schema is given as response format:
{
"name": "debate_schema",
"strict": true,
"schema": {
"type": "object",
"properties": {
"rounds": {
"type": "array",
"description": "Three rounds of debate.",
"items": {
"type": "object",
"properties": {
"affirmative": {
"type": "object",
"description": "The statement made by the affirmative debater.",
"properties": {
"argument": {
"type": "string",
"description": "Argument proving the rating is correct."
},
"rating": {
"type": "integer",
"description": "Rating given by the affirmative debater, ranging from 1 to 5."
}
},
"required": [
"argument",
"rating"
],
"additionalProperties": false
},
"negative": {
"type": "object",
"description": "The statement made by the negative debater.",
"properties": {
"argument": {
"type": "string",
"description": "Argument proving the rating is incorrect."
},
"rating": {
"type": "integer",
"description": "Rating given by the negative debater, ranging from 1 to 5."
}
},
"required": [
"argument",
"rating"
],
"additionalProperties": false
}
},
"required": [
"affirmative",
"negative"
],
"additionalProperties": false
}
},
"moderator": {
"type": "object",
"description": "The moderator's final conclusion.",
"properties": {
"conclusion": {
"type": "string",
"description": "The summary conclusion drawn by the moderator."
},
"final_rating": {
"type": "integer",
"description": "Final rating decided by the moderator, ranging from 1 to 5."
}
},
"required": [
"conclusion",
"final_rating"
],
"additionalProperties": false
}
},
"required": [
"rounds",
"moderator"
],
"additionalProperties": false
}
}

For this i would like to create a DSPY module. I tried with the following code:

class RequirementToRating(dspy.Signature):
# Input
question: str = dspy.InputField()

# Outputs
debate_json: str = dspy.OutputField()  # Store JSON response as a string

class RatingRound(dspy.Module):
def init(self):
super().init()
self.rating_extractor = dspy.Predict(RequirementToRating)

def forward(self, question: str) -> dspy.Prediction:

    rating_output = self.rating_extractor(question=question)
    raw_json = rating_output.debate_json  # Extracted JSON as a string

    print("Raw LLM Output:", raw_json)  # Debugging output

    # Parse JSON string into a dictionary
    try:
        debate_data = json.loads(raw_json)  # Convert string to dict
    except json.JSONDecodeError:
        print("Error parsing JSON output")
        return None  

    # Extract relevant fields
    rounds = debate_data.get("rounds", [])
    moderator = debate_data.get("moderator", {})

    # Extract ratings from each debate round safely
    round1_affirmative = rounds[0]["affirmative"]["rating"] if len(rounds) > 0 else None
    round1_negative = rounds[0]["negative"]["rating"] if len(rounds) > 0 else None

    round2_affirmative = rounds[1]["affirmative"]["rating"] if len(rounds) > 1 else None
    round2_negative = rounds[1]["negative"]["rating"] if len(rounds) > 1 else None

    round3_affirmative = rounds[2]["affirmative"]["rating"] if len(rounds) > 2 else None
    round3_negative = rounds[2]["negative"]["rating"] if len(rounds) > 2 else None

    final_rating = moderator.get("final_rating", None)
    conclusion = moderator.get("conclusion", "")

    # Return extracted ratings
    return dspy.Prediction(
        round1_affirmative=round1_affirmative,
        round1_negative=round1_negative,
        round2_affirmative=round2_affirmative,
        round2_negative=round2_negative,
        round3_affirmative=round3_affirmative,
        round3_negative=round3_negative,
        final_rating=final_rating,
        conclusion=conclusion
    )

I would really appreciate it if you could give me ideas or any support how to do this

Would you like to contribute?

• Yes, I'd like to help implement this.
• No, I just want to request it.

Additional Context

No response

[okhat]

Hey @aiadvantageuser . I'll confess this was a bit long so I couldn't read everything but should you perhaps use typed outputs in DSPy? You can specify a lot of structure in the output fields and their types, and you can make the types of some or all output fields be Pydnatic models.

[aiadvantageuser]

Thank you for the fast answer @okhat . I tried out with the Pydantic models, specifically with this example:

Code:

from pydantic import BaseModel, Field, conint
from typing import List

class DebaterStatement(BaseModel):
argument: str = Field(description="Argument proving the rating is correct or incorrect.")
rating: conint(ge=1, le=5) = Field(description="Rating given by the debater, ranging from 1 to 5.")

class DebateRound(BaseModel):
affirmative: DebaterStatement = Field(description="The statement made by the affirmative debater.")
negative: DebaterStatement = Field(description="The statement made by the negative debater.")

class ModeratorDecision(BaseModel):
conclusion: str = Field(description="The summary conclusion drawn by the moderator.")
final_rating: conint(ge=1, le=5) = Field(description="Final rating decided by the moderator, ranging from 1 to 5.")

class DebateSchema(BaseModel):
rounds: List[DebateRound] = Field(description="Three rounds of debate.")
moderator: ModeratorDecision = Field(description="The moderator's final conclusion.")

class RequirementToRating(dspy.Signature):
# Input
question: str = dspy.InputField()

# Output
debate_json: DebateSchema = dspy.OutputField()

predict = dspy.Predict(RequirementToRating)

print(predict(question="Certificates Replacement shall be completed within max 5000 ms"))

---

I replaced TypedPrecidot with dspy.Predict as i saw in an earlier issue, but i still got the following error. Any idea what goes wrong?

Error:

ValueError Traceback (most recent call last)
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\dspy\adapters\base.py:25, in Adapter.call(self, lm, lm_kwargs, signature, demos, inputs, _parse_values)
24 for output in outputs:
---> 25 value = self.parse(signature, output, _parse_values=_parse_values)
26 assert set(value.keys()) == set(signature.output_fields.keys()), f"Expected {signature.output_fields.keys()} but got {value.keys()}"

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\dspy\utils\callback.py:202, in with_callbacks..wrapper(instance, *args, **kwargs)
201 if not callbacks:
--> 202 return fn(instance, *args, **kwargs)
204 # Generate call ID as the unique identifier for the call, this is useful for instrumentation.

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\dspy\adapters\chat_adapter.py:85, in ChatAdapter.parse(self, signature, completion, _parse_values)
84 if fields.keys() != signature.output_fields.keys():
---> 85 raise ValueError(f"Expected {signature.output_fields.keys()} but got {fields.keys()}")
87 return fields

ValueError: Expected dict_keys(['debate_json']) but got dict_keys([])

During handling of the above exception, another exception occurred:

TypeError Traceback (most recent call last)
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\dspy\adapters\json_adapter.py:37, in JSONAdapter.call(self, lm, lm_kwargs, signature, demos, inputs, _parse_values)
36 provider = lm.model.split("/", 1)[0] or "openai"
---> 37 if "response_format" in litellm.get_supported_openai_params(model=lm.model, custom_llm_provider=provider):
38 outputs = lm(**inputs, **lm_kwargs, response_format={"type": "json_object"})

TypeError: argument of type 'NoneType' is not iterable

During handling of the above exception, another exception occurred:

AttributeError Traceback (most recent call last)
Cell In[20], line 76
72 debate_json: DebateSchema = dspy.OutputField()
74 predict = dspy.Predict(RequirementToRating)
---> 76 print(predict(question="Certificates Replacement shall be completed within max 5000 ms"))
79 """class RatingRound(dspy.Module):
80 def init(self):
81 super().init()
(...)
132 print(response)
133 """
136 """def load_debate_dataset(excel_path, sheet_name='Requirements_1'):
137 df = pd.read_excel(excel_path, sheet_name=sheet_name)
138 df['Correct'] = df['Correct'].astype(int)
(...)
157 )
158 tiny_evaluater(pr, metric=dspy.evaluate.answer_exact_match)"""

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\dspy\utils\callback.py:202, in with_callbacks..wrapper(instance, *args, **kwargs)
200 # If no callbacks are provided, just call the function
201 if not callbacks:
--> 202 return fn(instance, *args, **kwargs)
204 # Generate call ID as the unique identifier for the call, this is useful for instrumentation.
205 call_id = uuid.uuid4().hex

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\dspy\predict\predict.py:154, in Predict.call(self, **kwargs)
152 @with_callbacks
153 def call(self, **kwargs):
--> 154 return self.forward(**kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\dspy\predict\predict.py:188, in Predict.forward(self, kwargs)
185 import dspy
187 if isinstance(lm, dspy.LM):
--> 188 completions = v2_5_generate(lm, config, signature, demos, kwargs, _parse_values=self._parse_values)
189 else:
190 warn_once(
191 "\t* In DSPy 2.5, all LM clients except dspy.LM are deprecated, "
192 "underperform, and are about to be deleted. ***\n"
(...)
197 " \t\thttps://github.com/stanfordnlp/dspy/blob/main/examples/migration.ipynb"
198 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\dspy\predict\predict.py:295, in v2_5_generate(lm, lm_kwargs, signature, demos, inputs, _parse_values)
291 import dspy
293 adapter = dspy.settings.adapter or dspy.ChatAdapter()
--> 295 return adapter(
296 lm, lm_kwargs=lm_kwargs, signature=signature, demos=demos, inputs=inputs, _parse_values=_parse_values
297 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\dspy\adapters\base.py:33, in Adapter.call(self, lm, lm_kwargs, signature, demos, inputs, _parse_values)
31 from .json_adapter import JSONAdapter
32 if _parse_values and not isinstance(self, JSONAdapter):
---> 33 return JSONAdapter()(lm, lm_kwargs, signature, demos, inputs, _parse_values=_parse_values)
34 raise e

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\dspy\adapters\json_adapter.py:42, in JSONAdapter.call(self, lm, lm_kwargs, signature, demos, inputs, _parse_values)
39 else:
40 outputs = lm(**inputs, **lm_kwargs)
---> 42 except litellm.UnsupportedParamsError:
43 outputs = lm(**inputs, **lm_kwargs)
45 values = []

AttributeError: module 'litellm' has no attribute 'UnsupportedParamsError'