Dify misidentified MARKDOWN file segmentation

[yu154]

Self Checks

• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

Dify version

0.15.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Parent segmentation：### content1
Subclass segmentation：#### content2

✔️ Expected Behavior

content1 （Parent ）
content2（Subclass）

❌ Actual Behavior

content1 （Parent ）
#content2（Parent ）

[dosubot]

Hello, @yu154! I'm here to help you with any bugs, questions, or contributions you have for Dify. Let's work together to resolve the issue you're experiencing.

The issue with Dify misidentifying the segmentation of a MARKDOWN file likely stems from an incorrect JSON payload structure. Specifically, the 'parent_mode' and 'subchunk_segmentation' parameters might be misplaced. They should be placed under the 'rules' object, not directly under 'process_rule'. Here's the correct structure:

{
    "process_rule": {
        "mode": "hierarchical",
        "rules": {
            "segmentation": {
                "separator": "whatever",
                "max_tokens": 2000
            },
            "parent_mode": "paragraph",
            "subchunk_segmentation": {
                "separator": "***",
                "max_tokens": 500
            }
        }
    }
}

Ensure that the segmentation rules are set according to the document format and context requirements. Proper configuration is crucial for accurate content classification and retrieval ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[DavideDelbianco]

I confirm the same behavior; I've also tried with regular expressions.
Considering that most tools tend to convert scraped / extract content to markdown, this is an issue.
Using \n as suggested in the UI to separate chunks is a bad practice in RAG, because you will end up with very little information in each chunk vector and considering that retrieval in Dify is based only on a max TopK of 10 this ends up generating useless response from the AI.