Parameter update for use of Firecrawl V1 API

[ftonato]

Self Checks

• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

Dify version

NA

Cloud or Self Hosted

Self Hosted (Source)

Steps to reproduce

When trying to use Include & Exclude parameters the API makes no correct use of them

✔️ Expected Behavior

Include and Exclude parameters are passed correctly to the requests to /crawl

❌ Actual Behavior

There are some incorrect parameters to use the V1 version of the Firecrawl

[dosubot]

Hey there, @ftonato! I'm here to help you with any bugs, questions, or contributions you have for the project. Let's get this sorted out together.

To ensure the Include and Exclude parameters are correctly used with the Firecrawl V1 API, make sure they are passed as lists of strings. Here's how you can handle these parameters:

1. Extract Parameters: Use the split(",") method to convert comma-separated strings into lists.
2. Default to Empty Lists: If the parameters are not provided, default them to empty lists.

Here's a code snippet demonstrating this:

includes = options.get("includes").split(",") if options.get("includes") else []
excludes = options.get("excludes").split(",") if options.get("excludes") else []
params = {
    "includePaths": includes,
    "excludePaths": excludes,
    "generateImgAltText": True,
    "limit": options.get("limit", 1),
    "scrapeOptions": {"onlyMainContent": only_main_content},
}

Ensure you use the correct parameter names includePaths and excludePaths as specified in the API documentation ^[1][2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}