Files variable in Custom tool cannot be recognized correctlly

[zqy22]

Self Checks

• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

Dify version

0.11.1

Cloud or Self Hosted

Self Hosted (Source)

Steps to reproduce

I defined a new tool and tried to upload the file in the tool return value, but it kept being recognized as text
here is my code:

import requests
from flask import Flask, request, jsonify, url_for
import os
import mimetypes

app = Flask(name)

@app.route('/segment-image_6002', methods=['POST'])
def segment_image():
input_data = request.json
print(f"Received input_data: {input_data}")

if not input_data:
    return jsonify({"error": "Missing required parameter"}), 400

image_path = input_data.get("pic_need_judge")
if not image_path:
    return jsonify({"error": "Missing pic_need_judge parameter"}), 400

image_path = image_path.strip()
if not os.path.exists(image_path):
    return jsonify({"error": f"Invalid or missing image path: {image_path}"}), 400

# 调用本地深度估计服务
api_url = "http://localhost:6000/api/v1/depth-estimation"
data = {"image_path": image_path}

try:
    response = requests.post(api_url, json=data)
except requests.exceptions.RequestException as e:
    return jsonify({"error": f"Request failed: {str(e)}"}), 500

if response.status_code == 200:
    result_data = response.json()
    message = result_data.get("result_text", "No message returned")
    processed_image_path = result_data.get("image_path")

    if not processed_image_path or not os.path.exists(processed_image_path):
        return jsonify({"error": f"Invalid or missing processed image path: {processed_image_path}"}), 400

    # 准备文件信息
    file_name = os.path.basename(processed_image_path)
    file_size = os.path.getsize(processed_image_path)
    mime_type, _ = mimetypes.guess_type(processed_image_path)

    file_info = {
        "name": file_name,
        "size": file_size,
        "type": "image",
        "extension": file_name.split('.')[-1],
        "mime_type": mime_type,
        "transfer_method": "url",
        "url": url_for('static', filename=file_name, _external=True)  # 确保文件可以通过URL访问
    }

    # 构造返回值
    return jsonify({
        "text": message,
        "files": [file_info],
        "json": {"result": "success"}
    }), 200
else:
    error_message = response.text
    print(f"Error response from depth estimation service: {error_message}")
    return jsonify(
        {"error": f"Request failed with status {response.status_code}: {error_message}"}), response.status_code

if name == "main":
app.run(host='0.0.0.0', port=6002, debug=True)

✔️ Expected Behavior

I want to upload the image file to the "files" of the output variable

❌ Actual Behavior

My uploads are all recognized as text

[dosubot]

To ensure that the file is correctly recognized as a file rather than text in the output, you need to ensure that the files variable is structured correctly according to Dify's expected format. Here are a few things to check and adjust in your code:

1. Transfer Method: Ensure that the transfer_method in your file_info dictionary is set correctly. If you're using a URL, it should be "transfer_method": "REMOTE_URL" instead of "url".

2. File Structure: The file_info dictionary should include all necessary fields that Dify expects for file handling. Here's a refined version of your file_info dictionary:

   file_info = {
       "name": file_name,
       "size": file_size,
       "type": "image",
       "extension": file_name.split('.')[-1],
       "mime_type": mime_type,
       "transfer_method": "REMOTE_URL",  # Correct transfer method
       "url": url_for('static', filename=file_name, _external=True)
   }

3. Ensure URL Accessibility: Make sure that the URL generated by url_for is accessible and correctly points to the file location. The file should be placed in a directory that Flask's static file handler can serve.

4. Validation: Double-check that the file is being correctly validated and processed in the Dify workflow. The files variable should be a list of dictionaries, each representing a file with the correct metadata.

By ensuring these elements are correctly implemented, the file should be recognized as a file in the output rather than text. If the issue persists, verify that the Dify environment is correctly configured to handle file uploads and that there are no additional constraints or settings affecting file recognition.

To continue talking to Dosu, mention @dosu.

[zqy22]

To ensure that the file is correctly recognized as a file rather than text in the output, you need to ensure that the files variable is structured correctly according to Dify's expected format. Here are a few things to check and adjust in your code:

1. Transfer Method: Ensure that the transfer_method in your file_info dictionary is set correctly. If you're using a URL, it should be "transfer_method": "REMOTE_URL" instead of "url".
2. File Structure: The file_info dictionary should include all necessary fields that Dify expects for file handling. Here's a refined version of your file_info dictionary:

   file_info = {
       "name": file_name,
       "size": file_size,
       "type": "image",
       "extension": file_name.split('.')[-1],
       "mime_type": mime_type,
       "transfer_method": "REMOTE_URL",  # Correct transfer method
       "url": url_for('static', filename=file_name, _external=True)
   }

3. Ensure URL Accessibility: Make sure that the URL generated by url_for is accessible and correctly points to the file location. The file should be placed in a directory that Flask's static file handler can serve.
4. Validation: Double-check that the file is being correctly validated and processed in the Dify workflow. The files variable should be a list of dictionaries, each representing a file with the correct metadata.

By ensuring these elements are correctly implemented, the file should be recognized as a file in the output rather than text. If the issue persists, verify that the Dify environment is correctly configured to handle file uploads and that there are no additional constraints or settings affecting file recognition.

To continue talking to Dosu, mention @dosu.

It doesn't work. Output variable:
It doesn't work, output variable:
{
"text": "{"files": [{"extension": "png", "mime_type": "image/png", "name": "8_detected.png", "size": 625178, "transfer_method": "REMOTE_URL", "type": "image", "url": "http://localhost:6002/static/8_detected.png"}], "json": {"result": "success"}, "text": ["Label: car 0.76, Score: 0.98, Depth: 7.20m", "Label: rail track 0.68, Score: 0.91, Depth: 5.87m", "Label: human 0.55, Score: 0.91, Depth: 1.30m"]}",
"files": [],
"json": []
}
@dosu.

[dosubot]

It seems like the issue might be related to how the output is being structured or processed. Here are a few things to check and adjust:

1. Output Structure: Ensure that the output JSON is structured correctly. The files key should contain a list of file dictionaries, and it should not be nested within the text key. Here's an example of how the output should be structured:

   {
       "text": "Label: car 0.76, Score: 0.98, Depth: 7.20m",
       "files": [
           {
               "name": "8_detected.png",
               "size": 625178,
               "type": "image",
               "extension": "png",
               "mime_type": "image/png",
               "transfer_method": "REMOTE_URL",
               "url": "http://localhost:6002/static/8_detected.png"
           }
       ],
       "json": {
           "result": "success"
       }
   }

2. Ensure Correct JSON Formatting: Make sure that the JSON response is correctly formatted and that the text field is not mistakenly containing the entire JSON structure as a string. The text field should only contain the textual information you want to return.

3. Check for Processing Errors: Verify that there are no errors in the processing logic that might be causing the files list to be empty in the final output. Ensure that the file information is correctly added to the files list before returning the response.

4. Debugging: Add logging to check the state of the file_info and the output variable before returning the response. This can help identify where the structure might be going wrong.

By ensuring these elements are correctly implemented, the file should be recognized as a file in the output rather than text. If the issue persists, consider reviewing the Dify environment configuration to ensure it is set up to handle file uploads correctly.

To continue talking to Dosu, mention @dosu.

[zqy22]

It doesn't work and seems like a bug @dosu.

It seems like the issue might be related to how the output is being structured or processed. Here are a few things to check and adjust:

1. Output Structure: Ensure that the output JSON is structured correctly. The files key should contain a list of file dictionaries, and it should not be nested within the text key. Here's an example of how the output should be structured:

   {
       "text": "Label: car 0.76, Score: 0.98, Depth: 7.20m",
       "files": [
           {
               "name": "8_detected.png",
               "size": 625178,
               "type": "image",
               "extension": "png",
               "mime_type": "image/png",
               "transfer_method": "REMOTE_URL",
               "url": "http://localhost:6002/static/8_detected.png"
           }
       ],
       "json": {
           "result": "success"
       }
   }

2. Ensure Correct JSON Formatting: Make sure that the JSON response is correctly formatted and that the text field is not mistakenly containing the entire JSON structure as a string. The text field should only contain the textual information you want to return.
3. Check for Processing Errors: Verify that there are no errors in the processing logic that might be causing the files list to be empty in the final output. Ensure that the file information is correctly added to the files list before returning the response.
4. Debugging: Add logging to check the state of the file_info and the output variable before returning the response. This can help identify where the structure might be going wrong.

By ensuring these elements are correctly implemented, the file should be recognized as a file in the output rather than text. If the issue persists, consider reviewing the Dify environment configuration to ensure it is set up to handle file uploads correctly.

To continue talking to Dosu, mention @dosu.

[dosubot]

If the issue persists despite ensuring the correct JSON structure and formatting, it might be related to how Dify processes the files variable. Here are additional steps to consider:

1. Environment Configuration: Double-check the Dify environment settings to ensure they are configured to handle file uploads correctly. Key settings include STORAGE_TYPE, STORAGE_LOCAL_PATH, and file size limits like UPLOAD_FILE_SIZE_LIMIT ^[1].

2. File Handling in Dify: Ensure that the file handling logic in your Dify setup is correctly implemented. Dify requires specific configurations for file uploads, and any misconfiguration might lead to issues with file recognition ^[2].

3. Legacy Features: Be aware that certain features like sys.files are marked as LEGACY and might not function as expected. It's recommended to use custom file variables instead ^[2].

4. Debugging: Add detailed logging to track the flow of data and identify where the file information might be getting lost or misinterpreted. This can help pinpoint the exact step where the issue occurs.

5. Review Documentation: Refer to the Dify documentation for any additional guidance on handling files, especially if there are updates or changes in the version you are using.

If these steps do not resolve the issue, it might be beneficial to review any recent changes in the Dify version you are using or consult the documentation for any known issues or updates related to file handling.

To continue talking to Dosu, mention @dosu.

[crazywoola]

Please upgrade to latest version, we added file support for tool in later version #10796

[zqy22]

Please upgrade to latest version, we added file support for tool in later version #10796

I have upgraded to 12.1, it doesn't work neither.
This is my OPENAPI docs:
openapi: 3.0.0
info:
title: Image Segmentation API
version: 1.0.0
description: This API segments an image based on a provided URL and returns a segmented image along with a text message.
servers:

• url: http://localhost:6002
  description: Local development server
  paths:
  /segment-image_6002:
  post:
  summary: Segment an image and return the segmented image and a text message.
  description: This endpoint accepts an image URL and returns a segmented image along with a text message in a multipart response.
  requestBody:
  required: true
  content:
  application/json:
  schema:
  type: object
  properties:
  pic_need_judge:
  type: string
  format: uri
  description: The URL of the image to be segmented.
  required:
  - pic_need_judge
  example:
  pic_need_judge: "http://example.com/image.jpg"
  responses:
  '200':
  description: Successful operation. Returns the segmented image and a text message.
  content:
  multipart/form-data:
  schema:
  type: object
  properties:
  text:
  type: string
  description: A text message describing the result.
  file:
  type: string
  format: binary
  description: The segmented image file.
  encoding:
  file:
  contentType: image/jpeg
  example:
  text: "Segmentation successful"
  file: ""
  '400':
  description: Bad request. Missing or invalid parameters.
  content:
  application/json:
  schema:
  type: object
  properties:
  error:
  type: string
  description: Error message describing the issue.
  example:
  error: "Missing required parameter: pic_need_judge"
  '500':
  description: Internal server error. An unexpected error occurred.
  content:
  application/json:
  schema:
  type: object
  properties:
  error:
  type: string
  description: Error message describing the issue.
  example:
  error: "Request failed with status 500: Internal Server Error"
  And this is my python file:
  from flask import Flask, request, jsonify, Response
  import requests
  import mimetypes
  import os # 导入 os 库

app = Flask(name)

固定的 boundary 字符串

BOUNDARY = 'fixedboundary1234567890'

@app.route('/segment-image_6002', methods=['POST'])
def segment_image():
input_data = request.json
print(f"Received input_data: {input_data}")

if not input_data:
    return jsonify({"error": "Missing required parameter"}), 400

image_url = input_data.get("pic_need_judge")
if not image_url:
    return jsonify({"error": "Missing pic_need_judge parameter"}), 400

# 调用本地深度估计服务
api_url = "http://localhost:6000/api/v1/depth-estimation"
data = {"image_path": image_url}

try:
    response = requests.post(api_url, json=data)
except requests.exceptions.RequestException as e:
    return jsonify({"error": f"Request failed: {str(e)}"}), 500

if response.status_code == 200:
    result_data = response.json()
    message = result_data.get("result_text", "No message returned")

    # 确保 message 是字符串
    if isinstance(message, list):
        message = " ".join(map(str, message))  # 如果是列表，拼接成字符串
    elif not isinstance(message, str):
        message = str(message)  # 如果不是字符串，强制转换

    processed_image_path = result_data.get("image_path")

    if not processed_image_path or not os.path.exists(processed_image_path):
        return jsonify({"error": f"Invalid or missing processed image path: {processed_image_path}"}), 400

    # 构造多部分响应
    def generate_multipart_response():
        # 文本部分
        yield f'--{BOUNDARY}\r\n'
        yield 'Content-Disposition: form-data; name="text"\r\n'
        yield 'Content-Type: text/plain; charset=UTF-8\r\n'
        yield '\r\n'
        yield message + '\r\n'  # message 已确保是字符串

        # 图片部分
        with open(processed_image_path, 'rb') as img_file:
            filename = os.path.basename(processed_image_path)  # 确保这是一个字符串
            content_type = mimetypes.guess_type(processed_image_path)[0]  # 获取文件的 MIME 类型
            if not content_type:
                content_type = 'application/octet-stream'  # 如果无法猜测 MIME 类型，则使用默认值

            yield f'--{BOUNDARY}\r\n'
            yield f'Content-Disposition: form-data; name="file"; filename="{filename}"\r\n'
            yield f'Content-Type: {content_type}\r\n'
            yield '\r\n'
            yield img_file.read()
            yield b'\r\n'

        # 结束边界
        yield f'--{BOUNDARY}--\r\n'.encode('utf-8')

    # 使用正确的边界值来创建响应
    headers = {
        'Content-Type': f'multipart/form-data; boundary={BOUNDARY}'
    }
    return Response(generate_multipart_response(), headers=headers), 200
else:
    error_message = response.text
    print(f"Error response from depth estimation service: {error_message}")
    return jsonify(
        {"error": f"Request failed with status {response.status_code}: {error_message}"}), response.status_code

if name == "main":
app.run(host='0.0.0.0', port=6002, debug=True)

the output still be classified into text @crazywoola

[JimHeo]

Same issue here in v0.15.3.

For a single file, the typeFile is displayed correctly, but for multiple files, the type String is exposed instead of Array[File], so can't associate the correct variable.

I tried to test custom tools using FastAPI below:

@app.post("/upload-image/")
async def upload_image(file: UploadFile = File(...)):
    try:
        if not file.content_type.startswith("image/"):
            return {"success": False}
        
        content = await file.read()
        
        return {"success": True}
    except Exception as e:
        return {"success": False}

@app.post("/upload-images/")
async def upload_images(files: List[UploadFile] = File(...)):
    if not files:
        raise HTTPException(status_code=400, detail="No files uploaded.")
        
    for file in files:
        if not file.content_type.startswith("image/"):
            return {"success": False}
        await file.read()
        
    return {"success": True}

And part of the schema

"components": {
    "schemas": {
      "Body_upload_image_upload_image__post": {
        "properties": {
          "file": {
            "type": "string",
            "format": "binary",
            "title": "File"
          }
        },
        "type": "object",
        "required": [
          "file"
        ],
        "title": "Body_upload_image_upload_image__post"
      },
      "Body_upload_images_upload_images__post": {
        "properties": {
          "files": {
            "items": {
              "type": "string",
              "format": "binary"
            },
            "type": "array",
            "title": "Files"
          }
        },
        "type": "object",
        "required": [
          "files"
        ],
        "title": "Body_upload_images_upload_images__post"
      },
      "HTTPValidationError": {
        "properties": {
          "detail": {
            "items": {
              "$ref": "#/components/schemas/ValidationError"
            },
            "type": "array",
            "title": "Detail"
          }
        },
        "type": "object",
        "title": "HTTPValidationError"
      },
      "ValidationError": {
        "properties": {
          "loc": {
            "items": {
              "anyOf": [
                {
                  "type": "string"
                },
                {
                  "type": "integer"
                }
              ]
            },
            "type": "array",
            "title": "Location"
          },
          "msg": {
            "type": "string",
            "title": "Message"
          },
          "type": {
            "type": "string",
            "title": "Error Type"
          }
        },
        "type": "object",
        "required": [
          "loc",
          "msg",
          "type"
        ],
        "title": "ValidationError"
      }
    }
  },