PytorchEngine multi-node support v2

[grimoire]

1. build ray cluster:

# on driver node
ray start --head --port 6379

# on other node
ray start --address=$DRIVER_ADDR:6379

2. run pipeline on driver, model should be placed on same location.

from lmdeploy import pipeline, PytorchEngineConfig
with pipeline(model_path, backend_config=PytorchEngineConfig(tp=16)) as pipe:
    ....

Users can choice the distributed_executor_backend when using a single node.

Note

It is encouraged to build the cluster in docker container with --network host like vLLM to ensure each node has the same environment.
and automatic model downloading is not supported for now. Models should be pre-downloaded on each node.

[lvhan028]

Fantastic job!
@RunningLeon Please help check if it works for dsv3.

[lvhan028]

cc @jinminxi104

[lvhan028]

Please prepare a guide about the multi-node deployment, including offline pipeline and online serving

[grimoire]

Please prepare a guide about the multi-node deployment, including offline pipeline and online serving

https://github.com/InternLM/lmdeploy/blob/f1a8a08064087ee360c42e264f69d76cde6582d2/docs/en/advance/pytorch_multinodes.md

[grimoire]

Automatic model downloading has not been supported on the ray backend. The user should put the model in the same location on each node.

[RunningLeon]

Fantastic job! @RunningLeon Please help check if it works for dsv3.

@lvhan028 tested Ok with deepseek-ai/DeepSeek-R1 + tp16. The conversation output from serving is normal after this commit 95b2249

[lvhan028]

May merge the latest main so that I can request a full test