Post Syndicated from Wang Rui original https://aws.amazon.com/blogs/architecture/how-to-build-custom-nodes-workflow-with-comfyui-on-amazon-eks/
ComfyUI is an open-source node-based workflow solution for Stable Diffusion and increasingly being used by many creators. We previously published a blog and solution about how to deploy ComfyUI on AWS.
Typically, ComfyUI users use various custom nodes, which extend the capabilities of ComfyUI, to build their own workflows, often using ComfyUI-Manager to conveniently install and manage their custom nodes.
Following our blog post, we received numerous customer requests to integrate ComfyUI custom nodes into our solution. This post will guide you through the process of integrating custom nodes within ComfyUI-on-EKS.
Architecture overview
Figure 1. Architecture diagram showing the ComfyUI integration with Amazon EKS
To integrate custom nodes within ComfyUI-on-EKS solution, we need to prepare custom nodes codes and environment, as well as needed models:
- Code and Environment: Custom node code is placed in
$HOME/ComfyUI/custom_nodes, and the environment is prepared by running pip install -r on all requirements.txt files in the custom node directories (any dependency conflicts between custom nodes need to be handled separately). Additionally, any system packages required by the custom nodes also should be installed. All these operations are performed through the Dockerfile, building an image containing the required custom nodes. - Models: Models used by custom nodes are placed in different directories under
s3://comfyui-models-{account_id}-{region}. This triggers a Lambda function to send commands to all GPU nodes to synchronize the newly uploaded models to local instance store.
We’ll use the Stable Video Diffusion (SVD) – Image to video generation with high FPS workflow as an example to illustrate how to integrate custom nodes (you can also use your own workflow).
Build docker image
When loading this workflow, it will display the missing custom nodes. Next, we will build the missing custom nodes into the docker image.
Figure 2. Error message showing the missing node types
There are two ways to build the image:
- Build from GitHub: In the Dockerfile, download the code for each custom node and set up the environment and dependencies separately.
- Build locally: Copy all the custom nodes from your local Dev environment into the image and set up the environment and dependencies.
Before building the image, please switch to the corresponding branch
git clone https://github.com/aws-samples/comfyui-on-eks ~/comfyui-on-eks
cd ~/comfyui-on-eks && git checkout custom_nodes_demoBuild from GitHub
Install custom nodes and dependencies with RUN command in the Dockerfile. You’ll need to find the GitHub URLs for all missing custom nodes.
...
RUN apt-get update && apt-get install -y \
git \
python3.10 \
python3-pip \
# needed by custom node ComfyUI-VideoHelperSuite
libsm6 \
libgl1 \
libglib2.0-0
...
# Custom nodes demo of https://comfyworkflows.com/workflows/bf3b455d-ba13-4063-9ab7-ff1de0c9fa75
## custom node ComfyUI-Stable-Video-Diffusion
RUN cd /app/ComfyUI/custom_nodes && git clone https://github.com/thecooltechguy/ComfyUI-Stable-Video-Diffusion.git && cd ComfyUI-Stable-Video-Diffusion/ && python3 install.py
## custom node ComfyUI-VideoHelperSuite
RUN cd /app/ComfyUI/custom_nodes && git clone https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite.git && pip3 install -r ComfyUI-VideoHelperSuite/requirements.txt
## custom node ComfyUI-Frame-Interpolation
RUN cd /app/ComfyUI/custom_nodes && git clone https://github.com/Fannovel16/ComfyUI-Frame-Interpolation.git && cd ComfyUI-Frame-Interpolation/ && python3 install.py
...Refer to comfyui-on-eks/comfyui_image/Dockerfile.github for the complete Dockerfile.
Run following command to build and push Docker image
region="us-west-2" # Modify the region to your current region.
cd ~/comfyui-on-eks/comfyui_image/ && bash build_and_push.sh $region Dockerfile.githubBuilding from GitHub provides a clear understanding of the installation method, version, and environmental dependencies for each custom node, providing better control over the entire ComfyUI environment.
However, when there are too many custom nodes, installation and management can be time-consuming, and you need to find the URL for each custom node yourself (on the other hand, this can also be seen as a pro, as it makes you more familiar with the entire ComfyUI environment).
Build locally
Often, we use ComfyUI-Manager to install missing custom nodes. ComfyUI-Manager hides the installation details, and we cannot clearly know which custom nodes have been installed. In this case, we can build the image by COPY the entire ComfyUI directory (except the input, output, models, and other directories) into the Dockerfile.
The prerequisite for building the image locally is that you already have a working ComfyUI environment with custom nodes. In the same directory as ComfyUI, create a .dockerignore file and add the following content to ignore these directories when building the Docker image
ComfyUI/models
ComfyUI/input
ComfyUI/output
ComfyUI/custom_nodes/ComfyUI-ManagerCopy the two files comfyui-on-eks/comfyui_image/Dockerfile.local and comfyui-on-eks/comfyui_image/build_and_push.sh to the same directory as your local ComfyUI, like this:
ubuntu@comfyui:~$ ll
-rwxrwxr-x 1 ubuntu ubuntu 792 Jul 16 10:27 build_and_push.sh*
drwxrwxr-x 19 ubuntu ubuntu 4096 Jul 15 08:10 ComfyUI/
-rw-rw-r-- 1 ubuntu ubuntu 784 Jul 16 10:41 Dockerfile.local
-rw-rw-r-- 1 ubuntu ubuntu 81 Jul 16 10:45 .dockerignore
...The Dockerfile.local builds the image by COPY the directory
...
# Python Evn
RUN pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu121
COPY ComfyUI /app/ComfyUI
RUN pip3 install -r /app/ComfyUI/requirements.txt
# Custom Nodes Env, may encounter some conflicts
RUN find /app/ComfyUI/custom_nodes -maxdepth 2 -name "requirements.txt"|xargs -I {} pip install -r {}
...Refer to comfyui-on-eks/comfyui_image/Dockerfile.local for the complete Dockerfile.
Run the following command to build and upload the Docker image
region="us-west-2" # Modify the region to your current region.
bash build_and_push.sh $region Dockerfile.localWith this method, you can easily and quickly build your local Dev environment into an image for deployment, without paying attention to the installation, version, and dependency details of custom nodes when there are many of them.
However, not paying attention to the deployment environment of custom nodes may cause conflicts or missing dependencies, which need to be manually tested and resolved.
Upload models
Upload all the models needed for the workflow to the s3://comfyui-models-{account_id}-{region} corresponding directory using your preferred method. The GPU nodes will automatically sync from Amazon S3 (triggered by Lambda). If the models are large and numerous, you might need to wait. You can log into the GPU nodes using the aws ssm start-session --target ${instance_id} command and use the ps command to check the progress of the aws s3 sync process.
To set up this demo, you need to download the following models to s3://comfyui-models-{account_id}-{region}/svd/:
Test the Docker image locally (optional)
Since there are many types of custom nodes with different dependencies and versions, the runtime environment is quite complex. We recommend testing the Docker image locally after building it to ensure it runs correctly.
Refer to the code in comfyui-on-eks/comfyui_image/test_docker_image_locally.sh. Prepare the models and input directories (assuming the models and input images are stored in /home/ubuntu/ComfyUI/models and /home/ubuntu/ComfyUI/input respectively), and run the script to test the Docker image:
bash comfyui-on-eks/comfyui_image/test_docker_image_locally.shRolling update K8S pods
Use your preferred method to perform a rolling update of the image for the online K8S pods, and then test the service.
Note, to run this demo, you need to:
- use g5.2xlarge GPU node
- set lower num_frames in Load Stable Video Diffusion Model (for example to 6)
- set lower decoding_t in Stable Video Diffusion Decoder node (for example to 1)
Figure 3. Screenshot showing the rolling update demo
Conclusion
Custom nodes empower creators to unleash the full potential of ComfyUI by seamlessly integrating a wide range of capabilities into their own workflows.
This article demonstrate how to build custom nodes into ComfyUI-on-EKS solution, you can build your own ComfyUI CI/CD pipeline following the instructions.
