Multi Control Net

FluxPipeline provides support for multiple image generation pipelines with and without controlnets, including text-to-image (txt2img), image-to-image (img2img), and inpainting.

Ensure that a volume mount is added to the deployment, as all images generated are dumped inside /data/outputs directory in the container.

Model Optimization Configuration

Optimization Settings

For optimization, under the optimization config, use:

"optimisations": {
    "attention_caching": {
      "type": "auto",
      "enabled": true,
      "extra_params": {
        "threshold": 0.1
      }
    }
}

Higher threshold values result in greater speed gains but may degrade image generation accuracy.
We recommend a threshold of 0.1, which can provide up to a 40% speed improvement during inference while maintaining reasonable quality.

Pipeline Settings

For optimization, under the optimization config, use:

Multiple ControlNet models can be added under the controlnets section.
Each ControlNet model requires a name, source, and authentication details if needed.

{
  "type": "flux",
  "loras": [],
  "lora_repo": {
    "path": "",
    "type": "",
    "secret": {
      "type": ""
    },
    "ownership": ""
  },
  "pipelines": [
    "txt2img"
  ],
  "controlnets": [
    {
      "name": "canny",
      "source": {
        "path": "InstantX/FLUX.1-dev-Controlnet-Canny",
        "type": "hf",
        "secret": {
          "type": "hf",
          "token": ""
        }
      }
    },
    {
      "name": "depth",
      "source": {
        "path": "InstantX/FLUX.1-dev-Controlnet-Depth",
        "type": "hf",
        "secret": {
          "type": "hf",
          "token": ""
        }
      }
    }
  ],
  "model_choice": {
    "flux_type": "flux"
  },
  "custom_pipeline_config": [],
  "custom_pipeline_resources": ""
}

Here are some key pointers for understanding and structuring controlnet requests:

Understanding ControlNet Parameters

ControlNet Name Convention:

The parameters follow a structured pattern:

<controlnet_name>_<parameter_name><controlnet-name>_control_image
<controlnet-name>_weightage

Example for Canny:

"canny_control_image" - The input image processed with the **Canny edge detection** model.
"canny_weightage" - Defines the influence of the **Canny edge map** on the final image generation.

Example for Depth:

"depth_control_image" → The input image processed with the `Depth estimation` model.
"depth_weightage" → Determines how strongly the depth control image impacts the generation.

Extensibility for Multiple ControlNets:

This pattern allows easy extension to additional ControlNet models in a structured way.
If you add a new ControlNet (e.g., OpenPose), you’d include:

"openpose_control_image": "URL_to_openpose_image",
"openpose_weightage": 0.5

How Weightage Works::

Each weightage parameter (canny_weightage, depth_weightage, etc.) determines the degree of influence that specific ControlNet has on the final image.
Higher values make the model adhere more strictly to the control image, potentially sacrificing flexibility.
Lower values allow more artistic freedom but reduce adherence to structured inputs.

Combining Multiple ControlNets:

You can combine multiple ControlNets in a single request to layer different structural constraints.
In this example:
- Canny edge detection helps maintain sharp edges in the image.
- Depth estimation preserves 3D structural information.
- By adjusting the weightages, you can balance between these two influences.

Generalized Pattern for Other ControlNets:

"<controlnet-name>_control_image": "<URL_to_controlnet_input>",
"<controlnet-name>_weightage": <float_value>

Example with Pose and Normal Map:

"pose_control_image": "URL_to_pose_estimation_image",
"pose_weightage": 0.5,
"normal_control_image": "URL_to_normal_map_image",
"normal_weightage": 0.3

Supported Pipelines

txt2img - Generates an image from text input.
txt2img_controlnet - Generates an image from text input with controlnet support.
img2img - Generates an image based on an input image and a given prompt.
img2img_controlnet - Generates an image based on an input image and a given prompt with controlnet support.
inpaint - Modifies specific regions of an image based on a mask and a given prompt.
inpaint_controlnet - Modifies specific regions of an image based on a mask and a given prompt with controlnet support.

Example Requests

txt2img

{
    "prompt": "A girl in city, 25 years old, cool, futuristic <lora:multimodalart/plstps-local-feature:0.3> <lora:XLabs-AI/flux-RealismLora:0.3>",
    "negative_prompt": "canvas frame, (high contrast:1.2), (over saturated:1.2), (glossy:1.1), cartoon, 3d, ((disfigured)), ((bad art)), ((b&w)), blurry, ((bad anatomy)), (((bad proportions))), ((extra limbs)), cloned face, (((disfigured))), extra limbs, (bad anatomy), gross proportions, (malformed limbs), ((missing arms)), ((missing legs)), (((extra arms))), (((extra legs))), mutated hands, (fused fingers), (too many fingers), (((long neck))), Photoshop, video game, ugly, tiling, poorly drawn hands, 3d render",
    "height": 1024,
    "width": 1024,
    "num_images_per_prompt": 4,
    "num_inference_steps": 20,
    "seed": 2064977189,
    "guidance_scale": 4.5,
    "strength": 0.8,
    "scheduler": "EULER-A",
    "model_type": "txt2img"
}

txt2img_controlnet

{
    "prompt": "A girl in city, 25 years old, cool, futuristic <lora:multimodalart/plstps-local-feature:0.3> <lora:XLabs-AI/flux-RealismLora:0.3>",
    "negative_prompt": "canvas frame, (high contrast:1.2), (over saturated:1.2), (glossy:1.1), cartoon, 3d, ((disfigured)), ((bad art)), ((b&w)), blurry, ((bad anatomy)), (((bad proportions))), ((extra limbs)), cloned face, (((disfigured))), extra limbs, (bad anatomy), gross proportions, (malformed limbs), ((missing arms)), ((missing legs)), (((extra arms))), (((extra legs))), mutated hands, (fused fingers), (too many fingers), (((long neck))), Photoshop, video game, ugly, tiling, poorly drawn hands, 3d render",
    "height": 1024,
    "width": 1024,
    "num_images_per_prompt": 4,
    "num_inference_steps": 20,
    "seed": 2064977189,
    "guidance_scale": 4.5,
    "canny_control_image": "https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Canny/resolve/main/canny.jpg",
    "canny_weightage": 0.4,
    "depth_control_image": "https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Depth/resolve/main/depth.jpg",
    "depth_weightage": 0.4,
    "strength": 0.8,
    "scheduler": "EULER-A",
    "model_type": "txt2img_controlnet"
}

img2img

{
    "prompt": "cat wizard, gandalf, lord of the rings, detailed, fantasy, cute, adorable, Pixar, Disney, 8k <lora:multimodalart/plstps-local-feature:0.3> <lora:XLabs-AI/flux-RealismLora:0.3>",
    "negative_prompt": "canvas frame, (high contrast:1.2), (over saturated:1.2), (glossy:1.1), cartoon, 3d, ((disfigured)), ((bad art)), ((b&w)), blurry, ((bad anatomy)), (((bad proportions))), ((extra limbs)), cloned face, (((disfigured))), extra limbs, (bad anatomy), gross proportions, (malformed limbs), ((missing arms)), ((missing legs)), (((extra arms))), (((extra legs))), mutated hands, (fused fingers), (too many fingers), (((long neck))), Photoshop, video game, ugly, tiling, poorly drawn hands, 3d render",
    "height": 1024,
    "width": 1024,
    "num_images_per_prompt": 4,
    "num_inference_steps": 20,
    "image": "https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg",
    "seed": 89395930,
    "guidance_scale": 7.0,
    "strength": 0.5,
    "scheduler": "EULER-A",
    "model_type": "img2img"
}

img2img_controlnet

{
    "prompt": "cat wizard, gandalf, lord of the rings, detailed, fantasy, cute, adorable, Pixar, Disney, 8k <lora:multimodalart/plstps-local-feature:0.3> <lora:XLabs-AI/flux-RealismLora:0.3>",
    "negative_prompt": "canvas frame, (high contrast:1.2), (over saturated:1.2), (glossy:1.1), cartoon, 3d, ((disfigured)), ((bad art)), ((b&w)), blurry, ((bad anatomy)), (((bad proportions))), ((extra limbs)), cloned face, (((disfigured))), extra limbs, (bad anatomy), gross proportions, (malformed limbs), ((missing arms)), ((missing legs)), (((extra arms))), (((extra legs))), mutated hands, (fused fingers), (too many fingers), (((long neck))), Photoshop, video game, ugly, tiling, poorly drawn hands, 3d render",
    "height": 1024,
    "width": 1024,
    "num_images_per_prompt": 4,
    "num_inference_steps": 20,
    "image": "https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg",
    "canny_control_image": "https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Canny/resolve/main/canny.jpg",
    "canny_weightage": 0.4,
    "depth_control_image": "https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Depth/resolve/main/depth.jpg",
    "depth_weightage": 0.4,
    "seed": 89395930,
    "guidance_scale": 7.0,
    "strength": 0.5,
    "scheduler": "EULER-A",
    "model_type": "img2img_controlnet"
}

inpaint

{
    "prompt": "Face of a yellow cat, high resolution, sitting on a park bench <lora:multimodalart/plstps-local-feature:0.3> <lora:XLabs-AI/flux-RealismLora:0.3>",
    "height": 1024,
    "width": 1024,
    "num_images_per_prompt": 4,
    "num_inference_steps": 20,
    "image": "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png",
    "mask_image": "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png",
    "seed": 89395930,
    "guidance_scale": 7.0,
    "strength": 0.5,
    "scheduler": "EULER-A",
    "clip_skip": 0,
    "use_foocus": true,
    "model_type": "inpaint"
}

inpaint_controlnet

{
    "prompt": "Face of a yellow cat, high resolution, sitting on a park bench <lora:multimodalart/plstps-local-feature:0.3> <lora:XLabs-AI/flux-RealismLora:0.3>",
    "height": 1024,
    "width": 1024,
    "num_images_per_prompt": 4,
    "num_inference_steps": 20,
    "image": "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png",
    "mask_image": "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png",
    "canny_control_image": "https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Canny/resolve/main/canny.jpg",
    "canny_weightage": 0.4,
    "depth_control_image": "https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Depth/resolve/main/depth.jpg",
    "depth_weightage": 0.4,
    "seed": 89395930,
    "guidance_scale": 7.0,
    "strength": 0.5,
    "scheduler": "EULER-A",
    "clip_skip": 0,
    "use_foocus": true,
    "model_type": "inpaint_controlnet"
}

Example Response

{
    "response_id": "afbc439946a44d98bb8062c8b36ec16d",
    "inference_time_taken": 6.336474418640137,
    "lora_time": 1.8092551231384277,
    "total_time_taken": 7.176232099533081,
    "request_id": "6b060ab415d84117b7b6403d622414f5",
    "error": null
}

Key Notes

Ensure volume mounting in deployment for image storage.
ControlNet models are not loaded by default.
Supports multiple pipelines for text-to-image, image-to-image, and inpainting.

Integrations

Flux Compilation

LLM Compilation

Cluster Management

Deployment Guides

Model Optimization Configuration

Optimization Settings

Pipeline Settings

Understanding ControlNet Parameters

ControlNet Name Convention:

Extensibility for Multiple ControlNets:

How Weightage Works::

Combining Multiple ControlNets:

Generalized Pattern for Other ControlNets:

Supported Pipelines

Example Requests

txt2img

txt2img_controlnet

img2img

img2img_controlnet

inpaint

inpaint_controlnet

Example Response

Key Notes

​Model Optimization Configuration

​Optimization Settings

​Pipeline Settings

​Understanding ControlNet Parameters

​ControlNet Name Convention:

​Extensibility for Multiple ControlNets:

​How Weightage Works::

​Combining Multiple ControlNets:

​Generalized Pattern for Other ControlNets:

​Supported Pipelines

​Example Requests

​txt2img

​txt2img_controlnet

​img2img

​img2img_controlnet

​inpaint

​inpaint_controlnet

​Example Response

​Key Notes

Model Optimization Configuration

Optimization Settings

Pipeline Settings

Understanding ControlNet Parameters

ControlNet Name Convention:

Extensibility for Multiple ControlNets:

How Weightage Works::

Combining Multiple ControlNets:

Generalized Pattern for Other ControlNets:

Supported Pipelines

Example Requests

txt2img

txt2img_controlnet

img2img

img2img_controlnet

inpaint

inpaint_controlnet

Example Response

Key Notes