# Flux Dev API Source: https://docs.simplismart.ai/api-reference/inference/flux-dev openapi-specs/playground/flux-dev.yaml POST /model/infer/flux Generate high-quality images using the Flux model # Gemma 3 27B API Source: https://docs.simplismart.ai/api-reference/inference/gemma3-27B openapi-specs/playground/gemma3-27b.yaml POST /chat/completions Create a chat completion for given messages with streaming support # Gemma 3 4B API Source: https://docs.simplismart.ai/api-reference/inference/gemma3-4B openapi-specs/playground/gemma3-4B.yaml POST /chat/completions Create a chat completion for given messages with streaming support # Llama 3.3 70B API Source: https://docs.simplismart.ai/api-reference/inference/llama-3.3-70b-instruct openapi-specs/playground/llama-3p3-70b-instruct.yaml POST /chat/completions Create a chat completion for given messages with streaming support # Llama 4 Maverick 17B API Source: https://docs.simplismart.ai/api-reference/inference/llama-4-maverick-17b openapi-specs/playground/llama-4-maverick-17b.yaml POST /chat/completions Create a chat completion for given messages with streaming support # Llama 3.1 8B API Source: https://docs.simplismart.ai/api-reference/inference/llama3.1-8B openapi-specs/playground/llama-8b.yaml POST /chat/completions Create a chat completion for given messages with streaming support # LLM Metric API Source: https://docs.simplismart.ai/api-reference/inference/llm-metric openapi-specs/playground/llm-metrics.yaml GET /get/metrics/{request_id} Retrieve detailed metrics about token counts, processing time, and performance statistics # Qwen 14B API Source: https://docs.simplismart.ai/api-reference/inference/qwen-14b openapi-specs/playground/qwen-14b.yaml POST /chat/completions Create a chat completion for given messages with streaming support # Qwen 32B API Source: https://docs.simplismart.ai/api-reference/inference/qwen-32b openapi-specs/playground/qwen-32b.yaml POST /chat/completions Create a chat completion for given messages with streaming support # Whisper V2 API Source: https://docs.simplismart.ai/api-reference/inference/whisper-v2 openapi-specs/playground/whisper-v2.yaml POST /model/v2/infer/whisper Process audio files for transcription or translation with advanced options # Whisper V3 API Source: https://docs.simplismart.ai/api-reference/inference/whisper-v3 openapi-specs/playground/whisper-v3.yaml POST /model/infer/whisper Process audio files for transcription or translation with enhanced language support. Supports multiple audio formats and provides detailed word-level timestamps and speaker diarization. # Simplismart APIs Source: https://docs.simplismart.ai/api-reference/introduction Get started with the Simplismart API, including authentication, endpoints, and workflows for model training and inference. Simplismart offers two powerful sets of APIs to streamline your workflow: * **Inference APIs**: Effortlessly leverage pre-trained GenAI models for a wide range of tasks, including ASR transcription, image generation, and LLM-based applications. These APIs allow you to directly run inference and get task-specific outputs with minimal configuration. * **Training APIs**: Fine-tune and customize GenAI models to fit your unique requirements. With these APIs, you can train models on top of existing base models, optimizing them for your specific tasks, data, and desired outputs. ## API Key Generation Navigate to the **Settings** page, then select the **API Key** section and generate a new API key. ## Authentication All API endpoints are authenticated using Bearer tokens and picked up from the specification file. ```json theme={null} "auth": { "method": "bearer", "name": "Authorization" }, ``` # Retrieve Flux Job Details By Request ID and Job ID Source: https://docs.simplismart.ai/api-reference/training/flux/retrieve-flux-job-details-by-request-id-and-job-id openapi-specs/playground/flux-training.yaml GET /api/flux/training_job/get/ Fetch metadata and configuration details of a Flux training job by job_id the and org_id. # Retrieve List of Flux Jobs By Org ID Source: https://docs.simplismart.ai/api-reference/training/flux/retrieve-list-of-flux-jobs-by-org-id openapi-specs/playground/flux-training.yaml GET /api/flux/training_job/list/ Overview of Flux training capabilities and endpoints. # Start a new Flux Training Job Source: https://docs.simplismart.ai/api-reference/training/flux/start-a-new-flux-training-job openapi-specs/playground/flux-training.yaml POST /api/flux/train/ Submit a new Flux training job with the specified configuration, training data, and metadata. # Retrieve List of LLM/VLM Jobs By Org ID Source: https://docs.simplismart.ai/api-reference/training/llm/get-job openapi-specs/playground/llm-training.yaml GET /job/get/ Fetches a list of all LLM or VLM training jobs associated with the specified organization. # Retrieve LLM/VLM Job Details By Request ID Source: https://docs.simplismart.ai/api-reference/training/llm/get-job-by-req openapi-specs/playground/llm-training.yaml GET /job/get/ Fetch metadata and configuration details of a training job identified by the provided request_id and org_id. # Start a new LLM/VLM training job Source: https://docs.simplismart.ai/api-reference/training/llm/start-a-new-llm-vlm-training-job openapi-specs/playground/llm-training.yaml POST /job/ Submit a new training job with the specified configuration, training data, and metadata. # Advanced Benchmarking Source: https://docs.simplismart.ai/benchmarking/advanced-benchmarking Perform in-depth, customizable evaluations of LLM outputs using custom datasets and a range of evaluator types including programmatic, human, and AI-based. # Creating an Advanced Evaluation LLM Benchmark ## Start a New Benchmark 1. Go to **Benchmarking → Create**. 2. Choose **Advanced** as the benchmark type. Choose Advanced Benchmark 3. Select **LLM** as the model type. Select LLM Model Type ## General Information * **Benchmark Name** — Give the run a clear, unique name. * **Select Deployments** — Pick **one** deployment to benchmark. > Only one LLM deployment can be chosen at once. Deployment Selection ## Dataset Configuration * **Presigned Dataset Link** — Provide a presigned URL path to your dataset file. * Only **JSON** files are supported. * If the dataset has more than **1000 rows**, only the **first 1000 datapoints** will be used. * You can use the provided \[[sample dataset format](https://simplismart-public-assets.s3.ap-south-1.amazonaws.com/benchmarking/eval.json)] as a reference. ## LLM Configuration * **Max Tokens** — Defines the maximum number of tokens the model can generate in a response.\ Example: `1024` means the response will be capped at 1024 tokens. > A higher value allows longer outputs but also increases resource usage. * **Temperature** — Controls the randomness/creativity of the model’s output.\ Range: `0` to `1` * Lower values (e.g., `0.2`) → More deterministic and focused responses * Higher values (e.g., `0.8`) → More diverse and creative responses * Example: `0.7` balances creativity and consistency LLM Configuration ## Evaluation Configuration We provide a collection of **pre-built evaluators** that you can use immediately for your AI evaluation needs. Choose **up to 3 evaluators** for assessing model outputs. Evaluators can be selected from the following categories: * **Programmatic**\ Uses custom JavaScript or Python code to programmatically evaluate quality.\ Useful for deterministic checks (e.g., regex validation, schema conformance, rule-based scoring). * **Human**\ Relies on human reviewers to assess outputs based on subjective or nuanced criteria like: * Readability * Tone * Clarity * Relevance * Factual correctness * **Statistical**\ Uses traditional ML metrics for text comparison. Helpful for benchmarking against reference outputs. * **AI-based**\ Uses LLMs as judges with carefully designed prompts.\ Provides **automated, scalable evaluation** with high alignment to human judgment. # Introduction Source: https://docs.simplismart.ai/benchmarking/introduction Discover Simplismart’s benchmarking suite to evaluate model performance & quality Simplismart’s benchmarking suite lets you evaluate any deployment for **Performance** (speed & throughput), **Quality** (accuracy & relevance), or **Advanced evaluation** (predefined evaluator-based assessment). Follow these steps to create, configure, and run benchmarks end-to-end. Measures **Throughput**, **TTFT** (time‑to‑first‑token), and **TPOT** (time‑per‑output‑token). ## Evaluates model responses for **accuracy, relevance, and output quality** on selected datasets. Measures quality using a suite of advanced predefined evaluators for deeper assessment. ## Model Types | Model | Type | Status | | ----------- | -------------------- | ------------- | | **LLM** | Text generation | ✅ Available | | **Whisper** | Speech-to-Text (STT) | ⏳ Coming soon | | **Flux** | Image generation | ⏳ Coming soon | ## **Prerequisites** You have at least one **Deployment** in an active **Cluster**. # Performance Benchmarking Source: https://docs.simplismart.ai/benchmarking/performance-benchmarking Set up and run load tests to evaluate LLM deployment speed, throughput, and scalability using chat datasets. # Creating a Performance Benchmark ## Start a New Benchmark 1. Go to **Benchmarking → Create**. 2. Choose **Performance** as the benchmark type. Performance Benchmark Type 3. Select **LLM** as the model type. Select LLM Model Type ## General Information * **Benchmark Name** — Give the run a clear, unique name. * **Select Deployments** — Pick one or more deployments to benchmark. * **Region** — (Shown in Available Regions) choose where the run executes. General Information Setup ## Supported Inputs (Chat Dataset Input) * **Dataset Name** — Choose a dataset (currently we support only *Ultrachat* dataset). * **Num Samples** — Limit how many rows/samples from the dataset to use. * **Sampling Temperature** — Set generation randomness between 0 and 1. * **Maximum tokens to generate** — Sets the upper limit on response length in tokens. * **Whether to stream responses** — Toggle on/off to stream model outputs during the run. ## Execution Configuration * **Users** — Number of virtual users to simulate during the benchmark. * **Duration** — How long the benchmark should run (e.g., 30s). * **Load Type** — Pattern for load generation (e.g., Constant). * **Metrics** — Select the metrics to capture (e.g., *throughput*, *ttft*, *tpot*). Execution Configuration ## Search Space (Hyperparameter Sweep) **Search Space** allows you to run the same benchmark multiple times with different parameter values, so you can compare how deployments behave under varied conditions. Instead of running separate benchmarks manually, you can provide multiple values and the suite will automatically generate runs for each combination. * `inputs[name=messages].preprocess.input_tokens` — Vary the input size (number of tokens in the prompt).\ **Example**: `1000`, `2000`, `3000`. * `inputs[name=max_tokens].value` — Vary the maximum output length.\ **Example**: `100`, `200`, `300`. * `execution.profile.users` — Vary the number of concurrent users.\ **Example**: `1`, `2`, `3`. **Example**:\ If you specify `input_tokens: [1000,2000]` and `users: [1,2]`, the suite will generate 4 runs: * `1000×1`, `1000×2`, `2000×1`, `2000×2`. > Type a value and press `Enter` or `Add` — values appear as tags. Leave blank to skip. ## Run the Benchmark Click **Create Benchmark**. The run is queued and executed in the selected **Region** with the chosen **metrics** and **sweep settings**. # Quality Benchmarking Source: https://docs.simplismart.ai/benchmarking/quality-benchmarking Measure the accuracy and quality of LLM outputs using curated datasets and standardized evaluation metrics. # Creating a Quality Benchmark ## Start a New Benchmark 1. Go to **Benchmarking → Create**. 2. Choose **Quality** as the benchmark type. Choose Quality Benchmark 3. Select **LLM** as the model type. Select LLM Model Type ## General Information * **Evaluation Name** — Name for this evaluation. * **Select Deployments** — Choose one or more deployments to evaluate. ## Dataset Configuration * **Select Datasets** — Pick one or more datasets (e.g., *gsm8k*). Dataset Configuration ## Generation Configuration * **Max Tokens** — Maximum tokens the model can generate per response. * **Temperature** — Controls randomness; lower = more focused, higher = more creative. * **Top P** — Nucleus sampling; limits token choices to the top probability mass (e.g., `0.9` = top 90%). Generation Configuration ## Execution Configuration * **Batch Size** — Requests processed together. * **Evaluation Limit** — Limit number of dataset samples to evaluate (e.g., `10`). ## Run the Evaluation * Click **Create Benchmark** to start. # Compilation Configurations Source: https://docs.simplismart.ai/configurations/compilation-configurations Reference guide for model compilation configurations, including Optimization settings and Quantization options for different hardware environments. ## **Optimization Configuration** ```json theme={null} { "warmups": { "enabled": true, "iterations": 5, "sample_input_data": [] }, "backend": { "name": "auto", "version": "latest", "extra_params": {} }, "optimisations": { "speculative_decoding": { "enabled": false, "type": "auto", "extra_params": {} }, "attention_caching": { "enabled": false, "type": "auto", "extra_params": {} } }, "tensor_parallel_size": 1, "quantization": "float16" } ``` *** ## **Quantization Types** 1. **Float 32 (FP32)** * Full precision. * Highest accuracy. * Maximum memory usage. 2. **Float 16 (FP16)** * Reduced precision. * Minimal accuracy loss. * Recommended for most use cases. * Balances performance and accuracy. 3. **Float 8 (FP8)** * Advanced reduced precision. * **Hardware Limitations** * Not supported on **A100 GPU** architecture. * Only available on **H100 GPUs**. 4. **INT4 Quantization** * Extreme compression. * Substantial memory reduction. * Noticeable accuracy degradation. 5. **AWQ (Activation-aware Weight Quantization)** * Advanced compression technique. * Maintains model performance. * Minimal accuracy loss. *** ## Model Configuration ```json theme={null} { "type": "llm", "loras": [], "lora_repo": { "type": "", "path": "", "ownership": "", "secret": { "type": "" } }, "quantized_model_path": { "type": "", "path": "", "ownership": "", "secret": { "type": "" } } } ``` # LLM Training Configuration Source: https://docs.simplismart.ai/configurations/llm-training-configuration Detailed guide for configuring LLM training parameters, including input/output features, quantization settings, and prompt templating for combining multiple data columns. ## Advanced Configuration ```json theme={null} { "input_features": [ { "name": "question", "type": "text", "preprocessing": { "max_sequence_length": 4096 } } ], "output_features": [ { "name": "answer", "type": "text", "preprocessing": { "max_sequence_length": 4096 } } ], "quantization": { "bits": 4, "llm_int8_threshold": 6, "llm_int8_has_fp16_weight": false, "bnb_4bit_compute_dtype": "float16", "bnb_4bit_use_double_quant": true, "bnb_4bit_quant_type": "nf4" }, "trainer": { "type": "finetune", "learning_rate_scheduler": { "warmup_fraction": 0.01, "decay": "linear" } }, "preprocessing": { "sample_ratio": 1 }, "backend": { "type": "local" } } ``` The training configuration provides a flexible way to define the inputs, outputs, and other advanced settings for your model. * Max Sequence Length (input and output features) : The ‘max\_sequence\_length’ parameter in the config refers to the maximum tokens in the prompt and response respectively. * If Quantisation of the model is not required (for smaller models like 3B, 2B etc) we can remove the quantization key from the advanced configuration, so that all the parameters are used in full precision which yields better accuracy (however training time would increase). ## **Quantization Config** ```json theme={null} { "bits": 4, "llm_int8_threshold": 6, "llm_int8_has_fp16_weight": false, "bnb_4bit_compute_dtype": "float16", "bnb_4bit_use_double_quant": true, "bnb_4bit_quant_type": "nf4" } ``` * Ensure that the names in the input and output features match exactly with the corresponding columns in the dataset. (e.g., "question" for input and "answer" for output) ## Prompt Template In cases where you want to form a prompt using multiple columns you can use a prompt template to combine them. **config with Prompt Template:** ```json theme={null} { "prompt": { "template": "{system_prompt}\n\n {question}\n\n Response:\n" }, "input_features": [ { "name": "prompt", "type": "text", "preprocessing": { "max_sequence_length": 4096 } } ], "output_features": [ { "name": "response", "type": "text", "preprocessing": { "max_sequence_length": 4096 } } ], "quantization": { "bits": 4, "llm_int8_threshold": 6, "llm_int8_has_fp16_weight": false, "bnb_4bit_compute_dtype": "float16", "bnb_4bit_use_double_quant": true, "bnb_4bit_quant_type": "nf4" }, "trainer": { "type": "finetune", "learning_rate_scheduler": { "warmup_fraction": 0.01, "decay": "linear" } }, "preprocessing": { "sample_ratio": 1 }, "backend": { "type": "local" } } ``` # Image Generation Models Source: https://docs.simplismart.ai/get-started/playground/image-generation-models Generate fast, high-quality AI images using an intuitive playground The image generation playground is focused on image generation and customization. You can: * **Create Images from Text**: Generate images based on text descriptions or prompts. * **Set Generation Parameters**: Configure steps, image size, seed, and other parameters to refine the image generation process. * **Apply Custom Models**: Optionally use LoRA weights from the Simplismart Training Suite to customize outputs. * **Review and Download Results**: View the generated images, assess their quality, and export them as needed. ### Flux ## Settings Explained **prompt**: The text description guiding the image generation, crucial for defining the desired outcome. **height & width**: Dimensions of the generated image, available in various aspect ratios:
| **Image Dimensions** | **Aspect Ratio** | **Resolution** | | :------------------: | :--------------: | :------------: | | Square | 1:1 | 1024x1024 | | Portrait | 2:3 | 768x1152 | | Landscape | 4:3 | 1152x768 | | Widescreen | 16:9 | 1024x576 | | Vertical Widescreen | 9:16 | 576x1024 | | Cinematic | 2:1 | 1024x512 | | Vertical Cinematic | 1:2 | 1024x512 | | Portrait | 4:5 | 816x1024 |
**seed**: Initial input for the random generator, ensuring reproducibility of generated images. **num\_inference\_steps**: Number of iterations the model performs during generation; more steps improve quality at the cost of processing time. (recommended: 28) **num\_images\_per\_prompt**: Number of images to be generated for a single prompt. **guidance\_scale**: Controls how closely the model follows the input prompt. Higher values increase adherence to the prompt while lower values allow more creative freedom. (recommended: 3) **lora\_weights** (optional): URL pointing to custom LoRA weights from the Simplismart Training Suite. **lora\_scale** (optional): Determines the impact of LoRA weights on the generation process. (default: 1) *** Access the Flux model API documentation [**here**](/api-reference/inference/flux-dev) for endpoints, parameters, and code examples. # Large Language Models Source: https://docs.simplismart.ai/get-started/playground/large-language-models Generate and evaluate text across multiple powerful LLMs using a fast, interactive interface The LLM (Large Language Model) playground offers a versatile space for text generation and conversational AI. Users can: * **Chat**: Engage in interactive conversations with the model, simulating real-world dialogue. * **Configure Output**: Adjust settings like output length, temperature and top-P to customize the responses generated by the model. * **Experiment with Prompts**: Input different prompts and scenarios to see how the model responds, allowing for creative and practical applications. * **Evaluate Interactions**: Analyze the generated text for coherence, creativity, and relevance to ensure it meets the required standards. Llama_Playround *** ### Settings explained `output tokens:` The maximum length of the generated response, important for controlling the verbosity of the output. `temperature:` Controls randomness in the output; higher values produce more creative results, while lower values yield more deterministic responses. `top-P:` Uses nucleus sampling to choose tokens from the top P cumulative probability mass, balancing creativity and coherence. `stop sequence:` Specific sequences that, when generated, will halt further output. `system prompt:` The initial instruction or context setting the behaviour of the model. # Transcription Models Source: https://docs.simplismart.ai/get-started/playground/transcription-models Get real-time, high-quality speech-to-text output for any audio with fast STT processing, live results, and support for various audio formats. The Whisper playground is designed for audio processing and transcription. You can: * **Upload Audio Files**: Test the model by uploading audio files and receiving transcriptions. * **Set Advanced Parameters**: Configure settings such as initial prompts, the number of speakers, beam size, audio sample rate, and more to fine-tune transcription accuracy. * **Process Audio Real-Time**: Experience real-time audio processing and transcription to evaluate performance in various scenarios. * **Evaluate Results**: Review and analyze transcriptions to ensure they meet the desired accuracy and quality. ### Whisper *** ### **Settings explained** `language`: Language spoken in the audio, specify None to perform language detection. `task`: Determines if the Whisper model should perform translation or transcription. `initial prompt`: Optional starting text prompt for the model, useful for guiding the initial context. e.g. custom vocabularies or proper nouns to make it more likely to predict those word correctly. `best of`: Specifies how many decoding paths to consider and choose the best from, higher values can improve quality. `no of speakers`: The number of speakers in the audio, important for separating dialogues. `diarization`: Assignment of speakers to different parts of the text. `word timestamps`: Indicates if word-level timestamps should be in the output. `without timestamps`: Option to exclude timestamps in the output. `beam size`: Controls the breadth of search in beam search decoding, larger values improve accuracy but increase computation. `length penalty`: A factor that penalizes longer predictions, helps control output length. `batch size`: The number of audio samples processed together in one batch. `patience`: The duration to wait before making a prediction, useful for adjusting responsiveness. `minimum duration on`: Minimum duration of speech to consider it as an active segment. `minimum duration off`: Minimum duration of silence to consider it as a break. `maximum duration`: The longest duration of speech to process in one go, prevents excessive processing time. `maximum speakers`: The maximum number of speakers expected in the audio. `minimum speakers`: The minimum number of speakers expected in the audio. `vad onset`: Sensitivity for detecting the start of speech. `vad offset`: Sensitivity for detecting the end of speech. `pad onset`: Additional padding time added to the start of detected speech. `pad offset`: Additional padding time added to the end of detected speech. *** Access the Whisper model API documentation [here](/api-reference/inference/whisper-v3) for endpoints, parameters, and code examples. Need help with VAD parameter tuning or Whisper troubleshooting? Check our detailed guides on [VAD tuning](/troubleshooting-faq/vad-parameter-tuning) and [Whisper troubleshooting](/troubleshooting-faq/whisper-troubleshooting). *** ### Supported Languages with their Codes ```json theme={null} LANGUAGES = { "en": "english", "zh": "chinese", "de": "german", "es": "spanish", "ru": "russian", "ko": "korean", "fr": "french", "ja": "japanese", "pt": "portuguese", "tr": "turkish", "pl": "polish", "ca": "catalan", "nl": "dutch", "ar": "arabic", "sv": "swedish", "it": "italian", "id": "indonesian", "hi": "hindi", "fi": "finnish", "vi": "vietnamese", "he": "hebrew", "uk": "ukrainian", "el": "greek", "ms": "malay", "cs": "czech", "ro": "romanian", "da": "danish", "hu": "hungarian", "ta": "tamil", "no": "norwegian", "th": "thai", "ur": "urdu", "hr": "croatian", "bg": "bulgarian", "lt": "lithuanian", "la": "latin", "mi": "maori", "ml": "malayalam", "cy": "welsh", "sk": "slovak", "te": "telugu", "fa": "persian", "lv": "latvian", "bn": "bengali", "sr": "serbian", "az": "azerbaijani", "sl": "slovenian", "kn": "kannada", "et": "estonian", "mk": "macedonian", "br": "breton", "eu": "basque", "is": "icelandic", "hy": "armenian", "ne": "nepali", "mn": "mongolian", "bs": "bosnian", "kk": "kazakh", "sq": "albanian", "sw": "swahili", "gl": "galician", "mr": "marathi", "pa": "punjabi", "si": "sinhala", "km": "khmer", "sn": "shona", "yo": "yoruba", "so": "somali", "af": "afrikaans", "oc": "occitan", "ka": "georgian", "be": "belarusian", "tg": "tajik", "sd": "sindhi", "gu": "gujarati", "am": "amharic", "yi": "yiddish", "lo": "lao", "uz": "uzbek", "fo": "faroese", "ht": "haitian creole", "ps": "pashto", "tk": "turkmen", "nn": "nynorsk", "mt": "maltese", "sa": "sanskrit", "lb": "luxembourgish", "my": "myanmar", "bo": "tibetan", "tl": "tagalog", "mg": "malagasy", "as": "assamese", "tt": "tatar", "haw": "hawaiian", "ln": "lingala", "ha": "hausa", "ba": "bashkir", "jw": "javanese", "su": "sundanese", "yue": "cantonese", } ``` # Model Deployment Guide Source: https://docs.simplismart.ai/guides/deployment-guides A guide to deploying models with best practices/ checks for reliable model serving Before deploying any machine learning model, it is critical to perform a series of infrastructure checks to ensure optimal performance and cost-efficiency. Below are the key considerations to evaluate: ## 1. Model Specifications * **Model Size**: Determine the model's parameter size (e.g., 8B, 13B, etc.). * **Precision Format**: Know the floating-point format (e.g., FP16, INT8), as this impacts memory requirements. * **Tensor Parallelism:** Distribute model layers across **multiple GPUs** to handle models too large for single GPU memory. ## 2. GPU Memory Requirements * For large models (e.g., an 8B model using FP16), ensure a minimum of **16 GB GPU memory** to avoid Out-of-Memory (OOM) errors. * In such cases, opt for higher-spec GPUs: * **NVIDIA L4**: 24 GB VRAM * **NVIDIA L40s**: 48 GB VRAM ## 3. GPU vs. CPU RAM Clarification * It's important to distinguish between **CPU RAM** (displayed as system memory) and **GPU VRAM**. * For example, instances like `g4dn.xlarge` and `g4dn.2xlarge` offer: * `g4dn.xlarge`: 4 vCPUs, 16 GB CPU RAM * `g4dn.2xlarge`: 8 vCPUs, 32 GB CPU RAM * **Note**: Across a given instance family, the **GPU VRAM typically remains constant**, even though CPU resources scale up. ## 4. Resource Allocation Best Practices * To ensure system stability and allow room for background processes: * Allocate **only 80%** of the available CPU and RAM to the model or service. * Example: On a `g4dn.2xlarge` (8 vCPUs, 32 GB RAM), limit allocation to: * **6–7 vCPUs** * **\~26 GB RAM** ## 5. Deployment Considerations * Identify the **deployment region** and **preferred instance family**. * Define **scaling ranges and metrics** (e.g., CPU/GPU utilization, request latency) to enable autoscaling effectively. # Dynamic Lora Compilation Source: https://docs.simplismart.ai/guides/dynamic-lora-compilation Use dynamic LoRA compilation on Simplismart to run multiple task-specific LoRAs in a single deployment, improving efficiency and simplifying workflows. ## Introduction ### What's New? This feature enables users to dynamically load multiple **LoRAs (Low-Rank Adaptations)** into a single model deployment. With this enhancement, you can tailor your models to diverse tasks and domains without creating multiple separate deployments. By leveraging multiple LoRAs simultaneously, you can optimize model performance, reduce inference time, and streamline your model management workflows. ## Available Flags & Options | Flag Name | Type | Default | Description | | ------------------- | ------- | ------- | -------------------------------------------------------------------------------------------- | | `loras` | list | `[]` | List of LoRA configurations to load. See **Example Configuration** for the schema. | | `lora_repo` | dict | `null` | Cloud storage path (e.g. S3, GCP) containing multiple LoRAs to load dynamically. | | `load_lora_dynamic` | boolean | `false` | Enables dynamic loading of LoRAs into the base model. `false` will merge all provided LoRAs. | *** ## Example Pipeline Configuration ### Using `loras` list: When specifying LoRAs manually: ```json theme={null} { "type": "llm", "loras": [ { "id": "lora_id_0", "source": { "path": "s3://simplismart-model-repository/dobby-test-loras/Llama-3.1-8B-Instruct-GRPO-gsm8k-ft-lora", "type": "s3", "secret": { "type": "aws" } } }, { "id": "lora_id_1", "source": { "path": "s3://simplismart-model-repository/dobby-test-loras/llama3.1_text2sql_instruct_tuned", "type": "s3", "secret": { "type": "aws" } } }, { "id": "lora_id_2", "source": { "path": "raaec/llama3.1-8b-instruct-lora-model", "type": "hf", "secret": { "type": "hf" } } } ], "lora_repo": { "type": "", "path": "", "ownership": "", "secret": { "type": "" } }, "quantized_model_path": { "type": "", "path": "", "ownership": "", "secret": { "type": "" } }, "load_lora_dynamic": false } ``` ### Using `lora_repo`: When pulling LoRAs dynamically from a cloud directory: ```json theme={null} "pipeline_config": { "type": "llm", "loras": [], "lora_repo": { "type": "s3", "path": "s3://simplismart-model-repository/dobby-test-loras", "ownership": "", "secret": { "type": "aws" } }, "quantized_model_path": { "type": "", "path": "", "ownership": "", "secret": { "type": "" } }, "load_lora_dynamic": false } ``` *** ## Important Notes ✅ If `load_lora_dynamic` is `false` but the `loras` list contains more than one LoRA, then `load_lora_dynamic` will automatically be set to `true`.\ ✅ The `id` specified in each LoRA will be the model's name during inferencing.\ ✅ When using a `lora_repo`, each subfolder inside the specified path will become a separate model.\ ✅ LoRAs will be dynamically merged or loaded at inference time depending on the `load_lora_dynamic` flag.\ ✅ To understand how to structure secrets, refer to the [Secret Management](/model-suite/integrations/secrets) documentation. Here are some sample secrets for LoRAs. ```json AWS theme={null} { "type": "aws", "access_key_id": "", "secret_access_key": "" } ``` ```json Azure theme={null} { "type": "azure", "auth_json": { "client_id": "", "client_secret": "", "subscription_id": "", "tenant_id": "" } } ``` ```json GCP theme={null} { "type": "gcp", "auth_json": { "auth_provider_x509_cert_url": "", "auth_uri": "", "client_email": "", "client_id": "", "client_x509_cert_url": "", "private_key": "", "private_key_id": "", "project_id": "", "token_uri": "", "type": "", "universe_domain": "" } } ``` ## LoRA Repository Structure \ When using the **LoRA Config → LoRA Repository** option, the S3 bucket should contain the LoRA adapter files in the parent directory. At a minimum, the parent folder **must** contain the following files: ```text theme={null} / ├── adapter_config.json └── adapter_model.safetensors ``` ### Required Files | File | Description | | :-------------------------- | :-------------------------------------------------------------------- | | `adapter_config.json` | Contains the LoRA adapter configuration required to load the adapter. | | `adapter_model.safetensors` | Contains the trained LoRA adapter weights. | > **Note:** Both `adapter_config.json` and `adapter_model.safetensors` must be present in the **parent folder** of the LoRA repository. These are the minimum required files for loading a LoRA adapter. Additional files (such as tokenizer files, training metadata, metrics, etc.) may be present but are not required for compilation. ### Example S3 Structure Image (35) ## Recommendations * **Performance:** Use `load_lora_dynamic = true` if you want the system to load LoRAs on-demand and minimize startup time. * **Organizational Structure:** When using a `lora_repo`, name the directories intuitively, as those names will serve as model identifiers. * **Security:** Configure the appropriate `secret` for S3, GCP, HF, or any supported cloud source to ensure proper authentication and authorization. # Overview Source: https://docs.simplismart.ai/guides/flux-compilation-guide Comprehensive guide for compiling Flux models This page serves as a guide for Flux Pipeline Compilation and Inference using Simplismart. Below are the links to the specific types of pipelines covered in detail. * [Multi-Pipeline](/guides/multi-pipeline) * [Multi-ControlNet](/guides/multi-control-net) * [Multi-Mode ControlNet](https://docs.simplismart.ai/guides/multi-mode-control-net) # LiveKit Integration with Simplismart Source: https://docs.simplismart.ai/guides/livekit Learn how to build real-time voice AI agents using LiveKit with Simplismart's high-performance inference APIs ## What is LiveKit? LiveKit is an open-source platform that enables scalable, multi-user conferencing with WebRTC. It provides the tools you need to add real-time video, audio, and data capabilities to your applications. By combining LiveKit with Simplismart's optimized inference, you can build responsive voice AI agents that handle conversations with minimal latency. Learn more at [LiveKit.io](https://livekit.io). ## Prerequisites Before you begin, ensure you have: * **Simplismart API Key** - Get your API key from [Settings > API Keys](/model-suite/settings/api-keys) * **LiveKit Account** - Visit [LiveKit Cloud](https://cloud.livekit.io/) and create an account to get your API credentials * **Python 3.11 - 3.13** - LiveKit agents require Python \< 3.14. Verify your version with `python --version` Simplismart provides comprehensive AI model serving including STT (Speech-to-Text), LLM (Language Models), and TTS (Text-to-Speech) - all optimized for ultra-low latency in real-time applications. ## Configure LiveKit with Simplismart Set up an isolated Python environment for your project. This keeps dependencies organized and prevents conflicts with other projects. ```bash theme={null} python -m venv .venv source .venv/bin/activate # On Windows: .venv\Scripts\activate ``` Install the LiveKit agents framework with the Simplismart plugin. This includes voice activity detection (VAD) and all necessary components. ```bash theme={null} pip install livekit-plugins-simplismart 'livekit-agents[silero]' python-dotenv ``` The `simplismart` plugin provides native support for Simplismart's STT and TTS services, while the `openai` plugin (included by default) allows you to use any OpenAI-compatible LLM API. Create a `.env` file in your project directory with your API credentials. These credentials authenticate your application with Simplismart and LiveKit services. ```bash theme={null} SIMPLISMART_API_KEY=your-simplismart-api-key-here LIVEKIT_URL=your-livekit-url-here LIVEKIT_API_KEY=your-livekit-api-key-here LIVEKIT_API_SECRET=your-livekit-api-secret-here ``` Get your LiveKit credentials from the [LiveKit Cloud dashboard](https://cloud.livekit.io/): Livekit API Keys Fetch following credentials these in Settings → API Keys → Create key → Copy `Environment Variables` and paste it in the `.env` file. * **LIVEKIT\_URL**: Your project URL (starts with `wss://`) * **LIVEKIT\_API\_KEY** * **LIVEKIT\_API\_SECRET** Build a complete voice AI agent that uses Simplismart for speech-to-text, language processing, and text-to-speech. Create a file named `voice_agent.py`: ```python theme={null} import logging import os from dotenv import load_dotenv from livekit import agents, api from livekit.agents import AgentSession, Agent from livekit.plugins import openai, silero, simplismart load_dotenv() logging.basicConfig(level=logging.INFO) logger = logging.getLogger("voice-agent") # Load Simplismart credentials SIMPLISMART_API_KEY = os.getenv("SIMPLISMART_API_KEY") SIMPLISMART_BASE_URL = "https://api.simplismart.live" # Load LiveKit credentials LIVEKIT_API_KEY = os.getenv("LIVEKIT_API_KEY") LIVEKIT_API_SECRET = os.getenv("LIVEKIT_API_SECRET") LIVEKIT_URL = os.getenv("LIVEKIT_URL") class Assistant(Agent): def __init__(self) -> None: super().__init__(instructions="You are a helpful voice AI assistant.") # Initialize Simplismart STT (Speech-to-Text) model stt = simplismart.STT( base_url=f"{SIMPLISMART_BASE_URL}/predict", api_key=SIMPLISMART_API_KEY, model="openai/whisper-large-v3-turbo" ) # Initialize Simplismart LLM llm = openai.LLM( model="google/gemma-3-4b-it", api_key=SIMPLISMART_API_KEY, base_url=SIMPLISMART_BASE_URL, ) # Initialize Simplismart TTS (Text-to-Speech) model tts = simplismart.TTS( base_url=f"{SIMPLISMART_BASE_URL}/tts", api_key=SIMPLISMART_API_KEY, model="canopylabs/orpheus-3b-0.1-ft" ) async def entrypoint(ctx: agents.JobContext): logger.info(f"Starting agent in room {ctx.room.name}") session = AgentSession( stt=stt, llm=llm, tts=tts, vad=silero.VAD.load(), ) await session.start( room=ctx.room, agent=Assistant(), ) await session.generate_reply( instructions="Greet the user and offer your assistance." ) if __name__ == "__main__": # Generate and display token if not LIVEKIT_API_KEY or not LIVEKIT_API_SECRET: print("Missing LIVEKIT_API_KEY or LIVEKIT_API_SECRET in .env file") print("Get these from your LiveKit Cloud dashboard: https://cloud.livekit.io/") else: token = api.AccessToken(LIVEKIT_API_KEY, LIVEKIT_API_SECRET) \ .with_identity("test_user") \ .with_grants(api.VideoGrants( room_join=True, room="test_room", )) jwt_token = token.to_jwt() print("\n\nLiveKit Agent Ready to Connect!\033[0m\n\033[94m" + "="*50 + "\033[0m") print(f"\033[94mConnect at: https://agents-playground.livekit.io/\033[0m\n\033[94m") print(f"\033[94mURL: {LIVEKIT_URL}\033[0m") print(f"\033[94mToken: {jwt_token}\033[0m\n" + "="*50 + "\033[0m") agents.cli.run_app(agents.WorkerOptions(entrypoint_fnc=entrypoint)) ``` This example uses Simplismart's Whisper for speech-to-text, Gemma 3 4B for language understanding, and Orpheus TTS for natural-sounding speech synthesis - all optimized for real-time performance. Start your voice agent with the LiveKit CLI. The agent will connect to your LiveKit room and wait for a user to join. ```bash theme={null} python voice_agent.py dev ``` To test your agent: 1. Go to the [LiveKit Agents Playground](https://agents-playground.livekit.io/) 2. **If authenticated**: You'll see available rooms and can join directly. **Otherwise**: Use manual connection by entering the URL and token from your terminal (displayed in blue when you run the agent) 3. Click **Connect** 4. **Approve microphone access** when your browser prompts you (required for voice interaction) 5. Speak into your microphone - the agent should respond! Ensure your browser has microphone permissions enabled for the playground to function properly. ## Using Different Simplismart Models **OpenAI Compatibility**: Simplismart's LLM API is fully OpenAI-compatible. This means you can use LiveKit's built-in `openai` plugin to connect to any Simplismart LLM endpoint, no custom plugin needed. Just point `base_url` at `https://api.simplismart.live` and set your `SIMPLISMART_API_KEY`, then pick any model from the [Simplismart Marketplace](https://app.simplismart.ai/model-marketplace). ```python theme={null} from livekit.plugins import openai llm = openai.LLM( model="", api_key=SIMPLISMART_API_KEY, base_url="https://api.simplismart.live", ) ``` You can easily swap models based on your needs. Choose faster models for lower latency or more capable models for complex reasoning tasks. For ultra-fast responses with a compact model, use Gemma 3 1B: ```python theme={null} from livekit.plugins import openai # For faster responses with smaller model llm = openai.LLM( model="google/gemma-3-1b-it", api_key=SIMPLISMART_API_KEY, base_url="https://api.simplismart.live" ) ``` For a balance between speed and capability, use Gemma 3 4B (recommended): ```python theme={null} from livekit.plugins import openai # Balanced performance llm = openai.LLM( model="google/gemma-3-4b-it", api_key=SIMPLISMART_API_KEY, base_url="https://api.simplismart.live" ) ``` For complex reasoning and longer context, use Llama 3.3 70B: ```python theme={null} from livekit.plugins import openai # For maximum capability and complex reasoning llm = openai.LLM( model="meta-llama/llama-3.3-70b-instruct", api_key=SIMPLISMART_API_KEY, base_url="https://api.simplismart.live" ) ``` Try different Whisper models for various accuracy/speed tradeoffs: ```python theme={null} from livekit.plugins import simplismart # Ultra-fast transcription with Whisper v3 Turbo stt = simplismart.STT( base_url="https://api.simplismart.live/predict", api_key=SIMPLISMART_API_KEY, model="openai/whisper-large-v3-turbo" ) # Or use standard Whisper v3 for maximum accuracy stt = simplismart.STT( base_url="https://api.simplismart.live/predict", api_key=SIMPLISMART_API_KEY, model="openai/whisper-large-v3" ) ``` ## Advanced Configuration ### Custom Agent Instructions Customize your agent's behavior by modifying the system instructions: ```python theme={null} class CustomAssistant(Agent): def __init__(self) -> None: super().__init__( instructions="""You are a professional customer support agent for TechCorp. You help customers with product inquiries, troubleshooting, and order tracking. Always be polite, concise, and solution-oriented.""" ) ``` ### Adding Function Tools Enable your agent to perform actions using function tools: ```python theme={null} from livekit.agents import function_tool, RunContext @function_tool async def check_order_status( context: RunContext, order_id: str, ): """Check the status of a customer order.""" # Your order lookup logic here return {"status": "shipped", "tracking": "ABC123"} # Add to your agent session session = AgentSession( stt=stt, llm=llm, tts=tts, vad=silero.VAD.load(), tools=[check_order_status], # Add your tools here ) ``` ## Troubleshooting **Check your microphone permissions** - Ensure your browser or application has access to your microphone. **Verify VAD settings** - The Silero VAD may need tuning for your audio environment. Try adjusting `min_speech_duration` and `min_silence_duration` parameters. **Test STT independently** - Make a direct API call to Simplismart's Whisper endpoint to verify your audio is being transcribed correctly. **Use a smaller model** - Try `google/gemma-3-1b-it` instead of larger models for faster responses. The 1B model typically responds 2-3x faster. **Check network connectivity** - Ensure stable connections to both LiveKit and Simplismart endpoints. Use `ping` and `traceroute` to diagnose network issues. **Optimize instructions** - Shorter, more focused system instructions lead to faster generation. Aim for instructions under 200 words. **Monitor token usage** - Longer conversations accumulate context. Consider implementing context window management to keep prompts concise. **Verify API keys** - Double-check that your `SIMPLISMART_API_KEY` and LiveKit credentials are correct and not expired. **Check base URLs** - Ensure you're using the correct Simplismart endpoints: * STT: `https://api.simplismart.live/predict` * LLM: `https://api.simplismart.live` * TTS: `https://api.simplismart.live/tts` **Review firewall settings** - LiveKit requires WebRTC connections which may be blocked by some firewalls. Ensure UDP ports 50000-60000 are open. **Enable noise cancellation** - Configure noise cancellation in your audio input settings if working in noisy environments. **Check sample rates** - Ensure your audio input matches the expected sample rate for Whisper (16kHz). Mismatched sample rates can cause quality degradation. **Monitor bandwidth** - Poor audio quality can result from insufficient bandwidth. LiveKit automatically adjusts quality, but ensure you have at least 1 Mbps available. **Try different TTS voices** - Simplismart offers multiple TTS models. Experiment to find the best quality for your use case. **Verify Python version** - LiveKit agents require Python 3.11 or later (but \< 3.14). Check your version: ```bash theme={null} python --version ``` **Use pyenv for version management** - If you need multiple Python versions: ```bash theme={null} pyenv install 3.11.5 pyenv local 3.11.5 ``` **Check async compatibility** - Ensure you're using `async`/`await` syntax correctly. LiveKit agents are fully asynchronous. ## Additional Resources * [LiveKit Documentation](https://docs.livekit.io/) - Complete guide to LiveKit features and APIs * [LiveKit Agents Repo](https://github.com/livekit/agents) - Source code and examples * [Simplismart TTS Plugin for LiveKit](https://docs.livekit.io/agents/models/tts/simplismart/) - Livekit Text-to-Speech Plugin integration documentation * [Simplismart TTS Plugin for LiveKit](https://docs.livekit.io/agents/models/tts/simplismart/) - Livekit Speech-to-Text Plugin integration documentation * [LiveKit Community](https://community.livekit.io/) - Get help from the LiveKit community * [Simplismart Support](https://support.simplismart.ai/) - Contact our team for deployment assistance # Multi Control Net Source: https://docs.simplismart.ai/guides/multi-control-net Guide to combining ControlNets with FluxPipeline across different image generation pipelines for more guided and flexible outputs. FluxPipeline provides support for multiple image generation pipelines with and without controlnets, including text-to-image (txt2img), image-to-image (img2img), and inpainting. Ensure that a volume mount is added to the deployment, as all images generated are dumped inside `/data/outputs` directory in the container. ## Model Optimization Configuration ### Optimization Settings For optimization, under the optimization config, use: ```json theme={null} "optimisations": { "attention_caching": { "type": "auto", "enabled": true, "extra_params": { "threshold": 0.1 } } } ``` * Higher threshold values result in greater speed gains but may degrade image generation accuracy. * We recommend a threshold of 0.1, which can provide up to a 40% speed improvement during inference while maintaining reasonable quality. ## Pipeline Settings For optimization, under the optimization config, use: * Multiple ControlNet models can be added under the controlnets section. * Each ControlNet model requires a name, source, and authentication details if needed. ```json theme={null} { "type": "flux", "loras": [], "lora_repo": { "path": "", "type": "", "secret": { "type": "" }, "ownership": "" }, "pipelines": [ "txt2img" ], "controlnets": [ { "name": "canny", "source": { "path": "InstantX/FLUX.1-dev-Controlnet-Canny", "type": "hf", "secret": { "type": "hf", "token": "" } } }, { "name": "depth", "source": { "path": "InstantX/FLUX.1-dev-Controlnet-Depth", "type": "hf", "secret": { "type": "hf", "token": "" } } } ], "model_choice": { "flux_type": "flux" }, "custom_pipeline_config": [], "custom_pipeline_resources": "" } ``` Here are some key pointers for understanding and structuring controlnet requests: ## Understanding ControlNet Parameters ### **ControlNet Name Convention:** * The parameters follow a structured pattern: ``` __control_image _weightage ``` * Example for Canny: ``` "canny_control_image" - The input image processed with the **Canny edge detection** model. "canny_weightage" - Defines the influence of the **Canny edge map** on the final image generation. ``` * Example for Depth: ``` "depth_control_image" → The input image processed with the `Depth estimation` model. "depth_weightage" → Determines how strongly the depth control image impacts the generation. ``` ### **Extensibility for Multiple ControlNets:** * This pattern allows easy extension to additional ControlNet models in a structured way. * If you add a new ControlNet (e.g., OpenPose), you'd include: ```json theme={null} "openpose_control_image": "URL_to_openpose_image", "openpose_weightage": 0.5 ``` ### **How Weightage Works::** * Each weightage parameter (canny\_weightage, depth\_weightage, etc.) determines the degree of influence that specific ControlNet has on the final image. * Higher values make the model adhere more strictly to the control image, potentially sacrificing flexibility. * Lower values allow more artistic freedom but reduce adherence to structured inputs. ### **Combining Multiple ControlNets:** * You can combine multiple ControlNets in a single request to layer different structural constraints. * In this example: * `Canny edge detection` helps maintain sharp edges in the image. * `Depth estimation` preserves 3D structural information. * By adjusting the weightages, you can balance between these two influences. ### **Generalized Pattern for Other ControlNets:** ```json theme={null} "_control_image": "", "_weightage": ``` * Example with Pose and Normal Map: ```json theme={null} "pose_control_image": "URL_to_pose_estimation_image", "pose_weightage": 0.5, "normal_control_image": "URL_to_normal_map_image", "normal_weightage": 0.3 ``` ## Supported Pipelines 1. txt2img - Generates an image from text input. 2. txt2img\_controlnet - Generates an image from text input with controlnet support. 3. img2img - Generates an image based on an input image and a given prompt. 4. img2img\_controlnet - Generates an image based on an input image and a given prompt with controlnet support. 5. inpaint - Modifies specific regions of an image based on a mask and a given prompt. 6. inpaint\_controlnet - Modifies specific regions of an image based on a mask and a given prompt with controlnet support. ## Example Requests ## txt2img ```json theme={null} { "prompt": "A girl in city, 25 years old, cool, futuristic ", "negative_prompt": "canvas frame, (high contrast:1.2), (over saturated:1.2), (glossy:1.1), cartoon, 3d, ((disfigured)), ((bad art)), ((b&w)), blurry, ((bad anatomy)), (((bad proportions))), ((extra limbs)), cloned face, (((disfigured))), extra limbs, (bad anatomy), gross proportions, (malformed limbs), ((missing arms)), ((missing legs)), (((extra arms))), (((extra legs))), mutated hands, (fused fingers), (too many fingers), (((long neck))), Photoshop, video game, ugly, tiling, poorly drawn hands, 3d render", "height": 1024, "width": 1024, "num_images_per_prompt": 4, "num_inference_steps": 20, "seed": 2064977189, "guidance_scale": 4.5, "strength": 0.8, "scheduler": "EULER-A", "model_type": "txt2img" } ``` ## txt2img\_controlnet ```json theme={null} { "prompt": "A girl in city, 25 years old, cool, futuristic ", "negative_prompt": "canvas frame, (high contrast:1.2), (over saturated:1.2), (glossy:1.1), cartoon, 3d, ((disfigured)), ((bad art)), ((b&w)), blurry, ((bad anatomy)), (((bad proportions))), ((extra limbs)), cloned face, (((disfigured))), extra limbs, (bad anatomy), gross proportions, (malformed limbs), ((missing arms)), ((missing legs)), (((extra arms))), (((extra legs))), mutated hands, (fused fingers), (too many fingers), (((long neck))), Photoshop, video game, ugly, tiling, poorly drawn hands, 3d render", "height": 1024, "width": 1024, "num_images_per_prompt": 4, "num_inference_steps": 20, "seed": 2064977189, "guidance_scale": 4.5, "canny_control_image": "https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Canny/resolve/main/canny.jpg", "canny_weightage": 0.4, "depth_control_image": "https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Depth/resolve/main/depth.jpg", "depth_weightage": 0.4, "strength": 0.8, "scheduler": "EULER-A", "model_type": "txt2img_controlnet" } ``` ## img2img ```json theme={null} { "prompt": "cat wizard, gandalf, lord of the rings, detailed, fantasy, cute, adorable, Pixar, Disney, 8k ", "negative_prompt": "canvas frame, (high contrast:1.2), (over saturated:1.2), (glossy:1.1), cartoon, 3d, ((disfigured)), ((bad art)), ((b&w)), blurry, ((bad anatomy)), (((bad proportions))), ((extra limbs)), cloned face, (((disfigured))), extra limbs, (bad anatomy), gross proportions, (malformed limbs), ((missing arms)), ((missing legs)), (((extra arms))), (((extra legs))), mutated hands, (fused fingers), (too many fingers), (((long neck))), Photoshop, video game, ugly, tiling, poorly drawn hands, 3d render", "height": 1024, "width": 1024, "num_images_per_prompt": 4, "num_inference_steps": 20, "image": "https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg", "seed": 89395930, "guidance_scale": 7.0, "strength": 0.5, "scheduler": "EULER-A", "model_type": "img2img" } ``` ## img2img\_controlnet ```json theme={null} { "prompt": "cat wizard, gandalf, lord of the rings, detailed, fantasy, cute, adorable, Pixar, Disney, 8k ", "negative_prompt": "canvas frame, (high contrast:1.2), (over saturated:1.2), (glossy:1.1), cartoon, 3d, ((disfigured)), ((bad art)), ((b&w)), blurry, ((bad anatomy)), (((bad proportions))), ((extra limbs)), cloned face, (((disfigured))), extra limbs, (bad anatomy), gross proportions, (malformed limbs), ((missing arms)), ((missing legs)), (((extra arms))), (((extra legs))), mutated hands, (fused fingers), (too many fingers), (((long neck))), Photoshop, video game, ugly, tiling, poorly drawn hands, 3d render", "height": 1024, "width": 1024, "num_images_per_prompt": 4, "num_inference_steps": 20, "image": "https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg", "canny_control_image": "https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Canny/resolve/main/canny.jpg", "canny_weightage": 0.4, "depth_control_image": "https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Depth/resolve/main/depth.jpg", "depth_weightage": 0.4, "seed": 89395930, "guidance_scale": 7.0, "strength": 0.5, "scheduler": "EULER-A", "model_type": "img2img_controlnet" } ``` ## inpaint ```json theme={null} { "prompt": "Face of a yellow cat, high resolution, sitting on a park bench ", "height": 1024, "width": 1024, "num_images_per_prompt": 4, "num_inference_steps": 20, "image": "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png", "mask_image": "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png", "seed": 89395930, "guidance_scale": 7.0, "strength": 0.5, "scheduler": "EULER-A", "clip_skip": 0, "use_foocus": true, "model_type": "inpaint" } ``` ## inpaint\_controlnet ```json theme={null} { "prompt": "Face of a yellow cat, high resolution, sitting on a park bench ", "height": 1024, "width": 1024, "num_images_per_prompt": 4, "num_inference_steps": 20, "image": "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png", "mask_image": "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png", "canny_control_image": "https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Canny/resolve/main/canny.jpg", "canny_weightage": 0.4, "depth_control_image": "https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Depth/resolve/main/depth.jpg", "depth_weightage": 0.4, "seed": 89395930, "guidance_scale": 7.0, "strength": 0.5, "scheduler": "EULER-A", "clip_skip": 0, "use_foocus": true, "model_type": "inpaint_controlnet" } ``` ## Example Response ```json theme={null} { "response_id": "afbc439946a44d98bb8062c8b36ec16d", "inference_time_taken": 6.336474418640137, "lora_time": 1.8092551231384277, "total_time_taken": 7.176232099533081, "request_id": "6b060ab415d84117b7b6403d622414f5", "error": null } ``` ## Key Notes * Ensure volume mounting in deployment for image storage. * ControlNet models are `not loaded by default`. * Supports `multiple pipelines` for text-to-image, image-to-image, and inpainting. # Multi Mode Control Net Source: https://docs.simplismart.ai/guides/multi-mode-control-net FluxPipeline provides support for multiple image generation pipelines with and without control nets including text-to-image (txt2img), image-to-image (img2img), and inpainting. Ensure that a `volume mount` is added to the deployment, as all images generated are dumped inside the `/data/outputs` directory in the container. ## Optimization Settings For optimization, under the optimization config, use: ```json theme={null} "optimisations": { "attention_caching": { "type": "auto", "enabled": true, "extra_params": { "threshold": 0.1 } } } ``` * Higher threshold values result in greater speed gains but may degrade image generation accuracy. * We recommend a threshold of 0.1, which can provide up to a 40% speed improvement during inference while maintaining reasonable quality. ## Pipeline Settings ### For optimization, under the optimization config, use: * Multi-mode ControlNet model can be added under the controlnets section. * Each ControlNet model requires a name, source, and authentication details if needed. * Instead of using multiple single-mode ControlNets, all conditioning sources are unified within a single model. ### Current Limitations * Supports only one multi-mode ControlNet model. * Does not support multi-mode ControlNet with additional single-mode ControlNets. * This means if using a multi-mode model (FLUX.1-dev-ControlNet-Union-Pro), you cannot combine it with other standalone ControlNets. ```json theme={null} { "type": "flux", "loras": [], "lora_repo": { "path": "", "type": "", "secret": { "type": "" }, "ownership": "" }, "pipelines": [ "txt2img" ], "controlnets": [ { "name": "canny", "source": { "path": "Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro", "type": "hf", "secret": { "type": "hf", "token": "" } } } ], "model_choice": { "flux_type": "flux" }, "custom_pipeline_config": [], "custom_pipeline_resources": "" } ``` ## Understanding ControlNet Parameters ### When using MultiMode Controlnet, these parameters define the multi-mode behavior: | Parameter | Description | | ------------------------------- | ------------------------------------------------------------------- | | `control_image` | List of images used as conditioning (e.g., depth map, canny edges). | | `controlnet_conditioning_scale` | List of `scaling factors` for each conditioning image. | | `control_mode` | Defines the `influence level` for each control image. | ## Supported Pipelines 1. **txt2img** - Generates an image from text input. 2. **txt2img\_controlnet** - Generates an image based on an input image and a given prompt. 3. **img2img** - Modifies specific regions of an image based on a mask and a given prompt. 4. **img2img\_controlnet** - Generates an image based on an input image and a given prompt with controlnet support. 5. **inpaint** - Modifies specific regions of an image based on a mask and a given prompt. 6. **inpaint\_controlnet** - Modifies specific regions of an image based on a mask and a given prompt with controlnet support. ## Example Requests ### txt2img ```json theme={null} { "prompt": "A girl in city, 25 years old, cool, futuristic ", "negative_prompt": "canvas frame, (high contrast:1.2), (over saturated:1.2), (glossy:1.1), cartoon, 3d, ((disfigured)), ((bad art)), ((b&w)), blurry, ((bad anatomy)), (((bad proportions))), ((extra limbs)), cloned face, (((disfigured))), extra limbs, (bad anatomy), gross proportions, (malformed limbs), ((missing arms)), ((missing legs)), (((extra arms))), (((extra legs))), mutated hands, (fused fingers), (too many fingers), (((long neck))), Photoshop, video game, ugly, tiling, poorly drawn hands, 3d render", "height": 1024, "width": 1024, "num_images_per_prompt": 4, "num_inference_steps": 20, "seed": 2064977189, "guidance_scale": 4.5, "strength": 0.8, "scheduler": "EULER-A", "model_type": "txt2img" } ``` ### txt2img\_controlnet ```json theme={null} { "prompt": "A girl in city, 25 years old, cool, futuristic ", "negative_prompt": "canvas frame, (high contrast:1.2), (over saturated:1.2), (glossy:1.1), cartoon, 3d, ((disfigured)), ((bad art)), ((b&w)), blurry, ((bad anatomy)), (((bad proportions))), ((extra limbs)), cloned face, (((disfigured))), extra limbs, (bad anatomy), gross proportions, (malformed limbs), ((missing arms)), ((missing legs)), (((extra arms))), (((extra legs))), mutated hands, (fused fingers), (too many fingers), (((long neck))), Photoshop, video game, ugly, tiling, poorly drawn hands, 3d render", "height": 1024, "width": 1024, "num_images_per_prompt": 4, "num_inference_steps": 20, "seed": 2064977189, "guidance_scale": 4.5, "control_images": [ "https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro/resolve/main/assets/depth.jpg", "https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro/resolve/main/assets/canny.jpg" ], "control_weightages": [0.2, 0.4], "control_modes": [2, 0], "strength": 0.8, "scheduler": "EULER-A", "model_type": "txt2img_controlnet" } ``` ### img2img ```json theme={null} { "prompt": "cat wizard, gandalf, lord of the rings, detailed, fantasy, cute, adorable, Pixar, Disney, 8k ", "negative_prompt": "canvas frame, (high contrast:1.2), (over saturated:1.2), (glossy:1.1), cartoon, 3d, ((disfigured)), ((bad art)), ((b&w)), blurry, ((bad anatomy)), (((bad proportions))), ((extra limbs)), cloned face, (((disfigured))), extra limbs, (bad anatomy), gross proportions, (malformed limbs), ((missing arms)), ((missing legs)), (((extra arms))), (((extra legs))), mutated hands, (fused fingers), (too many fingers), (((long neck))), Photoshop, video game, ugly, tiling, poorly drawn hands, 3d render", "height": 1024, "width": 1024, "num_images_per_prompt": 4, "num_inference_steps": 20, "image": "https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg", "seed": 89395930, "guidance_scale": 7.0, "strength": 0.5, "scheduler": "EULER-A", "model_type": "img2img" } ``` ### img2img\_controlnet **Note** : Currently `num_images_per_prompt = 1` is supported. ```json theme={null} { "prompt": "cat wizard, gandalf, lord of the rings, detailed, fantasy, cute, adorable, Pixar, Disney, 8k ", "negative_prompt": "canvas frame, (high contrast:1.2), (over saturated:1.2), (glossy:1.1), cartoon, 3d, ((disfigured)), ((bad art)), ((b&w)), blurry, ((bad anatomy)), (((bad proportions))), ((extra limbs)), cloned face, (((disfigured))), extra limbs, (bad anatomy), gross proportions, (malformed limbs), ((missing arms)), ((missing legs)), (((extra arms))), (((extra legs))), mutated hands, (fused fingers), (too many fingers), (((long neck))), Photoshop, video game, ugly, tiling, poorly drawn hands, 3d render", "height": 1024, "width": 1024, "num_images_per_prompt": 1, "num_inference_steps": 20, "image": "https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg", "control_images": [ "https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro/resolve/main/assets/depth.jpg", "https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro/resolve/main/assets/canny.jpg" ], "control_weightages": [0.2, 0.4], "control_modes": [2, 0], "seed": 89395930, "guidance_scale": 7.0, "strength": 0.5, "scheduler": "EULER-A", "model_type": "img2img_controlnet" } ``` ### inpaint ```json theme={null} { "prompt": "Face of a yellow cat, high resolution, sitting on a park bench ", "height": 1024, "width": 1024, "num_images_per_prompt": 4, "num_inference_steps": 20, "image": "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png", "mask_image": "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png", "seed": 89395930, "guidance_scale": 7.0, "strength": 0.5, "scheduler": "EULER-A", "clip_skip": 0, "use_foocus": true, "model_type": "inpaint" } ``` ### inpaint\_controlnet **Note** : `Currently num_images_per_prompt = 1` is supported. ```json theme={null} { "prompt": "Face of a yellow cat, high resolution, sitting on a park bench ", "height": 1024, "width": 1024, "num_images_per_prompt": 1, "num_inference_steps": 20, "image": "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png", "mask_image": "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png", "control_images": [ "https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro/resolve/main/assets/depth.jpg", "https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro/resolve/main/assets/canny.jpg" ], "control_weightages": [0.2, 0.4], "control_modes": [2, 0], "seed": 89395930, "guidance_scale": 7.0, "strength": 0.5, "scheduler": "EULER-A", "clip_skip": 0, "use_foocus": true, "model_type": "inpaint_controlnet" } ``` ## Example Response ```json theme={null} { "response_id": "afbc439946a44d98bb8062c8b36ec16d", "inference_time_taken": 6.336474418640137, "lora_time": 1.8092551231384277, "total_time_taken": 7.176232099533081, "request_id": "6b060ab415d84117b7b6403d622414f5", "error": null } ``` ## Key Notes * Ensure volume mounting in deployment for image storage. * ControlNet models are `not loaded by default`. * Ensure all 3 `control_` params are sent out in request. * Supports `multiple pipelines` for text-to-image, image-to-image, and inpainting. # Multi Pipeline Source: https://docs.simplismart.ai/guides/multi-pipeline Guide to using multiple image generation pipelines with FluxPipeline on Simplismart. FluxPipeline provides support for multiple image generation pipelines, including text-to-image (txt2img), image-to-image (img2img), and inpainting. Ensure that a volume mount is added to the deployment, as all images generated are dumped inside `/data/outputs` directory in the container. ## Model Optimization Configuration * The standard base configuration presented in the UI is sufficient for optimal performance. * ControlNet models will not be loaded by default. ## Optimization Settings For optimization, under the optimization config, use: ```json theme={null} "optimisations": { "attention_caching": { "type": "auto", "enabled": true, "extra_params": { "threshold": 0.1 } } } ``` * Higher threshold values result in greater speed gains but may degrade image generation accuracy. * We recommend a threshold of `0.1`, which can provide up to a `40%` speed improvement during inference while maintaining reasonable quality. ## Supported Pipelines 1. **txt2img** - Generates an image from text input. 2. **img2img** - Generates an image based on an input image and a given prompt. 3. **inpaint** - Modifies specific regions of an image based on a mask and a given prompt. ## Example Requests ## txt2img ```json theme={null} { "prompt": "A girl in city, 25 years old, cool, futuristic ", "negative_prompt": "canvas frame, (high contrast:1.2), (over saturated:1.2), (glossy:1.1), cartoon, 3d, ((disfigured)), ((bad art)), ((b&w)), blurry, ((bad anatomy)), (((bad proportions))), ((extra limbs)), cloned face, (((disfigured))), extra limbs, (bad anatomy), gross proportions, (malformed limbs), ((missing arms)), ((missing legs)), (((extra arms))), (((extra legs))), mutated hands, (fused fingers), (too many fingers), (((long neck))), Photoshop, video game, ugly, tiling, poorly drawn hands, 3d render", "height": 1024, "width": 1024, "num_images_per_prompt": 4, "num_inference_steps": 20, "seed": 2064977189, "guidance_scale": 4.5, "strength": 0.8, "scheduler": "EULER-A", "model_type": "txt2img" } ``` ## Img2Img ```json theme={null} { "prompt": "cat wizard, gandalf, lord of the rings, detailed, fantasy, cute, adorable, Pixar, Disney, 8k ", "negative_prompt": "canvas frame, (high contrast:1.2), (over saturated:1.2), (glossy:1.1), cartoon, 3d, ((disfigured)), ((bad art)), ((b&w)), blurry, ((bad anatomy)), (((bad proportions))), ((extra limbs)), cloned face, (((disfigured))), extra limbs, (bad anatomy), gross proportions, (malformed limbs), ((missing arms)), ((missing legs)), (((extra arms))), (((extra legs))), mutated hands, (fused fingers), (too many fingers), (((long neck))), Photoshop, video game, ugly, tiling, poorly drawn hands, 3d render", "height": 1024, "width": 1024, "num_images_per_prompt": 4, "num_inference_steps": 20, "image": "https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg", "seed": 89395930, "guidance_scale": 7.0, "strength": 0.5, "scheduler": "EULER-A", "model_type": "img2img" } ``` ## Inpaint ```json theme={null} { "prompt": "Face of a yellow cat, high resolution, sitting on a park bench ", "height": 1024, "width": 1024, "num_images_per_prompt": 4, "num_inference_steps": 20, "image": "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png", "mask_image": "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png", "seed": 89395930, "guidance_scale": 7.0, "strength": 0.5, "scheduler": "EULER-A", "clip_skip": 0, "use_foocus": true, "model_type": "inpaint" } ``` ## Example Response ```json theme={null} { "response_id": "afbc439946a44d98bb8062c8b36ec16d", "inference_time_taken": 6.336474418640137, "lora_time": 1.8092551231384277, "total_time_taken": 7.176232099533081, "request_id": "6b060ab415d84117b7b6403d622414f5", "error": null } ``` ## Key Notes * Ensure volume mounting in deployment for image storage. * ControlNet models are not loaded by default. * The default configuration in the UI should work optimally for most use cases. * Supports multiple pipelines for text-to-image, image-to-image, and inpainting. # Optimization Guide Source: https://docs.simplismart.ai/guides/optimization-guide ## Overview This document details all configuration parameters available for the backend server, a high-performance framework designed for serving **Large Language Models**. It covers optimization and pipeline settings, including backend selection, quantization, parallelism, and advanced features such as speculative decoding and LoRA integration, supporting over 140 configurable options for flexible and efficient model deployment. ## Table of Contents * [Quick Start](#quick-start) * [Optimization Configuration](#optimization-configuration) * [Backend Settings](#backend-settings) * [Name](#name) * [Extra Parameters](#extra-parameters) * [V4 Backend](#v4-backend) * [V3 Backend](#v3-backend) * [Model Settings](#model-settings) * [Quantization](#quantization) * [Optimisations](#optimisations) * [Speculative Decoding](#speculative-decoding) * [Parallelism](#parallelism) * [Tensor Parallel Size](#tensor-parallel-size) * [Pipeline Configuration](#pipeline-configuration) ## Quick Start ### Default Optimization Configuration ```json theme={null} { "backend": { "name": "auto", "version": "latest", "extra_params": { "enable_torch_compile": true } }, "warmups": { "enabled": true, "iterations": 5, "sample_input_data": [] }, "model_type": "llm", "quantization": "float16", "optimisations": { "model_type": "llm", "attention_caching": { "type": "auto", "enabled": false, "extra_params": {} }, "speculative_decoding": { "type": "auto", "enabled": false, "extra_params": {} } }, "tensor_parallel_size": 1 } ``` ### Default Pipeline Configuration ```json theme={null} { "mode": "chat", "type": "llm", "loras": [], "lora_repo": { "path": "", "type": "", "secret": { "type": "" }, "ownership": "" }, "load_lora_dynamic": false, } ``` ## Optimization Configuration ### Backend Settings ### Name * **default**: "auto" * **description**: This takes backend name to compile your model with, with value `auto`, it chooses best backend based on configuration provided. * **values**: "auto", "v3", "v4" ### Extra Parameters This takes dicitonary as params, and these params are the configuration for server started with backend v3 and v4 ### V4 Backend | Parameter | Type | Default | Description | | ------------------------ | ------- | ------- | ------------------------------------------- | | `context_length` | integer | `null` | Maximum context length for the model | | `gpu_memory_utilization` | float | `null` | Fraction of GPU memory to reserve (0.0-1.0) | | `max_running_requests` | integer | `null` | Maximum concurrent requests | | `max_total_tokens` | integer | `null` | Maximum total tokens across all requests | | `chunked_prefill_size` | integer | `null` | Chunk size for prefill processing | | `max_prefill_tokens` | integer | `16384` | Maximum tokens processed in prefill phase | | `reasoning_parser` | string | `null` | Parser for reasoning task outputs | | `enable_torch_compile` | bool | `false` | Enable PyTorch 2.0 compilation | | `torch_compile_max_bs` | integer | `32` | Max batch size for compilation | | `cuda_graph_max_bs` | integer | `32` | Max batch size for CUDA graphs | | `tool_call_parser` | string | `null` | Parser for function call parsing | ### Notes * Possible Values for Reasoning Parser: `deepseek-r1`, `qwen3` * Possible values for tool\_call\_parser: `mistral`, `llama4`, `llama3`, `qwen25` ### V3 Backend | Parameter | Type | Default | Description | | ------------------------ | ------- | ------- | ------------------------------------------- | | `context_length` | integer | `null` | Maximum context length for the model | | `gpu_memory_utilization` | float | `null` | Fraction of GPU memory to reserve (0.0-1.0) | | `reasoning_parser` | string | `null` | Parser for reasoning task outputs | | `tool_call_parser` | string | `null` | Parser for function call parsing | ### Notes * Possible Values for Reasoning Parser: `deepseek_r1`, `qwen3` * Possible values for tool\_call\_parser: `mistral`, `llama3_json`, `llama4_json`, `hermes` ## Model Settings ### Quantization * **default**: `float16`, for no quantization * **values**: `float16`, `float8`, `w4` * **description**: Data type or quantization format to use for model weights. Use `float16` for standard precision, or select a quantized format for reduced memory usage. ## Optimisations ### Speculative Decoding * **type**: "auto" * **enabled**: true * **extra\_params**: * **algorithm**: * **type**: string * **default**: `"draft"` * **values**: `"draft"`, `"n-gram"`, `"eagle"`, `"eagle3"`, `"nextn"` * **description**: Decoding algorithm to use for speculative decoding. * **draft\_model\_path**: * **type**: string or null * **default**: `null` * **description**: Path to the draft model used in speculative decoding. If not set, uses the main model. * **num\_draft\_tokens**: * **type**: integer * **default**: `64` * **description**: Number of tokens to generate in each speculative draft step. * **num\_steps**: * **type**: integer * **default**: `5` * **description**: Number of speculative decoding steps to perform. * **topk**: * **type**: integer * **default**: `4` * **description**: Top-k sampling parameter for candidate selection during decoding. * **tp\_size**: * **type**: integer * **default**: `1` * **description**: Tensor parallel size for distributed speculative decoding. ## Parallelism ### Tensor Parallel Size * **Parameter**: `tensor_parallel_size` * **Type**: integer * **Default**: `1` * **Description**:\ Specifies the number of tensor parallel processes to use for model inference. Increasing this value enables the model to split its computations across multiple GPUs or nodes, which can accelerate inference and allow for larger models to be served. * Set to `1` for no tensor parallelism (single process). * Set to a higher integer (e.g., `2`, `4`, `8`) to enable tensor parallelism across that many devices. * Ensure your hardware setup matches the specified `tensor_parallel_size` (e.g., you have at least as many GPUs as the value you set). ## Pipeline Configuration * **mode**: * **type**: string * **default**: `"chat"` * **values**: `"chat"`, `"completion"`, `"embedding"` * **description**: Specifies the pipeline mode. * `"chat"`: For conversational/chat-based interactions. * `"completion"`: For standard text completion tasks. * `"embedding"`: For generating vector embeddings from input text. * **loras**: * **type**: list of objects * **default**: `[]` * **description**: List of LoRA (Low-Rank Adaptation) adapters to load with the model. Each object should specify the LoRA's unique ID and its source location. See the [documentation](https://docs.simplismart.ai/guides/dynamic-lora-compilation) for more details. * **Note**: If only 1 lora is provided then it will be merged with base model, else they will loaded dynamically. # Whisper Deployment Guide Source: https://docs.simplismart.ai/guides/whisper-deployment-guide Learn how to deploy Whisper V3 for transcription and translation, featuring VAD, diarization, and advanced hallucination-reduction capabilities ## **Overview** This API provides speech-to-text transcription and translation services using OpenAI's **Whisper V3** model with advanced features like voice activity detection (VAD), speaker diarization, and hallucination reduction. *** ## **Endpoint** ```python theme={null} POST "/predict" ``` *** ## **Authentication** Include Bearer token in headers: ```json theme={null} headers = {"Authorization": "Bearer YOUR_JWT_TOKEN"} ``` *** ## **Request Parameters** ### **Required Parameters** * `audio_file` (string): Audio input - either Base64-encoded audio file data or publicly accessible audio URL. ### **Core Processing Parameters** * `language` (string): Source language code. (e.g., "hi" for Hindi, "en" for English, `None` for auto-detection) * `task` (string): Processing task - **transcribe** (speech-to-text in same language) or **translate** (translate to English) Here is the list of [Supported Language](https://docs.simplismart.ai/get-started/playground/transcription-models#supported-languages-with-their-codes) in the Whisper Model. **Voice Activity Detection (VAD) Parameters** * `vad_model` (string): VAD model to use - "`silero`" (recommended for accuracy and speed) * `vad_onset` (float): Threshold for speech start detection (0.0-1.0, default: 0.5) * `vad_offset` (float): Threshold for speech end detection (0.0-1.0, default: 0.3) Need help with VAD parameter tuning or Whisper troubleshooting? \ \ Check our detailed guides on [VAD tuning](/troubleshooting-faq/vad-parameter-tuning) and [Whisper troubleshooting](/troubleshooting-faq/whisper-troubleshooting). ### **Timestamp Parameters** * `word_timestamps` (boolean): Enable word-level timestamps in output (`true`/`false`). * `without_timestamps` (boolean): Exclude timestamps from transcription text. Only sample text tokens from whisper model. (`true`/`false`). ### **Speaker Identification** * `diarization` (boolean): Enable speaker Diarization to identify different speakers (true/false) ### **Hallucination Reduction** * `strict_hallucination_reduction` (boolean): Apply post-processing filters to remove repeated phrases in the transcription (true/false) *** ## **Audio Format Support** * **MP3**, **WAV**, **M4A**, **AAC** * URLs must be publicly accessible or a pre-signed URL. *** ## **Example Request** ### **Option 1: Base64-encoded audio file** ```python theme={null} import requests import base64 import json # Read and encode audio file with open("audio_file.mp3", "rb") as f: audio_data = f.read() audio_base64 = base64.b64encode(audio_data).decode("utf-8") # API request with base64 headers = {"Authorization": "Bearer YOUR_JWT_TOKEN"} payload = { "audio_file": audio_base64, # Base64-encoded audio data "language": "hi", # Hindi audio "task": "translate", # Translate to English "vad_model": "silero", # Use Silero VAD "word_timestamps": True, # Include word timestamps "without_timestamps": False, # Keep timestamps in text "diarization": True, # Identify speakers "vad_onset": 0.5, # Speech detection threshold "vad_offset": 0.3, # Speech end threshold "strict_hallucination_reduction": True # Reduce false content } response = requests.post( "/predict", json=payload, headers=headers ) ``` *** ### **Option 2: Audio URL** ```python theme={null} import requests import json # API request with URL headers = {"Authorization": "Bearer YOUR_JWT_TOKEN"} payload = { "audio_file": "https://example.com/audio.mp3", # Publicly accessible audio URL "language": "en", # English audio "task": "transcribe", # Transcribe in same language "vad_model": "silero", # Use Silero VAD "word_timestamps": True, # Include word timestamps "without_timestamps": False, # Keep timestamps in text "diarization": False, # Single speaker "vad_onset": 0.5, # Speech detection threshold "vad_offset": 0.3, # Speech end threshold "strict_hallucination_reduction": True # Reduce false content } response = requests.post( "/predict", json=payload, headers=headers ) ``` *** ## **Response Format** ### **Successful Response (200)** ```json theme={null} { "duration": [ 3.02 ], "transcription": [ { "text": "Hello World", "start": 0.0, "end": 3.02 } ], "word_timestamps": [ { "word": " Hello", "start": 0.0, "end": 0.34, "probability": 0.92 }, { "word": " World", "start": 0.34, "end": 0.6, "probability": 0.92 } ], "diarization": [ { "start": 0.0, "end": 0.34, "text": "Hello World", "speaker": 1 } ], "info": { "language": "en", "probability": 1 }, "metrics": { "audio_loading_preprocessing": 0.009385824203491211, "audio_chunk": 0.014404058456420898, "audio_prediction": 0.08741092681884766, "audio_word_processing": 0.07052230834960938, "audio_diarization": 0.07052850723266602, "hallucination_reduction": 0.0004951953887939453 } } ``` *** ### **Error Response (4xx)** ```json theme={null} { "error": "Invalid or expired authentication token", "code": "401" } ``` *** ### **Error Response (5xx)** ```json theme={null} { "error": "Internal server error", "code": "500" } ``` *** ## **Error Codes** * **400**: Invalid request format or parameters * **401**: Authentication failed * **413**: File too large * **422**: Unsupported audio format * **500**: Internal server error # Overview Source: https://docs.simplismart.ai/inference/bring-your-own-compute Deploy and run models on your own cloud infrastructure. Bring-your-own-compute (BYOC) allows you to use your own compute solution for deploying and running your ML models. Dedicated Endpoint *** ## **Benefits of BYOC** * **Control over infrastructure**: Utilize your existing resources and configurations. * **Cost savings**: Avoid additional costs associated with using our infrastructure. * **Scalability**: Scale deployments according to your own infrastructure capabilities. *** ## **Deploying a model using your own compute** To deploy your model using your own compute resources, complete the necessary prerequisite integrations and set up your workflow seamlessly on the model suite using the processes outlined below: 1. **Integrating your accounts** You can add your cloud accounts, hosted zones and create secrets for your secure information, within the [integrations](/model-suite/integrations/secrets) page. 2. **Adding your cluster** Add your compute cluster by navigating to the [Clusters](/model-suite/clusters) page and providing the necessary details. 3. **Model Optimisation** * To optimise a model, you can either choose an available model from our [model marketplace](https://app.simplismart.ai/model-marketplace), or add your own. * Optimise your model for deployment by visiting the [Models](/model-suite/optimise-a-model) page and using our optimisation tools. 4. **Model Deployment** Deploy your optimised model through the steps mentioned in the [Deployments](/model-suite/deployments/creating-a-deployment) page. 5. **Monitoring your deployment** Keep track of your deployment using the [monitoring tools](/model-suite/deployments/inference-and-monitoring) available on the deployments page. 6. **Running a benchmarking job** Additionally, you can perform [benchmark](/model-suite/deployments/benchmarking) your deployed model for specific scenarios. 7. **Inferencing** You can invoke your deployed models from the **API tab** of the model deployment. For **BYOC (Bring Your Own Cloud)** deployments, pricing may vary based on your setup and resource usage. Please contact our [support team](https://support.simplismart.ai/support/tickets/new?ticket_form=ask_a_question) for pricing information. # Dedicated Endpoint Source: https://docs.simplismart.ai/inference/dedicated-endpoint Deploy models on exclusive, isolated infrastructure with stable performance and full customization Dedicated endpoints provide exclusive infrastructure for your model deployments, ensuring stable performance, reliability, and isolation from shared workloads. They also enable the deployment of **proprietary or custom models**, allowing greater flexibility in configuration, scaling, and security. Private Endpoint ## **Benefits of using a Dedicated endpoint** * **Dedicated resources**: No sharing of compute resources with other users. * **Bring your own models**: Deploy your own custom or finetuned models. * **Enhanced performance**: Improved response times and throughput. * **Higher reliability**: Reduced risk of downtime and performance degradation. *** ## Deploying your model on a Dedicated Endpoint To deploy your model on a dedicated endpoint, follow these outlined processes. Each step includes a link for detailed instructions, ensuring a smooth launch and use of your deployed model. *** 1. Optimise your model for deployment by visiting the [Models](/model-suite/optimise-a-model) page and using our optimisation tools. 2. You can either choose a pre-optimized model from our [Model Marketplace](https://app.simplismart.ai/model-marketplace), or add your own. **Pre-optimized models** are faster to deploy and support one-click deployment for quick setup and usage. You can deploy your optimised model on a dedicated endpoint by selecting your cloud provider as `Simplismart Cloud`, click here for detailed [Deployment](/model-suite/deployments/creating-a-deployment) steps. You can invoke your deployed models from the API tab of the model deployment. Want the deployment to run in your own cluster? Here's [how](/inference/bring-your-own-compute). For dedicated endpoints, pricing depends on GPU usage, with different GPUs priced separately. For detailed pricing information, refer to the [Pricing](https://www.simplismart.ai/pricing) section on our website. # Shared Endpoint Source: https://docs.simplismart.ai/inference/shared-endpoint Access models through Simplismart’s shared playground endpoints. Shared endpoints allow you to invoke ML models using our shared infrastructure. This is a cost-effective solution for users who do not require dedicated resources. Shared Endpoint *** ## **Using a shared endpoint** Inferencing any model on a shared endpoint is straightforward. You can directly use our [Playground](/get-started/playground/large-language-models) or perform an [API](/api-reference/introduction) call. Want a dedicated deployment for better SLAs? Check out [**Dedicated Endpoints**](/inference/dedicated-endpoint.mdx). For shared endpoints, pricing is determined by the number of API calls and actual usage, For detailed pricing information, refer to the [Pricing](https://www.simplismart.ai/pricing) section on our website. # Add Container Source: https://docs.simplismart.ai/model-suite/add-container Add a container image from Docker Hub, Depot, Nvidia NGC, Quay, or Amazon ECR to Simplismart Simplismart supports containers from five registries: **Docker Hub**, **Depot**, **Nvidia NGC** (for NIM containers), **Quay**, and **Amazon ECR**. This page covers setting up credentials and adding your image. Once added, see [Deploy Your Container](/model-suite/deployments/deploy-container) to configure and launch the deployment. ## Prerequisites * A container image hosted on Docker Hub, Depot, Nvidia NGC, Quay, or Amazon ECR. * For **private images**: credentials configured in Simplismart (see [Setting Up Registry Credentials](#setting-up-registry-credentials)). Public Docker Hub and Quay images do not require any credentials. You can skip the credentials section and go straight to [Adding Your Image](#adding-your-image). ## Setting Up Registry Credentials Skip this section if you are using a public image. Navigate to **Integrations** → [**Secrets**](https://app.simplismart.ai/integrations/secrets) in your Simplismart dashboard. Click **Create Secret** and fill in the following fields: * **Secret Name**: A descriptive name for this secret. * **Secret Type**: Select **Docker Hub** from the dropdown. * **Username**: Your Docker Hub username. * **Access Token**: Your Docker Hub access token. Click **Save**. The secret appears in your secrets list and can be selected when adding a private image. Navigate to **Integrations** → [**Secrets**](https://app.simplismart.ai/integrations/secrets) in your Simplismart dashboard. Click **Create Secret** and fill in the following fields: * **Secret Name**: A descriptive name for this secret. * **Secret Type**: Select **Depot** from the dropdown. * **Username**: Enter `x-token` (this is always the username for Depot). * **Access Token**: Your Depot access token. Click **Save**. The secret appears in your secrets list. Navigate to **Integrations** → [**Secrets**](https://app.simplismart.ai/integrations/secrets) in your Simplismart dashboard. Click **Create Secret** and fill in the following fields: * **Secret Name**: A descriptive name for this secret. * **Secret Type**: Select **Nvidia Secret Docker** from the dropdown. * **API Key**: Your NVIDIA NGC API key. Nvidia NGC secret configuration Click **Save**. The secret appears in your secrets list and can be selected when adding an NGC image. Only **public** Quay images are currently supported. No credentials are required. You can skip to [Adding Your Image](#adding-your-image). Navigate to **Integrations** → [**Secrets**](https://app.simplismart.ai/integrations/secrets) in your Simplismart dashboard. Click **Create Secret** and fill in the following fields: * **Secret Name**: A descriptive name for this secret. * **Secret Type**: Select **ECR** from the dropdown. * **Secret Data**: Paste your AWS credentials in the following JSON format: ```json theme={null} { "region": "us-west-2", "accessKey": "", "secretKey": "" } ``` Click **Create**. The secret appears in your secrets list and can be selected when adding an ECR image. ## Adding Your Image Go to the **My Models** page and click **Bring Your Own Container**. Fill in the fields for your registry. The **Model Name** and **GPU Per Container** fields are the same for all registries. Add model-— Docker Hub * **Model Name**: A descriptive name for your application. * **Platform Type**: Select **DockerHub** from the dropdown. * **Registry Path**: Your Docker Hub registry path (e.g. `myorg/myimage`). * **Container Image Tag**: The image version or tag (e.g. `latest`). * **GPU Per Container**: Number of GPUs per container. Default is `1`; maximum is `8`. * **Registry Credentials**: Select the Docker Hub secret configured earlier. Leave blank for public images. First, locate your registry path in Depot: 1. Go to your **Depot** dashboard and navigate to **Registry**. 2. Find your image in the **Image Reference** column. It will look like: ``` registry.depot.dev/YOUR-ORG-ID:CONTAINER-IMAGE-TAG ``` Split this into two parts: * **Registry Path**: `registry.depot.dev/YOUR-ORG-ID` (everything before the `:`) * **Container Image Tag**: `CONTAINER-IMAGE-TAG` (everything after the `:`) Depot image reference Then fill in the model fields: * **Model Name**: A descriptive name for your application. * **Platform Type**: Select **Depot** from the dropdown. * **Registry Path**: The path copied from Depot (e.g. `registry.depot.dev/YOUR-ORG-ID`). * **Container Image Tag**: The tag copied from Depot. * **GPU Per Container**: Number of GPUs per container. Default is `1`; maximum is `8`. * **Registry Credentials**: Select the Depot secret configured earlier. * **Model Name**: A descriptive name for your application. * **Platform Type**: Select **NGC** from the dropdown. * **Registry Path**: The NGC image path (e.g. `nvcr.io/nvidia/nim/meta/llama-3.1-8b-instruct`). * **Container Image Tag**: The image tag (e.g. `latest`). * **GPU Per Container**: Number of GPUs per container. Default is `1`; maximum is `8`. * **Registry Credentials**: Select the Nvidia NGC secret configured earlier. * **Model Name**: A descriptive name for your application. * **Platform Type**: Select **Quay** from the dropdown. * **Registry Path**: The Quay image path (e.g. `quay.io/myorg/myimage`). * **Container Image Tag**: The image tag (e.g. `latest`). * **GPU Per Container**: Number of GPUs per container. Default is `1`; maximum is `8`. * **Registry Credentials**: Leave blank, only public Quay images are supported. Add model — Amazon ECR * **Model Name**: A descriptive name for your application. * **Platform Type**: Select **Amazon ECR** from the dropdown. * **Registry Path**: The full ECR image URI without the tag (e.g. `123456789012.dkr.ecr.us-west-2.amazonaws.com/my-repo`). * **Container Image Tag**: The image tag (e.g. `latest`). * **GPU Per Container**: Number of GPUs per container. Default is `1`; maximum is `8`. * **Registry Credentials**: Select the Amazon ECR secret configured earlier. Expand **Container Configuration** to define how the platform interacts with your container. These settings carry over to the deployment step and can be overridden there. Container configuration Click **Add HTTP**, **Add gRPC**, or **Add Monitoring** to enable the respective services. * **HTTP Service** (Required): The port your server listens on (e.g. `8000`). * **Public Access**: Enable to make the endpoint reachable externally. When disabled, the service is accessible only within the cluster. * **gRPC Service**: Enable if your application uses gRPC (must use a different port than HTTP). * **Monitoring Service**: Enable for enhanced monitoring and observability. * **Health Check Configuration**: If any field is filled, all fields are required. * **Health Check Path**: e.g. `/health` * **Port**: Must match your HTTP service port. * **Initial Delay**: Time in seconds before the first check. * **Period**: Interval in seconds between checks. * **Timeout**: Maximum seconds to wait for a response. * **Environment Variables**: Runtime key-value pairs for your container. Container configuration continued * **Command Override**: (Optional) Custom command to replace the container's default startup command. * **Model Endpoint**: (Optional) Inference path for observability (e.g. `/v1/chat/completions`). Click **Add Model**. For Docker Hub, the image path is verified immediately. For other registries, the image is confirmed when the deployment starts. Once your image is added, continue to [Deploy Your Container](/model-suite/deployments/deploy-container) to configure and launch the deployment. # Adding a Custom Model Source: https://docs.simplismart.ai/model-suite/adding-a-custom-model Prepare and add a custom model or custom pipeline to the Simplismart platform. For **custom models or custom pipelines**, you must **prepare the model configuration before adding the model to the platform**. This includes defining the model logic, dependencies, and runtime environment. The platform expects all required files to be packaged together and provided as a single artifact (a ZIP file). This page describes how to implement the model interface, define runtime configuration, package your model, and add it through the Simplismart platform. ## Model interface (`model.py`) Your custom model must implement a standard interface so the platform can load and run it correctly. **Method requirements:** * **`load()`**: Handles model initialization and weight loading * **`preprocess()`**: Optional input preprocessing * **`predict()`**: Core inference logic * **`postprocess()`**: Optional output formatting Example: ```python theme={null} class Model: def __init__(self): self.model = None def load(self): # Initialize or load model weights self.model = "Model Initialization" def preprocess(self, request): """ Preprocess the incoming request. Input can be a Pydantic BaseModel, dict, or string. """ return request def predict(self, request): # Run inference output = self.model.predict() return output def postprocess(self, request): # Postprocess the model output return request ``` ## Runtime configuration (`config.yaml`) The `config.yaml` file defines the execution environment for the custom model. | Section | Purpose | | ----------------------- | ------------------------------------------- | | `python_version` | Python runtime version | | `environment_variables` | Custom environment variables (if any) | | `requirements` | Python dependencies | | `system_packages` | OS-level packages | | `custom_setup_script` | Optional setup script executed during build | Example: ```yaml theme={null} python_version: "3.10" environment_variables: {} requirements: - accelerate==0.20.3 - bitsandbytes==0.39.1 - peft==0.3.0 - protobuf==4.23.3 - sentencepiece==0.1.99 - torch==2.0.1 - transformers==4.30.2 system_packages: - wget - curl custom_setup_script: "script.sh" ``` ## Packaging the custom model Before adding the model to the platform, package all required files into a single ZIP file. Place the following in one folder: * `model.py` * `config.yaml` * Any additional scripts or assets (e.g. `script.sh`) Create a **ZIP file** containing the directory contents. Upload the ZIP to one of the supported model sources: * **AWS S3** * **GCP GCS** * **Public URL** Upload your trained model to **AWS S3** or **GCP GCS**, share the [access credentials](model-suite/integrations/secrets), and the platform will compile and prepare it for deployment. Models built to your specifications are integrated into the platform. ## Adding the custom model to the platform In the UI, point the platform to your ZIP, choose **Custom Pipeline** as the model type, and add the model. The platform then unpacks the archive and loads your model. On the Simplismart platform, provide your ZIP file, choose **Custom Pipeline** as the model type, and add the model. The platform then unpacks the archive and loads your model. Go to [**My Models**](https://app.simplismart.ai/my-models) and click **Add a Model** (top-right). My models * **Model name**: A name for your model. * **Model source**: Hugging Face, AWS S3, GCP GCS, or Public URL (use the source where you uploaded the ZIP). * **Model path**: Path to the ZIP file (e.g. S3 URI, GCS URI, or public URL). * If using AWS or GCP, select the linked [**Cloud credentials**](/model-suite/integrations/secrets). Add model details form Under **Model Class**, choose **Custom Pipeline** (or **Custom Model**). Choose **Simplismart Cloud** or **Bring Your Own Cloud**. Select **Accelerator type** and machine type based on your model size and compute requirements. Use the **Pipeline Config Editor** or **Extra Params** to tune deployment. For custom models, set `type` to `"custom"`. See the table and example below. Click **Add Model** to start compilation. The platform unpacks the archive, sets up the environment, and loads the model. ### Extra parameters (optional) Based on your model pipeline, you can add extra parameters in JSON format under **Extra Params**. | Field | Type | Default | Description | | -------------------- | ------ | ---------- | ----------------------------------------------------------------------------- | | `workers_per_device` | Int | `1` | Parallel workers per device (higher can improve inference speed). | | `device` | string | `cpu` | `"cpu"` or `"cuda"`. | | `endpoint` | string | `/predict` | URL path for inference requests. | | `type` | string | (required) | Use `"custom"` for custom models; other values: `"whisper"`, `"llm"`, `"sd"`. | For custom models, set `type` to `"custom"` in the pipeline configuration. Example: ```json theme={null} { "type": "custom", "extra_params": { "workers_per_device": 2, "device": "cuda", "endpoint": "/predict" } } ``` *** ## Next steps Once the model is compiled, see [Deploy a custom model](/model-suite/deployments/deploy-a-custom-model) to deploy your custom model. # Create a Cluster Source: https://docs.simplismart.ai/model-suite/clusters Learn how to create a cluster on your own cloud using Simplismart, configure compute resources, and prepare for model deployments. Through the [**Clusters**](https://app.simplismart.ai/clusters) section, you can efficiently manage your compute resources, allowing for optimal allocation and monitoring of your computational power. ## **Enter Basic Details** * **Name**: Provide a name for your cluster. This name will become your cluster slug. * **Cloud Account**: Select the cloud account where the cluster will be created. * **Region**: Choose the region for your cluster. * **Hosted Zone**: Select the appropriate hosted zone. After entering all the necessary details, click on the **Create Cluster** button to create the cluster. Alt Text *** ## Node Group Addition * When you see the **Success** status, it means your cluster including the **Auxiliary node pool** is fully provisioned, allowing you to move forward with creating a **Node Group**. Alt Text * Click on the **Add Node Group** button on the top-right. * **Name**: Provide a descriptive name for your node group. * **Accelerator**: Select the appropriate accelerator for your workloads. * **Capacity Type:** You can choose the node to be a **On Demand** instance or a **Spot instance**. * **Machine Type**: Choose the machine size that meets your performance requirements. * **Scaling Configuration**: Define the minimum and maximum number of nodes. Alt Text *** ## **Viewing Cluster Details** * The status will show `success` once the node group is successfully created and the cluster is now ready for use. * Once the **Node Group** is created, you can view its information by clicking on the **Overview** tab. * Click on the **Deploy** button to deploy the model to the node. You can find the deployment page [here](/model-suite/deployments/creating-a-deployment). Alt Text *** ## **Monitoring your cluster** You can track the overall usage and performance of your cluster to ensure optimal operation, by clicking on the **Monitor** tab. Alt Text * **Date Range**: Select the date range for which you want to view monitoring metrics. * **Pod Info**: View detailed information about the pods in your cluster. * **GPU and GRAM Utilization**: Monitor the usage and performance of GPUs and GRAM. Alt Text # Create AWS EKS Cluster on AWS Console Source: https://docs.simplismart.ai/model-suite/clusters/import-cluster/create-aws-eks-cluster Step-by-step guide to create an AWS EKS cluster and configure node groups on the AWS platform ## Overview This guide walks you through the process of creating an AWS EKS (Elastic Kubernetes Service) cluster and setting up node groups. Follow these steps if you need to create a new cluster before importing it into the Simplismart platform. ## Prerequisites Before starting, ensure: * You have access to the AWS Management Console and EKS service ## Create an AWS EKS Cluster Go to **Amazon Elastic Kubernetes Service** > **Clusters** > **Create cluster** Go to **AWS Management Console** → **EKS**. Click **Add Cluster** → **Create Cluster**. * Enter a **Cluster Name** (e.g., `test-cluster`) * Choose **Kubernetes version** (e.g., `1.33`) * Select an existing **IAM Role** or create a new one with EKS permissions * Click **Next** * Choose an existing VPC or create a new one * Select public and private subnets across different Availability Zones * Ensure proper routing and internet access for nodes Enable control plane logging for audit and debugging. Click **Next**, review, and click **Create**. ⏳ Cluster creation typically takes 15–20 minutes. For more details, check out the AWS documentation: [AWS EKS Cluster Creation Guide](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html) *** ## Create Node Groups Once your cluster is active, you'll need to create node groups to run your workloads. ### Create a Standard Node Group Once the cluster is active: * In the EKS cluster view, go to **Compute** → **Add Node Group** * Enter a **Node Group Name** (e.g., `worker-nodes`) * Select an **IAM Role** for nodes * Choose an **Instance Type** (e.g., `t3.large` or minimum `m6a.xlarge`) and **Node Count** * Label nodes with: ```bash theme={null} simplismart.ai/node-group-name: worker-nodes ``` * Click **Create** For Simplismart platform integration, minimum machine size is `m6a.xlarge` for auxiliary node groups where cluster tools will be installed. Create a new node group with GPU-enabled instances (e.g., `g5.xlarge`) following the same steps above, but: * Select a GPU-enabled instance type * Ensure NVIDIA GPU drivers are installed via DaemonSet after creation *** ## Next Steps After creating your cluster and node groups: Import your newly created AWS EKS cluster into the Simplismart platform Set up Kubernetes credentials to authenticate with your cluster ## Troubleshooting If you encounter issues during cluster creation, please refer to: * [AWS EKS Troubleshooting Guide](https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting.html) * Contact Simplismart support at [support@simplismart.ai](mailto:support@simplismart.ai) # E2E Cluster Import Prerequisites Source: https://docs.simplismart.ai/model-suite/clusters/import-cluster/e2e-cloud-prerquisites ## Importing a Kubernetes Cluster from E2E Cloud This page outlines the steps and requirements to create and import a Kubernetes cluster from the E2E cloud onto the Simplismart platform. Follow these steps to ensure your cluster is set up correctly and all necessary tooling is deployed. ## Basic Cluster Set-up & Requirements Before importing your cluster, please ensure that it meets the following prerequisites: ## Steps to Create a Kubernetes Cluster ### 1. Navigate to the Compute Dashboard * Go to the E2E Cloud dashboard. * Select the Kubernetes section. *** ### 2. Create a New Kubernetes Cluster * Click on **Get Started**. * Enter a cluster name (e.g., `test-cluster`). * Select an existing **VPC** or create a new one if necessary. title *** ### 3. Add a Node Pool * Click on **Add Node Pool**. * Choose a node configuration (e.g., 3.32 GB, 1516, 32 RAM). * **Node Labels:** Label every node in the cluster to indicate its node group. Use the following format for each node’s label: ```yaml theme={null} simplismart.ai/node-group-name: ``` * Replace `` with the appropriate group name for the node. title * Specify the worker count (number of nodes required). * (Optional) Add a GPU pool if required (e.g., GDC A100). title For clusters that include **GPU nodes**, verify that the *NVIDIA daemon set* is deployed. This ensures proper management and scheduling of GPU resources. *** ### **4. Create the Cluster** * Click on **Create Cluster**. * The cluster creation process generally takes approximately **15-20 minutes**. title The cluster should include **at least one node pool** with a minimum of **one CPU**. ### Configuring Storage (Optional) If volume mount is needed, please follow the below steps: * While the cluster is being created, navigate to the **Storage** section. * Go to **Scalable File System (SFS)**. * Click **Create**. * Select the same **VPC** as the cluster. * Specify the required **storage size**. * Click **Create SFS**. title * Grant `[all]` access permissions. * Retrieve the **IP address** of the volume and keep it handy. *** ### Post-Creation Steps Once the cluster is created, add **Load Balancer IP (LBIP)** and **Persistent Volume Claim (PVC)**. * Create a Persistent Volume Claim (PVC) in your cluster: title * Update the storage class to Default by adding this annotation to the storage class manifest: ```yaml theme={null} storageclass.kubernetes.io/is-default-class":"true" ``` * Or alternatively, run this command to update the storage class: ```bash theme={null} kubectl patch storageclass -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' ``` * Allocate a Load Balancer (LB) IP pool to manage inbound traffic: title Once you have completed the above steps and your cluster has been successfully created, please follow these steps to [import your cluster](/model-suite/clusters/import-cluster/import-kubernetes-cluster). # Import AWS EKS Cluster to Simplismart Platform Source: https://docs.simplismart.ai/model-suite/clusters/import-cluster/import-aws-cluster Learn how to import an existing AWS EKS cluster into the Simplismart platform and configure its settings ## Overview This guide walks you through the process of importing an existing AWS EKS cluster into the Simplismart platform. By importing your cluster, you can leverage Simplismart's deployment, monitoring, and scaling capabilities with your existing infrastructure. **Coming Soon**: Warmpool functionality and automatic node group creation are currently only available for clusters created directly on the Simplismart platform. In an upcoming release, these features will be extended to imported clusters. You'll be able to provide limited IAM access, and Simplismart will manage warmpool and node group creation automatically, just like it does for Simplismart-created clusters. ## Prerequisites Before importing your cluster, ensure you have the following: ### EKS Cluster You need an AWS EKS cluster. If you don't already have one, follow the [Create AWS EKS Cluster](/model-suite/clusters/import-cluster/create-aws-eks-cluster) guide. ### Node Group Your cluster must have at least one node group with minimum 1 vCPU and the required label (see below). For auxiliary node groups where Simplismart cluster tools will be installed, use a minimum machine size of `m6a.xlarge`. #### Node Group Label All nodes in your cluster must have the required label. To add a label, navigate to `Amazon Elastic Kubernetes Service` > `Clusters` > `` > `` > `edit` and add a label in the format given below: ```bash theme={null} simplismart.ai/node-group-name: ``` Node Group Label This label is essential for the Simplismart platform to identify and manage your nodes correctly. If you are importing a cluster, you need to create a node group yourself on AWS. Only then you can register it on the Simplismart platform. See the [Create AWS EKS Cluster guide](/model-suite/clusters/import-cluster/create-aws-eks-cluster#create-node-groups) for detailed instructions on creating node groups with the required label. ### Kubernetes Credentials Configure your Kubernetes credentials as a secret in Simplismart to authenticate with your AWS EKS Cluster. AWS supports both token-based and certificate-based authentication. This guide covers token-based authentication using three different approaches. **Important Timing Constraint**: AWS EKS tokens have a 15-minute validity period. Since cluster import on Simplismart takes approximately 10 minutes, you must generate the kubeconfig credentials and start the import process within 5 minutes. If the token expires during import, the cluster import will fail with an authentication error. If this happens, simply regenerate the credentials and retry the import. This method uses the AWS CLI to generate kubeconfig credentials programmatically. Ensure you have AWS CLI installed and configured with appropriate credentials: ```bash theme={null} aws configure ``` Enter your AWS Access Key ID, Secret Access Key, default region, and output format when prompted. Update your local kubeconfig file to include the EKS cluster credentials: ```bash theme={null} aws eks update-kubeconfig --region --name ``` Replace `` with your AWS region (e.g., `us-east-1`) and `` with your EKS cluster name. This command will add the cluster context to your `~/.kube/config` file. Test the connection to your cluster: ```bash theme={null} kubectl get nodes ``` If successful, you should see a list of nodes in your cluster. Extract the kubeconfig content to add as a secret in Simplismart: ```bash theme={null} cat ~/.kube/config ``` Copy the entire output. You'll use this in the next step. Go to the [Secrets](/model-suite/integrations/secrets#json-format-for-adding-secrets) section in Simplismart and: * Click on the **Kubernetes** tab * Add a new secret with the kubeconfig content you copied * Follow the detailed configuration instructions in the Secrets documentation This method uses the AWS Management Console to retrieve cluster connection information and build the kubeconfig manually. * Open the **AWS Management Console** * Go to **Amazon EKS** service * Select your cluster from the list In the cluster details page, note down the following: * **API server endpoint**: Found in the **Configuration** tab * **Certificate authority data**: Found in the **Configuration** tab (click "Show" to reveal) Copy both values—you'll need them to construct the kubeconfig. Create a kubeconfig file with the following structure: ```yaml theme={null} apiVersion: v1 kind: Config clusters: - cluster: certificate-authority-data: server: name: contexts: - context: cluster: user: name: current-context: users: - name: user: exec: apiVersion: client.authentication.k8s.io/v1beta1 command: aws args: - eks - get-token - --cluster-name - - --region - ``` Replace the placeholders: * ``: Certificate authority data from Step 2 * ``: API server endpoint from Step 2 * ``: Your EKS cluster name * ``: Your AWS region (e.g., `us-east-1`) Save the kubeconfig file and test the connection: ```bash theme={null} export KUBECONFIG=/path/to/your/kubeconfig kubectl get nodes ``` If successful, you should see a list of nodes in your cluster. Go to the [Secrets](/model-suite/integrations/secrets#json-format-for-adding-secrets) section in Simplismart and: * Click on the **Kubernetes** tab * Add a new secret with the kubeconfig content you created * Follow the detailed configuration instructions in the Secrets documentation This method uses eksctl, a simple CLI tool for creating and managing EKS clusters. If you haven't installed eksctl yet, follow the [eksctl installation guide](https://eksctl.io/installation/). For macOS: ```bash theme={null} brew tap weaveworks/tap brew install weaveworks/tap/eksctl ``` For Linux: ```bash theme={null} curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp sudo mv /tmp/eksctl /usr/local/bin ``` Use eksctl to write the cluster credentials to your kubeconfig: ```bash theme={null} eksctl utils write-kubeconfig --cluster --region ``` Replace `` with your EKS cluster name and `` with your AWS region. This will automatically update your `~/.kube/config` file. Test the connection to your cluster: ```bash theme={null} kubectl get nodes ``` If successful, you should see a list of nodes in your cluster. Extract the kubeconfig content: ```bash theme={null} cat ~/.kube/config ``` Copy the entire output. Go to the [Secrets](/model-suite/integrations/secrets#json-format-for-adding-secrets) section in Simplismart and: * Click on the **Kubernetes** tab * Add a new secret with the kubeconfig content you copied * Follow the detailed configuration instructions in the Secrets documentation For production environments, it's recommended to create a dedicated IAM role with minimal required permissions for Simplismart access, rather than using admin credentials. *** ## Import Cluster Basic details Provide the following information about your cluster: | Field | Description | | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | **Cluster Name** | A unique name to identify your cluster in the Simplismart platform | | **Cloud Provider** | Select AWS as your cloud provider | | **Secret** | Select a secret to authenticate with AWS. See the [Secrets](/model-suite/integrations/secrets) section for configuration details | | **Region** | The AWS region where your EKS cluster is deployed | | **Hosted Zone** | The DNS hosted zone for your cluster. See [Hosted Zone](/model-suite/integrations/hosted-zone) documentation | | **Environment** | Select your environment type: Production, Development, Demo, or Custom | | **Custom Environment Description** | Provide a description if you selected "Custom" environment | Select the tools to install on your cluster during the import process. All tools shown below are mandatory for core cluster functionality. Cluster Tools ### Scaling Tools Collects basic pod and node resource usage (CPU/memory) to enable Kubernetes Horizontal Pod Autoscaling and efficient resource management. Automatically adds or removes nodes in your cluster based on workload demand, optimizing costs by scaling infrastructure dynamically. The [Prometheus Adapter](https://github.com/kubernetes-sigs/prometheus-adapter) is a fallback in case we want to switch to use HPA and scale based on number of requests or custom metrics. Our main event-driven autoscaler that scales applications based on workload activity (e.g., queue length, message count), improving performance and reducing costs during idle periods. It is required for scale to 0. ### Observability Tools Long-term metrics storage system that provides reliable, scalable storage for time-series data, enabling historical analysis and trend monitoring. Centralized log aggregation system that collects and stores logs from all applications, making it easy to search, filter, and debug issues across your cluster. Log shipping agent that collects logs from your applications and forwards them to Loki, making all logs searchable in the Simplismart platform. Complete monitoring solution providing dashboards, metrics collection, and alerting capabilities to monitor workload health and system performance. GPU monitoring tool that tracks NVIDIA GPU utilization, temperature, and health metrics, essential for optimizing AI/ML workloads. Simplismart's internal monitoring agent that collects operational metrics and system health data like disk pressure, node readiness, degraded pod, etc for platform integration. Configure your node groups to be managed by the Simplismart platform. Node group configuration allows you to manage your cluster resources based on workload types, hardware requirements, and scaling policies. Using the given configuration, Simplismart is able to intelligently manage the nodes and effectively distribute the resources. Register Node Group ### Node Group Label Provide the node group label you configured earlier in the [Prerequisites](#node-group-label) section. This label allows Simplismart to identify and manage your node group. You can register multiple node groups with different configurations to use your cluster resources effectively on the Simplismart platform. ### Node Group Configuration Enable **Use Node Group Configuration** to have Simplismart automatically allocates the resource during the deployment. If not selected, you need to provide node configuration every time you deploy. | Field | Description | | --------------------- | -------------------------------------------------------------------- | | **Accelerator Type** | Select either **CPU** or **GPU** based on your workload requirements | | **Accelerator Count** | Number of accelerators (GPUs) per node | | **Min Node Count** | Minimum number of nodes to maintain in this node group | | **Max Node Count** | Maximum number of nodes allowed in this node group for autoscaling | | **CPU** | Number of CPU cores per node | | **Memory** | Memory allocation in GB per node | When this option is enabled, Simplismart will manage the specified node group in your EKS cluster based on the resource configuration provided. The node group must already exist in your cluster. ### Auxiliary Node Group Enable **Mark as Auxiliary** if this node group should be reserved for supporting workloads rather than primary AI/ML operations. **Common use cases for auxiliary node groups:** * Monitoring and logging services (refer to Step 2 for details) * Internal tooling and platform services Add custom tags to manage and identify your cluster for billing, cost allocation, and resource management. Tags will be auto-populated in the [billing](/model-suite/settings/billing) section as well as the event center for better tracking and visualization. Moreover, you can create environment tags for Testing, Staging or Production based on your requirements. Environment specific tags will help you in billing and how much each environment is generating the bills. Click **Add Tag** and provide key-value pairs as needed. *** ## Post-Import Steps After successfully importing your cluster, follow these steps to get started: Check that the cluster status shows as "Success" in the Simplismart dashboard. Verify all cluster tools are running correctly. Start deploying AI/ML models to your imported cluster using the Simplismart platform. Use the observability tools (Grafana, Prometheus, Loki) to monitor cluster health and performance metrics. Deploy custom Docker containers to your cluster for custom models. *** ## Deployment Capabilities ### Current Support Imported clusters currently support **container-based deployments**. You can deploy Docker/Depot containers with full integration into the Simplismart platform, including: * Monitoring via the observability stack (Grafana, Prometheus, Loki) * Auto-scaling via the scalability stack (Metrics Server, Cluster Autoscaler, KEDA) * Resource management and optimization ### Coming Soon In future releases, imported clusters will support: Full support for custom Helm chart deployments with complete lifecycle management. Your Helm deployments will be: * **Fully monitored** via the Simplismart observability stack * **Fully scalable** via the Simplismart scalability stack Once warmpool support is enabled for imported clusters, you'll benefit from: * **Rapid auto-scaling** for faster response to workload demands * **Reduced cold-start times** through pre-warmed resources * **Same capabilities** as clusters created directly on the Simplismart platform This will enable the same rapid scaling performance you get with Simplismart-created clusters. *** ## Troubleshooting If you encounter issues during the cluster import process, please feel free to reach out to our support team at [support@simplismart.ai](mailto:support@simplismart.ai). # Import a Kubernetes Cluster Source: https://docs.simplismart.ai/model-suite/clusters/import-cluster/import-kubernetes-cluster Learn how to import your existing Kubernetes cluster into the Simplismart platform This guide walks you through the process of importing your existing Kubernetes cluster into Simplismart. Before proceeding, ensure your cluster meets the [cloud-specific prerequisites](/model-suite/clusters/import-cluster/e2e-cloud-prerquisites). ## Prerequisites Before importing your cluster, make sure you have: * A Kubernetes cluster that meets the cloud-specific requirements * Administrative access to create secrets and configure node groups * Your cluster credentials and configuration details ## Step 1: Create a Kubernetes Secret To securely connect Simplismart to your cluster, you'll need to create a Kubernetes secret: 1. Navigate to the [Secrets](/model-suite/integrations/secrets) tab in the sidebar 2. Click **Create Secret** and select the type as Kubernetes. 3. Enter your cluster credentials and configuration Ensure your secrets are formatted correctly. Refer to the [Cloud specific format guide](/model-suite/integrations/secrets#json-format-for-adding-secrets) for proper structure. ## Step 2: Import the Cluster Now you're ready to import your cluster into Simplismart: 1. Navigate to the **Clusters** tab 2. Click **Import Cluster** 3. Fill in the required information: ### Basic Configuration * **Cluster Name**: Provide a descriptive name for your cluster * **Cloud Provider**: Select your cloud provider (AWS, GCP, Azure, OCI, etc.) * **Region**: Specify the region where your cluster is hosted * **Secret**: Choose the Kubernetes secret you created in Step 1 * **Environment**: Select your deployment environment: * `Development` * `Production` * `Demo Environment` * `Others` Import Cluster Configuration ## Step 3: Configure Cluster Tools Select the appropriate tools based on your use case: * **Training Tools**: For model training jobs * **Observability Tools**: For monitoring and logging * **Async Tools**: For asynchronous task management * **Scaling Tools**: For auto-scaling capabilities Cluster Tools Selection ## Step 4: Register Node Groups Configure your node groups to define the compute resources available for deployments. ### Node Group Labels Provide the labels that identify your node groups: * Use the format: `simplismart.ai/node-group:` * These labels must match the labels applied to your nodes in the existing cluster * Labels help Simplismart understand how to organize and schedule workloads ### Node Configuration Specify the following details for each node group: **GPU Configuration** * **GPU Type**: Select the GPU model available on your node group (e.g., A100, V100, T4) * **GPU Count**: Number of GPUs attached to each node **Scaling Configuration** * **Minimum Nodes**: Minimum number of nodes in the group * **Maximum Nodes**: Maximum number of nodes for auto-scaling **Resource Allocation** * **vCPU**: Number of virtual CPUs available per node * **Memory**: Amount of RAM available per node (in GB) The vCPU and memory values are critical as they determine the maximum resources that can be allocated during deployments on your cluster. Ensure these values accurately reflect your node specifications. Node Group Configuration ## Step 5: Automated Dependency Deployment Once you complete the import process, Simplismart automatically deploys essential infrastructure components to your cluster: * **Deployment Controller**: Manages model deployment lifecycle * **Cert Manager**: Handles TLS certificates * **NGINX Ingress Controller**: Routes external traffic to your deployments * **Additional Dependencies**: Other tooling required for platform operations ## Frequently Asked Questions The cluster import will fail if your secret is improperly formatted. Follow the [cloud-specific format guide](/model-suite/integrations/secrets#json-format-for-adding-secrets) and ensure all required fields (tokens, certificates, keys) match the expected structure for your cloud provider. To delete a cluster, contact [support@simplismart.ai](mailto:support@simplismart.ai). During deletion: * All tools installed by Simplismart will be removed from your cluster * Your cluster will continue running on your original cloud platform * Only the Simplismart integration components are removed Simplismart will not be able to detect your node groups. Each node must have the label in the following format: ``` simplismart.ai/node-group: ``` Apply these labels to your nodes before importing the cluster to ensure proper node group detection. No, node group configurations cannot be modified after import. The following settings are locked during the import process: * GPU type * GPU count per node * vCPU allocation * Memory allocation * Scaling limits (min/max nodes) If you need to change these settings, you'll need to reimport the cluster. Mismatched configurations can cause serious issues: * Deployment scheduling failures * Incorrect resource allocation * Under or over-provisioning of resources Always enter the exact specifications of your cluster nodes to ensure accurate resource management and scheduling. Simplismart automatically deploys the following essential components to your cluster: * **Deployment Controller**: Manages model deployment lifecycle * **Cert Manager**: Handles TLS certificate management * **NGINX Ingress Controller**: Routes external traffic to deployments * **Additional infrastructure components**: Required for platform operations These tools are mandatory and cannot be skipped during the import process. The automated deployment process typically takes 5-10 minutes. Monitor the progress in the Clusters dashboard. Simplismart installs its own versions of these tools. If your cluster has conflicting versions, you may experience issues with: * Traffic routing * TLS certificate management * Deployment orchestration **Best Practice**: Isolate Simplismart's tools in separate namespaces to prevent conflicts with your existing infrastructure. For each node group, you must specify: * **GPU Type**: The GPU model installed (e.g., A100, V100, T4, A10, H100) * **GPU Count**: Number of GPUs attached to each node This information determines which models can be scheduled on your cluster and ensures optimal resource allocation. Yes, you can import a CPU-only cluster. However: * GPU-accelerated model deployments will not be possible * You can only deploy CPU-based models * GPU node groups can only be added during the initial import, not afterward If you plan to deploy GPU models in the future, ensure you include GPU node groups during the import process. Common mistakes to avoid: * **Mixed authentication methods**: Don't combine token and certificate-based authentication in secrets * **Incorrect node specifications**: Ensure vCPU, memory, and GPU counts match your actual nodes * **Missing node labels**: All nodes must have the required `simplismart.ai/node-group` labels * **Expired credentials**: Verify your cluster credentials are current and valid * **Mismatched regions**: Ensure the region specified matches your cluster's actual location Once the import is successful, your cluster becomes fully operational in Simplismart. You can: * [Deploy models](/model-suite/deployments/deploy-on-an-imported-cluster) to your imported cluster * Monitor cluster health and resource utilization * View deployment logs and metrics * Scale your deployments based on configured node groups Note that node group configurations are locked after import and cannot be modified later. ## Next Steps After successfully importing your cluster, you can: * [Deploy models](/model-suite/deployments/deploy-on-an-imported-cluster) on your imported cluster * Configure additional node groups as needed * Set up monitoring and observability tools * Manage cluster resources and scaling policies # Autoscaling Source: https://docs.simplismart.ai/model-suite/deployments/autoscaling How Simplismart automatically grows and shrinks your deployment's pods to match demand Every deployment runs on a number of pods. Each pod is a running copy of your model, ready to serve requests. Too few pods, and requests queue up while response times climb. Too many, and you're paying for GPU capacity that sits idle. Autoscaling keeps this balanced automatically — Simplismart monitors your traffic and adds or removes pods accordingly. **If you're just getting started:** you don't need to read this whole page to configure autoscaling. Choose a metric and a threshold on the deployment form and leave the rest at their defaults — see [Picking a starting point](#picking-a-starting-point) for sensible defaults by workload type. The rest of this page explains the reasoning behind that behavior, for when you need finer control. This page walks through how autoscaling decides to add or remove pods. For the step-by-step form fields, see [Creating a Deployment](/model-suite/deployments/creating-a-deployment). Pick the metric, threshold, and min/max range that drive scaling. Guarantee capacity for traffic patterns you already know about. Control how fast pods are added or removed once a threshold is crossed. Drop idle deployments to 0 pods and stop paying for them. ## How it decides At regular intervals, Simplismart checks one number — your chosen metric — against a target you've set, called the **threshold**. If the number is above the threshold, it adds pods. If it's below, it removes pods. At the threshold, it leaves things as they are. Two settings bound this decision on either side: * **Min pods** — the floor. Your deployment never drops below this, even at zero traffic, unless Scale to Zero is enabled (covered below). * **Max pods** — the ceiling, capped by your account quota. Your deployment never grows past this, regardless of demand. The metric determines the direction; min and max determine how far it's allowed to go. ```mermaid theme={null} flowchart LR A["Measure metric\n(Memory · Latency · Throughput · Concurrency)"] --> B{"Above or below\nthreshold?"} B -- "Above" --> C["Candidate: scale up"] B -- "Below" --> D["Candidate: scale down"] C --> E["Autoscaling Policy\nhow fast, how much"] D --> E E --> F["Min / Max pods\nhard limits"] F --> G["Pod count applied"] ``` ### Choosing a metric The metric is simply the number autoscaling watches. It falls into one of two categories: * **Resource metrics** describe the pod itself — how hard the CPU or GPU is working, how much memory it's using. * **Request-shape metrics** describe your traffic — how many requests are arriving, how quickly they're answered. If you're unsure where to start, use **Concurrency** as your primary metric with **GPU Utilization** as a backup. This combination works well for most model deployments. **Resource metrics** | Metric | What it measures | Good fit for... | | -------------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------- | | **CPU Utilization** | Average CPU usage across all pods | Workloads that lean on CPU, like pre/post-processing or tokenization | | **GPU Utilization** | Average GPU compute usage across all pods | Inference that's GPU-bound, where compute (not memory) is the bottleneck | | **Memory Usage** | Average system memory across all pods | Catching memory leaks or workloads whose memory use keeps growing | | **GPU Memory Usage** | Average GPU memory (VRAM) usage across all pods | Getting an early warning before pods run out of VRAM (big batches, long-context LLMs) | **Request-shape metrics** | Metric | What it measures | Good fit for... | | --------------- | --------------------------------------------- | ----------------------------------------------------------------------------- | | **Concurrency** | How many requests are being handled right now | Bursty traffic made of many short requests | | **Latency** | How long each response takes | Chat or voice, where you'd rather add pods early than let responses slow down | | **Throughput** | Requests handled per second | Steadier traffic, where total volume matters more than any one request | Some model runtimes expose a few extra specialized metrics too (like queue depth for async workers). Available metrics can vary by model and container type — see [Scaling Parameters](/model-suite/deployments/creating-a-deployment#scaling-parameters) for the exact list on your deployment. Whichever metric you choose, the **threshold** is the value that determines the direction: above it, pods are added; below it, pods are removed. ## Try it: the autoscale console Scaling up and scaling down are deliberately asymmetric. Simplismart adds pods quickly when demand rises, so requests don't queue. It removes pods more cautiously, so a brief lull doesn't remove a pod you need again moments later. The console below runs a live simulation rather than a static illustration. Choose a traffic pattern, adjust the sliders, and watch the pod count respond in real time — no deployment required. Two things worth exploring once it's loaded: * The **Scaling ledger** logs every scale event as it happens. Select the **Flapping** pattern with a short scale-down period, and the ledger will flag (in rose) a scale-down that occurs sooner than expected. This is a real effect, explained in the next section. * Under **Advanced settings**, enable **Schedule window**. It guarantees a minimum pod count during a time window you define, regardless of what the traffic metric shows — you'll see it hold that floor (shaded on the chart) and return control to standard scaling once the window ends.