> ## Documentation Index > Fetch the complete documentation index at: https://docs.simplismart.ai/llms.txt > Use this file to discover all available pages before exploring further. # Model Optimization Troubleshooting Guide > Refer to this guide to ensure proper configuration of infrastructure settings for model compilation and inference. It also helps identify and prevent common issues related to hardware incompatibilities and performance bottlenecks. ## Required Infrastructure Configuration | **Parameter** | **Value** | **Notes** | | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Cluster** | You can either use the Simplismart Infrastructure or select your cluster where you want the model to be compiled. If you haven’t created a cluster yet, you can proceed with the Simplismart Infrastructure option. | - | | **Region** | If you have selected Simplismart Infrastructure as your cluster option, the region should be set to Global. If you are using your **own cluster**, you will need to select a region where you have sufficient quota available for the machine that will be used to compile the model. | Ensures access to globally available compute resources | | **Instance Type** | Model-dependent | Select a type based on model size (see below)
✅ Always choose the (any) variant for better availability (If you have selected Simplismart Infrastructure as your cluster option) | | **Machine Type** | Default | Always choose Default. (If you have selected Simplismart Infrastructure as your cluster option) | ## What is Tensor Parallelism (TP)? **Tensor Parallelism** allows a model's computation to be split across multiple GPUs or devices, enabling: * Faster inference for large models * Support for models too large to fit on a single device tensor\_parallel\_size controls how many devices the computation is split across: * 1 = **No tensor parallelism** (single GPU) * 2, 4, 8, etc. = Enable TP across multiple GPUs You must have **at least as many GPUs** as the value set in **tensor\_parallel\_size**. ## **Model Mode Types and Compatibility** Different model versions require specific *modes* to function correctly. When selecting models from Hugging Face, you'll typically find two variants: * Base models (e.g., *meta-llama/Meta-Llama-3-8B*) - designed for *completion* tasks. * Instruct models (e.g., *meta-llama/Meta-Llama-3-8B-Instruct*) - optimized for *conversational/chat interactions.* **Configuration Rule**: The pipeline mode must match the model type, otherwise compilation will fail. For instance, using a base completion model like **meta-llama/Meta-Llama-3-8B** with **chat mode** will cause errors - you must set the mode to **completion** instead. **Pipeline Mode Settings**: Configure the mode in your pipeline based on your intended use case: * Set mode to **embedding** when compiling embedding models * Set mode to **chat** when compiling conversational/instruct models * Set mode to **completion** when compiling base/completion models The key is ensuring alignment between your model choice and the corresponding pipeline mode configuration. ## Instance Type Selection Guide Use the table below to guide instance and TP configuration based on your model size: | **Model** | **Recommended Instance Type** | **Suggested TP** | **Notes** | | --------------------------- | ----------------------------- | ---------------- | --------------------------------------------- | | **Gemma 2B** | L40s (any) | 1 | Lightweight model; fits on a single GPU | | **LLaMA 3B** | L40s (any) | 1 | Also fits on single GPU with headroom | | **Gemma 7B** / **LLaMA 8B** | H100 (any) | 2–4 | Benefits from multi-GPU setup | | **LLaMA 70B** | H100 (any) | 2–4+ | Requires high TP and multi-GPU infrastructure | ## Common Issues & Fixes | **Issue** | **Cause** | **Fix** | | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | 🚫 **Job stuck in queue or not scheduled (machine not available)** | Instance type too specific or unavailable. | ✅ Use (any) variant of **instance type** | | 🚫 **Out of memory / crashes** | Model too large for single GPU. | ✅ Increase **tensor\_parallel\_size** or upgrade instance type | | 🚫 **TP value ignored or job fails to start** | TP set higher than available GPUs. | ✅ Ensure instance has **≥ GPUs** than **tensor\_parallel\_size** | | **🐢** **Slow inference** | Underutilised hardware or no parallelism. | ✅ Tune **tensor\_parallel\_size** and use multi-**GPU instances** | | **🚫** **Unsupported model type**


`Error message: The given model path is invalid` | The model which you trying to compile is currently not supported through the Simplismart Platform. | You can contact [support@simplismart.ai](mailto:support@simplismart.ai) we will check the feasibility and add support for the model. | | **Mode Types in Pipeline config Model compilation failing even if the correct model and other parameters are selected.** | The selected model *mode* in the pipeline config may be incorrect. | **Pipeline Mode Settings**: Configure the mode in your pipeline based on your intended use case:

- Set mode to **embedding** when compiling embedding models.

- Set mode to **chat** when compiling conversational/instruct models.

- Set mode to **completion** when compiling base/completion models. The key is ensuring alignment between your model choice and the corresponding pipeline mode configuration. | | **Machine Clean-up failed**


`Error Message: Cleanup Failure: Exception occured while cleaning up : Error cleaning up Azure resource group` | You can contact [support@simplismart.ai](mailto:support@simplismart.ai) we will check the reason for the failure. | You can contact [support@simplismart.ai](mailto:support@simplismart.ai) we will check the feasibility and add support for the model. | ##  FAQs 1. **Can I edit the model name later?** No, renaming a model after it's been added and compiled is not supported. The model name must be set during the initial setup. *** 2. **What options are available for model sources?** You can choose HuggingFace, where the base model is downloaded directly, or use AWS S3, GCS, or DockerHub by providing the appropriate path (S3 URL, GCS URL, or DockerHub registry link) along with the required credentials so we can retrieve your custom model. *** 3. **Do I need authentication keys for external sources?** Yes, for S3, GCS, and DockerHub, you must add your authentication keys on the Secrets page in the Simplismart platform and use those credentials during the compilation process. *** 4. **Why am I getting a “The given model path is invalid” error? How do I verify if my model path is valid?** While the platform supports most LLM models, this error can occur if a particular model type isn't supported yet. If you encounter this issue, please contact us at [support@simplismart.ai](mailto:support@simplismart.ai), and we’ll work on enabling support for your model. *** 5. **How do I link my AWS/GCP/Azure account?** You will have to your cloud account details in the integrations section. Refer to this [doc](https://docs.simplismart.ai/model-suite/integrations/secrets) on BYOC. *** 6. **Can I use multiple cloud accounts?** Yes, the Simplismart platform supports adding and managing multiple cloud accounts. *** 7. **Does region selection affect latency or costs?** \ For model compilation jobs, the selected region typically doesn't have a major impact on latency or cost. If you're using your own cloud account, you can run the compilation in any region where you have available resource quotas. *** 8. **How do I choose the right accelerator for my model?**\ \ The appropriate accelerator depends largely on the size of your model. Larger models require higher-spec machines for **optimization** and **deployment**. *** 9. **Why is my selected accelerator not available in the region I picked?**\ \ Some high-end GPUs like A100s or H100s are not available in all regions across major cloud providers such as AWS, Azure, or GCP. As a result, those accelerator options may not appear in the region's dropdown list. *** 10. **What happens if I run out of quota for GPUs in my cloud account?**\ \ If your cloud account lacks sufficient GPU quota, the optimization job may fail to start or could get stuck partway through, leading to a failed optimization process. *** 11. **What does \_accelerator count \_mean?**\ \ Accelerator count refers to the number of GPU instances allocated for a job. For example, if you select H100 as the accelerator and set the count to 2, two H100 machines will be provisioned. This is especially important when the tensor parallelism (TP) value is greater than 1. *** 12. **How do I know which machine type to select?**\ \ Even within the same instance family (e.g., AWS EC2 G5), all instances typically share the same underlying GPU for example, the NVIDIA A10G. However, they differ in the number of vCPUs and amount of RAM. It's generally recommended to choose a medium or large instance size to ensure sufficient CPU and memory resources for your ML workload. *** 13. **What is quantization and why should I use it?** Quantization is the process of reducing the precision of the numbers used to represent a language model’s parameters (e.g., from 32-bit floating point to 8-bit integers) to make the model smaller and faster, with minimal loss in accuracy, helpful for running large language models (LLMs) efficiently *** 14. **Which quantization levels are supported?**\ \ We support FP16, FP8, and AWQ quantization. \ \ Note that FP8 is not supported on Ampere architecture GPUs like A100 and A10G, as these devices do not natively support FP8 precision. *** 15. **Does quantization affect accuracy?**\ \ Yes, quantization can result in a slight reduction in model accuracy. ***