For about half a year I stuck with using 7B models and got a strong 4 bit quantisation on them, because I had very bad experiences with an old qwen 0.5B model.

But recently I tried running a smaller model like llama3.2 3B with 8bit quant and qwen2.5-1.5B-coder on full 16bit floating point quants, and those performed super good aswell on my 6GB VRAM gpu (gtx1060).

So now I am wondering: Should I pull strong quants of big models, or low quants/raw 16bit fp versions of smaller models?

What are your experiences with strong quants? I saw a video by that technovangelist guy on youtube and he said that sometimes even 2bit quants can be perfectly fine.

  • SGforceEnglish
    5·
    19 hours ago

    The technology for quantisation has improved a lot this past year making very small quants viable for some uses. I think the general consensus is that an 8bit quant will be nearly identical to a full model. Though a 6bit quant can feel so close that you may not even notice any loss of quality.

    Going smaller than that is where the real trade off occurs. 2-3 bit quants of much larger models can absolutely surprise you, though they will probably be inconsistent.

    So it comes down to the task you’re trying to accomplish. If it’s programming related, 6bit and up for consistency with whatever the largest coding model you can fit. If it’s creative writing or something a much lower quant with a larger model is the way to go in my opinion.

    • Smorty [she/her]@lemmy.blahaj.zoneOPEnglish
      2·
      18 hours ago

      Hmm, so what you’re saying is that for creative generations one should use big parameter models with strong quants but when good structure is required, like with coding and JSON output, we want to use a large quant of a model which actually fits into our VRAM?

      I’m currently testing JSON output, so I guess a small Qwen model it is! (they advertised good JSON generations)

      Does the difference between fp8 and fp16 influence the structure strongly, or are fp8 models fine for structured content?

      • SGforceEnglish
        1·
        17 hours ago

        fp8 would probably be fine, though the method used to make the quant would greatly influence that.

        I don’t know exactly how Ollama works but a more ideal model I would think would be one of these quants

        https://huggingface.co/bartowski/Qwen2.5-Coder-1.5B-Instruct-GGUF

        A GGUF model would also allow some overflow into system ram if ollama has that capability like some other inference backends.

        • Smorty [she/her]@lemmy.blahaj.zoneOPEnglish
          2·
          17 hours ago

          Ollama does indeed have the ability to share the memory between VRAM and RAM, but I always assumed it wouldn’t make sense, since it would massively slow down the generation.

          I think ollama already uses GGUF, since that is how you import the model from HF to ollama anyway, you gotta use the *.GGUF file.

          As someone who has experience with shader development in glsl, I know very well that communication between the GPU and CPU is super slow, and sending data from the GPU to the CPU is a pretty heavy task. So I just assumed it wouldn’t make any sense. I will try a full 7B model (fp16) model now using my 32GB of normal RAM to check out the speed. I’ll edit this comment once I’m done and share results