Llama 3.1
llama.meta.com
The open source AI model you can fine-tune, distill and deploy anywhere. Our latest models are available in 8B, 70B, and 405B variants.

Big day for people who use AI locally. According to benchmarks this is a big step forward to free, small LLMs.

    • just another dev@lemmy.my-box.devEnglish
      1·
      4 months ago

      Oof - not on my 12gb 3060 it doesn’t :/ Even at 48k context and the Q4_K quantization, it’s ollama its doing a lot of offloading to the cpu. What kind of hardware are you running it on?

      • brucethemoose@lemmy.worldEnglish
        2·
        4 months ago

        A 3090.

        But it should be fine on a 3060, with zero offloading.

        Dump ollama for long context. Grab a 5-6bpw exl2 quantization and load it with Q4 or Q6 cache depending on how much context you want. I personally use EXUI, but text-gen-webui and tabbyapi (with some other frontend) will also load them.