I’ve been playing around with Ollama in a VM on my machine and it is really useful.

To get started I would start by making sure you have capable hardware. You will need recent hardware so that old computer you have laying around may not be enough. I created a VM on my laptop with KVM and gave it 8gb of ram and 12 cores.

Next, read the readme. You can find the Readme at the github repo

https://github.com/ollama/ollama

Once you run the install script you will need to download models. I would download Llama2, Mistral and LLava. As an example you can pull down llama2 with ollama pull llama2

Ollama models are available in the online repo. You can see all of them here: https://ollama.com/library

Once they are downloaded you need to setup openwebui. First, install docker. I am going to assume you already know how to do that. Once docker is installed pull and deploy open web UI with this command. Notice its a little different than the command in the open web UI docs. docker run -d --net=host -e OLLAMA_BASE_URL="http://localhost:11434 -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

Notice that the networking is shared with the host. This is needed for the connection. I also am setting the environment variable in order to point open web UI to ollama.

Once that’s done open up the host IP on port 8080 and create an account. Once that’s done you should be all set.

  • Scrubbles@poptalk.scrubbles.techEnglish
    7·
    8 months ago

    Personally I’ve really enjoyed text-generation-webui. It made it really easy to ramp up and learn. Very cool stuff you got though, I’ll probably be looking at a comparison between them!

  • mozz@mbin.grits.dev
    5·
    8 months ago

    Does it work out okay with 12 cores purely on CPU? About how fast is the interaction?

    I played around a little with Ollama and gpt4all but it seemed to me like it wasn’t fast enough to be useful on pure CPU, but if I could just throw cores at it then I might revisit the issue.

    • Possibly linux@lemmy.zipOPEnglish
      3·
      8 months ago

      It wasn’t usable a few months ago. However, when I setup ollama it was “fast” and it works ok. It takes anywhere from instant to 5min for responses. LLava seems to take the longest which makes sense. For llama2 it is fairly fast unless you ask it for obscure information.

    • goosegooseboat@lemmy.worldEnglish
      2·
      8 months ago

      For the life of me can’t remember the scores I was getting on gpt4all. But given that you tried it I’m guessing you’ll most likely take a liking to LM studio or perhaps jan.ai. both GUI tools. If the lack open source bothers you go for Jan.ai, if not then go LM studio. LM studio in particular allows for full and partial GPU offloading. So if you have a semi capable but not quite enough vram on it you can load part of the model on the GPU to speed up inference. As a side note pure CPU on my old ryzen 1600 I was looking at 6/it. Which isn’t all that much but glass half full its still faster than the average typing speed and takes the load off of having to think about how to creatively word things

  • RedNight@lemmy.mlEnglish
    3·
    8 months ago

    Ollama has been great for self-hosting, but also checkout vLLM as its the new shiny self-hosting toy

  • cron@feddit.deEnglish
    41·
    8 months ago

    Thanks for sharing this. Its a shame that most AI tech is hidden behind steep price tags and cloud subscriptions, while even midrange PCs can run interesting AI models.

    • PerogiBoiEnglish
      3·
      8 months ago

      Check out LM Studio. Totally free and very nice GUI. You can do some very cool things with it. That and Kobold AI. Very simple self hosting no need for configuration either. Download and open essentially.

  • thantik@lemmy.worldEnglish
    41·
    8 months ago

    The biggest thing that I want to learn is how to either A: add “tools” for the AI to run, or B: “fine-tune” the model by feeding it data that’s relevant to me.