svnscha - ollama

Ollama Now Speaks Messages API

2026-01-22T00:00:00+00:00

The News: Ollama Learns a New Language

Claude Code can now talk to your locally-hosted models without any extra adapters, proxies, or dark magic.

"Ollama v0.14.0 and later are now compatible with the Anthropic Messages API, making it possible to use tools like Claude Code with open-source models." — Ollama Blog

Cue the excitement: Finally! Unlimited coding on steroids! No more watching your token budget evaporate! 24/7 AI-assisted development without the API bill anxiety!

...right?

Why Test This, You Ask?

Naturally, I had to try this immediately. What better excuse to fire up the DGX Spark and see what these local models can really do?

If you're running a DGX Spark (or any beefy GPU setup), this means you can now fire up Claude Code and have it talk to whatever model you've got running locally. Zero cloud dependency. Zero API costs eating into your coffee budget. Just pure, local AI goodness.

But does it actually work? Let's find out.

Setting It Up

Which Model?

The official docs recommend a few options:

Cloud models (if you're into that sort of thing):

glm-4.7:cloud, minimax-m2.1:cloud, qwen3-coder:480b

Local models (the fun part):

qwen3-coder - Excellent for coding tasks
gpt-oss:20b - Strong general-purpose model
glm-4.7-flash - Deep reasoning, needs Ollama 0.14.3

I tested several of these locally - qwen3-coder is genuinely impressive for coding tasks, and glm-4.7-flash (just released with Ollama 0.14.3) surprised me with its deep reasoning approach. Cloud models? Kind of defeats the purpose of running local, doesn't it?

Bump Ollama's Context Window

The docs recommend at least 32k context. Use systemctl edit for a persistent override that survives updates:

sudo systemctl edit ollama.service

Add this in the editor that opens:

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=32000"
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=f16"

Bonus performance tweaks:

OLLAMA_FLASH_ATTENTION=1 - Normal attention loads the entire context into memory at once. Flash Attention breaks it into chunks, processes them sequentially, and combines the results. Same output, way less memory.
OLLAMA_KV_CACHE_TYPE=f16 - Keeps the K/V cache at full precision (default). Use q8_0 to halve memory if you're tight on VRAM.

How much context can you actually fit? The VRAM Calculator is your friend. Play around with your model size, quantization, and available VRAM - it'll tell you the exact settings to max out each model.

Then restart:

sudo systemctl restart ollama.service

PowerShell Helper for Your Work Machine

This little function lives in my $PROFILE:

function Enter-ClaudeSpark {
   $env:ANTHROPIC_AUTH_TOKEN = "ollama"
   $env:ANTHROPIC_BASE_URL = "http://spark-1:8080"
   $env:CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC = "1"
}

Now I just type Enter-ClaudeSpark and Claude Code magically routes everything to my DGX Spark. The ANTHROPIC_AUTH_TOKEN is set to ollama because Ollama doesn't actually need a real token - it just checks if the header exists. Clever.

My Test: Meeting Destruction Therapy

Public benchmarks do not always reflect how a model feels in everyday use. I care more about how well it follows my language and understands the task, so I made a small practical test of my own.

I needed something creative, visual, and complex enough to separate the wheat from the chaff. What better way than asking these models to build a Breakout-style game where you "reschedule" your weekly meetings by smashing them with a paddle?

The Prompt:

# Breakout-Style Calendar Game Prompt

## Objective
Build a **simple, modern Breakout-inspired browser game** where a bouncing ball clears **calendar-style blocks representing weekly meetings**, and a horizontal paddle is used to “reschedule” (break) them.

## Core Concept
- The playfield should resemble a **weekly calendar view** rather than a classic arcade board.
- Each block represents a **meeting/event** (rounded rectangles, subtle borders).
- The paddle metaphorically “reschedules” meetings by bouncing the ball upward.
- The ball “clears” meetings as they are hit.

## Visual & UX Direction (Very Important)
- **Look & feel inspired by modern calendar apps** such as **Google Calendar** or **Outlook Calendar**:
  - Soft neutral background (light or dark mode acceptable)
  - Rounded corners, subtle shadows, clean typography
  - Muted but intentional color palette (event colors feel like calendar categories)
  - Gentle grid or column structure reminiscent of a weekly view
- Avoid retro/arcade styling (no pixel fonts, neon colors, or harsh outlines).
- Animations should feel **smooth and calm**, not frantic:
  - Subtle easing on ball movement
  - Soft fade/scale effects when meetings are cleared
  - Light hover or focus states where appropriate

## Technical Requirements
- Use **only vanilla HTML, CSS, and JavaScript** (no frameworks or libraries).
- Single-page implementation.
- **Responsive and centered layout** that works on desktop and mobile.
- Canvas or DOM-based rendering is acceptable, but visuals should remain crisp.
- Code should be clean, readable, and well-commented.

## Gameplay Requirements
- Paddle controlled via mouse/touch (keyboard optional).
- Ball physics should feel smooth and predictable.
- Winning state when all meetings are cleared.
- Optional: subtle sound effects or score labeled as **“Meetings Cleared”**.

## Polish Encouraged
- Thoughtful spacing and alignment
- Calendar-like typography (system UI fonts preferred)
- Micro-interactions and transitions that reinforce the scheduling metaphor

## Implementation
- Create index.html, style.css and game.js.

When finished, suggest that the user can host and test the game locally using:

python -m http.server

I threw this at several models through Claude Code and timed the results:

Model	Time	Rating
Claude Opus 4.5	2m 30s	⭐⭐⭐⭐⭐
`qwen3-coder:30b`	1m 48s	⭐⭐⭐⭐
`gpt-oss:20b`	4m 24s	⭐⭐⭐
`glm-4.7-flash`	8m 45s	⭐⭐⭐
`nemotron-3-nano:30b`	2m 46s	⭐
`ministral-3:14b`	-	🚩
`rnj-1`	-	🚩

Claude Opus 4.5 (Reference)

Not self-hosted - this is the cloud-based frontline model, included for comparison. Understood the creative brief, nailed the game mechanics, produced clean and maintainable code. No surprises there.

Your browser does not support the video tag.

qwen3-coder:30b

Faster than expected, and it actually produced something playable. At 30B parameters, genuinely impressive.

Your browser does not support the video tag.

gpt-oss:20b

Your browser does not support the video tag.

For this one I went all out - maxed context window at 95k with full precision KV cache:

Environment="OLLAMA_CONTEXT_LENGTH=95232"
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=f16"

Solid result! The 20B variant delivered a working game with proper paddle controls. Not as polished as qwen3-coder, but definitely playable. A good middle-ground option if you want something from the GPT family.

glm-4.7-flash

"As the strongest model in the 30B class, GLM-4.7-Flash offers a new option for lightweight deployment that balances performance and efficiency." - Bold claims require testing. This one needs Ollama 0.14.3 (just released!).

Your browser does not support the video tag.

Why the 8+ minute runtime? GLM spends serious time in its thinking stage. Watch the Ollama output live and you'll see deep, methodical reasoning - no "what if" self-doubt loops, just clear and consistent problem-solving. The result shows: best physics of all the OSS models I tested. The ball movement felt smooth and responsive. The catch? It completely ignored the meeting-themed UI brief and went with a generic breakout style instead. So close, yet so far.

nemotron-3-nano:30b

It rendered something! Got the calendar theme, colorful meeting blocks, even a "Meetings Cleared" counter. But... where's the ball? Where's the paddle? Apparently Nemotron thought breakout meant "break out of implementing game mechanics." Nice UI though.

This one makes me a bit sad, honestly. NVIDIA has been releasing incredible AI stuff lately - the voice models alone are mind-blowing - but Nemotron-3-Nano just flopped here. Maybe it shines elsewhere, but for Claude Code workflows? Not ready.

ministral-3:14b

I really tried with this one. Multiple attempts, different prompts, fresh sessions. Every single time: Claude Code just... completed. Immediately. No files, no code, no output. It's like the model and Claude Code looked at each other and mutually agreed to do nothing. 🤷

rnj-1

"8B parameter open-weight, dense models trained from scratch by Essential AI, optimized for code and STEM with capabilities on par with SOTA open-weight models." - 130k downloads, fresh release, sounded promising. The VSCode + Cline demo on their blog looked great - but that was a Python game, not our HTML/JS breakout challenge.

Unfortunately, no results here. Another one that just didn't produce anything usable with Claude Code. At least the resource usage was... well, not surprising for an 8B model.

Summary

So, Ollama now speaks the Messages API, and Claude Code can talk to local models. I put it through a fun benchmark - building a calendar breakout game - and the results were clear: Opus still delivers the goods, but the OSS models are catching up fast.

For Claude Code agentic workflows, qwen3-coder is the clear winner in the OSS space - fast, capable, and actually follows instructions. glm-4.7-flash has potential but needs work on following prompts. gpt-oss:20b is a solid middle-ground option.

This actually reflects my broader model philosophy: ChatGPT-style models for basic tasks - rephrasing, docs, research. Anthropic for serious coding work. And now in the open-source world, qwen3-coder is genuinely fun to work with.

Is this going to replace Claude Opus for serious work? Not today. But for experimentation, learning, and those times when you want to see what the open-source world can do? This setup is fantastic.

Next Steps

Want to try this yourself?

Install Ollama ≥ 0.14: curl -L https://ollama.com/install.sh | sh
Pull your model of choice: ollama pull qwen3-coder:30b
Bump the context window to at least 32k
Add the PowerShell helper to your $PROFILE
Fire up Claude Code and start breaking some meetings... I mean, coding

And who knows - at the rate these models are improving, maybe in a few months I'll have to eat my words about frontline models being irreplaceable.

In the meantime, building a game that lets you destroy your weekly meetings with a bouncing ball might be the most satisfying thing I've prompted this month.

Happy hacking! 🎮

DGX Spark: Hello World

2025-12-31T00:00:00+00:00

What Better Way to End 2025?

So here's how I'm closing out the year: playing with an NVIDIA DGX Spark. Yes, that DGX Spark - the "personal AI supercomputer" that makes my wallet cry but my inner tech nerd do a happy dance.

I have already spent some time testing LibreChat and ComfyUI on it. The courses at stable-diffusion-art.com also helped me get started with image-generation workflows.

And what better way to ring in the new year than with 128GB of unified memory and a Blackwell GPU sitting on my desk? I mean, some people do fireworks. I do neural networks.

(Yes, I'm aware that's possibly the nerdiest New Year's joke ever.)

Let me walk you through my first-day adventures getting this beautiful beast up and running. Fair warning: there may have been moments of childlike excitement, some creative problem-solving (read: frantically searching for a USB-C keyboard), and an unhealthy amount of "let's try this and see what happens."

The Unboxing

There's something special about unboxing new hardware. But unboxing a DGX? That hits different.

The packaging screams premium - NVIDIA clearly understands that when you're dropping serious money on a personal AI supercomputer, presentation matters. Opening the box felt like a tech ritual. Inside, nestled in protective foam, sat this compact powerhouse that somehow packs a Blackwell GPU and 128GB of unified memory into a form factor that actually fits on a desk.

The unit itself is surprisingly compact. I expected something massive and loud, but NVIDIA engineered this thing to be almost civilized. Almost. We'll see how that holds up once I start pushing it with larger models. Everything was included - power cables, documentation, and that unmistakable feeling of "I'm about to have way too much fun with this."

Alright, enough admiring the hardware. Time to make it do things.

The Setup Experience

I went with the local installation method - straightforward enough. Well, mostly. Here's a fun discovery: do you own a keyboard with USB Type-C? No? Neither did I. Fortunately, a laptop docking station saved the day. Crisis averted.

Once the initial setup was complete, first things first - let's give this machine a proper identity:

sudo hostnamectl set-hostname spark-1
sudo reboot

Why spark-1? Because who knows if there'll be a spark-2 someday. Better to be prepared. Future me will thank present me for this foresight. Or curse me for the temptation.

Installing Ollama

After updating the system, I started with something familiar. Ollama has become my go-to for local LLM deployment - it's absurdly simple to set up:

curl -fsSL https://ollama.com/install.sh | sh

Now, I know what you're thinking - "But what about vLLM? What about llama.cpp? What about those fancy self-compiled versions optimized for Blackwell's SM120 architecture?"

Yes, I've seen those posts. Yes, I'll explore them. But that's a future adventure. For now, let's start with the classics and make sure everything actually works before we go down the optimization rabbit hole.

With Ollama ready, time to pull a model:

ollama pull gpt-oss:20b

Could I run the massive 120B parameter models? Absolutely - that's kind of the whole point of having this hardware. But let's walk before we run. Download a smaller model to verify everything works, then let the big ones download overnight while I sleep. Strategy.

Setting Up LibreChat

Running models from the CLI is fine for testing, but I wanted a web interface. I previously used Open WebUI for my self-hosted LLM setup; this time I chose LibreChat, an open-source chat UI that supports local models.

Docker was pre-installed on the DGX Spark (nice touch, NVIDIA). Though my user wasn't in the docker group - but I'll assume you can figure that one out.

mkdir -p ~/workspaces
cd ~/workspaces/
git clone https://github.com/danny-avila/LibreChat.git
cd LibreChat/
cp .env.example .env

After some experimentation, I settled on these configuration changes:

Environment Configuration

In .env, the key change:

ENDPOINTS=custom

Docker Compose Override

Create a docker-compose.override.yml for your custom settings, and a librechat.yaml for the LibreChat-specific configuration.

docker-compose.override.yml

services:
  api:
    volumes:
      - type: bind
        source: ./librechat.yaml
        target: /app/librechat.yaml

librechat.yaml

---
version: 1.2.8
cache: true

endpoints:
  custom:
    - name: "Ollama"
      apiKey: "ollama"
      baseURL: "http://host.docker.internal:11434/v1/"
      models:
        default: [
          "gpt-oss:20b"
          ]
        fetch: true
      titleConvo: true
      titleModel: "current_model"
      summarize: false
      summaryModel: "current_model"
      forcePrompt: false
      modelDisplayLabel: "Ollama"

And that's... almost it.

The Reverse Proxy Dance

Here's where it gets slightly more involved. LibreChat enforces secure cookies by default - great for security, annoying when you're hitting F5 every five seconds during development and don't want to re-authenticate each time.

The solution? Set up a reverse proxy with HTTPS. I use Caddy in my home infrastructure because life's too short to manage SSL certificates manually.

chat.int.domain.tld {
    @lan remote_ip 172.16.29.0/24
    handle {
        reverse_proxy @lan http://172.16.2.123:3080
    }
    handle {
        templates
        respond "Access denied, {{.RemoteIP}}" 403
    }
}

This configuration does a few things:

Only allows access from my local network (the 172.16.29.0/24 range)
Proxies requests to the DGX Spark running LibreChat on port 3080
Returns a 403 for anyone trying to access from outside

It's not always open from outside but when it is (to refresh certificates) this is a great and easy way to deal with this.

I've also configured a custom DNS zone in my router to make chat.int.domain.tld resolve correctly, but that's infrastructure stuff for another post.

First Impressions

And just like that - we're live. Quick setup, everything working, ready to chat with local models running on actual Blackwell hardware. The response times are snappy, the interface is clean, and I can already tell this is going to be a fun playground.

Next Steps

Now that the basics are running, it's time to explore what this hardware can actually do. On my list:

Document Understanding: Feed it files and see how well it comprehends them
Voice integration: Testing spoken conversations with local models
Image generation with ComfyUI: Exploring image-generation workflows on the Blackwell GPU. The stable-diffusion-art.com courses have been a useful starting point.

These are just the fundamentals to get comfortable with the platform. But honestly? Between LibreChat's slick interface for chatting with local models and ComfyUI's node-based wizardry for image generation, I've already had more fun than should be legal.

It was a good way to end 2025: experimenting with new hardware and learning what it can do.

Here's to closing out the year with new adventures, and to many more in 2026! 🎉

I will share more results as I continue testing the DGX Spark.

Exploring AI: Self Hosted LLM

2025-03-17T00:00:00+00:00

Why, You Ask?

So, I decided to self-host an LLM. Why? Every time you use an online AI model, you're handing over your data to some company. Whether it's casual conversations, coding snippets, or business-related queries, everything you type is potentially being logged, analyzed, or even used to train future models. No thanks.

Instead, I prefer to keep things local. Also, I have an RTX ADA 4000 with 20GB of memory sitting here, so why not put it to good use?

Ollama makes it easy to run LLMs locally. I combine it with Open WebUI for the user interface and Nginx for remote access.

Here is the setup.

Step 1: Install Ollama

Ollama makes deploying LLMs locally ridiculously simple. Here's how to install it:

curl -fsSL https://ollama.com/install.sh | sh

This will install Ollama and set up everything you need to start running models locally. Want to make sure it's working? Just run:

ollama run codellama:13b

If you see an interactive prompt, congrats - you've got a local LLM running!

Step 2: Install Open WebUI

Ollama is great, but a web interface makes it even better. That's where Open WebUI comes in. It gives you a sleek, chat-like interface to interact with your models.

To install Open WebUI manually without Docker, follow these steps:

1. Create a Virtual Environment

python3 -m venv ~/openwebui-venv
source ~/openwebui-venv/bin/activate

2. Install Open WebUI

pip install open-webui

3. Create a Systemd Service

To make sure Open WebUI runs on startup, create a systemd service file:

sudo nano /etc/systemd/system/openwebui.service

Paste the following content:

[Unit]
Description=Open WebUI Service
After=network.target

[Service]
User=$USER
Group=$USER
WorkingDirectory=/home/$USER/openwebui-venv
ExecStart=/home/$USER/openwebui-venv/bin/open-webui
Restart=always

[Install]
WantedBy=multi-user.target

Save and exit, then reload systemd and enable the service:

sudo systemctl daemon-reload
sudo systemctl enable openwebui.service
sudo systemctl start openwebui.service

Step 3: Reverse Proxy with Nginx

Now, let's make accessing our LLM easier by setting up an Nginx reverse proxy. This way, we can reach Open WebUI without exposing it directly.

Here's a basic Nginx config:

server {
    listen 443 ssl;
    server_name chat.example.com;

    location / {
        proxy_pass http://localhost:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Reload Nginx with:

sudo systemctl restart nginx

Now, you can access your self-hosted LLM via https://chat.example.com. Fancy.

You Own Your Data Now

One of the biggest advantages of self-hosting an LLM? Your data stays with you.

No sending queries to an external API, no third-party tracking what you're asking, no potential leaks of sensitive information. It's all running on your hardware, fully under your control. Whether you're experimenting with code, processing confidential documents, or just having fun chatting with AI, everything stays local.

Model Sizes & Performance

Of course, different models come with different memory requirements. Here's what I'm running on my RTX ADA 4000 and how much VRAM they use:

NAME             ID              SIZE      PROCESSOR    UNTIL              
codellama:7b     8fdf8f752f6e    9.4 GB    100% GPU     2 minutes from now    
codellama:13b    9f438cb9cd58    15 GB     100% GPU     4 minutes from now    
gemma3:12b       6fd036cefda5    13 GB     100% GPU     4 minutes from now

This means I can comfortably run mid-sized models like codellama:13b while keeping things snappy.

Choosing the Right GPU

Picking the right GPU is all about balancing performance, VRAM, and cost - because, let's be honest, unless you're running an AI research lab, you're not dropping $30,000 on an H100.

Here's a quick breakdown of solid options:

NVIDIA RTX 3090 - Powerful, with 24 GB of VRAM, but it's last-gen. ($$)
NVIDIA RTX 4090 - Even more powerful, with 24 GB of VRAM and better efficiency. ($$$)
NVIDIA RTX 5090 - Even more and more powerful, with 24 GB of VRAM and better efficiency. ($$$$)
NVIDIA RTX 4000 ADA Generation - Less powerful, but 20 GB of VRAM, and a low-profile card. ($)

Now, why the ADA 4000? While the RTX 5090 is the fastest in raw compute power, VRAM is king for training AI models. The ADA 4000's 20 GB VRAM gives you enough room for Stable Diffusion training, larger batch sizes, and AI experiments, without hitting the limits of other consumer GPUs such as 3080 (10 GB).

Performance-wise, the 4090 and 5090 has more horsepower, but for training workloads where memory matters more than raw speed, the ADA 4000 is the more practical and cost-efficient choice. Plus, lower power consumption makes it a better long-term option if you're running AI workloads frequently. Also, the physical size - I mean it's a low profile card. Small, fits perfectly into any case.

For my needs, the RTX 4000 Ada offered a useful balance of VRAM, price, power use, and size.

Example

Asking codellama:13b

Prompt: implement fibonacci in python and also some unit tests using pytest

Your browser does not support the video tag.

Asking gemma3:12b

Prompt: implement fibonacci in python and also some unit tests using pytest

Your browser does not support the video tag.

Summary

With an RTX ADA 4000, Ollama, Open WebUI, and an Nginx reverse proxy, I now have an AI-powered assistant running entirely on my own hardware. No subscriptions, no cloud dependencies, just raw, local AI power. If you're serious about AI and privacy, setting this up is a no-brainer. Give it a try, and let your GPU do some work.