svnscha - llm

DGX Spark: Hello World

2025-12-31T00:00:00+00:00

What Better Way to End 2025?

So here's how I'm closing out the year: playing with an NVIDIA DGX Spark. Yes, that DGX Spark - the "personal AI supercomputer" that makes my wallet cry but my inner tech nerd do a happy dance.

I have already spent some time testing LibreChat and ComfyUI on it. The courses at stable-diffusion-art.com also helped me get started with image-generation workflows.

And what better way to ring in the new year than with 128GB of unified memory and a Blackwell GPU sitting on my desk? I mean, some people do fireworks. I do neural networks.

(Yes, I'm aware that's possibly the nerdiest New Year's joke ever.)

Let me walk you through my first-day adventures getting this beautiful beast up and running. Fair warning: there may have been moments of childlike excitement, some creative problem-solving (read: frantically searching for a USB-C keyboard), and an unhealthy amount of "let's try this and see what happens."

The Unboxing

There's something special about unboxing new hardware. But unboxing a DGX? That hits different.

The packaging screams premium - NVIDIA clearly understands that when you're dropping serious money on a personal AI supercomputer, presentation matters. Opening the box felt like a tech ritual. Inside, nestled in protective foam, sat this compact powerhouse that somehow packs a Blackwell GPU and 128GB of unified memory into a form factor that actually fits on a desk.

The unit itself is surprisingly compact. I expected something massive and loud, but NVIDIA engineered this thing to be almost civilized. Almost. We'll see how that holds up once I start pushing it with larger models. Everything was included - power cables, documentation, and that unmistakable feeling of "I'm about to have way too much fun with this."

Alright, enough admiring the hardware. Time to make it do things.

The Setup Experience

I went with the local installation method - straightforward enough. Well, mostly. Here's a fun discovery: do you own a keyboard with USB Type-C? No? Neither did I. Fortunately, a laptop docking station saved the day. Crisis averted.

Once the initial setup was complete, first things first - let's give this machine a proper identity:

sudo hostnamectl set-hostname spark-1
sudo reboot

Why spark-1? Because who knows if there'll be a spark-2 someday. Better to be prepared. Future me will thank present me for this foresight. Or curse me for the temptation.

Installing Ollama

After updating the system, I started with something familiar. Ollama has become my go-to for local LLM deployment - it's absurdly simple to set up:

curl -fsSL https://ollama.com/install.sh | sh

Now, I know what you're thinking - "But what about vLLM? What about llama.cpp? What about those fancy self-compiled versions optimized for Blackwell's SM120 architecture?"

Yes, I've seen those posts. Yes, I'll explore them. But that's a future adventure. For now, let's start with the classics and make sure everything actually works before we go down the optimization rabbit hole.

With Ollama ready, time to pull a model:

ollama pull gpt-oss:20b

Could I run the massive 120B parameter models? Absolutely - that's kind of the whole point of having this hardware. But let's walk before we run. Download a smaller model to verify everything works, then let the big ones download overnight while I sleep. Strategy.

Setting Up LibreChat

Running models from the CLI is fine for testing, but I wanted a web interface. I previously used Open WebUI for my self-hosted LLM setup; this time I chose LibreChat, an open-source chat UI that supports local models.

Docker was pre-installed on the DGX Spark (nice touch, NVIDIA). Though my user wasn't in the docker group - but I'll assume you can figure that one out.

mkdir -p ~/workspaces
cd ~/workspaces/
git clone https://github.com/danny-avila/LibreChat.git
cd LibreChat/
cp .env.example .env

After some experimentation, I settled on these configuration changes:

Environment Configuration

In .env, the key change:

ENDPOINTS=custom

Docker Compose Override

Create a docker-compose.override.yml for your custom settings, and a librechat.yaml for the LibreChat-specific configuration.

docker-compose.override.yml

services:
  api:
    volumes:
      - type: bind
        source: ./librechat.yaml
        target: /app/librechat.yaml

librechat.yaml

---
version: 1.2.8
cache: true

endpoints:
  custom:
    - name: "Ollama"
      apiKey: "ollama"
      baseURL: "http://host.docker.internal:11434/v1/"
      models:
        default: [
          "gpt-oss:20b"
          ]
        fetch: true
      titleConvo: true
      titleModel: "current_model"
      summarize: false
      summaryModel: "current_model"
      forcePrompt: false
      modelDisplayLabel: "Ollama"

And that's... almost it.

The Reverse Proxy Dance

Here's where it gets slightly more involved. LibreChat enforces secure cookies by default - great for security, annoying when you're hitting F5 every five seconds during development and don't want to re-authenticate each time.

The solution? Set up a reverse proxy with HTTPS. I use Caddy in my home infrastructure because life's too short to manage SSL certificates manually.

chat.int.domain.tld {
    @lan remote_ip 172.16.29.0/24
    handle {
        reverse_proxy @lan http://172.16.2.123:3080
    }
    handle {
        templates
        respond "Access denied, {{.RemoteIP}}" 403
    }
}

This configuration does a few things:

Only allows access from my local network (the 172.16.29.0/24 range)
Proxies requests to the DGX Spark running LibreChat on port 3080
Returns a 403 for anyone trying to access from outside

It's not always open from outside but when it is (to refresh certificates) this is a great and easy way to deal with this.

I've also configured a custom DNS zone in my router to make chat.int.domain.tld resolve correctly, but that's infrastructure stuff for another post.

First Impressions

And just like that - we're live. Quick setup, everything working, ready to chat with local models running on actual Blackwell hardware. The response times are snappy, the interface is clean, and I can already tell this is going to be a fun playground.

Next Steps

Now that the basics are running, it's time to explore what this hardware can actually do. On my list:

Document Understanding: Feed it files and see how well it comprehends them
Voice integration: Testing spoken conversations with local models
Image generation with ComfyUI: Exploring image-generation workflows on the Blackwell GPU. The stable-diffusion-art.com courses have been a useful starting point.

These are just the fundamentals to get comfortable with the platform. But honestly? Between LibreChat's slick interface for chatting with local models and ComfyUI's node-based wizardry for image generation, I've already had more fun than should be legal.

It was a good way to end 2025: experimenting with new hardware and learning what it can do.

Here's to closing out the year with new adventures, and to many more in 2026! 🎉

I will share more results as I continue testing the DGX Spark.

AI Agents in .NET: Let's Build Something Smart

2025-12-21T00:00:00+00:00

Welcome to a new blog series about building AI agents in .NET.

Yes, you read that right - I just said .NET. Not Python. C#. Curly braces. Strong typing. With drums and trumpets.

Over the coming posts, we'll explore the Microsoft Agent Framework, build actual agentic systems, and - most importantly - have some fun doing it. This first post sets the stage: why I'm doing this, what we're building, and how to get started with the companion repository.

Here is how to get started.

Why, You Ask?

Honestly? I got a little restless. Every AI agent tutorial out there follows the same well-worn path:

"Build a weather agent!" - Revolutionary stuff. Though I hear windows also exist.
"Spam detection with AI!" - Gmail's been doing this since 2004.
"Customer service chatbot!" - Oh good, we definitely need more of those.

These examples are useful for learning the basics, but I wanted to build something closer to a real application.

This series is about building agents that are genuinely smart, genuinely fun, or ideally both. No boring examples. No wrapping API calls in agent clothing and calling it intelligent.

As for the language choice - I've always been a C-family person. Python is lovely for what it does, but give me curly braces and strong typing any day. When Microsoft released the Agent Framework for .NET? Yeah, that got me genuinely excited. First-class AI agents in C#. Native support, proper tooling, the whole package.

The Microsoft Agent Framework

So what exactly are we working with? The framework comes with everything you need to build proper agentic systems:

Native C# support (obviously)
A beautiful DevUI for testing and debugging your agents - still in preview and missing some Python-side features, but it gets the job done
Built-in patterns for common agentic scenarios (the fun stuff)
Integration with the broader .NET ecosystem (finally!)

And that's what this blog series is all about.

About DevUI

Speaking of DevUI - if you head over to Microsoft's documentation and switch to the C# tab, you're greeted with this gem:

"DevUI documentation for C# is coming soon. Please check back later or refer to the Python documentation for conceptual guidance."

Ah yes, the classic "refer to Python" move. You know, the language we specifically chose not to use for this series? The irony isn't lost on me. But hey, I'm not bitter - I'm just impressed they managed to ship a working DevUI before the documentation. That's actually kind of bold.

The DevUI is a useful web interface for chatting with agents and inspecting their behavior during development. It does not replace proper tracing with tools such as OpenTelemetry and Jaeger, but it works well for quick iteration and debugging. I will cover observability separately in a later post.

The Knowledge Repository

To support this series, I've created a companion repository called Knowledge. I briefly considered "AI-Stuff-I-Throw-Together-At-2AM" but that didn't seem like a good fit. So I thought about it - I'm deepening my own knowledge about AI as I create this series, and hopefully you'll gain some knowledge along the way too. Yeah, let's face it: Knowledge is simple. Sometimes simple works.

The repository is structured as a progressive learning path. Each branch builds on the previous one, taking you from zero to building sophisticated agentic systems. Here's what we're working with:

Branch	Topic	What You'll Learn	Codespace
`main`	Repository & Hello World Agent	Project setup, DevUI, your first conversational agent

More branches will be added as the series progresses. And honestly? That Codespaces button is kind of magical - click it, grab a coffee, and come back to a fully configured dev environment. No setup, no dependencies, just ready to go. I still find that ridiculously cool.

Star the repo if you want to follow along!

Development Environment

You've got two options here - pick whichever fits your workflow.

Dev Container Setup

First things first: reproducible development environments. The repository includes a .devcontainer configuration with:

.NET 10
PostgreSQL with pgvector for when we get into embeddings and semantic search
All the tooling pre-configured

Just open the repo in VS Code, click "Reopen in Container," and you're ready to go. The classic "works on my machine" problem? Solved.

You can also open it in a Codespace and work entirely in the browser without a local setup.

Classic

And for those of you rolling your eyes at all this container and cloud stuff - fair enough. Install the .NET 10 SDK, fire up VS Code or Visual Studio, clone the repo, and just go. Sometimes the old ways are the best ways. That said, you'll want to get PostgreSQL with pgvector set up sooner rather than later - we'll need it for embeddings and semantic search in upcoming posts.

What's on the Main Branch?

Let me walk you through what you'll find when you clone the repo. The main branch sets up everything you need to hit the ground running with AI agent development.

Project Structure

The solution is organized into two projects:

Knowledge - The main web application that hosts our agents and the DevUI
Knowledge.Shared - Shared configuration, extensions, and utilities

This separation keeps things clean as we add more complexity in future posts.

Configuration Done Right

One thing I'm particular about is handling configuration properly. In Knowledge.Shared, we have a KnowledgeSettings class that gives us strongly-typed access to our config values.

The important bits are these two properties:

public string ApiKey { get; set; } = string.Empty;

public string ApiEndpoint { get; set; } = string.Empty;

ApiKey is where your OpenAI (or compatible) API key goes. ApiEndpoint is optional - leave it empty and we'll hit OpenAI's default endpoint, or set it to point at Azure OpenAI, a local model, whatever you need.

Now, you definitely don't want to hardcode API keys in your source code. That's how keys end up on GitHub and suddenly you're funding a stranger's LLM experiments. Instead, we use .NET User Secrets:

cd src/Knowledge
dotnet user-secrets set "Knowledge:ApiKey" "your-api-key-here"

Your key stays on your machine, outside of source control. Sleep well at night.

Wiring Up the Agent

The following sections walk through Program.cs piece by piece.

First, the imports - nothing too surprising here:

using System.ClientModel;
using Knowledge.Shared.Extensions;
using Microsoft.Agents.AI.DevUI;
using Microsoft.Agents.AI.Hosting;
using Microsoft.Extensions.AI;
using OpenAI;

We're pulling in the Agent Framework (Microsoft.Agents.AI.*), Microsoft's AI abstractions (Microsoft.Extensions.AI), and the official OpenAI client. Standard .NET web app stuff otherwise.

Next, we create our builder and call into our configuration extension:

var builder = WebApplication.CreateBuilder(args);

builder.ConfigureKnowledgeDefaults((settings, logger) =>
{
    // ...
});

ConfigureKnowledgeDefaults is a helper method in our shared project. It handles loading appsettings.json, binding it to our KnowledgeSettings type, and then gives us a callback where we can access both the settings and a logger. This keeps Program.cs clean while still giving us full control.

Inside that callback, we set up OpenAI. First, a sanity check:

if (string.IsNullOrWhiteSpace(settings.ApiKey))
{
    logger.LogWarning("No API key configured. Set Knowledge:ApiKey in user secrets.");
}

Better to warn loudly than fail silently.

Then we create the OpenAI client options. If a custom endpoint is configured, we use it:

var options = new OpenAIClientOptions();

if (!string.IsNullOrEmpty(settings.ApiEndpoint))
{
    options.Endpoint = new Uri(settings.ApiEndpoint);
}

This is what lets you swap between OpenAI, Azure OpenAI, or even a local model running through something like Ollama with an OpenAI-compatible API.

Now we create the actual client and register it with dependency injection:

var client = new OpenAIClient(new ApiKeyCredential(settings.ApiKey), options);

builder.Services.AddChatClient(client.GetChatClient("gpt-4.1").AsIChatClient());

GetChatClient("gpt-4.1") gets us a chat client for that specific model. .AsIChatClient() wraps it in Microsoft's IChatClient abstraction, which is what the Agent Framework expects. Using this abstraction means we could swap out OpenAI for any other provider without changing our agent code.

And finally - the actual agent registration:

builder.AddAIAgent("Knowledge", "You are a helpful agent named Knowledge.");

That's it. One line. We give it a name and a system prompt, and the framework handles the rest. We'll make this more sophisticated in future posts, but for a "hello world" agent, this is all you need.

Finishing the Pipeline

After our configuration callback, we register a few more services:

builder.Services.AddOpenAIResponses();
builder.Services.AddOpenAIConversations();

These set up the Agent Framework's handlers for OpenAI's responses and conversations API patterns.

Then we build the app and configure the HTTP pipeline:

var app = builder.Build();

app.ConfigureKnowledgePipeline();

ConfigureKnowledgePipeline is another shared helper - it sets up logging, error handling, and other middleware that we'll want across all our experiments.

Finally, we map our endpoints:

app.MapOpenAIResponses();
app.MapOpenAIConversations();
app.MapDevUI();

app.MapGet("/", () => Results.Redirect("/devui/"));

app.Run();

MapOpenAIResponses() and MapOpenAIConversations() expose endpoints that follow OpenAI's API patterns. MapDevUI() enables the Agent Framework's built-in developer interface.

That last MapGet just redirects the root URL to the DevUI. Because when you're developing agents, that's where you want to be.

Running the Application

Once configured, getting started is simple:

dotnet run --project src/Knowledge

Navigate to http://localhost:5000/devui and you'll see the DevUI - a beautiful interface for interacting with your agents, inspecting their reasoning, and debugging when things inevitably go sideways.

Summary

We've covered a lot of ground in this introductory post. You now have:

A companion repository with a reproducible dev container setup
A working "hello world" agent using the Microsoft Agent Framework
Proper configuration with user secrets (no API keys in source control!)
The DevUI for testing and debugging your agents

The foundation is solid. Now comes the fun part.

Next Steps

We've got the foundation in place - a working agent, DevUI, proper configuration. But let's be real: a "hello world" agent isn't going to impress anyone, least of all ourselves.

Next, I will add tools so the agent can perform actions, explore multi-agent scenarios, and use PostgreSQL with pgvector for embeddings and semantic search. A later post will also add proper observability.

But first things first. Step by step.

If you're a .NET developer who's been watching the AI agent space from the sidelines, wondering when C# would get some love - welcome. If you're just someone who's bored of the same old tutorials and wants to see something different - also welcome.

Clone the repo, star it if you're feeling generous, and let's build something that's actually worth building.

See you in the next post. 🚀

Exploring AI: Self Hosted LLM

2025-03-17T00:00:00+00:00

Why, You Ask?

So, I decided to self-host an LLM. Why? Every time you use an online AI model, you're handing over your data to some company. Whether it's casual conversations, coding snippets, or business-related queries, everything you type is potentially being logged, analyzed, or even used to train future models. No thanks.

Instead, I prefer to keep things local. Also, I have an RTX ADA 4000 with 20GB of memory sitting here, so why not put it to good use?

Ollama makes it easy to run LLMs locally. I combine it with Open WebUI for the user interface and Nginx for remote access.

Here is the setup.

Step 1: Install Ollama

Ollama makes deploying LLMs locally ridiculously simple. Here's how to install it:

curl -fsSL https://ollama.com/install.sh | sh

This will install Ollama and set up everything you need to start running models locally. Want to make sure it's working? Just run:

ollama run codellama:13b

If you see an interactive prompt, congrats - you've got a local LLM running!

Step 2: Install Open WebUI

Ollama is great, but a web interface makes it even better. That's where Open WebUI comes in. It gives you a sleek, chat-like interface to interact with your models.

To install Open WebUI manually without Docker, follow these steps:

1. Create a Virtual Environment

python3 -m venv ~/openwebui-venv
source ~/openwebui-venv/bin/activate

2. Install Open WebUI

pip install open-webui

3. Create a Systemd Service

To make sure Open WebUI runs on startup, create a systemd service file:

sudo nano /etc/systemd/system/openwebui.service

Paste the following content:

[Unit]
Description=Open WebUI Service
After=network.target

[Service]
User=$USER
Group=$USER
WorkingDirectory=/home/$USER/openwebui-venv
ExecStart=/home/$USER/openwebui-venv/bin/open-webui
Restart=always

[Install]
WantedBy=multi-user.target

Save and exit, then reload systemd and enable the service:

sudo systemctl daemon-reload
sudo systemctl enable openwebui.service
sudo systemctl start openwebui.service

Step 3: Reverse Proxy with Nginx

Now, let's make accessing our LLM easier by setting up an Nginx reverse proxy. This way, we can reach Open WebUI without exposing it directly.

Here's a basic Nginx config:

server {
    listen 443 ssl;
    server_name chat.example.com;

    location / {
        proxy_pass http://localhost:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Reload Nginx with:

sudo systemctl restart nginx

Now, you can access your self-hosted LLM via https://chat.example.com. Fancy.

You Own Your Data Now

One of the biggest advantages of self-hosting an LLM? Your data stays with you.

No sending queries to an external API, no third-party tracking what you're asking, no potential leaks of sensitive information. It's all running on your hardware, fully under your control. Whether you're experimenting with code, processing confidential documents, or just having fun chatting with AI, everything stays local.

Model Sizes & Performance

Of course, different models come with different memory requirements. Here's what I'm running on my RTX ADA 4000 and how much VRAM they use:

NAME             ID              SIZE      PROCESSOR    UNTIL              
codellama:7b     8fdf8f752f6e    9.4 GB    100% GPU     2 minutes from now    
codellama:13b    9f438cb9cd58    15 GB     100% GPU     4 minutes from now    
gemma3:12b       6fd036cefda5    13 GB     100% GPU     4 minutes from now

This means I can comfortably run mid-sized models like codellama:13b while keeping things snappy.

Choosing the Right GPU

Picking the right GPU is all about balancing performance, VRAM, and cost - because, let's be honest, unless you're running an AI research lab, you're not dropping $30,000 on an H100.

Here's a quick breakdown of solid options:

NVIDIA RTX 3090 - Powerful, with 24 GB of VRAM, but it's last-gen. ($$)
NVIDIA RTX 4090 - Even more powerful, with 24 GB of VRAM and better efficiency. ($$$)
NVIDIA RTX 5090 - Even more and more powerful, with 24 GB of VRAM and better efficiency. ($$$$)
NVIDIA RTX 4000 ADA Generation - Less powerful, but 20 GB of VRAM, and a low-profile card. ($)

Now, why the ADA 4000? While the RTX 5090 is the fastest in raw compute power, VRAM is king for training AI models. The ADA 4000's 20 GB VRAM gives you enough room for Stable Diffusion training, larger batch sizes, and AI experiments, without hitting the limits of other consumer GPUs such as 3080 (10 GB).

Performance-wise, the 4090 and 5090 has more horsepower, but for training workloads where memory matters more than raw speed, the ADA 4000 is the more practical and cost-efficient choice. Plus, lower power consumption makes it a better long-term option if you're running AI workloads frequently. Also, the physical size - I mean it's a low profile card. Small, fits perfectly into any case.

For my needs, the RTX 4000 Ada offered a useful balance of VRAM, price, power use, and size.

Example

Asking codellama:13b

Prompt: implement fibonacci in python and also some unit tests using pytest

Your browser does not support the video tag.

Asking gemma3:12b

Prompt: implement fibonacci in python and also some unit tests using pytest

Your browser does not support the video tag.

Summary

With an RTX ADA 4000, Ollama, Open WebUI, and an Nginx reverse proxy, I now have an AI-powered assistant running entirely on my own hardware. No subscriptions, no cloud dependencies, just raw, local AI power. If you're serious about AI and privacy, setting this up is a no-brainer. Give it a try, and let your GPU do some work.