svnscha - self-hosted

AI Agents in .NET: Beyond DevUI - LibreChat Integration

2026-01-04T00:00:00+00:00

Welcome back to the AI Agents in .NET series. In the introduction, I built a basic conversational agent. In Part 1, I added database persistence, embeddings, and semantic search capabilities.

Today I'm breaking free from the built-in DevUI. Time to escape the developer dungeon.

Don't get me wrong - DevUI is fantastic for development and debugging.

This post connects those agents to LibreChat through OpenAI-compatible endpoints.

The first version failed when a tool-calling agent was used through a downstream client. The solution was a reusable middleware pattern called ToolCallFilterAgent, which I explain below.

The Problem with DevUI

DevUI served me well. It let me test agents, inspect tool calls, and debug issues. But it has limitations:

Single-purpose: It's a development tool, not a production interface
No multi-user support: One conversation at a time
Limited customization: You get what you get

For real-world use, I need something more flexible. And rather than building my own chat UI (been there, done that in early 2025, don't want to repeat), let's leverage existing open-source solutions.

Why LibreChat?

I mentioned LibreChat in my DGX Spark post, and I really enjoy it. Like, really like it.

It's:

OpenAI API compatible: Speaks the same language as ChatGPT
Self-hosted: Your data stays yours (take that, cloud overlords)
Feature-rich: File uploads, conversations, presets, the whole shebang
Beautiful: Actually looks like something you'd want to use - not like a developer accidentally vomited JSON onto a webpage

The key insight? LibreChat doesn't care what's behind the API. It just needs something that speaks OpenAI's protocol. And guess what the Microsoft Agent Framework already supports?

The Plan

Here's what I'm building:

%%{init: {"theme": "dark"}}%%
flowchart LR
    LC[LibreChat] -->|OpenAI API| K[Knowledge API]
    K -->|Agent Framework| A[Agents]
    A --> KA[Knowledge]
    A --> KSA[KnowledgeSearch]
    A --> KTA[KnowledgeTitle]
    KSA -->|Vector Search| PG[(PostgreSQL)]

LibreChat talks to our Knowledge API using the standard OpenAI chat completions format. Our API routes requests to our agents based on the model parameter. From LibreChat's perspective, it's just another OpenAI-compatible endpoint with multiple models to choose from.

Removing DevUI, Embracing Swagger

First things first - I'm removing the DevUI dependency and redirecting to Swagger for API exploration. DevUI is great for debugging, but for an API-first approach, Swagger makes more sense. Plus, Swagger has that nice "I'm a real API" energy.

The home page now redirects to Swagger:

app.MapGet("/", () => Results.Redirect("/swagger"));

Clean and simple. The API is now focused on being a backend service. Goodbye, training wheels.

Exposing OpenAI-Compatible Endpoints

The Agent Framework provides a beautiful MapOpenAIChatCompletions method that exposes agents as OpenAI-compatible endpoints. Each agent gets its own endpoint path. It's almost too easy (foreshadowing...).

I collect agent builders during registration and map them after building the app:

// Collect agent builders for endpoint mapping
var agentBuilders = new List<IHostedAgentBuilder>();

builder.ConfigureKnowledgeDefaults((settings, logger) =>
{
    // ... OpenAI client setup ...

    // Register agents and collect builders for endpoint mapping
    builder.Services.AddSingleton<KnowledgeSearchAgent>();
    agentBuilders.Add(builder.AddAIAgent("Knowledge", (services, key) => 
        AgentFactory.CreateKnowledgeAgent(chatClient, services, key)));
    agentBuilders.Add(builder.AddAIAgent("KnowledgeSearch", (services, key) => 
        AgentFactory.CreateKnowledgeSearchAgent(chatClient, services, key)));
});

var app = builder.Build();

// ... middleware setup ...

// Map OpenAI chat completions endpoint for each registered agent
foreach (var agentBuilder in agentBuilders)
{
    app.MapOpenAIChatCompletions(agentBuilder);
}

Now each agent has its own endpoint:

/Knowledge/v1/chat/completions
/KnowledgeSearch/v1/chat/completions

Pretty slick, right?

Setting Up LibreChat

Before I go further, let me get LibreChat running so I can test this integration. If you don't have LibreChat running yet, here's the quick setup. (If you do, feel free to skip ahead and judge my configuration choices.)

Docker Compose

mkdir -p ~/librechat
cd ~/librechat
git clone https://github.com/danny-avila/LibreChat.git .
cp .env.example .env

The LibreChat Configuration

Create librechat.yaml and point LibreChat at the API:

version: 1.2.8
cache: true

endpoints:
  custom:
    - name: "Agent Framework"
      apiKey: "not-used-but-required"
      baseURL: "http://host.docker.internal:5000/v1"
      models:
        default: ["Knowledge", "KnowledgeSearch"]
        fetch: false

      titleConvo: false
      summarize: false
      forcePrompt: false
      modelDisplayLabel: "Agent Framework"
      iconURL: "https://svnscha.de/svnscha.webp"

Docker Compose Override

Create docker-compose.override.yml to mount the config:

services:
  api:
    volumes:
      - type: bind
        source: ./librechat.yaml
        target: /app/librechat.yaml
    extra_hosts:
      - "host.docker.internal:host-gateway"

The extra_hosts line is crucial - it lets containers reach the host machine where the Knowledge API runs.

Fire It Up

docker compose up -d

Navigate to http://localhost:3080, create an account (it's local, don't worry), and you should see "Agent Framework" in the endpoint list.

But wait - if you select "Knowledge" and send a message, LibreChat hits /v1/chat/completions with "model": "Knowledge". My per-agent endpoints live at /Knowledge/v1/chat/completions instead. Houston, we have a routing problem.

The Model Routing Middleware

LibreChat (and most OpenAI clients) expect a single endpoint at /v1/chat/completions where the model parameter determines the underlying model but in this scenario it should determines routing:

{
  "model": "Knowledge",
  "messages": [{"role": "user", "content": "What's on your mind?"}]
}

I need middleware that reads the model from the request body and rewrites the path to the agent-specific endpoint. Nothing fancy, just some good old path mangling:

// Rewrite /v1/chat/completions to /{model}/v1/chat/completions based on request body
app.Use(async (context, next) =>
{
    var path = context.Request.Path.Value;
    if (path?.Equals("/v1/chat/completions", StringComparison.OrdinalIgnoreCase) == true)
    {
        context.Request.EnableBuffering();

        using var reader = new StreamReader(context.Request.Body, leaveOpen: true);
        var body = await reader.ReadToEndAsync();
        context.Request.Body.Position = 0;

        string? model = null;
        if (!string.IsNullOrEmpty(body))
        {
            try
            {
                var json = System.Text.Json.JsonDocument.Parse(body);
                if (json.RootElement.TryGetProperty("model", out var modelElement))
                {
                    model = modelElement.GetString();
                }
            }
            catch (System.Text.Json.JsonException)
            {
            }
        }

        if (string.IsNullOrWhiteSpace(model))
        {
            var logger = context.RequestServices.GetRequiredService<ILogger<Program>>();
            logger.LogWarning("Chat completions request missing required 'model' field");
            context.Response.StatusCode = StatusCodes.Status400BadRequest;
            context.Response.ContentType = "application/json";
            await context.Response.WriteAsJsonAsync(new { 
                error = new { 
                    message = "The 'model' field is required", 
                    type = "invalid_request_error", 
                    param = "model", 
                    code = "missing_required_parameter" 
                } 
            });
            return;
        }

        context.Request.Path = $"/{model}/v1/chat/completions";
    }
    await next();
});

The key points:

Enable buffering: I need to read the body to extract model, then reset it for the actual handler
Path rewriting: Transform /v1/chat/completions → /{model}/v1/chat/completions
Error handling: Return a proper OpenAI-style error if model is missing

Now when LibreChat sends a request, it gets routed to the right agent automatically. Back to LibreChat - select "Knowledge" from the dropdown, send a message, and... it works! Streaming responses, proper formatting, everything.

At this point, I thought I was done. Celebrated with a coffee. Time to test KnowledgeSearch...

The Tool Call Disaster (And How I Fixed It)

I switched to the KnowledgeSearch model in LibreChat, asked it to search for something, and... nothing. The request just hung. Then cancelled. No error message. No response. Just the cold, judgmental silence of broken software.

I dug through LibreChat's source code, traced the request flow, added logging everywhere. Console.WriteLine debugging like it's 2005.

Note: We should get to Observability very soon...

The Problem

My KnowledgeSearch agent uses tools - specifically the SearchConversationHistory function. When the agent executes a tool, the response stream includes FunctionCallContent and FunctionResultContent alongside the text. This is how the Agent Framework communicates "I'm calling a tool" and "here's what the tool returned."

LibreChat receives the stream and interprets the function call as something it should execute. That fails because the function exists in the backend rather than in LibreChat, so LibreChat cancels the request.

[Our Agent] → "I'll search for that" → FunctionCallContent{SearchConversationHistory}
[Our Agent] → [executes tool internally]
[Our Agent] → FunctionResultContent{results...}
[Our Agent] → "Based on the search, here's what I found..."
          ↓
[LibreChat] → Sees FunctionCallContent → "I need to call this function!"
[LibreChat] → Can't find function → Cancels request
          ↓
[User] → "Why isn't anything happening?" 😤

The frustrating part? You can't disable this behavior in LibreChat. It's doing the right thing for its use case - if it sends a function call to an upstream model, it expects to handle the response. But I'm the upstream model, and I've already handled my own tool calls. We're both right, and that's the most annoying kind of bug.

The Solution

Don't send tool call content downstream. Filter it out before it leaves the API. If LibreChat never sees the tool calls, it can't get confused by them. taps forehead

Meet ToolCallFilterAgent - a delegating agent that wraps any agent and strips FunctionCallContent and FunctionResultContent from responses. It's like a bouncer for your API responses:

/// <summary>
/// A delegating agent that filters out tool call content from responses.
/// This prevents downstream consumers from seeing FunctionCallContent and FunctionResultContent
/// that they cannot execute.
/// </summary>
public sealed class ToolCallFilterAgent : DelegatingAIAgent
{
    public ToolCallFilterAgent(AIAgent innerAgent) : base(innerAgent) { }

    public override async Task<AgentRunResponse> RunAsync(
        IEnumerable<ChatMessage> messages,
        AgentThread? thread,
        AgentRunOptions? options,
        CancellationToken cancellationToken)
    {
        var response = await InnerAgent.RunAsync(messages, thread, options, cancellationToken);
        response.Messages = FilterToolCalls(response.Messages);
        return response;
    }

    public override async IAsyncEnumerable<AgentRunResponseUpdate> RunStreamingAsync(
        IEnumerable<ChatMessage> messages,
        AgentThread? thread,
        AgentRunOptions? options,
        [EnumeratorCancellation] CancellationToken cancellationToken)
    {
        await foreach (var update in InnerAgent.RunStreamingAsync(messages, thread, options, cancellationToken))
        {
            yield return FilterToolCalls(update);
        }
    }

    private static IList<ChatMessage> FilterToolCalls(IEnumerable<ChatMessage> messages) =>
        messages.Select(m => new ChatMessage(m.Role,
            m.Contents.Where(c => c is not FunctionCallContent && c is not FunctionResultContent).ToList()
        )).ToList();

    private static AgentRunResponseUpdate FilterToolCalls(AgentRunResponseUpdate update) =>
        new(update.Role, update.Contents
            .Where(c => c is not FunctionCallContent && c is not FunctionResultContent).ToList());
}

Now in the AgentFactory, I wrap KnowledgeSearchAgent:

public static AIAgent CreateKnowledgeSearchAgent(IChatClient chatClient, IServiceProvider services, string key)
{
    var searchAgent = services.GetRequiredService<KnowledgeSearchAgent>();

    var agent = chatClient.CreateAIAgent(new ChatClientAgentOptions
    {
        Id = key,
        Name = key,
        ChatOptions = new ChatOptions
        {
            ConversationId = "global",
            Instructions = KnowledgeSearchSystemPrompt,
            Tools = [AIFunctionFactory.Create(searchAgent.SearchConversationHistoryAsync, "SearchConversationHistory")],
            ToolMode = ChatToolMode.Auto
        }
    });

    // Wrap with filter to prevent downstream consumers from seeing tool calls they can't execute
    return new ToolCallFilterAgent(agent);
}

The agent still uses tools internally, but clients only see the final text response. Clean and transparent. What happens in the backend stays in the backend.

Back to LibreChat - KnowledgeSearch now works! The agent searches, finds results, and responds - all without LibreChat ever knowing tools were involved. It's like magic, except it's just careful filtering.

Title Generation with KnowledgeTitleAgent

LibreChat has a feature called titleConvo - it automatically generates titles for conversations using the AI. But my main agents have tools and complex system prompts that are overkill for simple title generation. It's like using a flamethrower to light a candle.

The solution: a dedicated title agent. It's intentionally simple - no tools, no embeddings, just a focused system prompt:

/// <summary>
/// Simple agent for generating conversation titles.
/// No tools, no embeddings - just a basic helpful assistant.
/// Designed for use with LibreChat's title generation feature.
/// </summary>
public static class KnowledgeTitleAgent
{
    private const string TitleSystemPrompt = @"You are a helpful assistant that generates concise, descriptive titles for conversations.
When given conversation content, create a brief title (3-7 words) that captures the main topic or purpose.
Be specific and informative. Avoid generic titles like 'Chat' or 'Conversation'.";

    public static ChatClientAgent Create(IChatClient chatClient, IServiceProvider services, string key)
    {
        return chatClient.CreateAIAgent(new ChatClientAgentOptions
        {
            Id = key,
            Name = key,
            ChatOptions = new ChatOptions
            {
                ConversationId = "global",
                Instructions = TitleSystemPrompt,
            }
        });
    }
}

agentBuilders.Add(builder.AddAIAgent("KnowledgeTitle", (services, key) => 
    AgentFactory.CreateKnowledgeTitleAgent(chatClient, services, key)));

Now update librechat.yaml to use the title agent:

version: 1.2.8
cache: true

endpoints:
  custom:
    - name: "Agent Framework"
      apiKey: "not-used-but-required"
      baseURL: "http://host.docker.internal:5000/v1"
      models:
        default: ["Knowledge", "KnowledgeSearch", "KnowledgeTitle"]
        fetch: false

      titleConvo: true
      titleModel: "KnowledgeTitle"
      summarize: false
      forcePrompt: false
      modelDisplayLabel: "Agent Framework"
      iconURL: "https://svnscha.de/svnscha.webp"

Now LibreChat uses KnowledgeTitle specifically for generating conversation titles, while Knowledge and KnowledgeSearch handle the actual conversations.

The Complete Program.cs

Here's what Program.cs looks like after all these changes:

using System.ClientModel;
using Knowledge.Services;
using Knowledge.Shared.Agents;
using Knowledge.Shared.Extensions;
using Microsoft.Agents.AI.Hosting;
using Microsoft.Extensions.AI;
using OpenAI;

var builder = WebApplication.CreateBuilder(args);

// Collect agent builders for endpoint mapping
var agentBuilders = new List<IHostedAgentBuilder>();

builder.ConfigureKnowledgeDefaults((settings, logger) =>
{
    if (string.IsNullOrWhiteSpace(settings.ApiKey))
    {
        logger.LogWarning("No API key configured. Set Knowledge:ApiKey in user secrets or environment.");
    }

    var options = new OpenAIClientOptions();
    if (!string.IsNullOrEmpty(settings.ApiEndpoint))
    {
        options.Endpoint = new Uri(settings.ApiEndpoint);
    }

    var client = new OpenAIClient(new ApiKeyCredential(settings.ApiKey), options);
    var chatClient = client.GetChatClient("gpt-4.1").AsIChatClient();

    // Configure embedding service
    var embeddingClient = client.GetEmbeddingClient("text-embedding-3-small").AsIEmbeddingGenerator();
    builder.Services.AddEmbeddingService(embeddingClient);

    // Register agents and collect builders for endpoint mapping
    builder.Services.AddSingleton<KnowledgeSearchAgent>();
    agentBuilders.Add(builder.AddAIAgent("Knowledge", (services, key) => 
        AgentFactory.CreateKnowledgeAgent(chatClient, services, key)));
    agentBuilders.Add(builder.AddAIAgent("KnowledgeSearch", (services, key) => 
        AgentFactory.CreateKnowledgeSearchAgent(chatClient, services, key)));
    agentBuilders.Add(builder.AddAIAgent("KnowledgeTitle", (services, key) => 
        AgentFactory.CreateKnowledgeTitleAgent(chatClient, services, key)));
});

// Register background service for embedding processing
builder.Services.AddHostedService<EmbeddingBackgroundService>();

builder.Services.AddOpenAIResponses();
builder.Services.AddOpenAIConversations();

var app = builder.Build();

// Rewrite /v1/chat/completions to /{model}/v1/chat/completions based on request body
app.Use(async (context, next) =>
{
    var path = context.Request.Path.Value;
    if (path?.Equals("/v1/chat/completions", StringComparison.OrdinalIgnoreCase) == true)
    {
        context.Request.EnableBuffering();

        using var reader = new StreamReader(context.Request.Body, leaveOpen: true);
        var body = await reader.ReadToEndAsync();
        context.Request.Body.Position = 0;

        string? model = null;
        if (!string.IsNullOrEmpty(body))
        {
            try
            {
                var json = System.Text.Json.JsonDocument.Parse(body);
                if (json.RootElement.TryGetProperty("model", out var modelElement))
                {
                    model = modelElement.GetString();
                }
            }
            catch (System.Text.Json.JsonException) { }
        }

        if (string.IsNullOrWhiteSpace(model))
        {
            var logger = context.RequestServices.GetRequiredService<ILogger<Program>>();
            logger.LogWarning("Chat completions request missing required 'model' field");
            context.Response.StatusCode = StatusCodes.Status400BadRequest;
            context.Response.ContentType = "application/json";
            await context.Response.WriteAsJsonAsync(new { 
                error = new { 
                    message = "The 'model' field is required", 
                    type = "invalid_request_error", 
                    param = "model", 
                    code = "missing_required_parameter" 
                } 
            });
            return;
        }

        context.Request.Path = $"/{model}/v1/chat/completions";
    }
    await next();
});

app.UseRouting();

app.ConfigureKnowledgePipeline();

// Map OpenAI chat completions endpoint for each registered agent
foreach (var agentBuilder in agentBuilders)
{
    app.MapOpenAIChatCompletions(agentBuilder);
}

app.MapGet("/", () => Results.Redirect("/swagger"));

app.LogStartupComplete();

app.Run();

That's it. No CORS configuration needed (LibreChat makes server-side requests). No custom request/response models. No manual streaming code. The framework does the heavy lifting - I just wire it up. Sometimes the best code is the code you don't have to write.

The key insights:

Each agent gets its own endpoint via MapOpenAIChatCompletions
Inline middleware handles routing from /v1/chat/completions based on model
ToolCallFilterAgent hides internal tool execution from downstream clients
KnowledgeTitleAgent handles title generation without tool complexity

LibreChat Test

Open LibreChat, select "Agent Framework" as the endpoint, choose a model from the dropdown (Knowledge, KnowledgeSearch, or KnowledgeTitle), and start chatting. You should see:

Streaming responses (that satisfying typing effect)
Tool calls working transparently (KnowledgeSearch uses tools, but you only see the results)
Automatic title generation (thanks to KnowledgeTitle)

Architecture Overview

Let me step back and look at what I've built (and maybe pat myself on the back a little):

%%{init: {"theme": "dark"}}%%
flowchart TB
    subgraph Clients
        LC[LibreChat]
        API[Direct API Calls]
    end

    subgraph "Knowledge API"
        MR[Model Router Middleware]
        E1["/Knowledge/v1/chat/completions"]
        E2["/KnowledgeSearch/v1/chat/completions"]
        E3["/KnowledgeTitle/v1/chat/completions"]
    end

    subgraph Agents
        KA[Knowledge Agent]
        KSA[KnowledgeSearch Agent]
        KTA[KnowledgeTitle Agent]
        TCF[ToolCallFilterAgent]
    end

    subgraph Storage
        PG[(PostgreSQL)]
    end

    LC -->|"/v1/chat/completions"| MR
    API --> MR
    MR -->|"model: Knowledge"| E1
    MR -->|"model: KnowledgeSearch"| E2
    MR -->|"model: KnowledgeTitle"| E3
    E1 --> KA
    E2 --> TCF
    TCF --> KSA
    E3 --> KTA
    KSA -->|Vector Search| PG

The flow is clean:

Request comes in to /v1/chat/completions with "model": "KnowledgeSearch"
Middleware rewrites path to /KnowledgeSearch/v1/chat/completions
Framework routes to the agent, which is wrapped in ToolCallFilterAgent
Agent executes tools internally, filter strips tool content from response
Client sees clean text output

Multiple clients, multiple agents, one simple routing pattern. It's almost... elegant?

Swagger is still available at /swagger for API exploration and testing.

Summary

I've taken the agents from a development-only DevUI to something that can serve real users through a proper chat interface. And I did it with surprisingly little code - plus one hard-won debugging lesson that I'm still a bit salty about.

The main points are:

MapOpenAIChatCompletions does the heavy lifting - one line per agent, full OpenAI compatibility
Each agent gets its own endpoint - clean separation, easy to test directly
Simple inline middleware handles routing - LibreChat sends to /v1/chat/completions, I rewrite to /{model}/v1/chat/completions
Tool-calling agents need filtering - the ToolCallFilterAgent pattern is essential when exposing agents to downstream clients that don't understand your internal tool calls.
Dedicated agents for specific tasks - KnowledgeTitleAgent for titles, KnowledgeSearch for search, Knowledge for general chat

Next Steps

I've got a solid foundation now - persistence, embeddings, semantic search, a proper UI, and automatic title generation. But there's more to explore:

Observability: OpenTelemetry, Jaeger, understanding what's happening at scale
Agent Loop: Exploring the magic behind automated AI Agents
Authentication: Proper API key validation for production deployments (because "meh, whatever" isn't a security strategy)

But first, I'm letting this settle. Play with LibreChat, see how your agents perform with real conversations, and notice what's missing. The best features come from actual use - not from staring at code and imagining what users might want.

See you in the next post. 🚀

The companion repository has been updated with the part/02-connect-librechat branch containing all the code from this post. Check it out on GitHub.

DGX Spark: Hello World

2025-12-31T00:00:00+00:00

What Better Way to End 2025?

So here's how I'm closing out the year: playing with an NVIDIA DGX Spark. Yes, that DGX Spark - the "personal AI supercomputer" that makes my wallet cry but my inner tech nerd do a happy dance.

I have already spent some time testing LibreChat and ComfyUI on it. The courses at stable-diffusion-art.com also helped me get started with image-generation workflows.

And what better way to ring in the new year than with 128GB of unified memory and a Blackwell GPU sitting on my desk? I mean, some people do fireworks. I do neural networks.

(Yes, I'm aware that's possibly the nerdiest New Year's joke ever.)

Let me walk you through my first-day adventures getting this beautiful beast up and running. Fair warning: there may have been moments of childlike excitement, some creative problem-solving (read: frantically searching for a USB-C keyboard), and an unhealthy amount of "let's try this and see what happens."

The Unboxing

There's something special about unboxing new hardware. But unboxing a DGX? That hits different.

The packaging screams premium - NVIDIA clearly understands that when you're dropping serious money on a personal AI supercomputer, presentation matters. Opening the box felt like a tech ritual. Inside, nestled in protective foam, sat this compact powerhouse that somehow packs a Blackwell GPU and 128GB of unified memory into a form factor that actually fits on a desk.

The unit itself is surprisingly compact. I expected something massive and loud, but NVIDIA engineered this thing to be almost civilized. Almost. We'll see how that holds up once I start pushing it with larger models. Everything was included - power cables, documentation, and that unmistakable feeling of "I'm about to have way too much fun with this."

Alright, enough admiring the hardware. Time to make it do things.

The Setup Experience

I went with the local installation method - straightforward enough. Well, mostly. Here's a fun discovery: do you own a keyboard with USB Type-C? No? Neither did I. Fortunately, a laptop docking station saved the day. Crisis averted.

Once the initial setup was complete, first things first - let's give this machine a proper identity:

sudo hostnamectl set-hostname spark-1
sudo reboot

Why spark-1? Because who knows if there'll be a spark-2 someday. Better to be prepared. Future me will thank present me for this foresight. Or curse me for the temptation.

Installing Ollama

After updating the system, I started with something familiar. Ollama has become my go-to for local LLM deployment - it's absurdly simple to set up:

curl -fsSL https://ollama.com/install.sh | sh

Now, I know what you're thinking - "But what about vLLM? What about llama.cpp? What about those fancy self-compiled versions optimized for Blackwell's SM120 architecture?"

Yes, I've seen those posts. Yes, I'll explore them. But that's a future adventure. For now, let's start with the classics and make sure everything actually works before we go down the optimization rabbit hole.

With Ollama ready, time to pull a model:

ollama pull gpt-oss:20b

Could I run the massive 120B parameter models? Absolutely - that's kind of the whole point of having this hardware. But let's walk before we run. Download a smaller model to verify everything works, then let the big ones download overnight while I sleep. Strategy.

Setting Up LibreChat

Running models from the CLI is fine for testing, but I wanted a web interface. I previously used Open WebUI for my self-hosted LLM setup; this time I chose LibreChat, an open-source chat UI that supports local models.

Docker was pre-installed on the DGX Spark (nice touch, NVIDIA). Though my user wasn't in the docker group - but I'll assume you can figure that one out.

mkdir -p ~/workspaces
cd ~/workspaces/
git clone https://github.com/danny-avila/LibreChat.git
cd LibreChat/
cp .env.example .env

After some experimentation, I settled on these configuration changes:

Environment Configuration

In .env, the key change:

ENDPOINTS=custom

Docker Compose Override

Create a docker-compose.override.yml for your custom settings, and a librechat.yaml for the LibreChat-specific configuration.

docker-compose.override.yml

services:
  api:
    volumes:
      - type: bind
        source: ./librechat.yaml
        target: /app/librechat.yaml

librechat.yaml

---
version: 1.2.8
cache: true

endpoints:
  custom:
    - name: "Ollama"
      apiKey: "ollama"
      baseURL: "http://host.docker.internal:11434/v1/"
      models:
        default: [
          "gpt-oss:20b"
          ]
        fetch: true
      titleConvo: true
      titleModel: "current_model"
      summarize: false
      summaryModel: "current_model"
      forcePrompt: false
      modelDisplayLabel: "Ollama"

And that's... almost it.

The Reverse Proxy Dance

Here's where it gets slightly more involved. LibreChat enforces secure cookies by default - great for security, annoying when you're hitting F5 every five seconds during development and don't want to re-authenticate each time.

The solution? Set up a reverse proxy with HTTPS. I use Caddy in my home infrastructure because life's too short to manage SSL certificates manually.

chat.int.domain.tld {
    @lan remote_ip 172.16.29.0/24
    handle {
        reverse_proxy @lan http://172.16.2.123:3080
    }
    handle {
        templates
        respond "Access denied, {{.RemoteIP}}" 403
    }
}

This configuration does a few things:

Only allows access from my local network (the 172.16.29.0/24 range)
Proxies requests to the DGX Spark running LibreChat on port 3080
Returns a 403 for anyone trying to access from outside

It's not always open from outside but when it is (to refresh certificates) this is a great and easy way to deal with this.

I've also configured a custom DNS zone in my router to make chat.int.domain.tld resolve correctly, but that's infrastructure stuff for another post.

First Impressions

And just like that - we're live. Quick setup, everything working, ready to chat with local models running on actual Blackwell hardware. The response times are snappy, the interface is clean, and I can already tell this is going to be a fun playground.

Next Steps

Now that the basics are running, it's time to explore what this hardware can actually do. On my list:

Document Understanding: Feed it files and see how well it comprehends them
Voice integration: Testing spoken conversations with local models
Image generation with ComfyUI: Exploring image-generation workflows on the Blackwell GPU. The stable-diffusion-art.com courses have been a useful starting point.

These are just the fundamentals to get comfortable with the platform. But honestly? Between LibreChat's slick interface for chatting with local models and ComfyUI's node-based wizardry for image generation, I've already had more fun than should be legal.

It was a good way to end 2025: experimenting with new hardware and learning what it can do.

Here's to closing out the year with new adventures, and to many more in 2026! 🎉

I will share more results as I continue testing the DGX Spark.

Exploring AI: Self Hosted LLM

2025-03-17T00:00:00+00:00

Why, You Ask?

So, I decided to self-host an LLM. Why? Every time you use an online AI model, you're handing over your data to some company. Whether it's casual conversations, coding snippets, or business-related queries, everything you type is potentially being logged, analyzed, or even used to train future models. No thanks.

Instead, I prefer to keep things local. Also, I have an RTX ADA 4000 with 20GB of memory sitting here, so why not put it to good use?

Ollama makes it easy to run LLMs locally. I combine it with Open WebUI for the user interface and Nginx for remote access.

Here is the setup.

Step 1: Install Ollama

Ollama makes deploying LLMs locally ridiculously simple. Here's how to install it:

curl -fsSL https://ollama.com/install.sh | sh

This will install Ollama and set up everything you need to start running models locally. Want to make sure it's working? Just run:

ollama run codellama:13b

If you see an interactive prompt, congrats - you've got a local LLM running!

Step 2: Install Open WebUI

Ollama is great, but a web interface makes it even better. That's where Open WebUI comes in. It gives you a sleek, chat-like interface to interact with your models.

To install Open WebUI manually without Docker, follow these steps:

1. Create a Virtual Environment

python3 -m venv ~/openwebui-venv
source ~/openwebui-venv/bin/activate

2. Install Open WebUI

pip install open-webui

3. Create a Systemd Service

To make sure Open WebUI runs on startup, create a systemd service file:

sudo nano /etc/systemd/system/openwebui.service

Paste the following content:

[Unit]
Description=Open WebUI Service
After=network.target

[Service]
User=$USER
Group=$USER
WorkingDirectory=/home/$USER/openwebui-venv
ExecStart=/home/$USER/openwebui-venv/bin/open-webui
Restart=always

[Install]
WantedBy=multi-user.target

Save and exit, then reload systemd and enable the service:

sudo systemctl daemon-reload
sudo systemctl enable openwebui.service
sudo systemctl start openwebui.service

Step 3: Reverse Proxy with Nginx

Now, let's make accessing our LLM easier by setting up an Nginx reverse proxy. This way, we can reach Open WebUI without exposing it directly.

Here's a basic Nginx config:

server {
    listen 443 ssl;
    server_name chat.example.com;

    location / {
        proxy_pass http://localhost:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Reload Nginx with:

sudo systemctl restart nginx

Now, you can access your self-hosted LLM via https://chat.example.com. Fancy.

You Own Your Data Now

One of the biggest advantages of self-hosting an LLM? Your data stays with you.

No sending queries to an external API, no third-party tracking what you're asking, no potential leaks of sensitive information. It's all running on your hardware, fully under your control. Whether you're experimenting with code, processing confidential documents, or just having fun chatting with AI, everything stays local.

Model Sizes & Performance

Of course, different models come with different memory requirements. Here's what I'm running on my RTX ADA 4000 and how much VRAM they use:

NAME             ID              SIZE      PROCESSOR    UNTIL              
codellama:7b     8fdf8f752f6e    9.4 GB    100% GPU     2 minutes from now    
codellama:13b    9f438cb9cd58    15 GB     100% GPU     4 minutes from now    
gemma3:12b       6fd036cefda5    13 GB     100% GPU     4 minutes from now

This means I can comfortably run mid-sized models like codellama:13b while keeping things snappy.

Choosing the Right GPU

Picking the right GPU is all about balancing performance, VRAM, and cost - because, let's be honest, unless you're running an AI research lab, you're not dropping $30,000 on an H100.

Here's a quick breakdown of solid options:

NVIDIA RTX 3090 - Powerful, with 24 GB of VRAM, but it's last-gen. ($$)
NVIDIA RTX 4090 - Even more powerful, with 24 GB of VRAM and better efficiency. ($$$)
NVIDIA RTX 5090 - Even more and more powerful, with 24 GB of VRAM and better efficiency. ($$$$)
NVIDIA RTX 4000 ADA Generation - Less powerful, but 20 GB of VRAM, and a low-profile card. ($)

Now, why the ADA 4000? While the RTX 5090 is the fastest in raw compute power, VRAM is king for training AI models. The ADA 4000's 20 GB VRAM gives you enough room for Stable Diffusion training, larger batch sizes, and AI experiments, without hitting the limits of other consumer GPUs such as 3080 (10 GB).

Performance-wise, the 4090 and 5090 has more horsepower, but for training workloads where memory matters more than raw speed, the ADA 4000 is the more practical and cost-efficient choice. Plus, lower power consumption makes it a better long-term option if you're running AI workloads frequently. Also, the physical size - I mean it's a low profile card. Small, fits perfectly into any case.

For my needs, the RTX 4000 Ada offered a useful balance of VRAM, price, power use, and size.

Example

Asking codellama:13b

Prompt: implement fibonacci in python and also some unit tests using pytest

Your browser does not support the video tag.

Asking gemma3:12b

Prompt: implement fibonacci in python and also some unit tests using pytest

Your browser does not support the video tag.

Summary

With an RTX ADA 4000, Ollama, Open WebUI, and an Nginx reverse proxy, I now have an AI-powered assistant running entirely on my own hardware. No subscriptions, no cloud dependencies, just raw, local AI power. If you're serious about AI and privacy, setting this up is a no-brainer. Give it a try, and let your GPU do some work.