On this page
AT A GLANCE
A multilingual public tourism assistant built around agent orchestration, RAG, external tools, and locally served open-weight models.
My contribution
Applied AI architecture · model evaluation · routing and tools · inference infrastructure · production debugging.
Key facts
Key engineering decisions
Route work by task
Separate intent selection, specialist work, and synthesis so tools and models can be evaluated for the narrower jobs they actually perform.
Serve open weights locally
Use role-specific vLLM endpoints to make model fit, capacity, observability, and replacement part of the system design.
Evaluate behavior in layers
Treat answer quality, tool use, routing, and concurrent serving capacity as separate questions rather than one aggregate score.
Investigate concurrency at the state boundary
Load testing traced interrupted streams to mutable agent state shared across requests, revealing the real serving constraint after isolation.
Project evolution
- Model benchmarks
- Routed agent architecture
- MCP tools and RAG
- Multimodal voice
- Load testing
- Request-state isolation
- Serving optimization
A public service behind a conversation
Tourists rarely ask neat, single-topic questions. In one message, someone might ask for a gluten-free restaurant, a wheelchair-accessible beach route, and something to do nearby, in English, Spanish, or Valencian, by text or voice. Benidorm brings those capabilities together in a public tourism assistant for visitors and residents.
The municipal tender went beyond chat: official content, CMS and smart-destination integrations, route planning, accessibility, and operational measurement. Its emphasis on open-weight models and self-hosted, open-source infrastructure made model quality, tool use, hardware fit, and serving capacity product decisions rather than vendor defaults. Fitting models into 24 GB NVIDIA L4 GPUs, routing tools reliably, and keeping the system stable under concurrent traffic became central engineering challenges.
Engineering scope
This case study follows the applied AI work across orchestration, knowledge retrieval, tourism tools, multimodal interaction, local model serving, benchmarking, and operational testing. The central question is how to fit different open-weight models to specific roles while keeping the complete system observable.
From request to useful response
Simplified architecture
A simplified view of the shared assistant path. Local model serving supports routing and synthesis; the surrounding prose documents the implementation details.
Request
A visitor asks in their preferred language by text or voice.
Routing
The system identifies which capabilities and sources the request needs.
Knowledge and services
Official content and tourism tools provide relevant, current context.
Synthesis
A role-specific model combines the selected results into one response.
Response
The assistant returns a clear answer through the selected channel.
The visitor experience
You can explore these accessible beach options in Benidorm and open their locations in Maps:
| Name | Type | Directions |
|---|---|---|
| Accessible Beaches in Benidorm | Beach | View in Maps |
| Levante Beach | Beach | View in Maps |
| Poniente Beach | Beach | View in Maps |
One question, several sources
The PWA uses CopilotKit to connect a conversation to FastAPI and LangGraph. After preparing state and checking persistent safety controls, the graph detects language and runs routing alongside sentiment analysis. The router selects specialists or simple service nodes; a synthesizer combines their results into one answer.
Drupal-backed MCP tools handle structured tourism queries. A separate RAG service ingests official content, generates embeddings, and searches PostgreSQL with pgvector. CMS updates can trigger reprocessing, while content hashes avoid embedding unchanged material. Maps and live-data services supply routes, weather, and municipal datasets.
The graph calls role-specific open-weight models through vLLM. Qwen-family models handle answer generation and tool-oriented work; Gemma remains in a narrow final parsing stage. Voice is normalized through STT into the same text-based agent flow, then routed to a language-appropriate TTS model. This keeps the multimodal experience from requiring one large multimodal model in every request. Langfuse traces, metrics, and persisted conversations make the path inspectable.
Why a router emerged
- Decision
- Separate intent selection, specialist work, and response synthesis.
- Rationale
- The capability set grew across restaurants, accommodation, events, transport, maps, and knowledge retrieval. Structured routing makes tool selection explicit and gives each specialist a narrower task.
- Tradeoff
- Additional model calls create latency and more interfaces to maintain. Parallel execution helps independent work, but synthesis still waits for the results it needs.
- Outcome
- The router returns selected agents and a query for each. LangGraph dispatches those branches and brings their results back to a shared synthesizer.
Intent routing
Select specialists and prepare their queries.
Visitor request
Find a restaurant and a walking route.
Restaurants
Tourism tools
Maps & routes
Map services
One response
Open weights were an architectural requirement
- Decision
- Evaluate and serve open-weight models locally through standard, replaceable interfaces.
- Rationale
- The project needed control over deployment, model roles, hardware allocation, and language-specific components without making the product depend on one hosted model API.
- Tradeoff
- Local inference moves model fitting, quantization, licenses, GPU memory, throughput, upgrades, and monitoring into the system's own operational scope.
- Outcome
- vLLM exposes OpenAI-compatible endpoints for role-specific Qwen and Gemma workloads. Open-source retrieval, orchestration, speech, maps, observability, and storage components keep the surrounding platform portable.
How evaluation expanded
Evaluation evolved with the system. An early n8n workflow compared repeated responses to tourism prompts using an AI judge. Later testing expanded to tool use, structured outputs, and end-to-end agent behavior.
Near-equal answer quality at a fraction of the cost
Gemma 27B mean score
The larger Gemma model provided the comparison point for this tourism-answer task.
Internal Benidorm benchmark; results apply to its prompt set, evaluator, serving configuration, and scoring rubric.Gemma 4B mean score
Nearly matched Gemma 27B at a fraction of the usage cost for this tourism use case.
Internal Benidorm benchmark; not a universal ranking of model families.Tourism answer quality
Internal benchmark · answer quality and tool use evaluated separately.
Evaluation rubric
- Clarity
- Factuality
- Consistency
| Gemma 27B | 8.23 |
|---|---|
| Gemma 4B | 8.17 |
Near-equal quality. A fraction of the usage cost.
Gemma 4B for this tourism use case.
Scores apply to this prompt set and rubric.
Models by role
Separate tool-use tests- Qwen
Generation & tools
- Gemma
Final parsing
Gemma: inconsistent tool use. Qwen: strong tool use, lower multilingual scores.
The highest answer score did not choose every role
- Decision
- Use Qwen for answer generation and tool-oriented work, with Gemma in the constrained final parser.
- Rationale
- When we benchmarked pure answer quality on our tourism prompt suite, Gemma 4B (8.17/10) nearly matched Gemma 27B (8.23/10) at a fraction of the compute cost. On paper, Gemma 4B looked like an easy winner. In practice, once we wired it into the agent loop, Gemma's tool selection and argument formatting were inconsistent. Qwen scored slightly lower on pure multilingual prose, but was more reliable with tool calls so we gave Qwen the tool-calling and generation roles, and kept a small Gemma model strictly as a constrained final parser.
- Tradeoff
- Choosing Qwen for generation and tools prioritized reliable tool behavior despite its lower multilingual assessment scores. A multi-model path also adds serving and debugging complexity, but lets Gemma remain within the narrower final-parsing contract.
- Outcome
- Model selection became role-specific: broad generation and tool behavior were judged independently from narrow final parsing. Aggregate quality scores informed the decision but did not override task behavior.
Benchmark the model and the system
Evaluate answer quality
Score clarity, factuality, tone, consistency, category behavior, and the low-quality tail on representative tourism prompts.
Test tools and structured output
Exercise routing, tool selection, arguments, malformed output recovery, and final parsing separately from natural-language answer quality.
Fit the deployment
Validate model size, quantization, context, output length, and memory use against the available L4 hardware and each role's latency budget.
Measure serving capacity
Use GuideLLM against each vLLM endpoint with role-shaped prompt and output sizes, then step through concurrent in-flight request levels to locate the throughput and latency knee.
Replay real agent scenarios
Run end-to-end cases through routing, RAG, live tools, synthesis, streaming, and fallbacks. A fast model endpoint is useful only if the complete assistant remains correct.
What broke under concurrency
The misleading symptom
When early load tests started failing at surprisingly low concurrency, our first suspect was GPU inference capacity. But vLLM metrics showed plenty of headroom and almost no queuing.
The actual failure
Tracing the failed streams revealed the real culprit: concurrent requests were sharing a single mutable AG-UI agent instance, so one visitor's stream could overwrite another's state mid-flight.
The change
Once we isolated agent instances per request (while sharing the compiled LangGraph topology), those crashes vanished.
The next constraint
That exposed the actual next bottleneck in router inference, which we solved by adding a load-balanced router replica.
Recommendations: relevance and exposure
- Decision
- Keep recommendation ordering behind a configurable bias-control layer.
- Rationale
- Sorting only by rating or review volume repeatedly favors the same establishments. A public tourism assistant also needs to consider variety among relevant candidates.
- Tradeoff
- Random rotation can reduce repetition but weaken consistency. Rating-based, stratified, and hybrid strategies each make different compromises; none proves fairness on its own.
- Outcome
- The recommendation code supports several ranking strategies independently of the tourism tools. This concerns establishment recommendations; semantic RAG retrieval uses its own similarity filters.
One assistant flow across text and voice
Simplified architecture
The audio path stays modular so speech components can change without replacing the evaluated text-and-tools assistant.
Audio
The visitor records a question.
Transcription
Speech becomes normalized text.
Assistant
The shared routing, tools, and knowledge flow produces an answer.
Speech routing
The response is sent to a voice suited to its language.
Playback
The visitor hears the answer and can continue the conversation.
Text and voice
Two ways in. One shared assistant.
Text input
What can I visit nearby?
Voice input
Speech to text
“What can I visit nearby?”
faster-whisperShared assistant
- Route
- Retrieve & use tools
- Synthesize
Speech output
Language-aware TTS
Selects the appropriate engine for the response language.
Multimodal chat without one multimodal bottleneck
- Decision
- Convert speech to text, reuse the text-and-tools agent graph, then route the response to a language-specific speech model.
- Rationale
- A decomposed audio → STT → language ID → agent → TTS path reuses the evaluated text workflow and lets each language use an appropriate open model. It avoids keeping a heavier general multimodal model in the critical path for every request.
- Tradeoff
- Recording, cancellation, transcription, language detection, synthesis, and playback have independent latency and failure states. Voice quality also varies by language, accent, street noise, place names, and client hardware.
- Outcome
- The dedicated audio service supports swappable local and API backends. The implemented local path uses faster-whisper for STT and routes TTS across Kokoro and aHoTTS-family voices. Browser or on-device inference remains an optional optimization with a server fallback.
Optimizing the measured path
Measure where time goes
Traces separate routing, source access, and synthesis. An internal preproduction analysis identified synthesis as the largest average component, while some external tools contributed slow responses.
Reduce avoidable work
Simple greetings can bypass router inference. Specialist results are compacted, irrelevant fields removed, and prompt sections included only when needed. Tool-output limits keep context growth bounded.
Allocate models by role
vLLM endpoints separate routing, generation, parsing, and embeddings. The serving environment uses NVIDIA L4 GPUs with 24 GB of memory, making quantization, context, output length, batching, and concurrent workloads practical model-selection constraints.
An architecture that can be investigated
The resulting system exposes distinct places to improve: model quality, tool behavior, hardware fit, routing decisions, source freshness, recommendation ordering, audio behavior, and inference capacity. The concurrency investigation demonstrates why that matters: fixing request isolation made the actual serving constraint visible.
Open weights were valuable because models could be benchmarked, fitted, served, and replaced by role. They also made the team responsible for those decisions. The work remains ongoing, and the figures here describe bounded internal evaluations rather than universal model rankings or a production capacity guarantee.
What the work made clear
- Answer quality and tool-use reliability are distinct model capabilities and need separate evaluation.
- Open-weight models are most useful when matched to a role and its hardware, not selected by parameter count alone.
- Useful recommendations depend on diversity as well as relevance.
- Separating speech from language keeps each part of a multimodal system replaceable and easier to improve.
- Traces explain system behavior, while queues expose capacity constraints; together they show what to optimize next.