Inference

Models and local chat

Chat is the default Studio mode: pick a model, talk to it, keep sessions. The same packaged file can run locally, on a Compute host, or become the base for a fine-tune.

Model formats

FormatWhat it isWhere it works
.smmSharpMind packaged model: config, weights, tokenizer and metadata.Local chat, Work, Compute hosts, fine-tune base.
.ggufGGUF model from HuggingFace or another tool.Local inference where the shape is supported; convert to .smm for hosts, fine-tuning and packaging.
.smloraLoRA adapter trained against a specific base.Resume/fine-tune; merge into a base to run as a normal model.

Training exports a merged .smm by default. An adapter alone cannot load without its base; the merged model is the simplest thing to chat with.

Add a model

  1. Training → Models → Find GGUF models. Search HuggingFace, inspect license, quantisation, size and gated status, then download to your Models folder with progress and resume.
  2. Convert if needed. The converter is directly under the browser. It suggests the destination, verifies the result and refuses to overwrite without confirmation.
  3. Pick it in Chat. The model chooser shows a "runs well / runs slowly / won't fit" badge based on this machine.
  4. For a host, upload it. Settings → Compute → host card → Browse → Upload and use. Hosts keep models under /data/SharpMind/Models.

Local models live under %USERPROFILE%\SharpMind\Models. Adapters and checkpoints live under Training.

Local chat

  • Saved sessions with titles, a system prompt and a tools drawer.
  • Streaming replies; stop generation from the message controls.
  • Per-session inference settings, with the effective value and the layer that supplied it.
  • Model badges from the machine's memory and the model size.
  • Long contexts truncate from the oldest turns; the session shows what was sent.

Chat works with a local .smm, a local GGUF where supported, a remote router, or a SharpMind Compute host. The provider picker is on the welcome and chat screens.

On the new-chat screen, Attach and Send sit in one row under the prompt instead of a large send button with the attachment control below it. Attachments stay local for now; a remote router receives text only. A new chat starts on the router and model you used last. Chat lists the default project's conversations; Work lists every project that has work sessions, and choosing Work asks which project the session belongs to, including a new one. The assistant name follows the CUI's Greek tier convention: the sampling temperature picks Alpha to Epsilon, with Delta when no temperature is set.

Context compaction

The default profile uses Summarize. When the prompt reaches 70% of the session's context, the active model reads the older turns and writes a short summary. Those turns are marked ignored — kept in the transcript but left out of later prompts — and the summary is pinned as a system message. System prompts and pinned messages are never summarized, and the current turn (from the latest user message onward, including its tool calls) is always kept verbatim. Repeated compactions fold the previous summary into one rolling summary rather than adding another pinned message.

Choose None, Summarize, Truncate (drop the oldest turns) or a plugin compactor by name under the inference profile's Context compactor. The system, family, model and per-session layers can each override it; plugin compactors appear in the same list. A global plugin compactor selected under Settings → Plugin components is used as the fallback when the profile is set to None.

Inference tuning

Settings → Models and Settings → Model families expose the layered settings: System → Family → Model → Session. Each value shows its source, so you can tell whether a setting came from the family default or the session.

  • Sampling: temperature, top-p, top-k, repetition penalty, max tokens.
  • Context and KV-cache behaviour, where supported by the model shape.
  • Tool-call recovery: Strict, Lenient or Loose. Chat and Work can use different defaults.
  • Reasoning tags and prompt formatting through the family profile.
Model families are metadata, not magic. If a downloaded model is misdetected, set its family (LLaMA, Qwen, GPT-2, BERT, …) under Model families; the chat template and stop tokens follow it.

GPU inference

A Compute host with the GPU image runs inference on the device when the model shape is supported. The engine covers:

  • RMSNorm or LayerNorm.
  • RoPE or NoPE positional encoding.
  • Gated or dense GELU FFN.
  • LLaMA/Qwen and GPT-2/BERT-style decoders, within the arena limit.

/health reports the backend and the reason:

"accelerators": [{ "name": "ilgpu", "inference": true, "backend": "CUDA · cuBLAS" }],
"acceleratorReason": "Inference on ilgpu (CUDA · cuBLAS)."

An unsupported shape or a prompt that overruns the arena falls back to the CPU with the reason in /health. It does not fail the chat. Unlike training, GPU inference does not require RMSNorm or a gated FFN.

Local Studio uses the same optional plugin. The source build copies SharpMind.GPU into plugins/ilgpu, and local chat picks it up when a CUDA or OpenCL device is present. Without the plugin or a device, chat runs on the CPU.

Compute API

A SharpMind Compute host serves one active model at a time, OpenAI-style. The token is required on every route except /health.

GET  /health
GET  /v1/models
POST /v1/models/activate       {"name":"model.smm"}
PUT  /v1/models/{name}?activate=true
POST /v1/chat/completions      {"stream":true, ...}
PUT  /v1/files/{path}
POST /v1/training/jobs
  • One model per process. Activating another unloads the current one.
  • Chat is OpenAI-compatible with streaming. Tool calling on the Compute API is not there yet.
  • Uploads and training jobs are bearer-token protected; /health is open.
  • Runpod's proxy has a 100-second response timeout; training returns 202 and is polled, so long jobs are fine.

Local, host or OpenRouter

LocalCompute hostOpenRouter
WeightsOn this PCOn your serverProvider's
CostFreeHost + powerPer token
PrivacyFullYour hostProvider policy
Tool callingYesNot yetYes, via the model
TuningCPU or local GPUGPU/CPU hostInference only
Model choiceWhat fitsWhat fits the hostMarketplace

Problems

  • Model won't fit: use a smaller quantisation, a shorter context, or a host with more memory. The badge is advisory, not a hard stop.
  • GGUF loads but behaves poorly: convert to .smm; the converter verifies the package and keeps the metadata.
  • Wrong chat template: set the model family in Settings; "raw" output is usually a missing template.
  • 401 from a host: the Studio token and container token differ. See tokens.
  • Host model list is empty: upload a model, or activate one already in the host's Models folder.