Train

Training and fine-tuning

SharpMind has three training jobs. They share the data pipeline, checkpoints and export, but they train different things and use different devices.

Three modes

ModeStarts fromTrainsGPU
New model (pretrain) Random weights Every weight No — CPU only
Improve (LoRA) Packaged .smm Small adapters Yes, when the base fits
SFT Packaged .smm + chat JSONL LoRA adapters, assistant tokens only Yes, same shape rules
From-scratch pretraining cannot use the GPU engine. The GPU trainer is LoRA-only by design. A pretrain that names ilgpu is refused, then retried on the CPU with a loud log line.

GPU support

The GPU training engine accepts a base model with all of these:

  • RMSNorm (not LayerNorm) — both in the blocks and the final norm.
  • RoPE or NoPE positional encoding.
  • Gated FFN (SwiGLU/GeGLU-style), not MoE.
  • An attention layout with q_proj, k_proj, v_proj, o_proj LoRA targets.
  • A packaged .smm base. Convert a GGUF first.

LLaMA/Qwen-style bases satisfy this. GPT-2/BERT-style bases (LayerNorm, dense GELU) do not for training, although GPU inference now covers them.

On a supported base the log shows:

Fine-tune: 8 adapters (rank 8, alpha 16).
Engine: [Cuda] NVIDIA A16, 14962 MB, cuBLAS 12.6
Training 100 steps (float32) — full backprop (loss → gradients → AdamW)…

On an unsupported base it shows the refusal and falls back:

Accelerator 'ilgpu' declined: GPU engine (M1) does not support a
LayerNormLayer final norm — LayerNorm, only RMSNorm
Retrying on the CPU. Set SHARPMIND_ACCELERATOR_FALLBACK=false to fail instead.

Studio's source build copies the optional SharpMind.GPU plugin into plugins/ilgpu with its ILGPU dependencies. The Accelerator list then offers ilgpu beside the built-in CPU. The plugin needs a CUDA or OpenCL device; without one it declines and the CPU path runs. The plugin is not a NuGet package, so a packaged install only gets it when that folder is shipped.

The fallback is a host policy. Set SHARPMIND_ACCELERATOR_FALLBACK=false to fail the job instead of retrying.

Fine-tune a base

  1. Training → Improve.
  2. Base model: pick a packaged .smm. The model list is under Models; a GGUF must be converted first.
  3. Corpus: use Browse. With SFT off it filters *.txt; with SFT on, *.jsonl. A folder or glob also works.
  4. LoRA: rank 8–16, alpha 16–32, targets q_proj k_proj v_proj o_proj.
  5. Steps: 50–200 for a tiny test, 200–500+ for a real SFT run.
  6. Output: merged .smm, adapter .smlora, or both.
  7. Run on: This PC or a Compute host. The base and corpus upload with the job; the export downloads back.
  8. Start fine-tune and watch the engine line and loss curve.
Quick GPU check: a small LLaMA-preset base and a 1 MB text file can finish 10–100 steps in under a minute on an A16, and proves the engine, export and download path before you spend time on a real base.

SFT

Tick Instruction tuning (SFT) to train on chat JSONL with assistant-only loss. Each line is one record. Supported shapes:

{"messages":[{"role":"user","content":"…"},{"role":"assistant","content":"…"}]}
{"conversations":[{"from":"human","value":"…"},{"from":"gpt","value":"…"}]}
{"prompt":"…","response":"…"}
{"instruction":"…","input":"…","output":"…"}

The base supplies the tokenizer and chat template. Use Preview data before the run: it reports parsed, skipped, truncated and train/val/exam counts, plus sample rendered conversations.

  • Blank lines, malformed JSON and records without an assistant turn are skipped.
  • Long conversations are truncated to the SFT sequence length.
  • FuseChat's original file is a root JSON array, not JSONL; convert one record per line first.

Choosing data

DataUse
Plain .txtPlain LoRA fine-tune (SFT off). Shakespeare, stories, domain text.
Text JSONL ({"text":"…"})Not for SFT — no assistant turn. Add it with the JSONL source so each record's text is one document.
Chat JSONLSFT. Use messages, conversations, prompt/response or instruction/output.
ParquetNot read by Studio. Convert to JSONL first.

A JSONL file must go through the JSONL source. A text source reads the raw {"text":"…"} line including its JSON escapes, so the model learns to write \u0022 instead of quotes. The quick Add a text or JSONL corpus box picks the right source for a .jsonl path.

The job editor's Randomize button next to Seed changes the split, batching order and weight initialisation in one click.

For a quick test, a 1 MB text file is enough. For a useful instruction tune, a few thousand chat records are enough to see behaviour; 20k+ is a small real run. Prefer data that matches how you will use the model.

Remote runs

Select a Compute host in Run on. Studio uploads the source files and base, starts the job, polls its state and logs, then downloads the export back to the path shown on the form. The host needs no chat model for a fine-tune.

  • A GPU run needs a vcg- plan and the host's GPU host checkbox ticked; a vc2- plan runs on the CPU. See Vultr + SSH.
  • If the base is large, upload time dominates the first poll; the card logs progress in 25% steps.
  • The export is written under the host's /data/SharpMind/Training/jobs/…, then downloaded.
  • Rehearse 30 steps runs locally and estimates step time; it is not the remote job.
  • An interrupted run may leave the host job running; cancel it from the run record or the host API.

Progress and interrupt

The run panel follows the job: status, step, loss, validation loss, curve, log tail and an Interrupt button. Starting a run scrolls it into view. The panel repaints itself while the job is active, including remote jobs.

Leaving Training and coming back restores the form from the active run, or from the last draft when nothing is running, so the fields keep the values that started the run. A finished run is also saved under the job's name; Load latest brings it back after a restart.

Do not interrupt during "Downloading the trained model…" unless you mean it. The host has finished, but the local copy is cancelled. The export remains on the host; rerun or download it again.

Validation and exam splits

Set the validation split under New model → Checkpoints & splits (Validation split (%) and Held-out exam (%)), or on Improve (Validation split and Exam split). Both default to 10% and must stay below 100% together. The exam slice is excluded from training and validation and is scored after the run for the promotion verdict.

Val loss is evaluated at each log interval, so the line stays — until the first logged step. If the split produces no held-out documents, the run log says so and training continues without a validation curve; load the job to see the percentages it actually used.

Checkpoints and chat

Checkpoints are written under the job's export folder (Training/checkpoints-<name>). A pretrain checkpoint holds a complete model.smm; a fine-tune checkpoint holds adapters.smlora.

  • Add to Models copies a finished model or checkpoint into the Models folder, so Chat, Work and the labs see it.
  • Chat does that and opens a chat with the model selected.
  • Send to host uploads the model to a Compute host and activates it there; then pick that host in Chat.
  • Fine-tune checkpoints are adapters only and cannot load in Chat. Rerun the fine-tune with Output = merged (optionally resuming the checkpoint) to get a merged .smm.
  • For a remote run, the final export downloads back to this PC; the step checkpoints stay on the host.
  • Interrupting a run keeps the latest checkpoint. Resume from it instead of starting over; Start fresh ignores it.

The Models tab lists files in the Training and Models folders. Anything still in Training shows Add to Models; a model already in the Models folder only shows Chat.

Accelerator errors

MessageMeaning
only LoRA adapters are trainable in M1The job is a pretrain. Use Improve/SFT for GPU, or let it fall back to CPU.
LayerNorm, only RMSNormThe base is GPT-2/BERT-style. Use a LLaMA/Qwen base for GPU training.
No CUDA or OpenCL deviceThe container has no GPU access. Check --gpus all, the driver and the toolkit.
MoEMoE training is not supported by the GPU engine.
Val split produced no documentsThe corpus is too small for a 64-token window; training continues without a validation curve.