Architecture study · Draft

CyberBible: Cloud (A) vs. Local Box (B)

Every number links to its source. "unknown" means no public source was found; nothing is estimated without saying so.

CyberBible uses AI to build a person's second brain as plain Markdown files that the user can read. A keeps those files in the user's own cloud account with KMS envelope encryption, runs the AI on a server, and serves the files to any model over MCP and a CLI. B sells a box about the price of a Mac mini that runs the model itself and never sends the data anywhere.

1. Which local models can do the job

The job: understand a long onboarding conversation, then write well-structured Markdown.

What this means. RULER is a synthetic recall-and-tracking test. It shows how long a context a model can really use, but it does not measure Markdown quality. Up to about 32K tokens, which covers a typical onboarding transcript (the actual transcript length is unknown and should be measured), Qwen3 14B–32B scores about 94–96 [2]. That is in the same band as GPT-4-1106 (93.2 at 32K) [2]. Llama 3.1 70B and Mistral Large lose a lot of quality at 128K [2]. No public benchmark scores "structured Markdown from a conversation," so a CyberBible-specific eval is needed. IFEval, which checks format compliance, is the closest proxy, and Qwen3-30B-A3B-2507 is on par with GPT-4o there (84.7 vs 83.9) [1].

Best fits: Qwen3-30B-A3B-Instruct-2507 as the main model, because it has few active parameters and a long effective context. Mistral Small 3.2 24B is the backup.

2. Hardware

• Mac mini M5 Pro: from $1,699, up to 64 GB, 307 GB/s memory bandwidth [7]. Mac mini M6: from $899 [7]. Refurbished M4 Pro 64 GB, 273 GB/s: $2,209 [8].
• macOS lets the GPU use about 75% of unified memory by default [4]. A 70B model at q4 (40–43 GB [3]) therefore needs a 64 GB machine. A 24B–32B model fits in 24–32 GB [4].
• Mistral Small 3.2 at 4-bit: 10–18 tok/s on M5 Pro, 9–16 tok/s on M4 Pro [4]. Tokens per second for Qwen3-30B-A3B on these chips: unknown (no source checked).
• Apple raised Mac mini prices because of an AI-driven memory shortage [7], so memory-heavy BOMs carry price risk.

3. Power draw (wall, Apple-measured) [9]

Power draw during sustained LLM inference specifically: unknown (Apple's max is a CPU stress test [9]).

4. Quality gap vs. cloud frontier models

• Long-context recall: frontier closed models keep about 95 on RULER at 1M tokens (Gemini 2.5 Pro reference) [10]. Qwen3-30B-A3B-2507 falls from 89.1 at 128K to 72.8 at 1M [1]. Below 32K the gap is small (about 1–3 points) [1][2].
• General instruction and format following (IFEval): roughly at parity with GPT-4o [1]. Against current frontier models (GPT-5.x, Claude 4.x) on the same tasks: unknown (no shared benchmark found).
• How far this transfers: only partly. RULER uses synthetic needles, and IFEval uses short prompts. Neither tests turning a messy, multi-speaker conversation into a coherent personal knowledge base. The real gap on CyberBible's task is unknown until we run our own eval, for example 20 real transcripts graded by blind rubric.

5. Per-unit cost

6. Can it run fully offline?

• Questionnaire onboarding: yes. A local web UI talks to a local model.
• Voice onboarding in person (talking to the box): yes. whisper.cpp for speech-to-text and Piper for text-to-speech both run without internet after the models are downloaded. Whisper model RAM: about 280 MB for base and about 2 GB for medium [11].
• Phone-call onboarding: not offline. A phone call has to travel over the carrier network, and a SIP or PSTN trunk provider sees the audio. Transcription and the AI can still run on the box, but the call itself cannot stay on the device. Offline alternative: a voice session in the browser on the local network.
• Model updates and first setup need either a download or a factory pre-load.
• Legal points that still apply: California's two-party recording consent covers any recorded call. CCPA delete and export rights still apply if CyberBible stores data or provides support. A DPA with model vendors is needed only for the cloud parts.

7. Side-by-side

8. Recommendation: hybrid, cloud first

1. Ship A first. It is already in progress, needs no inventory, and gets frontier quality.
2. Keep the model layer swappable from day one. MCP is already model-agnostic, and Markdown is the source of truth, so a "local mode" is just another MCP client or model endpoint.
3. Offer B later as a "Local Mode." Make it software that runs on a Mac the user already owns (Qwen3-30B-A3B or Mistral Small 3.2 through Ollama or MLX), not a box CyberBible has to manufacture. Sell a pre-configured Mac mini only if demand proves out.
4. Before deciding, run a CyberBible eval: same transcripts through a frontier model and through Qwen3-30B-A3B, graded blind. Until then, the size of the quality gap is unknown.

Sources

1. Qwen3-30B-A3B-Instruct-2507 model card: https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507
2. NVIDIA RULER results table: https://github.com/NVIDIA/RULER
3. Ollama llama3.3 tags: https://ollama.com/library/llama3.3/tags
4. Mistral Small 3.2 on Mac (RAM, speed): https://mlx.app/run/mistral-small-3.2
5. Mistral Small 3.2 model card: https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506
6. RULER leaderboard (Mistral-7B): https://www.sota2.com/research/sota/long-context-language-modeling-on-ruler-test-4k-256k-sweep
7. MacRumors Mac mini comparison guide
8. Apple refurbished M4 Pro 64 GB: https://www.apple.com/shop/product/g1kznll/a/Refurbished-Mac-mini-Apple-M4-Pro-Chip-with-12-Core-CPU-and-16-Core-GPU-Gigabit-Ethernet-
9. Apple Mac mini power consumption: https://support.apple.com/en-us/103253
10. Presenc AI: open-weight long-context models (closed-model reference)
11. whisper.cpp + Piper offline: https://turingpi.com/whisper-cpp-piper-tts-arm64-turing-pi-rk3588/