DISTRIBUTED NEURAL RUNTIME // FASTAPI + GGUF + P2P ENCRYPTED RELAY + SOUP

One Engine. Local Pooling. Global Connectivity.

Run massive open LLMs locally by pooling GPU VRAM across your LAN or connecting securely over the decentralized SHARD Network. Pure open source, zero telemetry without consent, and drop-in OpenAI/Anthropic API parity.

ARCHITECTURE DUAL RUNTIME (PYTORCH & GGUF)
LOCAL POOLING UDP 8001 / -ts RPC SHARDING
GLOBAL NETWORK ED25519 P2P & BLIND RELAY
API PARITY OPENAI & ANTHROPIC
SHARD Hero Network Animation
HYBRID TOPOLOGY LAN POOLING & BLIND RELAY READY

What Is SHARD?

SHARD transforms fragmented personal hardware into a unified AI inference engine. No expensive enterprise GPUs or datacenter NVLink interconnects required.

01

The Problem

Frontier open models (14B, 32B, 70B) exceed the 6-12 GB VRAM capacity of consumer GPUs. Developers are forced into expensive per-token cloud bills or painfully slow single-CPU offloading.

02

The Solution

SHARD provides a two-tiered system: pooling video memory and RAM across nearby computers on your LAN, and establishing secure peer-to-peer or encrypted relay tunnels across the global Internet.

03

The Advantage

Rather than all-reduce tensor splitting that demands 100 Gbps cables, SHARD uses layer sharding via llama.cpp RPC. Activations hop sequentially at ~8 KB per token, running fluidly over Wi-Fi, Ethernet, or broadband.

INTERACTIVE HARDWARE FIT SIMULATOR

PRE-FLIGHT MATH
Node 1 (Master) Free VRAM8 GB
Node 2 (Worker) Free RAM / VRAM16 GB
Calculating...
Model Memory Required: 4.70 GB
Total Pooled Capacity: 21.8 GB
Calculated -ts Split: 36% / 64%

Two Independent Connectivity Systems

SHARD implements two distinct networking architectures designed for different scopes: local memory pooling on your subnet, and decentralized cross-Internet connectivity.

LOCAL NETWORK

LAN Compute Pool

Zero-configuration memory pooling for computers connected to the same physical Wi-Fi, Ethernet switch, or overlay VPN (Tailscale, WireGuard).

  • Automatic UDP discovery beacons broadcast on port 8001 every 2 seconds.
  • Web Dashboard radar detects nearby hosts with instant 1-click Join/Share (no manual IP typing).
  • Proportional tensor split (-ts) assigns model layers based on exact device VRAM/RAM capacity.
  • Pairing codes for private subnets or --open flag for frictionless local hackathons.
shard lan join <SERVER_IP> --share
UDP 8001 Beacon TCP 8000 REST TCP 50052 RPC llama.cpp -ts
DECENTRALIZED GLOBAL

SHARD Network

Cryptographic, identity-driven networking for connecting remote hosts, clients, and workers across firewalls, NATs, and the public Internet.

  • Local Ed25519 cryptographic keypair (node_<hex>) with canonical JSON envelope signing.
  • REST Discovery directory with 30s heartbeats to publish and find active AI servers.
  • Direct STUN P2P hole punching with automatic fallback to end-to-end encrypted Blind Relay.
  • Cryptographic invite tokens with roles (client/worker), bandwidth quotas, and rate limiting.
shard network connect <TARGET> --invite-code <CODE>
Ed25519 Identity STUN NAT Traversal ChaCha20-Poly1305 AES-256-GCM WebSocket Relay

How SHARD Works

From standalone PC to multi-node cluster and global connectivity in four transparent steps. Zero proprietary lock-in.

CREATE Server Animation
STEP 01 CORE RUNTIME

INITIALIZE

Launch your SHARD server. The engine initializes hardware telemetry, evaluates local GPU/RAM budgets, and serves the Web Dashboard and OpenAI/Anthropic REST gateway.

shard start
SHARE Node Animation
STEP 02 LAN & GLOBAL PATH

POOL OR CONNECT

On LAN: nearby machines join with shard lan join <SERVER_IP> --share (or 1-click from the dashboard) to pool memory. Over Internet: remote clients connect using shard network connect <TARGET>.

shard lan join <SERVER_IP> --share
SERVER_IP is the dynamic LAN IP (e.g. 192.168.x.x or Tailscale 100.x.x.x) shown in shard lan status. Dashboard auto-detects nearby nodes via UDP 8001 beacons.
CONNECT Client Animation
STEP 03 SMART ROUTING

NEGOTIATE & ROUTE

SHARD selects the optimal connection path: direct LAN RPC for local memory slices, direct STUN P2P for remote nodes, or end-to-end encrypted Blind Relay forwarding.

shard network status
USE AI Agents Animation
STEP 04 STANDARD /V1 API

CONSUME

Point Cursor, Claude Code, Aider, or your custom agents to http://localhost:8000/v1. Stream tokens with sub-millisecond local response and zero cloud subscriptions.

http://localhost:8000/v1

Engine Features

Engineered from the ground up for low-latency inference, memory pooling, and developer ergonomics.

NETWORKING

Zero-Config LAN Discovery

UDP beacons announce cluster availability on port 8001. Nearby nodes appear in the dashboard automatically with instant one-click pairing.

DISTRIBUTED RUNTIME

llama.cpp RPC Layer Sharding

Proportional tensor split (-ts) assigns memory slices based on device capacity. Layer hops transfer only ~8 KB activation buffers.

GLOBAL CONNECTIVITY

Decentralized SHARD Network

Ed25519 cryptographic identity, public server publishing, STUN NAT traversal, and end-to-end encrypted ChaCha20/AES-GCM blind relay fallback.

DUAL BACKEND

PyTorch & GGUF Dual Engines

Run pure PyTorch models for research and development or high-efficiency quantized GGUF models for peak tokens-per-second throughput.

NATIVE ACCELERATION

C++20 & MASM AVX2 SIMD

Handcrafted 64-bit assembler kernels deliver sub-microsecond vector operations and zero-copy ring buffers on Windows x64 and Linux.

INTELLIGENT ADVISOR

Pre-Flight Hardware Fit Advisor

Run shard fit before downloading gigabytes. The analyzer mathematically determines if a model fits single-node or requires LAN pooling.

COLLABORATIVE TRAINING

SOUP Fine-Tuning Integration

Consent-gated telemetry collection, automatic PII redaction, dataset generation, and background LoRA/QLoRA training adapter.

STANDARD COMPATIBILITY

OpenAI & Anthropic /v1 Parity

Complete drop-in compatibility for /v1/chat/completions and /v1/messages with streaming Server-Sent Events (SSE) and tool calling.

SHARD Web Dashboard Interface
SHARD CLI Pre-Flight Advisor

System Architecture

Transparent modular architecture. Click any subsystem on the left to inspect its role, communication protocols, and source location.

CLIENT LAYER
Cursor / Claude Code / Aider / REPL
Standard /v1 HTTP/SSE consumers
API GATEWAY
FastAPI REST Server (server.py:8000)
OpenAI & Anthropic router + LAN admin
CLUSTER MASTER
llama-server + -ts Splitter (lan.py)
Device 0 coordinator + weight distributor
DISCOVERY DAEMON
UDP Beacon (Port 8001)
2s broadcast + open network flag
COMPUTE WORKER
ggml-rpc-server (Port 50052)
Contributes VRAM/RAM to cluster pool
NATIVE ACCELERATION
MASM AVX2 + Win32 C++20 DLL
Zero-copy ring buffer + vector math
GLOBAL ROUTER
SHARD Network Strategy (net_strategy.py)
Ed25519 identity + STUN P2P router
ENCRYPTED TUNNEL
Blind Relay Hub (net_tunnel.py)
ChaCha20/AES-GCM WebSocket forwarder
COLLABORATIVE TRAINING
SOUP Fine-Tuning (soup_adapter.py)
Consent-gated feedback + LoRA adapter
SUBSYSTEM INSPECTOR Cluster Coordinator

Master llama-server & Tensor Splitter

Executes llama-server with --rpc targets and -ts weights. Hosts model weights, coordinates inference layers, and handles master GPU/RAM device budget allocation.

llama-server --rpc flag -ts split VRAM Budget lan.py

Complete CLI Reference

Explore every command supported by the shard CLI parser. All commands verified against the current repository implementation.

shard start [--host IP] [--port N] [--daemon] [--no-browser]

Start the local SHARD inference server, initialize hardware diagnostics, and serve the Web Dashboard.

--host: bind IP (default 127.0.0.1)--port: port number (default 8000)--daemon: run in background
shard stop

Safely terminate any running background SHARD server daemon and clean up socket locks.

shard restart [--foreground]

Restart the running SHARD instance. Pass --foreground to attach output directly to terminal.

shard status

Print detailed runtime diagnostic report: PID, uptime, active model ID, memory consumption, and port bindings.

shard doctor

Verify local environment prerequisites: Python version, PyTorch, CUDA GPU driver, llama-server runtime, and AVX2 SIMD support.

shard list [--category [all|ideal|coding|reasoning|compact]]

List cached downloaded models and browse curated Hugging Face models with local hardware fit ratings.

shard fit <model_id> [--ctx <tokens>] [--no-fetch]

Pre-flight hardware fit advisor. Mathematically computes memory requirements and recommends GGUF quantization.

shard run [model_id]

Load a specified model into memory and launch an interactive terminal chat REPL with streaming tokens.

shard models-dir [path] [--reset]

View, configure custom disk directory, or reset storage location for downloaded model weights.

shard key [--gen]

Display existing local API keys for IDE integration, or generate new cryptographically secure keys.

shard lan status

Display current LAN cluster state: host IP, network mode (open/private), active member list, and pooled VRAM capacity.

shard lan create [name] [--open] [--model ID]

Create a local cluster host. Broadcasts UDP beacons on port 8001. Pass --open to allow peers to join without a pairing code.

shard lan join <SERVER_IP> [code] [--share] [--port N]

Connect to an active LAN server. Pass --share to contribute spare GPU VRAM or system RAM to the cluster pool.

shard lan diagnose [--public]

Inspect local network adapters, Tailscale/VPN interfaces, UPnP IGD gateway, and STUN NAT type to determine best join IP.

shard lan off

Gracefully disconnect from a connected cluster or stop hosting the local LAN network.

shard network status

Show node identity, public key fingerprint, hosting state, active transport (Direct P2P or Relay), and connected peers.

shard network identity [--regen]

Display local Ed25519 node public key, node ID (node_<hex>), and cryptographic envelope signature.

shard network servers [--query Q]

Query the configured Discovery directory to browse available public AI servers, models, and capacity.

shard network publish [--name NAME] [--open] [--bandwidth-limit-mbps N]

Register and publish your local inference server to Discovery and Blind Relay with optional rate limiting.

shard network connect <target> [--invite-code CODE] [--force-relay]

Connect to a remote SHARD server using direct STUN hole punching or fallback to end-to-end encrypted Blind Relay.

shard network invite [--role client|worker] [--quota-mb N] [--ttl-hours N]

Generate signed cryptographic invite tokens with role permissions, bandwidth quotas, and expiration lifetimes.

shard network discovery-serve [--port 8090]

Run a standalone, self-hosted Discovery server to maintain server directories for your team or organization.

shard network relay-serve [--port 8091]

Run a self-hosted Blind Encrypted Relay server for high-throughput WebSocket traffic forwarding between NAT-blocked peers.

shard training status

Display SOUP adapter status, feedback queue counts (approved/pending), dataset snapshots, and background jobs.

shard training doctor

Inspect SOUP integration prerequisites: Python version, CUDA readiness, trl, peft, and datasets dependencies.

shard training feedback --prompt "..." --response "..." [--approved]

Record consent-gated alignment feedback with automated client-side PII scrubbing.

shard training dataset --format [chatml|alpaca|sharegpt]

Compile approved interaction feedback into an immutable training dataset snapshot ready for fine-tuning.

shard training run --dataset <id> [--task sft|dpo] [--base <model>]

Execute background LoRA fine-tuning run on the selected dataset snapshot.

Installation & Quick Start

Choose your preferred environment. SHARD runs anywhere from standalone developer laptops to dedicated LAN clusters and headless servers.

1. Clone Repository
bash
git clone https://github.com/avelone123123/shard.git
cd shard
2. Install Package with Dependencies
bash
python -m pip install -e .
python -m pip install ".[full]"
3. Launch Web Dashboard & API Server
bash
shard start
4. Verify Environment Health
bash
shard doctor
bash (Local Cluster)
docker compose up -d
bash (Discovery & Blind Relay Stack)
docker compose -f deploy/docker/docker-compose.network.yml up -d
powershell / cmd
python native/build_native.py
shard test

Real-World Use Cases

How developers and teams deploy SHARD to solve real memory bottlenecks without subscription costs.

HOMELAB / DEV CLUSTERS

The Laptop + Desktop Supernode

Combine an RTX 3060 gaming laptop (6 GB VRAM) with an office desktop (16 GB RAM). Together, they pool 22 GB to run 32B coding models collaboratively.

POOLED: 22 GB // 32B QUANT FIT: YES
LOCAL DEVELOPMENT

Private AI Coding Agent Backend

Point Cursor, Claude Code, and Aider to http://localhost:8000/v1. Enjoy unmetered local completions and refactoring sessions without cloud rate limits.

DROP-IN /V1 API // ZERO PER-TOKEN FEES
ENTERPRISE PRIVACY

Air-Gapped & Multi-Office Clusters

Run completely offline on isolated local subnets, or connect distributed branch offices across the Internet using end-to-end encrypted blind relays.

LOCAL RUNTIME // ZERO DATA EXPOSURE

Frequently Asked Questions

Technical specifics, requirements, and hardware compatibility.

You can start with a single PC running Windows or Linux with a modern CPU supporting AVX2. Adding an NVIDIA CUDA GPU accelerates inference, and connecting additional LAN computers pools their memory automatically.

SHARD leverages llama.cpp native RPC protocol. The master node analyzes available memory budgets across all connected machines and splits model layers proportionally using the -ts flag.

No. Because SHARD uses layer sharding rather than all-reduce tensor splitting, only the hidden state activations (~8 KB per token hop) are transmitted between nodes. Standard Gigabit Ethernet or Wi-Fi is fully sufficient.

Yes. SHARD provides 100% compatible /v1/chat/completions (OpenAI) and /v1/messages (Anthropic) endpoints with streaming SSE support. Simply point your tool base URL to http://localhost:8000/v1.

An open network broadcasts an open flag in its UDP beacon. Other machines on your local subnet can discover and join the cluster with a single click in the Web Dashboard without typing a pairing code.

Yes. SHARD is 100% open source under the permissive MIT License. You are free to inspect, modify, fork, and deploy it for personal and commercial applications.

LAN Compute Pool is for local subnet memory pooling using UDP 8001 beacons and llama.cpp RPC. SHARD Network is a global decentralized protocol with Ed25519 node identities, STUN NAT traversal, and ChaCha20/AES-GCM blind encrypted relays for cross-Internet connectivity.

In LAN mode, the Web Dashboard continuously listens to UDP 8001 discovery beacons and shows nearby active clusters with one-click Join/Share buttons. The CLI command shard lan diagnose also auto-detects and displays the host recommended join IP.

Your server. Your network. Your SHARD.

Experience genuine local AI autonomy. Pool your hardware and run frontier models today.