Extension Development Hands-On

This is an end-to-end extension tutorial — from an empty directory to an extension that NeoMind can load and the Agent can call. By the end you'll be able to write your own extensions.

Prerequisite: read Extension SDK first for the trait, macros, capability, and ML-model lifecycle concepts. This page is the hands-on build flow.

Goal

We'll build a Counter extension: maintains a counter, exposes an increment command (callable by the AI Agent) and a counter metric (displayable on the dashboard). Small but covers the complete metric / command / FFI flow.

Step 1: Scaffold the Project

cargo new --lib counter-extension
cd counter-extension

Step 2: Configure Cargo.toml

[package]
name = "counter-extension"
version = "1.0.0"
edition = "2021"

[lib]
name = "neomind_extension_counter"        # prefix MUST be neomind_extension_
crate-type = ["cdylib", "rlib"]

[dependencies]
neomind-extension-sdk = "0.6.6"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
async-trait = "0.1"
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }

[profile.release]
panic = "unwind"     # REQUIRED — runner catches panics via unwind
opt-level = 3
lto = "thin"

Two critical settings:

• lib name prefix neomind_extension_ — runner looks up extensions by this convention
• panic = "unwind" — so an extension panic is caught rather than crashing the runner

Step 3: Implement the Extension

src/lib.rs:

use async_trait::async_trait;
use neomind_extension_sdk::prelude::*;
use std::sync::atomic::{AtomicI64, Ordering};

pub struct CounterExtension {
    counter: AtomicI64,
}

impl CounterExtension {
    pub fn new() -> Self {
        Self { counter: AtomicI64::new(0) }
    }
}

#[async_trait]
impl Extension for CounterExtension {
    fn metadata(&self) -> &ExtensionMetadata {
        static META: std::sync::OnceLock<ExtensionMetadata> = std::sync::OnceLock::new();
        META.get_or_init(|| {
            ExtensionMetadata::new("counter", "Counter", "1.0.0") // version is a String
                .with_description("A minimal counter extension")
                .with_author("You")
                .with_license("MIT")
        })
    }

    fn metrics(&self) -> Vec<MetricDescriptor> {
        static METRICS: std::sync::OnceLock<Vec<MetricDescriptor>> = std::sync::OnceLock::new();
        METRICS.get_or_init(|| vec![
            MetricDescriptor {
                name: "counter".into(),
                display_name: "Counter".into(),
                data_type: MetricDataType::Integer,
                unit: String::new(),
                min: None, max: None, required: false,
            },
        ]).clone()
    }

    fn commands(&self) -> Vec<ExtensionCommand> {
        static COMMANDS: std::sync::OnceLock<Vec<ExtensionCommand>> = std::sync::OnceLock::new();
        COMMANDS.get_or_init(|| vec![
            ExtensionCommand { // alias of CommandDescriptor
                name: "increment".into(),
                display_name: "Increment".into(),
                description: "Increment the counter value".into(), // also serves as the LLM hint
                parameters: vec![/* amount: Integer, default 1 */],
                ..Default::default()
            },
        ]).clone()
    }

    async fn execute_command(
        &self,
        command: &str,
        args: &serde_json::Value,
    ) -> Result<serde_json::Value> {
        match command {
            "increment" => {
                let amount = args.get("amount").and_then(|v| v.as_i64()).unwrap_or(1);
                let new_value = self.counter.fetch_add(amount, Ordering::SeqCst) + amount;
                Ok(serde_json::json!({ "counter": new_value }))
            }
            _ => Err(ExtensionError::CommandNotFound(command.into())),
        }
    }

    fn produce_metrics(&self) -> Result<Vec<ExtensionMetricValue>> {
        Ok(vec![ExtensionMetricValue::new(
            "counter",
            ParamMetricValue::Integer(self.counter.load(Ordering::SeqCst)),
        )])
    }

    fn as_any(&self) -> &dyn std::any::Any { self }
}

// The crucial line — FFI export
neomind_extension_sdk::neomind_export!(CounterExtension);

Pattern highlights:

• OnceLock<T> makes metadata / metrics / commands process-level singletons (avoids per-call allocation)
• execute_command is the Agent's entry point — the LLM fills arguments per the schema declared in commands()
• produce_metrics is the polling entry — runner calls it periodically and writes the values into telemetry.redb
• The last line neomind_export! turns the entire impl into an FFI entry point

Step 4: Build

cargo build --release

Artifact path:

• macOS: target/release/libneomind_extension_counter.dylib
• Linux: target/release/libneomind_extension_counter.so
• Windows: target/release/neomind_extension_counter.dll

Step 5: Install into NeoMind

Compile and package the extension directory into a .nep with the NeoMind CLI, then install it:

# Compile the extension directory (release build that also produces the .nep)
neomind extension build ./counter-extension
neomind extension validate dist/counter-1.0.0.nep

# Install (or click Upload Extension on the Web UI Extensions page to upload the .nep)
neomind extension install dist/counter-1.0.0.nep

Alternatively, drop the .nep into the server data directory's extensions/ folder and trigger a scan:

curl -X POST http://localhost:9375/api/extensions/sync

Step 6: Verify

# List extensions — counter should appear
curl http://localhost:9375/api/extensions

# Call the command
curl -X POST http://localhost:9375/api/extensions/counter/command \
  -H 'Content-Type: application/json' \
  -d '{"command":"increment","args":{"amount": 5}}'
# → {"success": true, "data": {"counter": 5}}

Or in AI Chat say: "Call the counter extension's increment command, add 3" — the LLM will auto-locate the increment command and invoke it (because commands() already declared it to the Agent tool system).

On the dashboard, add a value card widget, pick data source extension:counter:counter, and the live value shows up.

Step 7: Cross-Platform .nep Packaging

A single-platform .dylib only runs on macOS. To distribute you must compile all targets and package:

# Use cross or a GitHub Actions matrix to build every target
cross build --release --target x86_64-unknown-linux-gnu
cross build --release --target aarch64-unknown-linux-gnu
cross build --release --target x86_64-pc-windows-msvc
# Apple Silicon macOS (arm64)
cross build --release --target aarch64-apple-darwin

# Package into .nep (a zip archive; platform dirs use underscores)
mkdir -p nep/{linux_amd64,linux_arm64,windows_amd64,darwin_aarch64}
cp target/x86_64-unknown-linux-gnu/release/libneomind_extension_counter.so nep/linux_amd64/
# ... other platforms
cat > nep/manifest.json <<EOF
{ "format": "neomind-extension-package", "format_version": "2.0", "abi_version": 3,
  "id": "counter", "version": "1.0.0", "type": "native",
  "binaries": { "linux_amd64": "binaries/linux_amd64/libneomind_extension_counter.so", ... } }
EOF
cd nep && zip -r ../counter-1.0.0.nep .

.nep is the conventional archive format (see the NeoMind-Extensions repo's CI scripts for the reference). When a user installs via the Web UI one-click flow, the runner picks the binary matching the current platform.

Advanced Patterns

The following cover the most common patterns in real-world extension development.

Pattern 1: Network Extension (Weather API)

A weather extension is the canonical network extension — fetches an external API and produces metrics.

use neomind_extension_sdk::prelude::*;
use neomind_extension_sdk::capabilities::CapabilityContext;

pub struct WeatherExtension {
    config: std::sync::Mutex<WeatherConfig>,
}

struct WeatherConfig {
    api_key: String,
    city: String,
}

#[async_trait]
impl Extension for WeatherExtension {
    fn metadata(&self) -> &ExtensionMetadata {
        static META: OnceLock<ExtensionMetadata> = OnceLock::new();
        META.get_or_init(|| {
            ExtensionMetadata::new("weather", "Weather", "1.0.0")
                .with_config_parameters(vec![
                    ParameterDefinition {
                        name: "api_key".into(),
                        display_name: "API Key".into(),
                        description: "OpenWeatherMap API key".into(),
                        param_type: MetricDataType::String,
                        required: true,
                        ..Default::default()
                    },
                ])
        })
    }

    fn metrics(&self) -> Vec<MetricDescriptor> {
        // temperature, humidity, pressure
        // DataSourceId: extension:weather-forecast:temperature etc.
    }

    async fn configure(&mut self, config: &Value) -> Result<()> {
        let cfg = self.config.lock().unwrap();
        // update api_key / city from config JSON
    }

    async fn execute_command(&self, cmd: &str, args: &Value) -> Result<Value> {
        match cmd {
            "fetch" => {
                // HTTP goes through the sync `ureq` crate directly (same as the
                // official weather-forecast extension — there is no "network"
                // capability; network access bypasses the capability system)
                let resp: Value = ureq::get(&format!(
                    "https://api.openweathermap.org/data/2.5/weather?q={}&appid={}",
                    cfg.city, cfg.api_key
                )).call()?.into_json()?;

                // Write virtual device metrics via the device_metrics_write capability
                let ctx = CapabilityContext::default();
                ctx.invoke_capability("device_metrics_write", &json!({
                    "device_id": "virtual-weather",
                    "metric": "temperature",
                    "value": resp["main"]["temp"]
                }));
                Ok(json!({"status": "ok"}))
            }
            _ => Err(ExtensionError::CommandNotFound(cmd.into())),
        }
    }

    fn as_any(&self) -> &dyn std::any::Any { self }
}

neomind_export!(WeatherExtension);

Pattern 2: Streaming (Video Frames)

A video analysis extension uses the streaming API — each frame enters process_chunk() and returns detections.

pub struct YoloVideoExtension {
    model: OnceLock<YoloModel>,
}

#[async_trait]
impl Extension for YoloVideoExtension {
    fn stream_capability(&self) -> Option<StreamCapability> {
        Some(StreamCapability {
            supported_data_types: vec![StreamDataType::Binary],
            max_chunk_size: 64 * 1024,
            preferred_chunk_size: 16 * 1024,
            max_concurrent_sessions: 5,
            mode: StreamMode::Stateless, // stateless: each frame independent
            direction: StreamDirection::Download,
            flow_control: FlowControl::default(),
            config_schema: None,
        })
    }

    async fn process_chunk(&self, chunk: DataChunk) -> Result<StreamResult> {
        let model = self.model.get_or_try_init(|| YoloModel::load("yolov8n.onnx"))?;
        let detections = model.infer(&chunk.data)?;

        // data is a byte stream (JSON-serialized), extras ride in metadata
        let mut result = StreamResult::json(
            Some(chunk.sequence),
            chunk.sequence + 1,
            serde_json::to_value(&detections)?,
            0.0,
        )?;
        result.metadata = Some(json!({"frame_id": chunk.sequence}));
        Ok(result)
    }

    // ... other methods
}

Pattern 3: Push Mode (Sensors)

Push mode is for extensions that proactively produce data (e.g. serial sensors). The extension pushes via output_sender.

use tokio::sync::mpsc;
use std::sync::Arc;

pub struct SensorPushExtension {
    sender: std::sync::Mutex<Option<Arc<mpsc::Sender<PushOutputMessage>>>>,
}

#[async_trait]
impl Extension for SensorPushExtension {
    fn set_output_sender(&self, sender: Arc<mpsc::Sender<PushOutputMessage>>) {
        *self.sender.lock().unwrap() = Some(sender);
    }

    async fn start_push(&self, session_id: &str) -> Result<()> {
        let sender = self.sender.lock().unwrap().clone()
            .ok_or(ExtensionError::ExecutionFailed("no sender".into()))?;

        // Start background collection task; PushOutputMessage.data is a byte
        // stream (data_type declares the MIME type)
        let session_id = session_id.to_string();
        tokio::spawn(async move {
            let mut sequence = 0u64;
            loop {
                let value = read_sensor(); // your collection logic
                let msg = PushOutputMessage {
                    session_id: session_id.clone(),
                    sequence,
                    data: serde_json::to_vec(&json!({ "temperature": value })).unwrap_or_default(),
                    data_type: "application/json".into(),
                    timestamp: chrono::Utc::now().timestamp(),
                    metadata: None,
                };
                if sender.send(msg).await.is_err() { break; }
                sequence += 1;
                tokio::time::sleep(Duration::from_secs(5)).await;
            }
        });

        Ok(())
    }

    async fn stop_push(&self, _session_id: &str) -> Result<()> {
        // Stop the background task (via channel close or cancel token)
        Ok(())
    }
}

Pattern 4: Event Subscriptions

Extensions can subscribe to platform events (e.g. device online, rule triggered) and respond in handle_event().

fn event_subscriptions(&self) -> &[&str] {
    &["device.online", "rule.triggered"]
}

fn handle_event(&self, event_type: &str, payload: &Value) -> Result<()> {
    match event_type {
        "device.online" => {
            let device_id = payload["device_id"].as_str().unwrap();
            // Initialize when a device comes online...
        }
        "rule.triggered" => {
            // Rule-triggered integration logic...
        }
        _ => {}
    }
    Ok(())
}

Pattern 5: Configuration Hot-Reload

When a user changes config in the Web UI, configure() is called. Use a Mutex to guard config state:

pub struct MyExtension {
    config: std::sync::Mutex<MyConfig>,
}

async fn configure(&mut self, config: &Value) -> Result<()> {
    let new_cfg = MyConfig {
        api_key: config["api_key"].as_str().unwrap_or("").to_string(),
        interval: config["interval"].as_u64().unwrap_or(60),
    };
    // Validate
    if new_cfg.api_key.is_empty() {
        return Err(ExtensionError::InvalidArguments("api_key required".into()));
    }
    *self.config.lock().unwrap() = new_cfg;
    Ok(())
}

.nep Package Structure

A complete .nep package is a ZIP archive containing multi-platform binaries + metadata:

my-extension-1.0.0.nep (ZIP)
├── manifest.json               ← extension metadata + binaries platform mapping
├── binaries/
│   ├── darwin_aarch64/
│   │   └── extension.dylib
│   ├── linux_x86_64/
│   │   └── extension.so
│   ├── linux_aarch64/
│   │   └── extension.so
│   └── windows_x86_64/
│       └── extension.dll
├── frontend/                   ← (optional) dashboard component bundle (ships with a frontend.json when bundled)
└── models/                     ← (optional) ML model files
    └── yolov8n.onnx

manifest.json example (excerpt, matching real packages):

{
  "format": "neomind-extension-package",
  "format_version": "2.0",
  "abi_version": 3,
  "id": "my-extension",
  "name": "My Extension",
  "version": "1.0.0",
  "sdk_version": "2.0.0",
  "type": "native",
  "binaries": {
    "darwin_aarch64": "binaries/darwin_aarch64/extension.dylib",
    "linux_x86_64": "binaries/linux_x86_64/extension.so",
    "windows_x86_64": "binaries/windows_x86_64/extension.dll"
  }
}

The runner automatically selects the binary matching the current platform on load.

Reference Implementations

The NeoMind-Extensions repo has complete working examples for every pattern:

• weather-forecast — a simple network extension (no ML model)
• image-analyzer — ML-model lazy-load pattern (YOLOv11)
• yolo-video — streaming video processing
• yolo-device-inference — integration with NE301/NE101 cameras
• home-assistant-bridge — third-party system integration

Reading one of these is more illuminating than any doc.

Common Pitfalls

Symptom	Cause
Load error "symbol not found"	lib name prefix is not neomind_extension_
Extension panic permanently disables it	panic = "abort" (must be unwind)
Command call returns "permission denied"	Missing the matching capability declaration
Agent can't see your command	commands() not implemented, or description is empty (the LLM relies on it to understand the command)
Dashboard can't find your metric	metrics() not implemented, or the name is misspelled vs the DataSourceId
Cross-platform distribution fails	Missing some target platform binary, or the binaries field in the .nep's manifest.json is incomplete

Next Steps

• Open a PR to NeoMind-Extensions so the community can use your extension
• Extension command HTTP invocation details → REST API Reference
• Dashboard component development (if your extension ships visualizations) → Dashboard Component Dev
• Device metrics as extension data sources → Device Type Development

Practical Case Studies

This page is the API reference. To see how these APIs are used in real engineering, and why designs were chosen, read:

• Case Studies Overview
• #1 weather-forecast — Starter data extension
• #2 yolo-device-inference — AI inference extension
• #3 yolo-video — Streaming extension
• #4 onvif-bridge — Standard protocol bridge
• #5 uink-rms-bridge — Production-verified bridge

---

Last updated: 2026-09-08