Troubleshooting

Start with the symptom instead of reading the page from top to bottom. Commands assume an SSH session on the device; curl commands require a valid Bearer token. Before flashing, deleting a model, or cleaning files, confirm the backup and rollback path.

Start here

Symptom	Go to	First check
Web console does not open	Device & Network	Device IP, subnet, ping, and HTTPS
Web preview is black or RTSP has no video	Video & Streams	Stream switch, port, and camera-daemon
Model is loaded but produces no result	AI & Models	Model state, permissions, input size, and threshold
App exits or keeps restarting after installation	Apps & Containers	App logs, resources, and manifest permissions
Event publish succeeds but the receiver gets nothing	Events & Integrations	Topic, permissions, and subscriber connection
Upload fails or the disk is full	Storage & Disk	/, /data, logs, and core dumps
Flashing fails or a peripheral does not respond	Flashing & Peripherals	Wiring, serial link, and firmware procedure
A service fails to start or a socket is unreachable	System & Services	systemctl, journalctl, and /run/aipc

1. Device & Network

1.1 Cannot Access the Web Console

Symptom: The device web page does not open.

Check in this order:

1. Confirm the computer and device are on the same subnet; the default device subnet is 10.0.0.x.
2. Run ping <device_ip>. If it fails, check the Ethernet or PoE connection, the computer IP, and the firewall.
3. If the device has used a DHCP router, check the current address in the router admin page; DHCP may have changed it.
4. Open https://<device_ip> and accept the self-signed certificate.
5. If the network is reachable but the page still does not open, check the platform service over SSH:

ssh root@<device_ip>
aipc-cli system health
systemctl status platform-api

1.2 Forgotten Web Password

Symptom: You cannot log in to the Web console.

• If you can still log in, use Device Info → Change Password, or call POST /api/v1/system/password.
• If the test device still uses the default credentials admin / password, log in and change them immediately.
• If the changed password is lost, aipc-cli has no password-reset command; contact technical support or reflash the firmware.

The device currently has no factory-reset API. Reflashing has the same effect as a factory reset, so back up anything that must be kept first.

1.3 Lens Control Malfunction

Symptom: Focus, zoom, or iris does not respond or behaves abnormally.

Check the camera service, then read the lens state; reset the lens zero point only when needed:

systemctl status camera-daemon
grpcurl -plaintext -d '{}' unix:///run/aipc/device-control.sock aipc.device.DeviceControl/GetLensStatus
grpcurl -plaintext -d '{}' unix:///run/aipc/device-control.sock aipc.device.DeviceControl/LensResetZero

If the lens still does not respond, the issue is usually in the lens motor or HAL control path. The interface is defined in platform/device-control/proto/device.proto.

1.4 Serial / RS-485 Communication Failure

Symptom: An external RS-485 device, such as a PTZ or sensor, does not respond.

Distinguish the two serial links: the external RS-485 baud rate is set by the app through Rs485Init; /dev/ttyS0 @ 921600 is the internal host link between the processor board and interface-board MCU, not a peripheral interface.

Troubleshoot peripherals in §7.4 Alarm Input / Wiegand / RS-485. Check A/B polarity, shared ground, power, baud rate, device address, and protocol frame.

1.5 Browser Compatibility & Fallback

Browser	Minimum	Support	Known issue	Handling
Chrome	88+	Full	—	Preferred
Edge	88+	Full	—	Use directly
Firefox	78+	Basic	No WebCodecs	Use MSE playback
Safari	14+	Partial	No WebCodecs	Falls back to MSE
Mobile browsers	—	Limited	Limited performance	Use a desktop browser

Chrome 88+ or Edge 88+ is recommended. Safari's MSE fallback may perform worse.

1.6 WebSocket Failures at the Access Layer

Symptom	Check first	Handling
1006 abnormal closure	platform-api and port 443	Check the service, firewall, and network
401/403	Whether the token has expired	Log in again for a new token
Black screen	WebSocket state and SPS/PPS reception	Refresh, then check service logs
Artifacts or mosaic	Packet loss and decoder	Switch browser and check network quality
High latency	Network delay and buffering	Ensure LAN bandwidth and check the encoding GOP

Web preview uses MJPEG by default. If HD preview is black, the verified temporary workaround is to set HD_PREVIEW_ENABLED=0 for the app so it falls back to MJPEG. The HD path can become the default again after the platform exposes H.264 through nginx with wss://.

2. Video & Streams

2.1 RTSP Pull Failure

Symptom: A player or puller cannot connect to RTSP, or connects without video.

Check in order:

1. Confirm the pulling side is on the same subnet and the device IP has not changed through DHCP.
2. Confirm aipc-cli stream list shows the target stream enabled and the RTSP switch saved in the Web console.
3. Use rtsp://<device_ip>:8554/main; the port is 8554.
4. Confirm the network policy allows port 8554.
5. If it still fails, check the service and client connection:

systemctl status camera-daemon
journalctl -u camera-daemon -n 50 --no-pager
ffmpeg -rtsp_transport tcp -i rtsp://<device_ip>:8554/main -t 10 -f null -

RTSP on :8554 currently has no authentication. Any client that can reach the device may pull the stream; isolate it at the network layer in production.

2.2 WebSocket Disconnections at the Integration Layer

Symptom: A third-party system frequently disconnects after joining the video stream.

Separate authentication failures from runtime disconnects: 401/403 usually indicates a token problem; a connection that drops after it is established points to client timeouts, network changes, or server errors.

journalctl -u platform-api --since "1 hour ago" | grep -i "websocket\|h264"
wscat --no-check -c wss://<device_ip>/api/v1/h264/main

For client reconnection and video configuration, see Video and Imaging.

2.3 Stuck in SIMULATION Mode with No Detections

Symptom: The app log contains Running in SIMULATION mode - no actual inference and detections remain at zero.

This is the SDK fallback when it cannot obtain a real video stream; it does not by itself prove an app logic error. Check whether camera-daemon is crashing:

ssh root@<device_ip> "systemctl status camera-daemon"
ssh root@<device_ip> "journalctl -u camera-daemon -n 20 --no-pager"

If the log contains dlopen(/data/aipc/lib/hal/hal-hailo15.so) failed, the firmware usually lacks the HAL library. Reflash an image that contains it. The app will produce real detections only after the camera service recovers.

3. AI & Models

3.1 Model Imported but No Detections

Symptom: The model is imported, the app is running, but no detections or inference results appear.

Check these five items instead of looking only at the NPU Worker:

1. Models or aipc-cli model list must show the model as Loaded. If not, scan the model library and load it to the NPU.
2. permissions.models in app.yaml must contain the actual model ID.
3. If the threshold is too high, lower it until the pipeline produces a result, then raise it gradually.
4. The model input must match the platform's fixed 384×640 NV12 output.
5. The app must have permission for the raw-frame stream third or sub; main publishes encoded H.264 and cannot replace the inference input stream.

The standard HEF path is /data/aipc/models/<category>/ → scan → load. The corresponding endpoints are POST /ai/models/scan and POST /ai/models/<id>/load.

3.2 Inference Reports Model Not Found

Symptom: The log contains Model not found, or app-manager reports requires model X, but not found.

The model name in the app configuration must exactly match the registered device name. Query the device first, then update app.py or app.yaml:

from neoruntime_ipc_sdk import InferenceClient, FdMediaClient
print(InferenceClient().list_models())
print(FdMediaClient().list_streams())

You can also query models through the authenticated API:

curl -k https://<device_ip>/api/v1/ai/models \
  -H "Authorization: Bearer <token>"

3.3 Zero Detections from a High Threshold

Symptom: The model and stream are healthy, but the detection count is zero.

Temporarily lower the confidence threshold in the Models Detail dialog or app configuration. First confirm that detections appear, then raise the value until false positives and missed detections are acceptable for the business case.

3.4 Input Size Mismatch

Symptom: The log contains byte_size mismatch, or inference results are abnormal.

The platform preprocessor outputs fixed 384×640 NV12 frames. Use an HEF matching that size, such as hailo_yolov8n_384_640.hef; a mismatched model cannot run end to end on the current input path.

3.5 Inference Running but Zero Frames / Events

Symptom: The model is Loaded, the app is Running, the NPU Worker is active, but the app receives no frames or events and the logs show no clear error.

The app may still be attached to an expired container session. Check the app status heartbeat first; no heartbeat means that an active Worker does not prove that results are being returned.

For deeper diagnosis, observe inference failures:

strace -f -p $(pidof ai-runtime) -e trace=sendmsg -s 200 2>&1 | grep "Inference failed"

If errors such as Inference failed: -2814 appear, restart ai-runtime, scan and load the model again, then start the app. To separate NPU firmware issues from the ai-runtime software stack, run hailortcli run -t 5 <hef> in a maintenance window.

4. Apps & Containers

4.1 App Installation Failure

Symptom: aipc-cli app install or package upload fails.

Classify the failure first:

Failure area	Check first	Handling
Image import or pull	Network and image source	Check external connectivity and install logs
app.yaml parsing	YAML syntax and fields	Run yamllint app.yaml locally
Permission	Whether the user belongs to the aipc group	Correct the user permission and retry

journalctl -u app-manager -n 100 --no-pager
yamllint app.yaml

The old single-file upload interface curl -F app=@.aipc no longer works. Use the upload-image → upload-manifest → install-package flow.

4.2 Container Startup Failure

Symptom: Installation succeeds, but the app exits immediately or keeps restarting.

1. Read the app error with aipc-cli app logs <app-id>.
2. Check Dashboard memory and storage, then confirm system resources with df -h and free -h.
3. Confirm permissions includes the models, streams, and events the app needs.
4. If the error mentions image import or containerd, continue with Storage & Disk.

aipc-cli app logs <app-id>
journalctl -u app-manager --since "1 hour ago" | grep -i "container\|error\|failed"
systemctl status containerd

4.3 Health Check Failure

Symptom: app-manager marks the app unhealthy after startup.

app.yaml supports HTTP, command, and TCP probes. Reproduce the same probe type inside the container, then correct the target address, command, or port based on the result:

journalctl -u app-manager | grep -i healthcheck
aipc-cli app exec <app-id> -- /path/to/healthcheck.sh
aipc-cli app info <app-id>

Use curl for an HTTP probe, aipc-cli app exec for a command probe, and a port-listening check for a TCP probe.

4.4 Field-Verified Quick Reference

Symptom	Cause	Handling
apk add ... I/O error during build	Intermittent Docker Desktop, buildx, or network failure	Retry the build and check the image source
Start returns DeadlineExceeded	First image import exceeds the 10-second gRPC timeout	Wait for import to finish, then retry start
Log fails with json.tool	Response is NDJSON, one JSON object per line	Parse line by line, not as an array
curl -F app=@.aipc returns a JSON parse error	Old single-file upload API is unavailable	Use the staged upload flow

5. Events & Integrations

5.1 Event Publish Failure

Symptom: Publishing an event fails, or the subscriber receives nothing.

Confirm three items:

1. event-bus is running.
2. The Topic uses app/<app_id>/<event>, for example app/person_alert/person_detected.
3. app.yaml declares the publish Topic in permissions.events.publish.

systemctl status event-bus
journalctl -u event-bus -n 100 --no-pager
aipc-cli event publish app/demo/started '{"message":"test"}'

5.2 Subscription Failure / No Events Received

Symptom: The subscriber connects but receives no expected event.

Confirm that the subscription Topic exactly matches the published Topic and that permissions.events.subscribe declares it. The subscriber must also connect to the device Event Bus; in isolated network mode, the device-internal bus is not an external service.

systemctl status event-bus
aipc-cli event subscribe "app/<your_app>/*"

Events are currently pushed only over WebSocket in real time; there is no REST history endpoint, and /api/v1/events returns 404. The server receives all Topics, so filter by the Topic field at the receiver.

5.3 Topic Permissions & Wildcards

Topics are hierarchical and separated by /:

Syntax	Match scope	Example
*	One level	app/demo/* matches app/demo/started, not a deeper path
#	Multiple levels	MQTT-style multi-level matching

Every Topic an app publishes or subscribes to must be declared in app.yaml under permissions.events; otherwise the runtime rejects it.

6. Storage & Disk

6.1 Low Disk Space

Symptom: Dashboard shows high disk usage, or app startup, upload, or logging fails.

Locate the full partition before touching files:

df -h / /data
du -sh /data/aipc/* /home/root/* 2>/dev/null | sort -rh | head

For routine cleanup, use Maintenance → File Manager to remove old logs, unused apps, models, and packages. If container logs keep growing, adjust the app log policy; consider microSD expansion above 80% usage.

6.2 containerd Partition Mismatch

Symptom: Starting a large image reports parent snapshot sha256:... or no space, while the large partition still has space.

Older devices may place the deployment root at /opt/aipc on the small root partition; factory devices usually use /data/aipc on the large partition. Check the containerd root first:

df -h / /data
grep '^root' /etc/containerd/config.toml

If root still points to the small partition, back up the configuration and stop the related services before following the migration procedure in Deployment & Operations. Clean the old directory only after the migration is verified; do not delete unconfirmed containerd data.

6.3 App Volume Directory Missing

Symptom: Startup reports error mounting "/data/aipc/data/<id>" ... no such file or directory.

app-manager does not create host directories declared in app.yaml. Confirm the app ID, then create the directories:

ssh root@<device_ip> "mkdir -p /data/aipc/data/<app-id> /data/aipc/logs/<app-id>"

6.4 Root Partition Filled by Logs or Core Dumps

Symptom: Upload returns no space left, although the app image is not large.

Locate large files on the root partition:

df -h /
du -sh /data/aipc/* /home/root/* 2>/dev/null | sort -rh | head

Confirm each file's purpose and export logs needed for support before removing stale logs or /home/root/*.core. Do not delete the evidence required to diagnose the original failure.

7. Flashing & Peripherals

7.1 Flashing Fails or Aborts Midway

Symptom: SPI Flash boot-chain flashing fails, times out, or crashes midway.

Common causes are an unstable UART connection, a wrong baud rate, a firmware-package mismatch, or missing mkenvimage. Follow System Flashing §2 to recover the boot chain, then repeat §3 to flash the system.

7.2 Recovering from an Interrupted Upgrade

Symptom: A power or network interruption during a U-Boot TFTP upgrade leaves the device unbootable.

Follow System Flashing §2 to recover the boot chain, then repeat §3 to flash the system. Confirm power, Ethernet, and the complete TFTP file before retrying.

7.3 U-Boot Won't Start

Symptom: There is no U-Boot output on the serial console after power-on, or boot is stuck.

Follow System Flashing to recover the boot chain. This affects the boot path, so use a firmware package that matches the hardware.

7.4 Alarm Input / Wiegand / RS-485 Runtime Issues

When a peripheral does not work, check the physical layer, app configuration, and platform service in that order:

1. Physical layer: verify power, terminal wiring, A/B polarity, and shared ground.
2. Configuration: call Rs485Init before use, with the correct baud rate, address, and protocol frame.
3. Service layer: inspect device-control logs; if the issue remains, collect logs and wiring details.

The following are current platform limits, not configuration errors:

Symptom	Current cause	Handling
Alarm input triggers but no event is reported	No platform consumer for EV_ALARM_IN	Currently unavailable; wait for firmware support
Web Alarm Input Level has no effect	The control changes local state only; no API or MCU AIN_SET command is connected	Currently unavailable
Wiegand cannot receive a card-reader input	The current path is GPIO output only; there is no reader input or protocol encoder	Use it only as an access-control output
On-device Pan/Tilt call fails	The MCU host-link protocol has no corresponding PTZ command	Implement the protocol in the app for an external RS-485 PTZ
RS-485 receives no data	Initialization, wiring, ground, or protocol frame is wrong	Follow the three-layer order above

See Hardware Wiring and Interface Board for terminal definitions.

8. System & Services

Use this section for service-level diagnosis. For a field issue, complete the matching symptom section first.

8.1 General Troubleshooting Flow

Order	What you see	Next step
1	A command or page fails	Record the time, error code, and affected function
2	Service is not active	Run systemctl status <service>, then inspect startup logs
3	Socket is missing	Run ls -la /run/aipc/*.sock, then inspect the service process
4	Service is online but requests fail	Check permissions, dependencies, parameters, and resources

8.2 Service Startup Failure

Start with failed services and their startup logs:

systemctl status ai-runtime camera-daemon app-manager event-bus device-control device-discovery platform-api
systemctl --failed
journalctl -u <service-name> -b --no-pager

If the service exists but its interface is unreachable, check the Unix socket:

ls -la /run/aipc/*.sock
nc -U /run/aipc/ai-runtime.sock

If the socket is missing, repair the service first. If access is denied, check the /run/aipc directory and socket owner/group. Do not kill a process or delete a socket without confirming which service owns it.

8.3 Common Startup Issues & Socket Permissions

Log or symptom	Check first	Handling direction
Dependency is not ready	systemctl status <upstream-service>	Restore the upstream service, then restart the current one
Socket is occupied	ls -la /run/aipc/*.sock and the owning process	Confirm ownership before handling a stale process
permission denied	Socket and the /run/aipc permissions	Correct the user/group; do not open access to everyone
Binary or config is missing	Service ExecStart and config path	Repair the package or config; do not hide the error with an empty file
YAML parse failure	The referenced YAML file	Back up first, then fix syntax with yamllint

8.4 API Request Failures

Status	Meaning	Check first
401	Unauthenticated or expired token	Log in again and replace the token
403	Insufficient permission	Check user and app permissions
404	Path or resource does not exist	Verify the API path and resource ID
500	Internal service error	Read the platform-api logs
503	Service temporarily unavailable	Check dependencies and service state

8.5 Log Level Adjustment

The device uses configuration files under /data/aipc/etc/*.yaml; the repository's configs/ directory contains templates only. For temporary extra logging, back up the configuration, adjust log_level, reload the service according to its startup method, and restore the original level afterward.

journalctl -u ai-runtime --since "1 hour ago" | grep -i error
journalctl -u ai-runtime | grep -E "timeout|connection refused|permission denied"

Common levels are debug, info, warn, and error. Do not leave debug enabled in production because logs can fill the disk quickly.

8.6 Performance Monitoring & Resource Checks

top -p $(pgrep -f ai-runtime)
free -h
df -h / /data
iostat -x 1 5

For service-specific status, query AI Runtime, the app, and device-control:

grpcurl -plaintext -d '{}' unix:///run/aipc/ai-runtime.sock aipc.inference.InferenceService/GetStats
aipc-cli app info <app-id>
grpcurl -plaintext -d '{}' unix:///run/aipc/device-control.sock aipc.device.DeviceControl/GetDeviceStatus

Appendix A: Error Codes

These are common business error codes returned by platform-api; the full definition is in platform/platform-api/handlers/response.go.

Code	Meaning	Code	Meaning	Code	Meaning
0	Success	1000	Unknown error	1001	Invalid request
1002	Invalid JSON	1003	Missing parameter	1004	Invalid parameter
2000	Unauthenticated	2001	Permission denied	2002	Token expired
2003	Invalid token	3000	Service unavailable	3001	Service timeout
3002	Service error	3003	gRPC error	3004	Database error
4000	Resource not found	4001	Resource already exists	4002	Resource exhausted
4003	Operation failed	5000	Model not found	5001	Model load failed
5002	Inference error	5003	Invalid model format	6000	App not found
6001	App installation failed	6002	App startup failed	6003	App stop failed
6004	App is running	6005	App is not running	7000	Device error
7001	PTZ error	7002	Camera error	7003	GPIO error
8000	File not found	8001	File upload failed	8002	File deletion failed
8003	Storage full	8004	Access denied	9000	SSH configuration error
9001	SSH service error	10000	Process not found	10001	Process termination failed

DELETE /ai/models/<id> may also delete the model file. If a factory HEF has no backup, it cannot be recovered after deletion; confirm the file copy before deleting.

Related Docs

• System Architecture · Platform Services Layer — service responsibilities and socket relationships
• System Flashing — boot-chain recovery and system flashing
• Hardware Wiring — peripheral wiring and terminal definitions
• Deployment & Operations — disk, OTA, and rollback