Troubleshoot a Node

Collect a small set of evidence before changing configuration: the running binary, configured network, current height, peer count, and recent logs. This lets you distinguish a local service problem from a network-wide halt or an application-state mismatch.

Confirm the process and endpoints

The following examples assume a Linux systemd service named paxd and default local ports. Substitute your service name and bound ports if they differ.

paxd version --long
systemctl status paxd --no-pager
systemctl show paxd -p ExecStart -p User
ss -lntp
curl --fail --silent --show-error http://127.0.0.1:26657/status | jq '.result | {node_info, sync_info}'
curl --fail --silent --show-error http://127.0.0.1:26657/net_info | jq '.result.n_peers'
curl --fail-with-body --silent --show-error http://127.0.0.1:8545   -H 'Content-Type: application/json'   --data '{"jsonrpc":"2.0","id":1,"method":"eth_chainId","params":[]}'

For the Paxeer network documented here, compare the Cosmos network with hyperpax_125-1 and the EVM result with 0x7d. A local development cluster can use a different identifier. The binary found in your shell can differ from the absolute path in the service’s ExecStart.

Match the symptom to the next check

Capture a sync stall

Capture the height and latest block time twice, a short interval apart. Compare both samples with a known-good node on the same network. A changing height with increasing lag points toward throughput; an unchanged height with a repeating error points toward a specific blocker.

curl --fail --silent --show-error http://127.0.0.1:26657/status   | jq '.result.sync_info | {latest_block_height, latest_block_time, catching_up}'
journalctl -u paxd --since '15 minutes ago' --no-pager -o short-iso
df -h
df -i
free -h

Check storage capacity and inode exhaustion separately. For slow disks, use the host’s I/O monitoring to compare read latency, write latency, and saturation with the stalled period. Avoid changing several performance settings at once; preserve one baseline to compare against.

Investigate AppHash mismatches

An AppHash mismatch means the node’s computed application state disagrees with the block header. Record the exact height, expected and actual hashes, binary revision, genesis identity, and snapshot provenance. Use the binary assigned to that network’s upgrade schedule at the failing height.

journalctl -u paxd --since '1 hour ago' --no-pager   | rg -n -i 'apphash|app hash|panic|corrupt|upgrade|handshake'

# Set PAXEER_NODE_HOME to the data directory used by the service.
sha256sum "${PAXEER_NODE_HOME:?Set the active node home}/config/genesis.json"

Handshake errors can follow a state mismatch; start with the earliest application error rather than repeatedly rotating peers. Reconcile the release, upgrade schedule, and snapshot with a known-good operator before choosing recovery steps.

Handle database locks and recovery

A database lock often means another process still holds the same data directory. Inspect the service and process list, including containers, before attempting another start. Keep one writer per data directory; stop the process holding the database before recovery.

Rollback, snapshot restore, and reset commands change durable state and depend on the storage engine and release. Preserve a backup and use the recovery procedure for that exact node version. For validator recovery, preserve the existing signing state and confirm that only one signer is active.

For a duplicate-signing alert, stop the affected signer and establish which instance owns signing before restarting. Retain the validator keys and signing history throughout recovery.

Prepare an actionable report

• Node role, network identifiers, binary version and checksum, and startup command.
• Stalled or failing height and a comparison with a healthy node.
• At least the logs leading into the first failure, including the complete panic stack when present.
• Disk, memory, peer count, recent upgrade or restore activity, and snapshot checksum.
• The relevant configuration changes, with RPC credentials and signing material removed.

Use Operators Guide for ongoing monitoring, Configuration for settings, and Run a Node for initial setup.