---
title: Troubleshooting
description: >-
  Learn how to diagnose and resolve common Paxeer node errors, including
  consensus, network, database, and AppHash mismatch issues.
keywords:
  - paxeer node
  - blockchain node
  - node configuration
  - bug
  - troubleshooting
---

> **For AI agents:** the complete documentation index is at [llms.txt](/llms.txt). Append `.md` to any page URL for its markdown version.

Understanding common errors and their solutions helps maintain a healthy node
operation.

## Common Error Codes

Here are the most frequent errors you might encounter and their solutions:

### Consensus Errors

When you encounter consensus errors, quick and appropriate action is essential:

```text
Error: "Consensus failure - height halted"
Solution: Check for network upgrades or chain halts
Command: paxd status
```

```text
Error: "Private validator file not found"
Solution: Restore validator key or check file permissions
Location: $HOME/.paxeer/config/priv_validator_key.json
```

```text
Error: "Duplicate signature"
Solution: IMMEDIATELY STOP NODE - potential double signing risk
Action: Check validator operation on other machines
```

### Network Errors

Network errors can prevent your node from participating in consensus:

```text
Error: "Dial tcp connection refused"
Solution: Check network connectivity and firewall rules
Commands:
  - netstat -tulpn | grep paxd
  - ufw status
```

```text
Error: "No peers available"
Solution: Verify peer connections and network config
Commands:
  - curl localhost:26657/net_info
```

### Database Errors

Database corruption can require immediate attention:

```text
Error: "Database is corrupted"
Solution: Reset database or restore from backup
Commands:
  - paxd tendermint unsafe-reset-all
  - cp -r backup/data $HOME/.paxeer/
```

### Diagnostic Commands

These commands help you investigate issues and monitor your node:

```bash
# Check node synchronization
paxd status

# Check validator status
paxd query staking validator $(paxd tendermint show-validator)

# Monitor real-time logs
journalctl -fu paxd -o cat

# View system resource usage
top -p $(pgrep paxd)
```

## AppHash Mismatch Errors

If you encounter an AppHash mismatch, you'll need to capture the state for comparison with a known good version:

```bash
# For state-commit (most non-archive nodes):
paxd debug dump-iavl <latest height>
```

Always include the app hash, commit hash, and block height from your logs when reporting issues.

### Identifying AppHash Errors

AppHash errors typically appear in logs as:

```text
ERR wrong Block.Header.AppHash. Expected [EXPECTED_HASH], got [ACTUAL_HASH]
block_id={"hash":"...","parts":{"hash":"...","total":1}} height=[HEIGHT]
```

**Common Causes:**

- Using incorrect node version during sync (ensure you're on the latest version)
- Corrupted or incorrectly applied snapshots
- Database inconsistencies from improper shutdowns
- Syncing with outdated or incompatible peers

**Resolution Steps:**

1. **Stop the node immediately.**
2. **Try a node rollback first:**, see [here](/node/troubleshooting#node-rollback)
3. **If rollback fails, restore from a fresh snapshot:**

   - Download a recent snapshot from trusted providers
   - Ensure you're using the correct node version
   - Verify peer configurations are up to date

4. **Restart the node and monitor logs for continued errors**

### Peer Connection Issues as AppHash Red Herrings

**Important Note:** Peer connection failures are often symptoms of underlying AppHash errors, not the root cause.

When you see extensive peer connection errors like:

```text
ERR failed to handshake with peer
ERR failed to send request for peers
ERR peer handshake failed endpoint={} err=EOF
```

**Don't focus solely on fixing peer connections first.** Instead:

1. **Scan your logs carefully** for AppHash errors that may appear intermittently
2. **Look for the actual error pattern:**
   ```text
   ERR wrong Block.Header.AppHash. Expected [HASH], got [HASH]
   ```
3. **Check if your node is stuck** at a specific height despite peer connection attempts

**Debugging Approach:**

1. **First, check for AppHash errors** in your logs (search for "wrong Block.Header.AppHash")
2. **If AppHash errors are found**, treat this as the primary issue
3. **Only focus on peer connection fixes** if no AppHash errors exist

### Peer Connection and Handshake Issues

**Identifying Peer Issues:**

Look for these error patterns in your logs:

```text
ERR failed to handshake with peer err="expected to connect with peer \"[EXPECTED_ID]\", got \"[ACTUAL_ID]\""
ERR failed to send request for peers err="no available peers to send a PEX request to (retrying)"
ERR peer handshake failed endpoint={} err=EOF module=p2p
```

**Common Causes:**

- Outdated peer configurations with mismatched node IDs
- Network infrastructure changes on peer side
- Firewall blocking connections on port 26656
- DNS resolution issues

**Resolution Steps:**

1. **Update peer configurations** with current node IDs. Use the seed peer:

   ```bash
   e9c56cbadc4a96b67f69dcaaa7b4691851e945ca@31.220.74.140:26656
   ```

2. **Verify network connectivity:**

   ```bash
   # Test connection to peer endpoints
   nc -zv 31.220.74.140 26656

   # Check if port 26656 is open for inbound connections
   netstat -tulpn | grep :26656
   ```

3. **Check current peer status:**
   ```bash
   curl http://localhost:26657/net_info | jq '.result.peers | length'
   curl http://localhost:26657/lag_status | jq .
   ```

### Sync Performance Issues

**Identifying Sync Problems:**

Monitor these indicators:

```bash
# Check sync status and lag
curl http://localhost:26657/lag_status | jq .

# Monitor if height is progressing
curl http://localhost:26657/status | jq '.result.sync_info'
```

**Common Solutions:**

1. **Increase packet payload size** for large block processing:

   ```toml
   # In config.toml [p2p] section
   max-packet-msg-payload-size = 1024000
   ```

2. **Optimize mempool settings** in `config.toml`:

   ```toml
   # In [mempool] section
   keep-invalid-txs-in-cache = true
   ttl-duration = "5s"
   ttl-num-blocks = 5
   ```

3. **If node gets stuck at specific height:**
   - Try restarting the node
   - If restart doesn't help, perform rollback
   - Consider taking a fresh snapshot

## Crash and Panic Debugging

For crashes, panics, or nil pointer exceptions:

- Capture at least 1,000 lines of logs leading up to the crash
- Or collect 15 minutes of log data, whichever provides more context
- Include the full stack trace if available

### Logging Configuration

```toml
# In config.toml
log_level = "debug"
log_format = "json"
```

Configure log rotation to manage storage effectively:

```bash
sudo tee /etc/logrotate.d/paxd << EOF
/var/log/paxd/*.log {
    daily
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 paxeer paxeer
    sharedscripts
    postrotate
        systemctl reload paxd
    endscript
}
EOF
```

## Other common Issues and Fixes

1. **Sync Problems**

   - Check available disk space (`df -h`)
   - Ensure proper peer connections (`curl http://localhost:26657/net_info`)
   - Verify firewall settings (port 26656 open)

2. **Performance Issues**

   - Monitor system resources (`htop` or `iotop`)
   - Check disk I/O performance (`iostat`)
   - Analyze network traffic (`iftop`)

3. **Database Issues**

   - Run database integrity checks using:

     ```bash
     paxd debug dump-db | grep -i error
     ```

     If errors are detected, consider restoring from a recent backup.

   - Consider pruning excessive historical data by adjusting `ss-keep-recent` in
     `app.toml` or running:

     ```bash
     paxd tendermint unsafe-reset-all --home $HOME/.paxeer --keep-addr-book
     ```

## Node Rollback

To rollback a node from an AppHashed state, you need to stop the node first.

Next, do a rollback with:

```bash
paxd rollback
```

Then, restart the node.

In case you see the following error while trying to rollback:

```bash
failed to initialize database: resource temporarily unavailable
```

This means that you did not shutdown the node properly. Try to shutdown or kill the `paxd` process directly in that case. If this doesn't help, restart your machine.

Then try the rollback steps again.