Paxeer Node Advanced Configuration & Monitoring
Optimize Paxeer node performance with advanced configuration, Prometheus and Grafana monitoring, and alerting.
Optimizing System Configuration
There are a virtually unlimited number of unique individual setups that cannot be covered in this document. As well, even similar builds and configurations can behave differently due to external factors, so your results may vary.
Here are some general guidelines to use as a starting point. Be cautious, make incremental changes, testing and observing before moving forward. Always focus on only one specific area at a time - avoid making changes to memory, storage, and CPU configs all at once. Diagnosing potential problems becomes nearly impossible otherwise.
Memory Management
The following settings in /etc/sysctl.conf can optimize memory usage and disk I/O patterns:
# Minimize swapping
vm.swappiness = 1
# Control disk write behavior
vm.dirty_background_ratio = 3
vm.dirty_ratio = 10
vm.dirty_expire_centisecs = 300
vm.dirty_writeback_centisecs = 100
Apply changes: sudo sysctl -p
Network Stack
The following settings in /etc/sysctl.conf may improve network performance:
# Increase connection handling capacity
net.core.somaxconn = 32768
net.core.netdev_max_backlog = 32768
net.ipv4.tcp_max_syn_backlog = 16384
# Optimize buffer sizes
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 87380 16777216
Storage Configuration
For NVMe drives, optimize I/O scheduling:
# Set IO scheduler
echo "none" > /sys/block/nvme0n1/queue/scheduler
# Set read-ahead buffer
blockdev --setra 4096 /dev/nvme0n1
# Set IO priority in systemd service
sudo tee -a /etc/systemd/system/paxd.service << EOF
[Service]
IOSchedulingClass=realtime
IOSchedulingPriority=2
EOF
# Configure disk mount options
sudo tee -a /etc/fstab << EOF
/dev/nvme0n1p1 /data ext4 defaults,noatime,nosuid,nodev,noexec,commit=60 0 0
EOF
Infrastructure Monitoring
Monitoring is one of the most critical components of network infrastructure. performance tuning, and alerting configuration for Cosmos-SDK/Tendermint nodes.
Prometheus Setup
First, install Prometheus:
wget https://github.com/prometheus/prometheus/releases/download/v2.42.0/prometheus-2.42.0.linux-amd64.tar.gz
tar xvf prometheus-2.42.0.linux-amd64.tar.gz
Example Prometheus configuration:
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'paxeer_node'
static_configs:
- targets: ['node1_ip:port']
metrics_path: /metrics
- job_name: 'node'
static_configs:
- targets: ['node2_ip:port']
Grafana Integration
Install and configure Grafana:
sudo apt install -y apt-transport-https software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
sudo apt update && sudo apt-get install grafana
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"gnetId": null,
"graphTooltip": 0,
"id": 1,
"links": [],
"panels": [
{
"alerting": {},
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fieldConfig": {
"defaults": {
"custom": {}
},
"overrides": []
},
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 0
},
"hiddenSeries": false,
"id": 2,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"nullPointMode": "null",
"options": {
"alertThreshold": true
},
"percentage": false,
"pluginVersion": "7.2.0",
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "tendermint_consensus_height",
"interval": "",
"legendFormat": "",
"refId": "A"
}
],
"thresholds": [],
"timeRegions": [],
"title": "Block Height",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"schemaVersion": 26,
"style": "dark",
"tags": [],
"templating": {
"list": []
},
"time": {
"from": "now-6h",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "Paxeer Node Metrics",
"uid": "paxeer_metrics",
"version": 1
}Alert Management
Install Alertmanager:
wget https://github.com/prometheus/alertmanager/releases/download/v0.25.0/alertmanager-0.25.0.linux-amd64.tar.gz
tar xvf alertmanager-0.25.0.linux-amd64.tar.gz
groups:
- name: validator_alerts
rules:
- alert: NodeDown
expr: up == 0
for: 5m
labels:
severity: critical
annotations:
summary: 'Node {{ $labels.instance }} down'
- alert: BlockProductionSlow
expr: rate(tendermint_consensus_height[5m]) < 0.1
for: 5m
labels:
severity: warning
annotations:
summary: 'Block production is slow on {{ $labels.instance }}'
- alert: ValidatorMissedBlocks
expr: increase(tendermint_consensus_validator_missed_blocks[1h]) > 0
labels:
severity: critical
annotations:
summary: 'Validator missing blocks'
- alert: ValidatorJailed
expr: tendermint_consensus_validator_status == 0
labels:
severity: critical
annotations:
summary: 'Validator has been jailed'
- alert: ConsensusStalled
expr: tendermint_consensus_height_status == 0
for: 5m
labels:
severity: critical
annotations:
summary: 'Consensus has stalled'Log Management
Loki Setup
Using Loki for log aggregation:
wget https://github.com/grafana/loki/releases/download/v2.8.0/loki-linux-amd64.zip
unzip loki-linux-amd64.zip
server:
http_listen_port: 9080
positions:
filename: /tmp/positions.yaml
clients:
- url: http://localhost:3100/loki/api/v1/push
scrape_configs:
- job_name: paxeer_logs
static_configs:
- targets:
- localhost
labels:
job: paxd_logs
__path__: /var/log/paxd/*.logLog Rotation
Configure logrotate to manage log files:
sudo tee /etc/logrotate.d/paxeer << EOF
/var/log/paxeer/*.log {
daily
rotate 14
compress
delaycompress
notifempty
create 0640 paxeer paxeer
sharedscripts
postrotate
systemctl reload paxd
endscript
}
EOF
Security Configuration
Network Security
UFW firewall configuration:
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 26656/tcp comment 'Paxeer P2P'
sudo ufw allow 26657/tcp comment 'Paxeer RPC'
sudo ufw allow 9090/tcp comment 'Paxeer gRPC'
sudo ufw enable
Rate Limiting
http {
limit_req_zone $binary_remote_addr zone=paxeer_rpc:10m rate=10r/s;
server {
listen 26657;
location / {
limit_req zone=paxeer_rpc burst=20 nodelay;
proxy_pass http://localhost:26657;
}
}
}
Validator-Specific Monitoring
Status Query
Query validator status through SDK:
paxd query staking validator $(paxd keys show --bech val -a <validator_keyfile_name>)
Query through REST API:
curl -s "http://localhost:1317/cosmos/staking/v1beta1/validators/<valoper_address>"
#!/bin/bash
MONIKER="$1"
API_URL="http://localhost:1317/cosmos/staking/v1beta1/validators?pagination.limit=500"
echo "Querying validators from $API_URL..."
VALIDATOR_DATA=$(curl -s "$API_URL" | jq -c --arg MONIKER "$MONIKER" '.validators[] | select(.description.moniker == $MONIKER)')
if [[ -z "$VALIDATOR_DATA" ]]; then
echo "No validator found with moniker: $MONIKER"
exit 1
fi
echo "Validator details:"
echo "$VALIDATOR_DATA" | jq '.'Critical Metrics
Monitor these validator-specific metrics:
# Check signing status
paxd query slashing signing-info $(paxd tendermint show-validator)
# Check current delegations
paxd query staking delegations-to $(paxd keys show -a $VALIDATOR_KEY)
Backup Management
#!/bin/bash
BACKUP_DIR="/backup/paxeer"
DATE=$(date +%Y%m%d)
NODE_HOME="/root/.paxeer"
# Create backup directory
mkdir -p $BACKUP_DIR
# Stop service
systemctl stop paxd
# Backup configuration
tar czf $BACKUP_DIR/paxeer-config-$DATE.tar.gz $NODE_HOME/config
# Backup data directory
tar czf $BACKUP_DIR/paxeer-data-$DATE.tar.gz $NODE_HOME/data
# Backup key files
tar czf $BACKUP_DIR/paxeer-keys-$DATE.tar.gz $NODE_HOME/keyring-file
# Start service
systemctl start paxd
# Remove backups older than 7 days
find $BACKUP_DIR -type f -mtime +7 -name '*.tar.gz' -delete
# Log backup completion
echo "Backup completed successfully on $(date)" >> $BACKUP_DIR/backup.logHost System Monitoring
Resource Usage Tracking
Install and configure node_exporter:
wget https://github.com/prometheus/node_exporter/releases/download/v1.5.0/node_exporter-1.5.0.linux-amd64.tar.gz
tar xvf node_exporter-1.5.0.linux-amd64.tar.gz
Add to Prometheus configuration:
scrape_configs:
- job_name: 'node'
static_configs:
- targets: ['localhost:9100']
For specific customizations or additional metrics, consult the Paxeer technical communities.