Prerequisites¶

MeshStor runs on bare-metal Kubernetes clusters with local NVMe drives. This page lists the hardware, software, and network requirements.

Kubernetes¶

• Kubernetes 1.27+ (tested with 1.34)

Hardware¶

Minimum node count equals replicaCount. 1 node is enough for replicaCount=1 (local storage mode — the volume goes through the MeshStor data path but only writes to one underlying partition). 2 nodes for replicaCount=2, 3 for replicaCount=3, and so on.

Each node needs at least one NVMe drive that:

• Has a GPT partition table with unpartitioned free space. MeshStor carves its own partitions out of that free space and can coexist with existing OS partitions — the whole drive is not required. Legacy MBR is not supported, and free space inside an LVM PV or LVM VG don't count, since MeshStor needs direct access to NVMe drive.
• Uses 4 KiB logical sector size. Recommended for best performance; 512-byte sectors also work.

Verify available drives:

lsblk -d -o NAME,MODEL,SERIAL,SIZE,LOG-SEC,TRAN | grep nvme

nvme0n1 WD_BLACK SN7100 1TB       25315F800925     931.5G    4096 nvme
nvme1n1 KINGSTON SKC3000S1024G    50026B7686E27E84 953.9G     512 nvme

Host OS¶

MeshStor requires a recent Linux kernel. The userspace tooling it invokes (xfsprogs, mdadm, nvme-cli) ships inside the CSI container image, so the kernel is the only host-side software floor.

• Kernel 6.11+ (7.0+ recommended).

!!! danger "Pre-6.11 kernels on NVMe-oF + MD RAID Data corruption. The LP#2075110 bug affects the exact topology MeshStor uses. RHEL / Rocky Linux / AlmaLinux 10.x, Ubuntu 24.04 and 26.04 have the fix backported.

udev: disable MD auto-assembly¶

MeshStor orchestrates MD RAID assembly itself. By default, udev auto-assembles MD arrays as soon as it sees devices tagged linux_raid_member, which races with MeshStor's reconciler and produces transient assembly failures. Install a udev rule that suppresses auto-assembly by flagging raid members with ANACONDA=1 (the same flag the RHEL installer uses):

sudo tee /etc/udev/rules.d/10-mdadm-no-udev-assemble.rules > /dev/null <<'EOF'
SUBSYSTEM!="block", GOTO="md_no_udev_end"
ACTION=="remove", GOTO="md_no_udev_end"
ENV{ID_FS_TYPE}=="", IMPORT{builtin}="blkid"
ENV{ID_FS_TYPE}=="linux_raid_member", ENV{ANACONDA}="1"
LABEL="md_no_udev_end"
EOF

sudo udevadm control --reload-rules
sudo udevadm trigger

Kernel Modules¶

MeshStor requires both NVMe-oF host and target kernel modules — each node exports its local partitions to peers (target side) and imports peer partitions in return (host side). Load them and ensure they persist across reboots:

# Required — NVMe-oF host and target modules for both TCP and RDMA transports
sudo modprobe nvme_tcp
sudo modprobe nvmet_tcp
sudo modprobe nvme_rdma
sudo modprobe nvmet_rdma

Persist across reboots:

sudo tee /etc/modules-load.d/meshstor.conf > /dev/null <<'EOF'
nvme_tcp
nvmet_tcp
nvme_rdma
nvmet_rdma
EOF

Verify the configfs target directory exists:

sudo ls /sys/kernel/config/nvmet/

discovery_nqn  hosts  ports  subsystems

Network¶

Node Addresses¶

Each node must be annotated with its NVMe-oF network address. The annotation doubles as an interface selector: set it to the IP of the NIC you want to carry replication traffic, which matters when a node has a dedicated storage NIC separate from its Kubernetes control-plane NIC:

# Required: TCP address (must be reachable from all other nodes)
kubectl annotate node <node-name> meshstor.io/nvme-over-tcp-address=<ip-address>

# Optional but recommended: RDMA address (enables RoCEv2 on Ethernet, or InfiniBand transport preference)
kubectl annotate node <node-name> meshstor.io/nvme-over-rdma-address=<ip-address>

If no annotation is set, MeshStor falls back to the node's InternalIP, which puts replication traffic on the same interface as Kubernetes control-plane traffic — acceptable for dev clusters, not recommended in production.

Ports¶

Both ports must be open on the host network (the node DaemonSet uses hostNetwork: true).

Example for firewalld-based systems (RHEL, Rocky Linux, AlmaLinux):

sudo firewall-cmd --permanent --add-port=4420/tcp
sudo firewall-cmd --permanent --add-port=4421/tcp
sudo firewall-cmd --reload

On ufw, nftables, or iptables, open the same ports with the equivalent command for your firewall.

Subnet Connectivity¶

MeshStor only pairs two nodes as replicas if they share a common subnet derived from their meshstor.io/nvme-over-tcp-address (and meshstor.io/nvme-over-rdma-address if set) annotations. Plan flat L2 or routed connectivity between any pair of nodes that may host replica partitions for the same volume — otherwise the controller will exclude one of them from the placement set and partition scheduling will silently skip that pair.

Block Device Selection¶

MeshStor only uses NVMe drives that are explicitly allowed on each node via a node label. Unlabelled nodes contribute no drives to the storage pool, so volume provisioning will skip them.

Option 1 — allow specific drives:

Identify drives by serial number, not kernel name. Kernel names like nvme0n1 are assigned in device-probe order and can shift across reboots or hardware changes — a renumbered drive would silently change which device MeshStor selects. Serials are stable; read them with lsblk -o +SERIAL:

# Use only these two drives, by serial (separated by ".." due to Kubernetes label value restrictions)
kubectl label node ms-01-01 meshstor.io/selected-block-devices=PHBT83440LGR016N..PHBT83450WA8016N

The label also accepts kernel names (meshstor.io/selected-block-devices=nvme0n1..nvme1n1), but only rely on those where device numbering is known to be stable.

Option 2 — allow any NVMe drive on the node:

kubectl label node <node-name> meshstor.io/selected-block-devices=any

The any value is safe: MeshStor carves partitions out of unallocated GPT free space only and never touches existing/unknown partitions.

Verification Checklist¶

Run on each node to confirm readiness:

# 1. Kernel 6.11+
uname -r

# 2. NVMe drives visible with 4 KiB LOG-SEC
lsblk -d -o NAME,MODEL,SIZE,LOG-SEC,TRAN | grep nvme

# 3. Each candidate drive is GPT and has unallocated free space
sudo parted /dev/<nvme> print free   # repeat per drive

# 4. NVMe-oF host and target modules loaded
lsmod | grep -E '^(nvme_tcp|nvmet_tcp|nvme_rdma|nvmet_rdma)\b'

# 5. Modules persist across reboots
cat /etc/modules-load.d/meshstor.conf

# 6. NVMe-oF target configfs available (it is important)
ls /sys/kernel/config/nvmet/

# 7. udev MD auto-assembly suppression rule installed
cat /etc/udev/rules.d/10-mdadm-no-udev-assemble.rules

# 8. Firewall ports 4420 (TCP) and 4421 (RDMA) open on the host network
sudo firewall-cmd --list-ports | tr ' ' '\n' | grep -E '^(4420|4421)/'   # or your firewall's equivalent

# 9. Nodes annotated with NVMe-oF addresses (RDMA optional)
kubectl get nodes -o custom-columns='NAME:.metadata.name,TCP:.metadata.annotations.meshstor\.io/nvme-over-tcp-address,RDMA:.metadata.annotations.meshstor\.io/nvme-over-rdma-address'

# 10. Nodes labelled for block device selection
kubectl get nodes -o custom-columns='NAME:.metadata.name,DEVICES:.metadata.labels.meshstor\.io/selected-block-devices'

What's Next¶

• Installation — deploy MeshStor to your cluster
• Tuning — optional performance knobs for fast fabrics
• Compatibility — supported Kubernetes versions, kernels, and Linux distributions
• Internals — understand the components before deploying