Vitastor

New features

Implement QCOW2 image/snapshot export via qemu-img (bdrv_co_block_status in the driver)
Remove OSDs from PG history during vitastor-cli rm-osd to prevent left_on_dead PG states after deletion
Add a new recovery_pg_switch setting to mix all PGs during recovery, to almost fully reduce the probability of ENOSPC during rebalance
Introduce partial ENOSPC (“OSD is full”) handling - now ENOSPC doesn’t turn into cascades of crashes
Add migration support to Proxmox VE Vitastor driver
Track last_clean_pgs on a per-pool basis thus reducing data movement in a cluster with pools remaining unclean/degraded for a long time

Implement a new “vitastor-disk purge” command to remove OSDs with safety checks
Implement a new “vitastor-cli rm-osd” command to only remove OSD metadata from etcd
Fix a bug where the monitor could ignore OSD removal and other /osd/stats key changes
Fix a bug where garbage could be returned when reading objects being written at the same time
Fix a rare write stall where journal space could be not reclaimed where there were no new operations in the flush queue
Fix a rare peering stall caused by a previous long listing operations queues limiting attempt
Fix total object count statistic in OSD on object creation
Add missing offset&len into vitastor-disk dump-journal for big_writes, fix JSON format
Make vitastor-cli print help on missing command
Make vitastor-cli translate all ‘-’ to ‘_’ in CLI options

Fix QEMU driver compatibility with QEMU 7.0 and < 2.9
Add patches for pve-qemu-kvm 7.1 (PVE 7.3) and pve-qemu-kvm 6.2 (PVE 7.2)
Fix Proxmox driver location in the pve-storage-vitastor package
Disable HDD autodetection in non-hybrid mode
Explicitly warn about a buggy kernels on -EAGAIN in io_uring
Final fix for the lack of zeroing out of old metadata entries (do not crash with “big_write journal_entry was allocated over another object” in some cases after an unclean OSD shutdown)
Wait for data writes before fsyncing data if data fsync is enabled
Never try to wait for free space inside blockstore thus stalling OSDs
Fix a rare crash in osd_peering due to callback ordering
Fix a rare duplication of ping & op message IDs
Fix a rare use-after-free during pings
Add --force to vitastor-disk read-sb
Make vitastor-disk dump metadata object IDs in hex, add forgotten commas
Fix vitastor-disk SCSI disk cache check

Remove an additional data copy operation when flushing journal (should slightly increase write performance)
Fix a bug where new writes in the inmemory_journal=false mode could overwrite the data currently read by a parallel read operation
Fix degraded parity writes for EC N+K when K>1 where the bug could also lead to an “assertion failed” error
Fix missing journal space check for “big” writes which could lead to “prefill_single_journal_entry(): assertion failed…” error in OSD
Fix possible “assertion failed: next->prev_wait >= 0” in client in rare cases
Fix missing “len” field in vitastor-disk write-journal big_writes
Fix possible crash of a full OSD (ENOSPC)
Fix CSI build scripts to include newest packages every time
Fix CSI endpoint in the liveness probe manifest

Implement automatic OSD activation via udev and simple on-disk superblock storage
Add a new vitastor-disk tool and merge all disk-related functionality there. Now it can prepare new OSD disks, upgrade plain old systemd units to the new scheme, resize OSD data area, manage OSD services by disk paths, manage superblocks, automatically check and disable disk cache, dump and write back journal and metadata.
Add a documentation section about vitastor-disk (read it if you want details!)
Install systemd services during package installation instead of the older method of manually creating them via separate shell scripts
Add a new make-etcd script that reuses /etc/vitastor/vitastor.conf to configure etcd
Allow to configure block_size, bitmap_granularity and immediate_commit per-pool
Fix “fatal error: tried to overwrite non-zero metadata entry” which was possible in some cases after unclean OSD shutdown (caused by old metadata entries not being zeroed)

Add ISA-L erasure code implementation, now used automatically instead of jerasure when available
Fix listings sending too many parallel requests to OSDs
Fix rm-data crashing with --wait-list
Remove empty inodes from statistics and ls output, after inode_vanish_time seconds after deletion
Make monitor delete pool statistics when the pool is deleted and thus remove them from df output
Log multiple etcd addresses in OSD logs correctly
Fix true/false parsing in json configs like no_recovery/no_rebalance
Show no_recovery, no_rebalance, readonly flags in status

Add documentation! :-) in Russian and English
Implement a NFS proxy for file-based access emulation to Vitastor images for non-QEMU based hypervisors like VMWare, as a better way than iSCSI
Implement primary_affinity_tags pool option
Add a patch for libvirt 6.0
Fix free_down_raw in cli status
Fix a rare bug where OSDs could drop unrelated connections on errors

Fix incorrect reading of extra metadata block leading to extra unknown objects in stats
Fix CSI driver volumeMode: Block support
Add block PVC and pod examples
Fix build under 32 bit architectures
Fix slow connection ramp-up caused by up_wait_retry_interval

Implement vitastor-cli status (print cluster status) command
Add a new make-osd-hybrid.js script to quickly prepare a lot of hybrid (HDD+SSD) OSDs
Implement snapshot deletion for Cinder driver (only works in a healthy cluster)
Fix a huge :) bug causing reads to return all zeroes during rebalance. Add a test to prevent it in the future
Disconnect NBD proxy correctly without leaving a zombie [vitastor-nbd] process in D state
Fix a rare write hang appearing with small write throttling enabled