Vitastor 3.0.5 released

A single urgent fix for the new store: metadata was written incorrectly when filled metadata area exceeded 4 GB due to a simple & stupid 32-bit integer overflow. 🤦 This was leading to OSDs corrupting metadata and being unable to start on the next run.

The bug affected only the new store (meta_format=3) and OSDs with metadata area size larger than 4 GB. You can check your OSD metadata area size by checking data_offset in vitastor-disk read-sb /dev/vitastor/osdXX-data for single-disk OSDs or blockdev --getsize64 /dev/vitastor/osdXX-meta for hybrid OSDs.

If you’re affected, it’s highly recommended to:

• Install updated packages without restarting OSDs
• Restart OSDs one by one
• If a restart of an OSD triggers “has_degraded” rebalance - wait for it to finish before restarting another OSD
• If an OSD fails to start with “double-claimed block” or “metadata is corrupt or too old” error in the log - purge and recreate it, and again wait for “has_degraded” rebalance to finish before proceeding to the next OSD

2026-03-07 Continue reading →

Vitastor 3.0.4 released

New store bug fixes

• Fix very slow OSD startup (hours) possible in unlucky cases
• Fix incorrect garbage entry validation on start in some cases, preventing OSD startup
• Fix small_write_throttling not functioning and either hanging or crashing OSDs
• Fix “different fact_versions returned from subops” error possible with EC in some cases
• Add a recovery option for the new store: skip_corrupted_meta_entries
• Disable intent writes (even 4k) for hybrid OSDs by default for better write buffering

2026-03-03 Continue reading →

Vitastor 3.0.3 released

• Fix csum_block_size > 0 unusable with atomic writes in the new store (almost all atomic write requests generated invalid checksums with csum_block_size > 0)
• Fix monitor sometimes randomly failing to optimize PGs with the “problem is infeasible or unbounded” message due to not waiting to read full stdout of lp_solve
• Remove reshard_abort optimisation to fix chunked resharding introduced in 3.0.2 possibly corrupting in-memory OSD state when handling multiple rapid PG count change requests
• Fix removed inodes not disappearing from OSD statistics in the new store, leading to bloating of the statistics with old inodes
• Increase test coverage for the new store and fix several minor bugs:
  • Enabling/disabling of used_for_app was recalculating inode space statistics incorrectly
  • Fix object metadata validation on OSD startup rejecting some correct sequences of entries, possibly leading to OSD being unable to start
  • Fix PG activation with EC failing in rare cases with EBUSY when requesting to commit an already committed write
  • Fix a theoretically possible metadata writeback issue on ENOSPC during commit
• Fix monitor failing to optimize PGs in presence of a host with name convertible to js Number (like 04e278988710) :D
• Fix vitastor-cli dd sometimes (rarely) truncating the image when writing to stdout
• Fix a theoretically possible client connection object leak when io_uring is full
• Fix a leak of RDMA-CM connection objects
• Fix crashes with data_block_size < 32KB (useless setup, but anyway) (#113)

2026-02-08 Continue reading →

Vitastor 3.0.2 released

• Antietcd is now officially safe to use: the release includes a fixed version of antietcd which passes Jepsen transaction serializability tests.
• Fix a huge checksum bug in the old store: incorrect checksums for small initial writes. The bug affected writes of exactly csum_block_size (4k by default) into new (unallocated) objects and generated invalid checksums in the store for the written 4k block. Moreover, generation of these invalid checksums was very slow because it was calculating CRC32 for 4 GB of zeroes. If the block wasn’t then overwritten as a part of a larger write request it became unreadable even though the stored data was correct. The bug affected all versions since 1.0.0, or since 2.3.0 because vitastor-disk didn’t allow to enable checksums prior to 2.3.0 because of another bug. O:-)
• Prevent OSD disconnections due to long blocking of event-loop caused by PG resharding (moving objects between old and new PGs in memory) when changing pool PG count or just on restart of an OSD with a large database (for example, with a filled 8 TB SSD). The issue should now be fixed because OSD now performs resharding in chunks with pauses between chunks.
• Prevent pools stuck in paused state on an aborted PG count change.
• Fix a possible OSD crash with “division by zero” when trying to handle an operation before pool PG count is applied to the in-memory store.

2026-01-25 Continue reading →

Small, but with transaction isolation: writing Jepsen tests for Antietcd

Since version 1.7.0, Vitastor has a built-in etcd replacement — Antietcd.

It’s implemented in Node.js and is very simple — it has just a couple thousand lines of code. It doesn’t implement all features of etcd, but it’s absolutely sufficient for a fully functional Vitastor cluster — all the essential features are present, and in some ways, it’s even better than etcd — for example, Antietcd allows to avoid storing “temporary” data on disk.

However, until recently, there was no answer to the question: can it really be used in production? Does it work correctly?

Below is the story of the search for an answer. A story with a happy ending :)

2026-01-22 Continue reading →