# AntiEtcd

Simplistic miniature etcd replacement based on [TinyRaft](https://git.yourcmc.ru/vitalif/tinyraft/)

- Embeddable
- REST API only, gRPC is shit and will never be supported
- [TinyRaft](https://git.yourcmc.ru/vitalif/tinyraft/)-based leader election
- [Strict serializable](https://jepsen.io/consistency/models/strict-serializable) transaction isolation,
  confirmed by [Jepsen tests](https://git.yourcmc.ru/vitalif/antietcd/src/branch/master/jepsen)
- Websocket-based cluster communication
- Supports a limited subset of etcd REST APIs
- With optional persistence

# Contents

- [CLI Usage](#cli-usage)
  - [CLI Client](#cli-client)
- [Options](#options)
  - [HTTP](#http)
  - [Persistence](#persistence)
  - [Clustering](#clustering)
- [Embedded Usage](#embedded-usage)
- [About Persistence](#about-persistence)
- [Supported etcd APIs](#supported-etcd-apis)
  - [/v3/kv/txn](#v3-kv-txn)
  - [/v3/kv/put](#v3-kv-put)
  - [/v3/kv/range](#v3-kv-range)
  - [/v3/kv/deleterange](#v3-kv-deleterange)
  - [/v3/lease/grant](#v3-lease-grant)
  - [/v3/lease/keepalive](#v3-lease-keepalive)
  - [/v3/lease/revoke or /v3/kv/lease/revoke](#v3-lease-revoke-or-v3-kv-lease-revoke)
  - [/v3/maintenance/status](#v3-maintenance-status)
  - [Websocket-based watch APIs](#websocket-based-watch-apis)
- [HTTP Error Codes](#http-error-codes)

## CLI Usage

```
npm install antietcd

node_modules/.bin/antietcd \
    [--cert ssl.crt] [--key ssl.key] [--port 12379] \
    [--data data.gz] [--persist_interval 500] \
    [--node_id node1 --cluster_key abcdef --cluster node1=http://localhost:12379,node2=http://localhost:12380,node3=http://localhost:12381]
    [other options]
```

Antietcd doesn't background itself, so use systemd or start-stop-daemon to run it as a background service.

### CLI Client

```
node_modules/.bin/anticli [OPTIONS] put <key> [<value>]
node_modules/.bin/anticli [OPTIONS] get <key> [-p|--prefix] [-v|--print-value-only] [-k|--keys-only] [--no-temp]
node_modules/.bin/anticli [OPTIONS] del <key> [-p|--prefix]
node_modules/.bin/anticli [OPTIONS] load [--with-lease] < dump.json
node_modules/.bin/anticli [OPTIONS] watch <key> [key...] [-p|--prefix]
```

For `put`, if `<value>` is not specified, it will be read from STDIN.

Options:

<dl>

<dt>--endpoints|-e http://node1:2379,http://node2:2379,http://node3:2379</dt>
<dd>Specify HTTP endpoints to connect to</dd>

<dt>--cert &lt;cert&gt;</dt>
<dd>Use TLS with this certificate file (PEM format)</dd>

<dt>--key &lt;key&gt;</dt>
<dd>Use TLS with this key file (PEM format)</dd>

<dt>--ca &lt;cert&gt;</dt>
<dd>Use this TLS CA certificate to verify server certificate</dd>

<dt>--timeout 1000</dt>
<dd>Specify request timeout in milliseconds</dd>

<dt>--base64 1</dt>
<dd>Use base64 encoding of keys and values, like in etcd (enabled by default)</dd>

<dt>--json or --write-out=json</dt>
<dd>Print raw response in JSON</dd>

</dl>

## Options

### HTTP

<dl>

<dt>--port 2379</dt>
<dd>Listen port</dd>

<dt>--ip 192.168.0.10</dt>
<dd>Listen address (by default listen on all interfaces)</dd>

<dt>--cert &lt;cert&gt;</dt>
<dd>Use TLS with this certificate file (PEM format)</dd>

<dt>--key &lt;key&gt;</dt>
<dd>Use TLS with this key file (PEM format)</dd>

<dt>--ca &lt;ca&gt;</dt>
<dd>Use trusted root certificates from this file.
Specify &lt;ca&gt; = &lt;cert&gt; if your certificate is self-signed.</dd>

<dt>--client_cert_auth 1</dt>
<dd>Require TLS client certificates signed by <ca> or by default CA to connect.</dd>

<dt>--ws_keepalive_interval 30000</dt>
<dd>Client websocket ping (keepalive) interval in milliseconds</dd>

<dt>--use_base64 1</dt>
<dd>Use base64 encoding of keys and values, like in etcd (enabled by default).</dd>

</dl>

### Consistency

<dl>

<dt>--use_locks 1</dt>
<dd>Enable lock-based transaction isolation to prevent <a href="https://jepsen.io/consistency/phenomena/g1a">G1a</a>.</dd>

<dt>--merge_watches 1</dt>
<dd>Antietcd merges all watcher events for a single transaction into a single websocket message to provide
more ordering/transaction guarantees. Set to 0 to disable this behaviour.</dd>

<dt>--stale_read 1</dt>
<dd>Allow to serve reads from replicas (potentially stale). Specify 0 to always contact leader on every read.</dd>

</dl>

### Persistence

<dl>

<dt>--data &lt;filename&gt;</dt>
<dd>Store persistent data in &lt;filename&gt;</dd>

<dt>--persist_interval &lt;milliseconds&gt;</dt>
<dd>Persist data on disk after this interval, not immediately after change</dd>

<dt>--persist_filter ./filter.js</dt>
<dd>Use persistence filter from ./filter.js (or a module). <br />
Persistence filter is a function(cfg) returning function(key, value) ran
for every change and returning a new value or undefined to skip persistence.</dd>

<dt>--compact_revisions 1000</dt>
<dd>Number of previous revisions to keep deletion information in memory</dd>

</dl>

### Clustering

<dl>

<dt>--node_id &lt;id></dt>
<dd>ID of this cluster node</dd>

<dt>--cluster &lt;id1&gt;=&lt;url1&gt;,&lt;id2&gt;=&lt;url2&gt;,...</dt>
<dd>All other cluster nodes</dd>

<dt>--cluster_key &lt;key&gt;</dt>
<dd>Shared cluster key for identification</dd>

<dt>--election_timeout 5000</dt>
<dd>Raft election timeout</dd>

<dt>--heartbeat_timeout 1000</dt>
<dd>Raft leader heartbeat timeout</dd>

<dt>--wait_quorum_timeout 30000</dt>
<dd>Timeout for requests to wait for quorum to come up</dd>

<dt>--leadership_timeout 3000</dt>
<dd>TinyRaft leadership timeout</dd>

<dt>--leader_priority &lt;number&gt;</dt>
<dd>Raft leader priority for this node (optional)</dd>

<dt>--reconnect_interval 1000</dt>
<dd>Unavailable peer connection retry interval</dd>

<dt>--dump_timeout 5000</dt>
<dd>Timeout for dump command in milliseconds</dd>

<dt>--load_timeout 5000</dt>
<dd>Timeout for load command in milliseconds</dd>

<dt>--forward_timeout 1000</dt>
<dd>Timeout for forwarding requests from follower to leader in milliseconds</dd>

<dt>--replication_timeout 1000</dt>
<dd>Timeout for replicating requests from leader to follower in milliseconds</dd>

<dt>--compact_timeout 1000</dt>
<dd>Timeout for compaction requests from leader to follower in milliseconds</dd>

</dl>

### Logging

<dl>

<dt>--logs keyword1,keyword2,...</dt>
<dd>
Enable logs by keywords:

- access: log all requests and incoming websocket messages.
- watch: log outgoing websocket watch messages.
-cluster: log clustering (TinyRaft) events.
</dd>

</dl>

## Embedded Usage

```js
const AntiEtcd = require('antietcd');

// Configuration may contain all the same options like in CLI, without "--"
// Except that persist_filter should be a callback (key, value) => newValue
const srv = new AntiEtcd({ ...configuration });

// Start server
srv.start();

// Make a local API call in generic style:
let res = await srv.api('kv_txn'|'kv_range'|'kv_put'|'kv_deleterange'|'lease_grant'|'lease_revoke'|'lease_keepalive', { ...params });

// Or function-style:
res = await srv.txn(params);
res = await srv.range(params);
res = await srv.put(params);
res = await srv.deleterange(params);
res = await srv.lease_grant(params);
res = await srv.lease_revoke(params);
res = await srv.lease_keepalive(params);

// Error handling:
try
{
    res = await srv.txn(params);
}
catch (e)
{
    if (e instanceof AntiEtcd.RequestError)
    {
        // e.code is HTTP code
        // e.message is error message
    }
}

// Watch API:
const watch_id = await srv.create_watch(params, (message) => console.log(message));
await srv.cancel_watch(watch_id);

// Stop server
srv.stop();
```

## About Persistence

Persistence is very simple: full database is dumped into JSON, gzipped and saved as file.

By default, it is written and fsynced on disk on every change, but it can be configured
to dump DB on disk at fixed intervals, for example, at most every 500 ms - of course,
at expense of slightly reduced crash resiliency (example: `--persist_interval 500`).

You can also specify a filter to exclude some data from persistence by using the option
`--persist_filter ./filter.js`. Persistence filter code example:

```js
function example_filter(cfg)
{
    // <cfg> contains all command-line options
    const prefix = cfg.exclude_keys;
    if (!prefix)
    {
        return null;
    }
    return (key, value) =>
    {
        if (key.substr(0, prefix.length) == prefix)
        {
            // Skip all keys with prefix from persistence
            return undefined;
        }
        if (key === '/statistics')
        {
            // Return <unneeded_key> from inside value
            const decoded = JSON.parse(value);
            return JSON.stringify({ ...decoded, unneeded_key: undefined });
        }
        return value;
    };
}

module.exports = example_filter;
```

## Consistency Guarantees

When `--stale_read` is enabled, Antietcd provides [SERIALIZABLE](https://jepsen.io/consistency/models/serializable) transaction isolation level.

When it's disabled, the level is [STRICT SERIALIZABLE](https://jepsen.io/consistency/models/strict-serializable).

Reconnected watchers don't receive all historical events individually, but they are guaranteed to receive
a consistent snapshot of keys in interest.

## Supported etcd APIs

NOTE: `key`, `value` and `range_end` are always encoded in base64, like in original etcd.

Range requests are only supported across "directories" separated by `/`.

It means that in range requests `key` must always end with `/` and `range_end` must always
end with `0`, and that such request will return a whole subtree of keys.

### /v3/kv/txn

Request:

```ts
type TxnRequest = {
  compare?: (
    { key: string, target: "MOD", mod_revision: number, result?: "LESS" }
    | { key: string, target: "CREATE", create_revision: number, result?: "LESS" }
    | { key: string, target: "VERSION", version: number, result?: "LESS" }
    | { key: string, target: "LEASE", lease: string, result?: "LESS" }
    | { key: string, target: "VALUE", value: string }
  )[],
  success?: (
    { request_put: PutRequest }
    | { request_range: RangeRequest }
    | { request_delete_range: DeleteRangeRequest }
  )[],
  failure?: (
    { request_put: PutRequest }
    | { request_range: RangeRequest }
    | { request_delete_range: DeleteRangeRequest }
  )[],
  serializable?: boolean,
}
```

`serializable` allows to serve read-only requests from follower even if `stale_read` is not enabled.

Response:

```ts
type TxnResponse = {
  header: { revision: number },
  succeeded: boolean,
  responses: (
    { response_put: PutResponse }
    | { response_range: RangeResponse }
    | { response_delete_range: DeleteRangeResponse }
  )[],
}
```

### /v3/kv/put

Request:

```ts
type PutRequest = {
  key: string,
  value: string,
  lease?: string,
}
```

Other parameters are not supported: prev_kv, ignore_value, ignore_lease.

Response:

```ts
type PutResponse = {
  header: { revision: number },
}
```

### /v3/kv/range

Request:

```ts
type RangeRequest = {
  key: string,
  range_end?: string,
  keys_only?: boolean,
  serializable?: boolean,
}
```

`serializable` allows to serve read-only requests from follower even if `stale_read` is not enabled.

Other parameters are not supported: revision, limit, sort_order, sort_target,
count_only, min_mod_revision, max_mod_revision, min_create_revision, max_create_revision.

Response:

```ts
type RangeResponse = {
  header: { revision: number },
  kvs: { key: string }[] | {
    key: string,
    value: string,
    lease?: string,
    mod_revision: number,
  }[],
}
```

### /v3/kv/deleterange

Request:

```ts
type DeleteRangeRequest = {
  key: string,
  range_end?: string,
}
```

Other parameters are not supported: prev_kv.

Response:

```ts
type DeleteRangeResponse = {
  header: { revision: number },
  // number of deleted keys
  deleted: number,
}
```

### /v3/lease/grant

Request:

```ts
type LeaseGrantRequest = {
  ID?: string,
  TTL: number,
}
```

Response:

```ts
type LeaseGrantResponse = {
  header: { revision: number },
  ID: string,
  TTL: number,
}
```

### /v3/lease/keepalive

Request:

```ts
type LeaseKeepaliveRequest = {
  ID: string,
}
```

Response:

```ts
type LeaseKeepaliveResponse = {
  result: {
    header: { revision: number },
    ID: string,
    TTL: number,
  }
}
```

### /v3/lease/revoke or /v3/kv/lease/revoke

Request:

```ts
type LeaseRevokeRequest = {
  ID: string,
}
```

Response:

```ts
type LeaseRevokeResponse = {
  header: { revision: number },
}
```

### /v3/maintenance/status

Request:

```{}```

Response:

```ts
type MaintenanceStatusResponse = {
  header: {
    member_id?: string,
    revision: number,
    compact_revision: number,
    raft_term?: number,
  },
  version: string,
  cluster?: { [string]: string },
  leader?: string,
  followers?: string[],
  raftTerm?: string,
  raftState?: 'leader'|'follower'|'candidate',
  // dbSize actually reports process memory usage
  dbSize: number,
}
```

### Websocket-based watch APIs

Client-to-server message format:

```ts
type ClientMessage =
  { create_request: {
    key: string,
    range_end?: string,
    start_revision?: number,
    watch_id?: string,
  } }
  | { cancel_request: {
    watch_id: string,
  } }
  | { progress_request: {} }
```

Server-to-client message format:

```ts
type ServerMessage = {
  result: {
    header: { revision: number },
    watch_id: string,
    created?: boolean,
    canceled?: boolean,
    compact_revision?: number,
    events?: {
      type: 'PUT'|'DELETE',
      kv: {
        key: string,
        value: string,
        lease?: string,
        mod_revision: number,
      },
    }[],
  }
} | { error: 'bad-json' } | { error: 'empty-message' }
```

### HTTP Error Codes

- 400 for invalid requests
- 404 for unsupported API / URL not found
- 405 for non-POST request method
- 408 for lock wait timeouts
- 501 for unsupported API feature - non-directory range queries and so on
- 502 for server is stopping
- 503 for quorum-related errors - quorum not available and so on

# Author and License

Author: Vitaliy Filippov, 2024

License: [Mozilla Public License 2.0](https://www.mozilla.org/media/MPL/2.0/index.f75d2927d3c1.txt)
or [Vitastor Network Public License 1.1](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/VNPL-1.1.txt)
