v1.3.2

Release notes

Performance was the main focus of this immudb release. The way durability guarantees were implemented until now i.e. fsync was called synchronously as the final commit step, resulted in a lot of overhead for each transaction.

A significant improvement in write performance (up to x10 faster compared to previous immudb release v1.3.1) was achieved by grouping transactions in the last step of the commit process. The gains are greatest when there are fewer key-value entries per transaction and multiple writers are simultaneously working. The cost of the sync operation pays for itself as the number of key-value pairs in each transaction increases, although some noticeable speed can still be achieved (up to x2 compared to previous immudb release v1.3.1).

Read performance was also significantly improved due to the introduction of reusable pools of objects, which reduced allocations needed during query resolution. The gains are quite significant in both single and multi-get operations.

Benchmarking

Stay tuned, benchmarks done using stresser2 tool (https://github.com/codenotary/immudb-tools) will be available soon, we're still collecting some results.

JSON Logging

Logs can also be generated in json format for easier ingestion by logging platforms.
It's quite simple to enable this useful feature, just run immudb with the logformat flag set to json e.g. ./immudb --logformat=json

Go 1.18

immudb binaries and Docker images are now built with Go 1.18.

Changelog

[v1.3.2] - 2022-08-25

Bug Fixes

• access tls value in global scope within ingress annotations
• company name in webconsole and other files
• build: Fix go-acc and goveralls invocations
• build: update go version to 1.18 in Dockerfiles
• build/RELEASING.md: Add note about updating playground
• embedded: use tmp folder for unit test cases
• embedded/sql: Support single BEGIN statement.
• embedded/store: Check precommitted state when replicating
• embedded/store: Optionally preallocate Tx pools
• embedded/store: Ensure ordering of transaction timestamps
• embedded/store: Assign blTxID within locked tx state
• embedded/store: ensure tx is released upon error
• embedded/store: Improved check for replicated transaction
• embedded/store: Return correct error on key length exceeded
• embedded/store: Protect against simultaneous replicators
• embedded/store: Reduce the amount of allocations for tx object
• embedded/tools/stress_tool: Fix compilation after recent update to tx holder pool
• getRandomTable: increase RNG range for table generation
• github: Remove unnecessary /test/ path when uploading perf results to s3
• github: Do not use yaml anchors in github workflows
• pkg/client: Invalid client state after connection refused
• pkg/client/clienttest: enforce mock client to interface
• pkg/database: Fix calculation of proof for VerifiableTxByID
• pkg/database: Correct revision for Scan requirests
• server: Show info text with a logger
• servertest: Allow accessing Server object before starting the server
• stdlib/rows: add colums to row response
• test/performance: Cleanup test directory

Changes

• pin google.golang.org/protobuf to v1.27.1 (currently used version for generated code).
• Introduce separate TxHolder pools
• update github.com/spf13/viper to v1.12.0.
• makefile formatting.
• update build constraint to new & future-proof syntax.
• format tools.go.
• deprecate ImmuClient.HealthCheck in favour of ServerInfo.
• reimplement ImmuClient.HealthCheck using rpc ServerInfo instead of (deprecated) Health.
• refactor TestServerInfo.
• Update main go versin to 1.18
• generate gRPC stubs.
• ignore schema_grpc.pb.go in coveralls.
• use go.mod version of github.com/grpc-ecosystem/grpc-gateway when building codegen.
• regenerate with correct version of protoc-gen-go.
• ignore schema_grpc.pb.go in code coverage.
• pin github.com/pseudomuto/protoc-gen-doc to 1.4.1 (currently used version for generated code).
• Makefile: Update webconsole to 1.0.16
• build: Update RELEASING.md doc
• build: Improve generation of build checksums
• cmd/immuadmin: Add support for max-commit-concurrency option
• cmd/immuadmin: Add support for read-tx-pool-size option
• cmd/immudb: Add support for max-sessions command line option
• database/sql: Delay txholder allocation on VerifiableSQLGet
• embedded/ahtree: threshold-based sync
• embedded/ahtree: use bigger default write buffer size
• embedded/ahtree: improve error handling
• embedded/ahtree: flushless append
• embedded/ahtree: improve validations and error handling
• embedded/ahtree: support newst appendable implementation
• embedded/ahtree: improve error message consistency
• embedded/ahtree: minor error message change
• embedded/appendable: autosync when write buffer is full
• embedded/appendable: autosync support in multi-appendable
• embedded/appendable: flush when no more writes are done in appendable
• embedded/appendable: improve explanatory comment inside sync method
• embedded/appendable: improve singleapp validation and error handling
• embedded/appendable: error tolerant seek
• embedded/appendable: improve validations and error handling
• embedded/appendable: inmem buffer offset
• embedded/appendable: wip remoteapp validation
• embedded/appendable: return io.EOF when offset is out of range
• embedded/appendable: upgrade mocked and remote appendable based on new flushing assumptions
• embedded/appendable: auto-sync options
• embedded/appendable: multi-appendable shared write buffer
• embedded/htree: improve error handling
• embedded/sql: Remove unnecessary tx holder buffer from SQLTx
• embedded/store: Better errors returned during replication error
• embedded/store: aht options
• embedded/store: Use dedicated error for replication conflicts
• embedded/store: sync AHT before tx commit log
• embedded/store: in-mem clog buffer written when synced
• embedded/store: wrap internal already closed errors
• embedded/store: handle appendable already close error
• embedded/store: Optimize ReadTxHeader method
• embedded/store: Do not write values if concurrency limit is reached
• embedded/store: Add dedicated error for tx pool exhaustion
• embedded/store: add TODO comment
• embedded/store: parametrize write buffer size
• embedded/store: wip retryable sync
• embedded/store: flush-less precommit
• embedded/store: multi-tx syncs
• embedded/store: use smaller default buffer size
• embedded/store: Add txDataReader to process transaction data
• embedded/store: avoid sync waiting if there are no new transactions
• embedded/store: improve error comparison with errors.Is(...)
• embedded/store: wip error declaration
• embedded/store: Add explicit ReadTxEntry method
• embedded/store: Add explicit ReadTxHeader
• embedded/store: Optimize ReadTxEntry method
• embedded/store: Add txPoolOptions to setup pool parameters upon creation
• embedded/store: set new default write buffer values
• embedded/store/txpool: Allocate pool entries separately
• embedded/store/txpool: Make txPoolOptions members private
• embedded/tbtree: improve error handling
• embedded/tbtree: use non-retryable sync
• embedded/tbtree: define using generic errors towards errors.Is(...) usage
• embedded/watchers: improve error handling
• github: Update ACTIONS_SECRETS.md file
• github: Allow selection of runner to run perf test
• github: Run perf test suite on pull requests
• github: Run performance test suite on push to master
• github: Add simple documentation of PERF_TEST_xxx secrets
• github: Install qemu using docker/setup-qemu-action
• github: Allow using multiple runners for perf test suite
• github: Upload perf results to AWS s3
• pkg/api: Add tx pool size to GRPC and stored db options
• pkg/api: milliseconds message type
• pkg/api: expose aht settings
• pkg/api: export syncFrequency database parameter
• pkg/api: deprecate rpc Health in favour of ServerInfo.
• pkg/database: Add tx pool size to db options
• pkg/database: Remove txHolder from get operation
• pkg/database: Do not allocate txholder for history scans
• pkg/database: allocate tx buffer before doing verified writes
• pkg/logger: Add memory logger
• pkg/logger: add json logger
• pkg/server: simplify ImmuServer.Health.
• pkg/server: Add pprof option
• test/performance: Add basic flags to the benchmark process
• test/performance: Add separate Write KV/s test.
• test/performance: Split benchmark list and run code
• test/performance: Move test seed out of configuration
• test/performance: Move random generator and key tracker to common coode
• test/performance: Add CPU time / memory stats gathering
• test/performance: Allow customized name for the benchmark
• test/performance: Add basic IO stats
• test/performance: Improve live IO display
• test/performance: Better logging and output
• test/performance: Correctly close random data generator

Features

• revert usages of ServerInfo that would break backwards compatibility.
• add test for HealthCheck.
• cmd/immuadmin: expose syncFrequency and WriteBufferSize db parameters
• cmd/immuclient: add info command to immuclient.
• pkg/api: expose write buffer parameter
• pkg/api: improve documentation of ServerInfo.
• pkg/api: remove ServerInfoResponse.status field.
• pkg/api: add ServerInfo rpc to deprecate Health.
• pkg/client: revert WaitForHealthCheck change to maintain backwards-compatibility.
• pkg/client: implement ImmuClient.ServerInfo.
• pkg/server: implement ImmuServer.ServerInfo.

Downloads

Docker image
https://hub.docker.com/r/codenotary/immudb

immudb Binaries

File	SHA256
immudb-v1.3.2-darwin-amd64	b4efc5fb979db7dd32ad244c9fd27b1b5c4443d243cde13360f47c3d731d74fa
immudb-v1.3.2-darwin-arm64	1d7655ccf78edea69620c09afcadeb22ab8cf666f56f833fd62ce44d0357bac6
immudb-v1.3.2-freebsd-amd64	4306df4189a55aba4c37913dfc78223e791a9acdf59b3a75cac490683499e2e7
immudb-v1.3.2-linux-amd64	6833dc4cdba2da6d438c45ecaa244ea10e2ce0175af61428a43e4117645cdf52
immudb-v1.3.2-linux-amd64-static	4093bd435c1006c16b62f6dadc0d1178a111c267c9432e709503d8e8cca061ad
immudb-v1.3.2-linux-arm64	e9badad821af93ccacaff68d4c949c26f76a9ba60616b45ffba6a1fa04dbb1fe
immudb-v1.3.2-linux-s390x	58798b1bbec98c81a84b328a8bfec62ce7b65ff6dd0a148a94a15acac32c0c33
immudb-v1.3.2-windows-amd64.exe	20a61136c566ba712927cf9779058b2ecf9845dc5901f16471d2017188396000

immuclient Binaries

File	SHA256
immuclient-v1.3.2-darwin-amd64	537fe5282b95f7a48f1030a2020203b6c0376bee0a56e0e22d67ed409d77520d
immuclient-v1.3.2-darwin-arm64	c84efdf222f09e31b1c839696fc12c9c0c04a772fd9bc9f3383747ee56ec55df
immuclient-v1.3.2-freebsd-amd64	a73130dacff5897abc0e3f08c27897d373f7b8e03cdd93ca73072e3f558d1978
immuclient-v1.3.2-linux-amd64	081d52d7105a58cd37302a16d90132f96c025f0e374ec18573d5ea2f8fe50cf9
immuclient-v1.3.2-linux-amd64-static	c1b68ba71e363fdb33f8eb15288cc910a520c0ddc54af711bee9a35dfb54ca14
immuclient-v1.3.2-linux-arm64	20912c92a6268829c19077c2c750a7d1d7a247e4529ce979a515f2d1c924cdb0
immuclient-v1.3.2-linux-s390x	f9a480bd242143f8bae1ea9ea565dee0f51e09768ca539d6c950323031702c34
immuclient-v1.3.2-windows-amd64.exe	10e4c27efee6b558332dba1768fba3cb053215e4e9ae1a463571ec32a9fc9636

immuadmin Binaries

File	SHA256
immuadmin-v1.3.2-darwin-amd64	a801d2ce7bdc1565698c9596dc4e75682dc672568669c77958ace47586030761
immuadmin-v1.3.2-darwin-arm64	2178cef636784914ba5b182cbab75d24e2472ab232a301bf438549f3ad639e21
immuadmin-v1.3.2-freebsd-amd64	d5de0d391537c4ca555838306d8c3cf8b112f1b69712e643999f91e23b552685
immuadmin-v1.3.2-linux-amd64	ed82417aa2a96ff6da76a2c262bfd46f9cc356acbfebce437591e5c9acd83d95
immuadmin-v1.3.2-linux-amd64-static	50bdc5d1b58d2c4114ef2e5e538db32183a85a7c9e88ee33bf8acede5026c4e7
immuadmin-v1.3.2-linux-arm64	eac45e448ee6661eb262d365297ba94ec6e1a94d91a5f986536504f5cc132b38
immuadmin-v1.3.2-linux-s390x	d37ec84157ec3130292e9d84bdb8080edaf74a13c44d585ddc0e2fd4f50a83f0
immuadmin-v1.3.2-windows-amd64.exe	8678122e3708691937ee3b28e1edda251ee6a3eb10f79e8dccedc5868e1485ff