Small Kafka: Tansu and SQLite on a free t3.micro

Summary

AWS has a free tier that can be used with the t3micro instance, having 1GiB of memory and an EBS baseline throughput of ~10MB/s. Great for kick starting an early stage project. A maximum EBS throughput of ~260MB/s, creates some headroom to flex up. Keep a beady eye on those CPU credits though! I thought I'd try the Tansu broker on a t3micro using the embedded SQLite storage engine. All meta and message data is stored in this database. To backup or restore an environment, is as simple as copying tansu.db. In a zero downtime environment, the S3 storage engine could be used, allowing multiple stateless brokers to use the same bucket concurrently. Let's spin up a t3micro instance with Amazon Linux 2023, to setup the following: The Supplementary Packages for Amazon Linux (SPAL) package repository Docker Compose to run the Tansu broker docker image Enable the containerd (docker) service Add the docker group to the ec2-user so that we can run docker commands Reboot after installing the services Using the following commands: sudo dnf install -y spal-release sudo dnf install -y docker-compose sudo systemctl enable containerd.service sudo usermod -a -G docker ec2-user sudo /sbin/shutdown -r now Log back in and create a compose.yaml, that will do the following: We are using the latest Tansu docker image from ghcr.io/tansu-io/tansu. The image is from scratch containing only the Tansu statically linked binary (well, maybe a LICENSE and some SSL certificates too!). Substitute the ADVERTISED_LISTENER_URL with the name of your instance Default logging to the warn level Use the SQLite STORAGE_ENGINE using the /data directory in the container Map the /data directory in the container to the current directory on the host Expose the Kafka API on port 9092 My compose.yaml, ensure that you change ADVERTISED_LISTENER_URL with the name of your instance: --- services: tansu: image: ghcr.io/tansu-io/tansu environment: ADVERTISED_LISTENER_URL: tcp://ec2-35-179-120-103.eu-west-2.compute.amazona...