I'm super excited to announce that we're making the source code for our microvm launcher available to the public.
It's all out there in the GitHub repo
Cradle is how we isolate pods from each other on the kraud platform using microvms in KVM.
The repo should contain enough of the host implementation to enable anyone to launch docker containers in isolation.
In contrast to other solutions such as firecracker, cradle is intended to run existing containers without changes.
Kraud is primarily focused on creating a docker like experience in the cloud, for maximum developer comfort.
It is not "cloud native" compliant and does not fit into the k8s ecosystem.
Instead cradles mission is to start a docker container within 100ms on an arbitrary host machine,
so we can schedule workloads dynamically without the massive economic and ecological cost of operating the
infrastructure usually required to do that.
While this is cool and interesting tech, the primary purpose of publishing cradle as open source is to enable our customers
to verify the complete boot stack using AMD SNP. Everything that runs inside the VM is open and can be attested to be exactly what we claim it to be.
This in addition to purchasing on premise cloud machines enables our customers to enjoy the flexibility and comfort of a cloud
service while also staying in full control of their data.
Running an attested confidential docker container locally is a bit involved, since this is not intended as a finished product.
Internally at kraud we use lots of things to deploy cloud and on-prem machines.
Old school dpkg and ansible is a whole different world than docker, kubernetes, etc.
One of the things that’s been missing has been a thing in the middle, where we can configure and deploy services
in a consistent and repeatable way like docker, but without depending on high level services like docker being available yet.
There is an uncountable amount of innovative approaches to building consistent system services, nix being a popular one for example.
But docker build is really the standard. It's not elegant, but it's simple, which is very important in an operational environment
where you need to quickly understand a large amount of moving parts while all the red alarm lights are blinking.
docker usually builds images in layers, which is an unfortunate design choice. distributing those layers also requires docker registry.
Registry is a docker container, with persistent storage. The recommended k8s architecture
is boostrapping k8s from a registry that runs on k8s. But i'd rather sleep well at night so we're not doing that.
Instead bali builds docker images to a single tarball. They can be distributed using fairly standard tools, even replicated.
They're also ephemeral, meaning no cleanup is required after run.
security is critical in cloud operations. Users give up control over the low level infrastructure and in turn expect us to do it correctly of course.
docker content trust exists but it's ... weird and i'm not entirely sure if its even cryptographically correct.
Fortunately, single files are easy to sign, so bali signs any image it builds by default with a local ed25519 key.
The signature is embedded, so that the image still works as a normal tar.gz you can extract with any other tool.
bali run then can yolo-get images from any untrusted storage, but checks the signature on open.
This is convenient for us to make distribution of images as low-tech as possible, so we can rely on it to bootstrap higher level systems.
docker was designed back when containers where not a thing really. it can keep poorly behaving software in check fairly well.
That plus it makes it very convenient for the user by hiding alot of complexity required to make that work.
When bootstrapping infrastructure, both of those features are counterproductive. While you can run docker with --net=host for example,
it still creates a namespace anyway, interfering with our own namespacing for vpcs. The only namespace we really want is for filesystem,
because it makes it convenient to just apk add dependencies without thinking of new build systems like nix.
An even bigger issue for us with docker was the daemon. When you do docker run, the calling process is really just an rpc client to the docker daemon,
which then runs the container. This means for example SIGKILL to the docker run process will not kill the process. it will just kill the docker cli, leaving the
service dangling. Any cgroup limits etc also do not actually apply to the system service. docker has its own solutions for each managment task,
but they're often intended for developers rather than operators.
bali instead calls execve after its done with the setup. It becomes the process you originally wanted to run.
This means any context coming from the caller directly applies to the service, making bali work great inside systemd units, nomad, etc.
bali is trivial by design, only intended for our narrow use case. there are other systems for a wider use case.
For example consider systemd-nspawn for services that have a persistent root system on the same machine.
We're happy to share bali with the community, if its useful to anyone else.
GP: our general purpose ceph cluster with 100TB of consumer grade SSDs.
GFS: global shared filesystem, very useful for just having files available in multiple pods concurrently. This is backed by GP with cephfs MD on top.
RED: a data garbage bin using 200TB of spining consumer disks. You should not be using this unless you want really cheap large slow storage.
The new LV tier will aid users of legacy applications that are built for more traditional virtual server deployments.
It is backed by a raid 1 of enterprise NVME drives and peaks at 1 million IOPS.
A 4TB volume will have 3GB/s bandwidth dedicated and can request pcie passthrough for low latency, while smaller allocations share the bandwidth and IOPS.
Unlike GP, LV does not survive host failure, meaning a loss of a host will result in the volume becoming unavailable.
During the last (bad) month this would have resulting in a 99.1% uptime, unlike GP which had 99.9% uptime.
There's a residual risk of total loss, due to the nature of being electrically connected to the same chassis.
We advise users to build their own contingency plan, similar to what you should have done with competing virtual machine offers.
While LV has very little benefits in modern applications, it pairs well with traditional VMs and will become the default storage in the kraud marketplace for VMs.
Most kraud users would rather not bother with details of how storage works. This is after all, what we've built kraud for.
However, as you noticed, we're not doing great in terms of uptime recently (still better than Azure, lol) and this is due to storage.
To reach our goal of carbon negative computing while also taking in zero venture funding, we must navigate the difficult path of
serving a variety of incompatible workloads.
Kraud is all about energy saving so we use lower clock EPYC cpus with the highest possible compute per energy efficiency.
Ceph was built for high clock speed XEONs with very little respect for energy efficiency, so it does not perform great in this scenario.
Adding to that, we treat physical servers as expendable and built all of our software for graceful recovery from loosing a node.
That allows building machines for a third of a cost of traditional OEMs like HP, etc.
We use things like cockroachdb, which performs well under frequent failures.
While ceph does also recover from such an event, it does NOT do it as graceful as you'd hope for,
resulting in several minutes of downtime for the entire cluster every time a single node fails.
As i keep saying, high availability is the art of turning a single node incident into a multi node incident.
In summary ceph is not the correct solution for the customer group that is currently the most important to the companies survival (paying customers, yes)
This is why we're moving that customer group away from ceph, so GFS can come back to its previously slow-but-stable glory.
In the future, once we become big enough, we hope to deliver a custom built storage solution that can work well within the energy targets.
Thank you for your patience and for joining us on this critical mission towards carbon negative compute.
From the beginning of the project we always strived for compatbility with your existing tools, be it docker or kubectl.
Your feedback is always greatly apprechiated, as it helps us clarify what that exactly means in practice.
How much compat is good, and where do the existing tools not work?
We haven't reached a stage where this is entirely clear yet, but the trend is pointing towards
Fully supporting the docker cli
Building a custom cli to supplement docker
Freezing kubectl at 1.24
Partially supporting the most popular of the many incompatible docker compose variants
Particularly kubectl is a difficult choice. Kubernetes is a standard. But unfortunately, it's not actually a standard,
and keeping up with upstream does not seem feasible at the moment.
Instead we will shift focus entirely on supporting docker and docker compose.
The compose spec is weird, and inconsistent, but it is simple and hence very popular.
Most of the confusion we've seen in practice is easily addressable with better tooling.
The kra commandline program works on docker-compose files and will implement some of the processes that docker does
not do at all (ingress config currently requires kubectl) or does incorrectly (local file mounts).
Specifically a pain point in some user setups has been CI. Since we don't support docker build yet,
users build on the ci machine and then use docker load. This is slow, because the docker api was never intended to be used remotely.
Instead kra push is very fast and should be used in CI instead.
name:Deployon:push:branches:[main,'staging']jobs:deploy:runs-on:ubuntu-lateststeps:-uses:actions/checkout@v3-name:'deploytokraud'env:KR_ACCESS_TOKEN:${{secrets.KR_ACCESS_TOKEN}}run:|curl -L https://github.com/kraudcloud/cli/releases/latest/download/kra-linux-amd64.tar.gz | tar xvzsudo mv kra /usr/local/bin/# get credentials for docker clikra setup docker# build the images localydocker compose build# push images to kraudkra push# destroy running pod# this is currently needed until we fix service upgradesdocker --context kraud.aep rm -f myapp# start new pods with new imagesdocker compose up -d
kra is open source and available here: https://github.com/kraudcloud/cli.
We're aways happy for feeedback. Feel free to open github issues or chat with a kraud engineer on discord.
Vdocker is how we call the thing that responds to docker api calls like log, attach, exec and cp.
Vdocker used to run on the host, and all the commands where carefully funneled through a virtio-serial.
The advantage is that cradle is small, and starts faster. However, we realized most people do start fairly large containers that
take a few seconds to start anyway. Hence sub-100ms startup time for cradle is no longer a priority
Instead we traded a few milliseconds of start time for much higher bandwidth by moving vdocker directly into cradle.
It listens inside of your pod on port 1 and accepts the nessesary docker commands from the api frontend.
docker cp now works properly and is much faster. Also docker attach no longer glitches.
Unfortunately this means docker run feels slower, although it really hasnt changed much.
Log output starts appearing roughly 80ms after download completed, but for larger container it may take several seconds to download layers,
which you can currently not see.
On the upside, all other commands now feel alot faster, because we skip vmm and just proxy the http call directly to vdocker.
Global file system can be mounted simultaneously on multiple containers,
enabling easy out of the box shared directories between services.
Similar to NFS, it does make files available over the network,
but GFS is fully managed and does not have a single point of failure.
It is also significantly faster than NFS due to direct memory access in virtiofs.
GFS enables an application developer to start multiple instances of the same application,
without implementing synchronization. This is specifically useful for traditional stacks like PHP where
horizontal scaling requires a separate network storage.
Any docker container works with GFS without changes. The same standard syntax used to mount volumes on your local computers dockers will simply work in the cloud.
Modern applications often choose to store shared files in object storages, specifically s3.
With GFS, you can simply store a file using unix file semantics without the need for a separate layer.
File i/o behaves identical using a docker volume on your local machine, and with kraud. This makes developing apps locally and deploying into the kraud seamless.
GFS is backed by cephfs on a 3 times redundant SSD cluster. Ceph is an open source object storage cluster backed by redhat, CERN and others. All pods/containers launched in the Falkenstein DC enjoy a 20MB/s transfer rate.
Users with hybrid regions should note that GFS data transfer counts towards external traffic.
Additionally, customers may choose a separate cluster for large intermediate data on magnetic disks.
This is intended for science applications working with large data sets and can easily scale to multiple petabytes.
