Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,15 @@
# Changelog

## Unreleased

- **CDK, Pulumi and SDK examples, verified:** [`examples/cdk/basic`](examples/cdk/basic/README.md) (`cdk bootstrap` with the default template, `cdk deploy`, `cdk diff`, `cdk destroy`; HomeCloud must be served over HTTPS because the bootstrap bucket policy denies plain HTTP), [`examples/pulumi/basic`](examples/pulumi/basic/README.md) (`pulumi up`, `preview`, `refresh`, `destroy` with a local backend) and [`examples/sdk`](examples/sdk) (JavaScript v3, Go v2, Java v2: S3, SQS, DynamoDB, STS), with configuration snippets in [docs/aws-compat.md](docs/aws-compat.md#sdks) (#99, #100, #101).
- IAM: the AWS managed policy `AWSCloudFormationReadOnlyAccess`; policy, group and instance profile names may be up to 128 characters, as in AWS (users and roles stay at 64); the `aws:ResourceAccount` condition key is set (to the server's account) for users and role sessions.
- Lambda: `LoggingConfig` (log group, `Text`/`JSON` format, application and system log levels) over the AWS API and in CloudFormation (`AWS::Lambda::Function` `LoggingConfig`, CDK's `logGroup`); it was ignored and logs always went to `/aws/lambda/<name>`. New runtimes `nodejs24.x`, `python3.14` and `java25`.
- CloudFormation: `DeleteChangeSet` with the name of a change set that does not exist succeeds when the stack exists, as in AWS (`cdk diff` and `cdk deploy` rely on it).
- awsJson responses carry `X-Amz-Crc32`, as DynamoDB's do; the AWS SDK for Go v2 logged a checksum warning on every DynamoDB call without it.
- The `--tls-self-signed` certificate also covers `*.localhost` and `*.<public-host>`, so S3 Control calls (`<account>.localhost`, made by the Terraform and Pulumi AWS providers) verify over HTTPS. Delete `<data-dir>/tls/` to replace a certificate made by an earlier version.
- CloudWatch Logs, CloudWatch metrics and the Lambda environment reaper only see this installation's containers; on a Docker host shared with test runs they listed (and the reaper removed) containers of other accounts.

## 0.5.0 (2026-10-14)

Trust and operations: signed releases, safe upgrades with rollback, backups that restore everything, Prometheus metrics and a status page, and Testcontainers for Java and Node.
Expand Down
8 changes: 5 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,9 +119,11 @@ Or use the ready-made pieces in **[docs/integrations.md](docs/integrations.md)**
| **boto3** (Python SDK) | Tested: the compatibility test suite runs boto3 in CI, including Cognito SRP sign-in through pycognito |
| **Terraform / OpenTofu** (`hashicorp/aws` provider) | Tested: opt-in Terraform tests in the suite (IAM, S3, Route 53, CloudFormation and more; `HC_TEST_TERRAFORM=1`), and [`examples/terraform/shop`](examples/terraform/shop/README.md) applies, re-plans clean and destroys on a fresh install. Nightly: 17 of 17 popular `terraform-aws-modules` scenarios apply, re-plan clean and destroy ([results](docs/compatibility.md)). |
| **CloudFormation** (`aws cloudformation deploy`) | Tested: stacks, change sets and the boto3 waiters ([details](docs/aws-compat.md#cloudformation)) |
| **AWS CDK** | Partly: CDK-synthesized templates deploy through CloudFormation change sets; `cdk bootstrap` / `cdk deploy` end to end is not yet verified |
| **AWS CDK** | Verified by hand: `cdk bootstrap` (default template), `cdk deploy`, `cdk diff` and `cdk destroy` with HomeCloud served over HTTPS ([`examples/cdk/basic`](examples/cdk/basic/README.md)); constructs that need CloudFormation custom resources do not deploy yet |
| **Testcontainers** (Go, Python, Java, Node.js) | Modules in [`integrations/`](docs/integrations.md) start HomeCloud from a test and clean up after it; the Java and Node.js modules are tested in CI with S3, SQS, DynamoDB, Lambda and STS through the AWS SDK for Java v2 and the JavaScript SDK v3 |
| **Other AWS SDKs** (Go, .NET, …) and **Pulumi** | Expected to work (same protocols and SigV4), not yet covered by tests. Reports welcome |
| **Pulumi** (`@pulumi/aws`) | Verified by hand: `pulumi up`, `preview` (no changes), `refresh` and `destroy` ([`examples/pulumi/basic`](examples/pulumi/basic/README.md)) |
| **AWS SDKs for JavaScript v3, Go v2, Java v2** | Verified by hand: S3, SQS, DynamoDB and STS ([configuration](docs/aws-compat.md#sdks), [`examples/sdk`](examples/sdk)); the Java and Node.js Testcontainers modules also run the Java and JavaScript SDKs in CI |
| **Other AWS SDKs** (.NET, Rust, …) | Expected to work (same protocols and SigV4), not yet covered by tests. Reports welcome |

Run `homecloud aws-env` to get the environment variables. For Terraform, point the provider's `endpoints {}` block at HomeCloud and use `s3_use_path_style = true`; the [shop example](examples/terraform/shop/README.md) shows a complete provider block.

Expand Down Expand Up @@ -229,7 +231,7 @@ Releases are signed (from v0.5.0): the release workflow signs `checksums.txt` an

- **Shipped in [0.4.0](https://github.com/solinode/homecloud/releases/tag/v0.4.0):** VM-backed EC2 instances (QEMU, KVM when available), the container image `ghcr.io/solinode/homecloud`, a GitHub Action and testcontainers modules for Go and Python, a nightly Terraform modules compatibility suite ([17 of 17 pass](docs/compatibility.md)), and fixes from two security audits ([platform](docs/security-audit-2026-10.md), [VM instances](docs/security-audit-vm-2026-10.md)).
- **Shipped in [0.5.0](https://github.com/solinode/homecloud/releases/tag/v0.5.0):** signed releases (Sigstore) with SBOMs and build provenance, verified by the installer and `homecloud upgrade`; automatic backups before upgrades and `homecloud upgrade --rollback`; restores that bring back databases, caches, load balancers and instances on a new machine; Prometheus metrics and a system status page ([docs/monitoring.md](docs/monitoring.md)); Testcontainers modules for Java and Node.
- **Next:** closing the AWS API gaps users hit most, verified CDK and Pulumi support, native VM acceleration on macOS, passt networking for VMs on Ubuntu 24.04 hosts ([#90](https://github.com/solinode/homecloud/issues/90)), and multiple accounts with SSO for teams and classrooms.
- **Next:** closing the AWS API gaps users hit most, native VM acceleration on macOS, passt networking for VMs on Ubuntu 24.04 hosts ([#90](https://github.com/solinode/homecloud/issues/90)), and multiple accounts with SSO for teams and classrooms.
- **Planned:** multi-node clusters ([#55](https://github.com/solinode/homecloud/issues/55), [design](docs/design/multi-node.md)), edge compute and hardware integrations ([#56](https://github.com/solinode/homecloud/issues/56), [design](docs/design/edge.md)).

See the [open issues](https://github.com/solinode/homecloud/issues) for everything planned.
Expand Down
22 changes: 16 additions & 6 deletions cli/internal/awsapi/awsapi.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,12 @@ import (
"encoding/json"
"errors"
"fmt"
"hash/crc32"
"io"
"log"
"net/http"
"net/url"
"strconv"
"strings"
"sync"
"time"
Expand Down Expand Up @@ -572,12 +574,22 @@ func (q *Req) fail(err error) {
}

func (q *Req) writeJSON(status int, v any) {
q.W.Header().Set("Content-Type", "application/x-amz-json-"+q.Svc.JSONVersion)
q.W.WriteHeader(status)
if v == nil {
v = struct{}{}
}
_ = json.NewEncoder(q.W).Encode(v)
q.writeJSONBody(status, "application/x-amz-json-"+q.Svc.JSONVersion, v)
}

// writeJSONBody writes an awsJson response with its X-Amz-Crc32 header: the
// AWS SDK for Go v2 checks DynamoDB response bodies against it and, when the
// header is missing, compares them with 0 and logs a warning on every call.
func (q *Req) writeJSONBody(status int, contentType string, v any) {
var buf bytes.Buffer
_ = json.NewEncoder(&buf).Encode(v)
q.W.Header().Set("Content-Type", contentType)
q.W.Header().Set("X-Amz-Crc32", strconv.FormatUint(uint64(crc32.ChecksumIEEE(buf.Bytes())), 10))
q.W.WriteHeader(status)
_, _ = q.W.Write(buf.Bytes())
}

func (q *Req) writeJSONError(e *Error) {
Expand All @@ -593,13 +605,11 @@ func (q *Req) writeJSONError(e *Error) {
if q.Svc != nil {
ver = q.Svc.JSONVersion
}
q.W.Header().Set("Content-Type", "application/x-amz-json-"+ver)
q.W.WriteHeader(e.Status)
body := map[string]any{"__type": e.Code, "message": e.Message}
for k, v := range e.Fields {
body[k] = v
}
_ = json.NewEncoder(q.W).Encode(body)
q.writeJSONBody(e.Status, "application/x-amz-json-"+ver, body)
}

func (q *Req) writeRESTJSONError(e *Error) {
Expand Down
28 changes: 28 additions & 0 deletions cli/internal/awsapi/crc32_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
package awsapi

import (
"hash/crc32"
"net/http/httptest"
"strconv"
"testing"
)

// awsJson responses carry X-Amz-Crc32 of the body, as DynamoDB's do: the Go v2
// SDK validates DynamoDB bodies against it (and warns on every call without it).
func TestJSONResponsesCarryCRC32(t *testing.T) {
for name, write := range map[string]func(q *Req){
"result": func(q *Req) { q.writeJSON(200, map[string]any{"Item": map[string]any{"id": map[string]string{"S": "1"}}}) },
"empty": func(q *Req) { q.writeJSON(200, nil) },
"error": func(q *Req) { q.writeJSONError(&Error{Status: 400, Code: "ResourceNotFoundException", Message: "no table"}) },
} {
w := httptest.NewRecorder()
write(&Req{W: w, Svc: &Service{Name: "dynamodb", JSONVersion: "1.0"}})
want := strconv.FormatUint(uint64(crc32.ChecksumIEEE(w.Body.Bytes())), 10)
if got := w.Header().Get("X-Amz-Crc32"); got != want || w.Body.Len() == 0 {
t.Errorf("%s: X-Amz-Crc32 %q, body CRC %s (%q)", name, got, want, w.Body)
}
if ct := w.Header().Get("Content-Type"); ct != "application/x-amz-json-1.0" {
t.Errorf("%s: Content-Type %q", name, ct)
}
}
}
35 changes: 35 additions & 0 deletions cli/internal/dockertest/dockertest_test.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package dockertest

import (
"context"
"os/exec"
"testing"
"time"
Expand Down Expand Up @@ -108,3 +109,37 @@ func TestRemoveOnlyMatchingAccount(t *testing.T) {
t.Error("Remove deleted another test account's volume")
}
}

// OwnContainers leaves out containers of other installations and of other
// tests on the same Docker host (log groups, metrics and the Lambda reaper use it).
func TestOwnContainers(t *testing.T) {
d := Start(t)
other := NewAccount()
t.Cleanup(func() { Remove(d, other, t.Logf) })
if err := d.EnsureImage(context.Background(), "alpine:3.20"); err != nil {
t.Skip("cannot pull alpine: ", err)
}
mk := func(name string, labels map[string]string) string {
t.Helper()
c, err := d.C.CreateContainer(docker.CreateContainerOptions{Name: name, Config: &docker.Config{Image: "alpine:3.20", Labels: labels}})
if err != nil {
t.Fatal(err)
}
return c.ID
}
suffix := core.RandHex(4)
mine := mk("hctest-own-"+suffix, runtime.Labels("lambda", "fn", nil))
theirs := mk("hctest-theirs-"+suffix, map[string]string{core.LabelManaged: "true", core.LabelService: "lambda",
core.LabelResource: "fn", core.LabelAccount: other})
cs, err := d.OwnContainers()
if err != nil {
t.Fatal(err)
}
seen := map[string]bool{}
for _, c := range cs {
seen[c.ID] = true
}
if !seen[mine] || seen[theirs] {
t.Fatalf("OwnContainers: mine %v, other account's %v", seen[mine], seen[theirs])
}
}
16 changes: 16 additions & 0 deletions cli/internal/runtime/docker.go
Original file line number Diff line number Diff line change
Expand Up @@ -451,6 +451,22 @@ func (d *Docker) ManagedContainers() ([]docker.APIContainers, error) {
return d.C.ListContainers(docker.ListContainersOptions{All: true, Filters: map[string][]string{"label": {core.LabelManaged + "=true"}}})
}

// OwnContainers lists the managed containers of this installation (see Mine),
// leaving out those of another account or a running test on the same host.
func (d *Docker) OwnContainers() ([]docker.APIContainers, error) {
cs, err := d.ManagedContainers()
if err != nil {
return nil, err
}
out := cs[:0]
for _, c := range cs {
if Mine(c.Labels) {
out = append(out, c)
}
}
return out, nil
}

// ---- stats ----

type Usage struct {
Expand Down
8 changes: 5 additions & 3 deletions cli/internal/server/tls.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,13 +39,15 @@ func SelfSignedCert(dataDir, publicHost string) (string, string, error) {
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
BasicConstraintsValid: true,
IsCA: true,
DNSNames: []string{"localhost"},
IPAddresses: []net.IP{net.ParseIP("127.0.0.1"), net.ParseIP("::1")},
// S3 Control (which the Terraform and Pulumi AWS providers call for bucket
// tags) puts the account ID in front of the host: 123456789012.localhost.
DNSNames: []string{"localhost", "*.localhost"},
IPAddresses: []net.IP{net.ParseIP("127.0.0.1"), net.ParseIP("::1")},
}
if ip := net.ParseIP(publicHost); ip != nil {
tmpl.IPAddresses = append(tmpl.IPAddresses, ip)
} else if publicHost != "" && publicHost != "localhost" {
tmpl.DNSNames = append(tmpl.DNSNames, publicHost)
tmpl.DNSNames = append(tmpl.DNSNames, publicHost, "*."+publicHost)
}
der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key)
if err != nil {
Expand Down
14 changes: 14 additions & 0 deletions cli/internal/server/tls_test.go
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
package server

import (
"crypto/x509"
"encoding/json"
"encoding/pem"
"net"
"net/http"
"os"
Expand Down Expand Up @@ -40,6 +42,18 @@ func TestSelfSignedTLS(t *testing.T) {
if err := c.Do("GET", "/", nil, &out); err != nil || !out["ok"] {
t.Fatalf("https call through the trusted self-signed cert failed: %v %v", out, err)
}
// Account-prefixed S3 Control hosts are covered too.
pemBytes, _ := os.ReadFile(cert)
blk, _ := pem.Decode(pemBytes)
parsed, err := x509.ParseCertificate(blk.Bytes)
if err != nil {
t.Fatal(err)
}
for _, h := range []string{"localhost", "123456789012.localhost", "homelab.local", "123456789012.homelab.local", "127.0.0.1"} {
if err := parsed.VerifyHostname(h); err != nil {
t.Errorf("%s: %v", h, err)
}
}
// A second call must reuse the existing certificate rather than regenerate it.
cert2, _, _ := SelfSignedCert(dir, "homelab.local")
if cert2 != cert {
Expand Down
10 changes: 10 additions & 0 deletions cli/internal/svc/cfn/aws.go
Original file line number Diff line number Diff line change
Expand Up @@ -876,7 +876,17 @@ func (s *Service) awsExecuteChangeSet(q *awsapi.Req) (any, error) {
}

func (s *Service) awsDeleteChangeSet(q *awsapi.Req) (any, error) {
ref := q.Param("ChangeSetName")
cs, err := s.csAuth(q, "cloudformation:DeleteChangeSet")
var ce *core.Error
if err != nil && errors.As(err, &ce) && ce.Code == "ChangeSetNotFound" && ref != "" && !strings.HasPrefix(ref, "arn:") {
// As in AWS, deleting a change set by name succeeds when the stack
// exists but the change set does not: cdk diff and cdk deploy
// clear their change set name this way before creating it.
if _, serr := s.authStack(q, "cloudformation:DeleteChangeSet", "StackName"); serr == nil {
return nil, nil
}
}
if err != nil {
return nil, err
}
Expand Down
6 changes: 6 additions & 0 deletions cli/internal/svc/cfn/aws_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -523,6 +523,12 @@ Outputs:
if o, err := e.AWSErr(t, "cloudformation", "describe-change-set", "--stack-name", "dep", "--change-set-name", "preview"); err == nil || !strings.Contains(o, "ChangeSetNotFound") {
t.Fatalf("deleted change set: %v %s", err, o)
}
// Deleting a missing change set by name succeeds while the stack exists (cdk diff relies on it),
// and fails for a missing stack.
e.AWS(t, "cloudformation", "delete-change-set", "--stack-name", "dep", "--change-set-name", "preview")
if o, err := e.AWSErr(t, "cloudformation", "delete-change-set", "--stack-name", "nostack", "--change-set-name", "preview"); err == nil {
t.Fatalf("delete change set of a missing stack: %s", o)
}
// An identical template makes a FAILED change set that cannot be executed.
e.AWS(t, "cloudformation", "create-change-set", "--stack-name", "dep", "--change-set-name", "same", "--template-body", "file://"+tf, "--parameters", "ParameterKey=Retention,ParameterValue=120")
d = e.AWSJSON(t, "cloudformation", "describe-change-set", "--stack-name", "dep", "--change-set-name", "same")
Expand Down
10 changes: 10 additions & 0 deletions cli/internal/svc/cfn/awstypes_lambda.go
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,16 @@ func lambdaFunctionProps(x *xctx, in map[string]any) (map[string]any, error) {
if dl := mv(in, "DeadLetterConfig"); dl != nil && has(dl, "TargetArn") {
out["dead_letter_target"] = sv(dl, "TargetArn")
}
// CDK's `logGroup` prop sets LoggingConfig.LogGroup.
if lc := mv(in, "LoggingConfig"); lc != nil {
cfg := map[string]any{}
for _, k := range []string{"LogFormat", "LogGroup", "ApplicationLogLevel", "SystemLogLevel"} {
if has(lc, k) {
cfg[k] = sv(lc, k)
}
}
out["logging_config"] = cfg
}
return out, nil
}

Expand Down
4 changes: 4 additions & 0 deletions cli/internal/svc/cfn/awstypes_lambda_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ Resources:
Variables: {STAGE: prod, N: 3}
Tags:
- {Key: team, Value: platform}
LoggingConfig: {LogGroup: fnstack-logs}
Code:
ZipFile: |
def handler(event, context):
Expand Down Expand Up @@ -203,6 +204,9 @@ func TestLambdaStack(t *testing.T) {
!strings.HasSuffix(cfg["Role"].(string), ":role/"+lastPart(cfg["Role"].(string))) {
t.Fatalf("configuration: %v", cfg)
}
if lc, _ := cfg["LoggingConfig"].(map[string]any); lc["LogGroup"] != "fnstack-logs" || lc["LogFormat"] != "Text" {
t.Fatalf("LoggingConfig: %v", cfg["LoggingConfig"])
}
vars := cfg["Environment"].(map[string]any)["Variables"].(map[string]any)
if vars["STAGE"] != "prod" || vars["N"] != "3" {
t.Fatalf("environment: %v", vars)
Expand Down
9 changes: 5 additions & 4 deletions cli/internal/svc/cfn/awstypes_update.go
Original file line number Diff line number Diff line change
Expand Up @@ -450,7 +450,7 @@ func init() {

// ---- Lambda ----
configure("AWS::Lambda::Function", "FunctionName", []string{"Code", "Runtime", "Handler", "Description", "Timeout", "MemorySize", "Environment", "Role",
"Layers", "Architectures", "Tags", "VpcConfig", "DeadLetterConfig", "ImageConfig", "ReservedConcurrentExecutions"},
"Layers", "Architectures", "Tags", "VpcConfig", "DeadLetterConfig", "ImageConfig", "ReservedConcurrentExecutions", "LoggingConfig"},
func(x *xctx, r *Resource, old, in map[string]any) (map[string]any, error) {
np, err := translate(x, "AWS::Lambda::Function", r, in)
if err != nil {
Expand All @@ -472,18 +472,19 @@ func init() {
}
}
if differs(old, in, "Runtime", "Handler", "Description", "Timeout", "MemorySize", "Environment", "Role", "Layers", "Architectures", "Tags", "VpcConfig",
"DeadLetterConfig", "ImageConfig") {
"DeadLetterConfig", "ImageConfig", "LoggingConfig") {
cfg := map[string]any{}
for _, k := range []string{"runtime", "handler", "description", "timeout_seconds", "memory_mb", "environment", "role", "layers", "architectures", "tags",
"subnet_id", "security_group_ids", "dead_letter_target", "image_config"} {
"subnet_id", "security_group_ids", "dead_letter_target", "image_config", "logging_config"} {
if v, ok := np[k]; ok {
cfg[k] = v
}
}
// what the template dropped goes back to the default
for k, reset := range map[string]map[string]any{"Description": {"description": ""}, "Timeout": {"timeout_seconds": 3}, "MemorySize": {"memory_mb": 128},
"Environment": {"environment": map[string]string{}}, "Layers": {"layers": []string{}}, "Tags": {"tags": core.Tags{}},
"VpcConfig": {"subnet_id": "", "security_group_ids": []string{}}, "DeadLetterConfig": {"dead_letter_target": ""}} {
"VpcConfig": {"subnet_id": "", "security_group_ids": []string{}}, "DeadLetterConfig": {"dead_letter_target": ""},
"LoggingConfig": {"logging_config": map[string]any{}}} {
if has(old, k) && !has(in, k) {
for rk, rv := range reset {
cfg[rk] = rv
Expand Down
4 changes: 2 additions & 2 deletions cli/internal/svc/cloudwatch/logs.go
Original file line number Diff line number Diff line change
Expand Up @@ -494,7 +494,7 @@ func (s *Service) containerGroups() []LogGroup {
if s.env.Docker == nil {
return out
}
cs, err := s.env.Docker.ManagedContainers()
cs, err := s.env.Docker.OwnContainers()
if err != nil {
return out
}
Expand All @@ -515,7 +515,7 @@ func (s *Service) containerFor(group string) string {
return ""
}
svcName, res, _ := strings.Cut(rest, "/")
cs, err := s.env.Docker.ManagedContainers()
cs, err := s.env.Docker.OwnContainers()
if err != nil {
return ""
}
Expand Down
2 changes: 1 addition & 1 deletion cli/internal/svc/cloudwatch/metrics.go
Original file line number Diff line number Diff line change
Expand Up @@ -287,7 +287,7 @@ func (s *Service) collect(ctx context.Context) {
if s.env.Docker == nil {
return
}
cs, err := s.env.Docker.ManagedContainers()
cs, err := s.env.Docker.OwnContainers()
if err != nil {
log.Printf("cloudwatch: list containers: %v", err)
return
Expand Down
Loading
Loading