Backend systems for credit risk, built to hold up under scrutiny.

Each service owns its own database and has no knowledge of the others — they only know the shape of the events on the Kafka topics they publish to or consume from. The Order Service writes the order to its database, then publishes an OrderCreatedEvent. The Payment Service, listening on that topic, picks it up, simulates a payment, and publishes a PaymentCompletedEvent of its own. The Notification Service consumes that and logs a confirmation — three services, no direct calls between them, fully decoupled.

Real problem I hit: a poison message stuck the consumer in a retry loop

While testing manually, I pushed a plain string onto the order-events topic instead of a valid JSON payload. The Payment Service's consumer tried to deserialize it into an OrderCreatedEvent, failed, and — because Kafka had already committed that the consumer was "at" that offset for its consumer group — kept retrying the same bad message indefinitely on every restart. The fix taught me how Kafka actually tracks read position: offsets are tracked per consumer group, not per message, so the two real options are moving the group to a fresh offset (effectively a new group-id) or deleting and recreating the topic. I went with the consumer-group reset for a clean fix without losing the topic's other test data, and it's the reason I'm comfortable explaining consumer offset behavior in an interview rather than just reciting it.

Design patterns in play

The system follows the producer-consumer pattern at the messaging layer, with each service applying a simple repository pattern over its own database. The overall data flow accepts eventual consistency by design — there's a deliberate small window where an order exists but payment hasn't processed yet, which is the correct tradeoff for this kind of asynchronous pipeline rather than trying to force synchronous consistency across three services.

The core feature is a POST /wallets/transfer endpoint that moves money between two wallets atomically — both balances update, or neither does. The naive approach (two separate UPDATE statements) breaks under concurrency and crash scenarios: if the process dies between the debit and the credit, money disappears. I implemented it as a single database transaction with row-level locking (SELECT ... FOR UPDATE) on both wallet rows before reading balances or applying updates.

Real problem I hit: lock ordering and a cross-process deadlock

The first version locked the sender's row, then the receiver's row, in whatever order they arrived in the request. Two concurrent transfers in opposite directions (A→B and B→A) could each lock one row and then block waiting for the other — a classic deadlock. The fix was to always lock the two wallet rows in a consistent order regardless of who's sending and who's receiving (lower user ID first), so two concurrent transfers can never form a circular wait.

Separately, when I moved audit logging off the request path into a background worker, I first tried passing events through an in-memory Go channel between the API and worker. That immediately hit fatal error: all goroutines are asleep - deadlock! — because cmd/server and cmd/worker are two separate OS processes with separate memory spaces; a channel only works within a single process. That was a genuinely useful mistake — it's what pushed me to replace the channel with a Redis-backed queue (LPUSH from the API, blocking BRPOP in the worker), which actually works across process boundaries.

Design and testing

The project follows Go's standard cmd/ + internal/ layout, with each domain (user, wallet, audit) split into model, repository, and handler files to keep HTTP concerns out of business logic. I added integration tests against a real Postgres instance covering the success path and the insufficient-balance rejection path, plus health checks for both Postgres and Redis and graceful shutdown handling for the worker process.