MongoDB + Kafka
Outbox Publisher
A common challenge in microservice development is the need to replicate data between services without introducing tight coupling. The solution to this offered by hexkit is a special form of MongoDbDao grafted with a KafkaEventPublisher, called a MongoKafkaDaoPublisher (“Outbox Publisher”). Contrasted with the other provider in the MongoKafka subpackage, the Outbox Publisher is used in place of the basic MongoDbDao provider and event publishing occurs “in the background”. In addition, instances are created via the factory class, MongoKafkaDaoPublisherFactory. More on that below. Like the Persistent Publisher, the Outbox Publisher also provides methods to publish stored events that haven’t been published yet or to publish all events regardless of publish status. Using this provider requires you to install hexkit with both the akafka and mongodb extras.
The Outbox Publisher is the implementation of the protocol-level DaoPublisher interface; that chapter describes the backend-independent contract, while this section covers the MongoDB + Kafka specifics.
Instantiation/Construction
Construction via factory: Use MongoKafkaDaoPublisherFactory.construct(...).get_dao(...) with:
name: MongoDB collection name.dto_model: The class representing the domain object, which should inherit from Pydantic’sBaseModel.id_field: Name of the ID field on the DTO (mapped to_idin MongoDB). This is used as the event key.dto_to_event(dto) -> JsonObject | None: Maps the DTO to the event payload. ReturnNoneto skip publishing for a given change.event_topic: Kafka topic where outbox events are emitted.autopublish(default: True): If False, changes are stored first and can be published later using publish_pending().fields_to_indexnot currently implemented.
Behavior
Domain object state changes are captured by the Outbox Publisher and categorized as either an upsertion or a deletion. This binary categorization is reflected in the Kafka event types that the Outbox Publisher uses, which are hardcoded by hexkit: CHANGE_EVENT_TYPE (“upserted”) and DELETE_EVENT_TYPE (“deleted”). But the Outbox Publisher is designed to be able to replay changes, so how are deletions preserved? This is accomplished through additional data that is automatically managed by hexkit and stored along with the domain data in the database. This data is stored in a field named __metadata__ and tracks whether the data has been published as an event, what the event ID was for the last time the data was published, the correlation ID for the data, and a boolean indicating whether or not the data was deleted. When data managed by an Outbox Publisher is deleted, it is not entirely removed from the database. Instead, the ID field and metadata are kept (all other fields are deleted) and the __metadata__.deleted field is set to True. Even though the data still partially exists in the database, the Outbox Publisher will raise errors just as if the data did not exist. get_by_id() will raise a ResourceNotFoundError in this case.
Metadata stored on documents:
__metadata__.deleted(bool): Whether the resource has been deleted.__metadata__.published(bool): Whether the most recent change has been published to Kafka.__metadata__.correlation_id(UUID): Correlation ID captured from context when the change was made; reused on (re)publish for tracing.__metadata__.last_event_id(UUID | None): Event ID of the last emitted outbox event.__metadata__.write_id(UUID): Random ID of the latest write. A write is marked as published only while the document still carries its write ID, so a publish that finishes after a newer write neither reverts that write nor marks it as published. Such a publish then repeats with the current state, so that the last event on the topic shows the newest state.
When data is either added or modified deleted, the Outbox Publisher feeds the data into the Factory’s dto_to_event() method to obtain the desired representation for an event. This might match the original data, or it might remove fields, e.g. to avoid transmitting superfluous information. If the result of dto_to_event() is None, no event is published. When data is deleted, the Outbox Publisher publishes an event with an empty payload but the key set to the ID. Consumers can subscribe to the given outbox topic and perform the suitable action based on whether the event type is “upserted” or “deleted” — hexkit’s interface for this is the DAO Subscriber protocol. All event publishing depends on autopublish being set to True.
Republishing events:
- publish_pending(): Publishes all documents where
__metadata__.published == Falseusing the stored correlation ID and appropriate event type. - republish(): Republishes the current state of all documents (change events for non-deleted, tombstones for deleted), again using each document’s stored correlation ID.
Correlation & Event IDs:
- The current correlation ID is captured and stored on each write. A write raises a CorrelationIdContextError if no correlation ID is set in the context.
- During (re)publish, the stored correlation ID is re-applied to the context so downstream consumers see a consistent trace.
- Event IDs are generated every time the data is (re)published, and the ID is stored after the event is published. This is different from the Persistent Publisher in that the latter will reuse the same event ID.
One DAO per collection
A collection served by an Outbox Publisher must be read and written only through Outbox Publishers created with the same dto_model, dto_to_event and event_topic. Several instances of that DAO, such as the replicas of a service, may write the collection at the same time. The rule applies to the collection in its database, so it binds every service and process that opens the same database, not only the one that created the DAO. Collections with the same name in different databases are independent. This includes the usual case of a service that consumes the outbox events and keeps a copy of the resources in its own database: the copy is a different collection, so that service may use any DAO for it, such as a plain MongoDbDao. Opening the outbox collection with another DAO, such as a plain MongoDbDao, or writing it directly is not supported, because such access bypasses the outbox:
- A write publishes no event, so consumers miss the change.
- A write replaces or omits
__metadata__, so an unpublished change drops out of publish_pending(), and a concurrent publish can no longer tell by the write ID that it was overtaken. - An upsert brings a deleted resource back, and a delete removes its stub, so republish() can no longer send the deletion.
- A read does not know deletion stubs, so it fails to validate a stub or returns it as a resource.
Migrations that change documents before the service starts are the exception; they must keep __metadata__ intact.
Usage example
The following shows a contrived example of how to use the Outbox Publisher:
from uuid import UUID
from pydantic import BaseModel
from hexkit.correlation import set_new_correlation_id
from hexkit.providers.mongokafka import MongoKafkaConfig, MongoKafkaDaoPublisherFactory
# Define a model
class User(BaseModel):
user_id: UUID
name: str
email: str
# Determine how to represent the data as a Kafka event
def user_to_event(user: User) -> dict | None:
return {
"user_id": str(user.user_id),
"name": user.name,
"email": user.email,
}
# Use MongoKafkaConfig to supply the Factory with the necessary information
config = MongoKafkaConfig()
# Construct the Factory as an async context manager
async with MongoKafkaDaoPublisherFactory.construct(config=config) as factory:
user_dao = await factory.get_dao(
name="users",
dto_model=User,
id_field="user_id",
dto_to_event=user_to_event,
event_topic="users.outbox",
autopublish=True,
)
# Writes need a correlation ID in the context. A service usually sets it
# per request or gets it from the consumed event; here, set a new one.
async with set_new_correlation_id():
# Create or update data -> publishes a CHANGE event
await user_dao.upsert(
User(
user_id=UUID("2e3975db-2c80-49f1-9f6b-cbb0174ca8f3"),
name="Jane",
email="jane@example.com",
)
)
# Delete data -> publishes a DELETE tombstone
await user_dao.delete(UUID("2e3975db-2c80-49f1-9f6b-cbb0174ca8f3"))
# Operational controls
await user_dao.publish_pending() # Drain any unpublished changes
await user_dao.republish() # Replay full current statePersistent Publisher
With the basic Kafka publish provider (KafkaEventPublisher), event publication is not repeatable. The outbox publisher (MongoKafkaDaoPublisher) allows for a version of event publication, but mandates the use of a DAO in domain logic. There are plausible scenarios where the desired behavior is to publish events and persist them in a database via some transparent mechanism. For this, hexkit offers the PersistentKafkaPublisher (“Persistent Publisher”). The Persistent Publisher replaces existing usage of the KafkaEventPublisher and uses the MongoDB features exposed by hexkit to store a copy of each event as it is published, along with supplementary information such as when the event was published, whether the event was successfully published, the correlation ID associated with the event, and more. Like the Outbox Publisher, the Persistent Publisher also provides methods to publish stored events that haven’t been published yet or to publish all events regardless of publish status. Using this provider requires you to install hexkit with both the akafka and mongodb extras.
To learn more about the distinction between the Persistent Publisher and Outbox Publisher, see this writeup.
Persistence Model
The Persistent Publisher stores data according to the Pydantic model PersistentKafkaEvent:
class PersistentKafkaEvent(BaseModel):
"""A model representing a kafka event to be published and stored in the database."""
compaction_key: str = Field(
...,
description="The unique ID of the event. If the topic is set to be compacted,"
+ " the ID is set to the topic and key in the format <topic>:<key>. Otherwise"
+ " the ID is set to the actual event ID.",
)
topic: Ascii = Field(..., description="The event topic")
payload: JsonObject = Field(..., description="The event payload")
key: Ascii = Field(..., description="The event key")
type_: Ascii = Field(..., description="The event type")
event_id: UUID4 | None = Field(default=None, description="The event ID")
headers: Mapping[str, str] = Field(
default_factory=dict,
description="Non-standard event headers. Correlation ID, event_id, and event"
+ " type are transmitted as event headers, but added as such within the publisher"
+ " protocol. The headers here are any additional header that need to be sent.",
)
correlation_id: UUID4 = UUID4Field(description="The event correlation ID")
created: datetime = Field(
..., description="The timestamp of when the event record was first inserted"
)
published: bool = Field(False, description="Whether the event has been published")Instantiation/Construction
The Persistent Publisher is meant to be used as an async context manager. The instance is created via the construct() method.
There are three parameters unique to the Persistent Publisher: compacted_topics, topics_not_stored, and collection_name.
compacted_topics(set of strings):- Reflects remote topic compaction locally
- Pass a set of topics as
compacted_topicswhen constructing the publisher. - For those topics, the database uses a deterministic
compaction_keyof<topic>:<key>as the document ID, so only the latest event per (topic, key) is stored. This mimics Kafka log compaction on the storage side and makes “republish latest” semantics trivial. - For non-compacted topics, the
compaction_keyis theevent_id(a UUID), so every event is stored independently. This does mean that, for events in non-compacted topics, the event ID is stored as a UUID4 in theevent_idfield and as a string in thecompaction_keyfield.
topics_not_stored(set of strings):- Provides a way to opt out of persistence for specified Kafka topics
- Provide a set of topics that should be published but not stored. Those events are delegated to the plain KafkaEventPublisher that underpins the Persistent Publisher and will not appear in MongoDB.
topics_not_storedandcompacted_topicsmust be disjoint; the provider will raise aValueErrorif there is any overlap.
collection_name:- Determines the name of the collection used for storing events. All events are stored in the same collection.
- By default, events are stored in the collection named
{service_name}PersistedEvents(derived fromMongoKafkaConfig.service_name). - The MongoDB document ID is
compaction_key; this is unique by definition and makes writes idempotent for compacted topics.
Stored information and Considerations
- Payload:
- The event content, the payload, is stored in the field of the same name.
- If developing a database migration that affects persisted events, take care to consider whether the payload itself needs to be updated as well.
- Correlation IDs and headers:
- If a correlation ID is present in the context, it’s captured and stored; otherwise, a new one is generated.
- On (re)publish, the stored correlation ID is set in context so downstream consumers see a stable trace.
- Custom headers you pass are persisted and sent on publish; standard headers (correlation ID, event ID, type) are handled by the publisher protocol.
- Event ID and ordering:
- The
createdfield contains the timestamp denoting when the event was first published. It is not updated upon republishing. - publish_pending loads all documents with
published == false, sorts bycreatedascending, and publishes them. This ensures that events are published in their original order. - republish iterates over all stored events; if an event lacks an
event_id, it assigns one and markspublished = falseso the update is persisted. Then all events are published as above, regardless of whether they have been published already. - Events published to topics listed in
topics_not_storedare never written to MongoDB, thus they won’t be affected by publish_pending or republish.
- The
- Considerations:
- DLQ: Avoid using a Persistent Publisher as the DLQ Publisher for an event subscriber instance. Events should only ever be published once to the DLQ.
- Idempotence: Think about the consumers of events published by the Persistent Publisher and utilize
compacted_topicsandtopics_not_storedstrategically. - Republishing is usually performed as a one-off command for a service, rather than somewhere in standard operation. But this is a convention, and the methods can be utilized as best fits a given use-case.
Usage example in service ‘abc’
from collections.abc import AsyncGenerator
from contextlib import asynccontextmanager
from hexkit.providers.mongodb import MongoDbDaoFactory
from hexkit.providers.mongokafka import MongoKafkaConfig, PersistentKafkaPublisher
# Normally, topics would be defined in configuration. This is only for conciseness:
COMPACTED_TOPICS = {"users"}
TOPICS_NOT_STORED = {"notifications"}
@asynccontextmanager
async def get_persistent_publisher(
config: MongoKafkaConfig,
) -> AsyncGenerator[PersistentKafkaPublisher, None]:
"""Construct and return a PersistentKafkaPublisher."""
async with (
MongoDbDaoFactory.construct(config=config) as dao_factory,
PersistentKafkaPublisher.construct(
config=config,
dao_factory=dao_factory,
compacted_topics=COMPACTED_TOPICS,
topics_not_stored=TOPICS_NOT_STORED,
collection_name="abcPersistedEvents",
) as persistent_publisher,
):
yield persistent_publisherMongoDB + Kafka Test Utils
Integration tests of the Outbox Publisher and the Persistent Publisher need a MongoDB and a Kafka container at the same time. hexkit.providers.mongokafka.testutils combines the MongoDB fixtures and the Kafka fixtures so that a test gets one MongoKafkaConfig for both containers. There is no separate extra for it; install the extras of both fixtures, which the test extra also includes:
pip install hexkit[test-mongodb,test-akafka]Fixtures
The module provides two fixtures:
mongo_kafka_fixture(fixture namemongo_kafka) is function-scoped and yields aMongoKafkaFixture.mongo_kafka_config_fixture(fixture namemongo_kafka_config) is session-scoped and yields only the combined MongoKafkaConfig, for use in session-scoped fixtures of your own.
Neither starts a container. mongo_kafka builds on the fixtures named mongodb and kafka, and mongo_kafka_config on those named mongodb_container and kafka_container. Import all of them into the test module or its conftest.py, hence the # noqa: F401 comments.
This test checks that the users DAO from the usage example publishes a change event on insert. The DAO’s writes raise a CorrelationIdContextError without a correlation ID in the context, so the test sets one with set_new_correlation_id():
from uuid import uuid4
import pytest
from hexkit.correlation import set_new_correlation_id
from hexkit.providers.akafka.testutils import (
ExpectedEvent,
kafka_container_fixture, # noqa: F401
kafka_fixture, # noqa: F401
)
from hexkit.providers.mongodb.testutils import (
mongodb_container_fixture, # noqa: F401
mongodb_fixture, # noqa: F401
)
from hexkit.providers.mongokafka import (
CHANGE_EVENT_TYPE,
MongoKafkaDaoPublisherFactory,
)
from hexkit.providers.mongokafka.testutils import (
MongoKafkaFixture,
mongo_kafka_fixture, # noqa: F401
)
@pytest.mark.asyncio()
async def test_user_outbox(mongo_kafka: MongoKafkaFixture):
user = User(user_id=uuid4(), name="Jane", email="jane@example.com")
async with MongoKafkaDaoPublisherFactory.construct(
config=mongo_kafka.config
) as factory:
user_dao = await factory.get_dao(
name="users",
dto_model=User,
id_field="user_id",
dto_to_event=user_to_event,
event_topic="users.outbox",
)
async with (
set_new_correlation_id(),
mongo_kafka.kafka.expect_events(
events=[
ExpectedEvent(
payload=user_to_event(user),
type_=CHANGE_EVENT_TYPE,
key=str(user.user_id),
)
],
in_topic="users.outbox",
),
):
await user_dao.insert(user)mongo_kafka does no cleanup of its own: the mongodb and kafka fixtures do it. With mongodb_fixture and kafka_fixture, as above, each test starts with no collections and empty topics. To keep state between tests, import persistent_mongodb_fixture and persistent_kafka_fixture instead.
Unlike the MongoDB and Kafka fixtures, these two have no factory functions, so their scopes and names are fixed. If you rename the MongoDB or Kafka fixtures with their factory functions, mongo_kafka and mongo_kafka_config no longer find them.
The MongoKafkaFixture
A MongoKafkaFixture is a named tuple with three fields:
mongodb: theMongoDbFixture, whoseclientlets a test inspect stored documents, including their__metadata__.kafka: theKafkaFixture, for publishing test events and asserting on published ones.config: the MongoKafkaConfig that combines the configs of both fixtures, to pass to MongoKafkaDaoPublisherFactory.construct() or PersistentKafkaPublisher.construct().
The Persistent Publisher also needs a DAO factory; pass it mongo_kafka.mongodb.dao_factory.