Providers
A provider implements a protocol for one technology: MongoDbDaoFactory, for example, implements the DaoFactoryProtocol for MongoDB, and KafkaEventPublisher implements the EventPublisherProtocol for Apache Kafka. A service writes its translators against the protocols and chooses the providers in its dependency injection. Switching to another backend, or to an in-memory provider in a test, then needs no change to the translators.
Providers work in one of two directions:
- Outbound providers, such as DAO factories, event publishers and key-value stores, implement a protocol, and the service calls them.
- Inbound providers, the event subscribers, call the service’s translators, which implement the protocol.
Each technology’s client library is an optional extra of hexkit, so a service installs only the ones it uses, for example pip install hexkit[akafka,mongodb]. The test-* extras add pytest fixtures that run the backend in a testcontainer. The test extra installs all of them except test-redis, and the all extra installs test and the opentelemetry extra.
Each provider for a real backend has a config class, such as MongoDbConfig, from which the service’s config class inherits; see Configuration.
Providers by Technology
Apache Kafka
Extra: akafka; test fixtures: test-akafka. See Apache Kafka.
| Provider | Protocol | Purpose |
|---|---|---|
KafkaEventPublisher |
EventPublisherProtocol |
Publishes events to Kafka topics. |
KafkaEventSubscriber |
Runs a translator implementing the EventSubscriberProtocol or the DLQSubscriberProtocol |
Consumes events and passes them to the translator; can retry failed events and send them to a dead letter queue. |
KafkaOutboxSubscriber |
Runs translators implementing the DaoSubscriberProtocol |
Consumes another service’s outbox events to keep a copy of its resources. |
MongoDB
Extra: mongodb; test fixtures: test-mongodb. See MongoDB.
| Provider | Protocol | Purpose |
|---|---|---|
MongoDbDaoFactory |
DaoFactoryProtocol; its DAOs implement Dao |
Stores resources as documents, one collection per resource type, with queries and indexes. |
MongoDb*KeyValueStore |
KeyValueStoreProtocol |
Stores each value as a document in a collection, for durable structured values. |
The same extra provides the migration tools, which migrate stored documents when a service’s data model changes and implement no protocol.
MongoDB + Kafka
Extras: akafka and mongodb; test fixtures: test-akafka and test-mongodb. See MongoDB + Kafka.
| Provider | Protocol | Purpose |
|---|---|---|
MongoKafkaDaoPublisherFactory |
DaoPublisherFactoryProtocol; its DAOs implement DaoPublisher |
Stores resources in MongoDB and publishes their changes as events, implementing the outbox pattern. |
PersistentKafkaPublisher |
EventPublisherProtocol |
Publishes events to Kafka and stores them in MongoDB, so they can be published again. |
For the difference between the two, see Outbox vs. Persistent Publisher.
S3
Extra: s3; test fixtures: test-s3. See S3.
| Provider | Protocol | Purpose |
|---|---|---|
S3ObjectStorage |
ObjectStorageProtocol |
Manages buckets and objects on S3-compatible storage, and creates presigned URLs for uploads and downloads. |
S3*KeyValueStore |
KeyValueStoreProtocol |
Stores each value as an object in a bucket, for large or opaque values. |
Redis
Extra: redis; test fixtures: test-redis. See Redis.
| Provider | Protocol | Purpose |
|---|---|---|
Redis*KeyValueStore |
KeyValueStoreProtocol |
Keeps values in Redis, for small values that are read often, such as cached results or tokens. |
HashiCorp Vault
Extra: vault; test fixtures: test-vault. See Vault.
| Provider | Protocol | Purpose |
|---|---|---|
Vault*KeyValueStore |
KeyValueStoreProtocol |
Stores each value as a secret in Vault, for credentials and other sensitive values. |
In-Memory (Testing)
Extra: mongodb, because hexkit.providers.testing imports the mock DAO, which reuses the MongoDB provider’s document conversion. See In-Memory (Testing).
| Provider | Protocol | Purpose |
|---|---|---|
new_mock_dao_class() |
Dao |
Creates a DAO class that keeps resources in a dictionary. |
InMemEventPublisher |
EventPublisherProtocol |
Collects published events in an InMemEventStore that tests read from. |
InMem*KeyValueStore |
KeyValueStoreProtocol |
Keeps values in a dictionary. |
InMemObjectStorage |
ObjectStorageProtocol |
Tracks buckets and object IDs, without file content. |
These providers are for unit tests only.
Adding Providers
A new provider implements an existing protocol for a backend. If no protocol fits, define one first; see Adding Protocols.
- Package: put the provider in the backend’s subpackage,
src/hexkit/providers/<backend>/, with the implementation inprovider/. Export its public names from the subpackage’s__init__.py. - Dependencies: for a new backend, add its client library as an optional extra in
pyproject.toml, named after the backend. Add atest-<backend>extra that adds testcontainers. - Configuration: for a new backend, add a
<Backend>Configclass inconfig.py, and store credentials asSecretStrorSecret[...]. A service’s config class inherits from the config classes of all its providers. Prefix the field names with the backend, as inredis_urlors3_endpoint_url, so they do not clash. - Errors: where the protocol defines errors, such as the DAO errors or ObjectStorageProtocol.BucketNotFoundError, raise those instead of the client library’s errors. The service’s error handling then keeps working with another provider.
- Connection: if the provider holds a client connection, add a construct() async context manager that opens the connection on entry and closes it on exit. The Kafka, MongoDB, Redis and Vault providers do this.
- Test utils: for a new backend, add a
testutils.pywith a session-scoped container fixture, and a function-scoped clean fixture with a persistent variant. Addget_*factory functions for all three, as the existing providers do. - Tests: test the provider in
tests/integration/providers/against a testcontainer. For a key-value store, subclass the shared test suites intests/integration/providers/kvstore/kvstore_test_suites.py. - Documentation: write a page in
user_guide/providers/, and add the provider to this page. Ingreat-docs.yml, add the page to theProviderssection ofuser_guide, and the classes to aProvider —section ofreference.