Skip to main content

Providers

A provider implements a protocol for one technology: MongoDbDaoFactory, for example, implements the DaoFactoryProtocol for MongoDB, and KafkaEventPublisher implements the EventPublisherProtocol for Apache Kafka. A service writes its translators against the protocols and chooses the providers in its dependency injection. Switching to another backend, or to an in-memory provider in a test, then needs no change to the translators.

Providers work in one of two directions:

Each technology’s client library is an optional extra of hexkit, so a service installs only the ones it uses, for example pip install hexkit[akafka,mongodb]. The test-* extras add pytest fixtures that run the backend in a testcontainer. The test extra installs all of them except test-redis, and the all extra installs test and the opentelemetry extra.

Each provider for a real backend has a config class, such as MongoDbConfig, from which the service’s config class inherits; see Configuration.

Providers by Technology

Apache Kafka

Extra: akafka; test fixtures: test-akafka. See Apache Kafka.

Provider Protocol Purpose
KafkaEventPublisher EventPublisherProtocol Publishes events to Kafka topics.
KafkaEventSubscriber Runs a translator implementing the EventSubscriberProtocol or the DLQSubscriberProtocol Consumes events and passes them to the translator; can retry failed events and send them to a dead letter queue.
KafkaOutboxSubscriber Runs translators implementing the DaoSubscriberProtocol Consumes another service’s outbox events to keep a copy of its resources.

MongoDB

Extra: mongodb; test fixtures: test-mongodb. See MongoDB.

Provider Protocol Purpose
MongoDbDaoFactory DaoFactoryProtocol; its DAOs implement Dao Stores resources as documents, one collection per resource type, with queries and indexes.
MongoDb*KeyValueStore KeyValueStoreProtocol Stores each value as a document in a collection, for durable structured values.

The same extra provides the migration tools, which migrate stored documents when a service’s data model changes and implement no protocol.

MongoDB + Kafka

Extras: akafka and mongodb; test fixtures: test-akafka and test-mongodb. See MongoDB + Kafka.

Provider Protocol Purpose
MongoKafkaDaoPublisherFactory DaoPublisherFactoryProtocol; its DAOs implement DaoPublisher Stores resources in MongoDB and publishes their changes as events, implementing the outbox pattern.
PersistentKafkaPublisher EventPublisherProtocol Publishes events to Kafka and stores them in MongoDB, so they can be published again.

For the difference between the two, see Outbox vs. Persistent Publisher.

S3

Extra: s3; test fixtures: test-s3. See S3.

Provider Protocol Purpose
S3ObjectStorage ObjectStorageProtocol Manages buckets and objects on S3-compatible storage, and creates presigned URLs for uploads and downloads.
S3*KeyValueStore KeyValueStoreProtocol Stores each value as an object in a bucket, for large or opaque values.

Redis

Extra: redis; test fixtures: test-redis. See Redis.

Provider Protocol Purpose
Redis*KeyValueStore KeyValueStoreProtocol Keeps values in Redis, for small values that are read often, such as cached results or tokens.

HashiCorp Vault

Extra: vault; test fixtures: test-vault. See Vault.

Provider Protocol Purpose
Vault*KeyValueStore KeyValueStoreProtocol Stores each value as a secret in Vault, for credentials and other sensitive values.

In-Memory (Testing)

Extra: mongodb, because hexkit.providers.testing imports the mock DAO, which reuses the MongoDB provider’s document conversion. See In-Memory (Testing).

Provider Protocol Purpose
new_mock_dao_class() Dao Creates a DAO class that keeps resources in a dictionary.
InMemEventPublisher EventPublisherProtocol Collects published events in an InMemEventStore that tests read from.
InMem*KeyValueStore KeyValueStoreProtocol Keeps values in a dictionary.
InMemObjectStorage ObjectStorageProtocol Tracks buckets and object IDs, without file content.

These providers are for unit tests only.

Adding Providers

A new provider implements an existing protocol for a backend. If no protocol fits, define one first; see Adding Protocols.

  1. Package: put the provider in the backend’s subpackage, src/hexkit/providers/<backend>/, with the implementation in provider/. Export its public names from the subpackage’s __init__.py.
  2. Dependencies: for a new backend, add its client library as an optional extra in pyproject.toml, named after the backend. Add a test-<backend> extra that adds testcontainers.
  3. Configuration: for a new backend, add a <Backend>Config class in config.py, and store credentials as SecretStr or Secret[...]. A service’s config class inherits from the config classes of all its providers. Prefix the field names with the backend, as in redis_url or s3_endpoint_url, so they do not clash.
  4. Errors: where the protocol defines errors, such as the DAO errors or ObjectStorageProtocol.BucketNotFoundError, raise those instead of the client library’s errors. The service’s error handling then keeps working with another provider.
  5. Connection: if the provider holds a client connection, add a construct() async context manager that opens the connection on entry and closes it on exit. The Kafka, MongoDB, Redis and Vault providers do this.
  6. Test utils: for a new backend, add a testutils.py with a session-scoped container fixture, and a function-scoped clean fixture with a persistent variant. Add get_* factory functions for all three, as the existing providers do.
  7. Tests: test the provider in tests/integration/providers/ against a testcontainer. For a key-value store, subclass the shared test suites in tests/integration/providers/kvstore/kvstore_test_suites.py.
  8. Documentation: write a page in user_guide/providers/, and add the provider to this page. In great-docs.yml, add the page to the Providers section of user_guide, and the classes to a Provider — section of reference.