Page MenuHomePhorge

D135.1732591027.diff
No OneTemporary

Size
19 KB
Referenced Files
None
Subscribers
None

D135.1732591027.diff

diff --git a/design-docs/Make libkazv more sans-io.md b/design-docs/Make libkazv more sans-io.md
deleted file mode 100644
--- a/design-docs/Make libkazv more sans-io.md
+++ /dev/null
@@ -1,128 +0,0 @@
-
-# Make libkazv more sans-io
-
-## Current problem
-
-The library is not sans-io enough. For example, although we made
-`ClientModel` return its jobs and triggers in the reducer, it is
-not easy to keep this pattern once we made our way into E2EE.
-
-Sending encrypted message itself must be done in at least 3 requests,
-and there is a dependency relationship between them:
-
- Send Megolm encrypted event
- Claim keys if needed -> Send session key encrypted with olm
-
-We cannot get such dependency flows in sans-io. We need to do it
-in the IO-integration layer. But how?
-
-## sans-io patterns
-
-Ideally we should have:
-
-```c++
-Promise submitJobs(jobs)
-{
- auto promises = ContainerT{};
- for (auto job : jobs) {
- promises += jobHandler.submit(job)
- .then([=](Response r) { ctx.dispatch(ProcessResponseAction{r}); });
- }
-
- auto p = Promise::all(promises);
-}
-
-Promise Client::doSomething()
-{
- const auto client = +clientReader;
- auto [jobs, triggers] = client.perform(Action);
-
- return submitJobs(jobs);
-}
-```
-
-## Problem with that
-
-But there is difficulty -- `client.perform(Action)` may need to
-modify `client`, for example, encrypting a message will change
-the state of `crypto`. To solve this problem, one can propose:
-
-```
-Promise Client::doSomething()
-{
- const auto client = +clientReader;
- auto newClient = client.perform(Action);
- auto jobs = newClient.popAllJobs();
- auto triggers = newClient.popAllTriggers();
-
- ctx.dispatch(SetClient(newClient));
-
- return submitJobs(jobs);
-}
-```
-
-But it is definitely not thread-safe because between we
-fetch the value in `clientReader` and `SetClient` is actually
-executed, the client may already be changed, resulting in a
-state conflict.
-
-It may be possible then, to wrap the `perform` into another
-action, and do:
-
-```
-??? Client::doSomething()
-{
- ctx.dispatch(PerformAction{Action});
-}
-
-ClientResult updateClient(Client c, PerformAction a)
-{
- auto newClient = client.perform(Action);
- auto jobs = newClient.popAllJobs();
- auto triggers = newClient.popAllTriggers();
-
- return { c, [=](auto &&ctx) { submitJobs(jobs); } };
-}
-```
-
-But where goes the `Promise`? As seen above, we have
-no way to represent "when the submitted jobs finished."
-
-Using the event loop directly?
-
-```
-Promise Client::doSomething()
-{
- return eventLoop.post([=]() {
- const auto client = +clientReader;
- auto newClient = client.perform(Action);
- auto jobs = newClient.popAllJobs();
- auto triggers = newClient.popAllTriggers();
-
- ctx.dispatch(SetClient(newClient));
-
- return submitJobs(jobs);
- });
-}
-```
-
-This actually looks exactly the same as the current
-implementation, except that the current one is done
-in the `SdkModel` reducer:
-
-```
-SdkResult updateSdk(SdkModel s, SdkAction a)
-{
- if (a.is<ClientAction>()) {
- auto newClient = updateClient(s.client, a.to<ClientAction>());
- auto jobs = newClient.popAllJobs();
- auto triggers = newClient.popAllTriggers();
- return { {newClient},
- [=](auto &&ctx) {
- submitJobs(jobs);
- emit(triggers);
- }
- };
- }
-}
-```
diff --git a/design-docs/architecture.md b/design-docs/architecture.md
new file mode 100644
--- /dev/null
+++ b/design-docs/architecture.md
@@ -0,0 +1,441 @@
+
+# The architecture of libkazv
+
+libkazv is a [sans-io](https://sans-io.readthedocs.io/) library based on [lager](https://github.com/arximboldi/lager) implementing the [Matrix client-server API](https://spec.matrix.org/v1.11/client-server-api/).
+
+## lager, enabling the unidirectional data flow
+
+You should probably read the lager docs first: <https://github.com/arximboldi/lager>.
+
+### Model, Action, Reducer
+
+The basics of lager is that you can think of it as a state machine. The **model** is a **value** that represents the application state.
+
+```cpp
+struct Model {
+ /* ... */
+};
+
+Model current;
+```
+
+**Actions** are **values** that describe changes that are to be made to the model. Usually, there are more than one kind of actions, and we can unify them with a `variant`:
+
+```cpp
+struct A1 { ... };
+...
+
+using Action = std::variant<A1, A2, ...>;
+Action action = A1{...};
+```
+
+A **reducer** is a function taking a model and an action, and returns another model and an **effect**. The reducer *applies* the changes described in the action onto the model:
+
+```cpp
+// The reducer function is by convention called update()
+auto [next, effect] = update(current, action);
+```
+
+The effect is then executed against a **context** to perform side-effects:
+
+```cpp
+effect(context);
+```
+
+Effects are optional, though. The reducer may also return only a model and no effect.
+
+```cpp
+auto next = update(current, action);
+```
+
+### Store, Context, Deps
+
+There is a layer above this: the **store** contains the model and tracks the changes to it. The store constructor takes an **event loop**, which accepts **posts** of functions. They are run in a *queue*, and no more than one post can run at the same time. The time frame in which one post is run is called a **tick** of the event loop.
+
+```cpp
+auto initialModel = ...;
+auto eventLoop = ...;
+auto store = makeStore(
+ initialModel,
+ withEventLoop(eventLoop),
+ ...);
+```
+
+A store contains a **context** which effects are executed against. Contexts can be used to **dispatch** actions. A **dispatch** schedules (posts) the following in one tick of the event loop:
+- Run the reducer
+- Update the value of the model
+- Execute the effect against the context.
+
+Here is roughly what happens when an action is **dispatched**:
+
+```cpp
+post([=]() {
+ auto [next, effect] = update(current, action);
+ setCurrent(next);
+ effect(context);
+});
+```
+
+The store can contain **dependencies** when it is created, and these dependencies are available in the context as well:
+
+```cpp
+auto store = makeStore(..., lager::with_deps(depA, depB));
+```
+
+These dependencies can then be retrieved from the context in the effects. This is commonly used to perform IO:
+
+```cpp
+std::pair<Model, Effect> update(Model m, SomeAction a)
+{
+ auto nextModel = ...;
+ auto request = getRequest(a);
+ auto effect = [request](const auto &ctx) {
+ auto ioHandler = std::get<IOHandler>(ctx);
+ ioHandler.httpRequest(request);
+ };
+ return std::make_pair(nextModel, effect);
+}
+```
+
+### Cursors and Lenses
+
+On the other hand, the current value of the model can be obtained from the store via a [**cursor**](https://sinusoid.es/lager/cursors.html). For the purpose of the Kazv Project, the most-used cursor is a read-only one called `lager::reader`. The `get()` method on the cursor returns the current value stored in it.
+
+```cpp
+lager::reader<Model> cursor = store;
+Model model = cursor.get();
+```
+
+Cursors can be **watched**. When the value in a cursor changes, the watch callback is invoked (in the event loop):
+
+```cpp
+lager::watch(cursor, [](auto newModel) {
+ ...
+});
+```
+
+Cursors can also be derived. There are plenty of derivations possible, but the most used ones are **lenses** and **map-transformations**. Map-transformations are simple, it takes a function and returns another cursor:
+
+```cpp
+struct Model {
+ int a;
+ int b;
+};
+
+lager::reader<Model> cursor = ...;
+lager::reader<int> cursorA = cursor.map([](Model m) {
+ return m.a;
+});
+```
+
+[Lenses](https://sinusoid.es/lager/lenses.html) are relationships between two values (usually a whole and a part). For the purpose of libkazv, you can see it as a similar transformation as `map`, but more concise. Commonly used lenses include:
+
+- Attribute lenses:
+ ```cpp
+ struct Model {
+ int a;
+ int b;
+ };
+
+ lager::reader<Model> cursor = ...;
+ lager::reader<int> cursorA = cursor[&Model::a];
+ ```
+- `at` lenses:
+ ```cpp
+ lager::reader<std::vector<int>> cursor = ...;
+ lager::reader<std::optional<int>> cursor0 = cursor[0];
+ ```
+
+ ```cpp
+ lager::reader<std::unordered_map<std::string, int>> cursor = ...;
+ lager::reader<std::optional<int>> cursorA = cursor["A"s];
+ ```
+- Lenses about `optional`:
+ ```cpp
+ lager::reader<std::optional<int>> cursor = ...;
+ lager::reader<int> cursorI = cursor[lager::lenses::value_or(42)];
+ lager::reader<int> cursorI = cursor[lager::lenses::or_default];
+ ```
+
+Cursors can be combined with `lager::with()`:
+
+```cpp
+lager::reader<int> a = ...;
+lager::reader<int> b = ...;
+lager::reader<int> c = lager::with(a, b).map(
+ [](int aValue, int bValue) {
+ return aValue + bValue;
+ });
+```
+
+### Thread-safety
+
+Note that all cursor operations (`get()` and transformations) need to happen in the thread of the event loop. (see <https://github.com/arximboldi/lager/issues/118>) If you need to operate on cursors in another thread, you need to create a **secondary root**. This is done by creating another store with an event loop running in the desired thread, with the action type being the same as the model type, and then watching the primary root:
+
+```cpp
+auto secondaryStore = makeStore(
+ initialModel,
+ lager::with_reducer([](Model, Model next) { return next; }),
+ ... // pass an event loop running in the desired thread
+);
+lager::watch(mainStore, [ctx=secondaryStore.context()](Model m) {
+ ctx.dispatch(std::move(m));
+});
+
+// in the thread of the secondary store's event loop
+auto cursor = secondaryStore.map(...);
+```
+
+This pattern can be used when, for example, the primary store's event loop is running in a worker thread, and the data needs to be passed to the UI thread.
+
+## sans-io
+
+libkazv is sans-io, as it has a business logic layer (the models/reducers in kazvclient and kazvcrypto) and an *abstract* IO layer (Sdk/Client/Room, JobInterface/EventInterface/PromiseInterface).
+
+## How lager is used in libkazv
+
+The models (SdkModel, ClientModel, RoomModel, etc.) represent the state of the client. `Kazv::ClientModel` not only contains the application state, but also describes temporary things such as which **jobs** (network requests) to be sent and which **triggers** (e.g. notifications) to be emitted.
+
+The out-most `Kazv::SdkModel` provides an interface to save and load the state with `boost::serialization`. Its reducer also takes out `Kazv::ClientModel::nextJobs` and `Kazv::ClientModel::nextTriggers`, and send them in the effect returned.
+
+libkazv uses a stronger store (`Kazv::Store`) than lager's builtin store, allowing **then-continuations** (see <https://github.com/arximboldi/lager/issues/96>). We take the concept of **Promises** from [Promises/A+](https://promisesaplus.com/). It is similar to C++'s `std::future`. A `Promise` can be **then**-ed with a **continuation callback**, which is executed after the Promise **resolves**. A Promise can contain data, and the Promise is said to have resolved when the data is ready. For the case of libkazv, the data contained is of type `Kazv::EffectStatus`.
+
+```cpp
+auto promise = ctx.dispatch(action);
+promise.then([](auto status) {
+ // this will be run after the reducer for action is run,
+ // and after the effect returned by the reducer is run.
+ // `status` will contain the value returned from the effect.
+});
+```
+
+The context can create a resolved Promise:
+
+```cpp
+ctx.createResolvedPromise(EffectStatus(...));
+```
+
+A Promise can be created by the context (`Kazv::Context`) through a callback:
+
+```cpp
+ctx.createPromise([](auto resolve) {
+ // do some heavy computations or IO
+ resolve(value);
+}); // this promise will resolve after `resolve` is called
+```
+
+The `resolve()` function can be also called with another Promise. This creates a Promise that is **set to wait for** that Promise. A Promise, `p1`, that waits for another Promise `p2` will resolve after `p2` has resolved, and will contain the same data as `p2`.
+
+```cpp
+ctx.createWaitingPromise([](auto resolve) {
+ // ...
+ auto anotherPromise = ...;
+ resolve(anotherPromise);
+});
+```
+
+Moreover, in libkazv's store, the effect can return a Promise as well. If this is the case, the Promise returned from the dispatch will be set to wait for the Promise returned from the effect.
+
+```cpp
+std::pair<Model, Effect> update(Model m, Action a)
+{
+ return {nextModel, [](const auto &ctx) {
+ auto ioHandler = getIOHandler(ctx);
+ auto p = ctx.createPromise([](auto resolve) {
+ ioHandler.request(..., [](auto value) {
+ auto data = doSomethingWith(value);
+ return resolve(data);
+ });
+ });
+ return p;
+ }};
+}
+
+auto promise = ctx.dispatch(Action{}); // will be resolved after p has resolved
+```
+
+Actually, this is what exactly the reducer of `Kazv::SdkModel` (`Kazv::SdkModel::update`) does: it returns an effect that returns a Promise that resolves after all the jobs in `Kazv::ClientModel::nextJobs` have been sent and we get a response for each of them.
+
+## How to add an action (API request) to libkazv
+
+### Action and Job
+
+First, choose an action name, and add it to src/client/clientfwd.hpp . You need to add both the forward declaration and the `Kazv::ClientAction` variant. They are grouped by types so try to add it next to related actions.
+
+Then, think of what parameters the action takes, and add the struct definition to src/client/client-model.hpp . If this is an API request, it should usually contain all the parameters by the corresponding job constructor in `Kazv::Api` (except for the server url and the access token, which are already given in the `ClientModel`.
+
+Start writing a reducer for your action. In libkazv, the reducer for `ClientAction`s lies in src/client/actions . They are also grouped by types into files, so choose a file with relevant actions to begin with. If there are none, create a new file. In the hpp file, write at least two function signatures:
+
+```cpp
+namespace Kazv
+{
+ ClientResult updateClient(ClientModel m, XXXAction a);
+ ClientResult processResponse(ClientModel m, YYYResponse r);
+}
+```
+
+where `XXXAction` is the name of your action, and `YYY` is the name of the job in `Kazv::Api`. For example, the response type of `LoginJob` is `LoginResponse`. Be sure to include relevant headers in src/api/csapi .
+
+### Reducer, part 1 (or: How to send the request)
+
+In the cpp file, write the definitions of the `updateClient()` function. It needs to add a job from the action.
+
+```cpp
+namespace Kazv
+{
+ ClientResult updateClient(ClientModel m, XXXAction a)
+ {
+ // maybe verify the parameters and return an effect
+ // that returns a failed EffectStatus
+ // do this only if needed
+ if (/* it has invalid parameters */) {
+ return { m, failEffect("MOE.KAZV.MXC_SOME_ERROR_CODE", "some error message") };
+ }
+
+ auto job = m.job<YYYJob>().make(
+ a.param1, a.param2, ...
+ );
+ m.addJob(std::move(job));
+ return { m, lager::noop };
+ }
+}
+```
+
+Write a **unit test** to ensure the correct request is made. This is important because we auto-generate the C++ code for the jobs, and the order of the parameters in the job constructor can differ from version to version of the Matrix spec docs. We use [Catch2-3](https://github.com/catchorg/Catch2) for unit testing.
+
+```cpp
+// src/tests/client/xxx-test.cpp
+#include "client/actions/xxx.hpp"
+
+using namespace Kazv;
+using namespace Kazv::Factory;
+
+TEST_CASE("XXXAction", "[client][xxx]") // add test tags as needed
+{
+ auto client = makeClient(); // see design-docs/libkazvfixtures.md
+ auto [next, _] = updateClient(client, XXXAction{...});
+ assert1Job(next);
+ for1stJob(next, [](const BaseJob &job) {
+ REQUIRE(job.url().find(...) != std::string::npos);
+ auto body = json::parse(std::get<BytesBody>(job.requestBody()));
+ REQUIRE(body.at("param1") == ...);
+ REQUIRE(body.at("param2") == ...);
+ });
+}
+```
+
+If needed, add the test cpp file to `src/tests/CMakeLists.txt`.
+
+### Reducer, part 2 (or: How to process the response)
+
+The next step is to think about how to process the response returned from the server. Does it need to modify any client state, or will the state change be returned from sync? In either case, you need to do **error handling**. This can be done conveniently with helper functions.
+
+```cpp
+// src/client/actions/xxx.cpp
+
+namespace Kazv
+{
+ ClientResult processResponse(ClientModel m, YYYResponse r)
+ {
+ if (!r.success()) {
+ return { std::move(m), failWithResponse(r) };
+ }
+
+ // here the response is successful
+ // modify client state as needed
+ ...;
+ return { std::move(m), lager::noop };
+ // Or, if some data from the response needs to be returned to the user, use:
+ // return { std::move(m), [=](const auto &ctx) {
+ // return EffectStatus(/* succ = */ true, json{
+ // {"result1", result1},
+ // });
+ // }};
+ }
+}
+```
+
+It is important to add the processing of the response to src/client/client-model.cpp , in the reducer for ProcessResponseAction.
+
+```cpp
+[&](ProcessResponseAction a) -> Result {
+ auto r = ...;
+ // ...
+
+ RESPONSE_FOR(YYY);
+ // ...
+}
+```
+
+If you find that processing the response is only possible if you have access to the data that is not present in the response, but only present in the action, you may add additional data to the job in the `updateClient()` function:
+
+```cpp
+ClientResult updateClient(ClientModel m, XXXAction a)
+{
+ auto job = (...).withData(json{
+ {"param1", a.param1},
+ });
+ m.addJob(std::move(job));
+ // ...
+}
+```
+
+And then the response will contain the data json from the corresponding job:
+
+```cpp
+ClientResult processResponse(ClientModel m, YYYResponse r)
+{
+ // ...
+ auto param1 = r.dataStr("param1"); // this gives data["param1"] converted to std::string
+ auto param2 = r.dataJson("param2"); // this gives data["param2"] as json
+ // ...
+}
+```
+
+Similarly, add a test to verify it has the correct handling:
+
+```cpp
+TEST_CASE("Process YYY response", "[client][xxx]")
+{
+ boost::asio::io_context io;
+ AsioPromiseHandler ph{io.get_executor()};
+ auto initialModel = makeClient(...);
+
+ auto store = createTestClientStoreFrom(initialModel, ph);
+ auto client = Client(store.reader().map([](auto c) { return SdkModel{c}; }), store, std::nullopt);
+
+ WHEN("Success response")
+ {
+ auto succResponse = makeResponse("YYY");
+ // or, if data is needed:
+ // auto succResponse = makeResponse("YYY", withResponseDataKV("param1", value) | withResponseKV(...));
+ store.dispatch(ProcessResponseAction{succResponse})
+ .then([client] (auto stat) {
+ REQUIRE(stat.success());
+ // validate that the client is modified accordingly
+ REQUIRE(client.someGetter().make().get() == ...);
+ });
+ }
+
+ WHEN("Failed response")
+ {
+ auto failResponse = makeResponse("YYY", withResponseJsonBody(R"({
+ "errcode": "ERROR_CODE",
+ "error": "error message"
+})"_json) | withResponseStatusCode(403)); // replace with the response body and error code specified in the matrix spec
+ store.dispatch(ProcessResponseAction{failResponse})
+ .then([] (auto stat) {
+ REQUIRE(!stat.success());
+ REQUIRE(stat.dataStr("error") == ...);
+ REQUIRE(stat.dataStr("errorCode") == ...);
+ });
+ }
+
+ io.run();
+}
+
+```
+
+And from here you are done.
+
+Here is a real-world example of adding an action to libkazv: <https://lily-is.land/kazv/libkazv/-/merge_requests/78/diffs>.

File Metadata

Mime Type
text/plain
Expires
Mon, Nov 25, 7:17 PM (11 h, 40 m)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
39874
Default Alt Text
D135.1732591027.diff (19 KB)

Event Timeline