Viper internal¶

Introduction¶

Viper is a runtime designed with a strong usage of metadata to implement many features like Type, Value, Collection, Function, Serialization, Database, Remote Procedure Call, Network Service, Remote Database, Scripting in Python and Web interoperability with JSON and Bson.

Viper was designed to simplify the way scripting language and C++ work together through metadata. The dynamic aspect of a script interpreter uses the dynamic aspect of the metadata defined in Viper for Type, Value and Function to implement a strong typed data access and function call in a fast and lazy way.

P_Viper (the Python C/API binding of Viper) is like a safe strong-typed layer over Python required for C++ interoperability. P_Viper raises an exception immediately when you use the wrong type, but P_Viper is smart enough to use native Python value and collection as input since the metadata drives everything in Viper.

This document presents the various ideas found in the implementation of the Viper Runtime. Viper is straightforward to understand, and the code is almost self-explanatory (like a pseudocode). The style used is academic, so the runtime could be ported (line by line) to another imperative programming language like swift and C#.

Coding Style¶

Viper doesn't use complicated C++ features. Most of the classes are immutable and the memory management is handled by std::shared_ptr<T>.

This approach allows a shared ownership policy with the Python's objects. It's the foundation of the Python interop used by the seamless wrapper.

File Naming Convention¶

The name of the file mimics the declaration of C++ namespace.

• namespace Viper { class Type { ... }} --> Viper_Type.hpp
• namespace Viper::DSMP { class Checker {...}} --> Viper_DSMP_Checker.hpp

File Content¶

Only one C++ definition per header file.

Object Instantiation¶

To impose the use of std::shared_ptr<T> required by the C++/Python interop, a constructor is always private, and a static method make(...) is used to instantiate an object. This function ensures that parameters are valid or raise an exception, constructs intermediate values, then calls the private constructor which should only initialize instance members (no logic / no exception).

Immutability¶

Since almost class are immutable and accessible by a shared pointer, we can use a public const member. const is used in the east coast revolution approach.

• we used std::shared<T const> const & v (logical/east coast)
• we don't use const std::shared<const T> & v (english dialect/west coast).

Explicit T const an Immutability¶

Since almost all classes are immutable, we use std::shared<T> const & v (implicit immutability) and reserve std::shared<T const> const & for explicit immutability.

Exceptions vs Error¶

Viper defines a single std::exception Viper::Error, and introduces the notion of host, process, component, domain and message. When an error is raised, host, process, component, domain and message are filled by the offended method and a detailed description is constructed to describe the context.

In this example:

[GraphEditor@bma.home]:P_Viper:DecoderErrors:0:expected type 'long', got 'str' while decoding 'checkValue.Vector.at(0).Vec.at(2).UInt32'.

• Process = GraphEditor@bma.home (process@host)
• Component = P_Viper
• Domain = DecoderErrors
• Error Code = 0
• Message = expected type 'long', got 'str' while decoding 'checkValue.Vector.at(0).Vec.at(2).UInt32

Error are transported by the RPC infrastructure. An error raised from a remote function is captured on the server and throws on the client.

Error declinations for a class or a feature are defined in the file <something>Errors.hpp

Examples of Errors files.

• Viper_Key.h --> Viper_KeyErrors.hpp
• Viper_Path.h --> Viper_PathErrors.hpp
• Viper_Function.h --> Viper_FunctionErrors.hpp
• Viper_Stream.h --> Viper_StreamErrors.hpp
• ...

Minimal Exposition and Anonymous Namespace¶

To reduce the exposition of the implementation, we use anonymous namespace to define inner class, and we try to expose a feature as a function, since functions are composable.

// Viper_DefinitionsEncoder.hpp
namespace Viper {

class Definitions;
class StreamCodecInstancing;

namespace DefinitionsEncoder {

Blob encode(std::shared_ptr<Definitions const> const & definitions,
            std::shared_ptr<StreamCodecInstancing> const & codec);

}

} // ns

Only the forward declarations required by the function feature are exposed in the header file, and in the implementation file, we introduce an anonymous namespace to define the class that really implements the feature. The exposed function use the inner class by calling the principal method.

// Viper_DefinitionsEncoder.cpp
namespace Viper::DefinitionsEncoder {

namespace { // ans

class Encoder final {
...
};

} // ans

Blob encode(std::shared_ptr<Definitions const> const & definitions,
            std::shared_ptr<StreamCodecInstancing> const & codec) {
    Encoder encoder {definitions, codec};
    encoder.writeDefinitions();
    return encoder.endEncoding();
}

} // ans

The Type System¶

The type system is explained in Digital_Substrate_Model.

The type system is a dynamic implementation of various types available in the STL. The type system supports bool, integers, reals, tuple<T0, ...>,Optional<T>,vector<T>, set<T>, map<K, V>, variant<T0, ...> and other types introduced by the DSM.

Since the runtime is dynamic, the runtime needs to check the type very often and in a rapid way. We replace the use of std::dynamic_cast<T> by a simple TypeCode to check the type and then we use std::static_cast<T>.

Type codes are generally used in a big switch case, so the compiler can generate a jump table.

All types derive from Viper::Type and implement the pure virtual methods.

namespace Viper {

class Type {
public:
    TypeCode const typeCode;

    // MARK: - Life Cycle
    Type(TypeCode code): typeCode {code} {};
    virtual ~Type() = default;

    // MARK: - Representation
    virtual std::string representation() const = 0;
    virtual std::string representationIn(NameSpace const & nameSpace) const = 0;

    // MARK:  - Description 
    virtual std::string description() const = 0;
    virtual std::string descriptionIn(NameSpace const & nameSpace) const = 0;

    // MARK: - Equality
    virtual bool isEqual(std::shared_ptr<Type> const & other) const = 0;

    // MARK: - Compare
    virtual Ordered compare(std::shared_ptr<Type> const & other) const = 0;
};

} // ns

Types are totally ordered by the TypeCode.

Specific types define the method cast for convenience.

    static std::shared_ptr<TypeSet> cast(std::shared_ptr<Type> const & type);

Generic Types¶

All generic container declare at least a member elementType for their generic parameters.

ex: TypeVector<T> <--> std::vector<T>)

class TypeVector final : public Type {
public:
    std::shared_ptr<Type> const elementType;
    ...

    static std::shared_ptr<TypeVector> make(std::shared_ptr<Type> const & elementType); 
    static std::shared_ptr<TypeVector> cast(std::shared_ptr<Type> const & type);
}

ex: TypeSet<T> <--> std::set<T>

class TypeSet final : public Type {
public:
    std::shared_ptr<Type> const elementType;
    ...
}    

ex: Map<K, V> <--> std::map<K, V>

class TypeMap final : public Type {
public:
    std::shared_ptr<Type> const keyType;
    std::shared_ptr<Type> const elementType;
    ...
}

Data Model, Definitions and RuntimeID.¶

Data modeling is carried out using the DSM (Digital Substrate Model) language. This DSL (Domain-Specific Language) allows you to define the different elements involved in the design of a data model. This DSM makes it possible to define aggregates of information which are linked by unique key to construct complex structured data.

To define a structured, flexible data model, the DSM introduces basically five fundamental notions.

1. Namespace: a space where types are defined.
2. Concept: an abstract thing (like an abstract Type)
3. Key: a way to identify the instantiation of a Concept (like a UUID<Concept>)
4. Document: a piece of information expressible in the type system (ex: struct Document { ... })
5. Attachment: a way to associate a Document with a Key (like a map)

Identifier terminology:

• Namespace ID: the uuid of the namespace used as the seed to generate other Runtime ID.
• Runtime ID: the uuid generated by the runtime for a concept, a club, a struct, an enum or an attachment computed from its definition.

A Viper::Definitions is a catalog of Viper::TypeConcept, Viper::TypeClub, Viper::TypeStructure , Viper::TypeEnumeration, or Viper::TypeAttachment.

Viper::Definitions can be merged and extracted. This capability is the foundation of the service system.

A Runtime ID is required to register those types and only Viper::Definitions can make those types. During the registration of a type, various checks are done to detect possible corruption of the type system and the Runtime ID is generated.

class TypeStructure {
...
private:
    friend Definitions;
    static std::shared_ptr<TypeStructure> make(NameSpace const & nameSpace,
                                               std::shared_ptr<TypeStructureDescriptor> const & descriptor);
...                                               
}

The Value System¶

The value system is a dynamic version of various classes found in the C++ STL. However, the C++ STL use a value semantic and Viper uses a reference semantic.

Viper is like a strong type version of a subset of the Python runtime implemented with the C++ STL. Values must be explicitly copied with Value::copy, like in Python with copy.deepcopy.

Value¶

Since the runtime is dynamic, the runtime needs to check the type of value. This is done by the member type(). All value inherit from Viper::Value and implement the pure virtual methods.

namespace Viper {

class Type;

class Value {
public:
    virtual ~Value() = default;

    // MARK: - Type 
    std::shared_ptr<Type> const type() const = 0;

    // MARK: - Representation
    virtual std::string representation() const = 0;

    // MARK: - Description
    virtual std::string description() const = 0;
    virtual std::string descriptionIn(NameSpace const & nameSpace) const = 0;

    // MARK: - Hash & Equality
    virtual std::size_t hash() const = 0;
    virtual bool isEqual(std::shared_ptr<Value const> const & other) const = 0;

    // MARK: - Compare
    virtual Ordered compare(std::shared_ptr<Value const> const & other) const = 0;

    // MARK: - Helper
    static std::shared_ptr<Value> make(std::shared_ptr<Type> const & type);
    static std::shared_ptr<Value> copy(std::shared_ptr<Value const> const & value);
};

} // ns

A value can be constructed from a type with the universal constructor Viper::Value::create(...).

For each type there is a corresponding value.

• Viper::TypeUInt8 --> Viper::ValueInt8
• Viper::TypeFloat --> Viper::ValueFloat
• Viper::TypeVector --> Viper::ValueVector
• Viper::TypeMap --> Viper::ValueMap

Since primitive values (ints, reals, uuid, blob) are immutable the Viper::Value::copy(...) do nothing. The reference counter is just incremented by the semantic of the std::shared_ptr<T>.

Each generic container uses the corresponding C++ STL containers. Performance are drastically reduced, since the instance is accessible by a std::shared_ptr<T> (allocated in the heap with an atomic reference counter), the type is checked at runtime and a virtual method call is used to collaborate with the C++ STL container API for value equality and hash.

Set<T> copy the element inserted, and Map<K, V> copy the inserted Key to ensure that the STL container is used correctly (value semantic) (no mutation of an element in the set, and no mutation for the key).

In most cases, an element of a set and a key are immutable types, so the copy is just an incrementation of the reference counter. This strong behavior is expressed by the use of a std::shared_ptr<Value const> in the API.

class Set { 
    ...
    void add(std::shared_ptr<Value const> const & value);

    struct cmp {
        // instance passed as a const reference of shared_ptr<>.
        bool operator()(std::shared_ptr<Value const> const & a, std::shared_ptr<Value const> const & b) const {
            return isLess(a->compare(b)); // Virtual function call for the method compare.
        }
    };

    std::set<std::shared_ptr<Value const>, cmp> _storage; // Container adaptation through struct cmp.
    ...

}  

class Map {
  ...
    void set(std::shared_ptr<Value const> const & key, std::shared_ptr<Value> const & value);
    ...

    void remove(std::shared_ptr<Value const> const & key);
    bool contains(std::shared_ptr<Value const> const & key) const;
    ...

    struct cmp {
        bool operator()(std::shared_ptr<Value const> const & a, std::shared_ptr<Value const> const & b) const {
            return isLess(a->compare(b));
        }
    };
    std::map<std::shared_ptr<Value const>, std::shared_ptr<Value>, cmp> _storage; 
  ...
} 

Key¶

A key implements the notion for an instance of a concept.

The tuple (instanceId, runtimeId) uniquely identify an instance.

Concept are possibly related with a parent concept by using the DSM keyword is a. A key can be up-casted and down-casted to a parent key or to a children key.

When a key is created the member Viper::Key::typeConcept keeps the original concept type at the instantiation point. When you upcast or downcast a key, only the member Viper::Key::typeKey changes to represent the actual concept used by the key.

class ValueKey final : public Value {
public:
    std::shared_ptr<TypeKey> const typeKey;
    std::shared_ptr<TypeConcept> const typeConcept;
    UUId const instanceId;
    ...

    // MARK: - Conversion
    bool hasParentKey() const;
    std::shared_ptr<ValueKey> toParentKey() const;
    std::shared_ptr<ValueKey> toConceptKey() const;
    std::shared_ptr<ValueKey> toClubKey(std::shared_ptr<TypeClub> const & typeClub) const;
    std::shared_ptr<ValueKey> toAnyConcept() const;

    bool isMember(std::shared_ptr<TypeConcept> const & targetConcept) const;
    std::shared_ptr<ValueKey> toMemberKey(std::shared_ptr<TypeConcept> const & targetConcept) const;

    std::shared_ptr<ValueKey> toKey(std::shared_ptr<TypeKey> const & typeKey) const;

    ...
}

TypeAnyConcept is not the parent concept of all concepts. it's a container for all key types.

Path¶

A Path is used to locate a piece of information in a composed value (like a struct within another struct...).

A Path is used by the Commit Evaluator to implement the collaborative mutators.

A Path is not typed; it's just a way to iteratively traverse a Viper::Value by resolving the current value at the current Viper::PathComponent until the end of the Path.

The Function System¶

The function system is used to expose a collection of functions to the dynamic runtime.

A Viper::Function is registered in a Viper::FunctionPool and Viper::FunctionPool are registered in a Viper::Service. A service can be promoted to a Viper::RPCServiceClientHandler to implement a public micro Service.

A function is an abstraction and a concrete function implements the pure virtual method checkedCall(...).

// Viper
class Function {
public:
    std::shared_ptr<FunctionPrototype> const prototype;
    std::string const documentation;

    Function(std::shared_ptr<FunctionPrototype> const & prototype,
             std::string const & documentation = {});

    virtual ~Function() = default;

    std::shared_ptr<Value> call(std::vector<std::shared_ptr<Value>> const & args) const;
    std::string representation() const;

protected:
    virtual std::shared_ptr<Value> checkedCall(std::vector<std::shared_ptr<Value>> const & args) const = 0;
};

The concrete implementation for a C++ lambda with <functional>.

class FunctionLambda final : public Function {
public:
std::shared_ptr<FunctionPrototype> const prototype;
std::function<std::shared_ptr<Value>(std::vector<std::shared_ptr<Value>> const &)> const function;
...

}

The C++ generated code for the function int64 add(int64 a, int64 b) declared in a function pool named Tools.

// MARK: - Tools::Add
class F_add final: public Viper::Function {
public:
    static std::shared_ptr<F_add> make() {
        std::vector<Viper::FunctionPrototype::Parameter> parameters;
        parameters.push_back({"a", ValueType::type_int64()});
        parameters.push_back({"b", ValueType::type_int64()});

        auto const returnType {ValueType::type_int64()};

        return std::shared_ptr<F_add>{new F_add {
            Viper::FunctionPrototype::make("add", parameters, returnType),
            "Return a + b."
        }};
    }

    std::shared_ptr<Viper::Value> checkedCall(std::vector<std::shared_ptr<Viper::Value>> const & args) const override {
        auto const a {ValueDecoder::decode_int64(Viper::Int64::cast(args.at(0)))};
        auto const b {ValueDecoder::decode_int64(Viper::Int64::cast(args.at(1)))};

        return ValueEncoder::encode(PoolBridge::Tools::add(a, b));
    }

private:
    F_add(std::shared_ptr<Viper::FunctionPrototype> const & prototype, std::string const & documentation)
    : Viper::Function {prototype, documentation} {}
};

Parsing DSM Definitions¶

Viper::DSMP is the private namespace for parsing the DSM language.

How it works...

1) The content of all DSM files are concatenated in a single content with Viper::DSMBuilder::append(...). 2) The parser transforms this content to an AST representation of Viper::DSMP::Node*. 3) Semantic checkers are run on the AST representation and add their errors with Viper::DSMParseReport::add(...). 4) The AST representation is converted to a Viper::DSMDefinitions if no errors are reported.

// Viper_DSMHelper.cpp

std::shared_ptr<DSMDefinitions> parse_(std::shared_ptr<DSMBuilder> const & builder,
                                       std::shared_ptr<DSMParseReport> const & report) {

    ANTLRInputStream input_stream(builder->source());
    DSMLexer lexer(&input_stream);

    CommonTokenStream token_stream(&lexer);
    DSMParser parser(&token_stream);

    ...

    tree::ParseTree * tree {parser.definitions()};
    auto const v {parser.getNumberOfSyntaxErrors()};
    if (v != 0) {
        return nullptr;
    }

    auto const listener {new Listener};
    tree::ParseTreeWalker walker;
    walker.walk(listener, tree);

    auto const definitions {listener->definitions};
    checkDefinitionsSemantic(definitions, builder, report);

    if (report->hasError()) {
        return nullptr;
    }

    return Converter::convert(definitions);
}

DSM Definitions¶

a Viper::DSMDefinitions is only created if the parsing of the DSM content is valid.

Many immutable Viper::DSM* classes are required for a DSM Definitions. Those classes can be recursively traversed to converter a DSM Definitions to other representations like GraphViz, ULM classes, C++ implementations...

Viper::DSMDefinitions and Viper::DSMDefinitionEncoder are used to transmit the DSM Definitions to the code generator kibo-1.2.0.jar written in java.

a Viper::DSMDefinition can regenerate a DSL representation (DSM Language).

The generated representation can be in plain text or in HTML.

The Codec System¶

Viper defines the notion of codec for converting an instanced object to a binary representation. This binary representation is used for persistence, copy/paste, C++/Value interoperability, Remote Procedure Call, and also to exchange data with external tools like kibo.

There are many encoders and decoders:

• Viper::TypeEncoder/Decoder
• Viper::ValueEncoder/Decoder
• Viper::PathEncoder/Decoder
• Viper::DefinitionEncoder/Decoder
• Viper::FunctionPrototypeEncoder/Decoder
• Viper::ServiceFunctionPoolEncoder/Decoder
• Viper::DSMDefinitionsEncoder/Decoder
• ...

An encoder takes an instance and returns a Blob. A decoder takes a Blob and returns an instance.

Encoder/Decoder are based on Writer/Reader to write/read a Stream.

StreamCodec, StreamEncoder and StreamDecoder¶

Viper implements a Viper::StreamBinaryEncoder and a Viper::StreamTokenBinaryEncoder.

The Viper::StreamTokenBinaryEncoder uses a Viper::StreamBinaryEncoder but inserts a Token before using a method of the Viper::StreamWriting interface. The Stream is Tokenized and a loose of synchronization between an encoder and decoder is detected immediately.

Viper implements Viper::StreamBinaryCodec and Viper::StreamTokenBinaryCodec.

Viper::StreamTokenBinaryEncoder was useful during the development of Viper. We continue to use Viper::StreamTokenBinaryEncoder/Decoder when we exchange data with an insecure world.

Json/Bson Encoder and Decoders¶

To interop with the Web, we use.

• Viper::JSonDSMDefinitionsEncoder to transmit DSM Definitions.
• Viper::JSonDSMDefinitionsEncoder to receive DSM Definitions.
• Viper::JSonValueEncoder to transmit a value.
• Viper::JSonValueDecoder to receive a value.

Bson is also supported.

The RPC system¶

Viper defines various classes to implement the Remote Procedure Call Infrastructure.

A Viper::RPConnection is a specialized BSD Socket for transmitting and receiving a Viper::RPCMessage which contain the binary representation for an instance of a derived class of Viper::RPCPacket registered in a Viper::RPCProtocol.

The RPC infrastructure usages are:

• Access and synchronize Commit databases.
• Browse and access Database in a Remote Database Repository.
• Browse and call Function in Remote Service.

The three associated protocols are defined in Viper::RPCProtocols

namespace Viper {

class RPCProtocol;

namespace RPCProtocols {

std::shared_ptr<RPCProtocol const> remoteCommitDatabasing();
std::shared_ptr<RPCProtocol const> remoteDatabasing();
std::shared_ptr<RPCProtocol const> remoteService();

}} // ns

General purpose packets are shared by protocols, and the shared common feature are implemented in:

• Viper::RPCSideClientCall handles the call/return for the client side.
• Viper::RPCSideServerCall handles the call/return for the server side.
• Viper::StreamReadingHelper implements the deserialization of the packet payload.
• Viper::StreamWritingHelper implements the serialization of the packet payload.

The RPC Infrastructure use the Codec StreamTokenBinary to encode Viper::RPCMessage

RPCPacket¶

All packets inherit from Viper::RPCPacket and implement the pure virtual method write.

namespace Viper {

class StreamCodecInstancing;

class RPCPacket {
public:
    UUId const code;
    std::string const name;

    RPCPacket(UUId const & code, std::string name);
    virtual ~RPCPacket() = default;

    virtual void write(std::shared_ptr<StreamWriting> const & writing) const = 0;
    virtual std::string representation() const;
};

} // ns

Packets are logically discriminated:

• Viper::RPacketCall<something> to call a specific function.
• Viper::RPacketReturn<something> to return a specific value.

Packet are identified by a packetId (UUId) and each packet defines a specific payload.

Hashing and hasher¶

Viper::Hashing defines the interface for hashing.

namespace Viper {

class Hashing {
public:
    virtual ~Hashing() = default;

    virtual void update(void const * data, std::size_t size) = 0;
    virtual void reset() = 0;

    virtual std::string name() const = 0;
    virtual std::size_t digestSize() const = 0;
    virtual std::size_t blockSize() const = 0;

    virtual Blob digest() const = 0;
    virtual std::string hexDigest() const = 0;
};

Concretes implementations are Hasher:

• Viper::HashCRC32
• Viper::HashMD5
• Viper::SHA1
• Viper::SHA3
• Viper::SHA256

SQLite Modules¶

SQLite3 usages are split in modules to be reused in various database implementations.

• Viper::SQlite the C++ wrapper that adapts C type and raise Error.
• Viper::SQliteStatement the C++ wrapper that adapts C type and raise Error.
• Viper::SQliteBlob the C++ wrapper that adapts C type and raise Error.
• Viper::SQliteTableMetadata the metadata API used by Database/Commit Database.
• Viper::SQliteTableBlob the blob API used by Database/Commit Database.
• Viper::SQliteTableAttachment the Instance/Document API use by Database.
• Viper::SQliteTableCommit the Commit API use by Commit database.

Database¶

A Database is a Key-Value database that associates a Document with a Key.

The Primary Key is the tuple formed by (attachmentRuntimeId, instanceId, runtimeId), and the Document is a blob that contains the encoded binary representation_ of the Document (see The codec system).

The implementation is trivial since we delegate the features to the reusable modules Viper::SQliteTableMetadata, Viper::SQliteTableAttachment and Viper::SQliteTableBlob.

Databasing Interface¶

The Viper::Databasing interface defines the Low Level API for an abstract Database.

The concrete implementation are Viper::DatabaseSQLite for a local SQLite3 Database and Viper::DatabaseRemote for a remote repository of Viper::Database.

An application can use this interface to abstract the location of a Database.

Commit¶

Commit is a technology that persists fine-grained mutations of values in a transactional approach.

CommitState & CommitMutableState¶

Values are accessible from an isolated readonly Viper::CommitState through the interface Viper::CommitGetting and mutations only occurs inside a Viper::CommitMutableState through the interface Viper::CommitMutating.

To achieve fine-grained collaborative mutations at any structural level, Commit is based on Path.

Viper::CommitMutating inherits from Viper::CommitGetting, so we can access a modified value in a Viper::CommitMutableState during mutations.

Mutations, Commands and Value¶

When a method of the interface Viper::CommitMutating is called, the arguments are captured in a specific command adapted for the method signature. For each method, there is a Viper::CommitCommandType but many methods have the same signature and use Viper::CommitCommandPathValue.

All commands inherit from Viper::CommitCommand to capture the tuple (attachmentRuntimeId, instanceId, runtimeId).

To reconstruct a Value the Viper::CommitState use the Viper::CommitEvaluator to apply all CommitCommand to a new default initialized value.

Persistence of Mutations¶

To persist the history of all tracked mutations produced in the life cycle of an application, a DAG of Commit is persisted in a Commit Database.

A Commit has a Viper::CommitHeader that identify and describe the Node in the DAG and an optional binary encoded representation of all Viper::CommitCommand if the Viper::CommitType of the Node is a Viper::CommitType::Mutations.

Each Commit is identified by a commitId and references it's parent Commit with parentCommitId to form the DAG of Commit.

To query arbitrary values at a specific Commit, the Commit Engine create a new Viper::CommitState by fetching the persisted data for all Commits involved by the required Commit, to construct a list of Viper::CommitEvalAction.

A Viper::CommitState is then initialized with the list of Viper::CommitEvalAction and implements the Viper::CommitGetting interface with the help of the Viper::CommitEvaluator.

When we query a specific value, the Viper::CommitState/CommitMutableState use the Viper::CommitEvaluator to apply all Viper::CommitCommand of the list of Viper::CommitEvalAction and return an optional reconstructed value.

The return value is optional since the value may not exist for this Commit.

CommitDatabasing Interface¶

Viper::CommiDatabasing is the interface use to save and fetch the data associated for a Commit.

There is an implementation for a local SQLite3 Database Viper::CommitDatabaseSQLite and an implementation for a remote database Viper::CommiteDatabaseRemote.

This interface is used by Viper::CommitDatabase and by the Viper::CommitSynchronizer to synchronize Commit databases.

The interface hides the location of the Database (local or remote).

Database Synchronization¶

Synchronizing means 'merge unknown definitions and add missing commits and blobs.'

Since commits and blob are immutable, the synchronization process is trivial, but to extend the embedded definitions, we need to compute the new definitions inside an exclusive transaction.

void CommitSQLiteDatabase::extendDefinitions(std::shared_ptr<Definitions const> const & other) {
    try {
        _db->beginTransaction(DatabaseTransactionMode::Exclusive);

        auto const defs {Definitions::make()};
        defs->extend(definitions());
        defs->extend(other);
        SQLiteHelper::updateDefinitions(_db, _codec, defs);

        _db->commit();

    } catch (Error const & /*e*/) {
        _db->rollback();
        throw;

    } catch (std::exception const & /*e*/) {
        _db->rollback();
        throw;
    }
}

How it works:

1) Fetch unknown source definitions, commits and blobs and add into the target. 2) Push unknown target definitions, commits and blobs and add into the source.

The algorithm is implemented in Viper::CommitSynchronizer with the help of Viper::CommitSyncronizerHelper.

• Commits are incrementally added by traversing the topological sorted list of Commit to add.
• Small Blobs referenced by a Commit are packed together to reduce the RPC traffic.

Database Conversion¶

Viper::DatabaseToCommitDatabaseConverter converts a Viper::Database to a Viper::CommitDatabase by creating a document in the first Viper::CommitMutableState for all the documents found in the Viper::Database.

Viper::CommitDatabaseToDatabaseConverter converts a Viper::CommitDatabase at a specific commit to a Viper::Database by creating a new document for all documents available in the Viper::CommitState.

Application Model¶

Redux inspires the application model recommended for the commit technology.

How it works.

1) Initialize a CommitStore with a CommitDatabase. 2) Dispatch a mutating function. 3) Persist the mutations into a new Commit. 4) Use the new Commit. 5) Update the undo/redo stack. 6) Notify the application that a new State is present and may update the GUI of represented objects.

This application model is implemented by the class Viper::CommitStoreBase and a concrete Store need to implement the pure virtual methods database(), state(), notifier() and reset().

The Viper::CommitStoreBase has a rich API to:

• Manage the Commit DAG (commit mutations, enable/disable commit, merge heads...)
• Dispatch all sort of executable code (C++ lambda, Commit Function, collaborative mutators...)
• Manage the undo/redo stack (canUndo, canRedo, undo, redo...)

Dispatching¶

Dispatching means 'executing an algorithm that mutates the current state in an isolated context.'

• If the algorithm has run without raising an exception, the mutations are commited to the database within a transaction, and the application is notified that a new state in present, and may update the GUI of all represented objects.

• If the algorithm has raised an exception, then the application is notified that an error has occurred. (It's just a nop from the point of view of the state).

Here is the code from Viper::CommitStoreBase that illustrate the many steps involved to dispatch a mutating function with the collaboration of the virtual methods for a concrete store.

// The dispatch of a C++ lambda
void CommitStoreBase::dispatch(std::string const & label,
                               std::function<void(std::shared_ptr<CommitMutating>const&)> const & function) {

    if (!database()) // use the pure virtual method database()
        throw CommitStoreErrors::noDatabase(module, "dispatch");

    try {                               // 2a) Handle exceptions                               
        auto const ms {mutableState()}; // 2b) Create a new CommitMutableState from the current CommitState.
        function(ms);                   // 2c) Call the function by injecting the CommitMutating interface.
        commitMutations(label, ms);     // 3,4,5,6) Persist the mutations in a new labeled Commit.

    } catch(Viper::Error const & e) {   // 6) Notify the application for an Error (CATCH_NOTIFY_ERROR expanded)
       notifyDispatchError(e);                  

    } catch(std::exception const & e) {
       notifyDispatchError(GeneralErrors::basic(module, e.what()));
    }
}

std::shared_ptr<CommitMutableState> CommitStoreBase::mutableState() const {
    return CommitMutableState::make(state()); // use the virtual methods state() to get the current CommitState.
}

void CommitStoreBase::commitMutations(std::string const & label, std::shared_ptr<CommitMutableState> const & mutableState) {
    if (!database()) // use the virtual method database()
        throw CommitStoreErrors::noDatabase(module, "commitMutations");

    auto const newCommitId {database()->commitMutations(label, mutableState)}; // 3) create a new Commit
    useCommit(newCommitId);            // 4) use the new Commit
    _undoStack.set(state()->commitId); // 5) Update the undo/redo stack.
    notifyStateDidChange();            // 6) Notify the application for a new State.
}

void CommitStoreBase::notifyStateDidChange() {
    if (auto const n {notifier()}) // 6a) use the virtual method to get the concrete Notifier
        n->notifyStateDidChange(); // 6b) notify the application for a new state 
}

void CommitStoreBase::notifyDispatchError(Error const & e) {
    if (auto const n {notifier()}) // 6a) use the virtual method to get the concrete Notifier
        n->notifyDispathError(e);  // 6c) notify the application for an error
}

The dispatch idiom variants:

 void dispatch(std::string const & label,
               std::function<void(std::shared_ptr<CommitMutating>const&)> const & function);

 void dispatchPool(std::string const & label,
                   std::shared_ptr<CommitFunctionPool> const & pool,
                   std::string const & funcName,
                   std::vector<std::shared_ptr<Value>> const & args);

void dispatchSet(std::string const & label,
                 std::shared_ptr<Attachment> const & attachment,
                 std::shared_ptr<Key> const & key,
                 std::shared_ptr<Value const> const & value);

void dispatchDiff(std::string const & label,
                  std::shared_ptr<Attachment> const & attachment,
                  std::shared_ptr<Key> const & key,
                  std::shared_ptr<Value const> const & value,
                  bool recursive);

void dispatchUpdate(std::string const & label,
                    std::shared_ptr<Attachment> const & attachment,
                    std::shared_ptr<Key> const & key,
                    std::shared_ptr<Path const> const & path,
                    std::shared_ptr<Value const> const & value);

void dispatchEnableCommit(CommitId const & commitId, bool enabled);

Notifying the application¶

The concrete implementation of the notifier() methods returns a concrete notifier that implement the Viper::CommitStoreBaseNotifying interface to communicate with the notification system used by the GUI of the application.

ex: A StoreNotifier based on the NotificationCenter for AppKit (Apple).

class StoreNotifier: public Viper::CommitStoreBaseNotifying {
public:
    static std::shared_ptr<StoreNotifier> make();

    void notifyDatabaseDidOpen() override;
    void notifyDatabaseDidClose() override;
    void notifyDatabaseWillReset() override;
    void notifyDefinitionsDidChange() override;
    void notifyStateDidChange() override;
    void notifyDispatchError(Viper::Error const & e) override;
    ... 

private:
    StoreNotifier() = default;
};

std::shared_ptr<StoreNotifier> StoreNotifier::make() {
    return std::shared_ptr<StoreNotifier> {new StoreNotifier()};
}

void StoreNotifier::notifyDatabaseDidOpen() {
    [NSNotificationCenter.defaultCenter postNotificationName:DSCommitStoreBaseDatabaseDidOpenNotification object:nil];
}

void StoreNotifier::notifyDatabaseDidClose() {
    [NSNotificationCenter.defaultCenter postNotificationName:DSCommitStoreBaseDatabaseDidCloseNotification object:nil];
}

void StoreNotifier::notifyDatabaseWillReset() {
    [NSNotificationCenter.defaultCenter postNotificationName:DSCommitStoreBaseDatabaseWillResetNotification object:nil];
}

void StoreNotifier::notifyDefinitionsDidChange() {
    [NSNotificationCenter.defaultCenter postNotificationName:DSCommitStoreBaseDefinitionsDidChangeNotification object:nil];
}

void StoreNotifier::notifyStateDidChange() {
    [NSNotificationCenter.defaultCenter postNotificationName:DSCommitStoreBaseStateDidChangeNotification object:nil];
}

void StoreNotifier::notifyError(Viper::Error const & e) {
    auto const alert {[NSAlert alertWithError:error_convert(e)]};
    [alert beginSheetModalForWindow:NSApp.windows[0] completionHandler:nil];
}
...