Getting Started with DSM¶

The Data Model¶

Data modeling is carried out using the DSM language (Digital Substrate Model). This DSL (Domain-Specific Language) allows you to define the different elements involved in the design of a data model. This DSM makes it possible to define aggregates of information which are linked by unique strong typed key to construct structured data.

To define a data model, the DSM introduces five fundamental notions.

1. Namespace: a space where types are defined.
2. Concept: an abstract thing (like an abstract Type)
3. Key: a way to identify the instantiation of a Concept (like a UUID<Concept>)
4. Document: a piece of information expressible in the type system (ex: struct Document { ... })
5. Attachment: a way to associate a Document with a Key (like a map)

Identifier terminology:

• Namespace ID: the uuid of the namespace used as the seed to generate other RuntimeId.
• Runtime ID: the uuid generated by the runtime for a concept, a club, a struct, an enum or an attachment computed from its definition.

First, we create a folder tuto.

> mkdir tuto
> cd tuto

We create and edit the file model.dsm to define a minimalist data model.

We can install Visual Studio Code Extension and create a Build Task.

namespace Tuto {f529bc42-0618-4f54-a3fb-d55f95c5ad03} {

concept User;

struct Login {
  string nickname;
  string password;
};

struct Identity {
  string firstname;
  string lastname;
};

attachment<User, Login> login;
attachment<User, Identity> identity;

};

The Tool: dsm_util.py¶

dsm_utils.py is a tool to handle tasks for DSM Definitions files.

We will use the dsm_util.py sub-commands:

• check: To check the syntax of the file model.dsm.
• create_commit_database: To create a Commit database.
• create_python_package: To Generate a Python package.

See Getting Started with dsm_util.py for the complete documentation.

We check the syntax of the model.dsm.

> ../tools/dsm_util.py check model.dsm

Commit Database¶

A Commit database keeps the history for all mutations of your data.

A Commit database is a sort of git repository for the documents present in attachments.

Create a Commit database¶

We create a Commit database that embeds the definitions from model.dsm. The database is ready to commit the history of your mutations in a DAG of commits.

> ../tools/dsm_util.py create_commit_database model.dsm model.cdb

The Static Way¶

We have seen the dynamic capabilities of a Commit database, but for an application with a wellknown data model, we prefer (as a developer) to use an approach where for each key and document we have a specific class.

Since the description of the data model is a DSM file (model.dsm), we can use a code generator to build a Python package with classes and type hint to reflect the types defined in the DSM Definitions.

Generate a Python Package¶

The dsm_utils.py sub-command create_python_package create a python package by following those steps.

1. Read, parse, encode and write the DSM Definitions to the file model.dsmb (binary format).
2. Create a folder 'model' and call kibo (the code generator) to render the templated features available for a python package into the folder.
3. Encode the defined types and attachments to a binary resource (resources/definitions.bin)

We can specify the location of the kibo-1.2.0.jar with --kiko and the location of the folder for the templated Python features with --template to render or just the parameter --repos for the root folder of the installed Digital Substrate repositories.

> ../tools/dsm_util.py create_python_package model.dsm

Use the Python package¶

The features of the generated package are:

• model.definitions: The Definitions expressed for the Viper type system.
• model.data: Classes for all concepts, clubs, enumerations and structures.
• model.commit: The Commit API.
• model.database: The Database API.
• model.value_types: The definitions of all types required by the data model.
• ...

We import the generated classes and the commit API from the module 'model' as mc.

> python3
# import the Commit API
>>> import model.commit as mc
>>> mc.Tuto_UserKey
<class 'model.data.Tuto_UserKey'>
>>> mc.Tuto_Login
<class 'model.data.Tuto_Login'>

We open the Commit Database.

>>> from dsviper import *
>>> db = CommitDatabase.open("model.cdb")
>>> state = db.state(db.last_commit_id())

We check that the Commit API use the generated class UserKey and the generated class Login.

# Retrieve all key from the attachment<User, Login>
>>> keys = mc.tuto_user_login_keys(state.commit_getting())
>>> type(keys[0])
<class 'model.data.Tuto_UserKey'>

# Retrieve an optional Document from the attachment
>>> login = mc.tuto_user_login_get(state.commit_getting(), keys[0])
>>> type(login)
<class 'model.data.Optional_Tuto_Login'>

>> type(login.unwrap()) 
<class 'model.data.Tuto_Login'>

# Get a representation of the document
>>> login
Optional({nickname='nick', password='robust'})

We use the generated classes and the commit API to commit mutations in the database.

# Create in key for User
>> key = mc.Tuto_UserKey.create()

# Create the document Login.
>> login = mc.Tuto_Login()
>> login.nickname = "zoopo"

# Commit mutations 
>>> mutable_state = CommitMutableState(db.state(db.last_commit_id()))
>>> mc.tuto_user_login_set(mutable_state.commit_mutating(), key, login) 
>>> db.commit_mutations("Another user", mutable_state)
e4d4de7a88ccd3498f236d55b9798ed48bedbcdd

>>> state = db.state(db.last_commit_id())
>>> mc.tuto_user_login_get(state.commit_getting(), key)
Optional({nickname='zoopo', password=''})

>>> mutable_state = CommitMutableState(db.state(db.last_commit_id()))
>>> mc.tuto_user_login_set_nickname(mutable_state.commit_mutating(), key, "john")
>>> db.commit_mutations("Update nickname", mutable_state)
8cf9f4725baefbff3c725620e857e882a4865679

>>> state = db.state(db.last_commit_id())
>>> mc.tuto_user_login_get(state.commit_getting(), key)
Optional({nickname='john', password=''})

Generate a Wheel for the Package.¶

We can share the generated implementation of your data model by creating a Python wheel.

The command line option --wheel create a minimal pyproject.toml (if is not already present) to boostrap the creation of the wheel.

> ../tools/dsm_util.py create_python_package --wheel --repos /Volumes/DigitalSubstrate model.dsm

We must edit the pyproject.toml to fix the mandatory fields name="" and email="" and all other descriptions required by our wheel (readme, license, classifiers ...).

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "model"
authors = [
    {name = "<name>", email = "<email>"},
]

maintainers = [
    {name = "<name>", email = "<email>"}
]

description = "Python Wrapper for model"
version = "1.0.0"
readme = "README.rst"
requires-python = ">=3.10"
license = {text = "Proprietary"}
classifiers = [
    "Development Status :: 5 - Production/Stable",
    "Programming Language :: Python :: 3",
    "Intended Audience :: Developers",
    "Topic :: Software Development",
]
keywords = ["model"]

[tool.setuptools.packages.find]
include = ["model"]

# Generated files and resources.
[tool.setuptools.package-data]
model = ["*.py", "resources/*.bin"]

We create the wheel.

> pip3 wheel .

We install the wheel in our environment.

> pip3 install --force-reinstall model-1.0.0-py3-none-any.whl

The Dynamic Way¶

Since a Commit database embeds the definitions of types and attachments, we can open, read document and commits mutations (create or update) in any database thanks to the Viper type and value system.

> python3
>>> from dsviper import *
>>> db = CommitDatabase.open("model.cdb")

# Retreive the types defined in the embedded definitions
>>> db.definitions().types()
[Tuto::User, Tuto::Login, Tuto::Identity]

>> db.definitions().attachments()
[attachment<User, Login> Tuto::login, attachment<User, Identity> Tuto::identity]

We inject constants in the python interpreter to have direct access to the types and the attachments defined in the embedded definitions.

The symbols for the constants are constructed from those rules:

• \_T_\ for concept, club.
• \_E_\ for enum.
• \_S_\ for struct.
• \_A_\_\ for attachments.
• \_P_\_\ for a Path to a field.

# Inject constants for types defined in the embedded definitions
>>> db.definitions().inject()

We create a new key from the attachment TUTO_A_USER_LOGIN

A key has an uniq identifier to identify the new instance of the concept

# Create a key from the attachment
>>> key = TUTO_A_USER_LOGIN.create_key()
>>> type(key)
<class 'dsviper.ValueKey'>

>>> key.instance_id()
fc472756-8f9e-42aa-a06f-051d330d0108

# Get the concept of the key.
>>> key.type_concept()
Tuto::User

# Get the runtimeID of the concept
>>> key.type_concept().runtime_id()
bfb135a1-49e6-132a-e693-b84be57e9726

# Get a descriptive representation of a key
>>> key.description()
'fc472756-8f9e-42aa-a06f-051d330d0108:key<Tuto::User>'

We create a new document from the attachment TUTO_A_USER_LOGIN

# Create a document from the attachment
>>> login = TUTO_A_USER_LOGIN.create_document()

>>> type(login)
<class 'dsviper.ValueStructure'>

>>> login
gin
{nickname='', password=''}

We modify the document.

>>> login.nickname = "zoop"
>>> login.password = "robust"

# A more description representation
>>> login.description()
"{nickname='zoop':string, password='robust':string}:Tuto::Login"

The viper type system always checks the type of value.

>>> login.password = 3
Traceback (most recent call last):
  File "<python-input-18>", line 1, in <module>
    login.password = 3
    ^^^^^^^^^^^^^^
dsviper.ViperError: [pid(8497)@mac.home]:P_Viper:P_ViperDecoderErrors:0:expected type 'str', got 'int' while decoding 'checkValue.string'.

To commit the association between key and login in the database, we use the set method of the CommitMutating interface obtained from the mutable state.

A MutableState tracks all the mutations executed by the CommitMutating interface.

Since it's our first commit, we need to create a MutableState from the initial state of the database.

# Create a MutableState from the initial state
>>> mutable_state = CommitMutableState(db.initial_state())

# Associate the document with key in the attachment
>>> mutable_state.commit_mutating().set(TUTO_A_USER_LOGIN, key, login)

We commit the mutations.

# Commit the mutations tracked by the MutableState 
>>> commit_id = db.commit_mutations("First Commit", mutable_state)
>>> commit_id
87b2b1d275f0ac3b2389b18f58d7abe0214c2493

We retrieve the document from an immutable state by using the method get from the CommitGetting interface obtained from the State.

# Get an immutable state from the commit_id.
>>> state = db.state(commit_id)
>>> state.commit_getting().get(TUTO_A_USER_LOGIN, key)
Optional({nickname='zoop', password='robust'})

We commit a new mutation to update the field nickname.

# Use a collaborative setters for the field 'Login.nickname'
>>> mutable_state = CommitMutableState(state)
>>> mutable_state.commit_mutating().update(TUTO_A_USER_LOGIN, key, TUTO_P_LOGIN_NICKNAME, "zoopy")
>>> db.commit_mutations("Update Nickname", mutable_state)
c780c0eb75395bdd8f4f293b70ac24575eebbde5

We retrieve the different values of the document from our commits.

# Get the document from the last_commit_id
>>> state = db.state(db.last_commit_id())
>>> state.commit_getting().get(TUTO_A_USER_LOGIN, key)
Optional({nickname='zoopy', password='robust'})

# Get the document from the first commit
>>> state = db.state(db.first_commit_id())
>>> state.commit_getting().get(TUTO_A_USER_LOGIN, key)
Optional({nickname='zoop', password='robust'})

# Get the document for the initial state.
>>> state = db.initial_state()
>>> state.commit_getting().get(TUTO_A_USER_LOGIN, key)
nil

And to finish, this introduction, we can inspect a CommitHeader.

>>> commit_header = db.commit_header(db.last_commit_id())
>>> commit_header.commit_id(), commit_header.label(), commit_header.parent_commit_id()
(c780c0eb75395bdd8f4f293b70ac24575eebbde5, 'Update Nickname', 87b2b1d275f0ac3b2389b18f58d7abe0214c2493)

For a deep exploration of a CommitData, see Getting Started With Viper