Kibo Template Model¶

Code Generation Explained¶

First, We need to read a pre-parsed binary representation of the DSM Definitions.

This representation is generated with the sub-command encode of the tool dsm_util.py

>>> dsm_util.py encode model.dsm model.dsmb

The steps for generating the model.dsmb are: - Assemble the DSM definition files from multiple dsm files. - Parse the assembled definitions to produce the AST. - Check the semantics of the AST. - Convert the validated definitions to the binary representation.

We read the binary representation to construct a hierarchy of objects where the root object is an instance of the class DSMDefinitions.

This Object-Oriented representation of the DSM Definitions is not adapted for the StringTemplate engine, so we convert it to a template model representation that drastically simplifies the way we implement feature with template.

DSM Model classes are embedded in the template model classes to implement the basic introspection API.

During the conversion, we recursively collect and decompose all types and construct a uniq symbol per type called the type suffix.

A decomposition by type suffix:

• float
  • _float
• vector<float>
  • _vector_float,
  • _float
• map<tuple<int64, float>, vector<key<User>>
  • _map_tuple_int64_float_to_vector_UserKey
  • _tuple_int64_float
  • _int64
  • _float
  • _vector_UserKey
  • _UserKey

The type suffix is used in the template rule to call the generated implementation for the inner type (recursion).

In this example, the rule implements the write function for the generic type map<keyType, elementType>. The keyTypeSuffix and elementTypeSuffix are used to call the implementation of the write function for the keyType and for the elementType.

void StreamWriter::write<v.typeSuffix>(<v.type> const & value) {
    writing->writeUInt64(static_cast\<std::uint64_t>(value.size()));
    for (auto const & [k, v] : value) {
        write<v.keyTypeSuffix>(k);
        write<v.elementTypeSuffix>(v);
    }
}

The generated code for a map<int8, string> is:

void StreamWriter::write_map_int8_to_string(std::map<std::int8_t,std::string> const & value) {
    writing->writeUInt64(static_cast<std::uint64_t>(value.size()));
    for (auto const & [k, v] : value) {
        write_int8(k);
        write_string(v);
    }
}

In_theory, we can implement any feature by recursively consuming the DSM Definitions. In practice, it's an art to elaborate the patterns used to implement a templated feature.

The best way to learn this art is by studying the templates from - com.digitalsubstrate.viper/template/cpp/Model/.stg - com.digitalsubstrate.viper/template/cpp/Stream/.stg

Template Naming Convention¶

Type annotations are not available in StringTemplate, so a good parameter naming convention improves the understanding for the decomposition of the problem into template rules.

There is no convention for rule naming, however the first rule is always named main(m) ::=<..> and the m parameter is a TemplateDefinitions class.

Parameter Name	Usage
m	Definitions (module)
ns	Namespace
a	Attachment
e	Enumeration
em	Enumeration Member
s	Structure
sf	Structure Field
c	Concept
cc	Concept Child
cd	Concept Descendant
c	Club
cm	Club Member
tm	Tuple Member
vm	Variant Member
po	Pool
f	Function
p	Function Parameter
v	other

The Template Model¶

During the construction of the template model, the converter substitutes complicated stuff by a simple string representation.

The string value substituted for the type map<int64, vector<string>> in an instance of the class TemplateMapFunction are indicated in comments.

class TemplateMapFunction {
    ...
    public String type;              // "std::map<std::int64, std::vector<string>>"
    public String typeSuffix         // "_map_int64_to_vector_string"
    public String keyTypeSuffix;     // "_int64"
    public String elementTypeSuffix; // "_vector_string"
    ...
}

In the next section, we use a pseudo class representation to illustrate the field accessible by the StringTemplate engine to generate code and propagate the recursion.

Since the fields for all Template Model classes are self-explanatory, we only explain the use of the fields for the root class TemplateDefinitions to boostrap the code generation.

TemplateDefinitions¶

The root class TemplateDefinitions exposes all the DSM Definitions of the model.dsmb from the Template Model perspective.

The public API seen by the StringTemplate engine.

public class TemplateDefinitions {

    public String generated;
    public String namespace;

    // Namespaces
    public ArrayList<TemplateNameSpace> nameSpaces;

    // Definitions
    public ArrayList<TemplateConcept> concepts;
    public ArrayList<TemplateClub> clubs;
    public ArrayList<TemplateStructure> structures;
    public ArrayList<TemplateEnumeration> enumerations;
    public ArrayList<TemplateAttachment> attachments;

    // Pools 
    public ArrayList<TemplateFunctionPool> functionPools;
    public ArrayList<TemplateCommitFunctionPool> commitFunctionPools;

    // Functions
    public ArrayList<TemplateVecFunction> vecFunctions;
    public ArrayList<TemplateMatFunction> matFunctions;
    public ArrayList<TemplateTupleFunction> tupleFunctions;
    public ArrayList<TemplateOptionalFunction> optionalFunctions;
    public ArrayList<TemplateVectorFunction> vectorFunctions;
    public ArrayList<TemplateSetFunction> setFunctions;
    public ArrayList<TemplateMapFunction> mapFunctions;
    public ArrayList<TemplateXArrayFunction> xarrayFunctions;
    public ArrayList<TemplateVariantFunction> variantFunctions;
}

The m root object is injected in the rule main(m) ::=<<..>> and the recursive code generation start by consuming the fields of the root object.

# Anatomy of a complete Feature 
# Depending of the Feature, you can ommit some fields.

main(m) ::= <<
// Copyright ...
// <m.generated>
...

# generate something by namespace.
<m.nameSpace:namespace(): separator="\n">

# generate something for decomposed types.
<m.vecFunctions:vec(); separator="\n">
<m.matFunctions:mat(); separator="\n">
<m.tupleFunctions:tuple(); separator="\n">
<m.optionalFunctions:optional(); separator="\n">
<m.vectorFunctions:vector(); separator="\n">
<m.setFunctions:set(); separator="\n">
<m.mapFunctions:map(); separator="\n">
<m.xarrayFunctions:xarray(); separator="\n">
<m.variantFunctions:variant(); separator="\n">

>>

namespace(ns) ::= <<
# generate something for enumerations, structures, concepts and clubs.
<ns.concepts:concept(); separator="\n">
<ns.clubs:club(); separator="\n">
<ns.enumerations:enumeration(); separator="\n">
<ns.structures:structure(); separator="\n">

# generate something for attachments.
<ns.attachments:attachments(); separator="\n">
>>

TemplateConcept¶

This class corresponds to the definition of a concept.

public class TemplateConcept {
    // DSM
    public DSMConcept dsmConcept;

    // Components 
    public TemplateConcept parent;
    public String parentNameInNamespace;

    public ArrayList<TemplateConcept> children;
    public ArrayList<TemplateConcept> descendants;
    public ArrayList<TemplateConcept> strictDescendants;
    public ArrayList<TemplateConceptInNamespace> strictDescendantsInNamespace;

    // Namespace 
    public String namespace;
    public String name;

    // Runtime ID 
    public String runtimeId;

    // Documentation 
    public bool hasDocumentation;
    public String documentation;

    // Type 
    public String type;
    public String typeSuffix;

    // Attachments
    public ArrayList<TemplateAttachment> attachments;

    // Viper 
    public String viperType;
    public String viperValue;

    // Python 
    public templatePythonType pythonType;
}

TemplateClub¶

This class corresponds to the definition of a club.

public class TemplateClub {
    // DSM
    public DSMClub dsmClub;

    // Components
    public ArrayList<TemplateConcept> members;
    public ArrayList<TemplateConceptInNamespace> membersInNamespace;
    public ArrayList<TemplateConcept> memberDescendants;

    // Namespace
    public String namespace;
    public String name;

    // Runtime ID
    public String runtimeId

    // Documentation
    public Boolean hasDocumentation;
    public String documentation;

    // Type
    public String type;
    public String typeSuffix;

    // Viper
    public String viperType;
    public String viperValue;

    // Python
    public TemplatePythonType pythonType;
}

TemplateEnumeration¶

This class corresponds to the definition of an enum.

public class TemplateEnumeration {
    // DSM
    public DSMEnumeration dsmEnumeration;

    // Components 
    public ArrayList<DSMEnumerationCase> members;

    // Namespace   
    public String typeId;
    public String name;

    // Documentation
    public Boolean hasDocumentation;
    public String documentation;

    // Type 
    public String type;
    public String typeSuffix;

    // Viper
    public String viperType;
    public String viperValue;

    // Python
    public TemplatePythonType getPythonType;
}

TemplateStructure¶

This class corresponds to the definition of a struct.

public class TemplateStructure {
    // DSM
    public DSMStructure dsmStructure;

    // Components
    public ArrayList<TemplateStructureField> fields;

    // Predicates
    bool isMovable;

    // Namespace 
    public String namespace;
    public String name;

    // Runtime ID
    public String runtimeId;

    // Documentation
    public Boolean hasDocumentation;
    public String documentation;

    // Type
    public String type
    public String typeSuffix;

    // Viper
    public String viperType;
    public String viperValue;

    // Python
    public TemplatePythonType pythonType;
}

public class TemplateStructureField {
    // DSM
    public DSMStructureField dsmField;

    public String name;
    public String passBy;
    public String defaultValue;

    // Predicates
    public bool isMovable;
    public boolean isTypeAny;

    // Documentation
    public Boolean hasDocumentation;
    public String documentation;

    // Type
    public String type;
    public String typeInNamespace;
    public String typeSuffix;

    // Commit 
    public TemplateCommit commit;

    // Viper
    public String viperValue;

    // Python 
    public TemplatePythonType pythonType;
}

TemplateAttachment¶

This class corresponds to the definition of an attachment.

public class TemplateAttachment {
    // DSM
    public String dsmAttachment;

    public String representation;
    public String identifier;

    // Namespace
    public String namespace;
    public String name;

    // Runtime ID 
    public String runtimeId; 

    // Documentation
    public Boolean hasDocumentation;
    public String documentation;

    // Type 
    public TemplateAttachedKeyType keyType;
    public TemplateAttachedDocumentType documentType

    // Python
    public String pythonIdentifier;
}

public class TemplateAttachedKeyType {
    // Namespace
    public String namespace;
    public String name;

    // Type
    public String type;
    public String typeInNamespace;
    public String typeSuffix;

    // Viper
    public String viperValue;

    // Python
    public TemplatePythonType pythonType;
}

public class TemplateAttachedDocumentType {
    // Predicates
    public Boolean isStructure;
    public Boolean useBlobId;

    // Component 
    public TemplateStructure structure;

    // Type
    public String type;
    public String typeInNamespace;
    public String typeSuffix;

    // Commit
    public TemplateCommit commit;

    // Python 
    public TemplatePythonType pythonType;
}

TemplateFunctionPool¶

Those classes correspond to the definition of a function_pool.

public class TemplateFunctionPool {
    // DSM
    public String name;
    public String uuid;

    // Components
    public ArrayList<TemplateFunction> functions;

    // Documentation 
    public bool hasDocumentation;
    public String Documentation;

    // Type 
    public String type;
    public Boolean usePrototypeViper;
    public Boolean usePrototypeData;
}

public class TemplateFunction {
    // DSM
    public String name;

    // Components
    public ArrayList<TemplateFunctionParameter> parameters;

    // Predicates
    public Boolean isVoid;

    // Documentation
    public Boolean hasDocumentation;
    public String documentation;

    // Type
    public String type;
    public String typeSuffix;

    // Viper
    public String returnViperValue;

    // Python
    public TemplatePythonType returnPytonType
}

public class TemplateFunctionParameter {
    // DSM
    public String name;

    public String passBy;

    // Type
    public String type;
    public String typeSuffix;

    // Viper 
    public String viperValue;

    // Python 
    public TemplatePythonType pythonType
}

TemplateCommit¶

This specific class is used to generate collaborative mutators per container.

public class TemplateCommit {
    public String passBy;

    // Predicates
    public Boolean isNotBox;
    public Boolean isBox;
    public Boolean isSet;
    public Boolean isMap;
    public Boolean isXArray;

    // Type 
    public String type;
    public String keyType;
    public String keyTypeSuffix;
    public String elementType;
    public String elementTypeSuffix;

    // Viper
    public String elementTypeviperValue;

    // Python
    public TemplatePythonType pythonKeyType;
    public TemplatePythonType pythonElementType;
}

TemplateCommitFunctionPool¶

Those classes correspond to the definition of a function_pool.

public class TemplateCommitFunctionPool {
    // DSM
    public String name;
    public String uuid;

    // Components
    public ArrayList<TemplateCommitFunction> functions;

    // Documentation
    public boolean hasDocumentation;
    public String documentation;

    // Type
    public String type; 
}

public class TemplateCommitFunction {
    // DSM
    public String name;

    // Components
    public ArrayList<TemplateFunctionParameter> parameters;

    // Predicates
    public boolean isVoid;

    // Documentation
    public Boolean hasDocumentation;
    public String documentation;

    // Type
    public String typeS;
    public String typeSuffix;

    // Commit 
    public String commitType;

    // Viper 
    public String returnviperValue;

    // Python
    public TemplatePythonType returnPythonType;
}

TemplateVecFunction¶

This class corresponds to the definition of a vec<elementType, size>.

public class TemplateVecFunction {
    // DSM
    public String dsmType;
    public String size;

    // Type 
    public String type;
    public String typeSuffix;
    public String elementTypeSuffix;

    // Viper
    public String viperType;
    public String viperValue;

    // Python
    public TemplatePythonType pythonType;
    public TemplatePythonType getPythonElementType;
    public String pythonTupleType;
}

TemplateMatFunction¶

This class corresponds to the definition of a mat<elementType, columns, rows>.

public class TemplateMatFunction {
    // DSM
    public String dsmType;
    public String columns;
    public String rows;

    // Type 
    public String type;
    public String typeSuffix;
    public String elementTypeSuffix;

    // Viper
    public String viperType;
    public String viperValue;

    // Python
    public TemplatePythonType pythonType;
    public TemplatePythonType pythonElementType;
    public String pythonTupleType;
    public String pythonColumnType;
}

TemplateTupleFunction¶

This class corresponds to the definition of a tuple<T0, ...>.

public class TemplateTupleFunction {
    // DSM
    public String dsmType;

    // Components
    public ArrayList<TemplateType> members;

    // Type
    public String type;
    public String typeSuffix;

    // Viper
    public String viperType;
    public String viperValue;

    // Python
    public TemplatePythonType pythonType;
    public ArrayList<TemplatePythonType> pythonMembers;
}

TemplateOptionalFunction¶

This class corresponds to the definition of an optional<elementType>.

public class TemplateOptionalFunction {
    // DSM
    public String dsmType;

    // Type
    public String type;
    public String typeSuffix;
    public String elementType;
    public String elementTypeSuffix;

    // Viper
    public String viperType;
    public String viperValue;

    // Python
    public TemplatePythonType pythonType;
    public TemplatePythonType pythonElementType;

TemplateVectorFunction¶

This class corresponds to the definition of a vector<elementType>.

public class TemplateVectorFunction {
    // DSM
    public String dsmType; 

    public String valueRef;

    // Type
    public String type;
    public String elementTypeSuffix; 

    // Viper
    public String viperType;
    public String viperValue;

    // Python
    public TemplatePythonType pythonType;
    public TemplatePythonType pythonElementType;
}

TemplateSetFunction¶

This class corresponds to the definition of a set<elementType>.

public class TemplateSetFunction {
    // DSM
    public String dsmType;

    // Type
    public String type;
    public String typeSuffix;
    public String elementTypeSuffix

    // Viper
    public String viperType;
    public String viperValue;

    // Python
    public TemplatePythonType pythonType;
    public TemplatePythonType pythonElementType;
}

TemplateMapFunction¶

This class corresponds to the definition of a map<keyType, elementType>.

public class TemplateMapFunction {
    // DSM
    public String dsmType;

    // Type
    public String type;
    public String typeSuffix;
    public String keyTypeSuffix;
    public String elementTypeSuffix;

    // Viper
    public String viperType;
    public String viperValue;

    // Python
    public TemplatePythonType pythonType;
    public TemplatePythonType pythonKeyType;
    public TemplatePythonType pythonElementType;
}

TemplateVariantFunction¶

This class corresponds to the definition of a variant<T0, ...>.

public class TemplateVariantFunction {
    // DSM
    public String dsmType;

    // Components
    public ArrayList<TemplateType> members;

    // Type
    public String type
    public String typeSuffix;
    public String viperType;
    public String viperValue;

    // Python
    public TemplatePythonType pythonType;
    public ArrayList<TemplatePythonType> pythonMembers;
}

TemplateXArray¶

This class corresponds to the definition of a xarray<elementType>

public class TemplateXArrayFunction {
    // DSM
    public String dsmType;

    // Type
    public String type;
    public String typeSuffix
    public String elementTypeSuffix;

    // Viper
    public String viperType;
    public String viperValue;

    // Python
    public TemplatePythonType pythonType;
    public TemplatePythonType pythonElementType;
}

TemplatePythonType¶

This class is used to generate Python proxy classes for Viper.

public class TemplatePythonType {
    // Proxy
    public Boolean useProxy;
    public String proxy;

    // Type
    public String typeSuffix;
    public String type;
}

The DSM Model¶

The DSM Model is the projection of the DSM Definitions in a hierarchy of java class.

The pseudo classes exposed here express the public API that the StringTemplate engine consumes when evaluating rules substitution.

Those classes are embedded in the Template Model class for the basic introspection.

DSMDefinitions¶

This root class references all the DSM Definitions found in the model.dsmb

public class DSMDefinitions {
    public ArrayList<DSMConcept> concepts;
    public ArrayList<DSMClub> clubs;
    public ArrayList<DSMEnumeration> enumerations;
    public ArrayList<DSMStructure> structures;
    public ArrayList<DSMAttachment> attachments;

    public ArrayList<DSMFunctionPool> functionPools;
    public ArrayList<DSMCommitFunctionPool> commitFunctionPools;
}

DSMTypeReference¶

This class is used when we need a reference to another type. To express the list of members of a club...

public class DSMTypeReference {
    public TypeName typeName;
    public DSMTypeReferenceDomain domain;
}

DSMConcept¶

This class corresponds to the definition of a concept.

public class DSMConcept {
    public TypeName typeName;
    public DSMTypeReference parent;
    public String documentation;
    public DSMTypeReference typeReference;
    public UUID runtimeId;
}

DSMClub¶

This class corresponds to the definition of a club with related membership.

public class DSMClub {
    public TypeName typeName;
    public ArrayList<DSMTypeReference> members;
    public String documentation;
    public DSMTypeReference typeReference;
    public UUID runtimeId;
}

DSMEnumeration¶

This class corresponds to the definition of an enum.

public class DSMEnumeration {
    public TypeName typeName;
    public ArrayList<DSMEnumerationCase> members;
    public String documentation;
    public DSMTypeReference typeReference;
    public runtimeId;
}

public class DSMEnumerationCase {
    public String name;
    public String documentation;
}

DSMStructure¶

This class corresponds to the definitions a struct and provides access to the definition of the structure fields.

public class DSMStructure {
    public TypeName typeName;
    public ArrayList<DSMStructureField> fields;
    public String documentation;
    public DSMTypeReference typeReference;
    public runtimeId;
}

public class DSMStructureField {
    public String name;
    public DSMType type;
    public DSMLiteral defaultValue;
    public String documentation;
}

DSM Literal¶

Those classes are used to express the literal value for the initialization of a field.

public abstract class DSMLiteral {
    public abstract String representation();
}

public class DSMLiteralValue extends DSMLiteral {
    public DSMLiteralDomain domain;
    public String value;
}

public final class DSMLiteralList extends DSMLiteral {
    public final ArrayList<DSMLiteral> members;
}

DSMType¶

This class is the base class of types.

public abstract class DSMType {
    public abstract String representation();
}

DSMTypeVec¶

This class corresponds to the definition of a vec<T, n>.

public class DSMTypeVec extends DSMType {
    public DSMTypeReference elementType;
    public long size;
} 

DSMTypeMat¶

This class corresponds to the definition of a mat<T, columns, rows>.

public class DSMTypeMat extends DSMType {
    public DSMTypeReference elementType;
    public long columns;
    public long rows;
}

DSMTypeTuple¶

This class corresponds to the definition of a tuple<T0, ...>.

public class DSMTypeTuple extends DSMType {
    public ArrayList<DSMType> types;
}

DSMTypeOptional¶

This class corresponds to the definition of a optional<T>.

public class DSMTypeOptional extends DSMType {
    public DSMType elementType;
}

DSMTypeVector¶

This class corresponds to the definition of a vector<T>.

public class DSMTypeVector extends DSMType {
    public DSMType elementType;
}

DSMTypeSet¶

This class corresponds to the definition of a set<T>.

public class DSMTypeSet extends DSMType {
    public DSMType elementType;
}

DSMTypeMap¶

This class corresponds to the definition of a map<K, V>.

public class DSMTypeMap extends DSMType {
    public DSMType keyType;
    public DSMType elementType;
}

DSMTypeVariant¶

This class corresponds to the definition of a variant<T0, ...>.

public class DSMTypeVariant extends DSMType {
    public ArrayList<DSMType> types;
}

DSMTypeXArray¶

This class corresponds to the definition of a xarray<elementType>.

public class DSMTypeXArray extends DSMType {
    public DSMType elementType;
}

DSMFunctionPool¶

Those class corresponds to the definitions of function_pool.

public class DSMFunctionPool {
    public UUID uuid;
    public String name;
    public ArrayList<DSMFunction> functions;
    public String documentation;
}

public class DSMFunction {
    public DSMFunctionPrototype prototype;
    public String documentation;
}

DSMFunctionPrototype {
    public String name;
    public ArrayList<DSMFunctionPrototypeParameter> parameters;
    public DSMType returnType;
}

public class DSMFunctionPrototypeParameter {
    public String name;
    public DSMType type;
}

DSMCommitFunctionPool¶

Those class corresponds to the definitions of a commit_function_pool and func.

public class DSMCommitFunctionPool {
    public UUID uuid;
    public String name;
    public ArrayList<DSMCommitFunction> functions;
    public String documentation;
}

public class DSMCommitFunction {
    public boolean isMutable;
    public DSMFunctionPrototype prototype;
    public String documentation;
  }