P_Viper Internal¶

Introduction¶

P_Viper is the Python C/API binding of Viper is like a safe strong-typed layer over python required for C++ interoperability. P_Viper raises an exception immediately when you use the wrong type, but P_Viper is smart enough to use native python value and collection as input since the metadata drives everything in Viper.

Filename Convention¶

Wrapped Classes¶

Wrapped class for Viper Classes.

• Viper::TypeSet --> "P_Viper_TypeSet.hpp"
• Viper::ValueSet --> "P_Viper_ValueSet.hpp"
• Viper::Definitions --> "P_Viper_Definitions.hpp"
• Viper::DSMDefinitions --> "P_Viper_DSMDefinitions.hpp"
• ...

Wrapper Features¶

Classes involved in the implementation of Python wrapper.

• P_ViperSRef.hpp
• P_ViperCaptureIO.hpp
• P_ViperWrapper.hpp
• P_ViperEncoder.hpp
• P_ViperEncoderDeep.hpp
• ...

The Wrapped Pattern¶

Since Python is in C and Viper is in C++, we need to:

• Use a pointer to hide the C++ implementation of Viper in a custom Python Object.
• Catch all c++ exceptions and return a value adapted for convention defined by Python for reporting errors.
• Handle the Python reference counter with Py_XDECREF in C++ block for Python new reference convention.

The Pattern for embedding a std::shared_ptr<T>¶

We used a pointer std::shared_ptr<T> * v to implement the shared ownership policy between the Viper runtime and the Python runtime.

#include "Viper_TypeVector.hpp"

typedef struct {
    PyObject_HEAD
    std::shared_ptr<Viper::TypeVector> * v; // embed a C++ std::shared_ptr<T> in a C struct;
} P_Viper_TypeVector;

...

// create a new C Python Object that embed a std::shared_ptr of a Viper Object.
PyObject * P_Viper_TypeVector_New(std::shared_ptr<Viper::TypeVector> const & v);
...

#endif

The Pattern for New¶

PyObject * P_Viper_TypeVector_New(std::shared_ptr<TypeVector> const & v) {
    auto const pn_o {_PyObject_New(reinterpret_cast<PyTypeObject *>(P_Viper_TypeVector_Type))};
    auto const self {reinterpret_cast<P_Viper_TypeVector *>(pn_o)};
    self->v = new std::shared_ptr(v); // new => increment the reference counter of the C++ Object.
    return pn_o;
}

The Pattern for Free¶

static void tp_dealloc(P_Viper_Blob * self) {
    delete self->v; // delete => decrement the reference counter of the C++ Object

    auto const type {Py_TYPE(self)};
    type->tp_free(self);
    Py_DECREF(type);
}

The Pattern for Catching C++ Exception¶

To protect the Python runtime from the C++ exception, we use the macro P_TRY to create a safe context, and the macro P_CATCH_P to catch the exception for a C/Python API that require a null pointer to indicate an error or the macro P_CATCH_I to catch the exception for a C/Python API that returns an error with an integer;

ex: return a PyObject * (nullptr => error)

static PyObject * tp_repr(P_Viper_TypeTuple * self) {
    P_TRY {
        auto const v {*self->v};
        return P_ViperWrapper::wrapString(v->representation());
    } P_CATCH_P
}

ex: return a integer (-1 => error)

static int tp_set_vector_size(P_Viper_Fuzzer * self, PyObject * pb_value, void *) {
    P_TRY {
        auto const v {*self->v};
        v->vectorSize = P_ViperWrapper::checkUInt64(pb_value);
        return 0;
    } P_CATCH_I
}

The Pattern for Python 'new reference' Convention¶

To automatically handle new reference returned from the C/Python API we use the wrapper P_ViperSRef.

The Python documentation for PySequence_GetItem.

PyObject *PySequence_GetItem(PyObject *o, Py_ssize_t i)
Return value: New reference. Part of the Stable ABI.

We use a P_ViperSRef value to handle the returned reference of PySequence_GetItem in a C++ block.

// Extract types
std::vector<std::shared_ptr<Type>> types;
auto const size {PySequence_Length(pb_types)};
for (Py_ssize_t index {}; index < size; ++index) {
    P_ViperSRef ps_item {PySequence_GetItem(pb_types, index)}; // use of P_ViperSRef in a C++ block.
    auto const itemType {P_ViperWrapper::unwrapTypeOrPyErr(ps_item.get())};
    if (!itemType)
        return nullptr;
    types.push_back(itemType);
}

The Pattern for Methods¶

Method Naming Convention tp_m_set.¶

• tp for type.
• m for method.

Variable Naming Convention¶

Variables used to interop with the C/Python API are named p<reference_type>_<name>.

• pb_<name> for a borrowed reference.
• pn_<name> for a new reference.
• ps_<name> for a new Reference managed by a C++ scope through P_ViperSRef (loop and exception).

The variables used to interop with C++/Viper API use lowerPascalCase convention.

Anatomy of a Wrapped Method¶

1) Collect the arguments with PyArg_ParseTupleAndKeywords with "O!" for registered wrapped type and "O" for seamless value. 2) Unwrap a Viper object from the Python object. 3) Catch C++ exception. 4) Call P_ViperWrapper::checkValue to potentially convert a Python Object to the corresponding Viper Object. 5) Unwrap C++ this from Python self. 6) Call the method. 7) return a wrapped value or Py_RETURN_NONE; 8) Or convert the C++ exception to the 'Return Error Convention' of Python.

static PyObject * tp_m_set(P_Viper_CommitMutating * self, PyObject * args, PyObject * kwargs) {
    // Naming convention for C variables
    PyObject * pb_attachment {}, * pb_key {}, * pb_value {};

    // 1) Parse Python Arguments with the help of 'O!' and *_Type and 'O'.
    static char const * const kws[] = {"attachment", "key", "value", nullptr};
    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O!O!O", (char **) kws,
                                     P_Viper_Attachment_Type, &pb_attachment,
                                     P_Viper_Key_Type, &pb_key,
                                     &pb_value))
        return nullptr;

    // 2) Unwrap viper object (naming convention for C++ variable).
    auto const attachment {*reinterpret_cast<P_Viper_Attachment *>(pb_attachment)->v};
    auto const key {*reinterpret_cast<P_Viper_Key *>(pb_key)->v};

    P_TRY { // 3) Catch exception
        // 4) Create a Viper Value from a python object by consuming the type of required value.
        auto const value {P_ViperWrapper::checkValue(pb_value, attachment->documentType)};

        // 5) Extract C++ this from C/Python self.
        auto const v {*self->v};

        // 6) call the method 
        v->set(attachment, key, value);

        // 7) Return something
        Py_RETURN_NONE;
    } P_CATCH_P // 8) convert the C++ exception to 'Return Error Convention'.
}

The Seamless Wrapper¶

The seamless wrapper, convert Python Object to Viper::Value and Viper::Value to Python Object.

The conversion from a Viper::Value to a Python object is only done when a Python statement constructs a new Python object from others Python objects like c = a + b.

The namespace P_Viper::Wrapper defines utility functions to handle the conversion when it's necessary. The conversion from Python Object to Viper::Value is driven by the Viper's type system.

Functions defined in the namespace P_ViperDecoder try to construct a new Viper::Value by decoding the mapping from Python Object to Viper::Value.

• int --> Viper::ValueUInt8, Viper::ValueUInt16 ...
• real --> Viper::ValueFloat, Viper::ValueDouble ...
• bytes --> Viper::ValueBlob
• dict --> Viper::ValueMap
• list --> Viper::ValueVector
• set --> Viper::ValueSet
• ...

The Function P_Wrapper::wrapValueOrEncode creates Python object bool, int, float, str only for Viper::ValueBool, Viper::ValueUInt[8|16|32|64], Viper::ValueInt[8|16|32|64], Viper::ValueFloat, Viper::ValueDouble and Viper::ValueString other Viper::Values are just wrapped in their corresponding Python object P_Viper_Value_... by embedding the std::shared_ptr.

• Viper::ValueVector --> struct P_Viper_ValueVector
• Viper::ValueSet --> struct P_Viper_ValueSet
• Viper::ValueMap --> struct P_Viper_ValueVector
• Viper::ValueBlob --> struct P_Viper_ValueBlob
• ...