View source for Passing Dispatcher Invoke Arguments to Worklet Arguments Using Signatures

This document is an implementation plan for the VTK-m infrastructure to take arguments given to a dispatcher Invoke, passing data to the execution environment, and loading-storing pieces of data for the call to the worklet’s functor call. How each Invoke argument is interpreted is based on the control and execution signatures in the worklet.

This mechanism is replicating a similar mechanism in Dax. However, one goal of the VTK-m implementation is to make it straightforward to define new worklet types and new ways to interpret arguments.

==High Level Overview==

The process of moving data from the control environment to the execution environment proceeds in three steps.

# A check object checks the type of incoming arguments to make sure they are valid for the associated tag in the control signature. There might also be the ability to do run-time checking (for example, making sure input arrays are the right length), although that is not specified here.
# A transport takes a control object, transports it to the execution environment, and returns an execution object representing the same data (such as an array portal). There is a one-to-one match of dispatcher invoke (control signature) arguments to transports. The type of transport is determined by the tags in the control signature.
# A fetch object uses the execution objects to get and set an argument for a worklet invocation. There is a one-to-one match of fetch objects to execution signature components (including return if non-void). The type of the fetch object is determined by the tag in the execution signature and the associated parameter in the control signature tags. (All execution signature tags will have an associated control signature parameter.)

==Type Check Objects==

The type checking of dispatcher Invoke arguments is somewhat optional and will probably be implemented last, but can be very helpful to users who are only human and make mistakes.

Type checking is done through specializations of the class vtkm::cont::arg::TypeCheck. The default implementation of this class, which belongs in the file vtkm/cont/arg/TypeCheck.h, can be implemented as follows.

<source lang="cpp">
template<typename TypeCheckTag, typename Type>
struct TypeCheck {
  static const value = false;
};
</source>

The first argument is a tag specifying the concept for the argument’s type. These tags by convention are also located in the vtkm::cont::arg namespace and begin with TypeCheckTag. Tags like vtkm::cont::arg::TypeCheckTagArray and vtkm::cont::arg::TypeCheckTagExecObject are certainly going to be defined. The second argument is the type of the dispatcher Invoke argument to be compared.

Type checks can be defined by performing a full specialization of the TypeCheck template. However, most type checks will likely partially specialize TypeCheck and use boost template meta-programming features to determine the type. For example, the implementation for the execution object type check might look like this.

<source lang="cpp">
template<typename Type>
struct TypeCheck<vtkm::cont::arg::TypeCheckTagExecObject, Type> {
  static const value =
      boost::mpl::is_baseof<vtkm::exec::ExecObject, Type>::value;
};
</source>

==Transport Objects==

Transport objects are specializations of the class vtkm::cont::arg::Transport. The prototype of this class belongs in the file vtkm/cont/arg/Transport.h. It can be declared as follows.

<source lang="cpp">
template<typename TransportTag, typename Type, typename DeviceAdapterTag>
struct Transport;
</source>

The first argument is a tag specifying the concept for the transport. These tags by convention are also located in the vtkm::cont::arg namespace and begin with TransportTag. Tags like vtkm::cont::arg::TransportTagArrayIn and vtkm::cont::arg::TransportTagExecObject are certainly going to be defined. The second argument is the control-side dispatcher Invoke argument to be passed to the execution environment. The third argument is the tag for the device adapter to send the data to.

The Transport class is expected to define a type named ExecObjectType that defines the type created in the execution environment. It also defines a const parenthesis operator that takes type type as a first argument and the number of threads in the schedule as a second argument and returns the execution argument.

Here are a few simple but viable examples. The first example is for an execution object argument, which just passes the argument on to the execution environment.

:<font color="forestgreen">Is it worthwhile to have the transport for ExecObject call some method on the control side object to get the execution object? Might be convenient for advanced users but confusing to newbies. --[[User:Kmorel|Kmorel]] ([[User talk:Kmorel|talk]]) 16:48, 11 February 2015 (EST)</font>

<source lang="cpp">
template<typename Type, typename Device>
struct Transport<vtkm::cont::arg::TransportTagExecObject, Type, Device> {
  typedef Type ExecObjectType;

  VTKM_CONT_EXPORT
  ExecObjectType operator(Type object, vtkm::Id) const {
    return object;
  }
};
</source>

This next example is for an input array (typically used for a field).

<source lang="cpp">
template<typename Type, typename Device>
struct Transport<vtkm::cont::arg::TransportTagArrayIn, Type, Device> {
  typedef typename Type::template ExecutionTypes<Device>::PortalConst
      ExecObjectType;

  VTKM_CONT_EXPORT
  ExecObjectType operator(Type arrayHandle, vtkm::Id) const {
    return arrayHandle.PrepareForInput(Device());
  }
};
</source>

This final example is for an output array, which is usually an output field of the same domain as the execution (to assure a one-to-one write).

<source lang="cpp">
template<typename Type, typename Device>
struct Transport<vtkm::cont::arg::TransportTagArrayOut, Type, Device> {
  typedef typename Type::template ExecutionTypes<Device>::Portal
      ExecObjectType;

  VTKM_CONT_EXPORT
  ExecObjectType operator(Type arrayHandle, vtkm::Id size) const {
    return arrayHandle.PrepareForOutput(size, Device());
  }
};
</source>

== Fetch Objects==

Fetch objects are specializations of the class vtkm::exec::arg::Fetch. The prototype of this class belongs in the file vtkm/exec/arg/Fetch.h. It can be declared as follows.

<source lang="cpp">
template<typename FetchTag,
         typename AspectTag,
         typename ExecObjectInterface,
         typename Invocation,
         int ParameterIndex>
struct Fetch;
</source>

The first template argument is a tag specifying the concept for the fetch. These tags by convention are also located in the vtkm::exec::arg namespace and begin with FetchTag. Tags like vtkm::exec::arg::FetchTagPortalDirectIn and vtkm::exec::arg::FetchTagExecObject are certainly going to be defined. The second template argument is an aspect tag that defines what aspect of the data is to be loaded or stored. They are located in the vtkm::exec::arg namespace and begin with AspectTag. The common vtkm::exec::arg::AspectTagDefault is associated with execution signature parameters of types like _1, _2, etc. Other aspects can be specified in the execution signature. For example, vtkm::exec::arg::AspectTagAsVertices can get vertex indices from a cell. The third template argument is a FunctionInterface type with all the execution objects that are associated with the data passed to the dispatcher’s invoke on the control side (and generated with the transports). The fourth template argument is an Invocation object that encapsulates information from the call like the execution objects created in the transport and information from the signatures.  The structure of the Invocation object is described later. The fifth template argument specifies the parameter index to pull data from/push data to.

Due to dependence among these template arguments it should always be the case (and can be assumed) that the FetchTag template argument is the same type as

<source lang="cpp">
typename Invocation::FetchTags::template ParameterType<ParameterIndex>::type
</source>

And likewise the AspectTag template argument is the same type as

<source lang="cpp">
typename Invocation::AspectTags::template ParameterType<ParameterIndex>::type
</source>

The fetch primarily gets its data from an execution object associated with the ParameterIndex. Thus, it should be customary for a fetch to create a typedef similar to as follows.

<source lang="cpp">
typedef
    typename Invocation::Parameters::template ParameterType<ParameterIndex>
    ::type ExecObjectType;
</source>

Some fetch objects might need to coordinate between multiple execution objects (for example, getting point data on a cell needs to get the cell vertex indices from another object as well). This is the reason for providing all execution objects rather than the associated one.

The fetch object is expected to define a type named ValueType that is the type returned from a load, passed to an invocation of a worklet, and passed back to a store.

The fetch object is also expected to define a Load method that takes two arguments. The first argument is an index of the value to be loaded (typically the index of the thread). The second argument is a reference of ExecObjectInterface containing all the execution objects for the thread. Note that it is a very good idea to use pass by reference for this argument to prevent unnecessary data copies. Load can essentially be a no-op for output data.

Finally, a latch object is expected to define a Store method that takes three arguments. The first two arguments are the same as the Load. The third argument is a value of type ValueType that should be stored. The Store method may be a no-op for input data.

Fetch objects are defined by partially specializing the vtkm::exec::internal::Fetch object with a particular FetchTag and a particular AspectTag. The following are some example latch implementations.

The first example is the fetch for an execution object that is passed directly from control to execution environment.

<source lang="cpp">
template<typename ExecObjectInterface,
         typename Invocation,
         int ParameterIndex>
struct Fetch<vtkm::exec::arg::FetchTagExecObject,
             vtkm::exec::arg::AspectTagDefault,
             ExecObjectInterface,
             Invocation,
             ParameterIndex>
{
  typedef
    typename ExecObjectInterface::template ParameterType<ParameterIndex>::type
    ExecObjectType;

  typedef ExecObjectType ValueType;

  VTKM_EXEC_EXPORT
  ValueType Load(vtkm::Id vtkmNotUsed(index),
                 ExecObjectInterface &execObjects) {
    return execObjects.template GetParameter<ParameterIndex>();
  }

  // A no-op because this argument is input only.
  VTKM_EXEC_EXPORT
  void Store(vktm::Id vtkmNotUsed(index),
             ExecObjectInterface &vtkmNotUsed(execObjects),
             const ValueType &vtkmNotUsed(value)) {  }
};
</source>

The next example is a simple fetch for loading an input value from an array corresponding to the given index. This is the type of fetch used in a map field worklet.

<source lang="cpp">
template<typename ExecObjectInterface,
         typename Invocation,
         int ParameterIndex>
struct Fetch<vtkm::exec::arg::FetchTagArrayDirectIn,
             vtkm::exec::arg::AspectTagDefault,
             ExecObjectInterface,
             Invocation,
             ParameterIndex>
{
  typedef
    typename ExecObjectInterface::template ParameterType<ParameterIndex>::type
    ExecObjectType;

  typedef typename ExecObjectType::ValueType ValueType;

  VTKM_EXEC_EXPORT
  ValueType Load(vtkm::Id index,
                 ExecObjectInterface &execObjects) {
    return execObjects.template GetParameter<ParameterIndex>().Get(index);
  }

  // A no-op because this argument is input only.
  VTKM_EXEC_EXPORT
  void Store(vktm::Id vtkmNotUsed(index),
             ExecObjectInterface &vtkmNotUsed(execObjects),
             const ValueType &vtkmNotUsed(value)) {  }
};
</source>

And now here is the related example of a fetch for the same type of array access except for output instead of input.

<source lang="cpp">
template<typename ExecObjectInterface,
         typename Invocation,
         int ParameterIndex>
struct Fetch<vtkm::exec::arg::FetchTagArrayDirectOut,
             vtkm::exec::arg::AspectTagDefault,
             ExecObjectInterface,
             Invocation,
             ParameterIndex>
{
  typedef
    typename ExecObjectInterface::template ParameterType<ParameterIndex>::type
    ExecObjectType;

  typedef typename ExecObjectType::ValueType ValueType;

  // Essentially no-op because this argument is output only.
  VTKM_EXEC_EXPORT
  ValueType Load(vtkm::Id vtkmNotUsed(index),
                 ExecObjectInterface &vtkmNotUsed(execObjects)) {
    return ValueType();
  }

  // A no-op because this argument is input only.
  VTKM_EXEC_EXPORT
  void Store(vktm::Id index,
             ExecObjectInterface &execObjects,
             const ValueType &value) {
    execObjects.template GetParameter<ParameterIndex>().Set(index, value);
  }
};
</source>

The next example provides the fetches for a cell in a topology. There are actually two fetches for cells because there are two valid types of aspects. It should be noted that the execution object associated with cell topology might want to cache indices as they may be requested by multiple fetches. The example implementation assumes objects consistent with Dax for a frame of reference. Topology classes are not implemented yet in VTK-m and might look quite different. It is also probable that several of these methods will also accept IJK indices to speed some computations.

<source lang="cpp">
template<typename ExecObjectInterface,
         typename Invocation,
         int ParameterIndex>
struct Fetch<vtkm::exec::arg::FetchTagCellIn,
             vtkm::exec::arg::AspectTagDefault,
             ExecObjectInterface,
             Invocation,
             ParameterIndex>
{
  // Assuming like the Topology* classes in Dax
  typedef
    typename ExecObjectInterface::template ParameterType<ParameterIndex>::type
    ExecObjectType;

  typedef typename ExecObjectType::CellTag ValueType;

  // Boring because ValueType is really just a tag.
  VTKM_EXEC_EXPORT
  ValueType Load(vtkm::Id vtkmNotUsed(index),
                 ExecObjectInterface &vtkmNotUsed(execObjects)) {
    return ValueType();
  }

  // A no-op because this argument is input only.
  VTKM_EXEC_EXPORT
  void Store(vktm::Id vtkmNotUsed(index),
             ExecObjectInterface &vtkmNotUsed(execObjects),
             const ValueType &vtkmNotUsed(value)) {  }
};

template<typename ExecObjectInterface,
         typename Invocation,
         int ParameterIndex>
struct Fetch<vtkm::exec::arg::FetchTagCellIn,
             vtkm::exec::arg::AspectTagAsVertices,
             ExecObjectInterface,
             Invocation,
             ParameterIndex>
{
  // Assuming like the Topology* classes in Dax
  typedef
    typename ExecObjectInterface::template ParameterType<ParameterIndex>::type
    ExecObjectType;

  typedef typename ExecObjectType::CellTag CellTag;

  typedef const vtkm::exec::CellVertices<CellTag> &ValueType;

  VTKM_EXEC_EXPORT
  ValueType Load(vtkm::Id index,
                 ExecObjectInterface &execObjects) {
    return execObjects.template GetParameter<ParameterIndex>().
        GetCellConnections(index);
  }

  // A no-op because this argument is input only.
  VTKM_EXEC_EXPORT
  void Store(vktm::Id vtkmNotUsed(index),
             ExecObjectInterface &vtkmNotUsed(execObjects),
             const ValueType &vtkmNotUsed(value)) {  }
};
</source>

As our final example we have a fetch that gets the field values for a point on a cell (point to cell). This operation requires the coordination of both the topology information and the array containing the field data.

<source lang="cpp">
template<typename ExecObjectInterface,
         typename Invocation,
         int FieldParameterIndex>
struct Fetch<vtkm::exec::arg::FetchTagFieldPointToCellIn,
             vtkm::exec::arg::AspectTagDefault,
             ExecObjectInterface,
             Invocation,
             ParameterIndex>
{
  typedef ExecObjectInterface::
    template ParameterType<FieldParameterIndex>::type FieldExecObjectType;

  // Get the parameter index for the cell data from the domain of the
  // worklet. How this is defined is described later in this document.
  static int CellParameterIndex = Invocation::INPUT_DOMAIN_INDEX;

  // Assuming like the Topology* classes in Dax
  typedef
    typename ExecObjectInterface::
    template ParameterType<CellParameterIndex>::type CellExecObjectType;

  typedef typename CellExecObjectType::CellTag CellTag;

  typedef 
    vtkm::exec::CellField<typename FieldExecObjectType::ValueType,CellTag>
    ValueType;

  VTKM_EXEC_EXPORT
  ValueType Load(vtkm::Id index,
                 ExecObjectInterface &execObjects) {
    const CellExecObjectType &cellExecObject =
        execObjects.template GetParameter<CellParameterIndex>();
    const vtkm::exec::CellVertices<CellTag> &vertexIndices =
        cellExecObject.GetCellConnections(index);

    const PointExecObjectType &pointExecObject =
        execObjects.template GetParameter<PointParameterIndex>();
    ValueType result;
    for (int vertexIndex = 0;
         vertexIndex < ValueType::NUM_VERTICES;
         vertexIndex++)
    {
      result[vertexIndex] = pointExecObject.Get(vertexIndices[vertexIndex]);
    }

    return result;
  }

  // A no-op because this argument is input only.
  VTKM_EXEC_EXPORT
  void Store(vktm::Id vtkmNotUsed(index),
             ExecObjectInterface &vtkmNotUsed(execObjects),
             const ValueType &vtkmNotUsed(value)) {  }
};
</source>

==Worklet Base Classes==

Worklet base classes should be located in the namespace vtkm::worklet. This is a departure from Dax where they are located in dax::exec. The rational for this move is firstly that worklets straddle the two environments. They are constructed in the control environment and run in the execution environment. Secondly they specify (through signatures) data movement in both environments and therefore the signature tags are located in both control and execution environments. Since the package dependency should be that the execution package does not at all depend on the control package, worklets have to be outside if the execution package. And since it makes no sense to put them in the control package, they should go in their own package. (This may or may not mean that dispatchers also have to move to the worklet environment, which may or may not be a bad thing.)

The ControlSignature and ExecutionSignature typedefs use tags to specify the concepts of the arguments. There is a separate set of tags used for the ControlSignature and the ExecutionSignature.

We call these signature objects tags because they are used in that way, but they are not strictly the empty structs you would expect for a tag. Instead, they are structs that contain subtags that specify actual behavior. The rational is to simplify the specification of tags, particularly with respect to the templated execution signature tags.

===Control Signature Tags===

Control signature tags are structs that contain 3 subtags specifying the behavior of the movement of data from control environment to worklet invocation in the execution environment. These three tags are a type check for the control-side argument (TypeCheckTag), a transport (TransportTag), and a fetch (FetchTag).

The control signature tags can be defined directly in the superclass for a given worklet type. It is a simple struct with typedefs for the appropriate subtags. An example of a signature tag is as follows.

<source lang="cpp">
class WorkletMapField : public vtkm::worklet::internal::WorkletBase
{
public:
  struct FieldIn {
    typedef vtkm::cont::arg::TypeCheckTagArray TypeCheckTag;
    typedef vtkm::cont::arg::TransportTagArrayIn TransportTag;
    typedef vtkm::exec::arg::FetchTagArrayDirectIn FetchTag;
  };
  ...
</source>

:<font color="forestgreen">The tags are protected in Dax, but might need to be made public in VTK-m since objects outside the worklet will take the tag and look at its internal tags. The compiler might not allow that in protected subclasses. Making them public should be no problem. --[[User:Kmorel|Kmorel]] ([[User talk:Kmorel|talk]]) 16:48, 11 February 2015 (EST)</font>

The names of the control signature tags will not use the fancy parenthesis convention to specify parameters, such as Field(Point, In), as was done for Dax. This was problematic for some compilers and confusing to some users. Instead, all valid combinations of argument conventions will be enumerated in their own structures.

By convention, the tag names will be the base concept (e.g. Field, Topology) followed by the domain (e.g. Point, Cell) followed by In or Out. For example, FieldPointIn and FieldCellOut are good tag names. The domain and In/Out part should be there in most cases but can be left out where they are obvious or ambiguous. For example TopologyIn and ExecObject reasonably leave out parts of the name.

===Execution Signature Tags===

Execution signature tags are structs that contain a single subtag specifying the behavior of the fetch operation.

The concept of the arguments is specified in the control signature. The function of the execution signature tags is to point to the appropriate control signature argument and provide the aspect to use with the fetch. The execution signature tags are structs that contain a static int specifying the control signature parameter they refer to and a subtag for the aspect of the fetch. The “basic” execution signature tag simply loads/stores the default aspect for a parameter.

<source lang="cpp">
namespace vtkm {
namespace exec {
namespace arg {

struct AspactTagDefault {};

template<int N>
struct BasicArg {
  static int INDEX = N;
  typedef vtkm::exec::arg::AspectTagDefault AspectTag;
};
</source>

All arguments should have a default value associated with them (even if that value is really an empty type). Thus, WorkletBase defines a set of arguments named _1, _2, _3, etc. up to at least _9 to define a basic fetch of that argument. They can be defined as follows.

<source lang="cpp">
class WorkletBase : public vtkm::exec::FunctorBase
{
public:
  template<int N>
  struct Arg : vtkm::exec::arg::BasicArg<N> {  };

  struct _1 : public Arg<1> { };
  struct _2 : public Arg<2> { };
  struct _3 : public Arg<3> { };
  ...

  typedef _1 InputDomain;
</source>

The previous definition also has a type named InputDomain. This is a tag that the dispatcher will use to identify which control signature argument specifies the domain of the call. This would be an input array for a WorkletMapField or a topology for a WorkletMapCell. This makes the control argument specifying the domain explicit and prevents us from having to search through all the tags with template magic to implicitly pick one, which became problematic in Dax. By default, the input domain is set to the first argument, but that is easily overridden in worklet classes by redefining InputDomain.

Some argument types have different values that can be retrieved. For example, a cell that comes from a topology might return either its tag or index as its default value, but a worklet might need its vertex indices. This can be specified with a templated tag that takes another argument tag as its template argument. For example, a tag to get the indices for cell vertices could be specified as follows.

<source lang="cpp">
class WorkletMapCell : public vtkm::worklet::internal::WorkletBase
{
public:
  template<typename BaseArg>
  struct AsVertices {
    static int INDEX = BaseArg::INDEX;
    typedef vtkm::exec::arg::AspectTagAsVertices AspectTag;
};
</source>

This tag would be used like AsVertices<_1> in the execution signature.

Unfortunately, there are also tags that do not refer to any specific control signature argument because the data itself is implicit. Examples of this include the worklet invocation index and the visit index. I propose handling these by using a placeholder tag in the execution signature and then having the dispatcher replace them with valid tags after the appropriate data is created and added to the control parameters.

===Full Example===

To better demonstrate how these arguments fit together with the control and execution signatures, here is a (mostly) complete code segment showing the definition of the worklet objects from the worklet base down to a tetrahedralize worklet definition (which is nontrivial). The implementation of many of the objects associated with these tags is an exercise left till development.

<source lang="cpp">
namespace vtkm {
namespace worklet {
namespace internal {

class WorkletBase : public vtkm::exec::FunctorBase
{
public:
  template<int N>
  struct Arg : vtkm::exec::arg::BasicArg<N> {  };

  struct _1 : public Arg<1> { };
  struct _2 : public Arg<2> { };
  struct _3 : public Arg<3> { };
  ...

  // This tag is magically replaced by the dispatcher to point to a real array
  typedef vtkm::exec::arg::WorkIndex WorkIndex;

  // Default input is the first argument.
  typedef _1 InputDomain;
};


}
}
} // namespace vtkm::worklet::internal 

namespace vtkm {
namespace worklet {

class WorkletGenerateTopology : public vtkm::worklet::internal::WorkletBase
{
public:
  struct FieldPointIn {
    typedef vtkm::cont::arg::TypeCheckTagArray TypeCheckTag;
    typedef vtkm::cont::arg::TransportTagArrayIn TransportTag;
    typedef vtkm::exec::arg::FetchTagFieldPointToCellIn FetchTag;
  };

  struct FieldCellIn {
    typedef vtkm::cont::arg::TypeCheckTagArray TypeCheckTag;
    typedef vtkm::cont::arg::TransportTagArrayIn TransportTag;
    typedef vtkm::exec::arg::FetchTagFieldCellPermuted FetchTag;
  };

  // The input domain has to point to this TopologyIn, and this TopologyIn 
  // won’t work if not the input domain. Internally, the dispatcher will
  // augment the control-side argument with a permuation array.
  struct TopologyIn {
    typedef vtkm::cont::arg::TypeCheckTagTopology TypeCheckTag;
    typedef vtkm::cont::arg::TransportTagTopologyPermutedIn TransportTag;
    typedef vtkm::exec::arg::FetchTagCellIn FetchTag;
  };

  // The output domain has to point to this TopologyOut, and this TopologyOut
  // won’t work if not the output domain.
  struct TopologyOut {
    typedef vtkm::cont::arg::TypeCheckTagTopology TypeCheckTag;
    typedef vtkm::cont::arg::TransportTagTopologyOut TransportTag;
    typedef vtkm::exec::arg::FetchTagCellOut FetchTag;
  };

  template<typename BaseArg>
  struct AsVertices {
    static int INDEX = BaseArg::INDEX;
    typedef vtkm::exec::arg::AspectTagAsVertices AspectTag;
  };

  // This tag is magically replaced by the dispatcher to point to a real array
  typedef vtkm::exec::arg::VisitIndex VisitIndex;

  // Default output is the second argument.
  typedef _2 OutputDomain;
};

}
} // namespace vtkm::worklet

namespace vtkm {
namespace worklet {

class Tetrahedralize : public vtkm::worklet::WorkletGenerateTopology
{
public:
  typedef void ControlSignature(TopologyIn, TopologyOut);
  typedef void ExecutionSignature(AsVertices<_1>,
                                  AsVertices<_2>,
                                  WorkIndex,
                                  VisitIndex);

  VTKM_EXEC_EXPORT
  void operator()(
      const vtkm::exec::CellVertices<vtkm::CellTagVoxel> &inVertices,
      vtkm::exec::CellVertices<vtkm::CellTagTetrahedron> &outVertices,
      const vtkm::Id outputCellIndex,
      const vtkm::Id visitIndex) const
  {...}
};

}
} // namespace vtkm::worklet
</source>

==Dispatcher Invoke Sequence==

This section describes the sequence of operations that use the aforementioned facilities to go from a call of the dispatcher’s Invoke method in the control environment to an invocation of the worklet in the execution environment.

The base dispatcher uses a curiously recurring template pattern to call the invoke facilities of its derived classes. It also takes as arguments the worklet type (containing the signatures) so that it can create FunctionInterface types containing the tags for type check and transport of each control argument as well as the fetch tag for each execution argument. It will also contain other arguments like a DeviceAdapterTag for other operations.

===Packing and Dynamic Casting of Arguments===

The Invoke argument is implemented in the base dispatcher class. This method requires a variable number of arguments. These are easy in Cxx11, but harder with compilers using an earlier standard, which are abundant. Thus, this method is a short as possible to minimize code duplication and complication. The method basically packs its arguments in a FunctionInterface and passes control on to a method with a single template parameter.

This method then uses the dynamic casting functionality (already implemented in VTK-m’s FunctionInterface to convert arguments like DynamicArrayHandle into concrete types. These types are then checked using the TypeCheck objects. This is achieved with a ForEach method in FunctionInterface. A check to make sure that the worklet base class is the right type would be good here to. (Doing it in this templated method rather than in the class itself should make it easier to base the functionality of one dispatcher on another since one dispatcher DoInvoke could call another’s with it’s own worklet, bypassing the type check.)

Along with a function interface for the parameters, the dispatcher base also builds an invocation object that encapsulates the transport and fetch tags as well as the parameter index for the input domain (which is specified with the InputDomain typedef in all worklets). The Invocation object, which would be defined in the vtkm::internal package, could look something like this.

<source lang="cpp">
template<typename _TransportTagsFunctionInterface,
         typename _FetchTagsFunctionInterface,
         typename _ExecutionFunctionInterface,
         int _InputDomainIndex>
struct Invocation
{
  typedef _TransportTagsFunctionInterface TransportTags;
  typedef _FetchTagsFunctionInterface FetchTags;
  typedef _ExecutionFunctionInterface ExecutionFunctionInterface;
  static int INPUT_DOMAIN_INDEX = _InputDomainIndex;

  template<typename NewTransportTags>
  struct ChangeTransportTagsType {
    typedef Invocation<NewTransportTags,LatchTags,FetchTags> type;
  };

  template<typename NewTransportTags>
  VTKM_EXEC_CONT_EXPORT
  typename ChangeTransportTagsType<NewTransportTags>::type
  ChangeTransportTags(NewTransportTags) {
    return typename ChangeTransportTagsType<NewTransportTags>::type();
  }

  // And similar for changing fetch tags, execution interface, and
  // input domain index.
};
</source>

===Derived Dispatcher Operations===

The DoInvoke method allows the derived dispatcher to perform custom operations for a particular worklet type. This works pretty much how it worked in Dax. However, the fact that the tags from the signatures are taken from the worklet and packed in the invocation object make it easier to make changes (for example by adding permutations or adding arguments for things like the visit index).

The DoInvoke method will often have to create auxiliary data (such as arrays of indices) and add them to the calling specification. Adding or replacing arrays in the control parameters is straightforward. These arrays might be used to for data corresponding to tags like VisitIndex in the execution signature. The DoInvoke method will have to find those tags and replace them with tags that point to these new arrays. This will probably be the trickiest template programming the DoInvoke method will have to do. We should be able to provide some helper classes for this.

To perform the actual invocation of the worklet in the control environment (along with all necessary memory movement), the DoInvoke method calls a BasicInvoke on the base dispatch, much like is done now.

===Transporting Arguments===

The BasicInvoke method has to take the list of parameters and run the transport on each one to get a list of execution objects. The transport requires both the transport tag and the control argument itself, which are located in separate FunctionInterface objects. Performing this operation using the current FunctionInterface facilities is not possible. However, it would be straightforward to add a zip functionality that combined two FunctionInterface objects into one (where each parameter is now a vtkm::Pair), and then use a static transform on the zipped parameters.

The result would be a FunctionInterface containing the execution objects. This pack of execution objects would be given to a functor for the execution environment that invokes the worklet. In Dax, this object is called Functor, but I find that name way to general. I suggest calling it something like WorkletInvoke.

===Fetching Arguments===

The WorkletInvoke, upon invocation, must fetch each parameter for the worklet and then call the worklet on those parameters. The WorkletInvoke, having been templated in the execution object function interface created when transporting the objects and the Invocation, can create a FunctionInterface containing the parameters for the worklet by doing a static transform on the Invocation’s execution function interface. The static transform will perform the fetch operations for each parameter.

Once this function interface for the parameters is created, it can be invoked on the worklet itself. When the worklet call returns, this function interface can be zipped with the execution function interface and a ForEach can call the store method on the fetch objects.

== Acknowledgements ==

Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energy's National Nuclear Security Administration under contract DE-AC04-94AL85000.

[[Image:SandiaLogo.png]]
[[Image:DOELogo.png]]

SAND2015-0898 O