About Legion Runtime Class

These notes are closely based on the set of Legion Runtime Class videos produced by the Legion developers. They are my own notes and code walks, and any errors or things that are just plain wrong represent my own mistakes.

Today's notes are based on the following video:

Overview

An InlineLauncher is useful when you want to get a physical instance of a logical region right now, like if you didn't think ahead. That might be the case for some programs, but it is also really useful when just debugging and playing around. Let's see how it works.

Say we have created a logical region and we want to access it. Let's start by describing some requirements, and then creates the launcher object parameterized on those requirements:

  RegionRequirement req(lr, READ_WRITE, EXCLUSIVE, lr);
  req.add_field(FID_X);

  InlineLauncher launcher(req);

When an InlineLauncher is created there isn't much that goes on. In fact, it is really just a chunk of state that can be passed around that records what we are going to be mapping:

    struct InlineLauncher {
    public:
      InlineLauncher(void);
      InlineLauncher(const RegionRequirement &req,
                     MapperID id = 0,
                     MappingTagID tag = 0);
    public:
      inline void add_field(FieldID fid, bool inst = true);
    public:
      RegionRequirement               requirement;
      MapperID                        map_id;
      MappingTagID                    tag;
    };

Both constructors fill in the public instance variables, and add_field simply adds the field to the requirements instance variable. Things get a bit more interesting when we ask the runtime to map the region described by the launcher and the associated requirements:

PhysicalRegion region = runtime->map_region(ctx, launcher);

Once we arrive inside Runtime::map_region the first thing we do is grab a fresh MapOp, which is a specialization of the Operation type used to perform inline mappings. As an aside, a dedicated cache of operation instances are maintained by the runtime and used to avoid frequent allocations. There are access methods for each cache holding a particular type of operation. The following line grabs a free MapOp from the MapOp pool:

MapOp *map_op = get_available_map_op();

Next thing we do when mapping a region is to initialize the MapOp operation. We'll dig into this just a bit right now. MapOp::initialize is called inside map_region:

PhysicalRegion result = map_op->initialize(ctx, launcher, false);

There are many things that happen during initialization, but some of the important concepts are that the requirements stashed away in the launcher object are copied into the operation. A termination event is created which is triggered when the physical instance is released in the future, and importantly, an instance of PhysicalRegion is created which represents the result of operation after it has gone through the execution pipeline.

Like most stuff we've been seeing, just creating objects doesn't do anything right away. This is part of the deferred execution model used in Legion. That is, when the PhysicalRegion object is created during MapOp::initialize it is only a handle to the final result of the operation which is handed back up to the application to interact with:

region = PhysicalRegion(legion_new<PhysicalRegion::Impl>(requirement,
   completion_event, true, ctx, map_id, tag, false, runtime));

Now let's jump back to our calling context, Runtime::map_region, where we have just finished initializing a MapOp object. Next thing to do is to check if there are any conflicts, or other problems. If everything checks out, we add this operation to the dependence queue toss the physical region handle back to the application:

add_to_dependence_queue(ctx->get_executing_processor(), map_op);
return region;

At this point the operation is now in the pipeline. The MapOp is relatively simple, and for this discussion we can focus on the two Operation methods that it overrides trigger_dependence_analysis and trigger_execution. Logical dependence analysis is for a future class, but generally we want to find out at this stage what other operations the MapOp depends on based on its requirements. This is all encoded in data structures pointed to by runtime->forest which has local information about the region trees.

begin_dependence_analysis();

runtime->forest->perform_dependence_analysis(parent_ctx->get_context(),
    this, 0/*idx*/, requirement, privilege_path);

end_dependence_analysis();

When the MapOp marks the dependence analysis stage complete mapping is usually the next thing that happens. As far as the MapOp goes the default implementation for trigger_mapping is fine, which just adds the operation to a local ready queue for execution. From here we wait until it is time to execute the operation.

The execution stage is specialized for the MapOp by MapOp::trigger_execution. This is the place where we perform physical dependence analysis (which are things like scheduling data copies, etc...). There are separate cases in trigger_execution for when a region is being re-mapped, but for the case where it is the first time being mapped the mapper is invoked via invoke_mapper_map_inline to provide its opinion about things, and then runtime->forest->map_physical_region tries to make things a reality. At any time there might be a failure to map, and false can be returned. Finally, given the mapped region we register ourselves as a user and get an instance via runtime->forest->register_physical_region.

Mapping complete!

There are quite a few different operations in Legion. Most of them are defined in legion_ops.[h,cc].

• PredicateOp: Tracks boolean values
• SpeculativeOp: Provides predicates as precondition for an operation
• FenceOp: Ensure previously issued operations finish mapping
• FrameOp: Scope outstanding operations
• DeletionOp: Triggered when various types of things are removed

Different types of operations are built up using a type hierarchy. For instance, the CopyOp inherits from SpeculativeOp and is used to copy data from one region to another. Importantly, the TaskOp is a SpeculativeOp representing a task that should run when a some conditions have been met. Take a look, because there are even different types of TaskOp's.