QueueManager and Resource Group in Presto

Presto is a fast distributed SQL engine originally developed by Facebook. From 0.153, pluggable resource group was introduced. This feature enables us to separate cluster resource in account workload. For example, you want to ensure daily batch job resource allocation even if you submit some adhoc queries. In short when 3 concurrent queries are permitted in total, you can ensure to keep some of query allocation even 3 or more queries of batch job are submitted in advance.

If batch job is submitted to batch resource group, it can be guaranteed to run at least 1 query any time.

So how can it be achieved? I found resource group is basically implemented like queue mechanism of Presto. Queue mechanism of Presto is realized by QueryQueueManager.

/**
 * Classes implementing this interface must be thread safe. That is, all the methods listed below
 * may be called concurrently from any thread.
 */
@ThreadSafe
public interface QueryQueueManager
{
    void submit(Statement statement, QueryExecution queryExecution, Executor executor);
}

This interface receives QueryExecution which stores various information needed to run query. Before resource manager, only SqlQueryQueueManager implements this interface. It only controls running queries and queued queries. Actual QueryQueueManager is injected in advance.

public class CoordinatorModule
        extends AbstractConfigurationAwareModule
{
  @Override
  protected void setup(Binder binder)
  {
    if (buildConfigObject(FeaturesConfig.class).isResourceGroupsEnabled()) {
      binder.bind(QueryQueueManager.class).to(InternalResourceGroupManager.class);
    }
    else {
      binder.bind(QueryQueueManager.class).to(SqlQueryQueueManager.class).in(Scopes.SINGLETON);
      binder.bind(new TypeLiteral<List<QueryQueueRule>>() {}).toProvider(QueryQueueRuleFactory.class).in(Scopes.SINGLETON);
    }
  }
}

Therefore from SqlQueryManager side, these two implementations can be used transparently. Actually all SqlQueryManager does is submitting.

// start the query in the background
queueManager.submit(statement, queryExecution, queryExecutor);

So actual handling and resource management can be delegated to each implementations (SqlQueueQueueManager and InternalResourceGroupManager).

I’ll describe the detail of resource management mechanism of InternalResourceGroupManager.

Thanks.