BigQuery: Automate limits and custom quotas through the Discovery API client

Leveraging the Google API Discovery service to use undocumented Google APIs

Why a new blog post?

This post is inspired by a blog post from 3 years ago: https://medium.com/google-cloud/bigquery-set-up-limits-and-custom-quotas-through-api-629f77438b7e from Guillaume Blaquiere.

When testing the aforementioned Invoking the URL as:

https://servicemanagement.googleapis.com/v1/services/
bigquery-json.googleapis.com/projectSettings/PROJECT_ID?
updateMask=quotaSettings.consumerOverrides%5B%22QueryUsagePerDay%22%5D

I received a 404 error. The solution as stated in the blog post is not documented and could stop working at any time:

Important note: The following explanation is not documented by Google and can change or be blocked at any time, without warning. Use with caution.

Automate the quota limits

As I was dealing with a use case of dynamically provisioning a large number of BigQuery projects that would hold several data assets such as datasets, tables, and views, automating BigQuery quotas in some way was a must for me, so I started looking at how to do something similar as the previous blog post.

If you are not familiar with Google Cloud Platform Quotas, here's a short summary from their docs:

GCP docs

For BigQuery there are several quota types that can be set through Cloud Console:

GCP docs

It was a nice surprise, that there is a new gcloud alpha command where you can programmatically set those up:

GCS docs

The gcloud command certainly helps, but in the environment, I would be running the automation I didn't have gcloud installed, then I asked myself… is there another way?

Google API Discovery service

After digging inside gcloud alpha servives quotacommand source code, and looking at what it was doing I went over to the APIs explore page, to try to run it in a similar fashion.

If you are not familiar with Google Discovery Service, you may read the text block below, otherwise, you may skip it:

Many users are not aware but sometimes the newest API operations or available features are not immediately available on Google SDKs, but you have something they call discovery api client:

In summary, the Google API Discovery service simplifies the process of working with Google APIs by providing structured and standardized documentation, which under the hood is utilized by their own client libraries:

Basically, it’s a document that tells machines how to interact with their APIs, which sometimes can be useful as documentation. I recommend always using each of Google services SDK’s, and relying on the discovery client if the operation is not available in the SDK or if you want to get more details on what is available for that service with its models.

Then how to use it?

First, you start by installing the google-api-python-client PyPI package.

Next after looking at the discovery JSON that you can get in this link, and finding what is the right service and operation you need to call, you build the service object:

Let's look at how it looks for our BigQuery quota use case.

Put it all together

import googleapiclient.discovery

# [user] change to your GCP project id
MY_PROJECT_ID = ""
quota_value_in_mb = "10485760"

api_service_name = "serviceusage"
api_version = "v1beta1"
discovery_url = f"https://{api_service_name}.googleapis.com/$discovery/rest?version={api_version}"
service = googleapiclient.discovery.build(
    api_service_name, api_version, discoveryServiceUrl=discovery_url
)
created_quota = (
    service.services()
    .consumerQuotaMetrics()
    .importConsumerOverrides(
        parent=f"projects/{MY_PROJECT_ID}/services/bigquery.googleapis.com",
        body={
            "inlineSource": {
                "overrides": [
                    {
                        "overrideValue": quota_value_in_mb,
                        "metric": "bigquery.googleapis.com/quota/query/usage",
                        "unit": "1/d/{project}/{user}",
                    }
                ]
            },
            "force": True,
        },
    )
    .execute()
)

So that turned out to be the translated version of the gcloud alpha command, that specific call uses the Query Usage Per Day Quota and sets a value of 10 terabytes per user, you may tweak that sample to use different quotas or change the value.

Use with caution

Now, you can script this quotas API using Python and the Google Discovery API service.

However, keep in mind that is still an undocumented version, which uses a v1beta1 API service version which may change without notice and is not guaranteed to still work with time.