understanding Google auths

tagged auth, how-to, software, python

tl;dr: I explain several ways to create and use Google API Console auth

read time: 10m (skim), closer to an hour to implement and test

cost: free


The Google API Console is home to a baffling array of 305 APIs (at time of writing), including Google Docs, Sheets, Drive, Calendar, Maps, Ads, Analytics, Youtube, and GCP offerings.

The setup and management of authentication is equally baffling. Below I will walk through a series of different ways to manage serverside authentication.

I don't touch on clientside Google OAuth, as it is well covered elsewhere

In the below code, I provide made up secret keys that reflect the approximate length of each value. As always, avoid committing your actual secret keys to plain text.

table of contents:


Here's a quick overview of navigating through the Google API Console

For each new API you want to start using, I recommend creating a new Project under the existing Organization & GSuite user.

The most useful overview of your active projects is here.

Finally, there's a quite-helpful API playground here that quickly generates temporary tokens with any permissions you select


initial setup - Service Worker

There are two overarching auth structures used for server-to-server auth: Service Workers and OAuth2 Clients. Service Workers are faster to get started with, so let's walk through their setup first.

The following explains setting up a new Project with associated Service Worker + JSON credentials:

#1.1 - create a new project

#1.2 - create a new service account

#1.3 - name and describe the service account. you can pick the prefix of the service account ID, which will be followed by a random string

It will generate a Service Account ID for you. This email is what you share Google Sheets / Google Analytics Views / etc with in addition to the normal auth in order to access programmatically

#1.4 - ignore IAM permissions

#1.5 - finish and download credentials

Double check the email you want to administer this service account is the same one you're logged in as. After you click CREATE KEY, it will generate a JSON string.

#1.6 - but wait, you're not done yet

You still need to enable any specific APIs that you intend to you for that specific API Console project.

let's quickly go to the Library and enable the Google Sheets API, for rexample

#1.6.1 - search for the relevant API and click Enable

At this step, private_key based auths are possible (JWT and some SDK auths like GSpread). I generally recommend subsequently creating a refresh_token (detailed below)


initial setup - OAuth2 Client

I recommend you go through the following onetime steps in Postman + a browser:

#2.1 - Create a OAuth 2.0 Client ID

#2.2 - Create a OAuth Consent Screen + Scope

The Redirect URL should probably be a domain you control. Make sure to select Internal Application type.

#2.3 - start Oauth flow

Once you've created the OAuth Client, you now have an OAuth flow that you will go through exactly once to generate a refresh token. Take the URL below, change the client_id, redirect_uri, and specific-API-scopes scope to your specific versions, and paste it into the browser. It should start a typical Google OAuth.

https://accounts.google.com/o/oauth2/auth?response_type=code&prompt=consent&access_type=offline
&client_id=n6niRQ3208BX-tT2lGoH0sd9jqAf4jkr2389dwk4EeVg0Qr0.apps.googleusercontent.com
&redirect_uri=https://alec.fyi
&scope=https://www.googleapis.com/auth/analytics.readonly
&state=RANDOM-STRING-THE-OAUTH-SERVER-RETURNS

#2.4 - after authenticating, get the code from the URL it has redirected you to

https://www.alec.fyi/?state=RANDOM-STRING-THE-OAUTH-SERVER-RETURNS&scope=https://www.googleapis.com/auth/analytics.readonly
&code=4/4QHo6v4hZmcuilP9GWWqvimnClacQJeZjkFzLzHsrK4LzobInzMzCUZFFNXoITlcs1-otk5dNTEkr1q4rzf9IIY

#2.5 - usee that OAuth code to get the Refresh Token and an Access Token

Change the code, client_id, client_secret, and redirect_uri to your respective values.

(the new two calls you can do in Postman if you'd rather)

https://accounts.google.com/o/oauth2/token?grant_type=authorization_code
&code=4/4QHo6v5hZmcuilP9GWWqvimzClacQJeZjkFzLzHsrK4LzobInzMzCUZFFNXoITlcs1-otk5dNTEkr1q4rzfUIIY
&client_id=n6niRQ3208BX-tT2lGoH0sd9jqAf4jkr2389dwk4EeVg0Qr0.apps.googleusercontent.com
&client_secret=KwUoOAeoxIUAxDvQL_Vuf3A
&redirect_uri=https://alec.fyi

#2.6 - you can now use the Refresh Token (permanent) to get new Access Tokens. The Acccess Tokens last for 1 hour before expiring.

Change the client_id, client_secret, and refresh_token to your respective values.

https://accounts.google.com/o/oauth2/token?grant_type=refresh_token
&client_secret=KwUoOAeoxIUAxDvQL_Vuf3A
&client_id=n6niRQ3208BX-tT2lGoH0sd9jqAf4jkr2389dwk4EeVg0Qr0.apps.googleusercontent.com
&refresh_token=1//nFtvooivBOKHBhG3XJpJbitJxldsiufdmrAu1wci9D32980pIlEcvYpfA7CjZV7EZF9rSWKg5s6r9aadHHeGefqp07X7eX


refresh token auth

#3 - Once you've gone through the onetime setup steps, you can generate a new short-lived (1 hour) access_token at any time with the long-lived refresh_token.

def service_account_exchange_refresh_token_for_access_token(refresh_token_json):
    refresh_token_json = json.loads(refresh_token_json)

    url = "https://accounts.google.com/o/oauth2/token?grant_type=refresh_token"
    url += "&client_secret=" + refresh_token_json["GA_CLIENT_SECRET"]
    url += "&client_id=" + refresh_token_json["GA_CLIENT_ID"]
    url += "&refresh_token=" + refresh_token_json["GA_REFRESH_TOKEN"]

    resp = requests.post(url)
    return resp.json().get("access_token")

#4 - You can then use the refresh_token to get a short-lived access_token (1 hour before it expires), and that use access_token as the Bearer Token to make API calls.

For example, calling the Google Analytics Core Reporting API:

curl --location --request GET "https://www.googleapis.com/analytics/v3/data/ga?ids=ga:${GA_VIEW_ID}&start-date=2020-09-01&end-date=2020-10-01&metrics=ga:sessions,ga:bounces" \
--header "Authorization: Bearer ${GA_ACCESS_TOKEN}"


JWT auth

JWT refers to JSON Web Token, wherein the service account email and a bunch of metadata are encoded and signed with the associated private key

JWT auths allow you to avoid the onetime step of creating a refresh token, but are otherwise generally more cumbersome

Here's a partial implementation, showing the generation of the JWT token in Python:

iat = time.time()
exp = iat + 3600
payload = {
    "iss": sa_email, # "123456-compute@developer.gserviceaccount.com"
    "sub": sa_email, # "123456-compute@developer.gserviceaccount.com"
    "iat": iat,
    "exp": exp,
    "scope": "https://www.googleapis.com/auth/analytics",
    "aud": "https://oauth2.googleapis.com/token",
}

additional_headers = {"kid": opened_private_key["private_key_id"]}
signed_jwt = jwt.encode(
    payload,
    opened_private_key["private_key"],
    headers=additional_headers,
    algorithm='RS256'
)

Don't roll your own helpers; it's a huge hassle. You can use the Python/JS implementations of these JWT auth helpers here

Aside: JWT is pronounced jot for reasons


SDK auth

Various Google services have their own libraries that implement their own auths.

This can create the confusing situation where various services correspond to service workers in various stages of setup.

For example, GSpread (Google Sheets) can be run with this setup (no refresh token):

import gspread
from google.oauth2 import service_account

def auth_gspread():
    auth = {
        "private_key": os.environ["GSHEETS_PRIVATE_KEY"].replace("\\n", "\n").replace('"', ''),
        "client_email": os.environ["GSHEETS_CLIENT_EMAIL"],
        "token_uri": "https://oauth2.googleapis.com/token",
    }
    scopes = ["https://www.googleapis.com/auth/spreadsheets", "https://www.googleapis.com/auth/drive"] # /drive is for open by name
    credentials = service_account.Credentials.from_service_account_info(auth, scopes=scopes)
    return gspread.authorize(credentials)

For Google Adwords (the old one, not the newer Google Ads), you need a refresh_token and Adwords developer_token exported as a JSON string in environment variables:

export GOOGLEADS_SECRET=‘{
    "developer_token":"2390sdzf0y813900sd", # from the Adwords Manager Account
    "client_customer_id":"0966923613",      # The ID of the actually-running-ads Adwords Account
    "client_id":"n6niRQ3208BX-tT2lGoH0sd9jqAf4jkr2389dwk4EeVg0Qr0.apps.googleusercontent.com", # API Console
    "client_secret":"KwUoOAeoxIUAxDvQL_Vuf3A",                                                 # API Console
    "refresh_token":"1//nFtvooivBOKHBhG3XJpJbitJxldsiufdmrAu1wci9D32980pIlEcvYpfA7CjZV7EZF9rSWKg5s6r9aadHHeGefqp07X7eX"
}’

You then use that JSON env var to auth with the adwords Python client:

from googleads import adwords, oauth2, errors

def authenticate_to_adwords():

    adwords_auth_dict = json.loads(os.environ["GOOGLEADS_SECRET"])

    oauth2_client = oauth2.GoogleRefreshTokenClient(
        adwords_auth_dict["client_id"],
        adwords_auth_dict["client_secret"],
        adwords_auth_dict["refresh_token"],
    )

    adwords_client = adwords.AdWordsClient(
        adwords_auth_dict["developer_token"],
        oauth2_client,
        "My Specific Service Description",
        client_customer_id=adwords_auth_dict["client_customer_id"],
    )
    return adwords_client


API keys

Google has a notion of API keys, which are not at all like any other platform's.

Google's API keys are similar to a publishable client key, where they can access public data at high frequencies (that might be rate limited otherwise)

They are not usable in service worker use cases and are all around poorly documented.


Gotchas

#1. @gmail accounts cannot create OAuth2.0 Clients (and associated Refresh Tokens)! Try to create all service workers in one account attached to your own domain.

#2. Google may log you into the wrong account when using the API Console. Check to be sure you're in the correct one before creating auths! You can easily create service workers in the wrong account!

#3. Some services (Google Sheets, Google Analytics) require the Service Account email (123456-compute@developer.gserviceaccount.com) to be given access to a given Sheet/GA View for the associated auth to work.

#4. If you create >1 access_token per refresh_token in a given hour, it does not roll (cancel) the older access_token. Each will expire 1 hour after being created.

#5. If you get a 403 with a newly generated token with the following message, you need to enable the API in the API Console GUI

    "message": "Google Analytics API has not been used in project 164810568869 before or it is disabled.",


links to docs

Below are a series of links to various parts of Google's documentation that I found useful. If you made it to this section, good luck and Godspeed.