Thoughts on a personal journey into the modern world of data governance and DataOps.

Photo by Benjamin Davies on Unsplash

In April of 2021, I chose to leave a 10 year job at CI&T and join a nascent startup company as their first hire. …


Learn how to ingest SAP HANA metadata to Google Data Catalog and extend it with your user needs

Google Data Catalog has community-maintained tools like the open source connectors, to ingest metadata from different data sources:


Scripts and Terraform automation to help you ensure best practices in Google Data Catalog

Disclaimer: All opinions expressed are my own, and represent no one but myself…

There’s extensive documentation on what IAM Roles are available for Google Data Catalog. …


Best practices on two approaches, with code samples!

Disclaimer: All opinions expressed are my own, and represent no one but myself…. They come from the experience of participating in the development of fully operational sample connectors, available at: GitHub.

If you want to hear more about some of the Data Catalog connectors use cases, please check the official…


From design decisions to step by step execution learn how to ingest Apache Atlas metadata in Google Data Catalog doing full and incremental runs.

Image created on Canva.

Disclaimer: All opinions expressed are my own, and represent no one but myself…. They come from the experience of participating in the development of fully operational sample connectors, available on: github.

If you missed the latest post talking about how Apache Atlas and Data Catalog structure their metadata, please check…


Learn how your metadata is structured on both systems.

Image created on Canva.

Disclaimer: All opinions expressed are my own, and represent no one but myself…. They come from the experience of participating in the development of fully operational sample connectors, available on: github.

If you missed any of the latest posts on how to ingest metadata into Data Catalog, please check the…


How to Create Data Catalog tags by inspecting all your BigQuery data with Cloud Data Loss Prevention.

Background by Kelli Tungay on Unsplash

Do you know what kind of sensitive data your organization holds? Are you keeping track of every change applied across all your tables and columns? Are you confident to answer questions an auditor may have on data regulations?


Code samples with a practical approach on how to incrementally ingest metadata changes from an on-premise Hive server into Google Cloud Data Catalog

Background by JJ Ying on Unsplash

Disclaimer: All opinions expressed are my own, and represent no one but myself…. They come from the experience of participating in the development of fully operational sample connectors, available at: github.

The Challenge

Entering the big data world is no easy task, the amount of data can quickly get out of hand…


Code samples with a practical approach on how to ingest metadata from an on-premise Hive server into Google Cloud Data Catalog

Background by JJ Ying on Unsplash

Disclaimer: All opinions expressed are my own, and represent no one but myself…. They come from the experience of participating in the development of fully operational sample connectors, available at: github.

The Challenge

We hardly find large organizations storing all their data at the same place, sometimes it’s because of compliance or…


Code samples with a practical approach on how to ingest metadata from on-premise Relational Databases into Google Cloud Data Catalog

Background by JJ Ying on Unsplash

Disclaimer: All opinions expressed are my own, and represent no one but myself…. They come from the experience of participating in the development of fully operational sample connectors, available at: github.

Google Cloud Data Catalog Team has recently announced the product is GA, with the feature to accept custom (aka…

Marcelo Costa

software engineer & google cloud certified architect and data engineer | love to code, working with open source and writing @ alvin.ai

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store