New Source: Matomo Analytics

[jwest75674]

Tell us about the new connector you’d like to have

• Which source and which destination?
• Do you need a specific version of the underlying data source e.g: you specifically need support for an older version of the API or DB?
  A Matomo Analytics connector would be fantastic for companies and agencies like our own who need the use of a web/Google analytics like tool, but also have legislative or other requirements which inhibits the use of Google analytics.

Our (Canadian) agency leverages Matomo for clients who are restricted from storing data out of country, or clients who require their data accessible on prem.

Describe the context around this new connector

• Which team in your company wants this integration, what for? This helps us understand the use case.

The use case is as diverse as the use case for Google Analytics and it's related connector.

Describe the alternative you are considering or using

What are you considering doing if you don’t have this integration through Airbyte?

We'll be forced to write SQL queries against on-premise hosted installations, and/or working through data exports. Neither super fun, nor friendly to our junior staff.

[sherifnada]

@jwest75674 thanks for the request! which of the endpoints here would you be interested in using?

[jwest75674]

Absolutely, Happy to help!

Matomo 4.X, Reporting API, has a very similar role to play as the Google Analytics API of the same name. From the perspective of Airbyte, it is a data source, and is what I have in mind for my own use case.

Next in line would be one of the three Tracking APIs, likely the HTTP Tracking API.
I consider this to be lower priority, however the tracking API could allow Matomo to be used as a data destination. This would be a benefit to the organizations using Matomo for the reasons mentioned earlier, or otherwise to organizations who use Matomo as a data lake/warehouse.

[cheyura]

Webhook-based? (no/partially/yes)
No

Available authentication modes (API key/Oauth/other)
The token_auth acts as your password and is used to authenticate in API requests.

Security considerations
The token_auth is secret and should be handled very carefully: do not share it with anyone. Each Matomo user has a different token_auth.

Matomo 4 and newer
To generate a token_auth follow these steps:

Log in to Matomo
Go to the Matomo Admin through the top menu
Click on Personal -> Security
In the bottom of the page click on “Create new token”
Confirm your account password
Enter the purpose for this plugin as a description
Click on “Create new token”

Sandbox account?
No

How to populate the account with data?
API, UI

Available streams for sync
Tracking API https://developer.matomo.org/api-reference/tracking-api
Reporting API https://developer.matomo.org/api-reference/reporting-api

Integration supports incremental sync?
No

Other information/blockers?

[cheyura]

@sherifnada This connector is a smth like Google Analytics for websites. The most easy way is to gather analytics frpm Airbyte.io. Java Script tracking code(https://airbyte.matomo.cloud/index.php?module=CoreAdminHome&action=trackingCodeGenerator&idSite=1&period=day&date=2021-07-01&showtitle=1&random=2941) should be inserted into web-pages of Airbyte.io. And then we can call Tracking API on visits, clicks and so on.

[cheyura]

We have access though API to demo site, API token is generated for [email protected] but it is not needed for demo (public) site

[florent-martineau]

Any update on this connector?

[Dynnammo]

@florent-martineau Please add a 👍 on the first post, the more there is the more probable it'll be handle by Airbyte or volunteer contributors 😉 .

[mickaelandrieu]

Hi,

this connector is labeled as "Done" in the roadmap, but I don't find the connector in the list of sources or destinations.

Do we have any ETA ?

Regards

[sherifnada]

cc @YowanR as the person managing connector dev backlog

[YowanR]

@mickaelandrieu Could you point me to where you saw this connector marked as done, please? AFAIK, this connector has not been tackled yet so it's probably a typo from our side :)

[norbertorok92]

Hi, @YowanR do you have a rough estimation on the timeline of when you guys planning to have a connector for matomo?

[mickaelandrieu]

They don't have an ETA as no one is working on it AFAIK :(

[vchallier]

Looking forward to this connector !

[firehist]

Hey lovely team, is there any news regarding the ETA of this fabulous connector.

Thanks a lot!! 🚀

[mattab]

Great to see Matomo is the 3rd most popular new connector request!

Question to all: could you maybe give some example use cases of what you'd like to achieve with the Airbyte Matomo source?

As the founder of Matomo I just wanted to provide a basic information, that we have a few different APIs but maybe these would be the useful ones:

1. Matomo provides a simple way to fetch the reports called the Metadata Reporting API (see api reference). This is relevant when people using Airbyte want to fetch their analytics reports from Matomo. It's basically a 1-2 simple APIs that can be used to get any of Matomo 100+ reports.

2. there is also the Live API lets you request the Matomo RAW data (see faq with details). This is relevant if Airbyte users want to fetch the web analytics RAW data (the detailed list of visits, actions, with all the metadata columns)

3. if Airbyte can connect to Databases including Matomo database, then Airbyte users might be benefit from our Database Schema documentation here: https://developer.matomo.org/guides/database-schema

Happy to help with any question and hope that Matomo can be added as an Airbyte connector 👍

[mickaelandrieu]

Hi @mattab,

I've decided to consider Matomo just like any other PostgreSQL database (so option 3) and rely on the documented queries and relevant tables as a start.

For now and if buying a module is not possible, I'd be against an REST API connector because the available data from it is too limited for my needs.

We should document the option 3 in both Matomo and Airbyte docs: I can give a hand :)

Regards

[alxsbn]

Any news about the connector?

[atharva47dev]

Please provide the YAML file if not the connector.

[mickaelandrieu]

Hi,
I don't use any specific connector, I self host Matomo and ingest the data just like yet another MySQL database.

Soon, I will release a complete tutorial on how to self host Matomo and configure it for performance and scalability.
But I can't share any dbt models as it's not open source stuff :/

[atharva47dev]

Thanks Mickael, It would be super helpful. We also self hosted Matomo but since I am not aware of its schema, I am facing difficulty in terms of understanding how to merge the various tables to make the report.

[mickaelandrieu]

Hi @atharva47dev,

We started from this page into the official documentation if you haven't noticed it, yet ? https://matomo.org/faq/how-to/how-do-i-write-sql-queries-to-select-visitors-list-of-pageviews-searches-events-in-the-matomo-database/

I'll ask my boss if someday we will be allowed to open source some of our models, but tbh we have a little bit customised it to manage our custom dimensions so I'm not sure this will be useful "as it" :/

[mickaelandrieu]

I've published the first part of how to self host your Matomo using Docker : https://medium.com/gitconnected/how-to-install-and-configure-matomo-with-docker-fa410bf3f22c

Then, what you need to do in Airbyte is to declare a MySQL* source with your Matomo database credentials.

If you use Airbyte cloud, you will have to open a port. If you self host an instance of Airbyte on the same server you can "share" the same docker network and rely on the name of the container.

From Airbyte, many solutions are viable : here is my one (I wanted to write a complete tutorial for Airbyte, I've yet to find the time to contact them about it).

Then, you can (with dbt or something else) prepare your own views : the Matomo documentation have some good tips and useful queries.

STILL, we needs an official connector for Matomo : I don't know why this issue have been converted into a discussion.

[rGaillard]

Hello here,

The Matomo source connector is it planned in the Airbyte roadmap?

[mickaelandrieu]

no (Hello you, btw !)

[rGaillard]

hehe hello :)
So bad :( But i just see that they have a new feature to export raw data to bigquery & other data warehouses. I requested access to the beta version, maybe this can solve my issue! :)