Skip to content

Releases: mage-ai/mage-ai

0.9.63 | Halo 👾

08 Feb 04:06
ba3714d
Compare
Choose a tag to compare

What's Changed

🎉 Exciting New Features

🦆 MotherDuck Support


This one's for all the ducklings out there! In addition to supporting DuckDB, Mage now supports MotherDuck destinations!

By specifying a MOTHERDUCK_TOKEN and adding a md: prefix to your DuckDB database, you can read/write from/to MotherDuck locations! Check it out and get started here.

by @wangxiaoyou1993 in #4533

🤖 Support Thick mode in OracleDB


We like our OracleDB connections like we like our pancakes, THICK 🥞. By default, Mage's Oracle client runs in a "Thin" mode which connects directly to Oracle Database— this mode does not need Oracle Client libraries. However, some additional functionality is available when they're used.

Now, you can use the "Thick" mode in Mage to connect to OracleDB using the Oracle Client libraries!

Check out our docs to get started or read more about the differences between "Thin" and "Thick" modes here.

by @matrixstone in #4421

🗄️ Show unused block files in file browser

This is one of our most requested features and we're excited to announce that it's finally here! 🎉

You can now see which files are not being used in your pipeline! This is a great way to clean up your projects and remove any unnecessary files. Check out the gif below to see it in action!


Head over to our docs to learn more!

by @johnson-mage in #4449

🤐 Import functionality for pipeline zips

Like to share? So do we! You can now import pipelines (via .zip files) in your Mage projects! We're optimistic that this simple improvement will make it easier to share your pipelines or even borrow pipelines from your friends!


by @johnson-mage in #4453

🐛 Bug Fixes

💅 Enhancements & Polish

  • MSSQL data integration source - add support for DATETIMEOFFSET type by @hugabora in #4499
  • Fix: handle unprivileged user for postgres by @jdbranham in #4357
  • Optimize pipeline schedule queries by @dy46 in #4188
  • Remove pipeline's updated_at attribute by @johnson-mage in #4521
  • Create and view workspaces in different namespaces by @dy46 in #4513
  • Select status for pipeline runs that time out by @dy46 in #4519

New Contributors

Full Changelog: 0.9.62...0.9.63

0.9.62 | The Beekeeper 🐝

01 Feb 00:56
a9cacb0
Compare
Choose a tag to compare

What's Changed

🎉 Exciting New Features

🧵 [Mage IO] Weaviate Integration

🤔 Building AI apps with Mage? Perfect! Now you can use Weaviate as a data source. Weaviate is an open-source, AI-native vector database that helps developers create intuitive and reliable AI applications. With Mage, you can now read from and write to Weaviate databases! Read more in our docs here.

by @matrixstone in #4158

🔍 [Mage IO] Alogia Integration

Like search? Us too! That's why we've added support for Alogia as a data source in Mage. Alogia is a powerful search engine that helps you build fast and accurate search experiences. With Mage, you can now read from and write to Alogia! Read more in our docs here.

by @matrixstone in #4198

💥 Dynamic SQL Blocks

Another big improvement to dynamic blocks this week SQL Dynamic Blocks! That's right, you can now create dynamic outputs from SQL blocks in Mage. Previously, blocks had to be Python for dynamic outputs, but no more! This is a big step forward in making Mage more flexible and powerful. Give it a shot today. 🎉

by @tommydangerous in #4430

🌊 [Kafka Streaming Sources] Offset & Partitions

For our streaming aficionados, we have a few new Kafka features! You can now specify the offset and partitions for Kafka streaming sources. Offsets can be one of: Beginning, End, Int, & Timestamp. This allows users to set specific positions inside a topic to consume data.

by @Luishfs in #4226

⛴️ Configure Kubernetes Affinity & Tolerations

Last, but certainly not least, we've got some nitty Kubernetes configuration updates! You can now specify affinity and tolerations in your Kubernetes settings. ⚓

Node affinity is a set of rules used by the scheduler to determine where a pod can be placed. The rules are defined using custom labels on nodes and label selectors specified in pods. Node affinity allows a pod to specify an affinity (or anti-affinity) towards a group of nodes it can be placed on.

Tolerations are applied to pods and indicate that the pod can be scheduled on nodes with specific taints.

These configurations should help our Kubernetes power users make the most of Mage! 🧙

by @wangxiaoyou1993 in #4407

🐛 Bug Fixes

💅 Enhancements & Polish

New Contributors

Full Changelog: 0.9.60...0.9.62

0.9.60 | Yusuke Urameshi

19 Jan 17:45
27beeeb
Compare
Choose a tag to compare

What's Changed

🎉 Exciting New Features

🌊 [Streaming] Google Cloud Storage Destination

🎉 Google Cloud users rejoice! Streaming pipelines just even got better— Mage now supports Google Cloud Storage as a streaming destination! Check out the docs here and get started today!

by @wangxiaoyou1993 in #4340

👷‍♂️ Overwrite SQL types

For anyone with a data warehouse, listen up! (We assume that's most of you 😅)

You can now specify custom column types when exporting to SQL destinations. This is useful when you want to export a dataframe with a column that has a type that is not supported by the default mapping. You can read more about overwriting types here.

Here's an example of an exporter that overwrites column types for a PostgreSQL destination:

@data_exporter
def export_data_to_postgres(df: DataFrame, **kwargs) -> None:
    schema_name = 'your_schema_name'  # Specify the name of the schema to export data to
    table_name = 'your_table_name'  # Specify the name of the table to export data to
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'
    overwrite_types = {'column_name': 'VARCHAR(255)'}

    with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(
            df,
            schema_name,
            table_name,
            index=False,  # Specifies whether to include index in exported table
            if_exists='replace',  # Specify resolution policy if table name already exists
            allow_reserved_words=True,
            unique_conflict_method='UPDATE',
            unique_constraints=['col'],
            overwrite_types=overwrite_types,
        )

This feature is currently supported for PostgreSQL, Redshift, Trino, MSSQL, MySQL, Clickhouse, and BigQuery.

by @Luishfs in #3474

👨‍💻 [Command Center] Version Control & Files

The Mage Command Center can now be used for version control commands! You can both configure git and run your favorite version control commands directly from the Mage UI. Additionally, you can manage your files via nav and Mage's new file explorer (🧙 Arcane Library)!

As a reminder, to enable the command center, you can do so by going to Settings (the wizard icon) and click the Command Center toggle. It can be accessed via ⌘ + . (Mac) or Win + . (Windows).

by @tommydangerous in #4273

👾 [Command Center] Terminal App

Mage LEGEND @tommydangerous is back at it again— he's implemented a full terminal app into the command center. For those of you with this beta feature enabled, you'll now have a terminal at your fingertips at all times.

Check out the following video for sample usage:

CleanShot.2024-01-14.at.10.09.43.mp4

As a reminder, to enable the command center, you can do so by going to Settings (the wizard icon) and click the Command Center toggle. It can be accessed via ⌘ + . (Mac) or Win + . (Windows).

by @tommydangerous in #4365

JSON Logging

A huge shoutout to @dy46 for adding JSON logging to Mage! This will make it easier to parse logs and integrate with other logging tools. Just specify SERVER_LOGGING_FORMAT=json to change the output to something like the following:

by @dy46 in #4336

🐛 Bug Fixes

💅 Enhancements & Polish

Full Changelog: 0.9.59...0.9.60

0.9.59 | 🐲 🐉 Year of the Dragon

10 Jan 20:43
3dc128e
Compare
Choose a tag to compare

What's Changed

🎉 Exciting New Features

Note: many new features this week are in beta. You can enable them by navigating to your Mage settings and toggling the beta features there.

🎮 Multi-project Platform [BETA]

We've reworked our support for mulitple projects with a new multi-project platform! @tommydangerous is back at it again with this huge feature release, enabling nested projects, custom code paths, cross-project triggering, a split pipeline scheduler, and much more!

CleanShot.2024-01-08.at.02.33.54.mp4

If you'd like to try out the multi-project platform, you can check out this repo for a sample structure. Head over to your Mage settings to enable

🚀 Command Center [BETA]

Another new & exciting feature this week— the Mage Command Center. The command center is a floating search bar that can invoke actions like opening files & pages, perform actions within Mage, interacting with the page, and much more!

CleanShot.2024-01-08.at.03.15.15.mp4

Enable the Command Center in settings and give it a spin today!

🪣 Bitbucket Version Control

Shout out to @dy46 for continuing to crush the version control integrations!

You can now use Bitbucket as a version control provider! This is a great option for teams that use Bitbucket for their code repositories. To get started, navigate to the Mage Version Control app and select Bitbucket as your provider. You'll be prompted to authenticate with Bitbucket, and then you'll be able to select your repositories. Read more here.

by @dy46 in #4176

◔ Qdrant integration

Mage now supports Qdrant, an open-source vector search engine. Qdrant is a great tool for similarity search, and it can be used for a variety of use cases, including product recommendations, image search, and more. With this update, you can load/export data from Qdrant sources in your batch pipelines! Read more here.

by @matrixstone in #4081

🧱 dbt DX v2 + dbt Upgrade [BETA]

This release contains a huge dbt overhaul 🤯

Alongside a much-awaited upgrade to dbt 1.7, the dbt developer experience has been completely rebuilt. @tommydangerous has been hard at work crafting a dbt experience that is more intuitive, powerful, and flexible!

Here's a quick demo:

CleanShot.2024-01-08.at.03.08.21.mp4

🐛 Bug Fixes

SQL blocks

Data integration

Trigger and scheduling

dbt

File browser

Dynamic blocks

Git

Other

Read more

0.9.50 | Wonka 🎩

13 Dec 17:00
b0a3287
Compare
Choose a tag to compare

What's Changed

🎉 Exciting New Features

🌊 [Data Integration] Dremio Source


🥳 Dremio users, rejoice! Mage now supports Dremio as a data integration source, meaning you can now build data integration pipelines pulling from data lakes and more!

by @Luishfs in #4072

🏃‍♂️ Manually run pipeline once in same trigger

This update, courtesy of our frontend engineer, Johnson, is a big one!


First, he added a new button to the Trigger Detail page for running pipelines once in the same trigger! 👀

Next, some quality of life improvements:

  • For @once triggers, the trigger does not need to be active status before running the pipeline once using the Run@once button in the Trigger Detail page. Many users have told us this is confusing... No more!
  • The trigger will automatically be updated to active status. However, if the trigger is NOT an @once trigger (e.g. a recurring interval or API trigger), the trigger must be set to active status before manually running the pipeline once from the Trigger Detail page.

Finally, Johnson renamed the "start/pause" trigger on Trigger Detail page to "enable/disable" to be a bit more clear and communicative. Again, we've heard this is a bit misleading, so we did something about it! 🗣️

by @johnson-mage in #4133

[Streaming] ActiveMQ Sink


Shruti continues her epic tear of contributing magical ✨ pipelines. In this PR, she's added an ActiveMQ streaming sink to Mage. Apache ActiveMQ is an open source message broker written in Java... and now you can write data there via Mage! 💫

by @shrutimantri in #4141

🐛 Bug Fixes

💅 Enhancements & Polish

New Contributors

Full Changelog: 0.9.48...0.9.50

0.9.48 | The Boy and the Heron

07 Dec 01:56
83369a2
Compare
Choose a tag to compare

What's Changed

🎉 Exciting New Features

Incremental data integration in batch pipelines

🥳 Data integrations in batch pipelines now support incremental replication! You can read more here to get started!

by @tommydangerous in #4068

[Streaming] RabbitMQ Destination


Another community PR from @shrutimantri adds support for RabbitMQ as a streaming data sink. 🔥

Check it out today with your favorite streaming sources! You can find the configuration reference here.

by @shrutimantri in #4041

Chroma integration


Mage now has a ChromaDB IO Class, meaning you can use data loaders and exporters in your batch pipelines to read/write from Chroma sources. You can read more about configuration here or visit Chroma's site to learn more about their vector database.

by @matrixstone in #4017

Bookmark overrides


🎊 If you're creating a trigger on a data integration, you can now override bookmarks with your own custom values!

by @tommydangerous in #4073

SQL Block environment variable interpolation

For our fans of SQL blocks, you can now interpolate environment variables directly in your queries!

SELECT 
    '{{ env_var("ENV") }}' AS test
    , '{{ variables("test") }}' AS test2
    , {{ test }} AS test3

This should allow for much greater flexibility in pipelines with SQL!

by @tommydangerous in #4076

Additional upstream dependencies for dynamic children


Love dynamic blocks? 🤔 They dynamic children can now have additional upstream dependencies!

by @tommydangerous in #4104

Support caching block output in memory

Previously, pipelines with large Spark DataFrames faced out of heap space errors when persisting block outputs to disk. This PR allows the user to disable persisting output. The feature is only supported in standard batch pipeline (without dynamic blocks) for now.

cache_block_output_in_memory: true
run_pipeline_in_one_process: true

by @wangxiaoyou1993 in #4127

🐛 Bug Fixes

💅 Enhancements & Polish

New Contributors

Full Changelog: 0.9.46...0.9.48

0.9.46 | Wish 🪄

22 Nov 21:56
4228835
Compare
Choose a tag to compare

What's Changed

🎉 Exciting New Features

⚡️ Spark UI/UX 2.0 for AWS EMR

It's finally here! Mage now comes with a completely revamped custom Spark UI/UX for our AWS EMR users! This is a huge update that comes with a complete overhaul of every element possible for managing your Spark cluster!

Check out the PR for more screenshots and get started today (docs coming soon)!

by @tommydangerous in #3997

🌊 Streaming: ActiveMQ Source

🔥 @shrutimantri is on fire! In another community PR, she adds streaming support for ActiveMQ as a source! If you're an ActiveMQ user, give it a shot today!

by @shrutimantri in #3978

📁 Download Files & Pipelines via the UI

Another community PR, this one from @PopaRares, allows you to download files and pipelines via the right-click menu in the Mage UI.

This will be a game changer for collabrative projects and importing/exporting data from Mage!

by @PopaRares in #3813

✨ Streaming: NATS JetStream Source

A big shoutout to community member @mfreeman451 for adding the NATS JetStream message broker as a Streaming Source in Mage!

by @mfreeman451 in #3985

🎏 Data integration: Kafka destination

@Luishfs is back at it with another destination— this one for data integration. You can now write DI outputs to a Kafka topic. We can't wait to see what y'all cook up with this one!

by @Luishfs in #3976

🛟 Auto-save triggers in code

Last, but certainly not least, Mage is now able to auto-save triggers as code. That means (when enabled) you can update triggers and have them auto-save to your Mage project. WThis should help you keep track of your trigger changes across projects.

by @tommydangerous in #4009

🐛 Bug Fixes

💅 Enhancements & Polish

New Contributors

0.9.45 | Yuji Itadori 👹

16 Nov 02:24
345cf9b
Compare
Choose a tag to compare

What's Changed

🎉 Exciting New Features

🔐 New SSO/OAuth providers


With our latest release, Mage now supports SSO/OAuth from not one, but two providers— Okta & Google. Our engineers also thought ahead, laying the groundwork for supporting more providers in the future, too! Check out the docs— Google & Okta.

by @dy46 in #3921

🔥 Compute management for Apache Spark blocks

Tommy is back at it with another massive PR, this one adding full support for compute management in Apache Spark blocks. For those of you who leverage Spark, this PR will allow you fine-grained control over your compute. Keep an eye out for completely revamped EMR functionality in the near future!

by @tommydangerous in #3883

🤗 HuggingFace AI Client

Mage now supports using different AI models for interfaces within the applicaiton, not just OpenAI! The first we've added is a HuggingFace client... You can now use HuggingFace with Mage's AI functionality! Read more about getting started here.

by @matrixstone in #3850 and #3919

🧱 Azure Databricks Delta Lake Destination

🎉 Mage now supports Azure Databricks Delta Lake as a destination for data integration pipelines! That means you can write all of your favorite sources to the open, parquet-based storage system on Microsofts cloud infrastructure!

by @Luishfs in #3932

📊 Prometheus Metrics

Prometheus style metrics are a vendor neutral standard based on a pull model. Prometheus-enabled servers output time-series data on a route (usually /metrics), which can be scraped. Being an open standard, most monitoring tools know how to interface with Prometheus metrics (Open Telemetry support them too).

You can learn more about Prometheus here:

This PR enables the basic built-in metrics, which cover the Tornado server (http metrics) and the Python runtime. More metrics can be added in the future! Check out our docs here.

by @adelcast in #3910

🐛 Bug Fixes

💅 Enhancements & Polish

Full Changelog: 0.9.43...0.9.45

0.9.43 | Attack on Titan 💥

08 Nov 21:50
adde350
Compare
Choose a tag to compare

What's Changed

🎉 Exciting New Features

🌳 Dependency Tree 2.0

This one is huge— a complete rewrite of our dependency tree functionality! You'll notice an improved appearance and performance in addition to the following:

  • A full right click menu
  • The ability to add blocks between nodes
  • The ability to remove blocks using the menu
  • Improved connectivity through dragging lines
  • Block groups and subgroups
  • The ability to drag and drop blocks to connect
  • The ability to drag and drop groups of blocks
  • Double click to see all connections
  • New block execution animation

This new featureset is so expansive you'll need to upgrade to check it out. See this PR for a walkthough of all the features or update to the latest version to try it out today!

by @tommydangerous in #3886

🍷 Apache Pinot Data Loader

Pinot users rejoice!

Community member @shrutimantri has added a brand new data loader for the OLAP datastore! Update to the latest version to try it out.

by @shrutimantri in #3898

📊 Google Sheets Data Loader/Exporters

Who doesn't use Google Sheets?

Mage now supports reading and writing to individual Sheets/Tabs natively! This PR also includes some handy loader/exporter templates to make it easy.

Get started with your Sheets data in Mage today!

by @mattppal in #3828

🧱 Redis dependency added to Mage Helm chart

Thanks to community member @sriniarul for adding a Redis dependency to our Helm chart! Mage now supports multiple replicas. They also added a standalone scheduler and webserver option to the chart. It you use Redis you'll want to give this one a look!

by @sriniarul in mage-ai/helm-charts#22

🐛 Bug Fixes

💅 Enhancements & Polish

Full Changelog: 0.9.41...0.9.43

0.9.41 | Halloween 🎃👻

31 Oct 23:35
c75d6c9
Compare
Choose a tag to compare

What's Changed

🎉 Exciting New Features

Workspace Lifecycle Management

🎉 Mage now provides support for managing the workspace lifecycles in Kubernetes! That means you can control how Mage is deployed, start-to-finish, with the following options:

  1. Auto-termination
  2. Pre-start scripts
  3. Post-start scripts

Read more about lifecycle management here and give it a shot today!

by @dy46 in #3836

Elasticsearch Data Integration Destination

Mage now supports writing data to Elasticsearch for all of your search & LLM needs! 🧙🏻‍♂️

by @Luishfs in #3756

Block Detach for Shared Pipelines

Ok, this one is huge— say you have a block in multiple pipelines, but you need to change the logic in a single block instance... That sounds tricky, right? 🤔

Now you can with Block Detach! Simply click # Pipelines on the block, then Detatch to create a clone of the block in your current pipeline!

by @johnson-mage in #3816

🐛 Bug Fixes

💅 Enhancements & Polish

New Contributors

Full Changelog: 0.9.38...0.9.41