0.9.62 | The Beekeeper 🐝

What's Changed

🎉 Exciting New Features

🧵 [Mage IO] Weaviate Integration

🤔 Building AI apps with Mage? Perfect! Now you can use Weaviate as a data source. Weaviate is an open-source, AI-native vector database that helps developers create intuitive and reliable AI applications. With Mage, you can now read from and write to Weaviate databases! Read more in our docs here.

by @matrixstone in #4158

🔍 [Mage IO] Alogia Integration

Like search? Us too! That's why we've added support for Alogia as a data source in Mage. Alogia is a powerful search engine that helps you build fast and accurate search experiences. With Mage, you can now read from and write to Alogia! Read more in our docs here.

by @matrixstone in #4198

💥 Dynamic SQL Blocks

Another big improvement to dynamic blocks this week SQL Dynamic Blocks! That's right, you can now create dynamic outputs from SQL blocks in Mage. Previously, blocks had to be Python for dynamic outputs, but no more! This is a big step forward in making Mage more flexible and powerful. Give it a shot today. 🎉

by @tommydangerous in #4430

🌊 [Kafka Streaming Sources] Offset & Partitions

For our streaming aficionados, we have a few new Kafka features! You can now specify the offset and partitions for Kafka streaming sources. Offsets can be one of: Beginning, End, Int, & Timestamp. This allows users to set specific positions inside a topic to consume data.

by @Luishfs in #4226

⛴️ Configure Kubernetes Affinity & Tolerations

Last, but certainly not least, we've got some nitty Kubernetes configuration updates! You can now specify affinity and tolerations in your Kubernetes settings. ⚓

Node affinity is a set of rules used by the scheduler to determine where a pod can be placed. The rules are defined using custom labels on nodes and label selectors specified in pods. Node affinity allows a pod to specify an affinity (or anti-affinity) towards a group of nodes it can be placed on.

Tolerations are applied to pods and indicate that the pod can be scheduled on nodes with specific taints.

These configurations should help our Kubernetes power users make the most of Mage! 🧙

by @wangxiaoyou1993 in #4407

🐛 Bug Fixes

• Fix LDAP unpacking by @dy46 in #4399
• Fix reduce output and triple layout saving by @tommydangerous in #4409
• Fix file browser bug and Git push bug by @tommydangerous in #4411
• Fix serializing list and dict when formatting output by @tommydangerous in #4412
• Prevent index out of bounds by @tommydangerous in #4425
• Remove test print statements by @tommydangerous in #4431
• Removing Draft7 validation from Clickhouse destination by @Luishfs in #4424
• Fix global_vars context in pipeline executor by @wangxiaoyou1993 in #4435
• Fix a few global data product bugs by @tommydangerous in #4440
• Fix dynamic blocks OOM round 2 by @tommydangerous in #4445
• Fix incremental sync in chargebee source by @wangxiaoyou1993 in #4450
• Don’t count values if is None by @tommydangerous in #4454
• Fix policy issue updating settings by @tommydangerous in #4456
• Fix keyboard shortcuts when its empty array by @tommydangerous in #4458
• Run submodule sync by @dy46 in #4457
• Fixing Snowflake write_pandas issue by @Luishfs in #4395
• Fix writing to Snowflake with mixed int and str types by @wangxiaoyou1993 in #4460
• Fix some bugs and improve the edit page by @tommydangerous in #4462
• Fix error logging in pipeline executor by @tommydangerous in #4468
• When clicking show file versions in arcane library, show right panel by @tommydangerous in #4472
• Catch BigQuery if it fails to fix table names by @tommydangerous in #4479
• Fix pipeline detail prop passed for fetching files by @johnson-mage in #4485
• Always show edit pipeline button by @dy46 in #4484
• Revert change to PG IO by @tommydangerous in #4486
• Fix io redshift by @wangxiaoyou1993 in #4487
• Fix multi project flag by @wangxiaoyou1993 in #4490
• Fix Bigquery clean column name by @wangxiaoyou1993 in #4500
• Convert datetime type for s3 data integration destination by @wangxiaoyou1993 in #4501
• Fix dynamic child block outputs by @tommydangerous in #4422
• Fix database missing and serializing QueryJob by @tommydangerous in #4428
• Fix incorrect spelling by @ckfear in #4438
• Fix kafka type and doc by @wangxiaoyou1993 in #4466

💅 Enhancements & Polish

• Workspace improvements by @dy46 in #4469
• Scheduler improvements by @wangxiaoyou1993 in #4467
• Prevent unnecessary initial pipeline run by @johnson-mage in #4291
• Speed up monitor stats and reduce calls on overview by @tommydangerous in #4408
• Bump up dependency versions to resolve vulnerabilities by @wangxiaoyou1993 in #4433
• Limit dynamic block output in notebook by @tommydangerous in #4436
• Improve bigquery name parsing to help fill in full name by @tommydangerous in #4447
• Add error logging by @tommydangerous in #4448
• Fix app slowness due to project platform check by @wangxiaoyou1993 in #4483
• Add PVC retention policy by @dy46 in #4491
• E2e test for /pipelines by @edmondwinston in #4306
• Add query decorator for data integration blocks by @tommydangerous in #4465

New Contributors

• @ckfear made their first contribution in #4438

Full Changelog: 0.9.60...0.9.62