Version: 0.2.1
Author: Ossian Ravn Engmark ([email protected])
License: GPL v2 or later
Requires WordPress: 5.0+
Requires PHP: 7.4+
- Overview
- Key Features
- Installation & Requirements
- Basic Usage
- Admin Settings
- Per-Post Controls
- Context Files
- Caching System
- SEO Handling & Noindex
- Testing & Debug Tools
- FAQ / Troubleshooting
- Installation via Composer
- Changelog
- Contributing & Support
- License
The Markdown Mirror (llms.txt) plugin dynamically converts your WordPress posts and pages into Markdown on the fly and generates a root-level llms.txt file. This file acts as a machine-friendly "AI sitemap," guiding advanced language models and developer tools to your site's key content while ensuring your SEO remains intact.
Key benefits include:
- Dynamic Markdown Conversion: Converts posts/pages to Markdown (using the League HTML-to-Markdown library) when requested.
- AI-Friendly Index: Creates an
llms.txtfile that aggregates your content in a clean, Markdown-formatted list. - Extended Context Files: Generates
llms-ctx.txtandllms-ctx-full.txtfor enhanced content discovery and relationships. - SEO-Friendly: Adds canonical links and SEO headers without disallowing
.mdURLs inrobots.txt—ensuring both search engines and AI crawlers can access your content appropriately. - Efficient Caching: Speeds up repeated requests by caching both individual Markdown conversions and the global context files.
- Per-Post Control: Allows you to include or exclude specific posts from the Markdown mirror via a simple meta box.
- Robust Admin Interface: Provides detailed settings and built-in tests to ensure proper functionality.
- Dynamic Conversion: Serve your posts, pages, and taxonomy archives as Markdown files via custom rewrite rules.
- Automatic Context Generation: Generate three levels of content discovery:
llms.txt: Basic content listing with excerptsllms-ctx.txt: Core content without optional URLsllms-ctx-full.txt: Complete content including all referenced URLs
- Taxonomy Support: Include categories and tags in context files for better content organization, with optional Markdown versions of taxonomy archive pages
- Admin Settings & Meta Boxes: Easily select which post types to mirror and manage per-post inclusion.
- Caching System: Automatically caches converted Markdown and context files to improve performance.
- SEO Enhancements: Adds canonical links, alternate links, and X-Robots-Tag headers for proper content discovery and SEO optimization.
- Test Suite: Run diagnostic tests (available in WP_DEBUG mode) to verify rewrite rules, conversion accuracy, caching, and SEO headers.
- Custom Autoloader & Composer Support: Leverages a custom PSR-4 autoloader along with Composer for dependency management.
-
Download & Upload:
- Option 1: Download the latest release zip file directly
- Option 2: Upload the entire
markdown-mirrorfolder to your/wp-content/plugins/directory manually
-
Activate the Plugin:
In the WordPress admin, navigate to Plugins > Installed Plugins and activate Markdown Mirror (llms.txt). -
Configure Settings:
Visit Settings > Markdown Mirror to choose which post types to mirror, set a custom summary forllms.txt, and adjust caching and SEO options. -
Permalinks:
If your.mdURLs orllms.txtfile are not working as expected, go to Settings > Permalinks and click Save Changes to flush rewrite rules.
-
Clone the Repository:
git clone https://github.com/[your-username]/markdown-mirror.git cd markdown-mirror -
Install Dependencies:
composer install
-
Build the Plugin:
./build.sh
- WordPress 5.0+
- PHP 7.4+
- Pretty permalinks enabled
- Composer (for development)
Once activated, the plugin automatically:
-
Serves Markdown Versions:
Your content is available as Markdown files at URLs like:- Posts/Pages:
https://your-site.com/your-post.md - Categories:
https://your-site.com/category/your-category.md - Tags:
https://your-site.com/tag/your-tag.md
- Posts/Pages:
-
Generates
llms.txt:
A Markdown-formattedllms.txtfile is available at:
https://your-site.com/llms.txt -
Admin Controls:
- Global Settings: Configure which post types and taxonomy archives to include, custom summary, and SEO options.
- Per-Post Controls: Use the meta box on the post edit screen to include or exclude specific posts.
-
Testing:
(When WP_DEBUG is enabled) Run tests under Tools > MD Mirror Tests to verify rewrite rules, caching, conversion, and SEO header functionality.
Access the Markdown Mirror settings under Settings → Markdown Mirror in your WordPress dashboard. The settings are divided into three sections:
-
Include Content Types:
Select which content to make available as Markdown:- Post types (Posts, Pages, custom types, etc.)
- Category archives
- Tag archives
- Custom taxonomy archives
-
Include Categories & Tags:
Toggle inclusion of taxonomy information in context files. -
Custom Summary:
Enter a custom summary to appear at the top of the context files. If left blank, your site tagline is used.
- Basic Context Inclusion:
Choose what content to include inllms-ctx.txt:- Post titles
- Meta descriptions
- Excerpts
- Full content
- Full Context Depth:
Set how many levels deep to follow internal links inllms-ctx-full.txt
- Cache Duration:
Set how long to cache the converted Markdown content and context files (e.g., 1 hour, 1 day, or 1 week) to improve performance.
To manage Markdown inclusion on individual posts:
-
Edit a Post/Page:
Open the post you want to control in the editor. -
Locate the Meta Box:
Find the Markdown Mirror meta box on the right-hand side. -
Toggle Inclusion:
Choose "Include in Markdown Mirror" or "Exclude from Markdown Mirror".- Excluded posts return a 404 on
.mdURLs and are omitted fromllms.txt.
- Excluded posts return a 404 on
By default, all posts are included unless explicitly excluded.
The basic context file (llms-ctx.txt) provides a clean, URL-free version of your content organized by:
- Categories with descriptions and associated posts
- Tags with descriptions and associated posts
- Chronological listing of all posts and taxonomy archives
- Configurable content inclusion (titles, excerpts, meta descriptions)
The full context file (llms-ctx-full.txt) includes:
- Everything from the basic context
- Complete post content in Markdown format
- Taxonomy archive content when enabled
- Nested content from referenced internal links
- Configurable link depth for content expansion
- Full taxonomy relationships
Both context files can be customized through the admin settings to control:
- Which content elements to include
- Which taxonomy archives to include
- How deep to follow internal links
- Whether to include taxonomy information
- Cache duration for optimal performance
The plugin caches:
- Per-Post Markdown Content: To prevent repeated conversions.
- Global
llms.txtContent: To minimize processing on each request. - Context Files: To speed up repeated requests and improve content organization.
Cache Invalidation Triggers:
- Creating, updating, or deleting a post.
- Changing a post's publish status.
- Updating relevant meta data (e.g.,
_md_mirror_include, SEO descriptions). - Modifying plugin settings (post types, custom summary).
This ensures that content is current while reducing server load.
To safeguard your site's SEO, the plugin:
-
Noindex Markdown Pages:
When enabled in the settings,.mdpages receive anX-Robots-Tag: noindex, nofollowheader to prevent duplicate content issues. -
Canonical Links:
Each.mdpage includes a canonical link header pointing back to the original HTML page. -
Alternate Links:
Original HTML pages include arel="alternate"link to their Markdown versions, helping search engines and AI tools discover the Markdown content. The Markdown pages, in turn, include both a canonical link back to the HTML version and an alternate link to themselves. -
Robots.txt Adjustments:
The plugin adds references tollms.txtinrobots.txtbut no longer disallows.mdfiles, ensuring that both search engines and AI crawlers can access them if needed.
A built-in test suite is available under Tools → MD Mirror Tests (visible when WP_DEBUG is enabled). It checks:
- Rewrite Rules: Verifies that requests for
llms.txtand.mdURLs are processed correctly. - Conversion Accuracy: Ensures HTML is correctly converted to Markdown.
- Caching Behavior: Confirms that content is being cached and cleared as expected.
- SEO Headers: Validates that the appropriate HTTP headers (e.g., Content-Type, X-Robots-Tag, canonical link) are set.
- Settings & Meta Box Functionality: Checks that the default options and per-post controls work correctly.
Each test reports a Pass/Fail status with details for troubleshooting.
Q1: I get a 404 on my .md or llms.txt URLs.
- A: Flush your permalinks by going to Settings → Permalinks and clicking Save Changes. Also, ensure your server supports URL rewriting.
Q2: My .md pages are appearing in search results.
- A: Confirm that the NoIndex Markdown URLs option is enabled. Although the plugin no longer disallows
.mdfiles inrobots.txt, the appropriate noindex headers should prevent them from being indexed.
Q3: The conversion seems slow for very large posts.
- A: Enable caching (the default) to reduce processing time on subsequent requests.
Q4: Are physical .md files created on my server?
- A: No. All Markdown is generated dynamically without storing physical files.
Q5: How do I exclude a specific post?
- A: Use the meta box on the post editor screen to mark a post as excluded from the Markdown mirror.
Q6: Can I create Markdown versions of my category and tag pages?
- A: Yes! Go to Settings → Markdown Mirror and under "Include Content Types", check the boxes for "Category Archives" and "Tag Archives". Your taxonomy archives will then be available at URLs like
https://your-site.com/category/your-category.md.
If you prefer to manage dependencies with Composer, navigate to the plugin directory and run:
composer installThis will install the HTML-to-Markdown library and any other required packages. Ensure your environment meets the PHP 7.4+ requirement.
- Added Visual Composer shortcode support with proper content rendering
- Added convenient sitemap links to the admin settings page
- Improved admin UI with direct access to llms.txt, llms-ctx.txt, and llms-ctx-full.txt
- Enhanced shortcode cleanup for better Markdown output
- Added support for Markdown versions of taxonomy archive pages
- Added category and tag archives as optional content types in settings
- Implemented Markdown generation for taxonomy archives
- Added SEO headers and alternate links for taxonomy pages
- Updated rewrite rules to handle taxonomy URLs
- Added customizable context generation through admin settings
- Implemented expanded context files (llms-ctx.txt and llms-ctx-full.txt)
- Added automated context updates through caching system
- Initial beta release with core functionality
- Dynamic Markdown conversion and context file generation
- Added support for three context file types: llms.txt, llms-ctx.txt, and llms-ctx-full.txt
- Implemented category and tag support in context files
- Added per-post control via meta boxes
- Implemented caching, SEO enhancements, and rewrite rules
- Included a comprehensive test suite for verifying core functionalities
- Added admin settings for controlling context file content and behavior
- Note: The disallow rule for
.mdfiles inrobots.txthas been removed
- Use the GitHub issue tracker to report bugs
- Include as much detail as possible using the provided issue templates
- Follow the bug report template for bugs and feature request template for new features
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Make your changes
- Run the build script to test (
./build.sh) - Commit your changes (
git commit -m 'Add amazing feature') - Push to your branch (
git push origin feature/amazing-feature) - Open a Pull Request
build.sh: Creates a distributable zip file./build.sh
release.sh: Handles version bumping, changelog updates, and deployment./release.sh <new-version> # Example: ./release.sh 1.1.0
Options:
--skip-git: Skip Git operations--skip-svn: Skip WordPress.org SVN deployment
The repository includes GitHub Actions workflows that:
- Build the plugin on tag pushes
- Create GitHub releases
- Attach distribution files
- Generate release notes
- For advanced customization, refer to inline code comments
- Check the Wiki for detailed documentation
- Review available hooks and filters in the source code
This plugin is licensed under the GPL v2 or later.
Happy Markdowning!