Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revamp build process #56

Merged
merged 92 commits into from
Feb 6, 2025
Merged

Revamp build process #56

merged 92 commits into from
Feb 6, 2025

Conversation

ribose-jeffreylau
Copy link
Contributor

Fixes #55 and potentially resolves #53 .

@ribose-jeffreylau ribose-jeffreylau added the bug Something isn't working label Jan 27, 2025
@ribose-jeffreylau ribose-jeffreylau self-assigned this Jan 27, 2025
Makefile Outdated Show resolved Hide resolved
scripts/get-all-doc-paths-of-type Show resolved Hide resolved
@ronaldtse
Copy link
Contributor

I think the shell scripts are quite crazy and unsuitable because they are extremely hard to manage for being entirely semantic-less.

@ronaldtse
Copy link
Contributor

You probably want to read the "metanorma.yml" files in each to determine what files to publish, and only publish the "cc" flavor?

@ribose-jeffreylau
Copy link
Contributor Author

I think the shell scripts are quite crazy and unsuitable because they are extremely hard to manage for being entirely semantic-less.

How would a semantically aware solution look like? What kind of semantics do we require?

You probably want to read the "metanorma.yml" files in each to determine what files to publish, and only publish the "cc" flavor?

Yes, that's the direction I'm going with.

@ronaldtse
Copy link
Contributor

Necessary semantics:

  • is a document meant to be published
    • is the document a CC flavor document
  • is the document meant to be published in this category
    • what category does the document belong

I just don't think shell scripts are maintainable when working with content that spans formats. You're looking at Metanorma files and YAML at the same time.

@ribose-jeffreylau
Copy link
Contributor Author

Encountering some build issues when running metanorma site generate -o _site/documents/admin -c metanorma-admin.yml:

[info]: Compiling /Users/user/share/src/groups/paneron/standards.calconnect.org/src-documents/cc-admin-documents/sources/cc-0812-pres-meet-cc/pres-intro-01.adoc ...
/Users/user/.asdf/installs/ruby/3.3.7/lib/ruby/gems/3.3.0/gems/relaton-itu-1.14.3/lib/relaton_itu.rb:1: warning: mutex_m was loaded from the standard library, but will no longer be part of the default gems starting from Ruby 3.4.0.
You can add mutex_m to your Gemfile or gemspec to silence this warning.
/Users/user/.asdf/installs/ruby/3.3.7/lib/ruby/gems/3.3.0/gems/mechanize-2.8.5/lib/mechanize.rb:1369: warning: nkf was loaded from the standard library, but will no longer be part of the default gems starting from Ruby 3.4.0.
You can add nkf to your Gemfile or gemspec to silence this warning.
Fatal Error: undefined method `join' for an instance of String
bundler: failed to load command: metanorma (/Users/user/.asdf/installs/ruby/3.3.7/bin/metanorma)
/Users/user/.asdf/installs/ruby/3.3.7/lib/ruby/gems/3.3.0/gems/metanorma-standoc-2.4.2/lib/metanorma/standoc/blocks.rb:206:in `paragraph': undefined method `join' for an instance of String (NoMethodError)

        end.join("\n")
           ^^^^^
        from /Users/user/.asdf/installs/ruby/3.3.7/lib/ruby/gems/3.3.0/gems/metanorma-standoc-2.4.2/lib/metanorma/standoc/utils.rb:13:in `convert'
        from /Users/user/.asdf/installs/ruby/3.3.7/lib/ruby/gems/3.3.0/gems/asciidoctor-2.0.23/lib/asciidoctor/abstract_block.rb:76:in `convert'
        from /Users/user/.asdf/installs/ruby/3.3.7/lib/ruby/gems/3.3.0/gems/asciidoctor-2.0.23/lib/asciidoctor/abstract_block.rb:85:in `block in content'
        from /Users/user/.asdf/installs/ruby/3.3.7/lib/ruby/gems/3.3.0/gems/asciidoctor-2.0.23/lib/asciidoctor/abstract_block.rb:85:in `map'
        from /Users/user/.asdf/installs/ruby/3.3.7/lib/ruby/gems/3.3.0/gems/asciidoctor-2.0.23/lib/asciidoctor/abstract_block.rb:85:in `content'
        from /Users/user/.asdf/installs/ruby/3.3.7/lib/ruby/gems/3.3.0/gems/asciidoctor-2.0.23/lib/asciidoctor/block.rb:111:in `content'
        from /Users/user/.asdf/installs/ruby/3.3.7/lib/ruby/gems/3.3.0/gems/metanorma-standoc-2.4.2/lib/metanorma/standoc/section.rb:169:in `block (2 levels) in preamble'

Investigating why.

@ronaldtse
Copy link
Contributor

You can skip 0812 for now. Please refer to cc-admin-documents for which documents to compile because the build there passes.

@ribose-jeffreylau
Copy link
Contributor Author

@ronaldtse How much do we want to keep Jekyll? It turns out there's an incompatibility between latest jekyll and metanorma (via relaton-cli):

Resolving dependencies...
Could not find compatible versions

Because metanorma-standoc >= 2.9.0, < 2.10.3 depends on relaton-cli ~> 1.19.0
  and metanorma-standoc >= 2.10.3 depends on relaton-cli ~> 1.20.0,
  metanorma-standoc >= 2.9.0 requires relaton-cli ~> 1.19.0 OR ~> 1.20.0.
And because metanorma-cli >= 1.11.0 depends on metanorma-standoc ~> 2.10.0,
  metanorma-cli >= 1.11.0 requires relaton-cli ~> 1.19.0 OR ~> 1.20.0.
And because relaton-cli >= 1.15.1 depends on liquid ~> 5
  and jekyll >= 3.5.0 depends on liquid ~> 4.0,
  metanorma-cli >= 1.11.0 is incompatible with jekyll >= 3.5.0.
So, because Gemfile depends on metanorma-cli ~> 1.11.5
  and Gemfile depends on jekyll ~> 4.4,
  version solving has failed.

If we want to keep Jekyll (for building the non-metanorma HTMLs), an alternative solution may be to use separate Gemfiles for the different build stages. I am going to try that first.

@ronaldtse
Copy link
Contributor

Yes, please use different gem files for the document builds and Jekyll builds. This is not the first time we've encountered the issue where Jekyll is bound to older gems.

@ribose-jeffreylau ribose-jeffreylau force-pushed the 55-build branch 3 times, most recently from 89fe0e6 to 0e59b12 Compare February 4, 2025 10:55
.gitmodules Outdated Show resolved Hide resolved
Gemfile Outdated Show resolved Hide resolved

build-standards:
name: Build standards docs
needs: build-site
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the approach now to build 3 sites independently and then combining them in the directory to publish? Are we able to use the CalConnect styling for the metanorma mini-sites?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the approach now to build 3 sites independently and then combining them in the directory to publish?

Yes.

Are we able to use the CalConnect styling for the metanorma mini-sites?

Not at the moment. Will investigate.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems this approach requires a set of .liquid templates (via relaton, via metanorma-cli) for the CalConnect style. If there exists none, I'll create them...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Observation: {% include tag as used in Jekyll is not available in Relaton, as it is specific to Jekyll: https://github.com/jekyll/jekyll/blob/master/lib/jekyll/tags/include.rb

Meanwhile, Relaton uses straight liquid.

This makes the current set of Jekyll _includes and _layouts unreusable.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just in case you are not aware of the original approach, which is to build the documents, and input the Relaton Bib YAML files into Jekyll to render the site. This way Jekyll reads the Relaton files as native "data files".

@ribose-jeffreylau ribose-jeffreylau merged commit 8bc1d42 into main Feb 6, 2025
8 checks passed
@ribose-jeffreylau
Copy link
Contributor Author

Oops, merged by accident. Undid the merge, and now this PR remains closed...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Revamp doc and site build flow Restore missing artifacts?
2 participants