feat(asciidoctor): basic asciidoc plugin

[nicorikken]

The asciidoc plugin uses AsciidoctorJ from the Asciidoctor project
to bring Asciidoc rendering to Perun.
This ticket is related to issue 49 on GitHub:
#49

Left to do:

• Enable AsciidoctorJ libraries (like asciidoctor-diagram). In particular, how to handle the resulting image files? Help desired! Images are now generated on root, will have to be handled properly
• Forward already available metadata to the document. Metadata from the options is now forwarded
• Read metadata from the document, apart from the YAML
• Link AsciidoctorJ containers to a pod for improved performance. Help desired! Eventually linked to the fileset
• Get image generation by asciidoctor-diagram handled properly, to make sure they end up in the end result, and at the right place. Help desired!
• Change the metadata keywords for compatibility with the rest of Perun
• Have the defaults generate HTML similar to the Markdown HTML (which headers and H1's included).
• Expand readme with info on the Asciidoctor support and how to use it.

[nicorikken]

It is getting there. Basic tests are included.

[martinklepsch]

Link AsciidoctorJ containers to a pod for improved performance. Help desired!

This is relatively easy, right now you have:

(pod/with-call-in @pod (io.perun.contrib.asciidoctor/parse-asciidoc ~ad-files ~(merge +asciidoctor-defaults+ options)))

what you can do is make that parse-asciidoc function take a container as argument and define the container inside the pod like so:

(pod/with-call-in @pod (def container (new-container ~options)))

Then you can pass that defined container to the parse-asciidoc function.
There are also a few differences betwen with-call-in and with-eval-in (and probably some others so you might need to test which one you need).

Also here's a snippet where something similar is done.

[nicorikken]

Image rendering works, although they now get dropped in the root folder.

[nicorikken]

@martinklepsch I started working on linking the asciidoctor-container to the pod, but I guess I'm having issues with the namespaces. I'd like to leave the new-adoc-container function in the io.perun.contrib.asciidoctor namespace, to avoid bringing in other dependencies on AsciidoctorJ and the normalize-options helper. I run into either Unable to resolve symbol: container in this context or can't resolve symbol (def) errors. Also I'm not sure whether or not I should unquote the newly created container. I'll probably have to read up on the pods to get this straight.

[nicorikken]

@martinklepsch I committed my current work in progress, so you can have a look: nicorikken@6b53756

[nicorikken]

Metadata parsing is almost complete, only need to fix the existing tests. I'll commit it later today.

[nicorikken]

Will there be multiple pods handling the Asciidoctor fileset? Or will the files be parsed concurrently? Otherwise the asciidoctor container could also be defined in the parse-asciidoc function which takes the fileset and parses multiple files. At least it would make the container creation outside the pod structure easier to understand, if only for me.

[nicorikken]

I did a test to find out if some level of concurrency makes sense for conversion. Running the clojure.core.reducers/map instead of map makes little difference.

And of course I tried creating a JRuby container for each document to render. Other then being very busy, I killed it because it was still busy 15 minutes in.

[nicorikken]

All the previous checkmarks are checked! 🎉 I can still improve on the metadata, and ensure that the title is included. Also I'd like to align the keywords with the SPEC.md file for compatibility with other parts of Perun.

[nicorikken]

I've now got the metadata working properly (I had to strip the frontmatter before handing it over to the .readDocumentStructure for parsing the header information. The same goes for the .convert itself, as the skip-front-matter attribute fails to influence the document conversion somehow. Maybe that only affects a full-document parsing (.convertFile rather than .convert). On the same topic I'll have to look into the header and footer parsing. Now the H1 and author attributes are skipped fully, in contrast with the Markdown parser. There is an option for including the title (H1), but that would still leave out the author and revision information. Maybe that would would still be the most sane default.

[nicorikken]

Very close now, only image handling left.

[nicorikken]

The image generation using the `asciidoctor-diagram` library can use the `imagesoutdir` attribute to determine where the final images will end up. Otherwise they will end up at the directory from which the task is called. The `imagesdir` should match the `imagesoutdir` to some extent, to ensure the final HTML will refer to the correct files. _But how to get this working in Boot...?_ Multiple solutions exist to get to the image output directory https://github.com/asciidoctor/asciidoctor-diagram#image-output-location

[nicorikken]

I'm also working on an improved version of the YAML frontmatter strip action, to properly handle files without frontmatter. When those test run, I'll commit it, and only the image handling seems left.

[nicorikken]

Got the tests working (apparently map destructuring in the let block was giving issues), so now only documenting and image inclusion remains.

[nicorikken]

Regarding the image generation, I should be able to set the imagesoutdir to a new tmp-dir! and I should be able to commit them on the imagesdir and outdir stated in the main Asciidoctor file.

[nicorikken]

I was trying to get the path from the Asciidoctor file being processed by the Boot Fileset methods (tmp-path f) or (tmp-file f) but both result in stack-traces:

• clojure.lang.ExceptionInfo: No implementation of method: :path of protocol: #'boot.tmpdir/ITmpFile found for class: clojure.lang.PersistentArrayMap
• clojure.lang.ExceptionInfo: No implementation of method: :file of protocol: #'boot.tmpdir/ITmpFile found for class: clojure.lang.PersistentArrayMap
  Can somebody help me out to get the image handling compatible with Boot? Another option is to remove the asciidoctor-diagram library from the defaults and merge this PR with the mention that images handling is not yet worked out.

[nicorikken]

I'm getting there. Just fixed a but in merging maps (forgot to provide a function to merge-with 😛 )

TODO

• Convert options in lowest functions (normalize-options)
• Get it working for build-dev, not only for tests
• Make a tmpdir for the images and provide to the parse-adciidoc
• Add image directories in the in- and output for tmpdir

[nicorikken]

Now that I have fixed controlling the output directory, I could also set it to the :base_dir such that they are next to the parsed .adoc files. More in the way that the proposed PR #79 handles images. Maintainers, any preferences?

[nicorikken]

I did spend some more evenings to clean up and get images working. I found out that my render is somehow broken. Pages are rendered, committed to the fileset, but not synced to the target dir. This concerns the HTML renders, also for Markdown. Unless I get it working again, debugging Asciidictor fileset handling is impossible.

[Deraen]

Did you include target task in the pipeline? Boot 2.6 no longer automatically writes target dir.

[nicorikken]

No I did not. Thanks for mentioning, I'll give it another go very soon, hopefully this PR is now actually in a working state.

[nicorikken]

Alright, back on track. Somehow the image is currently ending up in the src directory, and the perhaps the metadata is not handled properly. Expect me back with fixes soon.

[nicorikken]

@bhagany thanks for taking my implementation issues and improving the general ecosystem!

[nicorikken]

Just a heads up, my progress stalled due to a new laptop and a temporary project. I'd like to get something up and running before Fosdem, so perhaps I'll split the PR to ignore the asciidoctor diagram extension for now. Or I'll generate them, but dump them in a fixed directory for now.

[bhagany]

@nicorikken glad to hear you're still in-progress on this, I think this feature is a great addition to Perun. I don't really know much about asciidoctor, but I like your idea of splitting the PR.

I don't know if you saw, but #132 changes the way content-parsing tasks are written. It should work for asciidoctor, but I haven't actually tried to make it work. Feel free to ping me here or on the Clojurians Slack if that gets merged and you run into issues.

Thanks for continuing to work on this!

[nicorikken]

Picked this project up again. As it has gotten terribly behind on master, and I gained much more Clojure experience since my last few commits, I'll restart from master. I'm glad the Pandoc converter has made its way into the project, as I can expect the abstractions to be more stable and sensible. E.g. I'll be using the generic YAML parser. Also I like the hardcoded set of markdown extension as a pattern. I'll probably do the same, rather then increasing complexity by making it more dynamic. Also I'll ignore the asciidoctor-diagram for now then, to further lower complexity.

[bhagany]

Sounds like a solid approach to me. Thanks again for working on this, and I look forward to getting this merged!

[nicorikken]

I just finished setting out the basic task structure: 62f409a...nicorikken:asciidoctor-plugin-redo
The structure is totally different now, but the abstraction from Boot filesets makes me more comfortable to add another tasks. I'd probably still have to dive in when dealing with the asciidoctor-diagram plugin in the future, but as said before, I'll leave that for now.
Next step will be to reintroduce the Asciidoctor code I wrote earlier to do the actual conversion.

[nicorikken]

I just finished the basic implementation: nicorikken@99283e5
It lacks some unit-tests, otherwise it is good to merge.

[nicorikken]

Just added some basic tests. They do however not fail when loading wrong content to test against, so perhaps I'm testing it wrong. Either way, the new PR can be found here: #183

[nicorikken]

After this basic task is merged, I'd like to extend it by syncing the Asciidoctor attributes and Perun meta. This would be something like:

;; For asciidoctor, YAML meta and Asciidoctor attributes to resulting meta
{:meta (->> meta
            (merge yaml-metadata)
            meta->attr
            (fn [attr] (.getAttributes (.getHeader (.readDocumenStructure container file-content {:attributes attr}))))
            attr->meta)}

;; For asciidoctor*, YAML meta and Asciidoctor attributes used in converting, not added to resulting meta
(->> meta
     (merge yaml-metadata)
     meta->attr
     (process-asciidoctor file-content {:attributes attr}))     

[nicorikken]

Reading about the YAML frontmatter handling in the Asciidoctor user manual, the section on the handling by Awestruct shows an increased complexity, as the precedence of attributes can be changed by adding an @ after the assignment: http://asciidoctor.org/docs/user-manual/#configuring-attributes-for-awestruct I believe the solution for Perun should be more straightforward than that. If necessary control can be provided what keys are passed on in the metadata.

[bhagany]

@nicorikken Is this PR still relevant, after merging the other one?

[nicorikken]

No, although I'll probably come back to these comments for implementing metadata and asciidoctor-diagram. But I can do so while this is closed. Let's keep it clean.

[nicorikken]

I spent the last few days rebuilding my blog, and as a result I now have a working solution for metadata extraction, and am working on getting a limited image-generation in place. Expect an asciidoctor PR soon.

[podviaznikov]

Great to hear! Waiting on your PR

…

On Aug 17, 2017, at 10:45 PM, Nico Rikken ***@***.***> wrote: I spent the last few days rebuilding my blog, and as a result I now have a working solution for metadata extraction, and am working on getting a limited image-generation in place. Expect an asciidoctor PR soon. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.