cltpt

<2025-10-19 Sun 02:48:14>
blog program
note that this website was generated using cltpt from org-mode files. so this page/webapp can be a testament to whether org->html conversion works properly (broken text elements is an indicator of parser/conversion failure).

introduction

https://github.com/mahmoodsh36/cltpt is both a tool and a library that is meant to serve as a base for the upcoming organ-mode package for the lem text editor. it is very much a WIP, it is unstable and breaking changes may be introduced without warnings.
the goal of cltpt is to maintain an editor-independent codebase and toolset (the existence of countless parsers for org-mode should be enough a motive for this). the core will be written in common lisp but will be independent of lem's libraries, but ofcourse this is supposed to (mainly) be a package for lem so some parts are bound to be lem-dependent. although lem itself can also be used as a library in common lisp, i think that this is still a good idea.

installation

nixos

if you are on nixos you may use the provided flake. running the commandline tool is as simple as something like: 
nix run github:mahmoodsh36/cltpt#cltpt -- -h

quicklisp

since the dependencies are all listed in cltpt.asd, cloning the code and then running (ql:quickload :cltpt) in the code directory should be enough to load the library.
to make this perist you may clone the repo into ~/.quicklisp/local-projects/. then you may overwrite main.lisp (which uses asdf directly without quicklisp) to make it work with quicklisp: 
(ql:quickload :cltpt)
(cltpt/zoo:init)
(cltpt/commandline:commandline-main (uiop:command-line-arguments))
then you may use the commandline like: run.sh -h. (in the following sections you may replace cltpt with run.sh and the commands will work.)

example commandline usage

this section will highlight what i think would be the most common usage for the commandline tool.

converting from org-mode to html

convert a single file from org to html (the % part is for "formatting" with lisp code): 
# this command writes the file to /tmp/test.html and the static files it links to to ~/tmp/~ aswell.
cltpt convert\
      -f 'test.org'\
      -d html\
      -o '/tmp/%(getf *file-info* :filename-no-ext).html'\
      -c '/tmp/%(getf *file-info* :filename)
convert a bunch of files from org to html: 
cltpt convert\
      -r '(:path "/home/<user>/dir1/" :glob "*.org" :format "org-mode")'\
      -r '(:path "/home/<user>/dir2/" :glob "*.org" :format "org-mode")'\
      -r '(:path "/home/<user>/dir3/file.org" :glob "*.org" :format "org-mode")'\
      -d html\
      -o '/tmp/%(getf *file-info* :filename-no-ext).html'
notice that you can stack multiple instances of the -r (or --rule) argument. this is also possible with the -f argument (which is used for single files): 
cltpt convert\
      -f ~/notes/file1.org\
      -f ~/notes/file2.org\
      -d html\
      -o '/tmp/%(getf *file-info* :filename-no-ext).html'
this would generate two files, /tmp/file1.html and /tmp/file2.html.

converting from org-mode to latex

notice that this is very similar to the previous commands, with slight differences in the arguments passed to the tool, such as the name of the destination format (the format we want to convert our files to).
convert a single file from org to latex (write it to /tmp/test.tex): 
cltpt convert\
      -f 'test.org'\
      -d latex\
      -o '/tmp/%(getf *file-info* :filename-no-ext).tex'
convert a bunch of files from org to latex: 
cltpt convert\
      -r '(:path "/home/<user>/notes/" :glob "*.org" :format "org-mode")'\
      -d latex\
      -o '/tmp/%(getf *file-info* :filename-no-ext).tex'

"roam" queries

the library provides an "org-roam-like" interface that works with different methods of identifying different types of objects in your text files. by default it recognizes org-id "identifiers" and links (the ones org-roam use for headers/files). denote-like identifiers and (more importantly) blk-like identifiers which includes the previous ones.
the following command queries a directory of org-mode files and prints the title, id and filepath for each one in the specified format (-o argument). 
cltpt roam\
      -r "(:path \"$ORG_DIR\" :glob \"*.org\" :format \"org-mode\")"\
      -o 'title: %title, id: %id, file: %file'

title: my doc, id: NIL, file: /home/mahmooz/work/cltpt/tests/test.org
title: header my secondary header, id: NIL, file: /home/mahmooz/work/cltpt/tests/test.org
title: send the professor a mail, id: NIL, file: /home/mahmooz/work/cltpt/tests/test.org
title: NIL, id: def-vector, file: /home/mahmooz/work/cltpt/tests/test.org
title: do something else, id: NIL, file: /home/mahmooz/work/cltpt/tests/test.org
title: header do something, id: NIL, file: /home/mahmooz/work/cltpt/tests/test.org
title: another due task, id: NIL, file: /home/mahmooz/work/cltpt/tests/test.org
title: due task, id: NIL, file: /home/mahmooz/work/cltpt/tests/test.org
title: NIL, id: def-ac-standard, file: /home/mahmooz/work/cltpt/tests/test.org
title: NIL, id: test-name, file: /home/mahmooz/work/cltpt/tests/test.org
title: NIL, id: NIL, file: /home/mahmooz/work/cltpt/tests/test2.org
title: some task, id: NIL, file: /home/mahmooz/work/cltpt/tests/test2.org
title: part 1, univariate linear regression, id: NIL, file: /home/mahmooz/work/cltpt/tests/test2.org

"agenda" queries

the following command prints an agenda "tree" after retrieving all TODOs from the org files found in the specified directory. the default range displayed is 7 days from now. 
cltpt agenda\
      -r "(:path \"$ORG_DIR\" :glob \"*.org\" :format \"org-mode\")"

├─ Tuesday 21 October
│ ├─ 00:00
│ ├─ 02:00
│ ├─ 04:00
│ ├─ 06:00
│ ├─ 08:00
│ ├─ 10:00
│ ├─ 12:00
│ ├─ 14:00
│ ├─ 16:00
│ ├─ 18:00
│ ├─ 20:00
│ └─ 22:00
├─ Wednesday 22 October
├─ Thursday 23 October
├─ Friday 24 October
├─ Saturday 25 October
├─ Sunday 26 October
└─ Monday 27 October
to print an agenda tree for a different range of timestamps, we use: 
cltpt agenda\
      -r "(:path \"$ORG_DIR\" :glob \"*.org\" :format \"org-mode\")"\
      -f '2025-10-13'\
      -t '2025-10-18'

├─ Monday 13 October
│ ├─ 00:00
│ ├─ 02:00
│ ├─ 04:00
│ ├─ 06:00
│ ├─ 08:00
│ ├─ 10:00
│ ├─ 12:00
│ ├─ 14:00
│ ├─ 16:00
│ ├─ 18:00
│ ├─ 20:00
│ └─ 22:00
├─ Tuesday 14 October
│ ├─ START: 00:00--00:00 part 1, univariate linear regression
│ └─ START: 10:00--12:00 part 1, univariate linear regression
├─ Wednesday 15 October
│ └─ START: 10:00--12:00 part 1, univariate linear regression
├─ Thursday 16 October
│ └─ START: 10:00--12:00 part 1, univariate linear regression
└─ Friday 17 October
  └─ START: 10:00--12:00 part 1, univariate linear regression

roadmap

notice that this roadmap is different from the roadmap for organ-mode.

todos

an X may not mean that the feature is completely implemented but that it is functional for the most part.
  • [-] org-header
    • [ ] priorities
    • [X] todo state
    • [X] tags
    • [X] properties
    • [X] timestamps, scheduling and deadlines
    • [ ] state history
    • [ ] completion status (e.g. completion percentage of children with tasks etc)
  • [ ] org-list
    • [ ] checkboxes
  • [-] agenda
    • [X] repeated tasks
    • [ ] tags
    • [ ] custom views
    • [ ] task hierarchy in agenda tree
    • [ ] state history tracking
  • [ ] markdown
    • [ ] support agenda for markdown
    • [ ] support roam for markdown
  • [X] inline lisp execution
  • [-] commandline
    • [X] conversion
    • [X] roam
    • [X] agenda
    • [ ] advanced conversion with prewritten webapp templates
  • [-] conversion (ideas from org-export)
    • [X] org to html
    • [X] org to latex
    • [ ] org to markdown
    • [ ] markdown to org
  • [ ] latex
    • [ ] recognize latex links (~\ref~)
    • [ ] recognize latex labels (~\label~)
  • [ ] babel
    • [ ] code tangling
    • [ ] code detangling
    • [ ] sessions
    • [ ] data pipelines
    • [ ] library of babel
    • [ ] noweb
  • [-] roam (idea from org-roam)
    • [-] node links
      • [X] links to files
      • [ ] links to headers
      • [ ] links to blocks
  • [ ] org-clock
  • [X] latex previews for html conversion
  • [ ] org-attach
  • [X] transclusions

org-element support

its not called org-element but a text-object in the source code.
org-element parsing highlighting conversion to html conversion to latex
list t t t
table t t t
header t t t
link t t t
timestamp t t
src-block t t t
export-block t t t
block t t t t
prop-drawer t
drawer t t t
latex-env t t t
keyword t
display-math t t t
inline-math t t t
italic t t t
emph t t t
inline-code t t t
comment t t t
comment-block
web-link t t
org-cite
underline
subscript
superscript
strike-through
footnote
dynamic-block
inline-task
paragraph

babel functionality (code execution)

this section is yet to be written, currently only support for running python code is implemented.

how markup is handled

yet to be written. this section will include:
  • what it means for some code to be format-agnostic.
  • how escape sequences are handled.

a lisp-based markup

after having used org-mode for years, i was fed up with its syntax. many people tend to say that org has a saner markup syntax than markdown, but i personally never saw the value in the syntax itself but in the rest of the features that org-mode provides.
in my mind the syntax i wanted had to be customizable, extensible, and similar to lisp code so that it is easy to parse. even better, i wanted the syntax to just be lisp code embedded within arbitrary text, so that lisp code is a second-class citizen, unlike in lisp code files.
after a bit of thinking i put something together that i think makes some sense. this section will highlight some example usage of this markup "language".

text macros and blocks

in my mind, a "block" of text is a portion of text that is specified by a opening and closing "tags". for example, the opening tag for a block in org-mode is #+begin_<type>. with this in mind, we consider the following org-mode text block: 
,#+begin_definition
this is my definition block
,#+end_definition
with the lisp-based markup cltpt provides, the alternative would simply be: 
#(cltpt/base::make-block :type 'definition)
this my definition block
#(cltpt/base::block-end)
both the opening and the closing tags are lisp expressions (which we call "text macros") that are evaluated to construct instances of text-object. block-end is a function that returns a value that the parser recognizes as the signal to close the region that was opened by make-block.
if a text macro isnt closed by another, it is taken to be its own text object without a "contained" region. for example, in the following text the macro is simply replaced with the evaluation result during conversion: 
1-1+1 equals #(+ (- 1 1) 1).

html templates

the lisp markup can serve as a templating engine for html files. we can simply write macros that will return read a file and return it as a text-object that simply stores the text that it read from the file, then during conversion, that text will be substituted inplace of the original macro. for example, we can use the following html file which reads another html file: 
<div class="header">
  #(uiop:read-file-string "header.html")
</div>
although this wouldnt work as expected because by default, the contents of the macro are escaped during conversion, so the read contents will not be interpreted as raw contents but as text to escape (for example < gets replaced with &lt; during html conversion). this is easy to overcome by writing a function that returns a text-object that overrides the conversion functionality to prevent escaping from happening, and using the function for the template: 
<div class="header">
  #(read-template-file-into-text-obj "header.html")
</div>

lexer text macros and post-lexer text macros

to be written. good to atleast note for now that % is used for post-lexer macros while # is used for lexer macros.

org files to webapp

this functionality is not yet available but i will write functionality for converting a bunch of org files into a website. (its already written for my website but i need to integrate it into this codebase.)

using the api

iterating through files

the repo https://github.com/mahmoodsh36/template could serve as a good example for how to work with files as it uses the api to generate this webapp.
parsing a file and using its tree can be done like: 
(let* ((tree (cltpt/base:parse-file cltpt/org-mode:*org-mode* "my/file.org")))
  (cltpt/tree:tree-show tree)
  (cltpt/base:map-text-object
   tree
   (lambda (obj)
     (format t
             "type is ~A, contents are ~A, position in original string is ~A~%"
             (cltpt/base:text-object-begin-in-root obj)
             (cltpt/base:text-object-text obj)
             (type-of obj))))
  tree)
iterating through files and their text-object trees can be done like: 
(defun my-show-nodes (rmr)
  (let* ((file-rules '((:path ("/home/mahmooz/brain/notes/")
                        :glob "*.org"
                        :format "org-mode")))
         (rmr (cltpt/roam:from-files rmr-files)))
    (loop for node in (cltpt/roam:roamer-nodes rmr)
          for this-tree = (cltpt/roam:node-text-obj node)
          do (cltpt/base:map-text-object
              this-tree
              (lambda (obj)
                (format t
                        "title is ~A, id is ~A, type is ~A~%"
                        (cltpt/base:text-object-property obj :title)
                        (cltpt/base:text-object-property obj :id)
                        (type-of obj)))))))

working with the code

this will highlight the main concepts and interfaces in the codebase. note that much of the code will likely change in the (near) future, but the main interfaces discussed here will probably remain the same.

the core interface

the core interface cltpt/base contains a few main CLOS interfaces:
  • text-object
  • text-format

~text-object~ interface

each text object has a "rule" that is passed to the parser. rules are simply trees composed of combinator functions. any class that inherits from text-object needs to have such a rule slot.
a few methods are defined for the text-object interface:
  • text-object-init: this is called immediately after parsing and recognizing the text object.
  • text-object-finalize: this is called once the final text-object tree has been constructed, each object in the tree is finalized starting from the root (an instance of document).
  • text-object-convert: this is called during conversion for each object in the tree, starting from the root. the method returns a plist that determines the behavior of the conversion, this plist is handled by the function convert-tree.

~text-format~ interface

the parser

the parser is based on a "parser combinator" that tries to simplify and abstract away the work of parsing different types of text objects. before creating instances of text-object's the results of parsing are instances of 'match', which is a tree-based data structure (just like text-object), except that its simpler and holds less properties and functionality.
the basis of both text-object and match is a buffer. a buffer is a tree-based structure (buffers nested within buffers) in which each node essentially pinpoints a region of text. the buffer data structure knows how to adapt to incremental/regional changes in the text and is used extensively in the conversion pipeline.
matches are used to construct text-objects, a match may or may not end up being a text-object depending on whether its id property corresponds to a text-object class.
some related functions are:
  • handle-match is a function that takes a match tree and turns it into a text-object tree.
  • cltpt/base:parse calls cltpt/combinator:parse, then on each match returned by the combinator it calls handle-match. then it proceeds to finalize the text-object tree.

the conversion pipeline

to be written.