cltpt

note that this website was generated using cltpt from org-mode files. so this page/webapp can be a testament to whether org->html conversion works properly (broken text elements is an indicator of parser/conversion failure).

introduction

cltpt is both a tool and a library that is meant to serve as a base for the upcoming organ-mode package for the lem text editor. it is very much a WIP, it is unstable and breaking changes may be introduced without warnings.
the goal of cltpt is to maintain an editor-independent codebase and toolset (the existence of countless parsers for org-mode should be enough a motive for this). the core will be written in common lisp but will be independent of lem's libraries, but ofcourse this is supposed to (mainly) be a package for lem so some parts are bound to be lem-dependent. although lem itself can also be used as a library in common lisp, i think that this is still a good idea.

installation

nixos

if you are on nixos you may use the provided flake. running the commandline tool is as simple as something like:

  nix run github:mahmoodsh36/cltpt#cltpt -- -h

quicklisp

since the dependencies are all listed in cltpt.asd, cloning the code and then running (ql:quickload :cltpt) in the code directory should be enough to load the library.
to make this perist you may clone the repo into ~/.quicklisp/local-projects/. then you may overwrite main.lisp (which uses asdf directly without quicklisp) to make it work with quicklisp:

  (ql:quickload :cltpt)
  (cltpt/zoo:init)
  (cltpt/commandline:commandline-main (uiop:command-line-arguments))

then you may use the commandline like: run.sh -h. (in the following sections you may replace cltpt with run.sh and the commands will work.)

example commandline usage

this section will highlight what i think would be the most common usage for the commandline tool.

converting from org-mode to html

convert a single file from org to html (the % part is for "formatting" with lisp code):

  # this command writes the file to /tmp/test.html and the static files it links to to ~/tmp/~ aswell.
  cltpt convert\
        -f 'test.org'\
        -d html\
        -o '/tmp/%(getf *file-info* :filename-no-ext).html'\
        -c '/tmp/%(getf *file-info* :filename)

convert a bunch of files from org to html:

  cltpt convert\
        -r '(:path "/home/<user>/dir1/" :glob "*.org" :format "org-mode")'\
        -r '(:path "/home/<user>/dir2/" :glob "*.org" :format "org-mode")'\
        -r '(:path "/home/<user>/dir3/file.org" :glob "*.org" :format "org-mode")'\
        -d html\
        -o '/tmp/%(getf *file-info* :filename-no-ext).html'

notice that you can stack multiple instances of the -r (or --rule) argument. this is also possible with the -f argument (which is used for single files):

  cltpt convert\
        -f ~/notes/file1.org\
        -f ~/notes/file2.org\
        -d html\
        -o '/tmp/%(getf *file-info* :filename-no-ext).html'

this would generate two files, /tmp/file1.html and /tmp/file2.html.

converting from org-mode to latex

notice that this is very similar to the previous commands, we slight difference in the arguments passed to the tool, such as the name of the destination format (the format we want to convert our files to).
convert a single file from org to latex (write it to /tmp/test.tex):

  cltpt convert\
        -f 'test.org'\
        -d latex\
        -o '/tmp/%(getf *file-info* :filename-no-ext).tex'

convert a bunch of files from org to latex:

  cltpt convert\
        -r '(:path "/home/<user>/notes/" :glob "*.org" :format "org-mode")'\
        -d latex\
        -o '/tmp/%(getf *file-info* :filename-no-ext).tex'

"roam" queries

the library provides an "org-roam-like" interface that works with different methods of identifying different types of objects in your text files. by default it recognizes org-id "identifiers" and links (the ones org-roam use for headers/files). denote-like identifiers and (more importantly) blk-like identifiers which includes the previous ones.
the following command queries a directory of org-mode files and prints the title, id and filepath for each one in the specified format (-o argument).

  cltpt roam\
        -r "(:path \"$ORG_DIR\" :glob \"*.org\" :format \"org-mode\")"\
        -o 'title: %title, id: %id, file: %file'

title: my doc, id: NIL, file: /home/mahmooz/work/cltpt/tests/test.org
title: header my secondary header, id: NIL, file: /home/mahmooz/work/cltpt/tests/test.org
title: send the professor a mail, id: NIL, file: /home/mahmooz/work/cltpt/tests/test.org
title: NIL, id: def-vector, file: /home/mahmooz/work/cltpt/tests/test.org
title: do something else, id: NIL, file: /home/mahmooz/work/cltpt/tests/test.org
title: header do something, id: NIL, file: /home/mahmooz/work/cltpt/tests/test.org
title: another due task, id: NIL, file: /home/mahmooz/work/cltpt/tests/test.org
title: due task, id: NIL, file: /home/mahmooz/work/cltpt/tests/test.org
title: NIL, id: def-ac-standard, file: /home/mahmooz/work/cltpt/tests/test.org
title: NIL, id: test-name, file: /home/mahmooz/work/cltpt/tests/test.org
title: NIL, id: NIL, file: /home/mahmooz/work/cltpt/tests/test2.org
title: some task, id: NIL, file: /home/mahmooz/work/cltpt/tests/test2.org
title: part 1, univariate linear regression, id: NIL, file: /home/mahmooz/work/cltpt/tests/test2.org

"agenda" queries

the following command prints an agenda "tree" after retrieving all TODOs from the org files found in the specified directory. the default range displayed is 7 days from now.

  cltpt agenda\
        -r "(:path \"$ORG_DIR\" :glob \"*.org\" :format \"org-mode\")"

├─ Tuesday 21 October
│ ├─ 00:00
│ ├─ 02:00
│ ├─ 04:00
│ ├─ 06:00
│ ├─ 08:00
│ ├─ 10:00
│ ├─ 12:00
│ ├─ 14:00
│ ├─ 16:00
│ ├─ 18:00
│ ├─ 20:00
│ └─ 22:00
├─ Wednesday 22 October
├─ Thursday 23 October
├─ Friday 24 October
├─ Saturday 25 October
├─ Sunday 26 October
└─ Monday 27 October

to print an agenda tree for a different range of timestamps, we use:

  cltpt agenda\
        -r "(:path \"$ORG_DIR\" :glob \"*.org\" :format \"org-mode\")"\
        -f '2025-10-13'\
        -t '2025-10-18'

├─ Monday 13 October
│ ├─ 00:00
│ ├─ 02:00
│ ├─ 04:00
│ ├─ 06:00
│ ├─ 08:00
│ ├─ 10:00
│ ├─ 12:00
│ ├─ 14:00
│ ├─ 16:00
│ ├─ 18:00
│ ├─ 20:00
│ └─ 22:00
├─ Tuesday 14 October
│ ├─ START: 00:00--00:00 part 1, univariate linear regression
│ └─ START: 10:00--12:00 part 1, univariate linear regression
├─ Wednesday 15 October
│ └─ START: 10:00--12:00 part 1, univariate linear regression
├─ Thursday 16 October
│ └─ START: 10:00--12:00 part 1, univariate linear regression
└─ Friday 17 October
  └─ START: 10:00--12:00 part 1, univariate linear regression

commandline format args

roadmap

notice that this roadmap is different from the roadmap for organ-mode.

todos

an X may not mean that the feature is completely implemented but that it is functional for the most part.

[-] org-header
- [ ] priorities
- [X] todo state
- [X] tags
- [X] properties
- [X] timestamps, scheduling and deadlines
- [ ] state history
- [ ] completion status (e.g. completion percentage of children with tasks etc)
[ ] org-list
- [ ] checkboxes
[-] agenda
- [X] repeated tasks
- [ ] tags
- [ ] custom views
- [ ] task hierarchy in agenda tree
- [ ] state history tracking
[ ] markdown
- [ ] support agenda for markdown
- [ ] support roam for markdown
[X] inline lisp execution
[-] commandline
- [X] conversion
- [X] roam
- [X] agenda
- [ ] advanced conversion with prewritten webapp templates
[-] conversion (ideas from org-export)
- [X] org to html
- [X] org to latex
- [ ] org to markdown
- [ ] markdown to org
[ ] latex
- [ ] recognize latex links (\ref)
- [ ] recognize latex labels (\label)
[ ] babel
- [ ] code tangling
- [ ] code detangling
- [ ] sessions
- [ ] data pipelines
- [ ] library of babel
- [ ] noweb
[ ] roam (idea from org-roam)
- [ ] node links
  - [ ] links to files
  - [ ] links to headers
  - [ ] links to blocks
[ ] org-clock
[X] latex previews for html conversion
[ ] org-attach

org-element support

its not called org-element but a text-object in the source code.

org-element	parsing	highlighting	conversion to html	conversion to latex
list	t
table	t		t	t
header	t	t	t
link	t		t	t
timestamp	t	t
src-block	t
export-block	t		t	t
block	t	t	t	t
prop-drawer	t
drawer	t
latex-env	t		t	t
keyword	t
display-math	t		t	t
inline-math	t		t	t
italic	t		t	t
emph	t		t	t
inline-code	t		t	t
comment	t		t	t
web-link	t		t
org-cite

using the api

iterating through files

the repo https://github.com/mahmoodsh36/template could serve as a good example for how to work with files as it uses the api to generate this webapp.
parsing a file and using its tree can be done like:

  (let* ((tree (cltpt/base:parse-file cltpt/org-mode:*org-mode* "my/file.org")))
    (cltpt/tree:tree-show tree)
    (cltpt/base:map-text-object
     tree
     (lambda (obj)
       (format t
               "type is ~A, contents are ~A, position in original string is ~A~%"
               (cltpt/base:text-object-begin-in-root obj)
               (cltpt/base:text-object-text obj)
               (type-of obj))))
    tree)

iterating through files and their text-object trees can be done like:

  (defun my-show-nodes (rmr)
    (let* ((file-rules '((:path ("/home/mahmooz/brain/notes/")
                          :glob "*.org"
                          :format "org-mode")))
           (rmr (cltpt/roam:from-files rmr-files)))
      (loop for node in (cltpt/roam:roamer-nodes rmr)
            for this-tree = (cltpt/roam:node-text-obj node)
            do (cltpt/base:map-text-object
                this-tree
                (lambda (obj)
                  (format t
                          "title is ~A, id is ~A, type is ~A~%"
                          (cltpt/base:text-object-property obj :title)
                          (cltpt/base:text-object-property obj :id)
                          (type-of obj)))))))

working with the code

this will highlight the main concepts and interfaces in the codebase. notice that much of the code will likely change in the (near) future, but the main interfaces discussed here will probably remain the same.

the core interface

the core interface cltpt/base contains a few main CLOS interfaces:

text-object
text-format

~text-object~ interface

each text object has a "rule" that is passed to the parser. rules are simply trees of composed of combinator functions. any class that inherits from text-object needs to have such a rule slot.
a few methods are defined for the text-object interface:

text-object-init: this is called immediately after parsing and recognizing the text object.
text-object-finalize: this is called once the final text-object tree has been constructed, each object in the tree is finalized starting from the root (an instance of document).
text-object-convert: this is called during conversion for each object in the tree, starting from the root. the method returns a plist that determines the behavior of the conversion, this plist is handled by the function convert-tree.

~text-format~ interface

the parser

the parser based on a "parser combinator" that tries to simplify and abstract away the work of parsing different types of text objects. before creating instances of text-object's the results of parsing are merely plists of the form:

  (:begin <int>
   :end <int>
   :id <symbol>)

these are called "matches" which are a result of applying parsing rules defined for text objects to a string. these matches are then used to construct the text object tree.

handle-match is a function that takes a match tree and turns it into a text-object tree.
cltpt/base:parse calls cltpt/combinator:parse, then on each match returned by the combinator it calls handle-match. then it proceeds to finalize the text-object tree.