I think that I've fixed this issue finally. We need to make the problematic commands robust. So for the future reference, instead of

\def\:temp#1{...}
\HLet\foo\:temp

it is better to use the following code for the commands that can be used in captions or sectioning commands:

\ProvideDocumentCommand\foo:temp{m}{...}
\HLet\foo\foo:temp

I think one would never want any \caption contents expanded when writing it to a file? Can't quite imagine when ...

Thank you, Michal. Will try the \protect thing. If that works, it may be worthwhile to try and add some logic to do that automatically. I'm actually using tex4ebook, but it uses tex4ht internally, and that's where I ran into this issue.

Recently when trying to convert a scientific paper to an eBook using tex4ht, I ran into some confusing errors. I tracked it down to a macro that used \left and \right inside of the caption of a labeled figure, which only seems to be a problem with hyperref included. A simple example to reproduce this:

\documentclass{article}

\usepackage{hyperref}


\begin{document}
\begin{figure}
\caption{$\left(A\right)$}
  \label{a}
\end{figure}
\end{document}

When compiled with `mk4ht htlatex main2.tex`, this gives:

--- file main2.css ---
 [1

] [2

]
LaTeX Font Info:    External font `cmex10' loaded for size
(Font)              <7> on input line 8.
LaTeX Font Info:    External font `cmex10' loaded for size
(Font)              <5> on input line 8.
l. 8 Writing main2.idv[1] (main20x.png)

! TeX capacity exceeded, sorry [input stack size=20000].
\Picture ->\leavevmode
                       \futurelet \:temp \:img
l.9   \label{a}

If you really absolutely need more capacity,
you can ask a wizard to enlarge me.


Here is how much of TeX's memory you used:
 7621 strings out of 492617
 101793 string characters out of 6129507
 880460 words of memory out of 5000000
 11452 multiletter control sequences out of 15000+600000
 3808 words of font info for 15 fonts, out of 8000000 for 9000
 1141 hyphenation exceptions out of 8191
 20000i,5n,11428p,758b,465s stack positions out of 20000i,500n,20000p,200000b,80000s
Output written on main2.dvi (2 pages, 9752 bytes).

Interestingly, a small change to \thefigure results in a completely different error:

\documentclass{article}

\usepackage{hyperref}


\begin{document}
\renewcommand{\thefigure}{S\arabic{figure}}
\begin{figure}
\caption{$\left(A\right)$}
  \label{a}
\end{figure}
\end{document}

Now gives:

(./main2.aux) [1] [2]
! You can't use `\relax' after \the.
<recently read> \c@

l.10   \label{a}

?

This only seems to happen with the very specific combination of factors I have outlined above. Not including \left and \right, not having them inside a \caption, not having the \label, or not including \hyperref all seem to make this go away, and it compiles with regular LaTeX just fine. I'm afraid I don't know enough about the macros you're using to be able to debug this.