No, you don't have to. The problem with caption package is that it changes lot of things, so it caused Tex4ht to fail until recently. It is still probably best to not use it with TeX4ht, unless you use specific commands provided by it.

I will look at this issue, but it is quite hard to debug

Indeed, with the minimal example I had attached (bug.tex) the TOC appears to be correct.

Further investigation shows that the bad TOC entries for tables are caused by some interaction with the caption package (version 2020/01/03 v3.4h). I am going to attach a new minimal example.

If I replace the caption package by the one that was shipped with TeX Live 2019 (version 2018/10/06 v3.3-154), the TOC entries are fine.

(file #348)

Yes, now the longtable entries in the TOC look like the section entries, i.e. the link does not include the table's number.

But as I said yesterday, with TeX Live 2020 the TOC entries for normal tables are broken as well. Not sure if that is a regression in tex4ht itself, or caused by some other change from TeX Live 2019 to 2020.

Update: With TeX Live 2020, the table's number is included inside of the link also for normal tables.

In other words, it is now consistent within the list of tables, but it is inconsistent with the table of contents.

In the list of tables, entries for longtables include the table's number inside of the link, whereas entries for normal tables (or for figures) don't include it.

The generated HTML for a table entry looks like this:

<span class="lotToc" >1&#x00A0;<a
href="#x1-10011">A table</a></span><br />

Whereas for a longtable it is:

<span class="lotToc" >&#x00A0;<a
href="#x1-1003">2 A longtable</a></span>

I am attaching a minimal example to reproduce the bug in bug.tex and the output generated with "mk4ht xhlatex bug" as bug.html.