I've done some more experiments and come with an interesting solution: convert MathML to HTML and render it using CSS. I've used this project as the basis for CSS: https://github.com/fred-wang/mathml.css/

Not everything is good. Fractions look OK, ordinary inline math too, but for example integrals are completely wrong. I've tried to set fence sizes using some simple heuristics, I am not sure how well it works for a more complicated stuff.

Anyway, sample.tex contains some dummy examples, sample.mk4 code for make4ht that does the conversion, and mathml.css is a modified version of Fred Wang's CSS file.

sample.tex can be compiled using the following command:

make4ht -m draft sample.tex "mathml"

(file #366, file #367, file #368)

I've modified the CSS code to produce a better result - the fractions are now vertically aligned and font sizes are better:

------
\Preamble{xhtml}
\catcode`\:=11
\def\IgnoreRule{\ht:special{t4ht@\string_}}
\def\EndIgnoreRule{\ht:special{t4ht@\string_\string_}}
\def\a:mathml{}
% \Configure{over}
% {\Send{GROUP}{0}
% {<\a:mathml mfrac><\a:mathml mrow>}\HCode{</\a:mathml mrow>}}
% {\HCode{<\a:mathml mrow>}\Send{EndGROUP}{0}
% {</\a:mathml mrow></\a:mathml mfrac>}}
\Configure{over}
{\Send{GROUP}{0}{<span class="fraction"><span class="top">}\IgnoreRule\HCode{</span>}}
{\HCode{<span class="bottom">}\EndIgnoreRule\Send{EndGROUP}{0}{</span></span>}}

\Css{
.fraction {
display: inline-flex;
flex-direction: column;
padding: 0 2px;
align-items: center;
font-size:70\%; % to get correct box size
vertical-align: 50\%;
}}

% reset font size, it will be downsized to 70% by HT fonts
\Css{.fraction span{font-size:1rem;}}

\Css{.top {
border-bottom: 2px solid grey;
}}

\Configure{$}{\HCode{<span class="math-inline">}\DviMath}{\EndDviMath\HCode{</span>}}{}
\catcode`\:=12
\begin{document}
\EndPreamble
-----------

We will definitely add this to TeX4ht sources once we find a good CSS that works in all cases. We may use this method also for \frac and other more complicated formatting that uses images at the moment. It should result in much better inline math than in the current state, where text and images are used. We may even try to use it for display math, where we use images by default.

The simplest thing you can do is to convert your whole inline math to image, using for example:

httex filename "xhtml,html5,pic-m"

Also, it works in MathML:

htttex filename "xhtml,html5,mathml,mathjax"

As it works in MathML, it is obvious that it is possible to configure \over to produce sensible output in plain HTML. Here is a sample HTML document:

\documentclass{article}
\begin{document}
${a+2 \over b} $

$ 1 + \begingroup a \over b \endgroup + 2$

$ 1 + \bgroup a \over b \egroup + 2$

$ 1 + { a \over b \egroup + 2$

$ 1 + \bgroup a \over b } + 2$
\end{document}

And here is a config file:

\Preamble{xhtml}
\catcode`\:=11
\def\IgnoreRule{\ht:special{t4ht@\string_}}
\def\EndIgnoreRule{\ht:special{t4ht@\string_\string_}}

\Configure{over}
{\Send{GROUP}{0}{<span class="fraction"><span class="top">}\IgnoreRule\HCode{</span>}}
{\HCode{<span class="bottom">}\EndIgnoreRule\Send{EndGROUP}{0}{</span></span>}}

\Css{
.fraction {
display: inline-flex;
flex-direction: column;
padding: 0 2px;
align-items: center;
font-size:130\%;
}}
\Css{.top {
border-bottom: 2px solid grey;
}}

\Configure{$}{\DviMath}{\EndDviMath}{}
\catcode`\:=12
\begin{document}
\EndPreamble

It uses CSS flex box to render fractions.

I don't really understand how \Send command works, as it is not really documented anywhere in TeX4ht sources. But it can be used in these cases to insert HTML code at beginning and end of the current group. It works only in \DviMath mode, which is why the following configuration is necessary:

\Configure{$}{\DviMath}{\EndDviMath}{}

I don't know if it will have any side effects on other math configurations, hopefully not.

Compile the example using

make4ht -c config.cfg sample.tex