Configure option for doxygen

So I’m building some Ruby releases I don’t want it to build any docs
for.

When running configure, I already add --disable-install-doc.

However, since I have doxygen, configure notices that and adds it to be
built. I normally just quick edit the resulting Makefile and remove
doxygen.

However, is there a way to tell configure to --disable-doxygen or
something similar?