In the simplest case, your .htm4l files include this file. In m4 there is an include macro that can be used as follows:
include(`/usr/local/www/m4/html-macros')
Notice that m4 uses grave and acute marks to delimit strings. It is normal to use these quotes around every argument to an m4 macro. If you know what you're doing, there are occasions when you may wish to omit them. See your documentation for m4 (if you have any) or follow links from my m4 page.
Once the above file is included in your .htm4l file, you can make use of the macros it defines. You can (and should) mix macro calls and ordinary html freely in the same document. Several examples of macro use can be found on the examples page.