In 1868, Carrie Martin and Elizabeth Jefferson formed the Ladies Literary Association to promote literacy and supply scholarly works. Jefferson offered her home as a meeting place for the association. Financed by annual dues, the organization purchased books and subscribed to many newspapers and periodicals. The Hudson Public Library was established under state law in 1903 and constructed a building the next year with a Carnegie grant. Upon opening the Ladies Literary Association disbanded and donated their collection to the library.

Noted Milwaukee architects VanRyn & DeGelleke designed the NeoClassical structure. A Bedford limestone parapet and cornice cap its flat roof. Fluted Doric columns support the projecting entry portico ornamented with a frieze and triglyphs derived from ancient Greek architecture. The double entrance doors are topped with a transom divided into three square lights with muntins that radiate from a central point. Rounded medallions featuring open books surrounded by wreaths of leaves are placed on either side of the portico.

The interior includes an entablature shelf, lintels, door frames, the circulation desk, and delicate spindle stair railing crafted by local artisan John A. Lee.

The library is still in operation and is open during regular hours.