In 1838, Joseph Goodrich, originally from the "Burned-over District" of New York, left for the West. After researching transportation routes, Goodrich filed a claim, speculating that his land would become a transportation hub. Upon arriving, Goodrich moved an 1837 log cabin from Lima Township to his property, thereby founding Milton. Goodrich expanded the cabin to create the original Milton Inn. Goodrich soon needed more space and began to erect the Milton House in 1844.

Goodrich designed a hexagonal tower and a long wing that extended from it; creating arguably the first concrete building in the United States. He constructed it with a mixture of sand, lime cement and gravel in a wet mixture, which he poured into wooden forms to create a fireproof material. The Milton House was the source and inspiration for Orson Squire Fowler's The Octagon House: A Home for All, which sparked the octagon fad of the mid-nineteenth century.

The cabin remained behind the new Milton House and served as a kitchen, while Goodrich rented rooms in the new structure to travelers. A tunnel connects the cabin to the Milton House. Originally dug from the earth and lined in limestone, it was just large enough for a person to crawl through. Based on oral accounts and the vocal objections of Joseph Goodrich to slavery, historians believe that this tunnel was a stop on the Underground Railroad, which delivered many slaves to freedom. Runaway slaves used the tunnel to move from the cabin to a more secure hiding place in the basement of the Milton House.

The Milton House is a museum and is open to the public. It was designated a National Historic Landmark in 1998.