The Madison Candy Company,established in 1899,manufactured a variety of candy, but was best know for its chocolate crèmes. They also sold crackers, cigars, cheese, and nuts. The company marketed its products throughout Wisconsin and later in Illinois as well.

Until this building was constructed in 1903, the factory was located at 623 Williamson Street. Architect and engineer John Nader designed the new building. The factory's new location was closer to the rail lines in an area emerging as a center of manufacturing and commerce. In the late nineteenth century, there was very little manufacturing in Madison due to a lack of skilled labor and the high freight rates. However, as Madison expanded, Madison's political and civic leaders pushed for the development of industry. Within this development, Joseph E. Kleiver formed The Madison Candy Company.

The factory ceased operation in 1927 and the building stayed vacant for several years until the Wisconsin Farm Bureau occupied it. It later served as the home of the Ela Industrial Supply Company, which occupied the building until the 1990s. In 1999 the building was rehabilitated for use as a restaurant and offices using historic preservation income tax credits.