In the early part of the Twentieth Century, armories were being constructed across the country for use by the National Guard, which was created at the end of the previous century to maintain civil control and contribute to war efforts. The Whitefish Bay National Guard Armory was built as part of this national trend. The Armory became the headquarters of the First Light Battery, later known as the 121st Field Artillery. This unit participated in six major campaigns during World War I, including Aisne-Marne, Alsace, Champaign, Lorraine, Meuse-Argonne, and Oisne-Aisne. During World War II, it served in both the Pacific and Europe. As part of its mission to serve in times of civil unrest, the unit was called to duty during the 1967 Milwaukee Riots, 1977 employee strike at Taycheedah Correctional Facility, and the 1977 and 1981 firemen's strikes in Milwaukee.

Martin Tullgren & Sons, a Milwaukee architecture firm, designed the Whitefish Bay National Guard Armory. Construction began in 1928 and was completed in 1929. This Late Gothic Revival/Collegiate Gothic building is the oldest existing National Guard facility in Milwaukee County. The first floor consists of a lounge, rest room, boiler room, work room, kitchen, and gymnasium. Offices, rest rooms, and a gym balcony, which was converted into additional office and storage space in 1968, comprise the second floor. A "radio room" occupies the third floor tower. Colonel Harry C. Hengles, who served as the State Military Architect and Engineer during World War I, designed an addition to the Armory that includes a locker room/classroom and vehicle storage space. The addition, which was completed in 1941, was funded, in part, by the Works Progress Administration, a federal agency created in 1935 to employ workers during the Great Depression. While the Armory has been used primarily as a military facility, during peacetime it has also served as a community center, recreational facility, and library.

Historically, the exterior appearance of an armory was intended to express its function as a military facility. For this reason, many armories were built in the castellated style, which was derived from medieval fortresses and features towers, battlements, and crenallated parapets. The Whitefish Bay National Guard Armory is no exception. It has also been commonly described as Late Gothic Revival or Collegiate Gothic due to its rectangular massing, symmetrical fenestration, keep-like entrance, and stone ornamentation.

The building was razed in 2004. The property was removed from the National Register on 4/6/2011.