Originally called the Infirmary and Guest House at St. Ann's Academy, the house was built in 1871-72 and was used as an infirmary of the Osage Mission, from 1872 till 1894, by the Jesuits and Sisters of Loretto. Then it served as a guest house for the Passionists. Aside from the St. Francis Church, St. Ann's is the oldest and only remaining structure of the original Osage Mission.

Originally, the infirmary and guest house stood behind the St. Francis Monastery, just west of the church and across the road from St. Ann's Academy. It was built in the same style as the rest of the St. Francis Institute - Jesuit Community complex. In 1912, the house was purchased by John Dowd who moved it on log rollers (about 1/2 mile) from its original location to 325 Main Street. This was done to make room for a new Passionist monastery next to the church.St. Ann's B & B

The property is in a remarkably original form. Only the front porch and a small kitchen attachment have been added on after the structure's move of 1912. The house is said to be a very early example of a rare two-story mansard-roof building.