The largest building on the site, towering above all the others, is the Faktorei, also known as the Herrenhaus (Mansion House).

The central section of the Faktorei as we see it today was built around 1560 by the Uthmann family. After it was sold to Prince Elector Augustus of Saxony in 1567, various rooms were used by the electoral family and by the Kurfürstliche Hofmeisterin, Sabine von Wedach.

The mansion was further extended on two occasions. Two wings were added in 1586 by Elector Christian I and in 1628. The Faktorei as we now know it was finalised in 1803.  

The Faktorei was used for administrative purposes, first under the Prince Electors, then by the private company Sächsische Kupfer- und Messingwerke Franz Adolph Lange, and finally by the state-owned VEB Blechwalzwerk rolling mill. After it was damaged by fire in 1964, it was converted into 18 separate apartments.

On the ground floor at the west gable end is a well-preserved room with beautiful cross vaulting.