This article describes the process of the development of Madrasah Jam'iyah Mahmudiyah to become a modern Islamic educational institution in Langkat in 1921-1950. The results of this study indicate that the Madrasah Jam'iyah Mahmudiyah experienced rapid progress throughout the years 1912-1942. With the number of students reaching 2,000 in 1930, the Madrasah Jam'iyah Mahmudiyah was able to compete with formal educational institutions established by Dutch colonial governments, such as Langkatsche School, Europese Lagere School (ELS) and Holland Chinese School (HCS). The progress of Madrasah Jam'iyah Mahmudiyah is inseparable from the role of Sultan Abdul Aziz who is very concerned about the progress of Islamic education in the area of the Langkat Sultanate. However, the condition of the madrasa began to decline since 1942, when Japan took over the authority of the Dutch. Even in 1950, this madrasah experienced a fire that consumed all classrooms. The fire was triggered by the explosion of weapons ammunition belonging to the Indonesian National Army which was stored in one of the classrooms.