Records on Castle Hill date back to 1636, when it was part of Derby Farm in South Salem. For a long period, an elegant summer home was situated on top of the hill. According to Fred Gannon's "Nicknames and Neighborhoods" the origin of the name is obscure. Colonists at one time had a watch tower on the summit as a precaution against attacks by Indians. Later, Salem merchants had summer homes on the hill. In the later part of the 1800's, many French Canadians, mostly from Quebec who found work in the shoe shops and cotton mills, settled in this part of Salem. In more recent years, the ledge was blasted to make stones for streets.
- Nicknames and Neighborhoods, by Fred Gannon.
- Vertical File in Salem Collection - Castle Hill
- "Changes in Salem Depicted by Old Time Photographs" Salem Evening News, March 6, 1926. p.16.