- Records on Castle Hill date back to 1636, when it was part of Derby Farm in South Salem. Colonists at one time had a watch tower on the summit as a precaution against attacks by Indians.
- According to Fred Gannon's "Nicknames and Neighborhoods" the origin of the name is obscure. In the 1700's, some Salem merchants had summer homes on the hill. One of these had the look of a castle, so people nicknamed the area "Castle Hill."
- Later, the ledge was blasted to make stones for streets.
- In the later part of the 1800's, many French Canadians, mostly from Quebec who found work in the shoe shops and cotton mills, settled in this part of Salem. St. Anne's Church was an important part of this neighborhood, even hosting an Athletic Club in their basement during the 1930's until as recent as 1982.
- Nicknames and neighborhoods and album of pictures of Old Salem by Fred Gannon, p.7
- Salem's Great Pasture: A History of the Southeast Extension of Jefferson Avenue to Loring Avenue by D. Michel Michaud, 1992.
- Vertical File in Salem Collection - Castle Hill
- "Changes in Salem Depicted by Old Time Photographs" Salem Evening News, March 6, 1926. p.16.
- History of French Canadians in Castle Hill Salem by Anthony V. Salvo