Cove were formed in 1922 and played in the local amateur leagues until 1985, when they became a junior team. However, the club applied and were accepted to the senior Highland Football League in 1986, where they have played ever since. They have also racked up a formidable reputation in the lower rounds of the Scottish Cup, reaching the 3rd round three times. In 2001 they won their first Highland League championship – but even more impressively won the ‘quadruple’ of Highland League, Aberdeenshire Shield, Scottish Qualifying Cup, and Highland League Cup.
In the 2007–08 season, Cove reached round four of the Scottish Cup for the first and only time, beating Brora Rangers away 7–0, Keith 3–0 and Edinburgh University 1–0 on the way, before losing 4–2 to Ross County.
However, that did not deter the south Aberdeen side from claiming their second ever Highland League Championship with a 3–0 home win against Lossiemouth on 3 May 2008.
Cove Rangers were one of the clubs preparing an application for entry into the Scottish Football League following Gretna relinquishing their league status on 3 June 2008. The other clubs were Annan Athletic, Preston Athletic and Spartans. Annan Athletic were eventually awarded the place, due to their superior facilities.
The club have started plans for a new stadium in preparation for a possible place in Scottish League Two.
In 2016 the club regained their Highland League crown in one of the greatest teams achievements in recent memory. Due to the introduction of the ‘pyramid system’, Cove were forced to say goodbye to their Allan Park home due to the stadium not meeting SPFL minimum stadia requirements. This in turn meant the club played all of their games at an away ground with Inverurie Locos, Formartine United and Banks O’Dee sportingly offering the use of their facilities so the club could compete. Despite this Cove went on to win the league and advance to the first play-off game against the winner of the Lowland League, Edinburgh City. The tie was won by City 4–1 on aggregate and progressed to Scottish League Two after beating East Stirlingshire in the second playoff tie.
One of the club’s most famous players is the young Scottish midfielder Paul Coutts who joined English League One club Peterborough United during the summer of 2008 for a nominal fee. Coutts won the clubs’s Young Player of the Year award for the 2008–09 season  and won eight caps for the Scotland under-21 squad, after being touting for a call up by his former manager Darren Ferguson.