The project will be funded by the Welsh Assembly's Heritage Tourism grant which has earmarked a total of £19m towards similar projects across Wales.
Work on Oystermouth Castle will include improving access to the Norman stone castle and to assure the long-term sustainability of the site.
To encourage more people to visit the castle there are also plans to launch an events programme.
A castle community co-ordinator will be appointed to manage a range of events, engage with community groups. The Alina's Chapel on the site will also be developed into a multi-functional space. The castle currently receives 5,000 visitors a year and the target is to increase this to 25,000 a year by 2014.
Minister for Heritage, Alun Ffred Jones said: "Tourism is an important part of our economy and this project will make a positive contribution to attracting visitors to Swansea.
"The project also aims to ensure the castle is accessible and relevant to the people of Swansea."