The suburb takes its name from the natural feature of Lavender Bay, on Port Jackson (Sydney Harbour), to the west of the Sydney Harbour Bridge. It lies between Milsons Point and McMahons Point and the suburb of North Sydney is located, to the north. Lavender Bay is a residential suburb with expansive views of Sydney Harbour.