Kandy is the last royal capital of Sri Lanka and it carries that weight well. The city is built around the lake and the temple, and both reward time spent. It's more urban than Ella, more cultural than the coast, and genuinely different from anywhere else on the island.
The city centre is busy and traffic is chaotic. Most guests stay in the hills above the lake where guesthouses have views and the noise drops away. The difference between a lakeside hill guesthouse and a city-centre hotel is significant - worth paying slightly more to be above it all.
The Esala Perahera festival in July-August is one of Asia's great spectacles - elephants, fire dancers, thousands of drummers. If your dates overlap, organise accommodation months in advance. If they don't, Kandy in any month is worth two nights.