Say to him/her that you would like to go, but that you have got too much to do this weekend. Perhaps another time.
Try to use 可 and 以后再。
星期天我的事情可多了，我们以后再找时间吧？
[Comments]
Click on the Chinese to hear the sentences spoken.
Alternatively one can say 星期天我的事情很多,我们以后再找机会去那儿吧。