Only inviting the closest people.

We're at the beginning stage of planning out wedding. Initially we wanted to elope as my Fiancé has quite severe social anxiety and I've never really thought I will get married till I met him and the idea of the big wedding isn't really US. Elopment sounded perfect but then after a little bit if thinking we decided we wanted just our clise family with us. This would include his parents and brother,my brother and Dad and my best friend ,her husband and 2 kids. Problem is with other people who won't be invited. I've been pretty much to all my friends weddings but the thing is I live in HK for the last 14 years and only really kept close friendship with my best friend who will be my Maid of Honour. Some people heard we've started to form guest list and reavh out asking if they will be on. They won't but I need to find a way to tell them politely. Any advice? I really want to just do what we feel happy and comfortable and not bend to other people wishes but don't know how not to offend others.