I have not noticed a significant impact to frames. Mine, inside and outside of the city, have only dropped below 60 a couple of times while I have been on the server (and I've been on for several hours now). There are a few "desync" issues where the server is freezing for a few seconds, however I have already made fixes to these in the soon™ to be released update.
For reference, I'm only running on a i7 4770k, GTX 970 and 16GB of RAM.