The Bible says it's better to conquer yourself than an entire city.
Maybe you have large ambitions. Are your appetites under control?
Maybe you dream of freedom for others. Are you free?
So, yes, worry about the town, the ministry, the business. But make you sure have you have yourself harnessed. God has first given you responsibility for you.