It sounds weird but usually is understood without much problem. However in some languages there are cases where different words or meanings are differentiated by gender and so those can be confusing. In my native German for instance, the word see when masculine mean lake, and when feminine means sea, so misgendering a word like that can produce unexpected meanings. Although we often can figure it out from context and your foreign accent.