The worst thing to do when learning a language is to be hesitant to speak. I've had students who whisper, or only give one word answers to questions. I understand how hard it is to learn a language. When I moved to Russia for a year, the first few months I avoided situations in which I'd have to speak in front of people or ask people for help in stores. I felt judged and nervous. However, the more times I was forced to speak, the more I realized no one was judging me! I found people were all happy I was trying to speak Russian. They didn't care about my incorrect grammar or heavy accent. They were just interested to meet me and learn about why I was learning their language. This made me more confident and inspired me to speak even more. The best way to improve is to not hesitate, and just speak as often as you can.
For those learning English I want to say- do not worry! All native English speakers have a lot of experience speaking with non-native speakers! Two of my closest friends are from Turkey and Hungary. They have accents, but they are not afraid to speak, loudly and often!
I find here in America we are very interested in people's backgrounds and respect the effort it takes to learn English. Americans are happy to help with any language issues, for we all have immigrant friends and most of us are learning another language ourselves!