Do I always have to carry my passport or residence card with me?（旅券や在留カードは必ず携帯しなければいけないの？）

Yes. You must carry your passport or residence card (if you already have had it issued) whenever you go out.
This is one of your obligations set forth in Japanese law, and you also have to present any of these IDs when asked by the police to do so.
Also note that you will be punished if you engage in an activity other than those permitted under your status of residence, or if you stay in Japan beyond your authorized period of stay.
Stick to the rules and regulations for your easy, comfortable life in Japan.