Both goals for Latvia were scored by Rūdolfs Balcers.
While the Latvian team played the first of seven group stage games, the Swiss kicked off the tournament on Friday with a victory over the defending champions, the United States.
Latvia's next game is on May 17 against Germany.