Brazil is far from the economic center of the Internet. South Korea, a country with 1/4 of the population of Brazil, is probably better connected because it's on the route to Japan, China, etc.
Brazil's poor latency may be partly explained by the fact that a lot of their connections have to go through the US and Canada:
"The shape of the Internet varies considerably when examining individual regions. In Europe, the majority of cross-border capacity is between European countries, while in Latin America and the Caribbean the majority of international capacity is connected to the U.S. & Canada."[1]
"More than 80% of Africa's and Latin America's international Internet bandwidth connect to cities outside their regions. In the case of Latin America, 60% of the region's interregional capacity connects to a single city, Miami."[2]
Also, because of Brazil's distance from the economic center of the Internet, they probably don't get the advantage of unpaid peering relationships.
