After installing the Postgres on my machine I had some issues regarding it’s connections through sockets. So after a long time of trying to configure correctly I’ve finally gave up and started from scratch. Here is a good post of how to install it (in russian): http://lug.nsk.ru/lugnskru/2010/10/ustanovka-postgresql-9-pgadmin-iii-v-ubuntu-10-04.html. So the steps were the following (for those who don’t know russian).
add-apt-repository tool will be used on the next step, so if you don’t have it, you should install python-software-properties package.

$ apt-get install python-software-properties

Adding the Postgres ppa for the latest stable version

$ add-apt-repository ppa:pitti/postgresql

Updating the apt and installing the required packages

$ apt-get update && apt-get install postgresql pgadmin3

Stoping and removing all that belong to a given PostgreSQL cluster

$sudo -u postgres pg_dropcluster --stop 9.1 main

Creating and starting it with UTF-8 encoding

$sudo -u postgres pg_createcluster --start -e UTF-8 9.1 main

Then we can create a new user or simply change the default user’s password. To do that you should run the psql tool with a postgres user on your machine.

$sudo -u postgres psql postgres

To change the password of the user postgres type the following command and enter the new password.

# \password postgres

You can create a new DB now:

$sudo -u postgres createdb mydb -e UTF-8

Everything is done, now you can connect to the DB by running the pgadmin3 GUI tool or simply by psql

$psql -hlocalhost mydb postgres