The forums tell you to run the adb-server as root. But you shouldn’t really do this. Instead create a udev-rule like the following one and put it in /etc/udev/rules.d/26-android.rules:
SUBSYSTEMS=="usb", ATTRS{idVendor}=="0bb4", ATTRS{idProduct} =="0c87", MODE="0660", GROUP="adm"
The vendor and product id here are for a HTC Desire, however yours may differ. You can use
sudo lsusb -v| grep -B 13 -i android
to get these ids. Also feel free to change mode and group settings of the device to your likings.