Tuesday, May 22, 2012

PostgreSQL sorting NULLS

Here is a handy feature I didn't know existed until today.

You can sort null values to the top or bottom of a list by adding "NULLS FIRST" or "NULLS LAST" at the end of the ORDER BY clause.

For a real world example,
SELECT DISTINCT p.permit_id,city_permit_num,street_no,
street, name, phone1, issue_date,
pl1.notes as work_description,
pl2.notes as work_type
FROM building.permit p
LEFT JOIN public.location l ON p.location_id = l.location_id
LEFT JOIN building.contact_permit cp ON cp.permit_id = p.permit_id
LEFT JOIN public.contact c ON c.contact_id = cp.contact_id
JOIN building.permit_line pl1 ON pl1.permit_id = p.permit_id AND pl1.status = 'A'
JOIN building.item i1 ON pl1.item_id = i1.item_id
  AND i1.description = 'Work Description'
JOIN building.permit_line pl2 ON pl2.permit_id = p.permit_id
  AND pl2.status = 'A'
JOIN building.item i2 ON pl2.item_id = i2.item_id
  AND i2.description ILIKE $1
WHERE street = $2 AND street_no = $3
ORDER BY issue_date DESC NULLS LAST


We get a nice date descending result with null dates at the bottom of the list!.
Yes, this is a real query in one of my apps. Sigh.

Wednesday, May 16, 2012

PostgreSQL Psql

PostgreSQL is a robust and powerful open source database. It has more advanced features than any other open source database and scales well with huge datasets and high traffic loads.

By default, PostgreSQL listens on TCP port 5432.


Dump all databases

pg_dumpall --clean > databases.sql

Dump a database with compression (-Fc)

pg_dump -Fc --file=database.sql --clean database

Dump a database, plain text, one schema only (-n)
pg_dump -Fp --file=filename.sql -n schema --clean database

Dump a single table

Specify the schema with the table name (if applicable) with pg_dump [-d database] -t schema.table

Dump a table definition (no data)

pg_dump -s [-d database] -t schema.table

Restore a database from a dump file

pg_restore -Fc database.sql

Restore a single table from a dump file

pg_restore -v -e -Ft -d database -t tablename dumpfile.tar
note: in this case, the dump file is in tar format, the database to restore to is after the -d switch and the table to restore is after the -t switch.

Copy data from a file into a table (from the psql client)

COPY table-name FROM '/path/to/filename' DELIMITER 'delimiter';
note: the file must be readable by postgresql (chmod 755), the default delimiter is tab.

Copy data from a table to a file (from the psql client)

COPY table-name TO '/path/to/filename' DELIMITER 'delimiter';
note: the directory and file must be writable by postgresql, the default delimiter is tab.



Start the PostgreSQL interactive terminal

psql

Psql - show a list of databases

\l
Lowercase L, not the number 1

Psql - show all users

select * from pg_user;

Psql - show all tables (including system tables)

select * from pg_tables;

Psql - show tables in the current context (database/schema)

\d

Psql - show description of tablename

\d tablename

Psql - show description of tablename, along with constraints, rules, and triggers

\d+ tablename

Psql - change current database

\c database;

Psql - show all schemas in the current database

\dn

Psql - Grant permissions on a schema to a user

GRANT ALL ON myschema TO user;

Psql - quit psql

\q

Psql - show help

\?

Psql - copy a table to a tab delimeted file

COPY table TO 'table.txt';

Psql - load a table from a tab delimeted file

COPY table FROM 'table.txt';

Psql - show permissions on database objects

\z [object]
r -- SELECT ("read")
w -- UPDATE ("write")
a -- INSERT ("append")
d -- DELETE
R -- RULE
x -- REFERENCES (foreign keys)
t -- TRIGGER
X -- EXECUTE
U -- USAGE
C -- CREATE
T -- TEMPORARY
arwdRxt -- ALL PRIVILEGES (for tables)
* -- grant option for preceding privilege
/yyyy -- user who granted this privilege

Psql - getting or setting sequence values
Get current value of a sequence:
SELECT currval('this_id_seq');

Set current value of a sequence to 1000:
SELECT setval('this_id_seq', 1000);


Run the vacuum utility (for version less than 9.0)


vacuumdb --verbose --analyze --all
Note: vacuum reclaims space from deleted records and updates indexes. It should be set up in cron. Newer versions of postgresql may run vacuum automatically.

Increase perfomance with shared memory

One effective performance tuning tip for Postgresql is to increase the shared memory buffers. This might require adding RAM to the server. Many Linux distros default to 32MB of shared memory, controlled by two kernel parameters:
/proc/sys/kernel/shmmax
/proc/sys/kernel/shmall

These values can be changed at run time, but it is better to set them at boot using the /etc/sysctl.conf file. This increases shared memory to 1GB:
# increase shared buffers for postgres at boot
kernel.shmmax=1073741824
kernel.shmall=2097152

Then, tell PostgreSQL to use 768MB of the 1GB available in the /var/lib/pgsql/data/postgresql.conf file:
shared_buffers = 98304 # min 16, at least max_connections*2, 8KB each

Restart PostgreSQL for the change to take effect.

Tuesday, April 3, 2012

LVM Basics

I've just spent a few hours with an EqualLogic iSCSI SAN disks and Linux Logical Volume Manager (LVM). The abstraction is even deeper than that, because Linux at work is running under VMware, so it is really VMware talking to the SAN and presenting a SCSI disk to Linux. Since I only get into the LVM weeds a few of times a year, I thought it would be helpful to list the steps to get usable disk space under Linux starting with the raw disk space.

Step One - create a new partition

Create a new partition with FDISK or PARTED. Mark the partition type hex 8E for LVM. In my case, the SCSI disk appeared as /dev/sdb and the partition using all space became /dev/sdb1. LVM is capable of using a raw device (no partition type), but I stayed in familiar partitioning territory.

Step Two - create LVM physical volume

pvcreate /dev/sdb1

Step Three - create LVM volume group in the physical volume

vgcreate new_volume_group /dev/sdb1

Step Four - create LVM logical volume in the volume group

lvcreate --name new_logical_volume --size 100G new_volume_group

Step Five - create a file system on the logical volume

mkfs -t ext4 /dev/mapper/new_volume_group-new_logical_volume
Note: Linux device mapper automatically creates a symlink to the disk in /dev/mapper using the volume group and logical volume names. If you choose more meaningful names than the example, the name won't look so awful.

Step Six - turn off automatic file system checks (optional)

tune2fs -c 0 /dev/mapper/new_volume_group-new_logical_volume

Step Seven - add mount point in /etc/fstab

Once the mount point is list in fstab, mount it manually and it is ready to use.

Saturday, March 31, 2012

File Timestamps [classic]

Each file has three timestamps associated with it (stored as the number of seconds since the Epoch, Jan 1, 1970). The three timestamps are:

  • Access time (atime) - the last time the file was read
  • Modify time (mtime) - the last time the file contents were changed
  • Change time (ctime) - the last time the file permissions were changed

In a long directory listing, the timestamp shown is the Modify time (mtime). To see all timestamps and a lot of other useful information, use the stat program with the verbose option (-x):
stat -x filename

Here is sample output from stat:
keithw$ stat -x "Mona Lisa Overdrive.mp3"
File: "Mona Lisa Overdrive.mp3"
Size: 6853358 FileType: Regular File
Mode: (0644/-rw-r--r--) Uid: (501/ keithw) Gid: (501/ keithw)
Device: 14,9 Inode: 10208 Links: 1
Access: Fri May 25 11:46:30 2007
Modify: Fri Dec 8 16:38:54 2006
Change: Fri Dec 8 16:38:54 2006

Deleting files with bad names [classic]

If a file with a bad name gets accidentally created, such as a name that begins with a hyphen "-", it can't be deleted with a normal remove command (rm). Use the "--" option to tell rm that no more options follow, then it can delete the file.

To delete a file whose file name begins with "-":

rm -- -bad-file-name
Or
rm ./-bad-file-name

To delete a file with non-printable characters in the name:

Use shell wildcards, '?' for one character and '*' for zero or more characters. For example, if the file name "bad file name" can't be deleted, one of the spaces may in fact contain a hexademical value. Try:
rm bad?file?name

caution: run ls bad?file?name first to make sure you are not matching more files than you think with wildcards before deleting them.

Thursday, March 29, 2012

Asus 1001-PXD Project: Keyboard Replacement

After I got Easy Peasy running, I ordered a replacement keyboard on eBay.

The keyboard that came with it was warped and bulging a little in spots, so I wanted to install a fresh one.

I found some instructions online that described the steps involved, but this YouTube video was very helpful.

You can use a spudger or a small standard screwdriver to make the switch. The key to getting the top part of the keyboard out is to push the small connectors at the top back a little, allowing the keyboard to pop out. I mangled one of the keys on the old keyboard trying to find the right angle. No permanent damage.

The keyboard I bought was for a 1001HA, but it worked perfectly in the 1001PXD. In fact, the colorized icons look even better than the original. The netbook is working liking a dream.

Saturday, March 24, 2012

Asus 1001-PXD Project: Installation

I bought an Asus 1001PXD netbook on eBay to create custom Linux "couch computer". The idea was to have someone light, portable, and secure and that didn't cost as much as an iPad or Galaxy tablet. I had done a little research on which netbooks were Linux compatible and on the various netbook oriented distributions, but it turned out to be a little trouble than I expected.

Updating the BIOS for USB book

The netbook came with Windows 7 starter edition. The first step was to update the BIOS so it supported booting from a USB flash drive. One of the guides I referenced was this one for Crunch Bang Linux.

One of the early steps to get ready for the BIOS update is:

prepare your USB flash drive with a: 16 mb FAT16 partition at the start of the disk
I first tried using a FAT32 USB drive, but that was never recognized by the Asus BIOS update utility. Next, I tried to partition the USB flash drive from Windows 7 and that was not supported. I ended up creating a 4MB FAT16 partition on a small USB flash drive and that didn't work. Finally, I used a program called BootIce to create a 1MB FAT16 partition on the same flash drive and at last it was recognized and updated the BIOS.

Installing Easy Peasy

After looking through some choices of netbook optimized distributions, I decided to try Easy Peasy. The interface is tablet like, one maximized window at a time, big icons, easy navigation, etc. I downloaded the ISO and unetbootin to create the USB boot image. The one trick here is to first reformat the USB flash drive partition back to FAT32 for booting. I tried leaving is as FAT16 and it didn't work. When booting, I hit ESC to bring up the boot menu, then chose USB and once it was up, chose to install on the hard drive with default options.

Out of the box, every appears to be working. Wireless networking, suspend/resume, camera, package installation and updates, etc. The online documentation for Easy Peasy seems a bit sparse, and the wiki had some signs of spam vandals, but I an pleased with the choice so far.