Saturday, June 9, 2012

Crontab [classic]

Crontab fields
Here is a typical crontab entry and the definition of cron fields:
00 03 * * * /tmp/zing.sh
  1. minute (00-59)
  2. hour (00-23)
  3. day of the month (01-31)
  4. month of the year (01-12)
  5. day of the week (0-6 with 0=Sunday)
  6. program or command to run
An asterisk (*) in any field means run every time for this field.
Ranges (X-Y) and steps (X-Y/Z) can also be defined.

Special @reboot option
Recent versions of cron support @reboot instead of time and day settings to run a command after boot. For example:
@reboot /usr/bin/command-to-run

I am not sure if I like this option, compared to using system start up scripts, like /etc/rc.d/rc.local.

User crontabs (including the root user)
To edit a user crontab (including root):
crontab -e
To delete a crontab:
crontab -r

User crontabs are stored as text files in /var/spool/cron/.

System crontab
The system crontab is stored in /etc/crontab. It can be changed (as root) with a text editor.
The system crontab has one extra field before the program to run, which is the user to run the command as (usually root).

Thursday, June 7, 2012

Rsync [classic]

Rsync is a file and directory syncronization tool. It can be used to keep local and/or remote files syncronized. What makes rsync powerful is that it can detect changes within large files and only transfer the parts that are different. This is much more efficient than transfering entire files, especially as the file sizes get bigger.

Note: examples that use a shell use ssh with public key authentication

To synchronize a local directory with a remote one (pull), use:

rsync -r -a -v -e "ssh -l username" --delete hostname:/remote/dir/ /local/dir/

To synchronize a remote directory with a local one (push), use:

rsync -r -a -v -e "ssh -l username" --delete /local/dir/ hostname:/remote/dir/

To synchronize a local file with a remote one, use:

rsync -a -v -e "ssh -l username" hostname:/filename /local/filename

To synchronize a remote file with a local one, use:

rsync -a -v -e "ssh -l username" /local/filename hostname:/filename

To synchronize a local directory with a remote rsync server:

rsync -r -a -v --delete rsync://rsync-server.com/stage/ /home/stage/

To synchronize a local directory with a local directory (make a backup), use:

rsync -r -a -v --delete /local/dir/ /backup/dir/

Tuesday, May 22, 2012

PostgreSQL sorting NULLS

Here is a handy feature I didn't know existed until today.

You can sort null values to the top or bottom of a list by adding "NULLS FIRST" or "NULLS LAST" at the end of the ORDER BY clause.

For a real world example,
SELECT DISTINCT p.permit_id,city_permit_num,street_no,
street, name, phone1, issue_date,
pl1.notes as work_description,
pl2.notes as work_type
FROM building.permit p
LEFT JOIN public.location l ON p.location_id = l.location_id
LEFT JOIN building.contact_permit cp ON cp.permit_id = p.permit_id
LEFT JOIN public.contact c ON c.contact_id = cp.contact_id
JOIN building.permit_line pl1 ON pl1.permit_id = p.permit_id AND pl1.status = 'A'
JOIN building.item i1 ON pl1.item_id = i1.item_id
  AND i1.description = 'Work Description'
JOIN building.permit_line pl2 ON pl2.permit_id = p.permit_id
  AND pl2.status = 'A'
JOIN building.item i2 ON pl2.item_id = i2.item_id
  AND i2.description ILIKE $1
WHERE street = $2 AND street_no = $3
ORDER BY issue_date DESC NULLS LAST


We get a nice date descending result with null dates at the bottom of the list!.
Yes, this is a real query in one of my apps. Sigh.

Wednesday, May 16, 2012

PostgreSQL Psql

PostgreSQL is a robust and powerful open source database. It has more advanced features than any other open source database and scales well with huge datasets and high traffic loads.

By default, PostgreSQL listens on TCP port 5432.


Dump all databases

pg_dumpall --clean > databases.sql

Dump a database with compression (-Fc)

pg_dump -Fc --file=database.sql --clean database

Dump a database, plain text, one schema only (-n)
pg_dump -Fp --file=filename.sql -n schema --clean database

Dump a single table

Specify the schema with the table name (if applicable) with
pg_dump -t schema.table database

Dump a table definition (no data)

pg_dump -s -t schema.table database

Restore a database from a dump file

pg_restore -Fc database.sql

Restore a single table from a dump file

pg_restore -v -e -Ft -d database -n schema -t tablename dumpfile.tar
note: in this case, the dump file is in tar format, the database to restore to is after the -d switch and the table to restore is after the -t switch.

Copy data from a file into a table (from the psql client)

COPY table-name FROM '/path/to/filename' DELIMITER 'delimiter';
note: the file must be readable by postgresql (chmod 755), the default delimiter is tab.

Copy data from a table to a file (from the psql client)

COPY table-name TO '/path/to/filename' DELIMITER 'delimiter';
note: the directory and file must be writable by postgresql, the default delimiter is tab.

List all schemas
select schema_name from information_schema.schemata

Start the PostgreSQL interactive terminal

psql

Psql - show a list of databases

\l
Lowercase L, not the number 1

Psql - show all users

select * from pg_user;

Psql - show all tables (including system tables)

select * from pg_tables;

Psql - show tables in the current context (database/schema)

\d

Psql - show description of tablename

\d tablename

Psql - show description of tablename, along with constraints, rules, and triggers

\d+ tablename

Psql - change current database

\c database;

Psql - show all schemas in the current database

\dn

Psql - Grant permissions on a schema to a user

GRANT ALL ON myschema TO user;

Psql - quit psql

\q

Psql - show help

\?

Psql - copy a table to a tab delimeted file

COPY table TO 'table.txt';

Psql - load a table from a tab delimeted file

COPY table FROM 'table.txt';

Psql - show permissions on database objects

\z [object]
r -- SELECT ("read")
w -- UPDATE ("write")
a -- INSERT ("append")
d -- DELETE
R -- RULE
x -- REFERENCES (foreign keys)
t -- TRIGGER
X -- EXECUTE
U -- USAGE
C -- CREATE
T -- TEMPORARY
arwdRxt -- ALL PRIVILEGES (for tables)
* -- grant option for preceding privilege
/yyyy -- user who granted this privilege

Psql - getting or setting sequence values
Get next value of a sequence:
SELECT nextval('this_id_seq');

Set current value of a sequence to 1000:
SELECT setval('this_id_seq', 1000);

Grant access to all tables in a schema


GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO user;


Run the vacuum utility (for version less than 9.0)


vacuumdb --verbose --analyze --all
Note: vacuum reclaims space from deleted records and updates indexes. It should be set up in cron. Newer versions of postgresql may run vacuum automatically.

Increase perfomance with shared memory

One effective performance tuning tip for Postgresql is to increase the shared memory buffers. This might require adding RAM to the server. Many Linux distros default to 32MB of shared memory, controlled by two kernel parameters:
/proc/sys/kernel/shmmax
/proc/sys/kernel/shmall

These values can be changed at run time, but it is better to set them at boot using the /etc/sysctl.conf file. This increases shared memory to 1GB:
# increase shared buffers for postgres at boot
kernel.shmmax=1073741824
kernel.shmall=2097152

Then, tell PostgreSQL to use 768MB of the 1GB available in the /var/lib/pgsql/data/postgresql.conf file:
shared_buffers = 98304 # min 16, at least max_connections*2, 8KB each

Restart PostgreSQL for the change to take effect.

Tuesday, April 3, 2012

LVM Basics

I've just spent a few hours with an EqualLogic iSCSI SAN disks and Linux Logical Volume Manager (LVM). The abstraction is even deeper than that, because Linux at work is running under VMware, so it is really VMware talking to the SAN and presenting a SCSI disk to Linux. Since I only get into the LVM weeds a few of times a year, I thought it would be helpful to list the steps to get usable disk space under Linux starting with the raw disk space.

Step One - create a new partition

Create a new partition with FDISK or PARTED. Mark the partition type hex 8E for LVM. In my case, the SCSI disk appeared as /dev/sdb and the partition using all space became /dev/sdb1. LVM is capable of using a raw device (no partition type), but I stayed in familiar partitioning territory.

Step Two - create LVM physical volume

pvcreate /dev/sdb1

Step Three - create LVM volume group in the physical volume

vgcreate new_volume_group /dev/sdb1

Step Four - create LVM logical volume in the volume group

lvcreate --name new_logical_volume --size 100G new_volume_group

Step Five - create a file system on the logical volume

mkfs -t ext4 /dev/mapper/new_volume_group-new_logical_volume
Note: Linux device mapper automatically creates a symlink to the disk in /dev/mapper using the volume group and logical volume names. If you choose more meaningful names than the example, the name won't look so awful.

Step Six - turn off automatic file system checks (optional)

tune2fs -c 0 /dev/mapper/new_volume_group-new_logical_volume

Step Seven - add mount point in /etc/fstab

Once the mount point is list in fstab, mount it manually and it is ready to use.

Saturday, March 31, 2012

File Timestamps [classic]

Each file has three timestamps associated with it (stored as the number of seconds since the Epoch, Jan 1, 1970). The three timestamps are:

  • Access time (atime) - the last time the file was read
  • Modify time (mtime) - the last time the file contents were changed
  • Change time (ctime) - the last time the file permissions were changed

In a long directory listing, the timestamp shown is the Modify time (mtime). To see all timestamps and a lot of other useful information, use the stat program with the verbose option (-x):
stat -x filename

Here is sample output from stat:
keithw$ stat -x "Mona Lisa Overdrive.mp3"
File: "Mona Lisa Overdrive.mp3"
Size: 6853358 FileType: Regular File
Mode: (0644/-rw-r--r--) Uid: (501/ keithw) Gid: (501/ keithw)
Device: 14,9 Inode: 10208 Links: 1
Access: Fri May 25 11:46:30 2007
Modify: Fri Dec 8 16:38:54 2006
Change: Fri Dec 8 16:38:54 2006

Deleting files with bad names [classic]

If a file with a bad name gets accidentally created, such as a name that begins with a hyphen "-", it can't be deleted with a normal remove command (rm). Use the "--" option to tell rm that no more options follow, then it can delete the file.

To delete a file whose file name begins with "-":

rm -- -bad-file-name
Or
rm ./-bad-file-name

To delete a file with non-printable characters in the name:

Use shell wildcards, '?' for one character and '*' for zero or more characters. For example, if the file name "bad file name" can't be deleted, one of the spaces may in fact contain a hexademical value. Try:
rm bad?file?name

caution: run ls bad?file?name first to make sure you are not matching more files than you think with wildcards before deleting them.