tr -d 'r' < winfile.txt > unixfile.txt
awk 'sub("$", "r")' unixfile.txt > winfile.txt
Hiring advice from Dee Hock
“Hire and promote first on the basis of integrity, second motivation, third capacity, fourth understanding, fifth knowledge, and last and least, experience. Without integrity, motivation is dangerous; without motivation, capacity is impotent; without capacity, understanding is limited; without understanding, knowledge is meaningless; without knowledge, experience is blind. Experience is easy to provide and quickly put to good use by people with all the other qualities.” Dee Hock, founder, VISA
Fast Company Article: The Trillion-Dollar Vision of Dee Hock
Reporting Tools
Having built and populated a reporting database I’ve been looking at reporting tools for the front end. In the past I’ve used Crystal Reports, Oracle Reports and MicroStrategy, but with the BI industry in flux over the past couple of years as a result of acquisitions and new open source businesses, I thought now is a good time to look at alternatives like BIRT, JasperSoft and Pentaho. My requirements are:
- easy and powerful report designer and engine with relevant examples
- secure report repository with end user web access including external authentication
- scheduled report execution and delivery by email
- value for money given tech savvy in-house support
Starting with report designers I looked at BIRT RCP Report Designer 2.3 vs. JasperSoft iReport 3.0. BIRT has a field guide included in the online help, report layouts may be fixed or automatic, and the BIRT report engine was described as stronger than JasperReports on a ServerSide forum. iReport struck me as undistinguished, and the user manuals need to be purchased as they are not available for download, which was irksome to perform a review. It does, however, provide a ‘pixel perfect’ layout and there were demo samples in jasperreports-3.0.0-project.zip. The first report I built failed to run, and I could not find any explanation.
Looking at report repositories I compared BIRT Report Server vs. JasperServer 2.1, Pentaho Reporting and OpenReports. BIRT Report Server was previously Actuate iServer Express, and is still a commercial product costing $1,995/core. There is a BIRT Deployment kit with repository (but no scheduler) available at $995/core. There is a 90 day evaluation available and manuals are available online. JasperServer 2.1 installation was bundled with Apache Tomcat 5.5, MySQL 5.0, Java JDK 1.5 and iReport 2.0.2. Novell SLES 10.2 is documented as a supported platform. I found the Pentaho Platform documentation to be jumbled and immature, with different dates more like an online support knowledgebase. The Pentaho wiki, however, does include articles on integrating BIRT and JasperReports. Also Pentaho is certified on Novell SLES 10 and has also been installed on Ubuntu. Finally OpenReports supports multiple report engines, including BIRT, JasperReports and Pentaho. External integration is supported using Spring Framework, but not well documented.
My conclusions from this research is that these products are still maturing, and I would not want to mix and match components from different products. I was not persuaded to pay for a commercial BIRT Report Server licence, even if the price is relatively modest compared to competing products a few years ago. That leaves Pentaho Reporting and JasperServer as the best alternative repositories, and they rate about equal. In my case I’m focusing on JasperServer as being easier to integrate seamlessly with our own software.
Populating an Oracle tablespace
Suppose that we want to move all the mid-size indexes for a given schema FOO, into their own tablespace. First, let’s list them all:
SELECT ext.owner, index_name, SUM (bytes) / (1024 * 1024) MB FROM dba_extents ext JOIN dba_indexes idx ON idx.owner = ext.owner AND idx.index_name = ext.segment_name WHERE ext.owner = 'FOO' AND segment_type = 'INDEX' AND index_type = 'NORMAL' GROUP BY ext.owner, index_name HAVING SUM (bytes) / (1024 * 1024) > 8 ORDER BY MB DESC
How large does the tablespace need to be?
SELECT 1.25 * SUM (bytes) / (1024 * 1024) MB FROM dba_extents ext JOIN dba_indexes idx ON idx.owner = ext.owner AND idx.index_name = ext.segment_name WHERE ext.owner = 'FOO' AND segment_type = 'INDEX' AND index_type = 'NORMAL' ORDER BY MB DESC
To populate the tablespace we could use:
SELECT 'alter index '||ext.owner||'.'||segment_name||' rebuild tablespace '||:new_ts||';' FROM dba_extents ext JOIN dba_indexes idx ON idx.owner = ext.owner AND idx.index_name = ext.segment_name WHERE ext.owner = 'FOO' AND segment_type = 'INDEX' AND index_type = 'NORMAL' GROUP BY ext.owner, segment_name, segment_type HAVING SUM (bytes) / (1024 * 1024) > 8 ORDER BY SUM (bytes) / (1024 * 1024) DESC
Now suppose we want to move all tables (including index organized) for a given schema FOO into their own tablespace. To list them:
SELECT owner, table_name, SUM (bytes) / (1024 * 1024) MB FROM (SELECT ext.owner, table_name, bytes FROM dba_extents ext JOIN dba_tables tbl ON tbl.owner = ext.owner AND table_name = ext.segment_name WHERE ext.owner = 'FOO' UNION ALL SELECT ext.owner, table_name, bytes FROM dba_extents ext JOIN dba_indexes idx ON idx.owner = ext.owner AND idx.index_name = ext.segment_name WHERE ext.owner = 'FOO' AND segment_type = 'INDEX' AND index_type = 'IOT - TOP') GROUP BY owner, table_name, bytes HAVING SUM (bytes) / (1024 * 1024) > 8 ORDER BY MB DESC;
How large does the tablespace need to be?
SELECT 1.25 * SUM (bytes) / (1024 * 1024) MB FROM (SELECT ext.owner, table_name, bytes FROM dba_extents ext JOIN dba_tables tbl ON tbl.owner = ext.owner AND table_name = ext.segment_name WHERE ext.owner = 'FOO' UNION ALL SELECT ext.owner, table_name, bytes FROM dba_extents ext JOIN dba_indexes idx ON idx.owner = ext.owner AND idx.index_name = ext.segment_name WHERE ext.owner = 'FOO' AND segment_type = 'INDEX' AND index_type = 'IOT - TOP')
To populate the tablespace we could use:
SELECT 'alter table '|| owner|| '.'|| table_name || ' move tablespace '|| :new_ts|| ';' FROM (SELECT owner, table_name FROM dba_tables tbl WHERE owner = 'FOO' UNION ALL SELECT owner, table_name FROM dba_indexes idx WHERE owner = 'FOO' AND index_type = 'IOT - TOP') SELECT 'alter index ' || owner || '.' || index_name || ' rebuild;' FROM dba_indexes WHERE owner = 'FOO' AND status = 'UNUSABLE'
VMware Timekeeping
A hopelessly inaccurate timeclock on my SLES host has been driving me nuts. I eventually found the relevant VMware white paper and used Virtual Center to set the ‘Advanced’ option to ‘Synchronize guest time with host’:
I also modified /boot/grub/menu.lst to add ‘clock=pit’ as an argument to the kernel invocation. Both changes required a reboot of the VM, but at least the issue is resolved now. Why on earth isn’t this option set by default?
CruiseControl and Subversion
Getting started with CruiseControl is relatively straightforward, see http://confluence.public.thoughtworks.org/display/CC/Getting+Started+With+CruiseControl
To place an application on CruiseControl it should have an Ant script to build it. The application Ant script does not need to interact with Subversion. The source for the application including the Ant script should be checked into Subversion.
I started to install CruiseControl by downloading and compiling the source, but my version of ant was too old. Instead I downloaded the binary (which ironically includes a newer ant also):
su - wget http://downloads.sourceforge.net/cruisecontrol/cruisecontrol-bin-2.7.2.zip unzip -d /opt cruisecontrol-bin-2.7.2.zip useradd --system --home /srv/cruisecontrol --create-home --gecos "System account to run CruiseControl" cruise usermod -s /bin/ksh cruise su – cruise mkdir checkout logs artifacts PATH=$PATH:/opt/subversion-1.3.2/bin/:/opt/cruisecontrol-bin-2.7.2 cruisecontrol.sh
Now we need to configure CruiseControl to monitor the Subversion repository. Create config.xml:
Build-myproj.xml contains Ant steps to checkout a clean copy of the application and build it:
To setup the dashboard, running at :8080
su - cruise cp /opt/cruisecontrol-bin-2.7.2/dashboard-config.xml . kill `cat cc.pid` cruisecontrol.sh
Installing Subversion on Ubuntu
There are many ways to do it, but this incantation worked for me on an Ubuntu host. This is an ‘entry level’ setup with one repository and simple authentication. Read the Subversion book and Ubuntu documentation to understand.
su - apt-get install subversion adduser --system --home /srv/svn --gecos "System account to run svnserve" svn svnadmin create /srv/svn chown -R svn:nogroup /srv/svn apt-get install xinetd cat >> /etc/xinetd.d/svn << "EOF" service svn { port = 3690 socket_type = stream protocol = tcp wait = no user = svn server = /usr/bin/svnserve server_args = -i -r /srv/svn } EOF /etc/init.d/xinetd restart # uncomment line to use default password file (~svn/conf/passwd) vi ~svn/conf/svnserve.conf cat >> ~svn/conf/passwd << "EOF" [users] fred = ***** ... EOF chmod 600 ~svn/conf/passwd
Installing Ubuntu Server 8.04 LTS Using LVM
The installation ISO for 8.04 LTS Server can be downloaded from http://www.ubuntu.com/getubuntu/download-server
Ubuntu 8.04 was released in April 2008 and will be supported until April 2013. The 64bit version is recommended.
I installed Ubuntu Server on a VM with a 2GB drive and 512MB RAM.
The default hostname given during the install is ‘ubuntu’. Be sure to change it to something more distinct.
The installation offers LVM, and even encrypted LVM, but if you select them it does not actually install the O/S on an LVM partition. For that you have to enter manual partitioning. I created a 100MB ext3 /boot partition (GRUB does not support booting from an LV) and allocated the rest of the drive to a PV.
Once the two disk partitions are created you can configure the Logical Volume Manager. Allocate LVs for the various filesystems: at least 200MB each for / and /var and 250 MB for /usr (installing VMware tools requires more disk storage, a 350MB /tmp to untar the installation bundle and 600MB /usr). I like using multiple separate filesystems because it reduces the risk that a runaway process will consume all available storage and render the system unavailable. Using LVM helps make multiple filesystems manageable because it becomes much easier to extend a filesystem when you need to.
Ubuntu Server installation offers a minimal menu of packages to install. I chose OpenSSH (OpenBSD Secure Shell) and PostgreSQL (v.8.3). UFW is installed, but not enabled by default.
After installation my filesystem utilization looked like this.
To extend LVM based filesystems see http://tldp.org/HOWTO/LVM-HOWTO/extendlv.html, eg.
lvcreate --size 150M --name lv_tmp vg01
mkfs.reiserfs --label tmp /dev/vg01/lv_tmp
lvextend -L+100M /dev/vg01/lv_tmp
resize_reiserfs -f /dev/vg01/lv_tmp
VMware Server Tips ‘n Tricks
Brian Downey has a good blog on VMware Server and clustering.
Drive Mappings with VBScript
I found this script at Programmer’s Corner.