Tag: SCM

Developer Cloud Tools Ripe For Consolidation

A dogfight is evolving for developer cloud tools.

The dominant player recently has been GitHub, raising $400M… GitHub achieved developer mindshare by giving away free hosting to open source projects such as Rails.

A close follower of GitHub has been Atlassian, a strong enterprise tools vendor with products such as JIRA, that bought BitBucket and offered free private git hosting. Atlassian went public end of 2015.

Other startups in this space that were popular a few years ago but have lagged include Unfuddle and CodebaseHQ.

More recently GitLab attended Y-Combinator and gained funding, building on an open source UI with many of the features of GitHub.

Large public cloud (IaaS/PaaS) vendors are now increasingly encroaching on this space. Why? Because the trend towards dev-ops and infrastructure as code is bringing developers more and more into the realm of infrastructure management. Projects like Docker are driving increasing commoditization of cloud infrastructure. To create and capture value IaaS/PaaS vendors need to gain and hold developer mindshare by offering more and better tools.

Amazon AWS has added CodeCommit.

AWS’s strongest competitor, Microsoft Azure offers hosted Visual Studio with free private git hosting and kanban boards for small teams, an attractive option to developers already used to Microsoft’s ecosystem.

Similarly, IBM has Bluemix DevOps Services, f.k.a. JazzHub that includes Rational Team Concert.

Google has now added Source Repositories that sync with GitHub and BitBucket and include a browser based source editor.

Meanwhile innovation in the developer tools space has shifted from repository hosting and issue tracking to continuous integration, cloud IDEs and developer chat.

CI startups include Travis CI, Circle CI and CodeShip, which recently raised a $7M series A. Atlassian BitBucket recently added CI pipelines. AWS has added CodePipeline/CodeDeploy for CI/CD.

In the cloud IDE space the strongest player appears currently CodeEnvy (IBM?). AWS has acquired Cloud9 (interestingly Atlassian was an early investor and customer) and GitLab has announced a collaboration with Koding.

Chat has been getting lots of attention recently because of Slack. Atlassian acquired HipChat. GitLabs partnered with Mattermost. Gitter? Microsoft has Skype for Business.

Given the forces at work more consolidation seems likely. Will Atlassian and GitHub remain independent from AWS, Google or IBM?

dbMaintain

Since reading about database refactoring some while ago I’ve been seeking to adopt a tool to support automated tracking and application of database changes.  The best known solution for this is LiquiBase (fka Sundog Refactoring Tool), with over 10,000 downloads of LiquiBase Core recorded on SourceForge as I write this.  In order to adopt LiquiBase on an existing project, however, I was faced with replacing dozens of SQL DDL scripts with LiquiBase specific XML files.  There is a generateChangeLog command, but it has limitations: for example it does not recognize the primary key of Oracle index organized tables, and the output is not arranged anything like the documented best practices.

Instead, for now, I have adopted a newer tool called dbMaintain, which was presented at a Belgian Java User Group last April (see below) and a ServerSide news article last July.  dbMaintain provides similar functionality to LiquiBase, but instead of XML it simply passes through native SQL scripts using JDBC, making it much easier to adopt with confidence on an existing project.  I did run into one known issue with setting the default schema, but I’m happy to say Tim Ducheyne responded to my emails and provided a fixed snapshot within days.  I hope this project gains momentum, and I’m looking forward to the next release.

Unit testing: unitils & dbmaintain

Technorati Tags:

PS. Other less mature alternatives include SQL based Thoughtworks dbdeploy and Carbon Five’s c5-db-migration.  Another custom XML based solution is MIGRATEdb.  Sony Pictures Imageworks have ported ActiveRecord migrations to Scala and there is a relatively inactive port of ActiveRecord migrations to Groovy called Bering.

Subversion on Ubuntu – Multiple Repositories

Eighteen months have passed since I setup Subversion on Ubuntu and I’ve found multiple repositories are valuable for providing finer grained access control and easier storage management. Here is a revised incantation:

su -
adduser --system --home /srv/svnrepos --gecos "System account to run svnserve" svn
apt-get install subversion xinetd
cat >> /etc/xinetd.d/svn << "EOF"
service svn
{
        port                    = 3690
        socket_type             = stream
        protocol                = tcp
        wait                    = no
        user                    = svn
        server                  = /usr/bin/svnserve
        server_args             = -i -r /srv/svnrepos
}
EOF
/etc/init.d/xinetd restart

# for each repository, eg. myrepo1
$repo=myrepo1
svnadmin create /srv/svnrepos/$repo
chown -R svn:nogroup /srv/svnrepos

# uncomment line to use default password file
vi ~svn/$repo/conf/svnserve.conf

cat >> ~svn/$repo/conf/passwd << "EOF"
[users]
fred = *****
...

Trac on Solaris

Trac is a web-based software project management and bug/issue tracking system. Example publicly accessible sites that use Trac include

Getting Trac installed on Solaris 10 is easy, see http://trac.edgewall.org/wiki/TracOnSolaris Question is, what next?

Blastwave packages are installed below /opt/csw (csw = Community SoftWare). /opt/csw/share/doc/trac/INSTALL provides the next steps. I used:

PATH=/opt/csw/bin:$PATH
MANPATH=/opt/csw/share/man:$MANPATH
tracenv=/var/opt/csw/trac
trac-admin $tracenv initenv
tracd --port 8000 $tracenv & # no authentication
firefox http://localhost:8000/trac

For a small number of users the tracd standalone server is good enough. Authentication can be managed with htdigest on Solaris 10 thusly:

/usr/apache2/bin/htdigest -c $tracenv/conf/users.htdigest
mydomain.com fred

Tracd can then be started like this:

PATH=/opt/csw/bin:$PATH
tracenv=/var/opt/csw/trac
nohup tracd --port 8000 --auth *,$tracenv/conf/users.htdigest,mydomain.com $tracenv &

To change the logo upload the new logo to /opt/csw/share/trac/htdocs and modify trac.ini

[header_logo]
…
link = https://trac.mydomain.com/
src = common/mylogo.png
…
[project]
descr = My Trac
footer = Visit the Trac open source project at 
http://trac.edgewall.org/
icon = common/trac.ico
name = My Project
url = https://myproject.mydomain.com/

Trac is much easier to administer with the WebAdmin plugin, which for Trac 0.10.4 requires downloading and installing from source:

easy_install http://svn.edgewall.com/repos/trac/sandbox/webadmin/

To get started a Trac administrator has to be empowered:

trac-admin /var/opt/csw/trac permission add fred TRAC_ADMIN

To simplify account administration we’ll also try an account manger plugin

/opt/csw/bin/easy_install http://trac-hacks.org/svn/accountmanagerplugin/0.10

…and change /var/opt/csw/trac/conf/trac.ini

[components]
webadmin.* = enabled
trac.web.auth.LoginModule = disabled
acct_mgr.api = enabled
acct_mgr.htfile.HtDigestStore = enabled
acct_mgr.web_ui.AccountModule = enabled
acct_mgr.web_ui.LoginModule = enabled
acct_mgr.web_ui.RegistrationModule = disabled
acct_mgr.admin.AccountManagerAdminPage = enabled
[account-manager]
password_format = htdigest
password_store = HtDigestStore
password_file = /var/opt/csw/trac/conf/users.htdigest
htdigest_realm = mydomain.com

Tracd can then be started like this:

PATH=/opt/csw/bin:$PATH
tracenv=/var/opt/csw/trac
nohup tracd --port 8000 $tracenv &

To manage custom fields we’ll add another plugin

easy_install http://trac-hacks.org/svn/customfieldadminplugin/0.10

…and change /var/opt/csw/trac/conf/trac.ini

[components]
customfieldadmin.* = enabled

CruiseControl and Subversion

Getting started with CruiseControl is relatively straightforward, see http://confluence.public.thoughtworks.org/display/CC/Getting+Started+With+CruiseControl

To place an application on CruiseControl it should have an Ant script to build it. The application Ant script does not need to interact with Subversion. The source for the application including the Ant script should be checked into Subversion.

I started to install CruiseControl by downloading and compiling the source, but my version of ant was too old. Instead I downloaded the binary (which ironically includes a newer ant also):

su -
wget http://downloads.sourceforge.net/cruisecontrol/cruisecontrol-bin-2.7.2.zip
unzip -d /opt cruisecontrol-bin-2.7.2.zip

useradd --system --home /srv/cruisecontrol --create-home --gecos "System account to run CruiseControl" cruise
usermod -s /bin/ksh cruise
su – cruise
mkdir checkout logs artifacts
PATH=$PATH:/opt/subversion-1.3.2/bin/:/opt/cruisecontrol-bin-2.7.2
cruisecontrol.sh

Now we need to configure CruiseControl to monitor the Subversion repository. Create config.xml:

Build-myproj.xml contains Ant steps to checkout a clean copy of the application and build it:

To setup the dashboard, running at :8080


su - cruise
cp /opt/cruisecontrol-bin-2.7.2/dashboard-config.xml .
kill `cat cc.pid`
cruisecontrol.sh

Installing Subversion on Ubuntu

There are many ways to do it, but this incantation worked for me on an Ubuntu host. This is an ‘entry level’ setup with one repository and simple authentication. Read the Subversion book and Ubuntu documentation to understand.

su -
apt-get install subversion
adduser --system --home /srv/svn --gecos "System account to run svnserve" svn
svnadmin create /srv/svn
chown -R svn:nogroup /srv/svn

apt-get install xinetd
cat >> /etc/xinetd.d/svn << "EOF"
service svn
{
        port                    = 3690
        socket_type             = stream
        protocol                = tcp
        wait                    = no
        user                    = svn
        server                  = /usr/bin/svnserve
        server_args             = -i -r /srv/svn
}
EOF
/etc/init.d/xinetd restart

# uncomment line to use default password file (~svn/conf/passwd)
vi ~svn/conf/svnserve.conf

cat >> ~svn/conf/passwd << "EOF"
[users]
fred = *****
...
EOF
chmod 600 ~svn/conf/passwd
Scroll to top