Jenkins on openshift #3 – building repo from github or bitbucket

Next task I wanted to do is to create jenkins on openshift and be able to build projects from github or bitbucket. It sounds simple, but you will face quite a lot of issues. So let’s do it:

Create Jenkins gear

  • this creates gear called “jenkins” using jenkins-1 application and includes ssh wrapper “git-ssh” which helps you to overcome ssh obstacles (openshift forbids to write into .ssh folder) – as you will see we will have to solve this problem several times later as well

Set jenkins slave

jenkins needs other linux machines (called slaves) to use them for building, anyway we don’t have so much free gears, so we will use jenkins itself for it.

Manage Jenkins -> Configure System -> 
# of executors = 1
Labels = put here anything

Generate ssh key

Add public key (id_rsa.pub) to github & bitbucket

search github/ bitbucket how to do it ūüôā

Add new Jenkins Job

  • in begining I mentioned that home folder is not writeable, so we are encountering several problems, like ssh can’t write into .ssh , maven can’t write into .m2 folder and so on. The only solution is to point all applications into writeable folder which is¬†$OPENSHIFT_DATA_DIR
  • Because of the problem above, you can’t use standard maven jobs but you have to use “Build a free-style software project”
  • Source Code Management: git: your git url (i.e. git@bitbucket.org:majecek/testtest.git)
  • Build: (add Execute shell): enter:
  • this creates setings.xml file and when runing maven commands – you have to specify where is the settings file
  • check this site for more info
  • next step is to add hooks in github/bitbucket – so after push in repo they will trigger jenkins to create new build – again check it in github/bitbucke or this site

Feed Sonar with data from Jenkins

This was quite a problem. I have to admit that I didn’t finish it, but found solution. Here is another problem, sonar and specially it’s mysql runs on different gear and openshift by default don’t allow any connection between gears nor from outside.

  • install Sonar plugin into jenkins, add URL & jdbc url and all info needed
  • I had to add into Sonar Aditional properties:¬†-DSONAR_USER_HOME=$OPENSHIFT_DATA_DIR
    • this is again, because sonar can’t write into home directory, so you have to point to writeable folder

Now you will face problem, that jenkins can’t connect to MYSQL DB – as said above, this is because gears can’t communicate to each other. There are 2 solutions:

  • set up SONAR as scaled application – scalled applications can communicate to each other
  • setup ssh port forwarding between applications

You can read more here.

Advertisements

SonarQube part #2 – let’s feed it with some data

In previous article, I have showed you how to build sonarqube 4.0 on openshift¬†, but let’s feed it with some data.

  • Let’s create dummy j2ee project using maven
    • mvn archetype:generate
    • cd <<project_name>>
    • mvn clean compile
  • openshift port forward to access database
    • rhc port-forward <<APPLICATION_NAME>>
    • you should see something like this:
      Service      Local                            OpenShift
      ——- ————– ¬† ¬† ¬† —- ¬† ¬† —————
      java     127.0.0.1:8080 => 127.X.X.X:8080
      mysql 127.0.0.1:3306 => 127.X.X.X:3306
    • database can be accessed on localhost:3306

Now we have few options how to feed sonar

  • maven – best for maven projects
  • sonarqube runner – best for java legacy code
  • jenkins,….

Maven projects

5 things to change or consider:

  1. jdbc url –¬†make sure you have correct url, port, application name
  2. mysql username
  3. mysql password
  4. sonar host url
  5. I am using maven version 3 – in case you have maven version 2 you have set different dependences

to run:

mvn clean compile sonar:sonar 

or (including profile name to be used)

mvn clean compile sonar:sonar -Dsonar.profile="Sun checks"

SonarQube Runner

4 things to change or consider:

  1. jdbc url – make sure you have correct url, port, application name
  2. mysql username
  3. mysql password
  4. sonar host url

to run:

  • you have to download SonarQube runner & set PATH variable
    export SONAR_RUNNER_HOME=/Path/to/SonnarRunner/sonar-runner-2.3
    export PATH=$SONAR_RUNNER_HOME/bin:$PATH
  • create file called “sonar-project.properties” and fill it with info mentioned above in gist
  • run:
    sonar-runner

Now change code (i.e. add empty private method, named with upper case letter, ….) and run analysis again to feed sonar. Check sonar – it shows you all violations and issues in your code, … ¬†(also it depends on plugins you have installed on your sonar)

Example project can be found on Bitbucket


How to run SonarQube 4.0 on openshift

I have managed to run latest SonarQube on openshift for free.

Because openshift has bug you can’t just have one-line command to do all setup for you, but I had to separate it into several commands and two git repos.

Bug

you can’t have .openshift folder in repo – so I have to have 2 git repos

  1. git repo with sonar without (.openshift folder)
  2. git repo with .openshift folder  with start & stop commands

How to get SonarQube 4.0 running on openshift

Description

  1. when RedHat fixes the bug, you should be fine, just with line #1
  2. cd into project
  3. add another git repo which holds .openshift folder with start & stop commands
  4. get changes from repo above
  5. pull from origin repo – git was complaining when I didn’t do pull
  6. push into openshift
  7. wait several minutes until sonar gets running

Thanks

Big thanks goes to Rui Rodrigues(@rodriguesrmb) as he managed to solve port binding problems and update java wrapper with new version


muCommander and external editor

If you want to use other editors/viewers with muCommander you have to do

  • find “preferences.xml” file (in Mac it ¬†“~/Library/Preferences/muCommander”) or check this page
  • Add this section:
  <editor>
     <use_custom>true</use_custom>
     <custom_command>open -a Sublime\ Text\ 2 $f</custom_command>
 </editor>
    • as you can see I am using “Sublime ¬†Text 2” text editor
    • don’t use ¬†quotation’s marks as it didn’t work
    • more can be found on this forum

 To note:

  • don’t use quotation marks – as mentioned above
  • use external editor to edit “preferences.xml” file
  • make sure mucommander is closed otherwise it likes to ¬†overwrite your changes
  • it it doesn’t work check whether you got your changes in that file (could be overwritten by muCommander)

GSL – GNU statistics library

If you need to do some calculation (like statistics, math, ….) check out the GSL library.

It has lots of method already written like standard derivation, correlation, mean, monte carlo analysis etc…

I have connected this library (which is written in C) to ruby.

steps:

  • download and install GSL (configure -> make -> make clean -> make install => see readme)
  • for ruby: ¬†gem install gsl ¬†(check ruby-gsl project for examples)
that’s it ūüôā

HSQLDB – some tips

I have been working these days with Spring3 and as DB I have used HSQLDB.

Types of catalog data

  • mem: stored entirely in RAM – without any persistence beyond the JVM process’s life
  • file: stored in filesystem files
  • res: stored in a Java resource, such as a Jar and always read-only

so the URL in java or spring context would looks like this:

"jdbc:hsqldb:file:testdb"
"jdbc:hsqldb:file:/opt/db/testdb"
"jdbc:hsqldb:mem:mymemdb"
"jdbc:hsqldb:res:org.my.path.resdb"

Turning on HSQLDB GUI in eclipse

in eclipse find the hsqldb.jar (usually lib folder) -> do right click and select “run as java” -> from the list select “DatabaseManagerSwing”

Get rid of “The database is already in use by another process”

For this error I have heard that you should use “shutdown=true” syntax in URL. So this would look like :

"jdbc:hsqldb:file:testdb;shutdown=true"  everything put in one single line (doc)

driver=org.hsqldb.jdbcDriver
url=jdbc:hsqldb:file:app.db;shutdown=true
username=sa
password=

That’s it. more can be found in google or HSQLDB doc.


google code with git

google code allows to use SVN to access the code and I was wondering whether there is way to access code with git.

  • Import
$ git svn clone --username your-name -s https://your-project.googlecode.com/svn
few other commands
git log
gitk
git diff HEAD^^
  • Developing
git add filename
git rm  filename
git commit -a     => commits to local repository
git push          => sends to server repository
  • Update – get online and check latest changes from google code
git svn rebase
  • Export – sends commits to google code
git svn dcommit       => sends commit
  • Other useful commads:
git branch -a         => shows all branches

You have noticed that svn command connected with git “git svn”. This is as already written google supports SVN, but git can comunicate “SVN” way ūüôā

You can read more here. For GIT tutorial check git pages or this tutorial.

You can use “egit” plugin to connect your eclipse to git repository.

So summary time - equivalent svn/git commands - Git for SVN User

svn checkout $SVNREP/trunk -> git svn clone $SVNREP -T trunk -b branches -t tags svn checkout $SVNREP/branches/release-0.3 -> git checkout -b local/release-0.3 release-0.3 svn commit -> git commit -a, or git add then git commit followed by git-svn dcommit svn diff|less, svn commit -> git commit -v svn merge -> git cherry-pick or use "gitk --all" svn status -> git status svn revert, --recursive -> git checkout , git checkout -f or git reset --hard svn diff -> git diff svn update -> git svn rebase --all (see also git stash)