Exporting bugs from Gforge.

At times you need to export data from a list of bugs for reporting purposes. A not-suggested way of One way of doing that is to export data from Gforge data tables using sql.
A quick way is to perform screen scrapping. I am giving to give example of this approach.

Here are the steps.

1. Log into Gforge account.
2. Get the details for a bug.
3. Fix the HTML to make a xhtml.
4. Use xslt transform to transform the xhtml into your desired format.


1) Login to GForge account.
You need to use wget to login. The wget invokation should be like specified below:
$ wget --keep-session-cookies --no-check-certificate --save-cookie=saved.cookie --post-data="return_to=%2fmy%2f&form_loginname=YourLoginName&form_pw=YourPassword2&login=Login" "https:/www.yourgforgeserver.com/account/login.php?return_to=%2Fmy%2F"
if you need to supply a http digest password then you should add this stanza in the beginning.
--auth-no-challenge --http-user=YourDigestUserName --http-password=YourDigestPassword
After this command the cookies will be saved in saved.cookie file.

2. Get the details for a bug.
$ export BUG_ID=7777
$ wget --keep-session-cookies --no-check-certificate -O $BUG_ID.details.html --load-cookie=saved.cookie "https://www.yourgforgeserver.com/tracker/?func=detail&aid=$BUG_ID&group_id=15&atid=165"
Note: The group_id and atid will be different for your project. Please specify correct values to get the details page.

3. Fix the HTML to make a xhtml.

The html generated by gforge is not wellformed so you need to clean it up. Use tagsoup to achieve this. Download the jar file. I used tagsoup version 1.2 from this (http://home.ccil.org/~cowan/XML/tagsoup/tagsoup-1.2.jar)location .
$wget http://home.ccil.org/~cowan/XML/tagsoup/tagsoup-1.2.jar
--2009-10-10 23:19:40-- http://home.ccil.org/~cowan/XML/tagsoup/tagsoup-1.2.jar
Resolving home.ccil.org... 192.190.237.100
Connecting to home.ccil.org|192.190.237.100|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 90023 (88K) [application/java-archive]
Saving to: `tagsoup-1.2.jar'


100%[===============================>] 90,023 248K/s in 0.4s


2009-10-10 23:19:41 (248 KB/s) - `tagsoup-1.2.jar' saved [90023/90023]

Now invoke this program to fix the HTML file.
$java -jar tagsoup-1.2.jar --nons --files 7777.details.html
src: 7777.details.html dst: 7777.details.xhtml

4. Use xslt transform to transform the xhtml into your desired format.
Create a xsl file to pick data from this xhtml file and create the output. A sample xsl file (names starter.xsl) can be as shown below:

<?xml version="1.0"?>

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"

version="1.0">

<xsl:template match="/">

<html><head><title><xsl:value-of select="//html/body/table[2]/tr[3]/td[2]/table[1]/tr[3]/td[2]/table[1]/form[1]/input[3]/@value"/></title></head>

<body>

<br/><xsl:text>BugId: </xsl:text>

<xsl:value-of select="//html/body/table[2]/tr[3]/td[2]/table[1]/tr[3]/td[2]/table[1]/form[1]/input[3]/@value"/>

<br/><xsl:text>Summary: </xsl:text>

<xsl:value-of select="//html/body/table[2]/tr[3]/td[2]/table[1]/tr[3]/td[2]/h2"/>

<br/><xsl:text>Resolution: </xsl:text>

<xsl:value-of select="//html/body/table[2]/tr[3]/td[2]/table[1]/tr[3]/td[2]/table[1]/form[1]/tr[3]/td[1]/select/option[@selected='selected']"/>

<br/><xsl:text>Severity: </xsl:text>

<xsl:value-of select="//html/body/table[2]/tr[3]/td[2]/table[1]/tr[3]/td[2]/table[1]/form[1]/tr[3]/td[2]/select/option[@selected='selected']"/>

<br/><xsl:text>Assigned to: </xsl:text>

<xsl:value-of select="//html/body/table[2]/tr[3]/td[2]/table[1]/tr[3]/td[2]/table[1]/form[1]/tr[8]/td[1]/select/option[@selected='selected']"/>

<br/><xsl:text>Priority: </xsl:text>

<xsl:value-of select="//html/body/table[2]/tr[3]/td[2]/table[1]/tr[3]/td[2]/table[1]/form[1]/tr[8]/td[2]/select/option[@selected='selected']"/>

<br/><xsl:text>State: </xsl:text>

<xsl:value-of select="//html/body/table[2]/tr[3]/td[2]/table[1]/tr[3]/td[2]/table[1]/form[1]/tr[9]/td[1]/select/option[@selected='selected']"/>

<br/><xsl:text>Details: </xsl:text>

<xsl:value-of select="//html/body/table[2]/tr[3]/td[2]/table[1]/tr[3]/td[2]/table[1]/form[1]/tr[11]/td[1]/table[1]/tr[1]/td[1]/table[1]/tr[2]/td[1]/pre[1]"/>

</body>

</html>

</xsl:template>

</xsl:stylesheet>

Now you can invoke xsltproc on these files:
$ xsltproc starter.xsl 7777.details.xhtml > 7777.out.html


Notes:
1. It is better to create text files instead of html because line breaks in detailed description are lost unless you escape them properly.
2. You can use lynx -dump 7777.out.html to create a text version of the html file.

XML to XSD in 5 seconds

Recently I created web page to create xsd file for a given xml content. Check it out.

http://sendit.bolona.us/tools/trang/

The UI is not so polished but it does the job.

Joining two mpeg files

Here is a small script to join to mpeg files.


#!/usr/bin/bash

FILE1=$1
FILE2=$2
OUTPUT=OUTPUT.mp2


ffmpeg -i $FILE1 -ss 0 -t 90 -y FILE1.wav
ffmpeg -v 0 -i $FILE1 -ss 0 -t 90 -sameq -y FILE2.m2v

ffmpeg -i $FILE2 -y FILE2.wav
ffmpeg -v 0 -i $FILE2 -sameq -y FILE2.m2v


sox FILE1.wav FILE2.wav all.wav


cat all.wav | mp2enc.exe -o all.mp2
cat FILE1.m2v FILE2.m2v > all.m2v


mplex -V -f 8 -L 48000:1:16 -o all.ps all.m2v $OUTPUT


# if you want to create an mpeg2 transport stream then use
#iso13818ts --fpsi 500 --ps $OUTPUT 1 > OUTPUT.ts

Implementing a Bandwidth Contorlled Pipe in Java

I implemented this one and though will post here so that I can reuse it in future.

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.util.Date;


import org.apache.commons.io.output.ThresholdingOutputStream;
public class Pipe extends ThresholdingOutputStream{

long LastTimeTheThresoldHasReached;
OutputStream OStream;

int BWCountPeriodInMiliSeconds;

public Pipe(int bwInBytes,int periodInSec, OutputStream oStream)
{
super(bwInBytes);
OStream =oStream;
BWCountPeriodInMiliSeconds = 1000* periodInSec;
LastTimeTheThresoldHasReached = 0;
}

@Override
protected OutputStream getStream() throws IOException {
return OStream;
}

@Override
protected void thresholdReached() throws IOException {
long now = System.currentTimeMillis();
long timeDiff = now - ( LastTimeTheThresoldHasReached + BWCountPeriodInMiliSeconds);
long bytesWritten = getByteCount();
long bytesExceeded = bytesWritten - getThreshold();
long timeThatWeNeedtoWait = (long) (((float)BWCountPeriodInMiliSeconds / getThreshold() ) * bytesExceeded) -timeDiff;

System.out.print("At " + new Date()+" Waiting for "+ timeThatWeNeedtoWait);

if ( timeThatWeNeedtoWait > 0)
{// We used the allocated quota quicker that we should have. So lets wait.
try {
Thread.sleep(timeThatWeNeedtoWait);
} catch (InterruptedException e) {
}

}

System.out.println("... " + new Date()+" Done");
resetByteCount();
LastTimeTheThresoldHasReached = now;
}



public static void main(String[] args) {

ByteArrayOutputStream baos = new ByteArrayOutputStream();
Pipe pipe = new Pipe(10,10,baos);
while(true)
{
try {
pipe.write("123456789012345".getBytes()); } catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}

Zoneedit with ddclient on debian

I just thought I should mention this for my own records that this configuration file works for zoneedit+ddclient combination on a debian machine as of 13th April 2009.

pid=/var/run/ddclient.pid,
protocol=zoneedit1,
use=web,
web=http://www.zoneedit.com/checkip.html,
web-skip='Current IP Address:',
server=dynamic.zoneedit.com,
login=MyName,
password='MyPassword',
yourdomain.com,www.yourdomain.com

37Signals

Today I tried free membership of 37Signals BaseCamp.
I have to say that I am not impressed. The app looks so amateurish, I wonder why would anyone pay for it. I was expecting a more sophisticated tool that just 5 screens.
At work I installed GForge few years back and we are very happy with it. The wiki is their weakest link, everything else I love, not that I hate wiki, but I wish the editing could be done using a better editor (like FCKEditor?) and I could attach images, files etc.

I have not upgraded to their latest version because I don't have time and what I have is good enough for our business.
They also have a VM, so try that and you will be happy.

Also GForge Group lets you create your own private projects free of cost, try them (I have not tried them).

Sending free SMS

Finally I created a website to send SMS message using the email to sms gateways.
Currently the US operators and few operators in India are working but I plan to add more.
I wanted this to be slightly different from the existing ones so that it remembers the numbers that you used earlier and is not so ad laden.
Check this out at,
http://sendit.bolona.us
There is also a link to suggest new cell phone operators.