Integration with File System

Scenario: An ESB hooks up to an application via a file system - mounted or local. It polls the file generated by the system. Once available, it reads it and processes it in order to send it to final destination. Challenge: 1. A partially generated file may get picked up by the ESB. 2. Non-transactional behavior. Options:

Time based polling: This approach is for ESB to wait for a few minutes before polling the file system. Assumption: We know the maximum size of the file that the application will generate and we are 100% sure that the file is completely generated after x number of minutes.

Risk: Suppose we polled for a file at a time t. So, the next polling starts at a time t + x assuming that the next file will be completely written to the disk by the time. However, there was a failure in the application generating the file and by the time it started generating the file again, it is already too late for it to finish writing the file completely by t+x. Thus, ESB gets a half-cooked file.

Conclusion: I think this approach is very risky and likely to generate data inconsistency very often in productive environment.

Size based polling: This approach is for ESB to poll several times before it "intelligently" concludes that the file has been written completely by looking at the file size. Assumption: After n number of polling for the same file, if the file size does not grow, we are 100% sure that the file is completely written by the application. 

Risk: There are chances when while writing the file, the application may fail. If the Exception/Compensation is not properly handled by the application, the partially generated file will not be deleted by the application and will stay there. The ESB after polling n number of times, will assume that the file is completely written because its size remains the same. However, actually it is not complete and might be missing its trailer or header.

Conclusion: This is a much better option that the first. However, even in this case there are some problems as mentioned above. So, unless there is a mechanism where the application gets notified by ESB that it has processed x number of records or there was a problem in this file (Which is difficult in file based Fire-And-Forget kind of asynchronous approach), or a proper exception handling in place within the application, there is always a great risk of losing the data - esp if the file structure and parsing is important from ESB point of view.

Polling for an "OK" file: This approach is for ESB to wait for a 0 KB file with same name as data file's name but appended with a an extension such as ".OK" or ".complete". Assumption: The application and ONLY application knows when it has FINISHED writing a file to the disk.

Risk: If not already implemented in the application, the additional functionality of generating the OK file needs to be implemented. The ESB also in this case will only deal with the archival of the OK file. Hence, it is possible for confusions to arise when there are lots of data file present in the ESB "Inbound" folder. Remember, in our previous options we used to archive the data files in order for ESB to not pick the files in duplicate.

Conclusion: Despite minimum risks involved in this option, in my opinion this should along with Option 2, might be a preferred implementation. Due to non-transactional behavior of the file system, the basic fact is that only an application knows when it finishes writing the file. Hence it is important that it provides another app (ESB) a signal when it finishes writing the file and option 3 is completely based on this theory. These options also assume that the application does not have any other means to communicate with ESB apart from a File System.

Convert your Windows 7 laptop into Wireless Hotspot

Situation: You have one internet connection via ethernet cable to your laptop but several Wifi mobile devices that need internet connections. Creating/Configuring a WiFi Virtual Miniport adapter: 1. Open cmd window with Admin privilage. 2. Type in "netsh" and then "wlan" 3. set hostednetwork ssid= mode=allow ssid= key= keyUsage=persistent 4. start hostednetwork 5. This will start the "Wireless Network Connection 2" which is your Wifi Mini Virtual port adapter. 6. Then you share your main LAN connection adapter by opening its property and selecting "Share connection..." checkbox in the "Sharing" tab. Also, select the "Wireless Network Connection 2" in the drop down list. You are done. Now, you should be able to connect your wifi enabled devices with the Access point described by , which should be picked up by the devices automatically.

PowerEdge 2950 Access From ANYWHERE

As I have been setting up and fiddling around with my development environment, I had quite a few learning in the due course. Firs thing I wanted to do is to access my home server from office (Yes, behind firewall, proxy, etc etc). To cut long story short, here are the steps taken:
1. Make a note of the ports on which the programs are running that are to be accessed remotely.
2. I got a DynDNS account already and hence got a public DNS name of my server.
3. Change the setting in the home modem router to forward the incoming TCP/UDP port request to this particular machine on the relevant program's ports. For instance:
Rule1: Forward TCP xxxx to Machine abc on TCP port yyyy.
4. Adjust the OS firewall running on the server machine to accept the connections on the port yyyy.

Then the real fun starts. After making all these configuration, I was able to access the programs using my public DNS name. However, I could not connect to the machine from behind my office firewall. Apparently the only outbound ports open from the proxy are: 80 and 443.

To get around this:
1. Run SSH on your machine on a particular port (May be 22 or may be 3456 does not matter).
2. Configure modem router to forward the request coming on to port 80 or 443 to the port configured in step 1.
3. Start an SSH tunnel from the office pointing to the machine on port 80 or 443. Two ways to start it:
3.1 - Dynamic port configuration that will enable to access the any ports on the remote server outside the proxy. Any program can be accessed by RemoteServer:

3.2 - Local to Remote port forwarding - Such as accept the request on localhost at port xx that will end up in the remote server on port yy. So any program can be accessed by localhost:xx.
4. Once you are able to start the above session, you are able to access your favorite programs over the internet.

Setting up SVN and Wiki on the one EC2 instance

As part of one of my own Android projects, I tried to configure an EC2 instance trying to setup both a wiki and subversion. Some interesting points were noted when I was trying to run the SVN standalone with svnserve service on default (Or any) port. After creating a security setting with opening the relevant inbound ports on EC2 (Like 80, 8888, 3690, 22, etc), I noted that:

1. When I was able to connect to subversion on 3690, I could not browse to the default apache hope page as it would give an error 503: "Server Temporarily Unavailable".
2. When I was able to setup the Wiki (From Bitnami), I was not at all able to telnet to the svn port (Even though I was using the SAME security group - that is the same inbound port configs).

In these two cases there were two different EC2 instances involved.

Finally, I decided to go with option 2 and instead of running the svnserve, I hosted it on Apache2 using the httpd.conf configuration file for the authentication, and repo root directory creation.

I am using Bitnami Wiki, which I secured using the LocalSetting.php file on the bitnami installation. No one can view the information at this point of time except for the registered users who are the project members. The registration option is also disabled.

Subversion is configured in Apache2's httpd config and can be accessed over HTTP instead of svn protocol. The biggest advantage is that both can be accessed from behind the proxy/firewall as they all are running on port 80. 

Setting up Ubuntu 11x on Dell PE

So, I thought I will give DynDNS a try and try to access my server from outside world. Once, after installing and updating my Ubuntu server, I took the following actions:

1. Registered with DynDNS and got a hostname.
2. Installed and configured SSH: openssh configuration and able to connect using putty and the external host name.

However, in order to secure the access through certificates, I had to produce the key pair from the client and in order to upload the public key, had to setup the FTP server.
3. Installation and setup the SFTPD server on Ubuntu: Took me very long. Experienced the issues such as:

3.1 sftpd.conf not available: purge the installation through apt-get command and reinstalled it.
3.2 Logging as anonymous user:
3.3 Unable to run with writeable anonymous user's home.

Dell PowerEdge 2950

So, I finally bought this big box from eBay and cost me AUD 1075. I got a Windows server 2003 R2 standard edition along with this big and heavy box. One fine Saturday morning I started installing the OS and failed a couple of times with the error message: "Cannot find the hard drive to install OS on". After Googling for some time, I found that there are some pre-requisite drivers missing that will create logical drives. Got this blog and tried for several hours with this and my USB stick: http://thebackroomtech.com/2008/03/13/a-dell-poweredge-server-windows-2003-x64-and-the-battle-over-sas-raid-drivers/.

Then I found a comment on the blog that said: "Eventually, I had to get a server management CD to get the installation successfully done!". Immediately, got some CDs went to the Dell site and burned it with the software, ran it and it did everything quite smoothly - Prepared for installation step by step and with the thorough GUI, then prompted for the OS CDs.

However, the 2003 R2 Standard Edition would not recognize the memory greater than 4 GB. I tried fiddling with the boot.ini including /PAE in there but no help. Finally wiped out everything and installed the Ubuntu 64 bit edition utilizing the complete hardware resources to the max! 

MVC framework development in Swing

Not sure whether there is any like JSF/Struts kinds are there for fat client development. I thought something similar should work well with the desktop apps. Here is what I think -

Develop a framework XML schema design that will map each of the users' screen into the event source, the listener, target business component, target business method and its parameters in terms of the source screens' saved properties. The XML will look something like this -


com.pkg.gui.Source
com.pkg.gui.listener.Listener
com.pkg.model.Model
methodName
sourceProperty

outcome
com.pkg.gui.TargetScreen
fileName

.....
Several other mappings
....

The main idea is to reduce the application code and not unnecessary handling the screen flow. The developer can concentrate on - Designing the front screen, modelling the business data and the objects containing the business logic.

Once a event is detected, the registered listener's corresponding method is invoked. Inside the method, developer can simply call - Controller.invoke(this, sourceScreen); This call will get Controller to read the XML file to find out the source screen mapping. It will take the action source by the help of the event (From the listener) and based on the mapping load the corresponding business object and invoke the designated method with the designated parameter "reflectively".

There are several gray area in this - such as - how the target screen will populate the screen's property? Essentially, what I try to do here is eliminate the developer's codes like -
if (event.source is button)
then
 if button id is search
  then
    take text from this field
    make the object of this class
    invoke this method
    if (result is x)
         initialize this screen
   else
       initialize screen x
 else
  ........
Instead of all these code under ONE listener's action method, one will write - Controller.invoke(params);

However, this seems to be a much convoluted XML design game. WIP.