Learning everywhere

World is your playground, play if possible; if not learn to play; alteast try to learn to play; you will appreciate things more.

This applies to programming as well. Programmers who know more than one programming paradigms deliver better outcomes. Very early on, I was misinformed that javascript doesn't care for invariants. The folks discussing this were my seniors, had thorough handson experience on C++ and Java. They are reknown for their sane and lucid code. They were working on a GUI generator (HTML UI) and using embedded js for validation. They did not knew unobstrusive JS. All they wanted was some validation on client UI that was realized using

 onclick="validateForm()" 

. The statement now seems silly, but from their point of view, it is akeen to writing business invariants in sed or grep. I couldn't grasp why one could not write class invariants in javascript, so asked them and got a response that in javascript everything was public (access scope). It was only a couple of years later I learnt the better parts.

My intent is not to demean any, but to highlight that even the best of the team is often blinded by competence. If one doesnot empty his pot, how can one fill fresh water! Another example comes from SQL, sql is declarative hence no imperative syntax. While it is true for the most part, certain implementations specific functions say

case...when, coallesce, nullif

can be used to impart imperativeness.

Java developers are a different breed, some just write procedural code, mask it under the name of OOPS and live with it. Some advance to proper OOPS, design and architect properly. A few others master AOP, MDD, ORM and try to be a bit more smart. JavaScript programmers on the other hand are the most eccentric ones. A true master uses functional, procedural, prototyping, event handling/messaging & OOP concepts to the fullest. While it is true that JS allows a whole gamut of things, it is still not fully seen beyond webservers. However, it is one of the few areas that indicates life beyond the know IDE realm.

Learning new languages may or may not be fun, but understanding and appreciating the concepts is definitely delightful. When I started with JRuby, the concept of blocks was the most enticing. The brevity of code was just not plausible in Java. Python had similar givings. When I moved to GWT, the concept of callbacks and message bus usage was obvious to me (naturally, they are defacto occurences in JS apps). I found certain developers crack their heads trying to propogate references/spoil interface definitions all in an attempt to meet the goals. I just wish they read a bit more.

Programmers who claim to be good with languages are not exempt. Ask them to come up with a proper word document, they do it - do it with style corruption. Most of us do not even understand what style/doc corruption is. We donot use styles. If we showed the same zeal towards MSWord, the world would be more pleasent to us. A close analogy is learning to type (finger typing or soft typing). My friends and collegues at times express their awe, well I have trained some portion of my brain to instinctively use a qwerty keyboard, (my friends can do it, but they never cared, if only you saw their stupendously long combo movements on arcade games).

My mom cooks great food. Serving several dishes working on them simultaneously is not a straight chore. I used to wonder how she did that. I observed that she planned ahead, cut her veggies and gathered her spices ahead. Even when the flame is up, she knew the order and quantity required; and accordingly batched her job. All good cooks are great planners, the plans don't always work - but planning is indispensable. Last time I was deploying a project, I took a leaf from her and prepared a check list. After a couple of reviews, we use it pre deployment, to identify what all needs to be done. A thorough deployment without a need to revisit is a small achievement, but a happy one. I am not saying that the deployment worked because I can cook things; all I am saying is that I can appreciate a intangible task in a better way.

All we need to do is keep an open mind to suggestions, and a realization that we are ignorant for the most part. Ignorance is a bliss, knowledge is power. I am happy that I have forgotten my calclus (Ignorance) but not my friends :)

Coder cannon

An article on /. about new developers getting better monetary benefits than seniors made me ponder about IT.

To give an example. In the world of stock market, if one were to estimate the worthiness of a stock/asset, it can be simply done by monitoring the topline growth for a given period. Another option would be find out the profitability of the company (Assets - Liabilities). However, one often finds stocks trading at a much higher price (indicated by P/E). This is very much like real-estate, where the price is driven not by the cost, but by the valuation.

IT valuations are taken more personally - to heart. It is expected that niche technologies deserve niche pays. For the most part, professional IT is about picking some data, transforming it and presenting it. Game devs and System programmers are prominent exceptions. I used to wonder why a SAP analyst makes more than a Java programmer, or a C++ programmer.

Indian IT companies often take a different route. They just force the programmers on to a technology. The results of cold, passionless work are the ones that give a bad name. No university can teach passion. A good organisation and leader can instill a drive, but lack of passion causes one to burnout. Another surprising facet of IT: I recollect seeing a job posting in 2009, asking for GWT experts with 5+yrs of experience. Did the job poster even know when GWT came into existance? On one hand, you can see new technologies as an opportunity to move ahead in life, on the otherhand you may have a "dejavu" moment. Indian IT's cream are adept at catching up with the game, and the whey just sulks.

Western IT is more passionate, they master the one prime thing they chose. I have consultants coming from far more diverse backgrounds into IT. While their IT skills are mediocre (beyond their comfort zone), the rich experience they bring is invalueable. Seniors are valueable, not because they have grown in age but because they can mentor and communicate better. New developers on the other hand can go anywhere. It is either a hit or miss scenario.

If you want your seniors for their soft-skills, make it clear to them. If you want them for their technical skills, tell them clearly the direction being taken. Who knows, some of them may just take a break - learn the new stuff and may prove even more valueable. One harsh thing about large IT projects is the fact that, no one is indispensable. All seniors were once n00bs, they remember that well enough. If they are comfortable in their current roles and don't want to move out, you can always blame the demand-supply curve. We still have cobol dinosaur dragon tamers around, don't we!

JMX - Remote applications

Post jdk 5, JMX has become part of the standard distribution. The default MBeans provided, provide quite a good view of the VM characteristics. However, connecting to remote applications (running on different VM) is not a very straight forward task. The tricky part is the difficult to find documentation about the rmiserver and the URL format to be used for lookup.

Sample JVM_ARGS:

-Dcom.sun.management.jmxremote.port=5000 (put your own non-conflicting port)
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=

which would look like
-Djava.rmi.server.hostname=144.203.88.87

The actual call would look like: java $JVM_ARGS MY_APP. This would invoke the JVM with an JMX Agent running on 5000 port, on a remote server whose ip-address is 144.208.88.87

Now open the jconsole application, invoke your application. Assuming that the application runs for a significant period of time (so that we can actually initiate the handshake and gather stats), enter the following in the Remote Process field

service:jmx:rmi://144.203.88.87:5000/jndi/rmi://144.203.88.87:5000/jmxrmi

Thats it! we are good to go.

Some good pointers on interwebs:

here
here and
here

About SqlLoader

SqlLoader is a utility provided by oracle, for loading data into database. This is analogous to db2export/import provided by db2. Most databases provide such utilities. Typical applications of such utilities: Migration of data (data only), creating a file based backup (as disk space is cheaper than the SAN disk space).

There are two prominent types of sql loader functionality.

• Conventional loading
  
• Direct path loading
  

Conventional loading

Sample command:

 msg=`sqlldr $oracleuser/$oraclepasswd@$ORACLE_SID control=${controlFile} data=${csvFile} direct=false bad=${badFile} log=${logFile} errors=${errors} rows=100000`

Conventional load acts as if someone is firing insert commands using oracle client. The indexes are preserved. Other users are not affected while the load happens.

Direct path loading

Sample command: (Note the direct=true part)

 msg=`sqlldr $oracleuser/$oraclepasswd@$ORACLE_SID control=${controlFile} data=${csvFile} direct=true bad=${badFile} log=${logFile} errors=${errors} skip_index_maintenance=false `

Sqlloader tries to write data directly into database, we can configure it to skip logging, bypass indices and write data directly. However this affects other users/transactions that are trying to read/write data to the same table.

Behind the scenes

Sql loader takes two major inputs. Control file, that dictates the nature of load. Data file that contains the data.
Sample control file for direct path load:

unrecoverable
load data
 APPEND

 into table vendor_data.ds_sp_data
 REENABLE DISABLED_CONSTRAINTS
 fields terminated by "," optionally enclosed by '|'
 trailing nullcols
(type_id FILLER, FEED_DATA_SOURCE FILLER, LINE_NUM, batch_id,  corp_actions_id, split_factor, ric CHAR(20) , record_type CHAR(2) , ds_type_id CHAR(8) , adjust_volume CHAR(1) , ds_file_date DATE "YYYY-MM-DD", split_date DATE "YYYY-MM-DD", file_code CHAR(4) , file_ext CHAR(1) , split_factor_val)

Sample control file for conventional load

OPTIONS (SKIP=1)
load data
 APPEND
 into table vendor_data.feed_run_status_record_type
REENABLE DISABLED_CONSTRAINTS
fields terminated by "," optionally enclosed by '|'
trailing nullcols
(ETL_SEP FILLER,RUN_ID, TYPE_ID, REGION_CODE, FEED_ID, FILE_NAME CHAR(255), RECORD_TYPE, RECORD_COUNT, FAILURE_COUNT, INSERT_COUNT,  STATUS_CODE, EXCEPTION CHAR(3000))

Sample of data loaded by the above:
(Using '------' as line separator of visual illustration only, it is not part of the data)

|21825|,|/u/datamgmt/TEST6/data/datascope/data//18251217.M|,|2|,|414146|,|SFE|,||,|1|,|21825|,|IRF|,|FUT|,|XSFE|,|M|,|NBBc1=|,|RY|,||,|NZ BANKBILL DEC9|,|2010-12-17|,|N|,|1825|,|0|
--------
|21825|,|/u/datamgmt/TEST6/data/datascope/data//18251217.M|,|3|,|414146|,|SFE|,||,|1|,|21825|,|IRF|,|FUT|,|XSFE|,|M|,|NBBc2=|,|RY|,||,|NZ BANKBILL MAR0|,|2010-12-17|,|N|,|1825|,|0|
--------
|21825|,|/u/datamgmt/TEST6/data/datascope/data//18251217.M|,|4|,|414146|,|SFE|,||,|1|,|21825|,|IRF|,|FUT|,|XSFE|,|M|,|NBBc3=|,|RY|,||,|NZ BANKBILL JUN0|,|2010-12-17|,|N|,|1825|,|0|

Note:

1. 'unrecoverable' is supported only in direct path load
   
2. APPEND is faster than INSERT
   
3. FILLER columns are skipped
   
4. CHAR(x) fields help sqlldr load fields that exceed default varchar limit of 127
   
5. SKIP indicates the number of lines to skip
   

A good article on SqlLoader

Extracting data using sql queries

I will just list down the commands that achieve this. The prerequisites for running these are to be managed by the users/admin.

For Db2:

Do a sanity check

db3 "connect to $DB_SVR user $db2user using $db2passwd"
connectionStatus=$?
 if [ $connectionStatus != 0 ]
 then
            #sendmail or raise an alert
        fi

Run the command:

fetchDataSql=" export to ${filePath}_data of del modified by coldel| ${query} "
 
 log "Extracting data using:: ${fetchDataSql}"
 db2 "${fetchDataSql}"
 extractionStatus=$?
 if [ $extractionStatus -eq 0 ] || [ $extractionStatus -eq 2 ]
 then
  log "Preparing the output file"
  echo ${title} > ${filePath}
  cat ${filePath}_data >> ${filePath}
  rm -f ${filePath}_data 
  log "########################################################################################"
  log "##  Data Extraction process completed successfully."
  log "########################################################################################"     
 else
  log "########################################################################################"
  log "##   Alert!!Problems extracting data from ${tableName}, return status::$extractionStatus "
  log "########################################################################################"
  send_mail "Alert!!Problems extracting data from ${tableName}, return status::$extractionStatus" ${FAILURE}
  terminate_db
  exit 4   
 fi

The terminate function just closes the db connection

function terminate_db {
 log "Close database connection"
 db2 "terminate"
}

For Oracle:

Load the oracle client variables into the environment. Ask you admin for details.

#Loading the oracle client
export ORACLE_BASE="/xyz/oracle/sqlnet/11.1.0/product/11.1.0/client_1"
export ORACLE_HOME="$ORACLE_BASE"
export PATH="$ORACLE_BASE/bin":"$ORACLE_BASE/perl/bin":"$ORACLE_BASE/sqldeveloper":$PATH
export LD_LIBRARY_PATH="$ORACLE_BASE/lib":$LD_LIBRARY_PATH
export PERL5LIB="$ORACLE_BASE/perl/lib/5.8.3":"$ORACLE_BASE/perl/lib/site_perl/5.8.3":$PERL5LIB

Query for results

values=`sqlplus  -s ${oracleuser}/${oraclepasswd}@${ORCL_SVR} << EOF
   SET LINESIZE 1000
   SET HEAD OFF
   SET FEEDBACK OFF
   ${query};
   exit;
   EOF`
extractionStatus=$?
if [ $extractionStatus -eq 0 ] || [ $extractionStatus -eq 2 ]
 then
  log "########################################################################################"
  log "##   Preparing the output file::${filePath}"
  log "########################################################################################"
  echo ${title} > ${filePath}
  echo ${values} > ${filePath}_data.txt
  sed 's/VDB_SEP/\n/g' < ${filePath}_data.txt >>${filePath}
  rm -f ${filePath}_data.txt
                log "########################################################################################"
  log "##   Data Extraction process completed successfully."
  log "########################################################################################"     
 else
  log "########################################################################################"
  log "##   Alert!!Problems extracting data from ${tableName}, return status::$extractionStatus "
  log "########################################################################################"
  exit 4   
fi

Paginated fetch queries

Without much fuss,

For DB2:

select * from 
    ( 
        select ROW_NUMBER() OVER () rowNum, t1.* from @tableName@ t1 
        where blah.. blah..        
    ) 
    as t00, 
    ( 
        select count(COL_1) as totalPageRecordCount from @tableName@ t2 
        where blah.. blah..
    ) 
    as t01 
where rowNum between @start@ and @end@

For Oracle:

select min_t.*, count_t.* from 
    ( 
        select rownum as row_num, max_t.* from 
            ( 
                select t1.* 
                from @data_tablename@ t1
                where blah.. blah..
                order by t1.rowid -- order by rowid for consistent results
            ) max_t 
        where  
           rownum <= @end@ 
    ) min_t, 
    ( 
        select count(*) as totalPageRecordCount 
        from @data_tablename@ t1
        where blah.. blah..
    ) count_t 
where 
    row_num >= @start@

Note that ordering by row_id ensures consistent data fetch results.