The art of mastering

Everyone wants experts, no one like mediocre. So what makes one an expert? Simple – deliberate practice. Malcolm gladwell, in ‘Outliers’ says that it takes 10000 hours to master any subject. How about programming? A person doing a 40hr week does around 2000 hours of programming, so that is 5 years. Hence the reason, most job positions ask for 5-9 yrs of experience. As simple as it sounds, it is disheartening to know that it is not true. I have seen programmers with 2-3yrs experience out perform their seniors. How is it possible then?

The answer lies in the fact that programming is a thought process; it is not a tangible entity. Once you know and believe that you can program, you need to apply it. The application is the process of mastery. Good programmers are good because, they spent time solving problems by pondering. The process of programming is not a straight forward verb. It is thought process that involves coming up with solutions or looking out for alternatives; and understanding them so that one can apply them in their truest form (not by pasting the code found elsewhere).

Good programmers try to minimize their technical debt (described by martin fowler). They try to be hands on. Most good programmers have the following traits.
1. Understand problems, and categorize them into known and unknown (or solvable on their own or need help)
2. Try to devise more than one solution, how so ever bad the stupid solution be
3. Discuss the hard case solutions with others to seek more inputs. (like a riddle/puzzle)
4. Read books (and gain insights from others who had been through the same) a smart cut approach.
5. Learn different programming paradigms (not languages aka syntax)

Ever wondered what is the difference between a programmer and a designer/architect? A good programmer is like a artist. Given a tough job, the goal of a good programmer is to realize it in its spirit. Good programmers circumvent problems and take solutions that burden them the least. A designer is a mature programmer, One can call oneself a designer when one devises a goal and achieves it.

Designers’ genuinely discover themselves to be so. They set forth goals and achieve them in a way that acts as an example to others. One can never be a good designer by virtue of a title or experience. Design is a discovery process, unlike programming. The true benchmark in my opinion is when a designer devises a solution all by himself and finds out that it is in fact a standard/recommended practice. Designers are above the implementation details and should realize that by achieving things the way they want , and not by the way the obvious solutions allow.
If a designer sets goals and cannot achieve them, then it is inadequate. Good designers preach and practice, not to substantiate their gospel but to discover new means.

Behind the interview door

Getting a job is a momentous occasion. No matter, how much you yearn - you always go through the dreaded interview phase. Now don’t get me wrong. I am not saying that interviews have to be miserable. I am used to sitting on either side of the desk. While it is a joy to interview candidates who fit the bill; It rarely happens. No amount of preparation can guarantee a success. Interviewees aspire to go by the ‘book’ (answering everything under the sun). However good interviewers don’t go by the book, they are great listeners. I had many a great opportunity to share the hall with some of them and I found the lessons invaluable. I am writing this for the benefit of many, with whom I shared and will share the experience.
While CV may say many things, there are certain things which become obvious only during the human interaction. The most prominent ones are below (in no particular order). The traits that make an interviewee most endearing are the ones I listed below.

Confidence/ Humility:

Do this by the correct measure. Don’t miss the hallmark listed in bold (the Job description). If it is for a sales/marketing/PR alike position, you can get away by overselling yourselves. For technical interview never promise things which you cannot deliver. I will let you a secret, the other side of the bench, also comprises of human beings, who themselves have done as many mistakes as you have done, if not more. They will be your future colleagues/supervisors. They reached their current position by committing those errors and are quite wary.
So what is the right measure? Simple tell them what you are good at, what you like and what you have done. It may not be a good idea to tell them what you don’t want, if you desperately need the job. In the long run however, you will find that the decent interviewer rejected you, not because you were bad but because you deserved better. I have been rejected roles in the final rounds with the comment from senior managers/leads saying there is an ‘aspiration mismatch’. Why hire a horse to do the donkeys’ work?

Presence of mind:

Listen, always listen first and ask for time if required. Ask sensible questions, If making a guess – spell it out loudly (I am not sure, can I guess!) It does wonders. There are dumb questions which get thrown to you, and smart answers which make the rational clear. Don’t jump the context unless you have struck a chord with the interviewer’s wavelength.
Interviewers also fumble; their wording is confusing at times, so ask questions so that you can think better. Do this before you admit defeat. If a interviewer is keen, hints will be thrown to you, counter points will be raised. This is your chance to take control. See someone is genuinely willing to hear you!

Honesty/Sincerity:

If above fails, admit your lack of knowledge or ignorance. It may be a morale breaker for you, but definitely not for the interview. People wish to judge on the merit and the intent. You won’t be punished for your lack of know how. No one can master every damn thing, but you definitely need to master the relevant. Even an indication that you are on the right course is a welcoming sight. There are only two things one has to do, communicate things which one has done and create trust about your ability to learn and adapt to new factors.

Never assume that you can drive the interview:

While it is true that most interviews have a defined timeline. It is never a good idea to gauge by the number of interviews and duration. The real metric is the quality of the hire. Technical folks tend to be adamant, so unless you give answers they don’t relent. It is never a good idea to procrastinate/show-off. Questions which ask you about your areas of comfort are to be viewed as the opening gambit. Sometimes pretty straight questions may come asking you to rate yourselves. The number is asked to judge ones perception of caliber and not as an answer. It is the interviewers’ job to do that. The best thing one can do is blend with the flow, like a sugar cube in milk, not obvious but definitely perceivable.
Interviewers also ask questions which sound dumb or outrageous; this is done just to set a context like a cold start.

1. 
   Think and communicate your thoughts/concerns
   


2. 
   Raise questions and point out flaws (politely)
   Wait for the control to be given to you, pounce by giving relevant examples from your experience.
   


3. 
   Always keep in mind that your preparation has started since the day you started working (or you joined college) and continual learning is indispensable.
   


4. 
   Never badmouth anyone. Infact this is one place where you can show-off you diplomatic skills.
   

Social Innovation - Communal participation (2)

In the earlier post, I listed out certain catalysts who have spent a significant time & effort in social causes. They are not to be seen as NGOs spending monies. These guys come under a different league. The inviting thing about social innovation, is that this can be copied/mimicked, and the plagiarist attitude is actually helpful. Anyone set up a franchise of a rural call center or courier service, and contribute in a meaningful way. I envy the HR in such organizations (near 0 attrition).

If deaf and dumb can run courier services, why not laundromats & cleaners? With 3G in India, why not have a call center that can transcribe and reinterpret the signs to verbal dialect targeted towards focus groups?. Why not come up with financial planning for rural farmers who have a different set of needs. Why not have a facebook for challenged/access deprived users? Can you have a broadcast station for deaf?

These ideas have great potential, and I for sure will not ask for any credit ;). Certain seemingly simple questions like transliteration of sign lang to text, have made me realize the complexity of the problem. Until then I would encourage and make an effort to participate in every effort that aids this cause.

Social Innovation - Communal participation (1)

Thanks to eenadu.net

Example 1

Who: Dhruv Lakra

What he did: Enabled the challenged folks to stand up for themselves

My moto for him: Teach a man how to fish...

Example 2

Who: Harish Hande

What he did: Brought sustainable power to masses

My moto for him: Your actions speak louder than your qualification.

Example 3

Who: Saloni malhotra

What she did: Proved that IT is not about A.C buildings

My moto for her: 'IT' makes sense

Example 4

Who: Ramana Babu Killi

What he did: Reached out to rural farmers

Maven and Ant

'Build' is more of a process than a tool. Build tools have come a long way, writing custom bat/shell scripts is no longer acceptable. Ant became the defacto standard for java shops. Then came Maven, for a developer working on a single project, the build tool would not be a major concern. It is just another tool in the arsenal.

A good Ant script is configurable, readable, modular and extensible. Properties and semantically sound naming conventions are must for a good script. This is analogous to writing maintainable code and carries the same penalty, none of the above conventions are mandatory, implying that hastily written scripts can end up in production environment.

Maven strongly enforces the concept of life-cycles. Plugins are used to carry out tasks (similar to ant tasks). Archetypes define the xml structure. All of these make it a bit more difficult to deviate from standards. Maven rules over Ant when it comes to standardisation. Almost anyone can use a maven build file (pom), where as certain build.xml files have known to have caused nightmares.

So why isn't everything mavenized? Answer: simplicity. Almost every developer working on a project know the libraries & versions required. Unless you are building a library/framework, one can be certain about build invariants (libraries). So 75.23455623% [random number generated by a fickle brain :)] of the projects meant for end-user consumption are happy with a working ant script. They don't need to haggle over dependency resolution. These are guys who don't care if SCM/CI are built-in or whether a GUI exists for viewing the jar files.

To conclude the discussion, decide in which environment you would be working. If it is a environment with diverse projects that may use your code one day, go with Maven as it is more polite to advanced users. If you work on personal projects/college assigments/maverick prototypes, Ant fits your bill.

Java HashMap with time base expiry policy

I recently had to come with this data-structure, later I found that google collections has a MapMaker which essentially does the same. Posting some ideas and the code I have written here.

There are broadly two different ideas:
1. Create a time tracker along with the hashMap. Then run a thread to detect and delete expired items
2. Create a new hashMap periodically while flushing the contents of old map.

I have implemented the first kind. This makes use of jdk 6 concurrency features

package test; import java.util.Collections; import java.util.HashMap; import java.util.HashSet; import java.util.Map; import java.util.Set; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.Executors; import java.util.concurrent.ScheduledExecutorService; import java.util.concurrent.ThreadFactory; import java.util.concurrent.TimeUnit; import java.util.concurrent.locks.ReentrantReadWriteLock; /** * Thread safe object map, which clears objects after a specified time. The objects are stored in the * underlying hashMap. * @author rongkan * */ public final class TimeLimitedCacheMap { private final ConcurrentHashMap<String, Object> objectMap = new ConcurrentHashMap<String, Object>(10); private final ConcurrentHashMap<String, Long> timeMap = new ConcurrentHashMap<String, Long>(); /* I need a shared lock, readwrite lock is an excellent candidate. * evcition is run with writeLock, put/remove with readLock */ private final ReentrantReadWriteLock accessLock = new ReentrantReadWriteLock(); //private final ReentrantLock accessLock = new ReentrantLock(); private final Runnable evictor = new Runnable() { /* evictor thread removes data, and changes map state. This is * in conflict with put() and remove(). So we need sync for these operations * * In case you are wondering why evictor needs sync (it just calls remove() right?) * eviction is a compound action that spans more than a single remove(). It enters * into a conflict with put() * * evictor: start looping ------------------------------> keyset is stale, evictor removes the recent put & armagedon comes * Thrd1:--------------------->put(same key, new object) * */ @Override public void run() { // avoid runs on empty maps if(timeMap.isEmpty()){ Thread.yield(); } long currentTime = System.nanoTime(); accessLock.writeLock().lock(); Set<String> keys = new HashSet<String>(timeMap.keySet()); accessLock.writeLock().unlock(); /* First attempt to detect & mark stale entries, but don't delete * The hash map may contain 1000s of objects dont' block it. The returned * Set returned may be stale, implying: * 1. contains keys for objects which are removed by user, using remove() (not a problem) * 2. contains keys for objects which are updated by user, using put() [a big problem] */ Set<String> markedForRemoval = new HashSet<String>(10); for (String key : keys) { long lastTime = timeMap.get(key); if(lastTime == 0){ continue; } long interval = currentTime - lastTime; long elapsedTime = TimeUnit.NANOSECONDS.convert(interval, expiryTimeUnit); if(elapsedTime > expiryTime){ markedForRemoval.add(key); } } /* Actual removal call, which runs on the objects marked earlier. * Assumption: marked objects.size() < hashmap.size() * Do not delete blindly, check for staleness before calling remove */ accessLock.writeLock().lock(); for (String key : markedForRemoval) { long lastTime = timeMap.get(key); if(lastTime == 0){ continue; } long interval = currentTime - lastTime; long elapsedTime = TimeUnit.NANOSECONDS.convert(interval, expiryTimeUnit); if(elapsedTime > expiryTime){ remove(key); } } accessLock.writeLock().unlock(); } }; private final ScheduledExecutorService timer = Executors.newSingleThreadScheduledExecutor(new MyThreadFactory(true)); private final class MyThreadFactory implements ThreadFactory { private boolean isDaemon = false; public MyThreadFactory(boolean daemon){ isDaemon = daemon; } @Override public Thread newThread(Runnable r) { Thread t = new Thread(r); t.setDaemon(isDaemon); return t; } }; private final long expiryTime; private final TimeUnit expiryTimeUnit; /** * Users can play around with evictionDelay and expiryTime. * 1. Large evictionDelay => less frequent checks, hence chances of finding expired Objects are more * 2. Lean evictionDelay => aggressive checks, and hence more sync overhead with put() and remove() * 3. Large expiryTime => increases the time object stays in object map and less chance of cache miss (cache miss is bad) * 4. Lean expiryTime => by itself does not force object to be removed aggressively, needs lean eviction to be configured * * In case you are wondering, above picture is not complete. * Another key aspect is "Arrival Periodicty", or the rate at which put() is called. * * Ideally: expiryTime == arrival periodicity + 1 [read '+1' as slightly greater] * evictionDelay == expiryTime + 1 * * For random arrival times, which is a more common scenario, use the following pointers * 1. eviction Delay > expiry Time * Here user needs to think of the impact of stale hit (define how stale is stale!) * 2. eviction Delay < arrival time * This has higher chances of cache miss and accidental treatment as failure * * 3. eviction Delay < expiry Time * Unwanted eviction run(s) resulting in sync overhead on map * 4. eviction Delay > arrival Time * Unwanted eviction run(s) resulting in sync overhead on map * * @param initialDelay, time after which scheduler starts * @param evictionDelay, periodicity with which eviction is carried out * @param expiryTime, age of the object, exceeding which the object is to be removed * @param unit */ public TimeLimitedCacheMap(long initialDelay, long evictionDelay, long expiryTime, TimeUnit unit){ timer.scheduleWithFixedDelay(evictor, initialDelay, evictionDelay, unit); this.expiryTime = expiryTime; this.expiryTimeUnit = unit; } /* The intention is to prevent user from modifying the object Map, * I want all adds/removals to be suspended. synchronizing on objectMap * would also work, but locks are easier to read without scope issues of {} * * Concurrent hashMap would have allowed put and remove to happen concurrently. * I did not use conc-map, because * 1. I want to update another map (timeMap) * 2. I want to sync the operation with evictor thread * * The unfortunate invariants: * 1. sync between evictor and put() * 2. sync between evictor and remove() * imply * 3. sync lock between put() and remove() * * 3. is unfortunate side effect, as you need to sync all exposed invariants on the same lock. * Lock duplication won't help. If I create putLock() and removeLock(), they will allow put() and remove() * to happen concurrently, but will not help two put()/remove() calls to happen in parallel. */ public void put(String key, Object value) { accessLock.readLock().lock(); Long nanoTime = System.nanoTime(); timeMap.put(key, nanoTime); objectMap.put(key, value); accessLock.readLock().unlock(); } /* Read comments for put() they apply here as well. * If had not allowed remove(), life would have been zillion times simpler. * However, an undoable action is quite bad. */ public Object remove(Object key) { accessLock.readLock().lock(); //accessLock.lock(); Object value = objectMap.remove(key); timeMap.remove(key); //accessLock.unlock(); accessLock.readLock().unlock(); return value; } /* Clone requires locking, to prevent the edge case where * the map is updated with clone is in progress */ public Map<String, Object> getClonedMap(){ accessLock.writeLock().lock(); HashMap<String, Object> mapClone = new HashMap<String, Object>(objectMap); accessLock.writeLock().unlock(); return Collections.unmodifiableMap(mapClone); } }

GXT Nuggets: Hipocrite's combo box

Bananas for the code monkey

Ever faced a situation where you want to show the user some information 'X' but you actually need 'Y' for the backend logic? Say you have an employee entity with a employee details and a auto-generated PK. You have designed a form to show the employee details, based on the user selection (in a drop-down combo). A hiprocrite's combo is up for this task. It shows employee name/number in the UI but secretly holds the PK deep down.

You may wonder why all the fuss, why not just set the state in the drop down. Works wonderfully provided you 'remember' the state's 'key'. If I have 10 such forms, this involves 10 if() stmts atleast. To overcome this and let the contract-driven design make life easier for us, I re-invented the combo box. I added an adaptive behaviour in form of DynamicProperyEditor. This enables us to query for various model properties without excessive intrusion.

public class DynamicPropertyEditor<Data extends BaseModelData> extends ListModelPropertyEditor<BaseModelData>{ private String propertyName; public DynamicPropertyEditor(String property) { this.propertyName = property; } public String getStringValue(Data value){ return value.get(propertyName); } public String getPropertyName() { return propertyName; } } public class X10ComboBox<D extends BaseModelData> extends ComboBox<D> { private DynamicPropertyEditor<D> dynamicPropertyEditor = null; public DynamicPropertyEditor<D> getDynamicPropertyEditor(){ return dynamicPropertyEditor; } public void setDynamicPropertyEditor(DynamicPropertyEditor<D> dynamicPropertyEditor) { this.dynamicPropertyEditor = dynamicPropertyEditor; } @Override public ListModelPropertyEditor<D> getPropertyEditor() { .... } }

GXT Nuggets: Preventive form validation

Bananas for the code monkey

It is always a good idea to prevent users from doing unwarranted things. Thats the whole idea of client side validation. (Client-side validation is one good way of catching errors before the code hits server, so....blah blah).

The general notion in UI is to accept user input using forms.(XSS, URL mangling etc left aside). In GXT, we can actually mandate/direct the hasty users to enter the correct data without cluttering the UI with error messages. The notion is simple, prevent the buttons from being enabled until all the validations are honored.

This is achieved using a FormButtonBinding object. The actual code:

private FormPanel createLabelSearchPanel(String formHeader, String keywordLabel, String submitLabel) { FormPanel formPanel = new FormPanel(); formPanel.setLabelAlign(FormPanel.LabelAlign.RIGHT); formPanel.setHeading(formHeader); formPanel.setFieldWidth(DashboardConstants.FIELD_WIDTH); formPanel.setLabelWidth(DashboardConstants.LABEL_WIDTH); formPanel.setFrame(true); formPanel.setAutoHeight(true); FormData formData = new FormData("-10"); TextField<BaseModelData> textField = TextFieldFactory.getDefault().getTextField(keywordLabel, "(Enter Search Key)", ""); formPanel.add(textField, formData); Button searchButton = new Button(submitLabel, new SubmitButtonListener(formPanel)); formPanel.getButtonBar().add(searchButton); FormButtonBinding binding = new FormButtonBinding(formPanel); binding.addButton(searchButton); formPanel.setButtonAlign(HorizontalAlignment.CENTER); return formPanel; } ------------------------------------------ private TextField<BaseModelData> createTextField(final String fieldLabel, final String defaultText, final String defaultValue) { final TextField<BaseModelData> textField = new TextField<BaseModelData>(); textField.setFieldLabel(fieldLabel); textField.setEmptyText(defaultText); textField.setSelectOnFocus(true); textField.setAllowBlank(true); return textField; }

Clutter free eclipse installations

The core principle of eclipse is plugin based architecture. This worked so well, that the eclipse eco-system is thriving with a plethora of addons. Earlier versions of eclipse (2.1 and earlier) were simple zip files, you could copy the whole directory structure and run it on any other machine. With the advent of 3.0, it is no longer the case.

Eclipse versions determine the target platform based on the configuration. Incompatible configurations can cause crashes. In such extreme cases, the only resort is to search for all conflicting workspace configurations, reinstall eclipse with required plugins and reload the workspace projects.

The latest versions come with P2 installer. P2 installations simplifies the burden of downloading large zip files. This ibm article describes how to share plugins across different installations.

However, advanced users prefer seggregating plugins under a single location. Like all EMF plugins under one directory and python plugins in a different directory. Then create links to such. The benefit of this approach, you can remove and add features and plugins by a simple add/removal of link file. This also implies less memory utilization, you will be able to tweak you VM args to make use of optimal memory, without burdening the system. Implying better overall performance.

Michael wrote an excellent article in this regards. Happy tweaking!

The punctuation of parsing

Awk scripts when interacting with shell tend to be problematic. Reason awk assumes a different parsing criteria which is incompatible with the shell's own expectation. For e.g: The quotes and whitespaces (',"," ") have a definitive meaning in korn shell, which conflicts with awks. Running the script with

 set -x 

flag shows the details.

So a script snippet that would work on command line

awk 'BEGIN {RS="_EOL@"}' 'END {print $NF}" a.txt

is to be transformed to:

ETL_RECORD_DELIMITER="_EOL@"
cmd=`awk BEGIN{RS="\"$ETL_RECORD_DELIMITER\""}" "END{print" "NR} a.txt`

The white spaces are to be demarcated using " ", if you missed the gist.

Learning everywhere

World is your playground, play if possible; if not learn to play; alteast try to learn to play; you will appreciate things more.

This applies to programming as well. Programmers who know more than one programming paradigms deliver better outcomes. Very early on, I was misinformed that javascript doesn't care for invariants. The folks discussing this were my seniors, had thorough handson experience on C++ and Java. They are reknown for their sane and lucid code. They were working on a GUI generator (HTML UI) and using embedded js for validation. They did not knew unobstrusive JS. All they wanted was some validation on client UI that was realized using

 onclick="validateForm()" 

. The statement now seems silly, but from their point of view, it is akeen to writing business invariants in sed or grep. I couldn't grasp why one could not write class invariants in javascript, so asked them and got a response that in javascript everything was public (access scope). It was only a couple of years later I learnt the better parts.

My intent is not to demean any, but to highlight that even the best of the team is often blinded by competence. If one doesnot empty his pot, how can one fill fresh water! Another example comes from SQL, sql is declarative hence no imperative syntax. While it is true for the most part, certain implementations specific functions say

case...when, coallesce, nullif

can be used to impart imperativeness.

Java developers are a different breed, some just write procedural code, mask it under the name of OOPS and live with it. Some advance to proper OOPS, design and architect properly. A few others master AOP, MDD, ORM and try to be a bit more smart. JavaScript programmers on the other hand are the most eccentric ones. A true master uses functional, procedural, prototyping, event handling/messaging & OOP concepts to the fullest. While it is true that JS allows a whole gamut of things, it is still not fully seen beyond webservers. However, it is one of the few areas that indicates life beyond the know IDE realm.

Learning new languages may or may not be fun, but understanding and appreciating the concepts is definitely delightful. When I started with JRuby, the concept of blocks was the most enticing. The brevity of code was just not plausible in Java. Python had similar givings. When I moved to GWT, the concept of callbacks and message bus usage was obvious to me (naturally, they are defacto occurences in JS apps). I found certain developers crack their heads trying to propogate references/spoil interface definitions all in an attempt to meet the goals. I just wish they read a bit more.

Programmers who claim to be good with languages are not exempt. Ask them to come up with a proper word document, they do it - do it with style corruption. Most of us do not even understand what style/doc corruption is. We donot use styles. If we showed the same zeal towards MSWord, the world would be more pleasent to us. A close analogy is learning to type (finger typing or soft typing). My friends and collegues at times express their awe, well I have trained some portion of my brain to instinctively use a qwerty keyboard, (my friends can do it, but they never cared, if only you saw their stupendously long combo movements on arcade games).

My mom cooks great food. Serving several dishes working on them simultaneously is not a straight chore. I used to wonder how she did that. I observed that she planned ahead, cut her veggies and gathered her spices ahead. Even when the flame is up, she knew the order and quantity required; and accordingly batched her job. All good cooks are great planners, the plans don't always work - but planning is indispensable. Last time I was deploying a project, I took a leaf from her and prepared a check list. After a couple of reviews, we use it pre deployment, to identify what all needs to be done. A thorough deployment without a need to revisit is a small achievement, but a happy one. I am not saying that the deployment worked because I can cook things; all I am saying is that I can appreciate a intangible task in a better way.

All we need to do is keep an open mind to suggestions, and a realization that we are ignorant for the most part. Ignorance is a bliss, knowledge is power. I am happy that I have forgotten my calclus (Ignorance) but not my friends :)

Coder cannon

An article on /. about new developers getting better monetary benefits than seniors made me ponder about IT.

To give an example. In the world of stock market, if one were to estimate the worthiness of a stock/asset, it can be simply done by monitoring the topline growth for a given period. Another option would be find out the profitability of the company (Assets - Liabilities). However, one often finds stocks trading at a much higher price (indicated by P/E). This is very much like real-estate, where the price is driven not by the cost, but by the valuation.

IT valuations are taken more personally - to heart. It is expected that niche technologies deserve niche pays. For the most part, professional IT is about picking some data, transforming it and presenting it. Game devs and System programmers are prominent exceptions. I used to wonder why a SAP analyst makes more than a Java programmer, or a C++ programmer.

Indian IT companies often take a different route. They just force the programmers on to a technology. The results of cold, passionless work are the ones that give a bad name. No university can teach passion. A good organisation and leader can instill a drive, but lack of passion causes one to burnout. Another surprising facet of IT: I recollect seeing a job posting in 2009, asking for GWT experts with 5+yrs of experience. Did the job poster even know when GWT came into existance? On one hand, you can see new technologies as an opportunity to move ahead in life, on the otherhand you may have a "dejavu" moment. Indian IT's cream are adept at catching up with the game, and the whey just sulks.

Western IT is more passionate, they master the one prime thing they chose. I have consultants coming from far more diverse backgrounds into IT. While their IT skills are mediocre (beyond their comfort zone), the rich experience they bring is invalueable. Seniors are valueable, not because they have grown in age but because they can mentor and communicate better. New developers on the other hand can go anywhere. It is either a hit or miss scenario.

If you want your seniors for their soft-skills, make it clear to them. If you want them for their technical skills, tell them clearly the direction being taken. Who knows, some of them may just take a break - learn the new stuff and may prove even more valueable. One harsh thing about large IT projects is the fact that, no one is indispensable. All seniors were once n00bs, they remember that well enough. If they are comfortable in their current roles and don't want to move out, you can always blame the demand-supply curve. We still have cobol dinosaur dragon tamers around, don't we!

JMX - Remote applications

Post jdk 5, JMX has become part of the standard distribution. The default MBeans provided, provide quite a good view of the VM characteristics. However, connecting to remote applications (running on different VM) is not a very straight forward task. The tricky part is the difficult to find documentation about the rmiserver and the URL format to be used for lookup.

Sample JVM_ARGS:

-Dcom.sun.management.jmxremote.port=5000 (put your own non-conflicting port)
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=

which would look like
-Djava.rmi.server.hostname=144.203.88.87

The actual call would look like: java $JVM_ARGS MY_APP. This would invoke the JVM with an JMX Agent running on 5000 port, on a remote server whose ip-address is 144.208.88.87

Now open the jconsole application, invoke your application. Assuming that the application runs for a significant period of time (so that we can actually initiate the handshake and gather stats), enter the following in the Remote Process field

service:jmx:rmi://144.203.88.87:5000/jndi/rmi://144.203.88.87:5000/jmxrmi

Thats it! we are good to go.

Some good pointers on interwebs:

here
here and
here

About SqlLoader

SqlLoader is a utility provided by oracle, for loading data into database. This is analogous to db2export/import provided by db2. Most databases provide such utilities. Typical applications of such utilities: Migration of data (data only), creating a file based backup (as disk space is cheaper than the SAN disk space).

There are two prominent types of sql loader functionality.

• Conventional loading
  
• Direct path loading
  

Conventional loading

Sample command:

 msg=`sqlldr $oracleuser/$oraclepasswd@$ORACLE_SID control=${controlFile} data=${csvFile} direct=false bad=${badFile} log=${logFile} errors=${errors} rows=100000`

Conventional load acts as if someone is firing insert commands using oracle client. The indexes are preserved. Other users are not affected while the load happens.

Direct path loading

Sample command: (Note the direct=true part)

 msg=`sqlldr $oracleuser/$oraclepasswd@$ORACLE_SID control=${controlFile} data=${csvFile} direct=true bad=${badFile} log=${logFile} errors=${errors} skip_index_maintenance=false `

Sqlloader tries to write data directly into database, we can configure it to skip logging, bypass indices and write data directly. However this affects other users/transactions that are trying to read/write data to the same table.

Behind the scenes

Sql loader takes two major inputs. Control file, that dictates the nature of load. Data file that contains the data.
Sample control file for direct path load:

unrecoverable
load data
 APPEND

 into table vendor_data.ds_sp_data
 REENABLE DISABLED_CONSTRAINTS
 fields terminated by "," optionally enclosed by '|'
 trailing nullcols
(type_id FILLER, FEED_DATA_SOURCE FILLER, LINE_NUM, batch_id,  corp_actions_id, split_factor, ric CHAR(20) , record_type CHAR(2) , ds_type_id CHAR(8) , adjust_volume CHAR(1) , ds_file_date DATE "YYYY-MM-DD", split_date DATE "YYYY-MM-DD", file_code CHAR(4) , file_ext CHAR(1) , split_factor_val)

Sample control file for conventional load

OPTIONS (SKIP=1)
load data
 APPEND
 into table vendor_data.feed_run_status_record_type
REENABLE DISABLED_CONSTRAINTS
fields terminated by "," optionally enclosed by '|'
trailing nullcols
(ETL_SEP FILLER,RUN_ID, TYPE_ID, REGION_CODE, FEED_ID, FILE_NAME CHAR(255), RECORD_TYPE, RECORD_COUNT, FAILURE_COUNT, INSERT_COUNT,  STATUS_CODE, EXCEPTION CHAR(3000))

Sample of data loaded by the above:
(Using '------' as line separator of visual illustration only, it is not part of the data)

|21825|,|/u/datamgmt/TEST6/data/datascope/data//18251217.M|,|2|,|414146|,|SFE|,||,|1|,|21825|,|IRF|,|FUT|,|XSFE|,|M|,|NBBc1=|,|RY|,||,|NZ BANKBILL DEC9|,|2010-12-17|,|N|,|1825|,|0|
--------
|21825|,|/u/datamgmt/TEST6/data/datascope/data//18251217.M|,|3|,|414146|,|SFE|,||,|1|,|21825|,|IRF|,|FUT|,|XSFE|,|M|,|NBBc2=|,|RY|,||,|NZ BANKBILL MAR0|,|2010-12-17|,|N|,|1825|,|0|
--------
|21825|,|/u/datamgmt/TEST6/data/datascope/data//18251217.M|,|4|,|414146|,|SFE|,||,|1|,|21825|,|IRF|,|FUT|,|XSFE|,|M|,|NBBc3=|,|RY|,||,|NZ BANKBILL JUN0|,|2010-12-17|,|N|,|1825|,|0|

Note:

1. 'unrecoverable' is supported only in direct path load
   
2. APPEND is faster than INSERT
   
3. FILLER columns are skipped
   
4. CHAR(x) fields help sqlldr load fields that exceed default varchar limit of 127
   
5. SKIP indicates the number of lines to skip
   

A good article on SqlLoader

Extracting data using sql queries

I will just list down the commands that achieve this. The prerequisites for running these are to be managed by the users/admin.

For Db2:

Do a sanity check

db3 "connect to $DB_SVR user $db2user using $db2passwd"
connectionStatus=$?
 if [ $connectionStatus != 0 ]
 then
            #sendmail or raise an alert
        fi

Run the command:

fetchDataSql=" export to ${filePath}_data of del modified by coldel| ${query} "
 
 log "Extracting data using:: ${fetchDataSql}"
 db2 "${fetchDataSql}"
 extractionStatus=$?
 if [ $extractionStatus -eq 0 ] || [ $extractionStatus -eq 2 ]
 then
  log "Preparing the output file"
  echo ${title} > ${filePath}
  cat ${filePath}_data >> ${filePath}
  rm -f ${filePath}_data 
  log "########################################################################################"
  log "##  Data Extraction process completed successfully."
  log "########################################################################################"     
 else
  log "########################################################################################"
  log "##   Alert!!Problems extracting data from ${tableName}, return status::$extractionStatus "
  log "########################################################################################"
  send_mail "Alert!!Problems extracting data from ${tableName}, return status::$extractionStatus" ${FAILURE}
  terminate_db
  exit 4   
 fi

The terminate function just closes the db connection

function terminate_db {
 log "Close database connection"
 db2 "terminate"
}

For Oracle:

Load the oracle client variables into the environment. Ask you admin for details.

#Loading the oracle client
export ORACLE_BASE="/xyz/oracle/sqlnet/11.1.0/product/11.1.0/client_1"
export ORACLE_HOME="$ORACLE_BASE"
export PATH="$ORACLE_BASE/bin":"$ORACLE_BASE/perl/bin":"$ORACLE_BASE/sqldeveloper":$PATH
export LD_LIBRARY_PATH="$ORACLE_BASE/lib":$LD_LIBRARY_PATH
export PERL5LIB="$ORACLE_BASE/perl/lib/5.8.3":"$ORACLE_BASE/perl/lib/site_perl/5.8.3":$PERL5LIB

Query for results

values=`sqlplus  -s ${oracleuser}/${oraclepasswd}@${ORCL_SVR} << EOF
   SET LINESIZE 1000
   SET HEAD OFF
   SET FEEDBACK OFF
   ${query};
   exit;
   EOF`
extractionStatus=$?
if [ $extractionStatus -eq 0 ] || [ $extractionStatus -eq 2 ]
 then
  log "########################################################################################"
  log "##   Preparing the output file::${filePath}"
  log "########################################################################################"
  echo ${title} > ${filePath}
  echo ${values} > ${filePath}_data.txt
  sed 's/VDB_SEP/\n/g' < ${filePath}_data.txt >>${filePath}
  rm -f ${filePath}_data.txt
                log "########################################################################################"
  log "##   Data Extraction process completed successfully."
  log "########################################################################################"     
 else
  log "########################################################################################"
  log "##   Alert!!Problems extracting data from ${tableName}, return status::$extractionStatus "
  log "########################################################################################"
  exit 4   
fi

Paginated fetch queries

Without much fuss,

For DB2:

select * from 
    ( 
        select ROW_NUMBER() OVER () rowNum, t1.* from @tableName@ t1 
        where blah.. blah..        
    ) 
    as t00, 
    ( 
        select count(COL_1) as totalPageRecordCount from @tableName@ t2 
        where blah.. blah..
    ) 
    as t01 
where rowNum between @start@ and @end@

For Oracle:

select min_t.*, count_t.* from 
    ( 
        select rownum as row_num, max_t.* from 
            ( 
                select t1.* 
                from @data_tablename@ t1
                where blah.. blah..
                order by t1.rowid -- order by rowid for consistent results
            ) max_t 
        where  
           rownum <= @end@ 
    ) min_t, 
    ( 
        select count(*) as totalPageRecordCount 
        from @data_tablename@ t1
        where blah.. blah..
    ) count_t 
where 
    row_num >= @start@

Note that ordering by row_id ensures consistent data fetch results.