Assessing Programming Skills of Coders Validly and Scientifically – Automated Simulated Programming Assessment

Simulated Assessments for Employment

It's an old saying that the best way to assess a candidate for any job is to put him into the job for ten days and evaluate his ability to fulfill job responsibilities successfully. Fortunately or unfortunately time and other constraints make this practically impossible to follow. But imagine if we could create an environment which could assess exactly the same things but in much lesser time. What would be a better way to assess a teacher than to leave her/him for 2 hours in a class and analyze her/his conduct. Similarly to assess a sales person one can just ask him to sell the product to the interviewer, or for an IT engineer to write a code and be evaluated on the same.

The argument for simulated assessments are far and many to be outnumbered by allegations that a simulated environment can never completely mirror the real thing. Sure it can't, but it's the closest competitor. Puzzles, group discussions or tricky questions can tell only give you an indicative idea of candidate's analytical and numerical skills, his trainability and power of understanding. But to assess whether this smart person knows coding or not, you would still have to ask him to write a code and evaluate it.

 

Developing Simulated Programming Assessment

Simulations typically include automated performance assessment to overcome the issues of uniformity and objectivity and eliminate the need for extensive one-on-one time with an assessor. However, simulations are much more complicated to develop than standard tests, and ensuring validity and reliability is a huge challenge. There are no standard guidelines for developing a simulation based assessment.

A simulated computer programming assessment should meet many parameters in order to qualify as a valid and reliable assessment for evaluating the programming skills of candidates at the time of recruitment. Let us explore the various important parameters in detail:

  1. Standardization- In order to have a fair and valid evaluation, the simulated programming assessment should be standardized. If two candidates see two different questions, then they should be of same difficulty level. This ensures that if A and B get a score of 100, it indicates that their coding skills are at par, irrespective of the kind of questions they got during test.
  2. Granularity- Most simulated programming assessment today give a binary score. If the code runs successfully then you score 100, otherwise 0. This would work only for a small population of candidates but what about candidates who could have scored between 1 to 99. The automated assessment should check the code for various parameters like:
    • Correctness of the code
      • Whether the code runs for basic as well as advanced test cases
      • Whether the code runs for boundary cases
    • Quality of the code
      • Use of industry-defined best practices in the code
    • Efficiency of the code
      • Average case time Complexity of the code.
      • How many warnings and errors were generated.
      • How many compilations witnessed run time and time-out errors etc.,
      • Time take to submit the final code etc.,
  3. Use case- Different companies would like to evaluate candidates on different skills sets of coding. For example an IT Services company would want to evaluate candidates on basic program design, assessing fundamentals in programming concepts like iteration, condition-based programs etc. Whereas an IT Product company would want to evaluate on not just fundamentals but also on Algorithm Design and Data Structures.

 

Evaluating Indian Engineers on their Coding Skills

Once a laggard in economic growth, India successfully rode the IT revolution to be one of the most favored locations for the big players of the IT world. To claim their share of the pie, a number of multinational giants have forayed into the Indian IT market, setting up their offshore development centers in the country. With more than 6 lac engineers graduating every year with the IT sector as one of their hottest career choice, the supply side seems quite promising and strong on paper. The reality however is quite different. Aspiring Minds' Computer Programming Learning Levels Report on Engineers is a study on 55,000+ engineers on their understanding of computer programming knowledge and application. Results indicate that around 50–60% CS/IT engineers do not understand subtleties of programming concepts, while more than 80% are unable to apply them to real-world situations. This clearly indicates that there is a strong need for using automated simulated programming assessments at the time of recruitment to select the candidates with strong coding skills. Manual evaluation of coding is not only time taking but also has limited usage as it can't evaluate a program on various quality and efficiency parameters. Automated assessments remove human bias and subjectivity from evaluation and provide an objective and detailed feedback to the interviewer on the candidate's programming skills, in a timely and efficient way.