Insertion Sort

• The array sorting problem is as follows:
  Given n numbers A[1..n], rearrange them so that A[1] ≤ A[2] ≤ … ≤ A[n-1] ≤ A[n].
• Insertion sort is a simple algorithm, and works well for very short files.

Iterative Insertion Sort

• High-level Algorithm: repeatedly insert the next unsorted number A[j] into its correct position in the sorted subarray A[1..j-1].

  for j ← 2 to n do
      insert A[j] in its correct position in A[1..j-1]

Example

• Given input array of numbers 12, 8, 1, 10, 4, 7, 9, 2, the array values at the beginning of each successive iteration are

  12   8←  1  10   4   7   9   2
   8  12   1← 10   4   7   9   2
   1   8  12  10←  4   7   9   2
       .           .           .
       .           .           .
       .           .           .
   1   4   7   8   9  10  12   2←
   1   2   4   7   8   9  10  12 

Iterative Insertion Sort, continued

• Low-level Algorithm:

  A[0] ← -∞
  for j ← 2 to n do
  {
      oldAj ← A[j]
      i ← j-1
      while A[i] > oldAj do
      {
          A[i+1] ← A[i]
          i ← i-1
      }
      A[i+1] ← oldAj
  }

• Note: This algorithm illustrates how a sentinel helps simplify the code.

Correctness

• It suffices to show that
  1. If the for loop terminates, A contains all and only the original elements in sorted order.
  2. The for loop does terminate.
• To prove (1), we come up with this claim.
  Loop Invariant: At the start of the iteration of the for loop, the subarray A[0..j-1] consists of the elements that were originally in A[0..j-1] but in sorted order.
• We prove the loop invariant by mathematical induction.
  Base Case. A[0..1] is sorted before the for loop starts, i.e., the invariant holds when j is 2.
  Induction Step. Assume the invariant holds at a particular value of j, it will continue to hold when j is incremented by 1.

Running Time Analysis

• High-level algorithm version:

  for j ← 2 to n do
      insert A[j] in its correct position in A[1..j-1]

  • There are \(O(n)\) iterations of the for loop.
  • Each iteration takes time \(O(n)\) in any reasonable implementation since we insert A[j] by comparing it to at most every other member.
  • So the total time is \(O(n)\) x \(O(n)\) = \(O(n^2)\).

Running Time Analysis

• low-level algorithm version:
  We do this by labeling each statement with the amount of time it takes to execute once. To do this, work outwards from the innermost blocks.

   1. O(1)   A[0] ← -∞
   2. O(n^2) for j ← 2 to n do
   3. O(n)   {
   4. O(1)       oldAj ← A[j]
   5. O(1)       i ← j-1
   6. O(n)       while A[i] > oldAj do
   7. O(1)       {
   8. O(1)           A[i+1] ← A[i]
   9. O(1)           i ← i-1
  10.            }
  11. O(1)       A[i+1] ← oldAj
  12.        }

  Thus, the worst-case running time of insertion sort is \(O(n^2)\).

• Note: Instead of labeling each statement by the amount of time it takes to execute once, we can also label it by the amount of time it takes to execute throughout the algorithm.

Recursive Insertion Sort

• The algorithm consists of the procedure insertion-sort that calls procedure insert.

  // Precondition: Array A contains integers.
  // Postcondition: Array A contains the original numbers in sorted order.
  insertion-sort(A) {
      n ←  A.length
      if n > 1 then {
          insertion-sort(A[1..n-1])
          insert(n)
      }
  }
  
  // Precondition: Subarray A[1..j-1] is sorted.  
  // Postcondition: Subarray A[1..j] contains original elements in A[1..j] in sorted order.  
  insert(j) {
      if j=1 then return
      else if A[j-1] < A[j] then return
      else {
          swap A[j-1] & A[j]
          insert(j-1)
      }
  }

Correctness

• First we show insert(j) works correctly, then show insertion-sort(A) works correctly.
• For each task we show that if the precondition holds before the procedure starts, then the postcondition holds when the procedure finishes.
• For correctness of insert(j) we prove by induction on j.
• For correctness of insertion-sort(A) we prove by induction on length of A.

Running Time Analysis

• Let I(j) be the worst-case running time of insert(j). The recurrence for I(j) is \[ I(j) = \left\{ \begin{array}{ll} O(1) & \mbox{if $j = 1$} \\ O(1) + I(j-1) & \mbox{if $j > 1$.} \end{array} \right. \] Iterating the recurrence gives I(j) = O(j).
• Let T(n) be the worst-case running time of insertion-sort(n). The recurrence for T(n) is \[ T(n) = \left\{ \begin{array}{ll} O(1) & \mbox{if $n = 1$} \\ T(n-1) + I(n) & \mbox{if $n > 1$.} \end{array} \right. \]
• Pluggging in I(n) = O(n) and iterating the recurrence gives T(n) = O(n²).

Notes

1. The items to be sorted do not have to be numbers. They can be characters, strings, etc. In general, they can come from any totally-ordered universe.
2. We can show that the (worst-case) running time of insertion sort is Theta(n²) by proving the big-Omega estimate separately from proving the big-Oh estimate. (Use arrays that are already sorted in decreasing order.)
3. When we can prove that the big-O estimate of an algorithm agrees with its big-Theta estimate, we say that we get a tight estimate.
4. The exact time bound for insertion sort is Theta(n+I) where I is the number of inversions in the input. This explains why insertion sort is fast on files that are almost-sorted.

Exercise

• Explain why it’s a bad idea to implement the phrase “insert A[j] in its correct position in A[1..j-1]” of the high-level algorithm by scanning the subarray A[1..j-1] from the left instead of from the right as we did in this handout.