- Intro
- Problem formulation
- A first attempt at constructing deepSVM Loss
- Tackle the first constraint
- Second constraint :
- Deriving sufficient conditions for equivalence: The first case
- The second case:

### Intro

For classification tasks, our deep net can be seen as having two parts: a feature extractor and linear classifier. Innovations usually comes from engineering the former - there is much more room to do so. It is therefore natural to wonder what if we rewire the later instead? A first experiment to do is replacing the usual softmax with some other well-known linear classifiers. Namely in this post I explore the soft-margined Support Vector Machine option, trained in One-Vs-Rest fashion.

Notice that since the classifier and feature extractor is co-trained, replacing the classifer will change the kind of features learnt by the feature extractor. So the impact is network-wide and thus, do worth some time to consider.

The rest of this post refreshes the mathematical foundation of SVM. I then proceed to investigate if this formulation is compatible with deep feature extractor.

### Problem formulation

Say you had this deep feature extractor and would like to plug onto it the final layer being an SVM. in this case is explicit and there is no way of computing without computing and first. There is no kernel trick this time and the only thing we want to do here is defining a loss that **1)** allow gradients flowing backwards and **2)** faithfully follow the definition of SVM. From the previous part, we know that amounts to the following problem:

### A first attempt at constructing deepSVM Loss

We can see that the construction of upon involves differentiable steps, thus back-propagation can comfortably flow the gradient all the way back from to . It is, however, unclear how to incorporate the constraints into this optimization procedure. The general consensus, I found out, is to simply forget about the constraints and add new penalty terms into the objective to discourage violation. That means, for multi-class hard-margin SVM, the objective is as followed:

Where , is the number of class and and is the corresponding of decision surface number - remember we are optimizing a One-vs-All SVM. The loss function for soft-margin is constructed in a similar fashion

### Tackle the first constraint

Apparently the above objectives are not mathematically equivalent to the original one, so plugging such a thing on top of is not exactly doing SVM. Here we try to refine the soft-margin objective presented above to get it as close to the original as possible. Observe that the first constraint simply establish a domain restriction on , but we know that auto differentiation will produce gradients that push below whenever needed. So an obvious thing to do is to clip the gradient whenever it makes falls below zero.

But there is a clever mathematical trick to this that accomplishes both: a function that allows optimization over whole-axis domain *and* produce stationary gradient at zero, thus get rid of the need of gradient clipping *as long as* the variable is initialised to be non-negative. This is the function , whose derivative becomes zero if and only if . By replacing , we are now able to optimise over freely and obtain a better approximation to our original Loss function, i.e. get clear of the penalty:

### Second constraint :

At this point, it is useful to look at what we are trying to approximate:

Going in the same direction as vanilla SVM to produce the Primal problem, we know that the following Lagrangian is equivalent to the original one:

Realise that is strikingly similar to . In fact, the similarity is not only syntactic, but also semantic. Suppose , then can not be positive, setting to zero is the only way to obtain maximization in . For it is clear that there is no room for optimization over since is always . So either way, .

This gives the incentive to prove that is also similar to in its behavior: a non-constrained objective that achieve the same result to the original one. In fact, we have already done one of the two checks: Whenever satisfies **all** constraints, minimizing is the same as minimizing since . This left us with the other, more difficult check: that **some** . We want this to work in a similar fashion, i.e. to prove that cannot be a local minimum. As a reminder, the primal objective can be summarised as:

where feasible.

elsewhere, clearly not a minimum.

While so far,

where feasible.

finite elsewhere, but hopefully not a local minimum.

### Deriving sufficient conditions for equivalence: The first case

Again, we look at the simpler case: Suppose violates **all** constraints. Now have a look at , where becomes identity:

Expand and get rid of the constants, we obtain an equivalent objective:

Taking the derivative with respect to and set them to zero to see the necessary conditions of local minima. Assuming is a non zero constant, we obtain:

From this point, you will quickly see that arguments supporting overpower its opponents.

Let’s consider cases where at least one of the above conditions is immediately wrong. is clearly not true if since by definition, the construction of requires , which cannot be zero as suggests if . To prove this, consider the aggregated matrix with entries: it has a single entry for each column and for all the remainings, and the summation is taking over all of its entries, thus the equality.

Some will close the case at this point and conclude victory for , but let’s just assume to see how far we can go. Note that does not means automatically becomes true, this would require another assumption: an equal number of positive and negative training data points. Assume this assumption is satisfied, it can easily be broken by augmenting data points to produce unbalanced size of training examples, or simply make sure that **N is odd**.

### The second case:

This is where things get messy. For some of the K classifiers in our One-vs-All scheme, there exists some data points that violates the second constraint, i.e. such that

Since the K classifiers are independent and the loss is simply a sum of their losses, we can consider separately one of them and get rid of the subscript for simplicity of notation. Let be the set of data points that violate the above constraint with respect to this specific classifier, then our objective becomes

Where if and if otherwise. Compute the derivatives and set them to zero and we obtain the necessary conditions:

We generally set , so the third condition essentially means if the current parameters are stationary, then all must be zero. By definition, for any data point , the following inequality holds

Subtitute by the first condition and we get:

Now since this holds for all , we can take the sum across all posible value of :

reduces to zero thanks to the second condition. Denote as the vector whose entries are all and the kernel for dot product . This inequality is vectorized to

Notice that each entry of is either or , and length of is exactly the size of , the final form of our violated constraint is thus:

We know with such value of , value of lies between and - the smallest and largest eigenvalue of positive definite . Since we are trying to prove this will not hold, the sure-fire way is to set