If you’ve read through how Support Vector Machines work, you probably know the linear simple SVM might not work in all cases… but how does it fail? Let’s take a look at an example I tried like to my simple example… but change it to be a larger space than just 4, and separated with a region in the middle, and the region around it (positive, negative labelled areas to learn):
#See also https://github.com/rogue-hack-lab/PythonMachineLearningSVMExamples/blob/master/coffee.py from __future__ import division # [row,col] coordinates: data = [[ 0,0 ], [ 0,1 ], [ 0,2 ], [ 0,3 ], [ 1,0 ], [ 1,1 ], [ 1,2 ], [ 1,3 ], [ 2,0 ], [ 2,1 ], [ 2,2 ], [ 2,3 ], [ 3,0 ], [ 3,1 ], [ 3,2 ], [ 3,3 ], ] #Separating x,y coordinates in nonlinear (circle center, only center positive): category = [ -1, -1, -1, -1, -1, 1, 1, -1, -1, 1, 1, -1, -1, -1, -1, -1, ] import numpy from sklearn.svm import SVC clf = SVC(kernel='linear', C=1) clf.fit(data, category) #Get m coefficients: coef = clf.coef_[0] b = clf.intercept_[0] print('This is the M*X+b=0 equation...') print('M=%s' % (coef)) print('b=%s' % (b)) print('So the equation of the separating line in this 2d svm is:') print('%f*x + %f*y + %f = 0' % (coef[0],coef[1],b)) print('The support vector limit lines are:') print('%f*x + %f*y + %f = -1' % (coef[0],coef[1],b)) print('%f*x + %f*y + %f = 1' % (coef[0],coef[1],b)) vertmatrix = [[x] for x in coef] good = 0 bad = 0 for i, d in enumerate(data): #i-th element, d in data: calculatedValue = numpy.dot(d, vertmatrix)[0] + b print( 'Mx+b for x=%s calculates to %s' % (d, calculatedValue) ) if calculatedValue > 0 and category[i] > 0: good += 1 elif calculatedValue < 0 and category[i] < 0: good += 1 else: bad +=1 #they should have matched category. print('accuracy=%f' % (good/(good+bad)) ) #The same as the builtin "score" accuracy: print('accuracy=%f' % clf.score(data, category) ) while True: #Try other points d = input('vector=') calculatedValue = numpy.dot(d, vertmatrix)[0] + b print(calculatedValue)
Now the result is not great, the linear separator can’t do anything in this case and the best found is something that makes all given points the negative:
This is the MX+b=0 equation… M=[0. 0.] b=-1.0 So the equation of the separating line in this 2d svm is: 0.000000x + 0.000000y + -1.000000 = 0 The support vector limit lines are: 0.000000x + 0.000000y + -1.000000 = -1 0.000000x + 0.000000*y + -1.000000 = 1
Mx+b for x=[0, 0] calculates to -1.0
Mx+b for x=[0, 1] calculates to -1.0
Mx+b for x=[0, 2] calculates to -1.0
Mx+b for x=[0, 3] calculates to -1.0
Mx+b for x=[1, 0] calculates to -1.0
Mx+b for x=[1, 1] calculates to -1.0
Mx+b for x=[1, 2] calculates to -1.0
Mx+b for x=[1, 3] calculates to -1.0
Mx+b for x=[2, 0] calculates to -1.0
Mx+b for x=[2, 1] calculates to -1.0
Mx+b for x=[2, 2] calculates to -1.0
Mx+b for x=[2, 3] calculates to -1.0
Mx+b for x=[3, 0] calculates to -1.0
Mx+b for x=[3, 1] calculates to -1.0
Mx+b for x=[3, 2] calculates to -1.0
Mx+b for x=[3, 3] calculates to -1.0
accuracy=0.750000
accuracy=0.750000
So what if we use a Kernel function as is often used with SVMs? I set this line to find out:
clf = SVC(kernel='rbf', C=1)
and removed the _coef and logic that of course can’t work with a non linear one:
from __future__ import division # [row,col] coordinates: data = [[ 0,0 ], [ 0,1 ], [ 0,2 ], [ 0,3 ], [ 1,0 ], [ 1,1 ], [ 1,2 ], [ 1,3 ], [ 2,0 ], [ 2,1 ], [ 2,2 ], [ 2,3 ], [ 3,0 ], [ 3,1 ], [ 3,2 ], [ 3,3 ], ] #Separating x,y coordinates in nonlinear (circle center, only center positive): category = [ -1, -1, -1, -1, -1, 1, 1, -1, -1, 1, 1, -1, -1, -1, -1, -1, ] import numpy from sklearn.svm import SVC clf = SVC(kernel='rbf', C=1) clf.fit(data, category) print('accuracy=%f' % clf.score(data, category) ) while True: #Try other points d = input('vector=') calculatedValue = clf.predict(eval(d)) print(calculatedValue)
With this revised code I can see that the output is 100% accuracy and separates inputs by that area centered on [1.5,1.5]:
accuracy=1.000000
vector=[[1,1]]
[1]
vector=[[-1,-1]]
[-1]
vector=[[1.5, 1.5]]
[1]
vector=[[3.1, 3.1]]
[-1]
For more information on how SVM can work in Scikit learn, check out their full documentation!