If you’ve read through how Support Vector Machines work, you probably know the linear simple SVM might not work in all cases… but how does it fail? Let’s take a look at an example I tried like to my simple example… but change it to be a larger space than just 4, and separated with a region in the middle, and the region around it (positive, negative labelled areas to learn):
#See also https://github.com/rogue-hack-lab/PythonMachineLearningSVMExamples/blob/master/coffee.py
from __future__ import division
# [row,col] coordinates:
data = [[ 0,0 ], [ 0,1 ], [ 0,2 ], [ 0,3 ],
[ 1,0 ], [ 1,1 ], [ 1,2 ], [ 1,3 ],
[ 2,0 ], [ 2,1 ], [ 2,2 ], [ 2,3 ],
[ 3,0 ], [ 3,1 ], [ 3,2 ], [ 3,3 ],
]
#Separating x,y coordinates in nonlinear (circle center, only center positive):
category = [ -1, -1, -1, -1,
-1, 1, 1, -1,
-1, 1, 1, -1,
-1, -1, -1, -1, ]
import numpy
from sklearn.svm import SVC
clf = SVC(kernel='linear', C=1)
clf.fit(data, category)
#Get m coefficients:
coef = clf.coef_[0]
b = clf.intercept_[0]
print('This is the M*X+b=0 equation...')
print('M=%s' % (coef))
print('b=%s' % (b))
print('So the equation of the separating line in this 2d svm is:')
print('%f*x + %f*y + %f = 0' % (coef[0],coef[1],b))
print('The support vector limit lines are:')
print('%f*x + %f*y + %f = -1' % (coef[0],coef[1],b))
print('%f*x + %f*y + %f = 1' % (coef[0],coef[1],b))
vertmatrix = [[x] for x in coef]
good = 0
bad = 0
for i, d in enumerate(data):
#i-th element, d in data:
calculatedValue = numpy.dot(d, vertmatrix)[0] + b
print( 'Mx+b for x=%s calculates to %s' % (d, calculatedValue) )
if calculatedValue > 0 and category[i] > 0:
good += 1
elif calculatedValue < 0 and category[i] < 0:
good += 1
else:
bad +=1 #they should have matched category.
print('accuracy=%f' % (good/(good+bad)) )
#The same as the builtin "score" accuracy:
print('accuracy=%f' % clf.score(data, category) )
while True:
#Try other points
d = input('vector=')
calculatedValue = numpy.dot(d, vertmatrix)[0] + b
print(calculatedValue)
Now the result is not great, the linear separator can’t do anything in this case and the best found is something that makes all given points the negative:
This is the MX+b=0 equation… M=[0. 0.] b=-1.0 So the equation of the separating line in this 2d svm is: 0.000000x + 0.000000y + -1.000000 = 0 The support vector limit lines are: 0.000000x + 0.000000y + -1.000000 = -1 0.000000x + 0.000000*y + -1.000000 = 1
Mx+b for x=[0, 0] calculates to -1.0
Mx+b for x=[0, 1] calculates to -1.0
Mx+b for x=[0, 2] calculates to -1.0
Mx+b for x=[0, 3] calculates to -1.0
Mx+b for x=[1, 0] calculates to -1.0
Mx+b for x=[1, 1] calculates to -1.0
Mx+b for x=[1, 2] calculates to -1.0
Mx+b for x=[1, 3] calculates to -1.0
Mx+b for x=[2, 0] calculates to -1.0
Mx+b for x=[2, 1] calculates to -1.0
Mx+b for x=[2, 2] calculates to -1.0
Mx+b for x=[2, 3] calculates to -1.0
Mx+b for x=[3, 0] calculates to -1.0
Mx+b for x=[3, 1] calculates to -1.0
Mx+b for x=[3, 2] calculates to -1.0
Mx+b for x=[3, 3] calculates to -1.0
accuracy=0.750000
accuracy=0.750000So what if we use a Kernel function as is often used with SVMs? I set this line to find out:
clf = SVC(kernel='rbf', C=1)
and removed the _coef and logic that of course can’t work with a non linear one:
from __future__ import division
# [row,col] coordinates:
data = [[ 0,0 ], [ 0,1 ], [ 0,2 ], [ 0,3 ],
[ 1,0 ], [ 1,1 ], [ 1,2 ], [ 1,3 ],
[ 2,0 ], [ 2,1 ], [ 2,2 ], [ 2,3 ],
[ 3,0 ], [ 3,1 ], [ 3,2 ], [ 3,3 ],
]
#Separating x,y coordinates in nonlinear (circle center, only center positive):
category = [ -1, -1, -1, -1,
-1, 1, 1, -1,
-1, 1, 1, -1,
-1, -1, -1, -1, ]
import numpy
from sklearn.svm import SVC
clf = SVC(kernel='rbf', C=1)
clf.fit(data, category)
print('accuracy=%f' % clf.score(data, category) )
while True:
#Try other points
d = input('vector=')
calculatedValue = clf.predict(eval(d))
print(calculatedValue)
With this revised code I can see that the output is 100% accuracy and separates inputs by that area centered on [1.5,1.5]:
accuracy=1.000000
vector=[[1,1]]
[1]
vector=[[-1,-1]]
[-1]
vector=[[1.5, 1.5]]
[1]
vector=[[3.1, 3.1]]
[-1]For more information on how SVM can work in Scikit learn, check out their full documentation!