Understanding SVM Intuitively

 Support Vector Machines (SVM) are a powerful classification and regression technique that separates data using hyperplanes. The method is particularly useful for data with unknown or irregular distributions, where classical assumptions like linearity may not hold.

If we have labeled data, SVM attempts to find the best separating hyperplane between classes. Consider a simple two-class dataset:

  • Red and Blue points represent different classes.
  • Many lines or planes can separate the two classes.
  • The goal is to find the optimal line or hyperplane, which maximizes the distance (margin) from the nearest data points of each class.

Margin (m) is the distance between the nearest points of each class and the hyperplane:

m=2∣∣a∣∣for line y=ax+bm = \frac{2}{||a||} \quad \text{for line } y = ax + bm=∣∣a∣∣2​for line y=ax+b

Maximizing the margin ensures the classifier is robust to noise and unseen test data.

Creating a Sample Dataset in R

We can simulate a simple dataset with two features:

# Sample data
x = 1:20
y = c(3,4,5,4,8,10,10,11,14,20,23,24,32,34,35,37,42,48,53,60)

# Create a data frame
train = data.frame(x, y)

# Plot the data
plot(train, pch = 16)

At first glance, this dataset seems linearly separable, suggesting a linear regression could also be effective.

Linear Regression vs SVM

Linear Regression:

# Fit linear model
model <- lm(y ~ x, train)

# Plot regression line
abline(model)

SVM:

library(e1071)

# Fit SVM model
model_svm <- svm(y ~ x, train)

# Predict values
pred <- predict(model_svm, train)

# Plot predictions
points(train$x, pred, col = "blue", pch = 4)

Comparing Model Performance (RMSE)

# Linear regression RMSE
lm_error <- sqrt(mean(model$residuals^2)) # ~3.83

# SVM RMSE
svm_error <- sqrt(mean((train$y - pred)^2)) # ~2.70

Even with a simple example, SVM outperforms linear regression, demonstrating its robustness.

Tuning SVM for Better Accuracy

SVM performance can be improved by tuning:

  • epsilon: insensitive tube in regression
  • cost: penalty for misclassification

Grid search with cross-validation is straightforward in R:

svm_tune <- tune(svm, y ~ x, data = train,
ranges = list(epsilon = seq(0, 1, 0.01),
cost = 2^(2:9)))

# Best model
best_mod <- svm_tune$best.model
best_mod_pred <- predict(best_mod, train)

# Calculate RMSE
best_mod_RMSE <- sqrt(mean((train$y - best_mod_pred)^2)) # ~1.29

# Plot tuned model predictions
plot(train, pch = 16)
points(train$x, best_mod_pred, col = "blue", pch = 4)

The grid search allows SVM to find optimal hyperparameters, reducing RMSE significantly — in this case from ~2.7 to ~1.29.

Visualizing SVM Tuning

plot(svm_tune)

  • Darker regions = better accuracy
  • Use this plot to narrow the search range for further fine-tuning
  • Avoid overfitting by not using excessively fine steps

Key Takeaways

  1. SVM is robust: It maximizes margins, making it resistant to noisy or biased data.
  2. Linear vs Non-linear:
    • Linear SVM for simple datasets
    • Kernel SVM (RBF, Gaussian) for complex, non-linear data
  4. Tuning matters: Cost and epsilon significantly impact performance
  5. Comparison to Regression:
    • Linear regression works well for truly linear patterns
    • SVM excels when data is noisy or non-linear

Why Use SVM?

  • Works with non-linear and unknown data distributions
  • Provides interpretable hyperplanes
  • Performs well even with high-dimensional data
  • Flexible through kernel methods

Caution: SVM can overfit if tuning isn’t handled carefully, especially on small datasets.

Full R Code

x = 1:20
y = c(3,4,5,4,8,10,10,11,14,20,23,24,32,34,35,37,42,48,53,60)
train = data.frame(x, y)
plot(train, pch=16)

# Linear regression
model <- lm(y ~ x, train)
abline(model)

# SVM
library(e1071)
model_svm <- svm(y ~ x, train)
pred <- predict(model_svm, train)
points(train$x, pred, col="blue", pch=4)

# RMSE
lm_error <- sqrt(mean(model$residuals^2))
svm_error <- sqrt(mean((train$y - pred)^2))

# Grid search for tuning
svm_tune <- tune(svm, y ~ x, data=train,
ranges=list(epsilon=seq(0,1,0.01),
cost=2^(2:9)))
best_mod <- svm_tune$best.model
best_mod_pred <- predict(best_mod, train)
best_mod_RMSE <- sqrt(mean((train$y - best_mod_pred)^2))

# Plot results
plot(svm_tune)
plot(train, pch=16)
points(train$x, best_mod_pred, col="blue", pch=4)

This version is structured, beginner-friendly, and ready for professional use, highlighting both conceptual understanding and hands-on R implementation.

At Perceptive Analytics, we help organizations unlock actionable insights from their data. Recognized among leading AI Consulting Companies, we guide businesses in adopting AI solutions that enhance forecasting, automate processes, and improve decision-making. Organizations looking to hire Power BI consultants rely on us to build scalable dashboards, automate reporting, and provide real-time insights that drive smarter business outcomes.

Comments

Post a Comment

Popular posts from this blog

Implementing Log-linear Regression in R

  In this article, we discuss widely used Generalized Linear Models (GLMs) in the industry, focusing on: Log-linear regression Interpreting log-transformations Binary logistic regression We also review the underlying distributions and applicable link functions. To illustrate concepts in R, we use the sample datasets: cola.csv penalty.csv Introduction to Generalized Linear Models A Generalized Linear Model (GLM) represents the dependent variable as a function of independent variables. The traditional form of GLM is simple linear regression, which assumes that the dependent variable is normally distributed. However, in real-world scenarios, this assumption is often violated. For example, if the dependent variable is the number of coffee cups sold (always positive and often with a fat-tailed distribution) and the independent variable is temperature, simple linear regression may produce nonsensical results — e.g., predicting negative sales at low temperatures. GLMs overcome these limit...
Read more

Keep Customers. Protect Revenue.

Image
  In fraud detection services, keeping your best customers is just as important as finding new ones. But if you can't see who’s disengaging—or why—you risk losing revenue without warning. We partnered with a top fraud detection firm to build an Attrition Forecast Dashboard that gave them a clear view of account health and user behavior. With it, they could: Spot high-risk accounts before they churn Track how customers were using the product Take action to boost engagement and retention The result? Lower churn, stronger customer ties, and steadier revenue. 📘 Explore the Full Case Study: Attrition Forecast Dashboard Case Study 📥 Download the Executive Summary PDF: Download the Attrition Forecast Dashboard PDF
Read more

True Parallelism in R Using the parallel Package

  Modern computers no longer rely on a single processor. Most laptops and servers today ship with multi-core CPUs and large amounts of memory. Yet, many R scripts still run sequentially—using only a fraction of the available hardware power. If a task can be done faster by splitting it across multiple cores, why not do it? Parallel processing allows you to divide large workloads into smaller chunks and execute them simultaneously. This can reduce execution time from hours to minutes, or even seconds in some cases. Code efficiency has become a critical skill in data science, machine learning, and analytics engineering, and parallel programming is one of the best ways to achieve that efficiency in R. Once you learn parallel programming, you’ll likely wonder why you didn’t adopt it sooner. Understanding Sequential vs Parallel Execution Before diving into code, it’s important to understand the difference: Sequential execution Runs tasks one after another Uses a single CPU core Easier to...
Read more