Distil | The Content Protection Network | www.distil.it

As a marketer, my life revolves around metrics. My work and day-to-day activities can be shown through how many visitors I bring to our site AND who they are. Obviously the more people I bring the better my team looks– so if I were to just bring thousands of pageviews each day my job would be done, right?

WRONG!

I’ve known many marketing teams that are solely judged on the amount of traffic they bring in – uniques per month, pageviews, duration of pageviews, etc. But are all pageviews equal?  Who’s to say the number reported isn’t correct? Hopefully my fellow marketers won’t hate me by saying this – but not all pageviews count equally as valuable interactions on your website.

Sure it looks great to your management team that you’re bringing in thousands or even 10s of millions of pageviews – but when the sales team turns around and asks where are the real customers and why none are converting, you are sent searching for answers.  What if your total pageview traffic is really a fraction of what your analytics tools are reporting? Yes, it surely is a lower number unless you are able to discern the bots and webscrapers that competitors and other data mining companies are running on your site…  Are the numbers you report actually accounting for malicious bot traffic visiting your site without you even knowing?

What if you are able to identify all the bot pageviews from the human pageviews visiting your website?  What if 10% of your traffic is bot generated pageviews… that means your data are diluted and distorted.  Well sit down, because the reality is that some sites have 20% to 60% of traffic from bots, making it very difficult to report on for marketing and driving up operational costs.

Bottom line for my fellow marketers: don’t just consider the number Google Analytics gives you every day for your website data.  You can save operational costs, improve your website performance, and measure the results of marketing when you have the ability to filter out bot traffic. With the correct data, it helps your marketing team properly set goals that are attainable and they can do their analysis more effectively.

For the CEOs, COOs, and CMOs – make sure your team has the right tools to do their jobs effectively – ask them how you are mitigating web scrapers and automated bot traffic sessions.  After all, bad data doesn’t help anyone and inbound bot traffic does not grow your revenue  – ensure the web traffic which you report is humans looking at your pages and not just malicious bots scraping your website.

To learn more about how bots may be infiltrating your website and marketing metrics download this white paper.

As I stepped back from the day-to-day meetings and started my planning for the coming weeks and months, it dawned on me suddenly that April was over. I realized amongst my many activities, I lost sight of one enormous milestone.

On April 13th, Distil Networks turned two years old … *&^%&.  How have two years flown by so quickly? I recognize in the big scheme of things we are just getting started with our company; but two years, two rounds of funding and tremendous customer growth are nothing to scoff at. How could I let myself forget such an important milestone?

As the CEO, during these past few months my travel schedule has increased dramatically and I find it hard to ever have time to relax – but it is important to step back and look around, acknowledge all our accomplishments and hard work…  I know it’s all worth it. Our team has tripled in size and everyone is moving at 100 miles/minute – it feels like just yesterday Andrew, Engin and I were sitting in my living room, developing the technology – talking to anyone who would listen, and signing up some of our earliest adopters.

We often lose sight of our past achievements as we constantly push one-another and look forward to stay on the leading edge and strategize our next move. Today served as a great lesson for me and hopefully will too for every startup founder. Stop. Take a moment to look back on what you’ve accomplished so far. Enjoy the progress you’ve made. Then give thanks to everyone who helped you get there, because you sure as hell didn’t do it on your own.  For me that means primarily my team but also our mentors. Thank you all for helping us to accomplish everything we’ve done so far:

Distil is born – April 2011

Distil Alpha – September 2011

Accepted into Techstars October 2011

Distil Beta – December 2011

$400k Raised – January 2012

Techstars Begins – January 2012

Distil General Availability – April 2012

$1.8M Raised – December 2012

Distil Rebranded to Distil Networks – April 2013

So here’s to the next two years- and beyond – may we grow in knowledge, success and of course revenue.

Cheers!

Before becoming a co-founder of Distil Networks, my background was in writing bots that scraped web pages. Every day I was deploying new bots that logged into websites, scraped their data and dumped it all in my local database. None of this was actually done for malicious reasons, but I was still launching 10,000+ requests an hour at a server that probably didn’t get that many requests a day.

Eventually, they blocked my IP. The hours of work I spent writing the perfect scraper went down the drain…

Until I took 10 seconds and changed my IP. After that I was back to scraping.

The problem is that (nowadays in particular) an IP address is cheap, easy to come by, and most importantly, on-demand. If every IP available at my house got blocked, I’d use my work IP. If all available work IPs got blocked: off to the cloud! I could spin up a new Amazon EC2 instance or SoftLayer Cloud instance or Windows Azure instance.  Exhausted all those options? Time for other cloud providers (Google, Rackspace, HP, etc) or the Tor network.

This is why we talk a lot with our customers about the cat and mouse game of bot detection and blocking. For every IP you block, there are millions of other potential IPs someone could be using to scrape your data or attack your website. Chasing down every IP and finding every possible entry point is beyond frustrating – it’s downright maddening. Worst of all, it’s something thousands upon thousands of website owners are doing daily.

This was a problem we knew we wanted to solve when we first started building Distil Networks. If we were really going to help our customers, we had to do more than just automated IP blocking. With that in mind, we stopped looking at where the request was coming from and we started looking at what the bot was doing, what capabilities the bot had, and how the bot was traversing your website. This led to the generation of our Distil Networks bot signatures – IP address agnostic combinations of all this information that we can use to define a bot, regardless of where it came from originally.

Currently, Distil Networks generates a signature for every device that accesses a page we protect. Once a bot has been detected, we take that bot’s specific signature and propagate it out to our worldwide network to ensure that no matter where a bot shows up, we can see it and stop it.

All of this probably makes it seem as though we don’t value IP information, but we do. If we don’t have any previous record of a device signature, we activate a level within our bot blocking technology that looks at the owner of a given IP address and uses that to determine a baseline likelihood that a request is coming from a bot. For example, if a request comes from Comcast in Washington DC, more often than not, it’s just a normal person browsing your webpage. If that same request came from Amazon East or SoftLayer Dallas our network will begin to monitor the connection more closely since the requestor is far more likely to be a bot. It’s by no means the end-all-be-all of detection, but it is a data point we’d be foolish to ignore.

Basically, blocking by IP address can be an effective, but really a temporary-at-best solution. Unless you catch one of these attacks as they’re occurring, there’s no guarantee that the IP you spent hours scouring your server logs for is even the IP that the next bot attempt will come from.  It’s like a game of Whac-A-Mole that never ends.

At Distil Networks, we truly believe that signature based blocking is going to be the future of bot security online. If you’re currently fighting a war against a bot that’s attacking or scraping your website, we can help.

Recently, Distil Networks released a new report, Trap Analysis and Statistics, in the Distil Admin Portal, which gives a more detailed overview of bot traps and the IP addresses triggering them. There are a number of traps in place for detecting malicious bots including User Agent checks, browser integrity checks, rate limiting, and many others. When a malicious bot attempts to access a site protected by Distil Networks, it triggers a trap. The bot’s request is then blocked, monitored or shown a CAPTCHA page to verify it’s a human visitor. The action taken can be configured within the Content Protection Settings page in the Distil Admin Portal.

The Trap Analysis and Statistics report displays a list of triggered traps and the number of violations for each. The IP address is captured for each violation allowing a more granular level of control into how the content is accessed. Traps are sorted by the number of violations in descending order with an accompanying bar chart for a quick visual overview into which traps are most frequently triggered.

Selecting a trap in the report will display a list of the IP addresses and the number of violations per IP. If a particular IP address is showing a high number of violations, selecting it will display the WHOIS information, which allows better insight into the bots origin.

IP addresses can be individually blacklisted or whitelisted. All requests from a blacklisted IP address will be blocked until the IP address is removed from the blacklist. Whitelisted IP addresses will never be blocked despite the traps it triggers. This can be especially useful to allow internal tools access, such as automation test tools, that can be mistaken as malicious bots.

Distil Networks releases product updates and enhancements frequently and they will periodically be featured on the blog. Be sure to subscribe to receive the latest posts in your inbox.

Here at Distil Networks, we do four things exceedingly well: coding, designing, arguing, and boozing.  Back before insanity struck us and we founded our company, we used to engage in two of these four activities on a pretty regular basis.  In between vodka tonics and under the tyranny of blasting glow stick (unce unce unce) music at a generic San Francisco lounge, our CEO Rami Essaid and I entered into a contest of opinions escalating to near gladiatorial proportions of hand-waving and hyperbole trying to figure out if big companies would feel comfortable moving their traffic through a third party security service.

Our core premise was simple: Would large enterprises ever trust their data to cloud security providers?  I don’t remember who won the argument (or anything else from that night, actually) but that hesitation of large enterprises committing to cloud infrastructure built and maintained by another company is a fact of life we deal with daily at Distil Networks.  We have a product that they want, but should they trust us with their data?  We say “absolutely!”  They say “weeellllll…”

Over the past year and a half we’ve seen some very high profile distributed cloud failures that have led to extended down times for customers and embarrassing headlines for companies.  Our team, as well, has had issues in the past that have either taken out our internal tools or caused our customers discomfort.  Maybe even severe discomfort.  At the scale that most cloud companies operate, when things go bad they tend to go spectacularly bad.

Despite these risks, I can’t help but believe that we’re on the fast track to a future where hardware will just be generic interconnected parts serving as hosts to diversify shared platforms (and judging by some predictions, I may be on the path to being prophetic).  Virtual servers, routers, and load balancers that can be brought online instantly and purely on-demand allow businesses to focus on their core products and grow naturally without being hampered by technological logistics.  Why then, spend hours configuring firewalls and caching layers when someone else has done that work for you?

The last question in particular applies to major enterprises now more than ever.  Unless your company provides hosting, networking or network security services, you really don’t need to invest vast resources into maintaining all these systems in-house.  Network security, for one, is an important aspect of modern businesses that is no longer the core product of nearly all of the companies that  need it the most.  This leads to damaging situations involving unauthorized access, and direct damage to your brand (and of course, even more bad headlines).

One immediate benefit that all of our customers see when they switch to Distil’s bot blocking network is actual insight into exactly how many bots are scraping them on a frighteningly regular basis — insight that’s usually followed up with a bit of shock and a lot of anger.  While we’re exceedingly good at detecting and blocking bots, the surprise customers have is rooted more from the fact that they didn’t know this was occurring in the first place. Since they weren’t aware it was a problem, they never paid close attention to their server logs or ran asynchronous analytics on connection behavior.  Not being aware of a security gap is almost always the reason those gaps are eventually exploited.

Because cloud security companies like Distil Networks have a razor focus on specific areas, we are aware of those gaps. While an hour outage can be extremely costly and embarrassing, a website that gets compromised by a bot attack can lead to private data theft, phishing attacks on your visitors, and malicious code running in your customer’s browser. The protection cloud security providers like Distil offer often saves a greater order of magnitude on the potential damages prevented than the worst previously linked cloud outages could have cost.  This makes cloud security a decided no brainer.