Digital media and publishing industries produce vast volumes of content for distribution through a diverse range of online formats. Publishers invest heavily on proprietary content that is consumed by audiences across the globe and is often syndicated to other publishers to reproduce on their portals. These industries rely on revenue from subscribers, consumers and syndicators of their valuable content ranging from news articles to various audio and video formats, as well as research reports, surveys, e-books, podcasts, and so much more.
Naturally, detecting and blocking bad bots is crucial for digital media and publishing enterprises to protect their revenue streams ─ which we’ve outlined further along this post. First, let’s take a look at the positive impact that good bots can have on media and publishing firms.
How good bots benefit media and publishing firms
While it’s fairly well known that bad bots cause much harm to media outlets and publishers, the industry also derives a range of benefits from good bots that promote their content, amplify their reach, expand audiences, provide a better user experience, and in turn increase overall revenue.
Radware Bot Manager has always recommended that webmasters allow good bots such as search engine crawlers, aggregator and feed-fetcher bots, partner bots, monitoring bots, and social media bots to visit their sites and carry out their programmed tasks. Let’s look at the functions they perform and the leading examples of each type:
Search Engine Crawlers
Making up nearly 55% of all good bots seen across our media and publishing customer base, these bots (also known as spiders or crawlers) index web content for search engines, helping searchers easily find relevant content that they are looking for. For media portals and publishers, they are among the most effective means of amplifying their reach. Googlebot is the best-known crawler, alongwith Bingbot and the Baidu Spider.
Aggregators & Feed-Fetchers
This category of bots performs the useful task of searching for and collating new or updated articles from media sites, to provide their users or subscribers with the latest news, alerts, and other desired content. Google Feedfetcher, Superfeedr, and Feedly are among the most popular aggregator bots.
Back-link checkers analyze the inbound URLs on media and publishing portals and deliver insights into campaign performance to marketers and search engine optimization teams, helping them develop well-informed content and marketing campaigns. SEMRushBot, UAS Link Checker, and AhrefsBot are examples of this type of bot
These bots provide essential business services and information to websites and their customers. They include bots run by vendors which provide transactional, CRM & ERP services, credit-scoring, geo-location, inventory and background checks, and many other business-related services. Alexa, Slackbot and IBM Watson are among the most common partner bots.
Social Network Bots
Social networks run their own bots to provide visibility to their advertisers’ and clients’ websites and can considerably boost engagement statistics on their platforms as well as on media and publishing sites. Facebook Bot, Pinterest Bot and Snapchat are among the best-known social media bots.
These bots generally monitor website and application uptime, page load times, downtime duration, and other parameters that help media and publishing companies keep tabs on their portals. Pingdom, AlertBot and StatusCake are some of the most widely used monitoring bots.
How bad bots harm media and publishing firms
Despite using several means of securing proprietary content against unauthorized reproduction and use (such as copyrights, subscriptions and paywalls, digital rights management, and even lawsuits) the media and publishing industry is intensively targeted by bots ─ and extensively harmed by their impact. Some bots scrape their proprietary content to republish elsewhere, while others carry out large-scale ad fraud. Spambots regularly disrupt discussion forums and comment threads by posting irrelevant comments and malware-laden links. In addition, bots also submit lead generation forms filled with gibberish, much to the irritation of marketing teams and webmasters.
In recent years, there has been a significant uptick in bad bots that carry out ad fraud through the largely automated (and poorly monitored) digital advertising ecosystem. Fraudsters are rampantly defrauding advertisers and publishers using bots and leveraging attack techniques such as cookie stuffing, traffic sourcing, ghost sites, domain spoofing, click farms, ad stacking, pixel stuffing, and ad injection.
Media and publishing portals are well-advised to implement dedicated Bot Management solutions that can prevent bad bot attacks on their online properties while letting useful good bots through. Specialized solutions such as Radware Bot Manager allow enterprises to decide which business-enhancing good bots are permitted and help precisely differentiate between human and bot traffic in real-time. They also give marketers and webmasters accurate and actionable statistics to help optimize their strategies. For a deep dive into good bots and how they can benefit a diverse range of industries, download our research report ‘Inside Good Bots.’