Techniques for obtaining geographically-relevant product inventory information, in real-time, from heterogeneous data sources are described. Product inventory information, including the volume of available products in specific geographical locations, is obtained from at least three different sources.

A system and methods for cybersecurity rating using active and passive external reconnaissance, comprising a web crawler that send message prompts to external hosts and receives responses from external hosts, a time-series data store that produces time-series data from the message responses, and a directed computational graph module that analyzes the time-series data to produce a weighted score representing the ...

A system and apparatus is described for directing a search engine web crawler's web view to refresh the top-level container that is currently displaying the content presented by a remote computer with the new content that a navigational link, within a remote desktop, remote application window, or remote graphical windowing user session, points to. Links can be modified so as ...

A system is described that includes an electronic program guide (epg) data receiver and a media content catalog enhancer. The epg receiver is configured to receive epg data from an epg data provider.

Methods, apparatus, systems and articles of manufacture are disclosed to throttle resource access by web crawlers. An example apparatus disclosed herein includes a data interface to obtain a media request message for media hosted by a server, the media request message requesting access to the media.

A method and a content validation and coding system (cvcs) for validating and coding content of an electronic document for search engine optimization (seo) are provided. The cvcs integrates with application programming interfaces (apis) of search engines and/or media platforms.

Systems and methods are disclosed for extending and reinforcing a knowledge graph using wiki-like web pages as a source of information. A web crawler parse a wiki-like source and obtain a topic entity from the source.

Systems and methods are provided for discovering advertisements on publisher web pages and for identifying placement pathways by which discovered advertisements have been placed on the publisher web pages. An advertisement tracking and discovery system may use multiple web crawler applications to explore multiple publisher websites.

Architecture for searching geospatial resources is provided. Geospatial web crawlers are used.

Systems and methods are provided for automatically generating and maintaining user profile cookie sets. The user profile cookie sets may be used by a web crawler when gathering data such as advertisement data associated with one or more websites.

The invention is a computer process in which a web crawler or other type of search process views visual media files on the world wide web, and initially scans or downloads the files contained in html or otherwise backend internet web site files. The process identifies objects in the visual media content as being mass production items, and places the ...

A method of detecting coordinated attacks on computer and computer networks via the internet. The method includes using a web crawler to crawl the world wide web to identify domains and subdomains and their associated ip addresses, and to identify links between domains and subdomains, and storing the results in a database.

The present invention extends to methods, systems, and computer program products for developing an item data model for an item. Aspects of the invention can automate the process of data collection of “facts” for “items” that information is needed about.

Systems for secure cloud-based collaboration over shared objects. Embodiments operate within systems in a cloud-based environment, wherein one or more servers are configured to interface with storage devices that store objects accessible by one or more users.

A method to provide health-related data analytics services via a web service may include crawling, via a web crawler, the internet to identify multiple websites with content related to human health. The method may also include obtaining, using text classification, multiple words associated with an occurrence in lives of people and multiple words associated with a health outcome in the ...

The claimed subject matter provides a system or method for web crawling hidden files. An exemplary method comprises loading a web page with a browser agent, and executing any dynamic elements hosted on the web page using the browser agent to insert pre-determined values.

A system for evaluating web pages for searchable content can be utilized to make an e-commerce search engine more effective by identifying pages with searchable value. In embodiments, a web page exhibiting “searchable value” is a page that provides useful information responsive to a user's query on an e-commerce search engine.

Multiple non-host client sites provide cached user copies of web pages and/or web content, or summaries thereof, to a server. Obtaining data from non-host sources for indexing purposes avoids masked web page content indexing errors for search engines.

A method and a system for scheduling web crawlers according to keyword search. The method comprises: a scheduling end receiving a task request command sent by a crawling node; the scheduling end acquiring a secondary download link address from a priority bucket, generating tasks, adding the generated tasks into a task list, acquiring keyword link addresses from a dynamic bucket, ...

Provided is a method and system for indexing documents in a collection of linked documents. A link log, including one or more pairings of source documents and target documents is accessed.

In accordance with various example embodiments, an m2m crawler service may support capabilities to enable m2m devices to be efficiently and effectively crawled by web crawlers. As a result, m2m devices may be indexed and searched by web search engines, and thus by web users making use of web search engines.