org.jscience.net
Class MediaCrawler

java.lang.Object
  extended by java.lang.Thread
      extended by org.jscience.net.MediaCrawler
All Implemented Interfaces:
java.lang.Runnable, CrawlerSetting

public class MediaCrawler
extends java.lang.Thread
implements CrawlerSetting

MediaCrawler is a single thread that searches the web for files that are of a given type.

Since:
10/29/2002
See Also:
Spider

Nested Class Summary
static interface MediaCrawler.Handler
          used to handle the media files found during the search of the MediaCrawler
 
Nested classes/interfaces inherited from class java.lang.Thread
java.lang.Thread.State, java.lang.Thread.UncaughtExceptionHandler
 
Field Summary
 
Fields inherited from class java.lang.Thread
MAX_PRIORITY, MIN_PRIORITY, NORM_PRIORITY
 
Constructor Summary
MediaCrawler(java.net.URL rootURL, int depth, java.lang.String mediaExtension, boolean currentSiteOnly, MediaCrawler.Handler handler, java.lang.String[] pattern)
           
MediaCrawler(java.net.URL rootURL, int depth, java.lang.String mediaExtension, boolean currentSiteOnly, java.lang.String[] pattern)
           
 
Method Summary
 void addHandler(MediaCrawler.Handler handler)
           
 boolean followLinks(java.net.URL url, java.net.URL referer, int depth, java.util.List<java.net.URL> resultURLList, java.util.List<java.net.URL> closedURLList, java.util.List<Spider.URLWrapper> searchURLWrapperList)
          followLinks() determines whether the given URL is to be searched for its links to be examined further in the next level.
 URLCache[] getFilesFound()
           
 boolean matchesCriteria(java.net.URL url, java.net.URL referer, int depth, java.util.List<java.net.URL> resultURLList, java.util.List<java.net.URL> closedURLList)
          This method decides whether either the URL itself or its content qualifies for what this CrawlerSetting searches for; as this function is also called on every URL encountered, it is also the place for any custom parsing this CrawlerSetting wants to do.
 void run()
           
 
Methods inherited from class java.lang.Thread
activeCount, checkAccess, countStackFrames, currentThread, destroy, dumpStack, enumerate, getAllStackTraces, getContextClassLoader, getDefaultUncaughtExceptionHandler, getId, getName, getPriority, getStackTrace, getState, getThreadGroup, getUncaughtExceptionHandler, holdsLock, interrupt, interrupted, isAlive, isDaemon, isInterrupted, join, join, join, resume, setContextClassLoader, setDaemon, setDefaultUncaughtExceptionHandler, setName, setPriority, setUncaughtExceptionHandler, sleep, sleep, start, stop, stop, suspend, toString, yield
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Constructor Detail

MediaCrawler

public MediaCrawler(java.net.URL rootURL,
                    int depth,
                    java.lang.String mediaExtension,
                    boolean currentSiteOnly,
                    java.lang.String[] pattern)

MediaCrawler

public MediaCrawler(java.net.URL rootURL,
                    int depth,
                    java.lang.String mediaExtension,
                    boolean currentSiteOnly,
                    MediaCrawler.Handler handler,
                    java.lang.String[] pattern)
Method Detail

addHandler

public void addHandler(MediaCrawler.Handler handler)

run

public void run()
Specified by:
run in interface java.lang.Runnable
Overrides:
run in class java.lang.Thread

getFilesFound

public URLCache[] getFilesFound()

followLinks

public boolean followLinks(java.net.URL url,
                           java.net.URL referer,
                           int depth,
                           java.util.List<java.net.URL> resultURLList,
                           java.util.List<java.net.URL> closedURLList,
                           java.util.List<Spider.URLWrapper> searchURLWrapperList)
Description copied from interface: CrawlerSetting
followLinks() determines whether the given URL is to be searched for its links to be examined further in the next level. The three List objects allow the CrawlerSetting to act on potential constrains that may result from e.g. a maximum number of total nodes to be examined (or any other custom checking imaginable). The url may include any URL, including non-HTTP protocols (such as mailto:, ftp:) and image or media URLs.

Specified by:
followLinks in interface CrawlerSetting
Parameters:
url - the URL that is to be examined for its links
referer - url's referer URL
depth - distance from the original root URL where the search began
resultURLList - List of URLs that have already been found to match this CrawlerSetting's criteria
closedURLList - List of URLs that have already been found not to match the CrawlerSetting's criteria
searchURLWrapperList - List of Spider.URLWrapper objects already identified to be examined in the next level
See Also:
Spider.URLWrapper

matchesCriteria

public boolean matchesCriteria(java.net.URL url,
                               java.net.URL referer,
                               int depth,
                               java.util.List<java.net.URL> resultURLList,
                               java.util.List<java.net.URL> closedURLList)
Description copied from interface: CrawlerSetting
This method decides whether either the URL itself or its content qualifies for what this CrawlerSetting searches for; as this function is also called on every URL encountered, it is also the place for any custom parsing this CrawlerSetting wants to do. The two List objects allow the CrawlerSetting to act on potential constrains that may result from e.g. a maximum number of total nodes to be examined (or any other custom checking imaginable). Note that it is the responsibility of the calling object to ensure that this function isn't called multiple times on the same URL if that's not desired. The url may include any URL, including non-HTTP protocols (such as mailto:, ftp:) and image or media URLs

Specified by:
matchesCriteria in interface CrawlerSetting
Parameters:
url - the URL in question to satisfy the criteria
referer - url's referer URL
depth - link distance from the original root URL where the search began
resultURLList - List of URLs that have already been found to match this CrawlerSetting's criteria
closedURLList - List of URLs that have already been found not to match the CrawlerSetting's criteria