Why does my crawler script suddenly end with no error?

Posted on 16th Feb 2014 07:03 pm by admin

Hi.

I have written a web crawler script. It will visit a large number of URL's with cURL.

After around 2-3 minutes of running, it will just stop, with no error output or notices.

I have these settings:
Code: [Select]set_time_limit(0);
ini_set('display_errors',1);
error_reporting(E_ALL|E_STRICT);
Any ideas why it would just stop?

Your Answer:

Login to answer

Other forums

cstdatomic (c++0x std::atomic) / g++ 4.4
Hello,

I'm trying to use cstdatomic (std::atomic in the upcoming c++0x standard) in g++

using file_get_contents??
okay how would i create a script using file_get_contents() in php have it grab search pages 1-20 and

Hits this week counter
I have a counter on my site that tracks hits, IP's, etc. into a mySQL database. It also includes the

PHP5/Zend 2.0 - Resources
PHP5 Snapshots
http://snaps.php.net/

ZEND 2.0 Feature Overview and Design (PDF File

WHYISNT THIS MYSQL STATEMNT NOT WORKING? (php)
caps because its absolutely rediculous.
i dontunderstand why its not working. there is a column c

Getting number of affected rows in SQLPLUS..
Hi everyone,
I have a shell script where i am invoking sqlplus, running a query and saving t

Add weeks in a query?
I have a SQL that summarizes the quantity per week. The weeks that has no value does not exist in th

extending tidy
I have problem with type-hinting and extending tidy. This code creates error:
Code: class cMyTidy

unoconv doc convert to pdf code prob
PHP/5.3.1

Hi. I am trying to use this code to convert docs to .pdf utilizing unoconv. Howe

PHP MySQL and DATE
Hi everyone

I have a databse and in one of the columns I have date values such as 2009-March-