Why does my crawler script suddenly end with no error?

Posted on 16th Feb 2014 07:03 pm by admin

Hi.

I have written a web crawler script. It will visit a large number of URL's with cURL.

After around 2-3 minutes of running, it will just stop, with no error output or notices.

I have these settings:
Code: [Select]set_time_limit(0);
ini_set('display_errors',1);
error_reporting(E_ALL|E_STRICT);
Any ideas why it would just stop?

Your Answer:

Login to answer

Other forums

Getting rid of quotes when printing data
Hi Guys, I use the filter_var FILTER_SANITIZE_STRING to filter the textarea input. The function esca

Require select-option functionality in Module Pool Programming
Hi Gurus,

I am doing a development by Module Pool on Warehouse Management. For that devel

md5 is it unique
I know this is most likely one of many simple questions that can be found semi easily. But I'm press

How to extract/download content from HTTPS page?
Hello to all the Members of this forum, Im Shoiab, A novice programmer in php.. for my first job I h

sql problems
I am having a small problem with my sql statement. it is inserting into 4 of the fields.

fie

Comparing MySql data and arrays.
I have a mysql table set up like this:
idsubjectbodyuseriddatetimetags1blog subjectblog body111|2

Display data returned from Ajaxservices!!
I have a question about displaying results using ajax. I have part of the code below.
$.ajax({

How to implement HTTP connection timeout
Hi All,I would like to implement an HTTP connection timeout.

I am working on a application wh

How to create a static html menu from a database
Hi,

I have built a small cms which allows me to create simple html pages and then upload them

Check something, wait, check again, do something!
Hi guys.

I wonder if someone can help me with this.

Basically, what I want to do is (