Monday, September 29, 2014

Google Scraper

I am in need for analyzing google search result, fortunately there are multiple opensource solution out there. But google hates scrapers and would block your IP should they determine that you are breaking their terms and condition. 

Possible Google Scraper: (Play with the sleep timing between request to prevent IP blocking)
https://github.com/NikolaiT/GoogleScraper
https://github.com/MarioVilas/google


//Example using MarioVilas's google scraper: 
python google.py --stop=20 "inurl:console filetype:php" > test.txt

//If you need to remove parameters, a simple bash script is perfect: 
vi removeparameter.sh
#!/bin/bash
while read p; do
FILE=$p
echo ${FILE%%\?*}
done < test.txt

Tuesday, September 2, 2014

Installing Evilgrade on Centos 6

Installing Evilgrade on Centos gave me abit of problem and took too much time for me thus i am documenting it.

Download Evilgrade from:
https://github.com/infobyte/evilgrade/archive/master.zip

unzip master.zip and go to the directory.

//if you have no error then gd else you will need to install the dependencies
./evilgrade

Required perl dependencies:
Data::Dump
Digest::MD5   
Time::HiRes
RPC::XML
How to install them on Centos using CPAN:
use CPAN, but if you have never used it, then you will need to configure it.

perl -MCPAN -eshell
//step thru the configuration

// key in the below command to automate the configuration
o conf init

cpan -i Data::Dump
cpan -i Digest::MD5
cpan -i Time::HiRes
cpan -i RPC::XML

if you have error with RPC::XML, 
yum whatprovides "perl(XML::Parser)"
yum install ""

yum install perl-IO-Socket-SSL