Monday, September 29, 2014

Google Scraper

I am in need for analyzing google search result, fortunately there are multiple opensource solution out there. But google hates scrapers and would block your IP should they determine that you are breaking their terms and condition. 

Possible Google Scraper: (Play with the sleep timing between request to prevent IP blocking)
https://github.com/NikolaiT/GoogleScraper
https://github.com/MarioVilas/google


//Example using MarioVilas's google scraper: 
python google.py --stop=20 "inurl:console filetype:php" > test.txt

//If you need to remove parameters, a simple bash script is perfect: 
vi removeparameter.sh
#!/bin/bash
while read p; do
FILE=$p
echo ${FILE%%\?*}
done < test.txt

Tuesday, September 2, 2014

Installing Evilgrade on Centos 6

Installing Evilgrade on Centos gave me abit of problem and took too much time for me thus i am documenting it.

Download Evilgrade from:
https://github.com/infobyte/evilgrade/archive/master.zip

unzip master.zip and go to the directory.

//if you have no error then gd else you will need to install the dependencies
./evilgrade

Required perl dependencies:
Data::Dump
Digest::MD5   
Time::HiRes
RPC::XML
How to install them on Centos using CPAN:
use CPAN, but if you have never used it, then you will need to configure it.

perl -MCPAN -eshell
//step thru the configuration

// key in the below command to automate the configuration
o conf init

cpan -i Data::Dump
cpan -i Digest::MD5
cpan -i Time::HiRes
cpan -i RPC::XML

if you have error with RPC::XML, 
yum whatprovides "perl(XML::Parser)"
yum install ""

yum install perl-IO-Socket-SSL

Monday, July 28, 2014

Setting up Wordpress on VPS

This is just to journal down the steps required for setting up a fresh installation of Wordpress on a VPS. FTP account is required and i assumed that your VPS has it.

Install mysql and mysqld
 yum install mysql mysqld  

Install PHP
 yum install php php-mysql  

Install Wordpress
 wget http://wordpress.org/latest.tar.gz  
 tar -xzvf latest.tar.gz   
 cp ./wordpress/wp-config-sample.php ./wordpress/wp-config.php  
 sudo cp -r ./wordpress/* /var/www/html  
 sudo yum install php-gd  
 sudo service httpd restart  

Install Mysql
sudo yum install mysql-server   
sudo service mysqld start   
sudo /usr/bin/mysql_secure_installation   

  Enter current password for root (enter for none):    
  OK, successfully used password, moving on...   
  By default, a MySQL installation has an anonymous user, allowing anyone   
  to log into MySQL without having to have a user account created for   
  them. This is intended only for testing, and to make the installation   
  go a bit smoother. You should remove them before moving into a   
  production environment.   
  Remove anonymous users? [Y/n] y              
  ... Success!   
  Normally, root should only be allowed to connect from 'localhost'. This   
  ensures that someone cannot guess at the root password from the network.   
  Disallow root login remotely? [Y/n] y   
  ... Success!   
  By default, MySQL comes with a database named 'test' that anyone can   
  access. This is also intended only for testing, and should be removed   
  before moving into a production environment.   
  Remove test database and access to it? [Y/n] y   
  - Dropping test database...   
  ... Success!   
  - Removing privileges on test database...   
  ... Success!   
  Reloading the privilege tables will ensure that all changes made so far   
  will take effect immediately.   
  Reload privilege tables now? [Y/n] y   
  ... Success!   
   $   

Create Mysql database
mysql -u adminusername -p  
CREATE DATABASE databasename;  
GRANT ALL PRIVILEGES ON databasename.* TO "root@localhost" IDENTIFIED BY "p@ssw0rd";  
FLUSH PRIVILEGES;     

Configure Mysql and link it with Wordpress wp-config.php
 define('DB_NAME', 'wordpress');  
 define('DB_USER', 'user');  
 define('DB_PASSWORD', 'password');  

Try surfing to Wordpress Site to complete the installation
 serverip/wp-admin/install.php  

Next configure so that themes can be installed automatically
 //add this line of code to wp-config.php  
 if(is_admin()) {  
      add_filter('filesystem_method', create_function('$a', 'return "direct";' ));  
      define( 'FS_CHMOD_DIR', 0751 );  
 }  

Some bugs with WP meant that inorder to create folders automatically i had to edit permission to WP-Content folder
 chmod 777 -R /var/www/html/wp-content  

References
http://www.wpmayor.com/the-ultimate-guide-to-setting-up-a-wordpress-vps-part-1/
https://www.digitalocean.com/community/tutorials/how-to-install-wordpress-on-centos-6--2
http://www.wpbeginner.com/wp-tutorials/how-to-fix-the-error-establishing-a-database-connection-in-wordpress/
http://wordpress.org/support/topic/your-php-installation-appears-to-be-missing-the-mysql-extension-which-is-require

Saturday, July 19, 2014

VBA Useragent Parser Project - Part 1 Iterate through the rows in a column & Part 2 Adding Regex to parse for information

Recently I have web logs handed to me in Excel format, not wanting to export it to text for processing, i thought i would work on a Excel VBA for parsing useragent data.

Apparently there isnt any that i can find from the internet, its probably a terrible idea to use excel for parsing web logs.

Anyway i will post my progress for the coding on this site for my future reference.

The code below is for looping through column B in the excel sheet.
 Sub Parse()  
 '  
 ' Parse Macro  
 '  
 '  
  Dim i As Long  
  Do While True  
  Count = Count + 1  
   If IsEmpty(Cells(Count, "B").Value) Then  
     Cells(Count, "B").Value = "end"  
     Exit Do  
   End If  
  Cells(Count, "B").Font.Color = vbRed  
  Loop  
 End Sub  


Based on the code above, i have added regular expression to parse for OS, Webkit and Browser information. Feel free to try it: (Place Useragent information in column B of your excel sheet and execute this macro)
 Sub Parse()  
  '  
  ' Parse Macro  
  '  
  '  
  Dim i As Long  
  Dim UACol As String  
  'Browser type and version'  
  Dim Browser(3) As String  
  Dim BrowserVer(4) As String  
  Dim BrowserCol As String  
  Dim BrowserVerCol As String  
  'OS type and version'  
  Dim OSVer(4) As String  
  Dim OSCol As String  
  Dim OSVerCol As String  
  'Device Type'  
  Dim DEVICEVer(0) As String  
  Dim DEVICEVerCol As String  
  'Webkit Version'  
  Dim WEBKITVer(0) As String  
  Dim WEBKITVerCol As String  
  UACol = "B"  
  BrowserCol = "C"  
  BrowserVerCol = "D"  
  OSCol = "E"  
  OSVerCol = "F"  
  DEVICEVerCol = "G"  
  WEBKITVerCol = "H"  
  Browser(0) = "Chrome"  
  Browser(1) = "IE"  
  Browser(2) = "Safari"  
  Browser(3) = "Firefox"  
  BrowserVer(0) = "(Chrome)\/([^\s\;]+)"  
  BrowserVer(1) = "(Firefox)\/([^\s\;]+)"  
  BrowserVer(2) = "(IE)[\/\s]([^\s\;]+)"  
  BrowserVer(3) = "(Trident)\/([^\s\;]+)"  
  BrowserVer(4) = "(Safari)\/([^\s\;]+)"  
  'iPad; CPU OS 7_1_1 like Mac OS X'  
  'iPhone; CPU iPhone OS 7_1_1 like Mac OS X'  
  'iPhone; U; CPU iPhone OS 2_1 like Mac OS X; en-us'  
  OSVer(0) = "(ip[honead]+)\; .+ ([^\s\;]+) like Mac OS X"  
  'Android 4.0.2'  
  OSVer(1) = "(android) ([^\s\;]+)"  
  'User Agent in BlackBerry 10'  
  'Mozilla/5.0 (BB10; <Device Type>) AppleWebKit/537.10+ (KHTML, like Gecko) Version/<BB Version #> Mobile Safari/537.10+'  
  OSVer(2) = "(BB10|BlackBerry).+ BB Version[/\b\s]([^\s\;]+)"  
  OSVer(3) = "(windows) (NT [^\s\;]+)"  
  'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.13+ (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2'  
  'Mozilla/5.0 (Macintosh; U; PPC Mac OS X; sv-se) AppleWebKit/85.7 (KHTML, like Gecko) Safari/85.5'  
  OSVer(4) = "(Macintosh).+ (MAC OS [^\s\;]+)"  
  'Mozilla/5.0 (Linux; Android 4.1.2; GT-N7000 Build/JZO54K) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.123 Mobile Safari/537.22 OPR/14.0.1025.52315'  
  'Android.+ (G[^\s\;]+-[^\s\;]+)'  
  'Mozilla/5.0 (Linux; U; Android 4.0.2; en-us; Galaxy Nexus Build/ICL53F) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30 :'  
  'Android.+ (Galaxy [^\s\;]+)'  
  DEVICEVer(0) = "Android.+ ((Galaxy |GT-|GN-)([^\s\;]+))"  
  WEBKITVer(0) = "(([^\s]+webkit)\/([^\s\;]+))"  
  Set objRegExp_1 = CreateObject("vbscript.regexp")  
  objRegExp_1.Global = True  
  objRegExp_1.IgnoreCase = True  
   'Regex Test Start'  
   'objRegExp_1.Pattern = "(Chrome)\/([^\s]+)" '  
   'objRegExp_1.Pattern = "Android.+ ((Galaxy |GT-|GN-)([^\s\;]+))"  
   'strToSearch = "Mozilla/5.0 (Linux; Android 4.1.2; GT-N7000 Build/JZO54K) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.123 Mobile Safari/537.22 OPR/14.0.1025.52315"  
   'strToSearch = "Mozilla/5.0 (Linux; U; Android 4.0.2; en-us; Galaxy Nexus Build/ICL53F) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.123 Mobile Safari/537.22 OPR/14.0.1025.52315"  
   'Set regExp_Matches = objRegExp_1.Execute(strToSearch)  
   'For Each objMatch In regExp_Matches  
   'DEVICE = objMatch.SubMatches(0)  
   'DEVICEVERSION = objMatch.SubMatches(1)  
   'MsgBox ("This string is a valid email address." & DEVICE & " " & DEVICEVERSION)  
   'Next  
   'Regex Test end'  
  Do While True  
  Count = Count + 1  
   If IsEmpty(Cells(Count, UACol).Value) Then  
    Cells(Count, UACol).Value = "end"  
    Exit Do  
   End If  
  'Browser Check'  
  For Each element In BrowserVer  
   objRegExp_1.Pattern = element  
   Set regExp_Matches = objRegExp_1.Execute(Cells(Count, UACol).Value)  
   For Each objMatch In regExp_Matches  
   Cells(Count, BrowserCol).Value = objMatch.SubMatches(0)  
   Cells(Count, BrowserVerCol).Value = objMatch.SubMatches(1)  
   GoTo OSCheck  
   Next objMatch  
  Next element  
 OSCheck:  
 'OS Check'  
  For Each element1 In OSVer  
   objRegExp_1.Pattern = element1  
   Set regExp_Matches = objRegExp_1.Execute(Cells(Count, UACol).Value)  
   For Each objMatch In regExp_Matches  
   Cells(Count, OSCol).Value = objMatch.SubMatches(0)  
   Cells(Count, OSVerCol).Value = objMatch.SubMatches(1)  
   GoTo DEVICECheck  
   Next objMatch  
  Next element1  
 DEVICECheck:  
 'Device Check'  
  For Each element In DEVICEVer  
   objRegExp_1.Pattern = element  
   Set regExp_Matches = objRegExp_1.Execute(Cells(Count, UACol).Value)  
   For Each objMatch In regExp_Matches  
   Cells(Count, DEVICEVerCol).Value = objMatch.SubMatches(0)  
   GoTo WEBKITCheck  
   Next objMatch  
  Next element  
 WEBKITCheck:  
 'WEBKIT Check'  
  For Each element In WEBKITVer  
   objRegExp_1.Pattern = element  
   Set regExp_Matches = objRegExp_1.Execute(Cells(Count, UACol).Value)  
   For Each objMatch In regExp_Matches  
   Cells(Count, WEBKITVerCol).Value = objMatch.SubMatches(0)  
   GoTo NextLoop  
   Next objMatch  
  Next element  
 NextLoop:  
  Loop  
  End Sub  

References required to produce this:

iOS Useragent info
http://www.webapps-online.com/online-tools/user-agent-strings/dv/operatingsystem51849/ios
http://www.enterpriseios.com/wiki/Complete_List_of_iOS_User_Agent_Strings

A more detailed Javascript UA parser
https://github.com/faisalman/ua-parser-js/blob/master/src/ua-parser.js
http://faisalman.github.io/ua-parser-js/

VBA Regular Expression
https://www.udemy.com/blog/vba-regex/
https://www.udemy.com/blog/vba-regex/

Thursday, June 26, 2014

Warning do not purchase from www.thaihosting.asia they do not honor their money back guarantee

I you value your time do not buy anything from www.thaihosting.asia. Its bad and terrible and a total waste of my time.
So i tried to buy a VPS from them, they requested my Identity document to be submitted for verification. I sent in my driving license with my identification number partially censored.
www.thaihosting.asia refused to accept my driving license for verification thus i wanted a full refund as they indicated that they have 100% money back guarantee.
They refused to honor it !!!

thaihosting.asia is a scam site.

a whois on thaihosting.asia revealed that it is registered under www.bakuri.com.
www.bakuri.com scam and that is how i see it.
www.bakuri.com warning do not buy.