Wednesday, February 24, 2016

php imap download email to eml a comparison of different approaches

Pieced together a simple php script to download emails using imap.
These are some of the different approaches to download the email as eml format. 
 Approach A   
 $headers = imap_fetchheader($connection, $k, FT_PREFETCHTEXT);  
 $body = imap_body($connection, $k);  
 file_put_contents("pathtoemail.eml", $headers . "\n" . $body);  
 Approach B  
  $body=imap_fetchbody($connection, $k, "");  
 file_put_contents("pathtoemail.eml", $body);  
 Approach C  
  imap_savebody($connection, "pathtoemail.eml", $k);  

Performance comparison when downloading 1 day worth of email. (lower is better)
 Approach A (mins)  
 1. 2.095  
 2. 1.886  
 3. 1.984  
 4. 1.990   
 Approach B (mins)  
 1. 1.447  
 2. 1.357  
 3. 1.399  
 4. 1.413  
 Approach C (mins)  
 1. 1.468  
 2. 1.409  
 3. 1.401  
 4. 1.367  
Conclusion:
Either approach B or C would make a good choice for downloading emails using php imap.
Sample script:
 <?php  
 $time_start = microtime(true);   
 //$server = '{imap.mail.yahoo.com:993/ssl}';  
 $server = '{imap.gmail.com:993/ssl}';  
 //$user = 'xxx@gmail.com';  
 $user = $argv[1];  
 //$pass = 'xxx';  
 $pass = "$argv[2]";  
 date_default_timezone_set('Asia/Shanghai');  
 $criteria = 'SINCE "'.date('d M Y', strtotime('- 1 days')).'"';  
 $connection = imap_open($server, $user, $pass) or die("can't connect: " . imap_last_error());  
 $mailboxes = imap_list($connection, $server, '*');  
 $folder = "./$user/";  
 if(!file_exists($folder)){  
   mkdir($folder, 0777, true);  
 }  
 foreach($mailboxes as $mailbox) {  
   $shortname = str_replace($server, '', $mailbox);  
   echo "$shortname\n";  
   //create folder  
   if(!file_exists("$folder$shortname")){  
     mkdir("$folder$shortname", 0777, true);  
     }  
   //enter into folder  
   imap_reopen($connection, $server.$shortname);  
   $count = imap_num_msg($connection);  
   $uids = imap_search($connection, $criteria, SE_UID, 'UTF-8');  
   if(is_array($uids)){  
     foreach($uids as $ud){    
       $k = imap_msgno ($connection, $ud);  
       //approach A  
       //$headers = imap_fetchheader($connection, $k, FT_PREFETCHTEXT);  
       //$body = imap_body($connection, $k);  
       //file_put_contents("$folder$shortname/$ud.eml", $headers ."\n". $body);  
       //approach B  
       //$body=imap_fetchbody($connection, $k, "");  
       //file_put_contents("$folder$shortname/$ud.eml", $body);  
       //approach C  
       imap_savebody($connection, "$folder$shortname/$ud.eml", $k);  
       echo "saved $ud.eml " . "($count-$k)\n";    
     }  
   }else{  
 //          echo "imap_search failed: " . imap_last_error() . "\n";  
     }  
 }  
   imap_errors();  
   imap_close($connection);  
 $time_end = microtime(true);  
 //dividing with 60 will give the execution time in minutes other wise seconds  
 $execution_time = ($time_end - $time_start)/60;  
 //execution time of the script  
 echo 'Total Execution Time: '.$execution_time.' Mins';  
 ?>  

Friday, September 4, 2015

Starting a VPS reseller business

Just some notes to remind me on how to setup my own VPS services

Pre-requisite:

1 x Server grade dedicated server
Multiple number of IP addresses; 1 for each VPS.
Visualization software:
a) http://solusvm.com/pricing/
b) http://www.virtualizor.com/NOCs

Payment portal
a) WHMCS
b) http://www.docs.modulesgarden.com/SolusVM_Extended_VPS_For_WHMCS

Monday, March 9, 2015

Manually setup Wifi AP with a captive portal on kali linux

As per the topic, this is a post to show how to create a (i) Wifi AP, (ii) redirect users to a captive portal, and (iii) how to allow user to use the internet after they accept the terms and condition on the captive portal.

This is the environment we will be using:
Local interface which is connected to the internet.

1)
interface configuration are located here /etc/network/interfaces
eth0
IP : 192.168.17.28
Mask: 255.255.255.0
GW : 192.168.17.2

Wireless Network
wlan0
IP : 192.168.0.1
Mask: 255.255.255.0
GW : 192.168.0.0

Assuming you already have eth0 setup properly, turn on wireless network interface
 ifconfig wlan0 192.168.0.1  


2)
Configure DHCPD /etc/dhcp/dhcpd.conf
      default-lease-time 300;  
      max-lease-time 360;  
      ddns-update-style none;  
      authoritative;  
      log-facility local7;  
      subnet 192.168.0.0 netmask 255.255.255.0 {  
      range 192.168.0.100 192.168.0.200;  
      option routers 192.168.0.1;  
      option domain-name-servers 192.168.17.2;  
      }  


3)
Enable IP forwarding
 echo "1" > /proc/sys/net/ipv4/ip_forward  


4)
Flush your iptables rules so that we are on the same settings
 iptables -t nat -F  


5)
Configure Iptables to enable traffic masquerading
 iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE   


6)
Configure all Wifi traffic from wlan0 to be redirected to the webserver
 iptables -t nat -A PREROUTING -i wlan0 -p tcp -j DNAT --to-destination 192.168.17.128:80  


7)
Configure your hotspot configuration. HOSTAPD is a utility for creating a AP.
create your configuration file /etc/hostapd/hostapd.conf
# Define interface  
interface=wlan0
driver=nl80211
# Select driver  
ssid=myhotspot
# Set access point name  
hw_mode=g
# Set access point harware mode to 802.11g  
# Enable WPA2 only (1 for WPA, 2 for WPA2, 3 for WPA + WPA2)  
# Set WIFI channel (can be easily changed)  
channel=6
#wpa=2
#wpa_passphrase=mypassword



8)
Time to start DHCPD and hostapd. once started use another device to see if there is a myhotspot.  Everytime you try to surf the internet, you will be redirected to your localhost webserver. Whatever your are serving is called the captive portal.
 dhcpd  
 hostapd /etc/hostapd/hostapd.conf  
 /etc/init.d/apache2 start  


9)
So how do you enable internet for users? One possible way is to use your captive portal to create Iptable rule which enable a particular device (mac address). For example if your device is 11:22:33:11:22:33
 iptables -t nat -I PREROUTING -m mac --mac-source 11:22:33:11:22:33 -j ACCEPT  

Still unclear on step 9? Try looking at pwnstar package.

Having trouble routing ur wifi to ur internet interface ?
https://prahladyeri.wordpress.com/2013/05/26/how-to-turn-your-linux-machine-into-a-wifi-access-point/

Monday, September 29, 2014

Google Scraper

I am in need for analyzing google search result, fortunately there are multiple opensource solution out there. But google hates scrapers and would block your IP should they determine that you are breaking their terms and condition. 

Possible Google Scraper: (Play with the sleep timing between request to prevent IP blocking)
https://github.com/NikolaiT/GoogleScraper
https://github.com/MarioVilas/google


//Example using MarioVilas's google scraper: 
python google.py --stop=20 "inurl:console filetype:php" > test.txt

//If you need to remove parameters, a simple bash script is perfect: 
vi removeparameter.sh
#!/bin/bash
while read p; do
FILE=$p
echo ${FILE%%\?*}
done < test.txt

Tuesday, September 2, 2014

Installing Evilgrade on Centos 6

Installing Evilgrade on Centos gave me abit of problem and took too much time for me thus i am documenting it.

Download Evilgrade from:
https://github.com/infobyte/evilgrade/archive/master.zip

unzip master.zip and go to the directory.

//if you have no error then gd else you will need to install the dependencies
./evilgrade

Required perl dependencies:
Data::Dump
Digest::MD5   
Time::HiRes
RPC::XML
How to install them on Centos using CPAN:
use CPAN, but if you have never used it, then you will need to configure it.

perl -MCPAN -eshell
//step thru the configuration

// key in the below command to automate the configuration
o conf init

cpan -i Data::Dump
cpan -i Digest::MD5
cpan -i Time::HiRes
cpan -i RPC::XML

if you have error with RPC::XML, 
yum whatprovides "perl(XML::Parser)"
yum install ""

yum install perl-IO-Socket-SSL

Monday, July 28, 2014

Setting up Wordpress on VPS

This is just to journal down the steps required for setting up a fresh installation of Wordpress on a VPS. FTP account is required and i assumed that your VPS has it.

Install mysql and mysqld
 yum install mysql mysqld  

Install PHP
 yum install php php-mysql  

Install Wordpress
 wget http://wordpress.org/latest.tar.gz  
 tar -xzvf latest.tar.gz   
 cp ./wordpress/wp-config-sample.php ./wordpress/wp-config.php  
 sudo cp -r ./wordpress/* /var/www/html  
 sudo yum install php-gd  
 sudo service httpd restart  

Install Mysql
sudo yum install mysql-server   
sudo service mysqld start   
sudo /usr/bin/mysql_secure_installation   

  Enter current password for root (enter for none):    
  OK, successfully used password, moving on...   
  By default, a MySQL installation has an anonymous user, allowing anyone   
  to log into MySQL without having to have a user account created for   
  them. This is intended only for testing, and to make the installation   
  go a bit smoother. You should remove them before moving into a   
  production environment.   
  Remove anonymous users? [Y/n] y              
  ... Success!   
  Normally, root should only be allowed to connect from 'localhost'. This   
  ensures that someone cannot guess at the root password from the network.   
  Disallow root login remotely? [Y/n] y   
  ... Success!   
  By default, MySQL comes with a database named 'test' that anyone can   
  access. This is also intended only for testing, and should be removed   
  before moving into a production environment.   
  Remove test database and access to it? [Y/n] y   
  - Dropping test database...   
  ... Success!   
  - Removing privileges on test database...   
  ... Success!   
  Reloading the privilege tables will ensure that all changes made so far   
  will take effect immediately.   
  Reload privilege tables now? [Y/n] y   
  ... Success!   
   $   

Create Mysql database
mysql -u adminusername -p  
CREATE DATABASE databasename;  
GRANT ALL PRIVILEGES ON databasename.* TO "root@localhost" IDENTIFIED BY "p@ssw0rd";  
FLUSH PRIVILEGES;     

Configure Mysql and link it with Wordpress wp-config.php
 define('DB_NAME', 'wordpress');  
 define('DB_USER', 'user');  
 define('DB_PASSWORD', 'password');  

Try surfing to Wordpress Site to complete the installation
 serverip/wp-admin/install.php  

Next configure so that themes can be installed automatically
 //add this line of code to wp-config.php  
 if(is_admin()) {  
      add_filter('filesystem_method', create_function('$a', 'return "direct";' ));  
      define( 'FS_CHMOD_DIR', 0751 );  
 }  

Some bugs with WP meant that inorder to create folders automatically i had to edit permission to WP-Content folder
 chmod 777 -R /var/www/html/wp-content  

References
http://www.wpmayor.com/the-ultimate-guide-to-setting-up-a-wordpress-vps-part-1/
https://www.digitalocean.com/community/tutorials/how-to-install-wordpress-on-centos-6--2
http://www.wpbeginner.com/wp-tutorials/how-to-fix-the-error-establishing-a-database-connection-in-wordpress/
http://wordpress.org/support/topic/your-php-installation-appears-to-be-missing-the-mysql-extension-which-is-require

Saturday, July 19, 2014

VBA Useragent Parser Project - Part 1 Iterate through the rows in a column & Part 2 Adding Regex to parse for information

Recently I have web logs handed to me in Excel format, not wanting to export it to text for processing, i thought i would work on a Excel VBA for parsing useragent data.

Apparently there isnt any that i can find from the internet, its probably a terrible idea to use excel for parsing web logs.

Anyway i will post my progress for the coding on this site for my future reference.

The code below is for looping through column B in the excel sheet.
 Sub Parse()  
 '  
 ' Parse Macro  
 '  
 '  
  Dim i As Long  
  Do While True  
  Count = Count + 1  
   If IsEmpty(Cells(Count, "B").Value) Then  
     Cells(Count, "B").Value = "end"  
     Exit Do  
   End If  
  Cells(Count, "B").Font.Color = vbRed  
  Loop  
 End Sub  


Based on the code above, i have added regular expression to parse for OS, Webkit and Browser information. Feel free to try it: (Place Useragent information in column B of your excel sheet and execute this macro)
 Sub Parse()  
  '  
  ' Parse Macro  
  '  
  '  
  Dim i As Long  
  Dim UACol As String  
  'Browser type and version'  
  Dim Browser(3) As String  
  Dim BrowserVer(4) As String  
  Dim BrowserCol As String  
  Dim BrowserVerCol As String  
  'OS type and version'  
  Dim OSVer(4) As String  
  Dim OSCol As String  
  Dim OSVerCol As String  
  'Device Type'  
  Dim DEVICEVer(0) As String  
  Dim DEVICEVerCol As String  
  'Webkit Version'  
  Dim WEBKITVer(0) As String  
  Dim WEBKITVerCol As String  
  UACol = "B"  
  BrowserCol = "C"  
  BrowserVerCol = "D"  
  OSCol = "E"  
  OSVerCol = "F"  
  DEVICEVerCol = "G"  
  WEBKITVerCol = "H"  
  Browser(0) = "Chrome"  
  Browser(1) = "IE"  
  Browser(2) = "Safari"  
  Browser(3) = "Firefox"  
  BrowserVer(0) = "(Chrome)\/([^\s\;]+)"  
  BrowserVer(1) = "(Firefox)\/([^\s\;]+)"  
  BrowserVer(2) = "(IE)[\/\s]([^\s\;]+)"  
  BrowserVer(3) = "(Trident)\/([^\s\;]+)"  
  BrowserVer(4) = "(Safari)\/([^\s\;]+)"  
  'iPad; CPU OS 7_1_1 like Mac OS X'  
  'iPhone; CPU iPhone OS 7_1_1 like Mac OS X'  
  'iPhone; U; CPU iPhone OS 2_1 like Mac OS X; en-us'  
  OSVer(0) = "(ip[honead]+)\; .+ ([^\s\;]+) like Mac OS X"  
  'Android 4.0.2'  
  OSVer(1) = "(android) ([^\s\;]+)"  
  'User Agent in BlackBerry 10'  
  'Mozilla/5.0 (BB10; <Device Type>) AppleWebKit/537.10+ (KHTML, like Gecko) Version/<BB Version #> Mobile Safari/537.10+'  
  OSVer(2) = "(BB10|BlackBerry).+ BB Version[/\b\s]([^\s\;]+)"  
  OSVer(3) = "(windows) (NT [^\s\;]+)"  
  'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.13+ (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2'  
  'Mozilla/5.0 (Macintosh; U; PPC Mac OS X; sv-se) AppleWebKit/85.7 (KHTML, like Gecko) Safari/85.5'  
  OSVer(4) = "(Macintosh).+ (MAC OS [^\s\;]+)"  
  'Mozilla/5.0 (Linux; Android 4.1.2; GT-N7000 Build/JZO54K) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.123 Mobile Safari/537.22 OPR/14.0.1025.52315'  
  'Android.+ (G[^\s\;]+-[^\s\;]+)'  
  'Mozilla/5.0 (Linux; U; Android 4.0.2; en-us; Galaxy Nexus Build/ICL53F) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30 :'  
  'Android.+ (Galaxy [^\s\;]+)'  
  DEVICEVer(0) = "Android.+ ((Galaxy |GT-|GN-)([^\s\;]+))"  
  WEBKITVer(0) = "(([^\s]+webkit)\/([^\s\;]+))"  
  Set objRegExp_1 = CreateObject("vbscript.regexp")  
  objRegExp_1.Global = True  
  objRegExp_1.IgnoreCase = True  
   'Regex Test Start'  
   'objRegExp_1.Pattern = "(Chrome)\/([^\s]+)" '  
   'objRegExp_1.Pattern = "Android.+ ((Galaxy |GT-|GN-)([^\s\;]+))"  
   'strToSearch = "Mozilla/5.0 (Linux; Android 4.1.2; GT-N7000 Build/JZO54K) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.123 Mobile Safari/537.22 OPR/14.0.1025.52315"  
   'strToSearch = "Mozilla/5.0 (Linux; U; Android 4.0.2; en-us; Galaxy Nexus Build/ICL53F) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.123 Mobile Safari/537.22 OPR/14.0.1025.52315"  
   'Set regExp_Matches = objRegExp_1.Execute(strToSearch)  
   'For Each objMatch In regExp_Matches  
   'DEVICE = objMatch.SubMatches(0)  
   'DEVICEVERSION = objMatch.SubMatches(1)  
   'MsgBox ("This string is a valid email address." & DEVICE & " " & DEVICEVERSION)  
   'Next  
   'Regex Test end'  
  Do While True  
  Count = Count + 1  
   If IsEmpty(Cells(Count, UACol).Value) Then  
    Cells(Count, UACol).Value = "end"  
    Exit Do  
   End If  
  'Browser Check'  
  For Each element In BrowserVer  
   objRegExp_1.Pattern = element  
   Set regExp_Matches = objRegExp_1.Execute(Cells(Count, UACol).Value)  
   For Each objMatch In regExp_Matches  
   Cells(Count, BrowserCol).Value = objMatch.SubMatches(0)  
   Cells(Count, BrowserVerCol).Value = objMatch.SubMatches(1)  
   GoTo OSCheck  
   Next objMatch  
  Next element  
 OSCheck:  
 'OS Check'  
  For Each element1 In OSVer  
   objRegExp_1.Pattern = element1  
   Set regExp_Matches = objRegExp_1.Execute(Cells(Count, UACol).Value)  
   For Each objMatch In regExp_Matches  
   Cells(Count, OSCol).Value = objMatch.SubMatches(0)  
   Cells(Count, OSVerCol).Value = objMatch.SubMatches(1)  
   GoTo DEVICECheck  
   Next objMatch  
  Next element1  
 DEVICECheck:  
 'Device Check'  
  For Each element In DEVICEVer  
   objRegExp_1.Pattern = element  
   Set regExp_Matches = objRegExp_1.Execute(Cells(Count, UACol).Value)  
   For Each objMatch In regExp_Matches  
   Cells(Count, DEVICEVerCol).Value = objMatch.SubMatches(0)  
   GoTo WEBKITCheck  
   Next objMatch  
  Next element  
 WEBKITCheck:  
 'WEBKIT Check'  
  For Each element In WEBKITVer  
   objRegExp_1.Pattern = element  
   Set regExp_Matches = objRegExp_1.Execute(Cells(Count, UACol).Value)  
   For Each objMatch In regExp_Matches  
   Cells(Count, WEBKITVerCol).Value = objMatch.SubMatches(0)  
   GoTo NextLoop  
   Next objMatch  
  Next element  
 NextLoop:  
  Loop  
  End Sub  

References required to produce this:

iOS Useragent info
http://www.webapps-online.com/online-tools/user-agent-strings/dv/operatingsystem51849/ios
http://www.enterpriseios.com/wiki/Complete_List_of_iOS_User_Agent_Strings

A more detailed Javascript UA parser
https://github.com/faisalman/ua-parser-js/blob/master/src/ua-parser.js
http://faisalman.github.io/ua-parser-js/

VBA Regular Expression
https://www.udemy.com/blog/vba-regex/
https://www.udemy.com/blog/vba-regex/