Daily Unix Commands

If you are a dev or a sys admin  you must rely on unix command to get things done.

These are my daily unix tools list,I am putting these together for myself as a future reference,if it helps anyone else I will be happy ūüôā

Screen

I use ‘screen’ ¬†for keeping myself alive in my remote cloud machines. If you are new to screen,please check this docs.

Checking list of screen in current machine 'screen -ls'
Resuming 'screen -r screen_name' or 'screen -r pid'
Create new screen 'screen -S screen_name'
Detach from screen using 'ctrl+a+d'
Copy mode start 'ctrl+a+{'
Kill the current window 'ctrl+k'

Screen cheatscheet1 screen cheat sheet2

File Compression

Using zip and unzip

Zip command example is like below:

zip backup.zip filename1 filename2
zip -r backup.zip dir_name

Unzip command example is like below:

unzip backup.zip

tar,gzip and bzip2

tar cfv backup.tar filename1 filename2
tar xvf backup.tar

for specific directory

tar xvf backup.tar -C /dir_name/

to use gzip compression use -z  and for bzip2 compressing use -j like this

for compression:

tar -zcfv backup.tar.gzip filename1
tar -jcfv backup.tbz2 filename

for uncompression

tar -zxvf backup.ta
tar -jxvf  backup.tbz2

Unix file compression cheat sheet

And if you want to learn details about this compression techniques please check this

Vim basics

I use vim for editing files in unix system(specially in remote machines).These are list of commands I use:

Basic Navigation:


j->down
k->up
l->left
r->right
H->first line of the file
G->last line of the file
$->end of the line

Changing Line:


yy->copy the current line
p->paste the copied line

check Vim Cheat sheet .

I rarely use ‘sed’ command,and I only used it for easy find replace of strings(its really handy when you have to work on multiple files for same find replace ).The syntax is like below:

sed s/search_string/replace+string old_file.txt > new_file.txt

check sed refernce

Advertisements

Rank SQL query Result

I was playing with mysql client,suddenly found that it would be really great if I can rank the query results.Lets say we have a table like below:

Now we want to rank  the students based on their Roll No.To do it,we have to define a variable with the initial value of 0 and in next query we will show it by as one of the resulted field  query

like below:


set @counter=0;

select @counter:=@counter+1 as Rank,LastName,Roll_no as Roll  from Students order by Roll_no ASC;

and in next query we will show it by as one of the resulted field ,that results:

Check mysql docs from here to know details about user defined variable in mysql

Build Periodic crawler with Celery

Its a very common use case when you build a crawler and it will have to run periodically.And generally we set a unix Cron Job to handle the crawler periodically.

But its really pain when you add new task you have to login to the server and add new cron task into the crontab.Its only feasible when you have to run only few cron jobs.

I thought it would be great if I can handle it from my python code and do some interesting things.I have heard about Celery¬†a lot as a meesage queue system.¬†Truly¬†speaking at first I couldn’t understand¬†¬†how it works or how I can integrate with my projects.After googling I understood what is Celery then I thought it will be really great if I can run crawlers around it and use Celery to scheduling periodic work.

Install  Celery by following this link,then you will have to install and configure RabbitMQ from this link. BTW dont forget to add user,vhosts it is described on SettingUp RabbitMQ sections(ou can use mongodb,resddis as broker too).

And then you can clone my git repo,change the celeryconfig.py file as per as your configuration.Add a new task into tasks.py following the first method.

I have added a sample method which requests this site and print the HTTP response status code .

To run the project run “celerybeat”,then it will start celerybeat and start to send tasks to the broker like below:

Run “celeryd”¬†into another terminal window to check the task output,you will see something like below:

It is printing the response status after every 5 seconds.

You can handle anything that you want todo after the crawling,like parsing the dom saving text,submitting form etc.

Btw dont forget to run


pip install -r requirements.txt

to install necessary packages for the projects.My Github Periodic Crawling Project

Unix sort & uniq

It was around 2 Am and I was working like a caveman,but its hard to escape bed time ūüė¶

Suddenly I found I set a wrong cron job in a cloud and it generated duplicate results.I have to make a report from the cron output and every line should be unique.The file is around 1.2 GB.

It was  a json file, that has several thousand lines,many of them are redundant.I have to remove the redundant  values and make a file which every line is unique.

I started to write a python script to do that,I was on the half way to finish my python script that takes file and create another file that contains uniqu¬†elements¬†from the input file.As I was too¬†tired,thought I should do a search is there any unix command to this job.And found exactly what I needed ūüôā

sort filename.txt | uniq

Or

cat filename.txt | sort -u

If the input file contans:

Line 1
Line 2
Line 2
Line 3
Line 1
Line 3

The command generates

Line 1
Line 2
Line 3

And I just redirected the output of the command into a new file like below:

sort filename.txt | uniq > result.txt

Explanation of the command:

sort’ command lists all the lines by default alphabetically and ‘uniq’¬†command can eliminate or count duplicate lines in a pre sorted file.

You can also use sort and uniq in different situation, for  details check following links:

Sort and Uniq

These two utility command will help me to sleep early ūüôā

Python Movie Data Crawler

Couple of days ago,I was talking with newly joined Engineer in our team.Suddenly I found he is a very resourceful person in terms of collecting movi.He has personal collection of movie archive which is 3TB !

So,I was really happy to find such a person as a teammate.He has three different lists of movies in text file.And shared one of the text file which contained more than couple of  hundreds movie name,I decided to write a script that scrapes movie rating from IMDB.com.I was googling if there is any public REST API available and found this website http://www.imdbapi.com/ which returns json as a search result.

The main task is couple of lines code like below:

import requests
import urllib
import json
BASE_URL = 'http://www.imdbapi.com/?'
query = {'i': '', 't': movi_name ,'tomatoes':'true'}
response = requests.get(BASE_URL+urllib.urlencode(query))
output = json.dumps(response.content, separators=(',',':'))

The output is the movie information,to grab specific result I have done some more formatting.You can check the whole script from here
Right now the script would print output like this:

Getting Movi The Pianist...
{'Plot': 'A Polish Jewish musician struggles to survive the destruction of the Warsaw ghetto of World War II.', 'Rating': '8.5', 'Title': 'The Pianist', 'Director': 'Roman Polanski', 'tomatoRating': '8.2', 'IMDB Rating': '8.5'}

In this script I have used amazing python module called Requests.

You can add more functionality like getting movie poster or writhing the movie data into
And if you want to run this script please add text file names movies.txt,like below:

The Pianist
The Avengers

Happy coding!

Activating Git Autocompletion

Git by default dont provide autocompletion,sometime its really seems boring to write the whole branch name or command.

Couple of days ago Rubayeet shared the way to activate it.

Go to this link ,copy the content of the file and save it as .git-completion.sh and add the path of the file to your bash profile.

like this

source ~/.git-completion.sh

And compile the bash profile by using following command:

source ~/.bash_profile

Write any git command and then press tab(like unix autocomplete)

you will see the branch name or command is automatically completing or showing suggestion like below

Happy coding ūüôā

PHP and MongoDB web development beginners guide review

As I am working with mongodb for different application development,the more I explore mongo the more I am amazed.
I got a review copy from Packt couple of weeks ago and started to read PHP and MongoDB Web Development Beginner’s Guide.
Chapter 1:As Nosql is a new technology yet,people talk more than using it in their production.And in the first chapter author described what is NoSql,Types of NoSQl and why mongdb.Ttraditional and very effective way to start exploring a technology.The important part of this chapter is mongodb concepts section.There is a mongodb is quiet different than any other traditional db and here there is a difference in basic terminology too.Author described those point with example.
Chapter2:As its a beginners guide author chose blog application development and here the mongo querying comes through returning article from mongo back-end.Interesting part was,author focused on mongodb details implementation and feature.Like Upsert,Update and save.
chapter3:Building a session manager,here a session manager was built with the blog implementation.

chapter4:Aggregation queries one of the most important chapter of the book.For db administrating,building complex business model application developers have to gain mastery in db he is using for his application.In this chapter MapReduce was explained and implemented also found different essential querying based on count(),distinct.
The mapreduce is very essential concepts the examples are real life problems solving using mapreduce,I found it very useful.

chapter5:A very simple version of web analytics was built and why we can choose mongodb for different analytics solution was described.Mapreduce implementation for analytics is the fun part and reader will learn to aggregation in different use cases for his analytics.
Chapter 6: Focuses on Mongodb with mysql.when to use,how it fits.mongodb as a data archive system.
Chapter7 : This chapter is about Gridfs,mongodbs file storage solution.A simple application is built to show its power and author also mentioned when should not mongodb.
chapter8:Building location aware application.How to store geo location data using mongodb is described,interesting part of geospatial indexing,this topic well described with couple of real life use case.Also covered performing location query
Chapter 9 and 10:Security and administration section.Useful section is implementation indexing in mongodb,indexing in embeded document fields.Optimization rules will be very help full section to boost the  mongo performance and the end of the chapter replication is described.Chapter 10 is all about mongo administration using GUI application like rock mongo and PHPmongo.

So overall this book is a good reference/training book who wants to learn mongodb using PHP for first time,beside that author explained many under the hood topics.
Clear instruction and several pop up quizes made the chapters enjoyable to read and learn.What I expected bit more about scaling monogo,partitioning,sharding different topics.May be as the book is written for the Beginner these topics has been avoided.

Quick Summary of 2011

So last year was  good enough for me,worked with new technologies and of course learned a lot.

  • Worked with Zend Framework for first time(¬†truly at first I was bit annoyed but later I discovered its power and flexibility.¬†)
  • Worked with Doctrine 2 and mongodb(Doctrine2 is my first ORM to work with,before I had worked with Django’s built in ORM and Doctrine2 is really made our life easy by handling every bits of ¬†SQL transaction and also optimizing silently)
  • Started coding Python again,which I love most ūüėČ
  • Played with Jquery Mobile and YII
  • And became a Mac Fan boy,bought MAC Air for development and Iphone 4 as cell phone
  • Had a wonderful sessions with the juniors of my department,took a workshop on Freelancing Web development and Python

2012 I have some goal that I should reach,btw I dont want to write it down.Bcoz a research found that “if you share your initiative with others,you get a little taste of success which sometime keep the person away from his target”!

Book Review : Yii 1.1. Application development cookbook.

As a Yii developer I love to explore the framework related books,blog posts.Recently PacktPub published a cookbook on Yii written by Alexander Makarov.

One of the best feature of Yii framework that you can develop applications really fast.The cookbook started with the chapter “Under The Hood”,beside rapid development its must to know the internal structure for maintaining good quality code.Alex tried to focus most common needs of a PHP programmer,like other frameworks you can write magic methods in Yii.In the first chapter he demonstrated that with getter,setter and event,components of Yii.Also wrote simple core method name by using which you can play with request response object.


In second,third chapter author worte abut how MVC pattern works in Yii with jquery based ajax implementation.Working with JSON is an interesting section where I learned how to handle JSON inside Yii.For REST Api based application the JSON tutorial will help a lot any Yii developer.
In third,fourth chapter he covered Yii form and testing.Common use cases like file uploading and captcha validation examples.
Chapter 6 is important in terms of application development,in the chapter how active recored pattern works,how to write and design models for db.But truly I expected ORM implementation in this chapter(like Doctrine or Propel).Highlighting code with Yii seems to be a not relevant example for this chapter in my opinion. Rather than this tricks example,data model and Yii data handling internal architecture can help a developer a lot.

Zii is a wonderful tool for handling data and make automated CRUD based basic application.I found this chapter really helpful because of writer tried to explain Under the hood of Zii,how all the classes are working.How to use grid and list view,this should help any developer.Custom grid implementation is a nice step by step example to customization Zii based code.Chapter 8,9 seems to be a well described chapters with some handy techniques for Yii core contribution and debugging.Other chapters seems interresting for advanced Yii developers,I personally liked chapter 12 as its described to add External classes,framework’s code into Yii application.Specially Zend Framework using Yii section is an exclusive section for any developer because following the steps you can easily migrate any other frameworks code to your Yii application and just use it.

And final chapter about deployment staffs which will help new Yii developers.Finally I will vote up for this book because this cookbook is well combination of “Under the Hood” and “Tricks and best Practices”.If you are a new Yii developer,I am sure it will pave your way to Yii based application development,if you are an advanced Yii expert,it will help you to extend your expertise ¬†further.Happy Coding ūüôā