Attaching new EBS volume in AWS EC2 instance.

Amazon AWS plus EC2 logo_scaled

In AWS, EC2 by default provide 8GB space, in a past project I had to extend the size of one of my development instance as the data was growing fast. From AWS console add new EBS volume. Then attach it to your instance by AWS console and log into you EC2 instance via ssh.

Run following command:


sudo fdisk -l

which will show list of volumes with the newly added volume as unpartitioned. Something like below:

Screen Shot 2012-09-19 at 1_up_2

Then next step is to build the file system of new EBS volume using unix command mkfs. Like below:


sudo mkfs -t ext4 /dev/xvdf

Screen Shot 2012-09-19 at 1_mkfs

Next you have to mount it in your desired path,  e.g. /mnt/ebs1. Run following command:


sudo mount /dev/xvdf /mnt/ebs1

Then add an entry into /etc/fstab. it would be something like this:


"/dev/xvdf  /mnt/ebs1 ext4 defaults 1 1"

There are facts if you add the EBS volume to your /etc/fstab and some how if there are  issue (like file system corruption, unavailability of zone  etc ) with the volume during booting the instance it will not be booted. Because while booting your system will look for the entry and when its not available the whole instance is down. Check AWS forum post for details.

And also check this whole SO discussion to resolve this issue in alternative way ( using a script for example).

Check following docs if you are more interested about the unix commands that used in this post.

fdiskmount and unmount  and mkfs.

Advertisements

Serving static files using Restify

I was working with Node.js for building a REST API. For REST API module I was using restify. The restify is a simple and yet powerful node module. One of the use case of the API was, I had to serve static file for specific routing. I went through the docs and tried different things but couldn’t figure yet out at first. After hustling for hours,  me and Christian started to go deep into it and figured it out!

So in my case the configuration was  something like this


server.get(/.*/, restify.serveStatic({
    'directory': 'static_content',
    'default': 'index.html'
 }));

When restify internally resolve the path, it looks for ‘static_content/index.html’.

I have coded a very basic application to show it works, sample application looks like below:

In this sample application the static content (index.html) is in the same root as server.js as why ‘directory’: ‘.’.

Use package.json to install necessary module and start playing 🙂

Counting line number of files recursively in Unix

I was working in a project for last couple of months, as the days are passing the codebase is getting larger. Suddenly I thought, It would be great if I can  know how many lines of code I have written so far for each module. And also in total. I know unix has a really awesome utils named wc.

After googling and trying different params and commands I managed to find it by merging to unix tool(wc and find), the full command for recursive line number counting is like below:

wc -l `find . -type f`

The command returns something like below:

Screen Shot 2013-09-25 at 1.39.13 AM

Using  find . -type f   listing all the files recursively and wc -l is counting the line numbers 🙂

For learning these tow unix command in details check wc and find manual.

MongoDB backup script

Last year while I was working in a project, I needed to automate the whole backing up process from taking snapshot of the current db to saving it to AWS S3 buckets. At that time I took most of the staffs from this blog post.

mongo_logo

Couple of days ago, I started to code for making small backup script that will backup to another cloud machine rather than to AWS S3.  Instead of  coding it from scratch, I reused my previously coded script. All I need to implement a bash function(save_in_cloud) which runs a simple scp command 🙂

The whole script look like below:

I reused this script, all I did just added a new function which copy the current backup data to a remote server.  And also updated do_cleanup, now it works in any year.

The backup script depends on other two js (fsync_lock.js and fsync_unlock.js) functions which responsible for locking mongo during  db snapshots and releasing lock  after the  snapshots.

Happy Coding 🙂

Painless deployment with Fabric

Deployment of code in test/staging/production  servers is one of the important part of modern web applications development cycle.

Deploying code were painful because its repetitive same tasks we have to do every time we want to push code, during deployment  if something goes wrong the application will go down too. But the scenario has changed, now we have many tools to make the deployment easier and fun. I have used Capistrano and Fabric for deployment. Found Fabric really painless and as its a Python battery, it was easier for me to adopt and get things done.

I am going to cover fundamental operations and finally a simple fabric script(like boilerplate) for writing your own fabric script.

env = its a Python dictionary like subclass where we define specific settings like password,user etc

local = runs command in  local host(where fabric script is being run)

run = runs command in a remote host

You can use these code tasks in many different ways, to do that check the Fabric Office Documentation from here.

First gist is a sample fabric script,second one is a bash script to install fabric in your ubuntu machine.

 After setting username,password and host information into the script you cab check your server’s access log by running  fab test_server latest_access_log 

I am using fabric for around two years and used for different small,medium and large projects.

There are many interesting open source projects going on top of Fabric. I found these two projects really promising.

1.Fabtools

2.Graphite_fabric

Search through github,you will find many advance level Fabric use.

Happy Coding!

Daily Unix Commands

If you are a dev or a sys admin  you must rely on unix command to get things done.

These are my daily unix tools list,I am putting these together for myself as a future reference,if it helps anyone else I will be happy 🙂

Screen

I use ‘screen’  for keeping myself alive in my remote cloud machines. If you are new to screen,please check this docs.

Checking list of screen in current machine 'screen -ls'
Resuming 'screen -r screen_name' or 'screen -r pid'
Create new screen 'screen -S screen_name'
Detach from screen using 'ctrl+a+d'
Copy mode start 'ctrl+a+{'
Kill the current window 'ctrl+k'

Screen cheatscheet1 screen cheat sheet2

File Compression

Using zip and unzip

Zip command example is like below:

zip backup.zip filename1 filename2
zip -r backup.zip dir_name

Unzip command example is like below:

unzip backup.zip

tar,gzip and bzip2

tar cfv backup.tar filename1 filename2
tar xvf backup.tar

for specific directory

tar xvf backup.tar -C /dir_name/

to use gzip compression use -z  and for bzip2 compressing use -j like this

for compression:

tar -zcfv backup.tar.gzip filename1
tar -jcfv backup.tbz2 filename

for uncompression

tar -zxvf backup.ta
tar -jxvf  backup.tbz2

Unix file compression cheat sheet

And if you want to learn details about this compression techniques please check this

Vim basics

I use vim for editing files in unix system(specially in remote machines).These are list of commands I use:

Basic Navigation:


j->down
k->up
l->left
r->right
H->first line of the file
G->last line of the file
$->end of the line

Changing Line:


yy->copy the current line
p->paste the copied line

check Vim Cheat sheet .

I rarely use ‘sed’ command,and I only used it for easy find replace of strings(its really handy when you have to work on multiple files for same find replace ).The syntax is like below:

sed s/search_string/replace+string old_file.txt > new_file.txt

check sed refernce

Rank SQL query Result

I was playing with mysql client,suddenly found that it would be really great if I can rank the query results.Lets say we have a table like below:

Now we want to rank  the students based on their Roll No.To do it,we have to define a variable with the initial value of 0 and in next query we will show it by as one of the resulted field  query

like below:


set @counter=0;

select @counter:=@counter+1 as Rank,LastName,Roll_no as Roll  from Students order by Roll_no ASC;

and in next query we will show it by as one of the resulted field ,that results:

Check mysql docs from here to know details about user defined variable in mysql

Unix sort & uniq

It was around 2 Am and I was working like a caveman,but its hard to escape bed time 😦

Suddenly I found I set a wrong cron job in a cloud and it generated duplicate results.I have to make a report from the cron output and every line should be unique.The file is around 1.2 GB.

It was  a json file, that has several thousand lines,many of them are redundant.I have to remove the redundant  values and make a file which every line is unique.

I started to write a python script to do that,I was on the half way to finish my python script that takes file and create another file that contains uniqu elements from the input file.As I was too tired,thought I should do a search is there any unix command to this job.And found exactly what I needed 🙂

sort filename.txt | uniq

Or

cat filename.txt | sort -u

If the input file contans:

Line 1
Line 2
Line 2
Line 3
Line 1
Line 3

The command generates

Line 1
Line 2
Line 3

And I just redirected the output of the command into a new file like below:

sort filename.txt | uniq > result.txt

Explanation of the command:

sort’ command lists all the lines by default alphabetically and ‘uniq’ command can eliminate or count duplicate lines in a pre sorted file.

You can also use sort and uniq in different situation, for  details check following links:

Sort and Uniq

These two utility command will help me to sleep early 🙂

Step by step setting up a Squid Proxy server

Squid is used as a proxy cache server.It makes web responses faster,minimize bandwidth.
Couple of days ago,I had set up a local squid proxy server for our dev environment.The first step was to configure a standalone squid in my Mac OS X lion.

In this post I will write down the minimal steps and configuration for local Squid proxy service.

The steps are simple:

1.Update your network settings

2.Downlading Squid source

3.Compile and make it

4.Configure and start it!

Go to “System Preferences”=>”NetWork”=>”Advanced”(lower right corner)=>”Proxies”=> Select Web Proxy.

And set the “Web proxy Server” value as “localhost” and port is 3128(default port of squid)

Download the source form here http://www.squid-cache.org/Versions/
Extract it using

tar zxvf packagename.tar.gz

then cd package name” then configure it using

./configure -prefix=/usr/local/squid

compile using

make all

then intall it using

sudo make install

“for configuring the squid go to the

cd /usr/local/squid/etc

and open the squid.conf using

sudo mate squid.conf

I used Textmate as an editor,you can use your favorite text editor.
then you have to add the following configurations

# disk and memory cache settings
 cache_dir ufs /usr/local/squid/var/cache 500 16 256
 maximum_object_size 4096 KB

 

#access log and cache log settings
cache_access_log /usr/local/squid/var/logs/access.log
cache_log /usr/local/squid/var/logs/cache.log
pid_filename /usr/local/squid/var/run/squid.pid

 

#assign your machine IP
acl allowed src 192.168.1.108
# Squid listening port
 http_port 8080

Then run the following commands

cd /usr/local/squid/sbin
 /.squid  -z

This command will create your all the swap directories.
And you need to change the permissions of the cache directories too.
Now time to start Squid using

./squid

Now,browse couple of pages and turn off your internet,then try to browse the pages again.

You can see the web pages and they are coming from cache.
That’s it.You can download my configuration file and change it as your settings. From here 
Hopefully in next post I will write down the steps how to configure Squid for a network and serving multiple computer as a Proxy server.

N.B:Please check the paths and change it accordingly for your installation directory.

Reset Mysql Root password in Ubuntu

For last couple of months when I tried to access my mysql,I sadly discover that I forgot the root password.
And google a lot to know the reset details.

I know in future I will need the steps again as why thought It would be better to write it down.
Its very simple,all I did.

1.Stop running Mysql


service mysql stop

or using

sudo /etc/init.d/mysql stop

2.Starting the mysql configuration ,login as a root in mysql and update the root user’s password.

sudo mysqld --skip-grant-tables &

Access to mysql using

mysql -u root mysql

And the type update the root user password using following command

UPDATE user SET Password=PASSWORD('new pass') WHERE User='root'; FLUSH PRIVILEGES; exit;

Then you will see “OK” confirmation like below:

Query OK, 3 rows affected (0.00 sec)
Rows matched: 3  Changed: 3  Warnings: 0

Query OK, 0 rows affected (0.00 sec)
Bye

Then you can access to your mysql using following command

mysql -u root -p

Thats it.

Check out mysql doc for details