mkatari-bioinformatics-august-2013-introlinuxnotes
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
mkatari-bioinformatics-august-2013-introlinuxnotes [2015/06/03 18:15] – mkatari | mkatari-bioinformatics-august-2013-introlinuxnotes [2015/06/11 11:50] (current) – [Controlling Jobs] mkatari | ||
---|---|---|---|
Line 33: | Line 33: | ||
====== Logging in with X Windows ====== | ====== Logging in with X Windows ====== | ||
+ | |||
+ | The standard user interface for personal computers is a GUI (Graphical User Interface). However for linux it is a command-line interpreter called shell. | ||
+ | It is simply a prompt the awaits your command. There are several different shells, but the one used often is called “bash”, which is a mixture of a bunch of other shells. | ||
In cases where a program requires a GUI, you should log in using the '' | In cases where a program requires a GUI, you should log in using the '' | ||
Line 64: | Line 67: | ||
Generally in this directory you have complete control over creating, modifying, and executing files in this or any sub directory you create. In order to return to your home directory simply type the command: '' | Generally in this directory you have complete control over creating, modifying, and executing files in this or any sub directory you create. In order to return to your home directory simply type the command: '' | ||
- | ====== The Linux Shell ====== | ||
- | |||
- | The standard user interface for personal computers is a GUI (Graphical User Interface). However for linux it is a command-line interpreter called shell. | ||
- | It is simply a prompt the awaits your command. There are several different shells, but the one used often is called “bash”, which is a mixture of a bunch of other shells. | ||
===== Command Line Editing ===== | ===== Command Line Editing ===== | ||
Line 99: | Line 98: | ||
+ | ===== Directing standard output ===== | ||
+ | Instead letting the output print to the screen we can save it to a file by using the ''>'' | ||
+ | The following command gets details about all users' home directories and saves them into a file called '' | ||
+ | < | ||
+ | [mkatari@hpc ~]$ ls -l /home/ > allusers.txt | ||
+ | [mkatari@hpc ~]$ ls -l allusers.txt | ||
+ | -rw-rw-r--. 1 mkatari mkatari 18897 Jun 3 21:26 allusers.txt | ||
+ | |||
+ | </ | ||
+ | |||
+ | |||
+ | ===== Command-line completion ===== | ||
+ | |||
+ | In some cases the commands or the file names that you need as arguments can be very long which increases the chance of spelling mistakes. | ||
+ | |||
+ | To prevent such mistakes simply type the enough letters to unambiguously identify the command or file and then pressing tab will complete it for you. | ||
+ | |||
+ | In the case you don’t know how many letters you need, simply press tab twice to see all your options. | ||
+ | |||
+ | In the example below, after typing the command and its options, the tab key was pressed twice to get this. The command will not be executed until the '' | ||
+ | |||
+ | < | ||
+ | [mkatari@hpc ~]$ ls /usr/bin/bz | ||
+ | bzcat | ||
+ | bzcmp | ||
+ | </ | ||
+ | |||
+ | ===== Wildcards ===== | ||
+ | |||
+ | In cases where you want to refer to multiple files you can use '' | ||
+ | |||
+ | In the example below, the first line gives all files/ | ||
+ | |||
+ | < | ||
+ | [mkatari@hpc ~]$ ls / | ||
+ | / | ||
+ | / | ||
+ | [mkatari@hpc ~]$ ls / | ||
+ | / | ||
+ | </ | ||
+ | |||
+ | ===== Finding Your Way ===== | ||
+ | |||
+ | Often you will get lost on the hpc and you will need to know where you are, which computer did you log into, or even which account have you logged into. Below are some simple commands that help you find your way. | ||
+ | |||
+ | < | ||
+ | [mkatari@hpc ~]$ whoami | ||
+ | mkatari | ||
+ | |||
+ | [mkatari@hpc ~]$ pwd | ||
+ | / | ||
+ | |||
+ | [mkatari@hpc ~]$ hostname | ||
+ | hpc.ilri.cgiar.org | ||
+ | |||
+ | </ | ||
+ | |||
+ | ===== File manipulation ===== | ||
+ | |||
+ | Useful commands for manipulating files and directories. To get details about how to use the commands type man < | ||
+ | |||
+ | ^Command ^Action ^ | ||
+ | |mkdir | make a directory | | ||
+ | |rmdir | remove a directory (only works if the directory is empty ) | | ||
+ | |cd | change directory | | ||
+ | |pwd | present working directory | | ||
+ | |ls | list of files and directories in the directory. You can use wild card to look for specific files. You can also use -l to see details such as permission for files and directories | | ||
+ | |cp | copy a file and/or directories. Use -r to recursively copy. | | ||
+ | |mv | move a file. It will copy and then delete the source. This can be used to rename files as well. | | ||
+ | |rm | remove a file | | ||
+ | |||
+ | < | ||
+ | [mkatari@hpc ~]$ mkdir temp | ||
+ | [mkatari@hpc ~]$ cd temp/ | ||
+ | [mkatari@hpc temp]$ ls | ||
+ | [mkatari@hpc temp]$ cp ../ | ||
+ | [mkatari@hpc temp]$ ls | ||
+ | allusers.txt | ||
+ | [mkatari@hpc temp]$ mv allusers.txt allusers.backup | ||
+ | [mkatari@hpc temp]$ ls | ||
+ | allusers.backup | ||
+ | [mkatari@hpc temp]$ rm allusers.backup | ||
+ | [mkatari@hpc temp]$ ls | ||
+ | [mkatari@hpc temp]$ cd ../ | ||
+ | [mkatari@hpc ~]$ rmdir temp/ | ||
+ | |||
+ | </ | ||
+ | |||
+ | ====== Permissions ====== | ||
+ | |||
+ | There are three levels of permissions that can be assigned to all files, programs, and directories | ||
+ | * Read: open the file and copy it | ||
+ | * Write: edit the file and delete it | ||
+ | * Execute: Run the commands in the file or change into the directory if it is a directory | ||
+ | |||
+ | There are also three different levels of users: | ||
+ | * User – you | ||
+ | * Group – A collection of users that are in a group | ||
+ | * Everyone - Not just the people who have accounts on the machine but if the directory is open to the public and any one. | ||
+ | |||
+ | The commands used to change owner, group, and specific permissions are: | ||
+ | * chown – changes the owner | ||
+ | * chgrp – changes the group | ||
+ | * chmod – change read, write, and execute permissions | ||
+ | * +/- r = read | ||
+ | * +/- w = write | ||
+ | * +/- x = execute | ||
+ | * u = user level | ||
+ | * g = group level | ||
+ | * o = others | ||
+ | * a = all | ||
+ | * chmod can also use three numbers to set permissions where the value of the number represents a specific combination of rwx and their order assigns it to the different levels (u,g,o) | ||
+ | * 0 = none | ||
+ | * 1 = execute only | ||
+ | * 2 = write only | ||
+ | * 3 = write and execute only | ||
+ | * 4 = read only | ||
+ | * 5 = read and execute only | ||
+ | * 6 = read and write only | ||
+ | * 7 = read, write and execute | ||
+ | |||
+ | < | ||
+ | |||
+ | [mkatari@hpc ~]$ tail allusers.txt > bottomusers.txt | ||
+ | [mkatari@hpc ~]$ ls -al bottomusers.txt | ||
+ | -rw-rw-r--. 1 mkatari mkatari 772 Jun 11 13:54 bottomusers.txt | ||
+ | |||
+ | [mkatari@hpc ~]$ chmod u=+r-wx, | ||
+ | [mkatari@hpc ~]$ ls -al bottomusers.txt | ||
+ | -r--r-----. 1 mkatari mkatari 772 Jun 11 13:54 bottomusers.txt | ||
+ | |||
+ | [mkatari@hpc ~]$ rm bottomusers.txt | ||
+ | rm: remove write-protected regular file `bottomusers.txt'? | ||
+ | [mkatari@hpc ~]$ ls -al bottomusers.txt | ||
+ | ls: cannot access bottomusers.txt: | ||
+ | |||
+ | </ | ||
+ | Notice that since we have taken away our own write permission to the file we can not automatically delete it. The shell is asking to change the permission so it can delete. If we have write permissions that it would have delete right away. | ||
+ | |||
+ | ====== History ====== | ||
+ | |||
+ | Your shell saves all your commands and you can access them using the up and down keys. | ||
+ | |||
+ | Typing the command “history” returns all the commands you have entered and a number assigned to it. | ||
+ | |||
+ | You can run a specific one again by typing ! (also called “bang”) Followed by the job number. | ||
+ | |||
+ | !! Will perform the most recent command. | ||
+ | |||
+ | In the example code below, '' | ||
+ | |||
+ | < | ||
+ | [mkatari@hpc ~]$ history | grep bottom | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | [mkatari@hpc ~]$ !1000 | ||
+ | tail allusers.txt > bottomusers.txt | ||
+ | [mkatari@hpc ~]$ ls -al bottomusers.txt | ||
+ | -rw-rw-r--. 1 mkatari mkatari 772 Jun 11 14:01 bottomusers.txt | ||
+ | [mkatari@hpc ~]$ !1002 | ||
+ | chmod u=+r-wx, | ||
+ | [mkatari@hpc ~]$ ls -al bottomusers.txt | ||
+ | -r--r-----. 1 mkatari mkatari 772 Jun 11 14:01 bottomusers.txt | ||
+ | </ | ||
+ | |||
+ | ====== Transferring Files ====== | ||
+ | |||
+ | There are several ways to transfer files to a server. The most reliable and consistent way is to use '' | ||
+ | |||
+ | One major advantage of using mobaxterm on windows machines is that it comes with some basic linux commands, including scp. On a mac, scp comes with the operating system. So if you want to copy a file that is on your Desktop to hpc from your laptop you can use the following command: | ||
+ | |||
+ | < | ||
+ | [2015-06-11 07: | ||
+ | [Manpreet.WIN-OAOVO3NM02E] ➤ cp allusers.txt Desktop/ | ||
+ | ✔ | ||
+ | ──────────────────────────────────────────────────────────────────────────── | ||
+ | [2015-06-11 07: | ||
+ | [Manpreet.WIN-OAOVO3NM02E] ➤ scp Desktop/ | ||
+ | Unauthorized access is prohibited. | ||
+ | allusers_local.txt | ||
+ | |||
+ | </ | ||
+ | the : at the end is very important because it tells the shell that is a server and not a file name. The : by itself puts the file in your home directory, but you can specify a specific path if you wanted to. | ||
+ | |||
+ | Now to copy the file back to my computer I will use the following command. Note that this time I am also renaming the file on my computer. | ||
+ | |||
+ | < | ||
+ | [Manpreet.WIN-OAOVO3NM02E] ➤ scp mkatari@hpc.ilri.cgiar.org: | ||
+ | Unauthorized access is prohibited. | ||
+ | allusers_local.txt | ||
+ | |||
+ | </ | ||
+ | |||
+ | ====== Controlling Jobs ====== | ||
+ | |||
+ | The following commands and keyboard short-cuts can come in handing when you need to cancel, suspend, or start a job. | ||
+ | |||
+ | ^Command^Action^ | ||
+ | |ctrl-C|Terminate current running job| | ||
+ | |ctrl-Z|Suspend Jobs| | ||
+ | | bg | Once a job has been suspended '' | ||
+ | | fg | In order to put a background job in the foreground type '' | ||
+ | | & | When executing a command and you want to put it in the background immediately put the ''&'' | ||
+ | | jobs | This gives a list of jobs (suspended, running, and terminating)| | ||
+ | | top | This starts an interface where you can see all jobs. To see all jobs under your account type '' | ||
+ | |||
+ | ====== Commands for manipulating and querying files ====== | ||
+ | |||
+ | Some more cool commands | ||
+ | |||
+ | ^Command^Action^ | ||
+ | |less/more | read through the file without loading the entire file. Press spacebar to continue or q to quit. | | ||
+ | |touch | create an empty file | | ||
+ | |head | show the first few lines of the file| | ||
+ | |tail | show the last few lines of the file| | ||
+ | |cat | read through the file(s)| | ||
+ | |grep | search for patterns in a file or files| | ||
+ | |cut | separate file based on columns| | ||
+ | |comm/diff | compare and see the difference between the files. The files have to be sorted before using either of these commands.| | ||
+ | |split | splits file into smaller files based on the options| | ||
+ | |sort | sort the file base on the options selected.| | ||
+ | |wc | wordcount| | ||
+ | |||
+ | |||
+ | |||
+ | |||
+ | |||
+ | |||
+ | ====== Exercise ====== | ||
+ | |||
+ | - Download the file to your computer from the following url [[ftp:// | ||
+ | - Transfer it to your LinuxJune2015 directory on HPC. | ||
+ | - Gunzip the file and rename it to : Cassavagenes.txt | ||
+ | - Determine the number of different genes are present in the file. Hint – check the third column | ||
+ | - Save the rows that contain the word “gene” in the third column as a separate file. Call it “GeneTypes.txt” | ||
+ | - Create a directory called “Share” | ||
+ | - Move GeneTypes.txt into Share and change permissions on Share so others can read the file. | ||
====== Some useful information about linux ====== | ====== Some useful information about linux ====== | ||
+ | |||
+ | |||
=== Environment variables and PATH === | === Environment variables and PATH === |
mkatari-bioinformatics-august-2013-introlinuxnotes.1433355314.txt.gz · Last modified: 2015/06/03 18:15 by mkatari