Learning notes for Linux enviroment construction
This my class note for CS35L, instructed by Paul Eggert and Ritam Sarmah, at UCLA. The main purpose of this project is to construct enviroment under Linux
system, practice working under Linux. The README contains most of the topics. More details are in the log files of each subdirectories. Any copy is prohibited. To view content of each project, please contact me
Everyting in Linux is a file or process
Files are collection of data
Process are an instance of an executing program
ln source linkln -s source linkShell is a interface where you can interact with Linux, bash, zsh
File Paths
- Root dir is /
- User’s homedir is ~ not a folder, but shell internally expand to home dir
- Absolute path, relative to root, start with /
- Relative path, relative to current dir
| Command | Definition |
|---|---|
| /bin | essential binaries (e.g. pwd, bash) |
| /dev | devices (and virtual ones like /dev/null) |
| /home | users’ home directories |
| /tmp | temporary files |
| /usr | user files and binaries (e.g. /usr/bin) |
echo $PATH default path for command devided by :pg1 2> pg2 write stderr from pg1 to pg2mkdir dstat d this shows Linksnumber 2, one is d, one is . inside dln -s d soft create sofelink soft to d ln is not allowed to hard link a dirC-l clear terminal screenC-u delete all text behind cursorps list current funning processkill PID kill processdu displays the amount of file space used by the specified files or directories
Enviroment variables
HOME ~PATH list of directories, separated by colons, where shell should look for executables
Assign variable using VAR=…
Access with dollar sign, e.g. $VAR
export 命令用于设置或显示环境变量。
在 shell 中执行程序时，shell 会提供一组环境变量。
export 可新增，修改或删除环境变量，供后续执行的程序使用。export 的效力仅限于该次登陆操作。export PATH=”$PATH:your path1:your path2 ...”
You can export in your ~/.profile file to do it every time you log in
| Action | Command |
|---|---|
| Exit | C-x C-c |
| Undo | C-x u |
| Save | C-x s |
| Copy | M-w |
| Cut(‘kill’) | C-w |
| Paste(‘Yank’) | C-y |
| Run shell Command | M-! COMMAND |
| Compile | M-x compile |
| Open dir | C-x d |
| Cancel comand | C-g |
Compile with -g flag(gdb) prompt(gdb) run arg1 --arg2 run
Breakpoint
When you reach the breakpoint, the line is not executed yet
(gdb) break file.c:6 This sets a breakpoint at line 6 of file.c.(gdb) break my_func This sets a breakpoint that pauses execution anytime my_func is called.(gdb) break my_func if expression Pause at my_func only when expression is true(gdb) info break List breakpoints(gdb) delete Delete all breakpoints(gdb) delete 3 Delete 3rd breakpoint(gdb) clear my_function Clear breakpoint at my_function(gdb) clear 6 Clear breakpoint at line 6(gdb) clear file.c:6 Clear breakpoint at line 6 in file.c
Once you hit a breakpoint, you can control how to continue execution
(gdb) continue Continue execution until the next breakpoint.(gdb) step [n] Step to next [n] line(s) of code. Steps into functions.(gdb) next [n] Same as step, but won’t step into functions.
<Enter> repeats the last command you typed in GDB
(gdb) print var Prints the value of var.
(gdb) watch var Create watchpoint that pauses execution when var value changes
Analyzing the program
(gdb) info frame Displays info about current stack frame, such as return address and register values.(gdb) info local List local variables and values of the function for the current stack frame.(gdb) info args List arguments for the current function call.(gdb) list [line/function] List 10 centered around line or function
Change variable value in current session
(gdb) print answer = 1 set the value of local variable answer to 1 in current gdb session(gdb) print sizeof(arr) run c code with gdb
valgrind ./file valgrind --leak-check=full ./file to show more detail Cipher example
- Caesar’s shift
- AES(symmetric) transform data a number of time key can be up to 128, 192, or 256 bits
- RSA(asymmetric) combine 2 large prime to generate public key the prime are kept secret in the private key for decrypt use asymmetric to send symmetric key is SSH
~/.ssh/known_hostClien authentication
Password(default)
SSH keys:
~/.ssh/authorized_keyssession keyGenerate SSH key pair
ssh-keygen <enter passphrase> Then you will see fingerprint cd .ssh id_rsa and id_rsa.pub is created Copy pubkey to server
- Manually append content of
id_rsa.pubtoauthorized_keys- or
ssh-copy-id username@host
ssh-copy-id username@lnxsrv10<enter password>Login with SSH key from lnxsrv09 to lnxsrv10
ssh username@lnxsrv10- By default, SSH uses defualt key
~/.ssh/id_rsa- Or,
ssh -i <path-to-private-key> username@lnxsrv10- If your key has passphrase, you need to enter it in prompt
Use ssh-agent a process need to redo everytime
- Back to lnxsrv09
exiteval `ssh-agent`start processssh-addtell which key to use- enter passphrase
- then you can ssh to server without entering passphrase
forwarded over SSH using ssh -A
You can modify./profileor.bashrcin~to automatically do thisUse X11 forwarding
ssh -Xxeyescheck successMultiple hop
- set up
ssh-agent- keep use
ssh -A
tr translate ro delete characterecho "hello world" | tr 'odw' 'aye' hella earlyecho "hello world" | tr -d 'lo' he wrdecho "hello world" | tr -s 'l' helo worldecho "hello world" | tr 'odw' 'o' hella aarlaecho "hello world" | tr 'o' 'aye' hella warld maps o to asort [FILE] also from stdin-u remove duplicatecomm FILE1 FILE2 generate output of 3 columns- as filename-1 supress Col1wc count lines, words, bytes-l only lineslocaleLC_ LC_COLLATE-order for comparing and sortingexport LC_ALL='C' Change all local variable to Csort grep Search for lines matching text-E use ERE, -F matches fixed string instead of regexls -l | grep "^d"sed reolace text using regex -Esed "s/ORIGINAL/REPLACEMENT/[FLAGS]" [FILENAME]sed "s/this/that/" file replace first this with thatsed "s/this/that/g" file replace allsed -E "s/hi|bye//g" file remove all hi or byefind search for files in a directory-name search by name-type search by file type-maxdepth DEPTH descend into directory to DEPTH-prune ignore a directory and files under itfind . -type d -exec echo {} >> matches.txt \; \; completes exec#!/usr/bin/bash#!/usr/bin/env bashman testecho Hello world! # => Hello world!
echo ‘This is the first line’; echo ‘This is the second line’
first=Bruce #no space!
last=Wayne
echo $first
name=”$first $last”
echo “${name} is Batman”
dirs=$(find . -maxdepth 1)
echo “My current directory is $(pwd)”
a=$((1+3))
b=$(($a+3)) #b = 7
echo “Last program’s return value: $?”
echo “Script’s PID: $$”
echo “Number of arguments passed to script: $#” #not include program name
echo “All arguments passed to script: $@”
echo “Script’s arguments separated into different variables: $1 $2… ${10} …”
a=pwd
echo ‘$a’ # $a single quotes use literal
echo “$a” # pwd double quotes expand backticks and $
echo $a # /usr/… backticks expand and execute as shell command
echo “There are ls | wc -l items here.”
echo “always executed” || echo “only execute if first fails”
echo “always executed” && echo “only execute if first not fails”
numer=$(($RANDOM % 61 - 30)) between -30 to 30
if [ $number -eq 0 ]; then
echo “Zero”
elif [ $number -gt 0 ]; then
echo “Positive”
else
echo “Negative”
fi
if [ -d abc ] #True is abc exist and is a directory
echo “What’s your name?”
read Name # Note that we didn’t need to declare a new variable
echo Hello, $Name!
if [ “$Name” == “Steve” ] && [ “$Age” -eq 15 ]
then
echo “This will run if $Name is Steve AND $Age is 15.”
fi
if [ “$Name” == “Daniya” ] || [ “$Name” == “Zach” ]
then
echo “This will run if $Name is Daniya OR Zach.”
fi
=~ operator, which tests a string against a Regex pattern:Email=me@example.com
if [[ “$Email” =~ [a-z]+@[a-z]{2,}.(com|net|org) ]]
then
echo “Valid email!”
fi
case statement that works similarly to switch in Java and C++:case “$Variable” in
#List patterns for the conditions you want to meet0) echo "There is a zero.";; #;; is like break1) echo "There is a one.";;*) echo "It is not null.";;
esac
for i in {1..3}; do
echo “$i”
done
for ((i=1; i <=3; i++)); do
echo “$i”
done
while [ true ]
do
echo “loop body here…”
break
done
greeting=”hello world” hello
for word in $greeting; do world
echo “${word}”
done #IFS is internal Filed separator, default is SP
cell=”123-456-789”
IFS=’-‘
for num in $cell; do 123
echo ${num} 456
done 789
for i in $(seq 0 ${#greeting}); do #each line is a character
echo “${greeting
1}” #IFS=’’ not work
done
fruit=(“apple” “banana” “cherry”)
echo ${fruit[0]} #apple
echo ${fruit[1]} #banana
echo ${fruit[@]} #apple banana cherry
echo ${fruit} #apple
for fruit in ${fruits[@]}; do
echo ${fruit}
done
declare -a hw #declare an array hw
hw[0]=”hello”
echo ${hw[@]} #hello
hw+=(“world”) #appending
echo ${hw[@]} #hello world you can add 1 to hello
equal() {
if [ $1 -eq $2 ]; then #arguments $1…
return 0 #exit success
else
return 1
fi
}
equal 5 9 no return, no x=equal 5 9
echo $? #1
equal 5 “5”
echo $? #0 they are equal
### Customize Command in ~/./profile`source filename` source命令通常用于重新执行刚修改的初始化文件，使之立即生效，而不必注销并重新登录。该命令通常用命令“.”来替代`./profile` will execute everytime login
up(){ #Usage: up 2 cd .. cd ..
local limit=1
if [ $# -eq 1 ]; then
limit=$1
fi
for i in $(seq 1 $limit); do
cd ..
done
}
### [Regular Expression Cheat Sheet](https://www.rexegg.com/regex-quickstart.html)- to match '-' in Character set [-abc] or [abc-]- POSIX Bracket Expression- [ab[:digit:]] is same as [ab0-9]- Capture groups (abc){3} matches abcabcabc, \# match precious capture group, start from 1`<([A-Z][A-Z0-9]*)\b[^>]*>.*?,/|1>` match HEML tag- `yes|no` match `yes` or `no`- `G(oo)+gle` match even number of `o`- `h(im|er)` him or her- `\d{3}-\d{3}-\d{4}` 123-456-7890 phone number> Basic vs. Entended Regular Expressions- Basic regular expression(BRE): ? + {} () | are nomal, escape as metacharacter- Extended reuglar expression(ERE)### Exercise- match favorite or favourite- `favorite|favourite`- `favou?rite`- match variation of Google with 2-7 os- `Go{2,7}gle`- match line where the last word ens in 'st' or 'de' and the line optitional end in punc- `^.*(st|de)[\.,';]?$`- `[(st)(de)] is invalid`- match binary string of 00111, 001 even 0 odd 1- `^(00)*1(11)*$`- match line that is exactly an int with an optitional leading + or -- `^[+-]?(0|([1-9][[:digit:]]*))$`- [Regex Test](https://regexr.com/)------------------------------------------------------------## Patches Compilation and MakefileBack to Content Tabble](#linux-enviroment-experiment)### Diff and patches- `diff -u file1 file2`- Comapre the difference in files line by line- -u output in diff formatc
—- path/to/originalfile last modified data
+++ path/to/modifiedfile last modified data
@@ -startingline,endingline +startingline,endline @@
patch -p[N] < patch_file
CC=g++ # compilerCFLAGS=-Wall -g # compiler flags# Running `make` (no target) defaults to the first target, conventionally "all"all: bookstorebookstore: coffee.o book.o store.o # final binary depends on object files$(CC) $(CFLAGS) -o bookstore coffee.o book.o bookstore.ocoffee.o: coffee.cpp coffee.h$(CC) $(CFLAGS) -c coffee.cpp # -c says not to run linker; so just creates .o filebook.o: book.cpp book.h$(CC) $(CFLAGS) -c book.cppstore.o: store.cpp book.h coffee.h$(CC) $(CFLAGS) -c store.cpp$Makefile for dynamic linkingrandmain : randmain.o randcpuid.ogcc $(CFLAGS) $^ -ldl -Wl,-rpath,. -o $@randlibhw.so : randlibhw.cgcc $(CFLAGS) $^ -shared -fPIC -o $@randlibsw.so : randlibsw.cgcc $(CFLAGS) $^ -shared -fPIC -o $@clean: # cleans generated build filesrm -f coffee.o book.o bookstore.o bookstore
int *ptr;int x = 77ptr = &x;*ptr++;void increment(int *num){ *num++; }int x = 10;increment(&x);#function pointerint add(int x, int y){ return x + y; }int main(void){int (*add_fn_ptr)(int, int) = &add;int sum = (*add_fn_ptr)(2, 3);}#qsortint compare(const void * a, const void * b){ //generic pointer type to fit qsortreturn (*(int *)a - *(int *)b); //cast to int pointer, then defference}int main(void){int values[] = {40, 10, 100, 90, 20, 25};qsort(values, 6, sizeof(int), compare);for(int i = 0; i < 6; ++i){printf("%d\n", values[i]);}}
struct Point {int x;int y;}struct Point p1 = {1, 2}; //Initialize (ordered)p1.x = 3;struct Point p3 = {.x = 5, .y = 6}; //alternative initializetypedef struct Point {int x;int y;} Mypoint;Mypoint p1 = {1, 2};
printf("x is %d", x) always to stdoutfprintf(FILE *fptr, const char * format, ...);sprintf(char *str, const char *format, ...);snprintf(char *str, size_t size, const char *format, ...);
char buffer[50];sprintf(buffer, "Jack");fprintf(stdout, "Hello, %s\n", buffer);
getchar putchar Functions to read/write a single character from stdin/stdout
#include <stdio.h>// implements cat: copies stdin to stdoutint main(void){char c;while ((c = getchar()) != EOF) // EOF indicates end of fileputchar(c);}
ptr = (type *) malloc(number_of_bytes)
char* letters; // Pointer for memory blockint n = 5; // Number of array elementsletters = (char *) malloc(n * sizeof(char));if (letters == NULL)// Check memory allocated correctlyprintf("Error allocating memory\n");free(letters); // Deallocate
ptr = realloc(ptr, new_byte_number)
int *arr = (int *) malloc(sizeof(int) * 10);arr = realloc(arr, sizeof(int) * 100);
strace ./program Intercepts and prints out system calls to stderr or to an output filestrace –o strace_output ./my_programstrace -c prints out a more readable summary reportwrite(1, "Hello world!\n", 14) write 14 byte to 1(stdout)syscall(SYS_write, 1, "Hello world!\n", 14) SYS_write is #define as 1, the syscall number, 1 is stdoutBuffered operations are more efficient, while unbuffered operations are instantaneous
time COMMAND of time ./PROGRAM
fflush(File *stream) — forces a write of all buffered data
#include <unistd.h>pid_t getpid(void) // returns the process id of the calling processint dup(int fd) // duplicates a file descriptor fdint fstat(int fd, struct stat *buf) // Returns information about the file with the descriptor fd into bufssize_t read(int fd, void *buffer, size_t n) // read n number of bytes from file described by fd to bufferssize_t write(int fd, void *buffer, size_t n) // write n number of bytes from file described by fd to bufferint open(const char *pathname, int flags) // given a pathname for a file, returns a file descriptor (int)int close(int fd) // closes given file descriptor
#include "add.h"
gcc -c add.c -o libadd.o ar rcs libadd.a libadd.o gcc -c main.c -o main.o gcc main.o libadd.o gcc -fPIC -c add.cgcc -shared -Wl,-soname,libadd.so.1 -o libadd.so.1.0 add.o ln -sf libadd.so.1.0 libadd.so.1 soname ln -sf libadd.so.1 libadd.so link name gcc -c main.c -o main.o gcc main.o -L. -Wl,-rpath,. -ladd ldd a.out show dynamic library included Dynamic loading: Works with shared library
use -ldl flaggcc main.c -ldl -Wl,-rpath,.
/* main.c /*int main(void){#include <stdio.h>#include <dlfcn.h>void *handle = dlopen("libadd.so", RTLD_LAZY);if(!handle){fputs(dlerror(), stderr);;return 1;}int (*add)(int, int) = dlsym(handle, "add");char *error = dlerror();if(error != NULL){fputs(error, stderr);return 1;}int sum = add(2, 4);printf("%d\n", sum);dlclose(handle);return 0;}
void * dlopen(const char *filename, int flag) flag:RTLD_NOW/RTLD_LAZYvoid * dlsym(void *handle, char *symbol)char * dlerror()
int dlclose(void *handle)__attribute__((__constructor__))void to_run_before(void){printf("Ran with dlopen\n");}__attribute__((__destructor__))void to_run_after(void){printf("Ran with dlclose\n");}
git checkout <commit> change status(files…) into status recorded by 
git initgit add READNE.mdgit commit -m "first commit"git remote add origin URLgit push -u origin master #if git push, shows "current breach master has no upstream branch"git branch -u origin/master current branch track origin/master
git initgit clone URL
git status
git add <file> Move files with changes to the staging areagit rm <file> Remove files from the staging areagit commit Creates a new commit finalizing all the changes in the staged area git commit -m "<message>" “Make xyzzy do frotz” or “Add file”git log show commit historygit log --stat show stats on file changesgit log -p show full detail on each commit change with diff format, can really see diff each line of srcgit log --pretty=oneline condense git log output to one line with just hash (SHA-1 hash)git push <remot> <branch> <remote>'s <branch> with the local <branch>git push origin mastergit remote add <name> <server> If you have not cloned an existing repository and want to connect to agit fetch <remote> retrieves info about any remote file changes git pull retrieves info and copies the changes, combine with current snapshot, Essentially does a git fetch and git merge
setting->SSH and GPG keys->new ssh key < car ~/.ssh/id_ras.pub
Use other branches for developing features, bug fixes, experiments, and merge them back to the master branch upon completion
HEAD Pointer
git branch
git branch <name><name> pointing to the current commit git branch -d <branch> Deletes a specified branchgit branch -m <branch> Rename current current to git checkout <name> Checks out (or switches) to the branch, by moving the HEAD pointer to it
git checkout -b <name> Quickly create a branch and check it outgit checkout -b <local> <remote>/<branch> Creates a local branch <local> that will be copied from and track <remote>/<branch>git checkout <branch> will automatically create a tracking branch for you if the branch name you try to checkout (a) doesn’t exist locally and (b) exactly matches a remote branch namegit branch -u <upstream> <local> makes an existing local branch track a remote upstream branchgit branch -vv displays which branches are tracked and which are notgit push -u <remote> <branch> Pushes to the remote branch and sets it as the upstream branch for the current local branch. We tend to do this the first time we push from a local branch to remote.git push origin master then git branch -u master origin/master Three way merge
git merge <branch>joins the branch specified into the current checked-out branch
- Merge conflicts happen when merging branches have different changes to the same part of the file and Git doesn’t know which version to keep You’ll need to resolve this manually (git mergetool)
- Uses three commits to generate the merge commit: the two branch tips and their common ancestor
Fast forward merge
- Instead of actually merging the branches to combine histories, Git moves (i.e. “fast forward”) the current branch tip up to the target branch tip
- A fast-forward merge can occur when there is a linear path from the current branch tip to the target branch
- A fast-forward is not possible if the two branches have diverged
git tag list all the current tagsgit tag -a <tag> -m "message" create an annotated taggit stash
- Stash the changes in a dirty working directory away- View stashes using `git stash list`- To re-apply an older stash, use `git stash apply stash@{#}` where # is the number of the stash as shown in the stash list- This is useful if you have some in-progress work in one branch that's not ready for staging, and you need to switch to another branch quickly
git diff
- Show all current changes in diff file format on any Git data source: commits, branches, files, etc.
.gitignore A gitignore file is a list of file paths that won’t show up as untracked files and won’t participate in version control unless you force them to, Supports wildcards for matching sets of files (e.g. *.txt)
git show assuming the hash is for the commit at the head of a branch, featuregit show 1c002dd4b536e7479fe34593e72e6c6c1819e53bgit show 1c002dd4b536e7479fgit show 1c002dgit show feature
git format-patch -n <object><object> can be a single commit or rangegit am < file.patch Apply the patch to your working directory.git/objects/.git/refs/.git/refs/heads/mastercat .git/refs/heads/master 487e009ed7294021dcc0b66c2b5a5046aea51ab4cat .git/HEAD ref: refs/heads/quote.git/index A binary file that acts as the staging area (also often called the index)
git ls-files --stage see what’s stored in the index file: a sorted list of the path names, each with permissions and the SHA1 of a blob object.git/hooks script that will run after each commit, you can put test in it
git commit history can be represented as a DAG
Topological sorting parent apears before child