oneline functions

make a function which returns every input to first three characters

truncate() { echo "${1:0:3}"; }
copy

the commands inside the braces need to be finish by semicolon if the closing brace is on the same line or in a script

truncate() {
echo "${1:0:3}"
}
copy

sort: sort csv files easily

To sort text files including comma delimited (csv) files.

• -t, sets delimiter to a comma, if left empty, uses space as a delimiter
• -kN,M from Nth-column to M-th column. if ,M is ommitted uses N and all columns after N for sorting
• f flag to sort alaphabetically while ignoring case (otherwise all capitals preced all the lowercase in sorting)
• n flag to sort numerically
• r flag reverses sorting (from biggest to lowest)
• -t$'\t' to set tab as a seprator

sort example

sort -t, -k1,1f -k3,3nr <file>
copy

Sorts a commma delimited file by first and third column alphabetically and numerically in descending order (respectively)

sort and keep the header unsorted

{ head -n1 file.csv; tail -n +2 file.csv | sort -t, -k1,1 ; } > sorted_file.csv
copy

read: read a file line by line and do something with it

Read a file and reverse all lines

while IFS= read -r line; do
    echo "$line" | rev
done < filename.txt
copy

-r flag ignores \ as escape characters IFS= prevents whitespace to be trimmed from front and back also -a flag will enable reading into an array which starts with an index of zero

heredocs: write line from command line

cat <<EOF > terminal-file.txt
# here write what you want on multiple lines
# finish by typing EOF
# In case you want to use a different signal instead of EOF use a different word
# cat <<END > terminal-file.txt will be ended by END
copy

sed

How sed operates

sed operates on individual lines of the input

1. reads first line - puts into a pattern space
2. excutes commands
3. prints what is in pattern to output
4. clears the pattern space
5. reads next line

So if you supply no commands to sed sed '' file it just outputs the content of the file. It is the same as running cat file

The -n switch prevents default printing

There is an important switch -n going ahead of the commands. When -n flag is present it negates the action of a 3rd step: negates content of pattern space to output. Example

sed -n '' file` 
copy

runs and prints nothing

Telling the sed what to do with which lines

The ’ ’ structure is used to define range of lines on which to operate specific commands.
The general structure is: sed 'ranges commands'. More commands can be included inside the ’ ’ to run consequtively when seprated by semicolon: sed 'ranges commands1; commands2'

Ranges

Ranges are defined by line numbers or pattern matching

Example

'1,2' first two lines
'1,+N' first line plus N more lines
'/pattern1/,/pattern2/' on lines from appearance of first pattern to the first occurence of the second pattern
copy

Commands print, delete, change, append

These commands tell sed what to do with the content of the pattern space. Caution: Remember that the default action of sed is to print everyting which is in the pattern space after the commands on the line are executed.
This will lead to suprising doubling of lines when using print.
Use -n flag to supress this behaviour.

print

Examples:

sed '1,2p' file
copy

Will print the first lines twice each then the rest as normal.

sed -n '1,2p' file
copy

Will print the first two lines, does not print the rest

### delete
`sed '1,2d' file` deletes first two lines
`sed '/pattern1/,/pattern2/d file` deletes lines from pattern1 containing line to pattern2 containing line

### change
`sed '1,2cXXXX file'` will replace the first two lines with a single line containing XXXX
In case you weant to have a multiline replacement use '\n' inside the string 

### read
`sed '1,2rfile2' file1` will put:

* first line of file1
* whole file2
* second line
* whole file2
* the rest of the file

### substitute `s/pattern1/pattern2/gI`
To replace a pattern1 with a pattern2.
**Examples**
`sed 's/hello/goodbye/' file` will replace the first occurence of hello with goodbye on all lines.\  
`sed 's/hello/goodbye/g' file` will replace all occurences (g goes for global).\
`sed 's/hello/goodbye/gI' file` will replace all occurences of hello case independently (Hello, heLLo and so on).\

To delete pattern use `sed 's/hello//' file` which will delete first occurence of hello.

To delete a pattern in specific range `sed '1,2s/hello//g' file` will replace all hello occurences on first two lines.

### Using regular expressions in substitute command
1. It is possible to use regular expressions for the patterns, like [a-Z][0-9], 
2. It is possible to group the regular expressions and use them for the substitution
**Example:**\ 
`sed 's/_\([0-9]\)/_a\1/' file` will insert "a" before a first digit following an underscore.

  


# YAML
## Background and use-cases
Looks YAML is perfect for some of my use-cases:

1. Make a human editable textfile lists which can be converted between plain text file and csv
2. Use the YAML for supplying setting for BASH scripts
3. Use the YAML for supplying settings for R scripts

## YAML syntax
### key value pairs 
`key: value`

### Lists(Arrays)
```YAML
fruits:
  -apple
  -orange
  -banana
copy

Dictionaries (mapping)

address:
  street: 123 Main st
  city: Brno
  zip: 12345

Nested Structures

person:
  name: John Doe
  age: 30
  children:
    - name: Jane Doe
      age: 10
    - name: Jake Doe
      age: 7

Multiline Strings

using | or >

description: |
  This is a multi-line string.
  Line breaks are preserved.  

Anchor and alias

& is used to set what gets anchored and * is used to recall.
Different parts can be overwritten

default: &default
  name: Default Name
  age: 30

person:
  <<: *default
  name: John Doe

YAML parsing by yq

Installation

sudo snap install yq\

Usage

Make the testfile

name: John Doe
age: 30
address:
  street: 123 Main St
  city: Springfield
copy

Networking

Resource

baeldung linux

What do I want?

Figure out why i am not able to connect to my pizero over wifi

1. Find out what are the ip addresses of the devices connected to a wifi network

Hypothesis:

1. I have the name of the zero wrong
2. It does not want to talk using hostname, only ip address
3. there is some specific problem/bug connected to a hotspot from a mobile deviceO

Finding wireless access point ip address

ip route | grep default default via 192.168.88.1 dev wlp2s0 proto dhcp metric 600 sudo nmap -sn 192.168.88.1/24 sudo nmap -sL 192.168.131.115/24 sudo nmap -vv -sP 192.168.131.115/24 sudo nmap -sS -sV -vv -n -Pn -T4 192.168.131.115/24

Hardware commands

Resource

Learn Linux TV\

The ls* suite

• lsusb gives you list of all usb devices, can be useful to figre out whether usb is even detected.\
• watch -d lsusb can enable you to go around the ports and test where it appears.\
• lsusb -t gives you a tree view of the devices under the different busest\
• lspci gives you a lot of info on the pci components.\
• lshw -html > hw.info creates an html file with all the hardware spec. Truly amazing.\
• lshw -short gives you shorter version.\
• lscpu info about the cpu characteristicsi\
• lsblk give you the info about the disks\
• watch lsblk is super useful to run in paralell terminal to see adding and removing hardware you want to format and so on.
  =======

Logging what happened for stack-overflow posts

1. Make a troubleshoot file

touch troubleshoot
copy

2. Save the command into the file

echo "your command" >> troubleshoot
copy

3. Run the command and collect both stdout and stderr into troubleshoot

$( tail -n1 troubleshoot) &>> troubleshoot
copy

4. Check what happened

cat troubleshoot
copy

Maybe there is an easier way, but this works well enough for me.

Mounting volumes

Motivation

1. Be able to find what devices are connected to my machine
2. Be able to mount the disk devices to a place where it is easy to use (for copying and stuff)
3. Learn how to create linux and windows compatible USB

Resources

video on learn linux tv

lsblk list block devices

• gives all the devices, the hard drives are sda and number\
• the usbs are sdb and number
• it also shows where it is mounted (for example the normal volume I have is mounted as / )
• if the device is not mounted automatically nothing is written there
• each disk device has a file in /dev/sda and like that
• lsblk -f get the type of filesystem

sudo fdisk -l gives info about the devices

• needs to be run with sudo
• gives more detailed information, make of the device, sectors size, disk label, disk identifier etc.
• the disk is split into devices (in case there are more partitions)

mount list the mounted devices

list all the mounted devices, there is a lot, so most usefully grep for the disk name to get what you want.

mount device dir mount the file where you want it

the trick is what you see in lsblk is sdc1 etc.
In linux everything is a file, and disks live in /dev/ folder, so when mounting it you must use the full path /dev/sdc1\

example command:
mkdir /home/vladimir/usb # make a folder where to mount
sudo mount /dev/sdb1 /home/vladimir/usb mount the usb

the proper place to mount to is /media for temporary disks like usebs
the proper place to mount a harddrive is /mnt\

sudo umount dir to unmount

just put the path to the directory you want to unmount, not the name of the device

df -h how much of a filesystem is used

there is also an ncdu utility which is more useful, but needst obe installed first whihc I did not do today

Reformatting USB to be windows and linux compatible

use parted to create a new partition table

• lsblk to find what are the devices (lets see we get our device on sdb)
• sudo parted /dev/sdb start parted on this device
• (parted) mklabel msdos will have some messages about:
  • use of partitions -> ignore
  • data will be lost -> yes
  • Error unable to inform the kernel -> Ignore
• (parted) mkpart primary ntfs 1MiB 100%
• (parted) quit

Format to NTFS

• sudo umount /dev/sdb1 need to unmount the volume otherwise throws an error
• sudo mkfs.ntfs -f /dev/sdb1 to format
• lsblk -f verify the partition and filesystem

if USB does not show up on Windows:

1. on windows: Open Disk Management by pressing Win + X and selecting Disk Management.
2. Look for the USB drive in the list of drives. If you see the USB drive but it doesn’t have a drive letter, you can assign one:
3. Right-click on the partition and select “Change Drive Letter and Paths…”
4. Click “Add” and assign a new drive letter.
5. System volume information gets added to the usb and solves the problem

Input/Output error

On an SD card not able to write an image, not able to reformat (win), not able to proceed with parted (Input/Output) error.

1. overwrite the SD card by zeros: sudo dd if=/dev/zero of=/dev/sdb bs=4M status=progress
2. try parted as above
3. check the kernel messages: sudo dmesg | tail

At the end decided to trash the SD card, was corrupted

Managing groups and users

Motivation

1. Make new users
2. Give them access to a shared folder
3. Enable them to execute the same programmes

Resources

• [link to video from linode](link will be here)\
• [link to video about permissions in general](link will be here)\
• chapters from the linux administration book

Basic ideas:

1. everything is a file
2. permissions to files are granted to a. owner b. group c. all
3. Permissions of the file are normally inherited, or can be user set or group set by setUID setGID
4. At creation every user is added to a group bearing its name
5. Each group can have multiple users
6. Each user can be in multiple groups
7. Many services (ssh, xrdp etc.) create a group, by adding the user there enables him to use that services

Basic commands:

useradd <username> adds a user with default values

• base home directory for the user is created in the home directory
• the account does not expire
• there is no password created at the time of creating the user
• bash is selected as a default shell
• the user gets a group of his/her own
• it is possible to make a system user with an useradd -r <username> which is non-login and used for automation of tasks

Example code:

sudo useradd -s /bin/bash -m <username> # to make the user
sudo passwd <username> # to set the password
su <username> change to the user account
xdg-user-dirs-update --force create the desktop/downloads directories

groupadd <groupname> creates new group

usermod -aG <groupname/s> <user> add user to group

to remove user from group run usermod -G <all groups but the one you want to remove the user from> user

groups [user] lists the groups of user

passwd [user] create password

Using process redirection

there is an interesting concept of creating temporary files using the process substitution
syntax:
diff -y <(sort file1) <(sort file2)\

echo diff -y <(sort file1) <(sort file2)\

Useful commands

true

true results in exit code 0

sleep

sleep gives you sleep for a given time for seconds minutes hours days.
sleep 1s or sleep 1m and so on.
works also for the floating points.\

shift

shift moves the positional parameters by [N] close to zero (defaults to 1 if not specified) and reduces the $# by [N].
Returns exit code of 0 unless N is negative or greater than $#.
So basically you put it at the end of the while loop defined as while [[ $# -eq 1 ]]
I used it in a while loop to assign the first parametr into username and then shifted, this way the while loop exited and the following code expanded the rest of arguments into the comment variable.\

basename

basename strips the string leading up to the last string after a /.
It works on strings so you can apply it to a string, and it will do its job despite the string being a path to a file which does not exist.\

dirname

Strips the filename and gets the last directory.\

cut

Splits the standard input or a file based on characters, fields, delimiters etc.
Enables using ranges which can lead to some confusions.
There may or may not be a space between the -c flag and thenumber or dash for ranges

Using the -c flag with a range or a single field

echo "apple" | cut -c 2-4 gives you characters 2 to 4 > ppl
echo "apple" | cut -c -4 gives you characters 0 to 4 > appl
echo "apple" | cut -c 2- gives you characters 2 to end > pple
echo "apple" | cut -c2 gives you character 2 > p\

Using the -c flag with a comma

echo "apple" | cut -c 2,4 gives you characters 2 to 4 > ppl
echo "apple" | cut -c 1-2,4-5 gives you characters 1 to 2 and 4 to 5 > aple\

using the fields by -f and -d flag

can be used to get the fields separated by a given separator, super useful for working with csv files
echo -e "1,2,3\na,b,c" > commatest to create a comma separated file
cut -d "," -f 2-3 commatest gives you the second and third column.
The delimiter needs to be a single character.
COnfusingly the -d flag cannot stand alone, if you are splitting on comma, it must be -d, or -d "," better stick to the more general syntax, because -d; does not work.

variables in bash

man bash and there /shell variables to jump right there for definitions.

executing commands and programmes

There ares several important concepts and commands to know when it comes to executing commands and your own programmes.

• PATH variable
• which command
• type command
• hash -r command
• /usr/local/bin location where to keep the personal scripts (can be kept apart from the rest of the system in case you upgrade)

PATH for search of the commands

PATH is a bash variable which specifies the folders where the operation system goes to search for the commands(programmes) to be executed.
It starts with the lowest (closest to the user ) such as /usr/local/bin and then goes more general to allow for the customization of the commands.\

which

the which command tells you where is the command you are interested in located.
which <command>\

type

type tells you what kind of a command it is whether it is a shell builtin or a proper programme outside of the shell.
type -a <command> gives you all the locations of the command which are available.

Putting your own command into the path to be general and then removing it

Being able to write some small scripts which are available all the time is super powerful feature of linux enabled by the abovementioned PATH variable.
Because PATH variable defines where the system looks for the commands, that allows you to create a command/script, copy it into a folder on the path and then use it from anywhere.

Example of creating a new head command

Create a script called head and put it into a /usr/local/bin/.
Change the permissions for it to be executable.
This command will be executed first when I call the head command.
Be careful when adding and removing your own commands.\ Their existence is not checked every time it is called. Instead it si hashed.
So after you remove your own command and you try to run it, it fails, because it goes to the place it remembers but does not find anything (instead of executing the next in line).
In case you run type <removed-command> it returns > the <removed-command> is hashed.
To make an update of these hashes run hash -r and everything is okey again.\

Positional parameters

First look at the positional parameters:\

echo "you executed this command: ${0}" gives you the name of the last command.
It is possible to omit the curly braces around the zero. It works the same but the curly braces are more foolproof.\

Expanding to number of positional parameters

${#} or $# expands to a number of the positional arguments supplied.
Can be very neatly used to give a help to the user.
In case the user does not supply any positional arguments tell them the usage of the command/script.\

if [[ "${#}" -lt 1 ]]
then
  echo "Usage: ${0} USER_NAME [USER_NAME]..."
  exit 1
fi
copy

For loop through the positional parameters

for USER in "${@}"
do 
  echo " User: ${USER} "
done
copy

Notes: it is important to quote the “${@}” in order to avoid splitting the user name which could contain spaces to separate fields.

While loop

Shell Scripting OCurse Adding the new users and modifications

Figure out what programmes is what

type <command>
find all types of the : type -a <command>

it is better to use the builtin is faster and will always work
to get help on builtin you cna use help

Understanding the UID and bash variables

UID is bash variable all bash variables are to be found in man bash
to get the username use the programme id get the all the info id
get the user UID id -u
get the user group id -g
get all the groups ids id -G
get just the user name id -un
get just the group name id -gn\

get the result of a command into a variable

USER_NAME=$(id -un)

check if the user is root:\

if [[ "${UID}" -eg 0 ]]; then echo "you are a root; else echo "you are not root"; fi
copy

also tested here the strings\

if [[ "${USER}" = "${USER_NAME_TO_TES_FOR}" ]]
  then
  echo "your username matechs ${USER_NAME_TO_TEST_FOR}"
fi
copy

Exit status

you can tell the script to exit by a specific exit code using exit[n]

the last exit status is stored in ${?} variable

getting user input

using the read builtin to get user input into a variable called VAR:
read -p 'this is a prompt: ' VAR
In case there are more variables defined:
read -p 'this is a prompt: ' VAR1 VAR2 VAR3
the input is split at the delimiter. In case there are more of the pieces then variables the leftover goes into VAR3.\

Permissions

order of bits is: user, group, all, other users who are not ina file group.\ the values for the bit codes are as below:

• r=4
• w=2
• x=1

The defaul permission is chmod 755 (owner can modifiy it, others can just read it and execute)

to run the bashscript you need to use ./script syntax, because it is a file, not a programme (that runs just with script)

How to add a new local user

using the useradd builtin. the LOGIN parameter is a username. By usage should not be longer then 8 characters, you can yse numbers also. The too long names still work, but get shortened in listings etc.

Creating the new name user: useradd dougstamper
Check that you can log into him: sudo su - dougstamper
The su - starts it as a login shell which is better somehow\

In case you use -c to get the comment field which can be used to put the whole name.
In case you want to create the home directory use the -m flag.
The creation of the home directory is decided by the /etc/login.def config file, where in my case the CREATE_HOME is set to yes. however it can be different on different systems so to make sure it always works, you should use the -m flag.

Deleting the user

to delete user run userdel <username>. This command removes the user, but keeps the home directory, so in case you make the same name of the user he/she will get the same home directory as the previous one.\

Set/change the password for the user

command passwd activates the change of the password for the interactive shell.
In case I want to supply the password from the script I need to use the --stdin flag.
To make sure that the user changes the his/her password I can use the -e expire flag which will force the user to change the password after first login.\

To change the password of the specific user you need to either be logged in as that user or do it as a root.
sudo passwd <user> works\

In case you want to change the password from the script use the possibility of password coming from standard input using echo: echo ${PASSWORD} | passwd --stdin\

Installing ivrtual box and vagrant

resources

virtual box link following the command line options wget -q https://www.virtualbox.org/download/oracle_vbox_2016.asc -O- | sudo apt-key add - sudo add-apt-repository “deb [arch=amd64] http://download.virtualbox.org/virtualbox/debian $(lsb_release -cs) contrib” sudo apt update && sudo apt install virtualbox-7.0

vagrant installation copy paste to the terminal wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg –dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg echo “deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main” | sudo tee /etc/apt/sources.list.d/hashicorp.list sudo apt update && sudo apt install vagrant

The rename command (needs to be installed on 20.04)

Motivation and use cases:

1. replace some common characters “-”, " “,”_",
2. remove a matching pattern - only one or greedy
3. add something to the start, to the end
4. remove something from the start, from the end
5. know how to do the dry run
6. know

Resources

link to article

Basic usage

rename [ -v ] [ -n ] [ -f ] perlexpr [ files ]

-v is verbose, prints the names of the files successfully renamed -n no act, shows the files that would be renamed -f force overwrite existing files

Perl syntax

The rename follows perl syntax s/pattern1/pattern2/[gi]\

metacharacters:

1. ^,$ to match the start and and
2. ., +, ?, any character, preceding characters once or more, zero or one
3. | or operator
4. (…) grouping
5. […] set of characters
6. [^…] set of characters to be excluded
7. \t is a tab, \n is a newline, \r is a carriage return
8. \d is a digit \D is a non-digit
9. \w is a word character (alphanumeric or underscore), \W any single non-word character
10. \s matches single whitespace (space, tab, newline), \S single non-whitespace
11. \b zero width matching word boundary \B zero width matching all but not word boundary
12. \
13. {n[,[n]]} are the quantifiers
    • (pattern){n} - matches exactly the number
    • (pattern){n,} - matchs n and more
    • (pattern){n,m} -matches from n to m occurences There is a lot, let me settle with this.

Examples

replace characters

files: a-b-c c-d-e

to replace the first matching character:
rename -n -v 's/-/_/' *

output:\

rename(a-b-c, a_b-c) rename(c-d-e, c_d-e)

-n -v flag tells you what gets renamed

to replace all the matching characters in all files ("*"):
rename -n -v 's/-/_/g' *

replace the extension

rename -n -v 's/\.jpg/\.jpeg/g'

remove two different extensions

rename -n -v 's/(\.jpg|\.jpeg)//g'

add extension

rename -n -v 's/$/\.jpg/g'

translate from one character set to another

rename 'y/[a-z]/[A-Z]/'

in case you try to rename to something which already exists, does not do it unless you use the -f flag.

using the regex lookups

replace the last dash preceding the last character, which is a digit.

rename -n -v 's/-(?=\d$)/_/'

replace every dash preceded by a digit.

rename -n -v 's/(?<=\\d)-/_/'

remove spaces replace with underscores

using the string substitution and enabeling substitution of one or more spaces with an underscore:\

rename -n -v 's/\s+/_/g' <files>

Be aware, that using translate does not work!
I guess it is because \s is not a character.\

rename 'y/\s/_/' <files> # doesnt work

remove spaces and dashes with underscore

rename -n -v ’s/\s+|-/_/g'

replace weird characters to underscores as alias

rename dot, comma,semicolon,space, plus, question mark and dash to underscore to get “word” formatted name. Dot preceding extension (composed of digits and letters) is kept

alias to_underscores_test="echo 'this is testmode, use to_underscore_run command to rename'; rename -v -n 's/(\,|\;|\s|\+|\?|-|(\.(?!(\w|\d)+$)))+/_/g'"

alias to_underscores_run="rename -v 's/(\,|\;|\s|\+|\?|-|(\.(?!(\w|\d)+$)))+/_/g'"

Authorizing pushing to github from command line gpg and pass

Resource

link to generate gpg public/private keys

Issue

• when I tried the git push from repo, wanted to get the \

Done

• searched for the gpg keys: find ~/ -name .gpg-id -print not found
• checked for the gpg presence by searching for version: gpg --version
• installed gpg by: apt install gpg
• generated the gpg key: gpg --full-gen-key, used gmail email
• installed pass by: sudo apt install pass
• initialized pass by: pass init [email protected]

Notes

• works
• the and passprase is stored in bitwarden
• is an email you put in when creating the password

How to set up the protected instance on linode 02-02-2023

• made linode mbd-lnx-01 user vld a root
• passwords are in bitwarden
• did the securing as in the docslink
• super useful to put an alias to log in
• useful to add the IP address of the linode to ~/.profile and ~/.bashrc
• learned how to generate the keys
• set the keys in github to be able to communicate direactl from the command line
• installed csvkit, youtube-dl, and some other stuff lost now in it
• was not happy with the filezilla, so will use the sftp link
• big idea is to go through it and make a script which would enable me to run it automaticaly (also with the .vimrc and aliases etc)

26-11-2022

need to learn how to use the arrays, expansions of indices etc.
However the thing I need to do most is to clean this directory to have proper organization of the files so that I can fing what i need. What I am not able to find/do not search because it is not comfortable, is lost and have no purpose to keep it.\

Get the version of your ubuntu

20-11-2022

lsb_release -a

remap the capslock into esc key

for xfce

I followed following instructions link

and added this:
/usr/bin/setxkbmap -option “caps:swapescape”

into the autorun startup from the applications menu

works very well so far

20-11-2022

I followed this:link
put ['caps:escape'] into the dconf tool which was activated in terminal running: dconf-editor.
dconf-editor installed using these commands:

1. sudo add-apt-repository universe
2. sudo apt update
3. sudo apt install dconf-cli dconf-editor

rks

Learning the basics of a find command

02-09-2022

Motivation

Parsing output of ls into a variable and then executing commands is not a safe way.
According to bash pitfalls website there is a better way, the find command.
The find command is able to find stuff and then do something with it.
So lets go and explore:)\

Pitfall No1

I have several files with .mp3 suffix made by following line:
touch "**.mp3" "normal.mp3" "with break.mp3" "with a \n.mp3"

ls fail

I want to use ls command to do something with these files:\

find00.sh #/bin/bash/ for i in $(ls *.mp3); do echo ${i} done `

Output ls fail: **.mp3
normal.mp3
with a \n.mp3
with break .mp3
normal.mp3
with
a
\n.mp3
with
break
.mp3\

This and other variations like ls or find . -type f etc are all wrong
The reason is that there can be spaces, newlines, globbing characters and even “`” characters in the filenames.
And that messes up the function of a script.\

Additionally, the ls command works differently on different systems,
its main role is to make a human readable output. Thats it all it does.\

Normal globbing

find01.sh

#/bin/bash for i in ./*.mp3; do\ echo "$i" #always use double-quotes\ done

Output normal globbing ./**.mp3
./normal.mp3
./with a \n.mp3
./with break .mp3\

However if there are no files found, the loop will still execute ones,
because there is the “./*.mp3” in the $i variable.\

Output normal globbing without matching files
./*.mp3

To avoid this behaviour, there should be a test for existence of the file.\

Normal globbing with a test for file match

*find03.sh

#/bin/bash/ for i in ./*.mp3; do [ -e "$i" ] || continue echo "$i" done

This gives no output and therefore I can say it works.

Testing steps summary:

1. touch "**.mp3" "normal.mp3" "with break.mp3" "with a \n.mp3"
2. ./find00.sh>>output
3. ./find01.sh>>output
4. ./find02.sh>>output
5. rm ./*.mp3
6. ./find01.sh>>output
7. ./find02.sh>>output

Pitfall-01: find

how to use find resource\

find03.sh #!/bin/bash find . -type f -name '*.mp3' -exec echo {} \;

works both on the empty and normal folder with *.mp3 files.\

Overall the find command is super useful and would be good to learn more, so lets go and do it now.

What I would like to achieve eventually is to be able to move around files and give them the name according to either my selection or based ont he name o fhteir parent folder if it is somehow possible.

Find in more detail

• find descends through a hieararchy of files
• matches files which meet specified criteria
• does stuff to them

Example 1 - most basic find

find .

• you should always specify, where you want to search!\
• if you do not specify any criteria, you get all the types of files blocks, sockets etc.* if you do not specify an action, most find versions will print, it is always better to specify thus: find . -print
• it does not sort the output at all

Example 2 - find files by criteria

find . -name '*.mp3' -print

• find uses globs. These globs need to be protected from the shell expansion, thus the single quotes

Example 3 - find files by criteria using logical operators

find . \(-name '*.mp3' -o -name '*.jpg'\) - prinfind . (-name ‘.mp3’ -o -name ‘.jpg’) - print t`

so the and operator is implicit, the OR needs to be specified by the -o flag.\

additionally the -name flag searches only for the basename of the file.
In case you want to search in the folder names you must use the flag -path.\

that is a lot I learned today must stop now.

26-08-2022

link

SCP

From my computer to the remote

scp myfile.txt remoteuser@remoteserver:/remote/folder/

From remote to my computer

scp remoteuser@remoteserver:/remote/folder/remotefile.txt localfile.txt

Also works recursively

scp remoteuser@remoteserver:/remote/folder/remotefile.txt localfile.txt

So lets try it and see now

need to get the address of the ssh connection

ss | grep -i ssh | awk 'F" " {print $NF}'> cloud-desktop-ssh gives me the number and :ssh as a file in home directory and here when I delete the :ssh I can also make it a variable this way:

CLOUD="vld@cat ../cloud-desktop-ssh" but it is super tiring, there must be something better, at least to have it as an environmental variable forever

SFTP

link

How to use it

1. Navigate to the folder with the documents you want to copy to remote/where you want to recieve the files
2. get into the remote machine using sftp
3. figure out where you are and navigate where you want to get
4. upload or download files

1. Navigate where you want to get on local machine

Note: whatever starts with a ‘#’ is a comment explaining the command

• pwd # to see where you are
• cd # to get where you want
  • cd ~ # gets you to home directory
  • cd ~/Desktop # gets you to Desktop
  • cd ~/Documents/ # gets you to Documents
  • cd ~/Music/ # gets you to Music
  • cd ~/Videos/ # gets you to Videos
  • cd .. # gets you to the parent directory
  • cd - # gets you to the last visited directory
• ls # to see what files are in the directory

2. Log into the remote machine using sftp

sftp [user@]<remote-ip>

Note: The prompt will change to > and lose colours

3. Navigate the remote and local directories

To navigate through the file system of the remote machine use the same commands as you used on the local machine before

Notes:

• In case you want to go back to the local machine and navigate the filesystem there, put l before the normal commands.

• The expansion for home directory does not work in the sftp mode

• All the paths in sftp are relative, start from pwd and append whatever path you say so say if you are in /home/username/Documents and you type put *.jpg /home/username/Videos the computer will create a path /home/username/Documents/home/username/Videos and throw an error.

• pwd is working directory on remote; lpwd is working directory on local

• cd change directory on remote; lcd change working directory on local

This works for all the “normal commands” (pwd, cd, cp, mv, rm etc.)

4. Upload (put) and download (get) files to and from remote

put *.jpg # uploads all .jpg files from local working directory to the remote working directory

Overview of some commands and flags

Note: whatever starts with a ‘#’ is a comment explaining the command

• put # to copy a file from local to remote
• get # to copy from remote to the local
• ! # drops me into the local
• -r # recursive
• -P # copy permissions
• df -h # tells you the space on the disk where the folder is!
• ? # gives you help
• cd # change remote directory
• lcd # change local directory
• pwd # remote working directory
• lpwd # local working directory
• cp # copy on the remote
• lcp # copy on the local
• mv # move/rename on the remote
• lmv # move/rename on the local

Little bit of awk

this gives you the size of all the files containing shiny in the given folder field number $5 is the size ls -la |grep shiny | tail -5 |awk 'F" " { size=size + $5; print $5 } END { print "total size is " size/(1024*1024) "M" }'

ALiases

14-08-2022

Motivation

Be able to execute the commands which I do more common without typing out so much
Examples:\

1. convert by pandoc from md to pdf, md to docx and docx to md
2. download the mp3 or mp4 from a youtube link

General

There is a ~/.bashrc file where I can put the aliases uses a convention alias CMD1='CMD2'.
The ~/.bashsrc file needs to be first copied to ~/.bashsrc.bak by cp ~/.bashrc ~/.bashrc.bak
After modifying the file needs to be sourced, not executed: . ~/.bashrc
Cleaner way to do it is: to make a file called ~/.bash_aliases where I put the aliases and put following in the ~/.bashrc:
if [ -f ~/.bash_aliases ]; then . ~/.bash_aliases fi

Resources:

link

Test

ll

alias ll='ls -l'

Steps

1. cp ~/.bashrc ~/.bashrc.bak
2. echo “alias ll=‘ls -l’” > ~/.bash_aliases
3. echo “if [ -f ~/.bash_aliases ]; then . ~/.bash_aliases; fi” » ~/.bashrc
4. . ~/.bashrc WORKED!

now do it by a script! MAybe later!

reverse back

1. mv ~/.bashrc.bak ~/.bashrc
2. . ~/.bashrc

write the create_aliases.sh script

1. #!/bin/bash/
2. echo “cp ~/.bashrc ~/.bashrc.bak” &&
3. echo “alias ll=‘ls -l’” > ~/.bash_aliases
4. echo “if [ -f ~/.bash_aliases ]; then . ~/.bash_aliases; fi” » ~/.bashrc
5. . ~/.bashrc WORKED!

The power of symbolic links

13-08-2022

Motivation

To have the files better organized, have them at one place and call them from various places if needed.

General

The symbolic links enable you to have one source file/folder and approach it from different places by symbolic links. YOu can modify the file content, but when you delete it, it is just the link, which was deleted.

Syntax

ln -s creates a symbolic link ln creates hard link, if you delete the hard link, the original is gone too.
Unless you know what are you doing, use the symbolic links!

TARGET only

To make a symbolic link leading to a file/folder in a place with [TARGET] path into current folder.
the name of the link is the same as the name of the original ln -s [TARGET]

New name of link

to make a symbolic link with a brand new name [LINK_NAME] which directs to file/folder in a place with [TARGET] ln -s [TARGET] [LINK_NAME]

Important Better use the full path strarting with ~/ because otherwise the symbolic links do not work from the desktop

link a directory under a new name

ln -s [TARGET] [DIRECTORY]
*the directory is created by this command, shell not have a “/” at the end otherwise it does not work

You can have multiple symbolic links leading to the original file

Exploring the for loops to rename and move files

11-08-2022

Motivation

1. Be able to name the files in the folders according to a name of the subfolder
2. Copy these files into a parent folder
3. Make one file containing all the information in the files

Prepend the names of the files in the folder to the filenames

1. make variable for directories DIRS="AA BB CC"
2. make directories for i in $DIRS; do mkdir ${i}; done
3. make test file in the directory for i in $DIRS; do touch ${i}/test1; done
4. get the variable with directories names as if I did not know it DIRS2= ls (without the space)
   do not know how to get only directories here, not files
5. write a text in each filefor i in $DIRS2; do echo ${i} > ${i}/test1; done
6. pre-pend the directory name and copy to parent folderfor i in $DIRS2; do cp ${i}/test1 ./${i}_test1
7. summarize everything into one file: touch summary.txt; cat * > summary.txt
   throws and error as there are directories involved in the expansion but gives the summary anyway
   would be great to be able to include the names of the files it comes from as a first line in the file

BASH history

link
another useful link on the bash history and rerunning commands stuff here

To see history

history

Add timestamps

export HISTTIMEFORMAT='%F %T '

Using for loop to manipulate the same file in multiple directories in working folder

2022-08-10

Motivation

Append the same line to an INFO.txt file which is in directories called CMDX_[and here it is a mess]

How to do it

DIRS=ls | grep CMDX
for d in $DIRS; do echo "Same line" >> "${d}INFO.txt"; done

To figure out

1. is "${d}INFO.txt" safe or is "$"{d}"INFO.txt" better?
   • "$"{d}"INFO.txt" does not work
2. when using with touch to make .txt files, why does it says that it does not have rights, but does the job anyway?
3. Why is the line not appended, but added to the end in those INFO.txt files? Worked on the new ones I created.