Topic: Linux

Oct 13, 2020

Linux Text Processing

  • awk:
    • Remove some columns: awk '{$6=$8=""; print $0}' file
  • cut:
    • Display first x characters: cut -c 1-35 <file>
    • Cut the first x characters: cut -c 1-35 --completement <file>
  • grep:
    • Treat binary file as text file: grep -a data fsstatus.log
  • sed:
    • sed on a string (not a file) using a here string: sed "s/,/','/g" <<< "A,B,C"
    • Extract last section from multiple top outputs: COLUMNS=1024 top -b -d 1 -n 2 -c | sed -n 'H;/^top/h;\${g;p;}'
  • sort
    • Sort numeric column 3 in CSV file: sort -t ',' -n -k 3,3 data.csv

Oct 13, 2020

Linux Sysadmin

  • alternatives command (e.g. to set to a specific default Java version)
  • User accounts:
    • Add user to sudoers
      One method is to add the user to wheel group: # usermod -aG wheel username. Now log in again as username and run groups command.
    • Create group: sudo groupadd ansible
    • Create user (also add user to sudoers via wheel group membership):
      sudo useradd -g ansible -G wheel ansible
    • Set password: passwd ansible
    • Remove prompt for password when using sudo (not most secure, but very handy for using ansible etc):
      Run visudo
      Comment-out and comment these lines, to leave them like this: 
      ## Allows people in group wheel to run all commands
#%wheel ALL=(ALL)       ALL
## Same thing without a password
%wheel  ALL=(ALL)       NOPASSWD: ALL

      To test, running sudo id should return root

  • Cron
  • ps
    • Seeing all of the command-line: ps auxww | grep java
  • System-wide LD_LIBRARY_PATH update:
    • LD_LIBRARY_PATH - how to update for a system service: Add the directory to /etc/ld.so.conf or a new file in /etc/ld.so.conf.d/, depending on distro. After that, you must run (at least on Redhat) ldconfig as root.
    • See also https://developer.ibm.com/technologies/linux/tutorials/l-lpic1-102-3/.
  • System-wide PATH update:
    • See files in /etc/profile.d
  • rpm:
    • List installed packages: rpm -qa
    • List files in an uninstalled rpm: rpm -qpl ./BaseOS/Packages/yum-utils-4.0.8-3.el8.noarch.rpm
    • List files in an installed rpm: rpm -ql postgresql12-server.x86_64 | grep conf
    • Idenitfy the package a file came from: rpm -qf /usr/bin/svn
    • Install an rpm, and automatically install any dependencies (via yum): yum --nogpgcheck localinstall VirtualBox-6.1-6.1.12_139181_el8-1.x86_64.rpm
  • systemd:
    • Debug service startup faiure: journalctl -xe -u redis
    • List all services: systemctl list-unit-files | grep -i postgres or sudo systemctl list-units --type=service
    • Example config file path: /usr/lib/systemd/system/MFSafeNet.service
  • top:
    • top -c and 200% cpu! (type shift+i to see shared per-cpu core usage): See here
    • Filter top output by process name: asroot top -c -p $(pgrep -d',' -f cascd) (see here
  • yum:
    • Yum cheatsheet
    • List files in an uninstalled package:
      yum repoquery --list postgresql12-contrib.x86_64
    • List all available versions of each package, rather than the most recent version:
      yum --showduplicates --disablerepo="*" --enablerepo="pgdg10" list available
    • Install a package version that is not the most recent:
      yum --showduplicates --disablerepo="*" --enablerepo="pgdg10" install postgresql10-server-10.12
      Another example:
      yum --showduplicates --disablerepo="*" --enablerepo="pgdg10" install postgresql10-odbc-10.03.0000-1PGDG.rhel7
      Note: Take the package base name (e.g. “postgresql10-server”) and then add the version (e.g. “10.12”). See also here.

Oct 13, 2020

Linux Screen

  • Basic usage example: 
    screen -S my-session bash
ctrl+a d
screen -ls
screen -r my-session
  • How to use Linux Screen
  • Manual (gnu.org)
  • Name session: screen -S session_name
  • Name session using a screenfile: screen -S sentinel-demo -c ~/scripts/screenfile-sentinel-demo
  • Rename the screen session (the name listed by “screen -ls”): Ctrl+a then type :sessionname MyNewSessionName
  • Re-name a screen window’s title: Hit Enter to open that window, and then ctl+a + shift+a
  • Detach: Ctrl+a then d
  • List sessions: screen -ls
  • Re-attach to session: screen -r <name> or <pid>

Oct 13, 2020

Red Hat Linux

Oct 13, 2020

Linux Processes

  • List threads:
    ps -p <pid> -L
  • environment variables in a process:
    [rhel7tpcc-hub]cat /proc/29640/environ | tr '\0' '\n' | grep COBPATH
    COBPATH=/home/hub/pkg_260324_es/lib/es
  • pstree:
    • See child processes: pstree -pa (see also /proc/<parent-pid>/task/)
  • Redirecting to stdout and stderr:
    • script:
      echo "hello world stdout"
      echo "hello world stderr" >&2
      echo "hello world tty" >/dev/tty
  • shared memory:
  • signals:
    • https://www-uxsup.csx.cam.ac.uk/courses/moved.Building/signals.pdf
    • SIGABRT (6): The program called the abort() function. This is an emergency stop.
    • SIGSEGV (11): An attempt was made to access memory not allocated to the process. This is often caused by reading off the end of arrays etc.
  • strace:
    • Follow child processes (-ff), capture child process output in separate files (-o), and don’t truncate the strings (-v -s 1024):
      strace -v -s 1024 -o test -ff casstart /rTPCCDBFH /uSYSAD /pSYSAD
      See also: https://www.thegeekstuff.com/2011/11/strace-examples/
  • sysctl (configure kernel parameters at runtime):
    • Updating /etc/sysctl.conf:
      1. Use sysctl -a to get the full name.
      2. Add the new value to /etc/sysctl.d/99-custom.conf (Note: On my system, this is a symbolic link to /etc/sysctl.conf).
      3. Reboot (alternatively, to avoid the reboot for now, run: sysctl -p /etc/sysctl.d/99-custom.conf).
  • UID/GID of running process:
    cat /proc/<PID>/status (the first two values are real and then effective id)
    Uid: 0 0 0 0
    Gid: 1 1 1 1
  • ulimit:
    • Soft and Hard limits:
      # ulimit -u -S
      # ulimit -u -H
    • Nice examples (from Oracle install)
    • General procedure: edit /etc/security/limits.conf (and/or files in /etc/security/limits.d) and then run sysctl -p
    • Names of common items to change
    • Check used network socket counts using lsof:
      [nwb-tpccrh76es2-hub]asroot lsof -a -U -uhub |wc -l
      17
      [nwb-tpccrh76es2-hub]for i in {1..1000}; do curl localhost:8222/esmac/casrdo00 >nul 2>&1; done
      [nwb-tpccrh76es2-hub]asroot lsof -a -U -uhub |wc -l
  • gdb:
    • Attach: gdb -p <pid>
    • List threads: (gdb) info threads
    • Switch to thread: (gdb) thread <num>
    • Show strack trac: (gdb) backtrace

Oct 13, 2020

Linux Performance

  • iftop: Displays per-network-connection traffic every 2,10,40 seconds:
    iftop -n -N -P -i ens192 -f 'port 9223' -t -L 15
    (remove “-t” to see curses-style ui, “-L” is the number of results to display, “-P” is to show ports.)
    Note: To see curses output property in putty, go to “Window > Translation” and tick “Enable VT100 line drawing even in UTF-8 mode”.
  • Load averages (e.g. top, uptime)
  • pidstat:
    • Installation: yum install sysstat
    • thegeekstuff.com
    • Example: 5 samples, one second between each, then give me an average:
      pidstat -C cassi -l 1 5
      Average: UID PID %usr %system %guest %wait %CPU CPU Command
      Average: 1001 27616 0.80 1.39 0.00 0.00 2.19 - pidstat -C cassi -l 1 5
      Average: 1001 31248 6.57 3.98 0.00 0.00 10.56 - cassi32 /rTPCCDBFH -rTPCCDBFH
      Average: 1001 31254 6.37 3.19 0.00 0.00 9.56 - cassi32 /rTPCCDBFH -rTPCCDBFH
      Average: 1001 31260 3.78 1.79 0.00 0.00 5.58 - cassi32 /rTPCCDBFH -rTPCCDBFH
      Average: 1001 31264 6.18 2.99 0.00 0.00 9.16 - cassi32 /rTPCCDBFH -rTPCCDBFH
      Average: 1001 31266 12.75 7.37 0.00 0.00 20.12 - cassi32 /rTPCCDBFH -rTPCCDBFH
  • top:
    • “top” in batch mode (i.e. can be redirected to a file) and include command-line args: top -b -n 1 -c

Oct 13, 2020

Linux Filesystem

  • find command:
    • Search for multiple file names: find . -type f \( -name "*.png" -o -name "*.jpg" -o -name "*.deb" -o -name ".pdf" \)
    • Follow symlinks: find -L $COBDIR -iname ...
    • find with compex exec: find . -iname console.log.tr4 -exec sh -c "echo {} && grep 'CASSI5030I PLTPI Phase 2 List(T1) Processing Completed' {} | wc -l" \;
  • List symlinks: find -type l
  • Mount cdrom using /etc/fstab: /home/hub/Downloads/rhel-8.1-x86_64-dvd.iso /media/cdrom/ iso9660 loop,ro 0 0
  • List symbols of binary: 
    # nm -D /usr/pgsql-10/lib/libpq.so.5.10 | grep PQconninfo
0000000000010150 T PQconninfo
0000000000010220 T PQconninfoFree
00000000000118b0 T PQconninfoParse
  • rsync
    • Exclude directory: rsync -avz --exclude VirtualBox_VMs /home/jcdc . (See here)
  • Determine file creation time (see also here)
    • Method 1: “stat filename” - But even on my ext4 filesystem (RH Linux) the “Birth” value was unfortunately blank (apparently, it is down to the kernel if the “Birth” value is populated): 
      [wibble]stat myfile.txt
File: '/home/fred/myfile.txt'
Size: 329066          Blocks: 648        IO Block: 4096   regular file
Device: 802h/2050d      Inode: 2229605     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/     hub)   Gid: ( 1000/     hub)
Access: 2023-09-20 09:58:38.000000000 +0100
Modify: 2023-09-20 09:58:38.000000000 +0100
Change: 2023-09-20 10:06:05.316755558 +0100
 Birth: -
    • Method 2: “debugfs” (Note this must be run as root) - looks for the “crtime” value (not “ctime”!): 
      [wibble]asroot debugfs -R 'stat myfile.txt' /dev/sda2
debugfs 1.42.9 (28-Dec-2013)
Inode: 2229605   Type: regular    Mode:  0644   Flags: 0x80000
Generation: 2201300265    Version: 0x00000000:00000001
User:  1000   Group:  1000   Size: 329066
File ACL: 0    Directory ACL: 0
Links: 1   Blockcount: 648
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x650ab5fd:4b853998 -- Wed Sep 20 10:06:05 2023
 atime: 0x650ab43e:00000000 -- Wed Sep 20 09:58:38 2023
 mtime: 0x650ab43e:00000000 -- Wed Sep 20 09:58:38 2023
crtime: 0x650ab5fd:4b0b269c -- Wed Sep 20 10:06:05 2023

Oct 13, 2020

Bash

  • Extract “hel” character range from variable:
    s1="hello world"
    echo ${s1:0:3}
  • Built-in variables (including the current function name)
  • Sort an array in bash
  • Use process redirection (to log output to multiple places):
    # Process substitution is not a POSIX compliant feature and so it may have to be enabled
    # See https://www.linuxjournal.com/content/shell-process-redirection
    set +o posix
    exec > >(tee /tmp/test.log | logger -t user-data -s) 2>&1
    echo "Send to stderr. This will be logged to a file, syslog, and the screen $(date)" >&2
    echo "Send to stdout. This will be logged to a file, syslog, and the screen $(date)"
  • Bash arrays:
    #!/usr/bin/bash
    declare -a HOSTS_ARRAY=('nwb-tpccrh76es1' 'nwb-tpccrh76es2' 'nwb-tpccrh76es3' 'nwb-mlora1' 'nwb-mlora2' 'nwb-mlora3');
    declare -a FILES_ARRAY=('lib/mfescache.jar');
    function update_files()
    {
    ` local _host _file `
    ` for _host in “${HOSTS_ARRAY[@]}” ; do for _file in “${FILES_ARRAY[@]}” ; do echo “scp -p $_file hub@${_host}:/home1/hub/pkg-es/$_file” scp -p $_file hub@${_host}:/home1/hub/pkg-es/$_file done done } update_files`
  • “set” command:
    #!/bin/ksh
    echo "Input args: $@"
    echo "Args count: $#"
    echo "Adding another arg..."
    set -- $@ wibble
    echo "New input args: $@"
    echo "New args count: $#"