[Bin] Fix issue downloading videos with ytdl

When downloading a whole channel, ytdl used to download everything
including playlists displayed on the user’s main page. Now, a check
is performed in order to verify if the URL passed is a channel, and
if it is only download the videos listed on their uploads page.

2021-01-06 16:10:39 +01:00

65 KiB

Raw Blame History

Executable scripts

Presentation
Autostart
cli utilities
Emacsmail
Media
- mp42webm
- youtube-dl wrappers
  - ytplay
  - ytdl - a youtube-dl wrapper
Plock
Rofi utilities
Wallpaper utilities
Weather

Presentation

This file will present all the executable scripts I wrote. It is also their original source code, all the following code snippets are exported and tangled from this file to the actual executables.

Autostart

Because I sometimes switch from window manager to window manager, creating a script that handles by itself autostarting things for me is way easier than rewriting every time the autostart part of my configuration. As you can every instance will be launched asynchronously, and only if there is no other instance of said command running.

set-screens is a custom script declared below.

Command	Arguments	Run once?
`mpc`	`stop`	no
`set-screens`		no
`xwallpaper`	`--zoom (cat $HOME/.cache/wallpaper)`	no
`picom`	`--experimental-backends`	yes
`pumopm`		yes
`xfce-polkit`		yes
`xss-lock`	`plock`	yes
`xidlehook`	`--not-when-audio --not-when-fullscreen --timer 3600 lock ''`	yes
`nm-applet`		yes
`numlockx`	`on`	yes

  (mapconcat (lambda (start-command)
               (let* ((command   (s-replace "~" "" (nth 0 start-command)))
                      (arguments (s-replace "~" "" (nth 1 start-command)))
                      (once?     (string= "yes"    (nth 2 start-command))))
                 (if once?
                     (format "pgrep -x %s 2&>/dev/null || echo (%s) 2&>/dev/null"
                             command
                             (s-collapse-whitespace (format "%s %s & && disown"
                                                            command
                                                            arguments)))
                   (format "%s %s &" command arguments))))
             table
             "\n")

set-screens  &
pgrep -x pumopm 2&>/dev/null || echo (pumopm & && disown) 2&>/dev/null
pgrep -x xfce-polkit 2&>/dev/null || echo (xfce-polkit & && disown) 2&>/dev/null
pgrep -x xss-lock 2&>/dev/null || echo (xss-lock plock & && disown) 2&>/dev/null
pgrep -x picom 2&>/dev/null || echo (picom --experimental-backends & && disown) 2&>/dev/null
pgrep -x xidlehook 2&>/dev/null || echo (xidlehook --not-when-audio --not-when-fullscreen --timer 3600 lock '' & && disown) 2&>/dev/null
pgrep -x nm-applet 2&>/dev/null || echo (nm-applet & && disown) 2&>/dev/null
pgrep -x numlockx 2&>/dev/null || echo (numlockx on & && disown) 2&>/dev/null
mpc stop &
xwallpaper --zoom (cat $HOME/.cache/wallpaper) &

  set -l PATH $PATH /usr/lib/xfce-polkit

  <<autostart-gen()>>

cli utilities

Backup

backup is a very simple, oneliner script that will create a local copy of a file and add the date at which it was copied in the filename. You can see its source code here:

  cp -r $argv[1] $argv[1].bak.(date +"%Y%m%d%H%M%S")

Development

Cnew

cnew is a small utility script similar to but simpler than cppnew that creates a CMake template C project from the template that already exists in ~/dev/templateC. If no argument was passed, display an error message and exit.

  if ! count $argv > /dev/null
      echo "Missing argument: PROJECT" && return -1
  end

Pass the first argument to a switch statement.

switch "$argv[1]"

If the argument is -h or --help, then display the help message and exit the script normally.

  case -h --help
      man ~/dev/fishfunctions/cnew.man
      exit 0

Else, the argument is the name of the project the user wants to create.

  case '*'
      set -g project_name $argv[1]

Let’s close the switch statement.

end

Now, let’s copy the template where the user is executing cnew from, give it the name of the project and move to the project.

  cp -r ~/dev/templateC $argv[1]
  cd $argv[1]

The default files have a placeholder for the name of the project. Let’s replace these placeholders with the project’s name.

  sed -i "s/PROJECTNAME/$argv[1]/g" CMakeLists.txt
  sed -i "s/PROJECTNAME/$argv[1]/g" README.org
  sed -i "s/CPROJECTNAME/$argv[1]/g" doc/Doxyfile

Now, let’s create a git repository and initialize it.

  git init
  git add .
  git commit -m "initial commit"

And we’re done!

Dart Language Server

Spacemacs' recommendations on how to use Dart with LSP is outdated, since dart_language_server is obsolete. As recommended by the repo owner, we should launch instead the following code:

  /usr/bin/dart $DART_SDK/snapshots/analysis_server.dart.snapshot --lsp

So, instead of using the obsolete executable, instead we will be calling the analysis server as requested.

UpdateFlutter

This is a simple utility to be ran when the flutter package is updated.

  sudo chown -R :flutterusers /opt/flutter
  sudo chmod -R g+w /opt/flutter
  sudo chmod a+rw /opt/flutter/version
  sudo chown $USER:(id -g $USER) /opt/flutter/bin/cache

set-screens

set-screens is a small script that allows the user to automatically set up an external monitor. First, let’s set some variables so we don’t have to type in hidden places some values that should be easily modifiable.

  set internal "eDP1"
  set external "HDMI1"

Now, let’s set the DETECTEDSCREEN variable with a simple grep. If the variable turns out to be empty, this means the display was not detected. However, if it’s not, then it will be an array with its second value that holds the maximum resolution the display can handle. It needs to be passed through awk in order to get only the resolution itself and not the refresh rate, but once we’ve got that, we can set our external monitor as the main monitor with its maximum resolution. i3 is also restarted in order to properly display the wallpaper and Polybar on the new screen.

  set externaldisplay (xrandr -q --current | grep -A 1 -i "$external connected")
  if test -n "$externaldisplay"
      set resolution (echo $externaldisplay[2] | awk '{$1=$1;print $1}')
      xrandr --output "$external" --primary --auto --mode "$resolution" --right-of "$internal"
  end

sshbind

Something that I did not know for quite some time but that is actually crazy useful about SSH is its ability to bind locally the port of a remote machine, and vice versa. The syntax is actually very simple, but I prefer a more intuitive way of writing it. Its usage is sshbind PORT FROMHOST TOHOST.

  ssh -L $argv[1]:$argv[3]:$argv[1] $argv[2] -N

Starwars

This is a one-liner that allows you to watch Star Wars episode 4 in ASCII art in your terminal. Here is the code:

  telnet towel.blinkenlights.nl

Toggle touchpad tapping

For some reasons, my firmware does not recognize the function key for toggling the touchpad. I’m not going to really complain about it since it lets me program it like I want. Since I often don’t need to completely deactivate the touchpad, I’ll instead toggle whether tapping is enabled or not when pressing XF86TouchpadToggle. And for that, I need this small script that will actually toggle it, and it will be used in my window manager configuration.

First let’s declare some variables to make this script more personal. With my current computer (a Gazelle by System76), the name of my touchpad is the following:

  set TPNAME "ELAN0412:00 04F3:3162 Touchpad"

Let’s now get the identifier of the touchpad for xinput:

  set TPID (xinput list | grep $TPNAME | awk '{print $6}' | sed 's|id=\(.*\)|\1|g')

Now, let’s detect the current status of the touchpad:

  set TPSTATUS (xinput list-props $TPID | grep "Tapping Enabled" | \
                grep -v "Default" | awk '{print $5}')

This will set TPSTATUS either to 0, meaning tapping is disabled, or to 1, meaning it’s enabled. I will consider any other value as being disabled.

  test [[ $TPSTATUS = "1" ]] && set NEWTPSTATUS 0 || set NEWTPSTATUS 1

Finally, let’s update the touchpad’s options:

  xinput set-prop $TPNAME "libinput Tapping Enabled" $NEWTPSTATUS

Wacom setup

I made a small and quick utility to set up my Wacom tablet so it is only bound to one screen.

Set our variables

Let’s first declare our function that will be called to set our variables.

function set_device

We need some variables in order to correctly set our tablet. First, let’s get declare what the name of our tablet is, and what the name of its touchpad is.

  set -g DEVICE "Wacom USB Bamboo PAD Pen stylus"
  set -g DEVICETOUCH "Wacom USB Bamboo PAD Finger touch"

We will also modify two settings: the speed of the cursor on the touchpad, and the scroll speed. Let’s declare the name of these two settings.

  set -g WACOMPROPTOUCHSPEED "Device Accel Velocity Scaling"
  set -g WACOMPROPSCROLLPSEED "ScrollDistance"

To get the correct values for the area it can cover, we’ll need to reset our tablet.

xsetwacom set "$DEVICE" ResetArea

Now we can get the X and Y areas.

  set -l AREATOT (xsetwacom get "$DEVICE" Area)
  set -g AREAX (echo $AREATOT | awk '{print $3}')
  set -g AREAY (echo $AREATOT | awk '{print $4}')

Now let’s close our function.

end

Select our screen

This function will allow us to select the screen on which the tablet will be active. We can also select the option “desktop” so that all screens are selected. Let’s declare our function.

function set_screen

First, let’s set what screens are available, including the desktop option.

  set CONNECTED_DISPLAYS (xrandr -q --current | \
  sed -n 's/^\([^ ]\+\) connected .*/\1/p') desktop

Now, let’s select the one we wish to use using rofi.

  set -g SCREEN (for d in $CONNECTED_DISPLAYS
      echo $d
  end | rofi -dmenu -i -p "Select your dispaly" | tr -d '\n')

Now, let’s get the resolution of our selected screen.

  set -l LINE (xrandr -q --current | if [ "$SCREEN" = "desktop" ]
      sed -n 's/^Screen 0:.*, current \([0-9]\+\) x \([0-9]\+\),.*/\1 \2/p'
  else
      sed -n "s/^$SCREEN"' connected \(primary \)\{0,1\}\([0-9]\+\)x\([0-9]\+\)+.*/\2 \3/p'
  end)

From that, let’s get the vertical and horizontal resolution of our screen.

echo $LINE | read -g WIDTH HEIGHT

If any of our WIDTH ou HEIGHT it empty, we’ll have to abort the script.

  if test -z $WIDTH || test -z $HEIGHT
      exit 1
  end

Let’s close our function now.

end

Adjust the tablet

This function will take care of adjusting our tablet to our screen. Let’s declare our function.

function adjust_device

If our screen is too high or too wide for our tablet, we will have to adjust the height or width of the area used by the tablet. So let’s get the theoretical new height and width of the area.

  set RATIOAREAY (math ceil \($AREAX \* $HEIGHT \/ $WIDTH\))
  set RATIOAREAX (math ceil \($AREAY \* $WIDTH \/ $HEIGHT\))

Now, if the current height of the tablet’s area is greater than the theoretical new area, it means the current area is too high. Otherwise, it should be the other way around. Let’s set NEWAREAX and NEWAREAY that will be used to set the new area for the tablet.

  if test $AREAY -gt $RATIOAREAY
      set -g NEWAREAX $AREAX
      set -g NEWAREAY $RATIOAREAY
  else
      set -g NEWAREAX $RATIOAREAX
      set -g NEWAREAY $AREAY
  end

Alright, now let’s set the new area with these new variables.

  xsetwacom set "$DEVICE" Area 0 0 $NEWAREAX $NEWAREAY
  xsetwacom set "$DEVICE" MapToOutput "$SCREEN"

Let’s slow down the cursor’s speed on the touchpad.

xinput set-float-prop $DEVICETOUCH $WACOMPROPTOUCHSPEED 0.5

Let’s also slow down the scroll speed of the touchpad.

  xsetwacom set $DEVICETOUCH $WACOMPROPSCROLLPSEED "90"

Now, let’s close the function.

end

Lauch the functions

Back to the main body of the script, we can now launch the functions sequencially.

  set_device
  set_screen
  adjust_device

Emacsmail

This short script is used in my ~/.local/share/applications/mu4e.desktop file in order to send to Emacs any mailto: requests made in my system.

  emacsclient -c --eval "(browse-url-mail \"$@\")"

Media

mp42webm

This function allows me to convert easily an mp4 video to the webm format. Nothing too fancy here.

  ffmpeg -i $argv[1] -c:v libvpx -crf 10 -b:v 1M -c:a libvorbis $argv[1].webm

youtube-dl wrappers

ytplay

ytplay is a simple script I’ve written that allows me to play in mpv any YouTube video at the desired resolution. The script relies on dmenu (or rofi in dmenu-mode), youtube-dl and of course mpv itself.

  set URL (rofi -dmenu -i -p "Video URL")
  if test -n "$URL"
      set FORMAT                              \
          (youtube-dl --list-formats "$URL" | \
          egrep "webm.*[0-9]+x[0-9]+"       | \
          awk '{print $3 " " $1}'           | \
          sort -gu                          | \
          rofi -dmenu -i -p "Resolution"    | \
          string split " ")
      set FCODE $FORMAT[2]
      mpv --ytdl-format=$FCODE+bestaudio/best "$URL"
  end

I’ll even add a .desktop entry for this script:

  [Desktop Entry]
  Type=Application
  Version=1.0
  Name=ytplay (YouTube in mpv)
  Comment=Play YouTube videos in mpv
  Exec=/home/phundrak/.local/bin/ytplay
  Path=/home/phundrak/.local/bin
  Terminal=false
  Categories=Media

ytdl - a `youtube-dl` wrapper

This script is a wrapper around youtube-dl which I use mainly for archiving YouTube videos on my NAS (at the time I’m writing this, I have already 2.1TB worth of videos archived). The principle behind this script is quite simple: I want to avoid as much as possible to redownload any video already downloaded in order to avoid pinging too much YouTube’s servers, 429 Too Many Requests errors are really annoying, and it comes really early when you have only a couple of new videos to download among the few 14k videos already downloaded.

Be aware this script was written for the Fish shell (3.1.0 and above), and makes use of youtube-dl 2020.03.24 and above, Fish getopts and ripgrep.

Setting default values

Some variables in this script will have default values, we do not want to have a mile-long command each time we wish to download a single video. We’ll also set some global variables that won’t change:

Variable Name	Default Value	String?
YTDL_SHARED_DIR	$HOME/.local/share/ytdl	no
FORMAT_DEFAULT	%(uploader)s/%(upload_date)s - %(title)s.%(ext)s	yes
DOWNFILE_DEFAULT	$YTDL_SHARED_DIR/downloaded	no
ERRFILE_DEFAULT	$YTDL_SHARED_DIR/video-errors	no
LOGFILE_DEFAULT	$YTDL_SHARED_DIR/ytdl.log	no
PREFFERED_FORMAT	bestvideo[ext=mp4]+bestaudio[ext=m4a]/bestvideo+bestaudio	yes
VERSION	0.3	yes

There is one more default variable pointing to ytdl’s root directory which depends on whether the videos directory has a French or English name:

  if test -d "$HOME/Vidéos"
      set -g ROOTDIR_DEFAULT "$HOME/Vidéos" # French name
  else
      set -g ROOTDIR_DEFAULT "$HOME/Videos" # English name
  end

  (mapconcat (lambda (var)
               (let ((varname  (car  var))
                     (varvalue (cadr var))
                     (string?  (string= (nth 2 var) "yes")))
                 (format "set -g %-16s %s" varname (if string? (format "\"%s\"" varvalue)
                                                   varvalue))))
             vars
             "\n")

set -g YTDL_SHARED_DIR  $HOME/.local/share/ytdl
set -g FORMAT_DEFAULT   "%(uploader)s/%(upload_date)s - %(title)s.%(ext)s"
set -g DOWNFILE_DEFAULT $YTDL_SHARED_DIR/downloaded
set -g ERRFILE_DEFAULT  $YTDL_SHARED_DIR/video-errors
set -g LOGFILE_DEFAULT  $YTDL_SHARED_DIR/ytdl.log
set -g PREFFERED_FORMAT "bestvideo[ext=mp4]+bestaudio[ext=m4a]/bestvideo+bestaudio"
set -g VERSION          "0.3"

  <<ytdl-default-vars-make()>>
  <<ytdl-default-vars-root>>

We’ll also create the directory pointed at by YTDL_SHARED_DIR if it doesn’t exist already:

  mkdir -p $YTDL_SHARED_DIR

Help message

The next step is displaying the help message for the script. For that, just a long string echo’d will do, wrapped in the function _ytdl_help.

  function _ytdl_help
      echo "Usage: ytdl [OPTION]... URL [URL]...

      -4, --ipv4
          Download with forced IPv4
          Default: no

      -6, --ipv6
          Download with forced IPv6
          Default: no

      -a, --batch-file <file>
          File containing URLs to download, one URL per line. Lines starting with
          '#', ';' or ']' are considered as comments and ignored.
          Default: None

      -c, --id-cache <file>
          File containing the video IDs that were already downloaded, one ID per
          line.
          Default: $DOWNFILE_DEFAULT

      -d, --directory <dir>
          Root directory in which to download videos.
          Default: $ROOTDIR_DEFAULT

      -e, --error-file <file>
          File containing the IDs of videos that failed to download, one ID per
          line
          Default: $ERRFILE_DEFAULT

      -f, --format <format>
          Format name for downloaded videos, including path relative to root
          directory
          Default: $FORMAT_DEFAULT

      -l, --logs <file>
          File in which to store logs.
          Default: $LOGFILE_DEFAULT

      -V, --verbose
          Show verbose output
          Default: no

      -v, --version
          Show version of ytdl.

      -h, --help
          Shows this help message"
  end

We also have the function _ytdl_version to display the current version of ytdl:

  function _ytdl_version
      echo "ytdl 0.3, developped for fish 3.1.0 and youtube-dl 2020.03.24 or newer"
      echo "requires Fish getopts <https://github.com/jorgebucaran/fish-getopts>"
      echo "and ripgrep <https://github.com/BurntSushi/ripgrep>"
  end

Arguments Handling

The function _ytdl_parse_ops is a little bit trickier: we use getopts to parse the arguments passed to the script in order to get some preferences from the user. Here is a quick reference on what options are available and what they do:

Short	Long	Takes a value?	Associated Variable	Default Value	What it does
4	ipv4	no	IPV4	None	Force IPv4
6	ipv6	no	IPV6	None	Force IPv6
a	batch-file	yes	FILE	None	Batch file
c	cache	yes	DOWNFILE	$DOWNFILE_DEFAULT	Cache file
d	directory	yes	ROOTDIR	$ROOTDIR_DEFAULT	Root directory
e	error-file	yes	ERRFILE	$ERRFILE_DEFAULT	Error logs
f	format	yes	FORMAT	$FORMAT_DEFAULT	Filename format
l	logs	yes	LOGFILE	$LOGFILE_DEFAULT	Logs
V	verbose	no	VERBOSE	1	Verbose output
v	version	command	None	None	Script version
h	help	command	None	None	Display this message

We can also pass individual YouTube URLs without any options or switches associated to them, they will be downloaded as part of a single queue.

  (mapconcat (lambda (arg)
               (let* ((short (format "%s" (nth 0 arg)))
                      (long  (nth 1 arg))
                      (arg?  (string= "yes" (nth 2 arg)))
                      (var   (unless (string= "None" (nth 3 arg))
                               (nth 3 arg))))
                 (format "case %s %s\n\t%s"
                         short long
                         (if var (format "set -g %s %s" var
                                         (if arg? "$value" ""))
                           (format "_ytdl_%s && exit"
                                   (if (string= "h" short) "help" "version"))))))
             args
             "\n")

case 4 ipv4
	set -g IPV4
case 6 ipv6
	set -g IPV6
case a batch-file
	set -g FILE $value
case c cache
	set -g DOWNFILE $value
case d directory
	set -g ROOTDIR $value
case e error-file
	set -g ERRFILE $value
case f format
	set -g FORMAT $value
case l logs
	set -g LOGFILE $value
case V verbose
	set -g VERBOSE
case v version
	_ytdl_version && exit
case h help
	_ytdl_help && exit

The following shows how getopts is used to catch the options and switches passed to the script:

  getopts $argv | while read -l key value
      switch $key
          <<ytdl-arg-handling-gen()>>
          case _
              for v in $value
                  set -g VIDEOS $VIDEOS $v
              end
      end
  end

  (let* ((args (-filter (lambda (arg)
                          (let* ((var     (unless (string= "None" (nth 3 arg)) (nth 3 arg)))
                                 (default (format "%s" (nth 4 arg)))
                                 (default (unless (string= "None" default) default)))
                            (and var default)))
                        args)))
    (mapconcat (lambda (arg)
                 (let* ((var     (nth 3 arg))
                        (default (format "%s" (nth 4 arg))))
                   (format "if set -q $%s\n\tset -g %s %s\nend"
                           var var default)))
               args
               "\n"))

if set -q $DOWNFILE
	set -g DOWNFILE $DOWNFILE_DEFAULT
end
if set -q $ROOTDIR
	set -g ROOTDIR $ROOTDIR_DEFAULT
end
if set -q $ERRFILE
	set -g ERRFILE $ERRFILE_DEFAULT
end
if set -q $FORMAT
	set -g FORMAT $FORMAT_DEFAULT
end
if set -q $LOGFILE
	set -g LOGFILE $LOGFILE_DEFAULT
end
if set -q $VERBOSE
	set -g VERBOSE 1
end

Some values need to be set to their default, so let’s assign them their value if no user value was passed:

  <<ytdl-arg-set-default-value-gen()>>
  set -g FORMAT "$ROOTDIR/$FORMAT"

Both these code blocks are executed in _ytdl_parse_ops:

  function _ytdl_parse_ops
      <<ytdl-getopts>>
      <<ytdl-arg-set-default-value>>
  end

Logging

_ytdl_log is a very simple function used for logging information for the user in the file pointed to by LOGFILE. The first argument the function should receive is its log level. I generally use either "INFO" or "ERR". The second argument is the message to log.

  function _ytdl_log
      set -l INFOLEVEL $argv[1]
      set -l MSG $argv[2]
      set -l LOG (printf "[%s] %s %s\n" $INFOLEVEL (date +"%F %T") $MSG)
      printf "%s\n" $LOG >> $LOGFILE
      if test $VERBOSE -eq 1
          echo $LOG
      end
  end

Download a Single Video

In order to download a single video, a simple function has been written for this that will display when downloaded how far it is down the list of videos to be downloaded and it will add its ID to the file listing all videos downloaded. The script will also try to download the video according to the PREFFERED_FORMAT variable, but if the download fails it will download the default format selected by youtube-dl. If both downloads fail, the ID of the video will be added to the list of failed videos. If one of the downloads succeeds, it will remove the ID from the list of failed downloads.

The first argument of the function is the video ID from YouTube, the second argument is the position of the video in the queue, and the third argument is the queue length –can be the amount of videos in a whole YouTube channel, the amount of videos in a playlist, or simply the amount of YouTube URLs passed as arguments to the script.

  function _ytdl_download_video
      set ID $argv[1]
      _ytdl_log "INFO" (printf "Downloading video with ID $ID (%4d/%4d)" $argv[2] $argv[3])
      if youtube-dl -f $PREFFERED_FORMAT -ciw -o $FORMAT "https://youtube.com/watch?v=$ID"
          echo $ID >> $DOWNFILE
      else if youtube-dl -ciw -o $FORMAT "https://youtube.com/watch?v=$ID"
          echo $ID >> $DOWNFILE
      else
          _ytdl_log "ERR" "Could not download $VIDEO"
          echo $ID >> $ERRFILE
      end
  end

Note that this function is not meant to be called without any checks before. It is meant to be called by _ytdl_download_queue described below.

Download a Queue of Videos

One of the main goals of this tool is to check if a video has already been downloaded. This is why, as you will see below, we use ripgrep to check if the ID of the video we want to download is already present in the list of downloaded videos. If not, it will then be downloaded though _ytdl_download_video described above.

  function _ytdl_download_queue
      for i in (seq (count $argv))
          rg -- $argv[$i] $DOWNFILE 2&> /dev/null
          if test $status -ne 0
              _ytdl_download_video $argv[$i] $i (count $argv)
          end
      end
  end

Download Videos From Arguments

The main aim of this function is to transform the URLs contained in the arguments passed to the script to a list of IDs usable later on by ytdl.

  function _ytdl_download_arg_urls
      set -g IDs
      for VIDEO in $argv
          _ytdl_log "Info" "Getting video ID for $VIDEO"
          set -g IDs $IDs (youtube-dl --get-id $VIDEO)
      end
      _ytdl_download_queue $IDs
  end

Download Videos From a Batch File

The final function to declare before the main body of the script is _ytdl_download_batch: it will look for each line, ignoring the ones beginning by #, ; and ] (just like youtube-dl) and will download them, assuming these are channel URLs or playlist URLs, however it should also work with direct video URLs.

What this function does is for each line, it will fetch the entierty of the video IDs found in a playlist or channel. Then, it will look each ID up the list of already downloaded videos and will add all new IDs to a queue of videos to be downloaded. It will then pass each new video ID to _ytdl_download_video directly. Beware that if you pass directly the URL of the channel, such as https://www.youtube.com/user/enyay if you want to download Tom Scott’s videos, it will download everything on the main page of their channel, which means it will even download videos from playlists they decided to put on their channel’s front page, even if it is not theirs. So in that case, we need to append /videos to any channel URL.

  function _ytdl_download_batch
      set -q $FILE
      if test $status -eq 1
          set -g NEW
          set CHANNELS (cat $FILE | grep -vE "#|;|\]")
          for c in $CHANNELS
              _ytdl_log "INFO" "Getting IDs for channel $c"
              if test (egrep '\/c\/|user|channel' (echo $c |psub))
                  set -g IDS (youtube-dl --get-id "$c/videos")
              else
                  set -g IDS (youtube-dl --get-id $c)
              end
              _ytdl_log "INFO" "Fetching new videos from channel"
              for i in (seq (count $IDS))
                  printf "\rsearching (%d/%d)" $IDn (count $IDS)
                  rg -- $IDS[$i] $DOWNFILE 2&> /dev/null
                  if test $status -ne 0
                      set -g NEW $IDS[$i] $NEW
                  end
              end
              printf "\n"
          end
          for i in (seq (count $NEW))
              _ytdl_download_video $NEW[$i] $i (count $NEW)
          end
      end
  end

Main Body

Now that we have all our functions declared, let’s call them! First, we need to parse our arguments. We’ll then download all files passed as arguments. Finally, we’ll download videos, playlists and channels specified from a batch file.

  _ytdl_parse_ops $argv
  _ytdl_download_arg_urls $VIDEOS
  _ytdl_download_batch

And that’s all! If you’re interested with a very simple interface for downloading one video once, I wrote a small rofi-ytdl script that calls the rofi utility to specify a single link and download it.

Plock

plock is a simple script that locks the screen with i3lock while setting as the background image of the locked screen a corrupted screenshot of the screen before it was locked.

  set TMPBG /tmp/screen.png
  scrot $TMPBG
  corrupter -add 0 $TMPBG $TMPBG
  i3lock -t -e -f -i $TMPBG
  rm $TMPBG

Rofi utilities

askpass

Askpass is a simple script that invokes rofi as a way to get from a GUI the user’s sudo password. It is inspired by this original tool, rewritten in fish and with rofi support instead of dmenu. As you can see, this is a oneliner if we ignore the initial shebang. This executable is pointed at by the

  rofi -dmenu -password -no-fixed-num-lines -p (printf $argv[1] | sed s/://)

awiki

awiki is a simple script used with rofi that relies on the arch-wiki-docs package in order to provide the user a way to quickly find and display any English page from the Arch Wiki in a browser. The advantage of using this over the wiki-search utility from the arch-wiki-lite package is you get instant suggestion in rofi using fuzzy-search. The downside is rofi will only help you find pages by their title, and it will not help you find keywords in the content of said pages.

The first step is to create the list of all the pages that are currently stored on disk. arch-wiki-docs stores them in /usr/share/doc/arch-wiki/html/en. A simple ls piped in three sed will give us a list of page titles. We then pipe that into rofi in dmenu mode in order to choose the page we want to display. By the way, setting the location of the HTML files will come in handy later.

  set WLOCATION /usr/share/doc/arch-wiki/html/en/
  set WPAGE (/bin/ls $WLOCATION                              | \
      sed -e 's/_/ /g' -e 's/\.html$//' -e 's|.*/\(.*\)|\1|' | \
      rofi -dmenu -p "Arch Wiki" -i)
  set WPAGE (echo $WPAGE | sed -r 's/\s+/_/g')

Now, all I need to do is to send this list into rofi and tell it to open the result with our favorite browser with xdg-open.

  xdg-open $WLOCATION$WPAGE.html

dmenu

I wrote this very simple script in order to replace dmenu with rofi’s emulation of dmenu, since I prefer rofi’s appearance. It basically calls rofi’s dmenu emulation with the arguments initially passed to dmenu.

  rofi -dmenu $argv

Emoji picker

The emoji picker is a simple fish script that uses rofi and ~/.config/emoji.txt to provide a small, local search for emojis. Once the emoji is selected, it is copied to the clipboard using xclipboard.

  grep -v "#" ~/.config/emoji.txt | rofi -dmenu -p "Select emoji" -i | \
  awk '{print $1}' | tr -d '\n' | xclip -selection clipboard

Also, let’s send a notification telling the user the emoji has been copied!

  set emoji (xclip -o -selection clipboard | tr -d '\n')
  test -z "$emoji" && notify-send "No emoji copied" -u low && exit
  set -a emoji "copied to clipboard"
  notify-send -u low $emoji

It is inspired from this video from Luke Smith, rewritten in Fish.

Partition mounting and unmounting

Rofi-mount

rofimount is a script inspired by this one, based on dmenu, which interactively asks the user what to mount, and where to mount it. What I did was replace dmenu with rofi, and fix a couple of bugs I encountered in the original script.

Get the mountable elements

  begin

What the script does first is detect everything that can be mounted. Between a begin and end, let’s set LFS as a local variable. This si in order to get sane variables in the current block.

set -l LFS

Now, let’s detect the amount of mountable Android filesystems, and if any are detected, let’s read them into a global variable.

  set -l a (math (jmtpfs -l | wc -l) - 2)
  test $a -ge 0 && jmtpfs -l 2> /dev/null | tail -n $a | read -zg anddrives

We’ll do the same for external and internal drives and partitions that can be mounted here.

  lsblk -rpo "name,type,size,mountpoint" | \
  awk '$2=="part"&&$4==""{printf "%s (%s)\n",$1,$3}' | \
  read -zg usbdrives

Finally, we look for any CD drive that could be mounted on our device.

  blkid /dev/sr0 | awk '{print $1}' | sed 's/://' | read -z cddrives

And that’s the end of our first block!

end

Alright, we’ll save what kind on drives we can mount in a temporary file called /tmp/drives. We’ll make sure it’s blank by erasing it then creating it again with touch, like so. The -f flag on rm is here so we get no error if we try to delete a file that doesn’t exist (yet).

  set -g TMPDRIVES /tmp/drives
  rm -f $TMPDRIVES
  touch $TMPDRIVES

Now, let’s write what type of drives we can mount in this temporary file.

  test -n "$usbdrives" && echo "USB" >> $TMPDRIVES
  test -n "$cddrives" && echo "CD" >> $TMPDRIVES
  test -n "$anddrives" && echo "Android" >> $TMPDRIVES

Now, we want to declare where to look for mount directories. For now, we’ll only look in /media, but you can add more if you wish.

  set -g basemount /media

Get the mount point

Now, let’s declare a function that will allow us to chose the drive we want to mount.

  function getmount

First, we want to get our mount point. We’ll run a find command on each of the directories listed in $basemount to look for folders on which our drive could be mounted. This list will be passed to rofi from which we will chose our mount point.

  set -g mp (for d in $basemount
      find $d -maxdepth 5 -type d
  end | rofi -dmenu -i -p 'Type in mount point.')

We should verify that something has been actually selected, otherwise we should abort the script.

  if test -z $mp || test $mp = ""
      return 1
  end

Now, if the selected mount point does not exist, we’ll ask the user whether the directory should be created. If no, the script will abort. If yes, an attempt will be made at creating the directory as the user; if that fails, a new attempt will be made as sudo.

  if test ! -d $mp
      switch (printf "No\\nYes" | rofi -dmenu -i -p "$mp does not exist. Create it?")
          case 'Yes'
              mkdir -p $mp || sudo -A mkdir -p $mp
          case '*'
              return 1
      end
  end

Finally, let’s close the function

end

Mount a USB drive, hard drive or partition

Alright, we want to mount a partition that answers by the name of /dev/sdXX, how do we do that? Let’s create first the function mountusb that will take care of it for us.

function mountusb

Now, the first thing we want to do is select the partition we want to mount. Remember, we stored those in $usbdrives earlier, so let’s pipe them into rofi so we can chose from it. Also, awk will get their path in /dev.

  set -g chosen (echo $usbdrives | \
  rofi -dmenu -i -p "Mount which drive?" | \
  awk '{print $1}')

As usual after a user selection, let’s verify something has actually been selected. If not, let’s abort the script.

test -z $chosen && return 1

Now, let’s select the mount point of our partition. We’ll call the function getmount described in Get the mount point to select it.

getmount

Let’s verify the variable mp set in getmount is not empty, otherwise abort the script.

test -z $mp && return 1

Now, let’s mount it! We’ll use a switch which will detect the filesystem used so we know how to mount the partition.

switch (lsblk -no "fstype" $chosen)

We have two named case: vfat filesystems.

  case "vfat"
      sudo -A mount -t vfat $chosen $mp -o rw,umask=0000

And ntfs filesystems.

  case "ntfs"
      sudo -A mount -t ntfs $chosen $mp -o rw,umask=0000

Else, we’ll let mount determine which filesystem is used by the partition (generally ext4).

  case '*'
      sudo -A mount $chosen $mp

We’ll also run a chown on this newly mounted filesystem so the user can access it without any issues.

  sudo -A chown -R $USER:(id -g $USER) $mp

Let’s close the switch block and send a notification the partition has been mounted.

end && notify-send -a "dmount" "💻 USB mounting" "$chosen mounted to $mp."

And let’s close the function.

end

Mount an Android device

The function that manages to mount Android filesystems is mountandroid. Let’s declare it.

function mountandroid -d "Mount an Android device"

We’ll select which Android we want to mount. We will be asked through rofi.

set chosen (echo $anddrives | rofi -dmenu -i -p "Which Android device?" | awk '{print $1 $2}' | sed 's/,$//')

Now, we need to get the bus of the Android device we want to mount. It will be useful later, after we authorized mounting from our device, to get the path to our partition.

set bus (echo $chosen | sed 's/,.*//')

Let’s temporarily mount our device.

jmtpfs -device=$chosen $mp

Now, we need to allow our computer to mount our Android device. Depending on the Android version it is running on, we either need to specify our device is USB connected in order to exchange files, or Android will explicitely ask us if it is OK for our computer to access it. Let’s inform the user of that.

  echo "OK" | \
  rofi -dmenu -i -p "Press (Allow) on your phone screen, or set your USB settings to allow file transfert"

Now, let’s get the actual path of our Android filesystem we wish to mount, and let’s unmount the previous temporary filesystem.

  set newchosen (jmtpfs -l | grep $bus | awk '{print $1 $2}' | sed 's/,$//')
  sudo -A umount $mp

Now we cam mount the new filesystem and send a notification if everything went well.

  jmtpfs -device=$newchosen $mp && \
  notify-send -a "dmount" "🤖 Android Mounting" "Android device mounted to $mp."

And now, we can close our function.

end

Mount a CD drive

This part is way easier than the previous functions. As we will see, the function mountcd's body is only three lines long. First, let’s declare the function.

function mountcd

Now, let’s chose the CD drive we want to mount using rofi.

  set chosen (echo $cddrives | rofi -dmenu -i -p "Which CD drive?")

We’ll also get the mountpoint thanks to the getmount function described earlier.

getmount

And finally, let’s mount it and send the notification everything went well.

  sudo -A mount $chosen $mp && \
  notify-send -a "dmount" "💿 CD mounting" "$chosen mounted."

Finally, let’s close our function.

end

Ask what type of drive we want to mount

The first thing we will be asked if different types of drives are detected is which of these types the user wishes to mount. This is done with the function asktype which is declared below.

function asktype

We will use a switch statement which will use our anwser to rofi about what we wish to mount.

switch (cat $TMPDRIVES | rofi -dmenu -i -p "Mount which drive?")

If we chose the option "USB", we’ll mount a hard drive, partition or USB drive. In which case we’ll call the mountusb function.

  case "USB"
      mountusb

If we chose the "Android" option, the mountandroid function is called.

  case "Android"
      mountandroid

Else if we chose the "CD" option, we’ll call the mountcd function.

  case "CD"
      mountcd

If nothing is selected, the function will naturally exit. Now, let’s close our switch statement and our function.

end
end

Launch the mounting functions

Now that we have declared our functions and set our variables, we’ll read the temporary file described in Get the mountable elements. The amount of lines is passed in a switch statement.

switch (wc -l < $TMPDRIVES)

If the file has no lines, i.e. it is empty, we have no mountable media. Let’s inform our user this is the case.

  case 0
      notify-send "No USB drive or Android device or CD detected" -a "dmount"

If we only have one line, we have only one type of mountable media. We’ll pass this line to a second switch statement.

  case 1
      switch (cat $TMPDRIVES)

This will allow the script to automatically detect what type of media it is, and mount the corresponding function.

  case "USB"
      mountusb
  case "Android"
      mountandroid
  case "CD"
      mountCD

Let’s close this nested switch case.

end

If we have more than one line, we’ll have to ask the user what type of media they want to mount.

  case '*'
      asktype

Now, let’s end our switch statement!

end

Finally, we’ll delete our temporary file.

rm -f $TMPDRIVES

And with that, this is the end of our script!

Rofi-umount

rofiumount is the counterpart of rofimount for unmounting our mounted partitions.

Get the unmountable drives

First, we will need to list all the drives that can be safely unmounted. Let’s run this.

  set -g drives (lsblk -nrpo "name,type,size,mountpoint" | \
  awk '$2=="part"&&$4!~/\/boot|\/home$|SWAP/&&length($4)>1{printf "%s (%s)\n",$4,$3}')

Now, let’s get the android devices that are mounted.

set -g androids (awk '/jmtpfs/ {print $2}' /etc/mtab)

And let’s get the CD drives that are mounted.

set -g cds (awk '/sr0/ {print $2}' /etc/mtab)

We’ll store all of our information in a temporary file, /tmp/undrives.

set -g undrivefile /tmp/undrives

Let’s make sure we begin with a clean, empty file.

  rm -f $undrivefile
  touch $undrivefile

Depending on if the related variables are set, write the different types of mounted drives in the temporary file.

  test -n "$drives" && echo "USB" >> $undrivefile
  test -n "$cds" && echo "CD" >> $undrivefile
  test -n "$androids" && echo "Android" >> $undrivefile

Unmount disk partitions

The function unmountusb will take care of unmounting any drive we can safely unmount. First, let’s declare the function.

function unmountusb

Let’s chose the drive to unmount with rofi.

  set chosen (echo $drives | \
  rofi -dmenu -i -p "Unmount which drive?" | \
  awk '{print $1}')

Let’s verify if the user actually selected any drive. If no, let’s abort the script.

test -z "$chosen" && exit 0

Now, let’s unmount the chosen drive and send a notification if it has been done.

  sudo -A umount $chosen && \
  notify-send "💻 USB unmounting" "$chosen unmounted." -a "dumount"

Now, let’s close the function.

end

Unmount Android device

The function unmountandroid will take care of unmounting any mounted Android device. First, let’s declare our function.

function unmountandroid

Let the user choose which Android device to unmount.

set chosen (echo $androids | rofi -dmenu -i -p "Unmount which device?")

We’ll verify the user chose any device.

  test -z "$chosen" && exit 0

If a device has been chosen, let’s unmount it and send a notification it has been successfuly unmounted.

  sudo -A umount -l $chosen && \
  notify-send "🤖 Android unmounting" "$chosen unmounted." -a "dumount"

Finally, let’s close the function.

end

Unmount CD drive

unmountcd will take care of unmounting any mounted CD drive. Let’s declare this function.

function unmountcd

As before, let the user chose which CD drive to unmount.

set chosen (echo "$cds" | rofi -dmenu -i -p "Unmount which CD?")

We’ll verify the user chose any device.

  test -z "$chosen" && exit 0

If a drive has been chosen, let’s unmount it and send a notification it has been successfuly unmounted.

  sudo -A umount -l $chosen && \
  notify-send "💿 CD unmounting" "$chosen unmounted." -a "dumount"

Now, let’s close the function.

end

Ask what type of drive to unmount

If several types of unmountable drives are available, let’s ask the user which type to unmount based on the content of the temporary file declared in Get the unmountable drives. First, let’s declare the function.

function asktype

Let’s create a switch statement to which will be passed the selection of the user from rofi.

  switch (cat $undrivefile | rofi -dmenu -i -p "Unmount which type of device?")

Three types of values can be returned: "USB", "CD", or "Android". These values will be used to launch their corresponding function.

  case 'USB'
      unmountusb
  case 'CD'
      unmountcd
  case 'Android'
      unmountandroid

Let’s close the switch statement.

end

Let’s now close the function.

end

Launch the unmounting functions

Now back to the body of our script, let’s input in a switch case the number of lines contained in our temporary file.

switch (wc -l < $undrivefile)

If the file containes no lines. i.e. it is empty, nothing is to be unmounted. Let’s inform the user of that.

  case 0
      notify-send "No USB drive or Android device or CD to unmount" -a "dumount"

Else, if there is only one type of drive, we’ll automatically let our script choose based on the content of this sole line.

  case 1
      switch (cat $undrivefile)
          case 'USB'
              unmountusb
          case 'CD'
              unmountcd
          case 'Android'
              unmountandroid
      end

And if there are more types than one, let’s ask the user.

case '*'
     asktype

Let’s close our main switch statement.

end

And finally, let’s delete our temporary file.

rm -f $undrivefile

Rofi-pass

rofi-pass is a simple utility that gets a password stored in the pass password manager with rofi as its interface, and then stores the password in the clipboard.

Let’s parse all the arguments passed to the script. If one of them is --type, -t or type, the script will attempt to type the password to the text area already selected without pasting the password to the clipboard.

  for arg in $argv
      switch $arg
          case '--type' '-t' 'type'
              set -g TYPE "yes"
          case '*'
              printf 'Unknown argument: %s\n.' $arg
              exit 1
      end
  end

Now, let’s get the list of the passwords that exist in our pass repository.

  set passwords (find $HOME/.password-store -type f -name "*.gpg" | \
  string replace -r ".*.password-store/" "" | \
  string replace -r ".gpg" "" | sort)

Let the user choose which password they wish to select.

  set password (for elem in $passwords
      echo $elem
  end | rofi -dmenu -i -p "Select your password")

Let’s verify we actually selected a password and not just exited. If no password was selected, let’s simply exit the script.

  if test -z $password
      exit
  end

Depending on the arguments passed earlier, we might want some different behavior.

  if test $TYPE = "yes"
      <<rofi-pass-type>>
  else
      <<rofi-pass-copy>>
  end

The default behavior is to copy the password to the clipboard for 45 seconds, so let’s do that.

  pass show -c $password 2> /dev/null

Else, if we passed --type, -t or type as an argument of the script, we want it to attempt to type our password in the currently selected text input. Let’s do that.

  set -l IFS
  <<rofi-pass-type-get-password>>
  printf %s $pass | xvkbd -file -

To correctly get the password from pass, we need to parse the output and only get the first line, hence the following command.

set pass (pass show $password | string split -n \n)[1]

rofi-ytdl

This is just a simple wrapper around ytdl so I can easily download a video from rofi, which we’ll use first to retrieve the URL of the video we want to download, be it from YouTube or other website supported by youtube-dl.

  URL=$(echo "Video to download:" | rofi -dmenu -i -p "Video to download:")

Now, if the variable URL is not empty (i.e. the user specified a link and did not abort the operation), we’ll proceed to teh download. Before it begins, we’ll send a notification saying the download is about to begin. When the ytdl process ends, we’ll also send a notification notifying the user on the success or failure of the download.

  if [ -n "$URL" ]; then
      notify-send -u normal "YTDL" "Starting downloading\n$URL"
      ytdl "$URL" \
          && notify-send -u normal "YTDL" "Finished downloading!" \
              || notify-send -u critical "YTDL" "Failed downloading\n$URL"
  fi

Wallpaper utilities

pape-update

This little tool sets a random wallpaper using xwallpaper.

  PAPESDIR=$HOME/Pictures/Wallpapers
  PAPE=$(find $PAPESDIR -type f | sort -R | tail -1)
  set-pape $PAPE

Select wallpaper

This script is base on what sxiv can do as an image viewer as well as xwallpaper.

  PAPE=$(sxiv -orbft ~/Pictures/Wallpapers/*)
  set-pape $PAPE

Set a wallpaper

This utility is not meant to be called by the user directly, but rather by scripts that may be written by the user. Its role is simple: check if the provided wallpaper exists and if it is an image. If both requirements are met, the path to this image is then stored in $XDG_CACHE_HOME/wallpaper, or if this variable is empty in $HOME/.cache/wallpaper.

  CACHEFILE=$([ -n "$XDG_CACHE_HOME" ] && echo $XDG_CACHE_HOME/wallpaper || echo $HOME/.cache/wallpaper)
  [[ -f $1 ]] &&                              \
      grep image <(file -b --mime-type $1) && \
      echo $1 > $CACHEFILE                    \
      && xwallpaper --zoom $1

Weather

A quick and useful script I often use is a curl request to v2.wttr.in to get a weather forecast in the terminal. By default, I want the request to be about the city I live in, but it is also possible for the script to accept as its arguments a search inquiry.

  if count $argv > /dev/null
      set -l SEARCH (string join '+' $argv)
      curl http://v2.wttr.in/~$SEARCH
  else
      curl http://v2.wttr.in/Aubervilliers
  end

65 KiB Raw Blame History Unescape Escape

Executable scripts

Presentation

Autostart

cli utilities

Backup

Development

Cnew

Dart Language Server

UpdateFlutter

set-screens

sshbind

Starwars

Toggle touchpad tapping

Wacom setup

Set our variables

Select our screen

Adjust the tablet

Lauch the functions

Emacsmail

Media

mp42webm

youtube-dl wrappers

ytplay

ytdl - a youtube-dl wrapper

Setting default values

Help message

Arguments Handling

Logging

Download a Single Video

Download a Queue of Videos

Download Videos From Arguments

Download Videos From a Batch File

Main Body

Plock

Rofi utilities

askpass

awiki

dmenu

Emoji picker

Partition mounting and unmounting

Rofi-mount

Get the mountable elements

Get the mount point

Mount a USB drive, hard drive or partition

Mount an Android device

Mount a CD drive

Ask what type of drive we want to mount

Launch the mounting functions

Rofi-umount

Get the unmountable drives

Unmount disk partitions

Unmount Android device

Unmount CD drive

Ask what type of drive to unmount

Launch the unmounting functions

Rofi-pass

rofi-ytdl

Wallpaper utilities

pape-update

Select wallpaper

Set a wallpaper

Weather

65 KiB

Raw Blame History

ytdl - a `youtube-dl` wrapper