2Shirt/GitHub-to-B2
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Initial rewrite to use BASH3 Boilerplate

Browse source
This commit is contained in:
2Shirt 2018-06-01 01:26:09 -06:00
commit 89e90f818a
7 changed files with 1025 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
LICENSE.txt Normal file
View file

@ -0,0 +1,7 @@
Copyright (c) 2018 Alan Mason
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

100
backup-git-repos-to-b2 Executable file
View file

@ -0,0 +1,100 @@
#!/bin/bash
# Script to backup git repo(s) to Backblaze B2
script_name="$(basename "${0}")"
# Usage
read -r -d '' __usage <<-'EOF' || true
Usage: backup-git-repos-to-b2 [options] repo(s)...
Options:
-b --b2-bucket [arg] Backblaze B2 bucket name. Required.
-i --b2-id [arg] Backblaze B2 account ID. Required.
-k --b2-key [arg] Backblaze B2 application key. Required.
-g --github-account [arg] GitHub account name. Required.
-t --temp-dir [arg] Dir to use for temp files.
-h --help This page.
EOF
# Help text
__helptext='NOTE: --temp-dir defaults to "$(mktemp -d)" if not specified.'
source bash3boilerplate/main.sh
b2_bucket="${arg_b}"
b2_cmd="b2"
b2_id="${arg_i}"
b2_key="${arg_k}"
date="$(date '+%F_%H%M%z')"
github_account="${arg_g}"
temp_dir="${arg_t:-}"
if [[ "$temp_dir" == "" ]]; then
temp_dir="$(mktemp -d)"
fi
# Check for backblaze-b2
# Check for b2 cmd
if which backblaze-b2 >/dev/null 2>&1; then
# Arch Linux uses this name
b2_cmd="backblaze-b2"
elif ! which b2 > /dev/null 2>&1; then
echo "Error B2 command-line tool not found"
exit 1
fi
# Setup repository to ${1}
repository="${1}"
if [[ "${repository: -4:4}" == ".git" ]]; then
repository="${repository:0: -4}"
fi
# Create the backup directory
mkdir -p "${temp_dir}"
pushd "${temp_dir}" > /dev/null
echo "Backing up ${repository}"
git clone --mirror --verbose \
"https://github.com/${github_account}/${repository}.git" \
"${repository}-${date}.git"
if [ $? -ne 0 ]; then
echo "Error cloning ${repository}"
exit 1
fi
tar cpzf "${repository}-${date}.git.tgz" "${repository}-${date}.git"
if [ $? -ne 0 ]; then
echo "Error compressing ${repository}"
exit 1
fi
# Authorize B2 account
"${b2_cmd}" authorize-account "${b2_id}" "${b2_key}"
if [ $? -ne 0 ]; then
echo "Error authorizing Backblaze B2 account"
exit 1
fi
if [[ -f "${repository}-${date}.git.tgz" ]]; then
"${b2_cmd}" upload-file "${b2_bucket}" \
"${repository}-${date}.git.tgz" \
"${github_account}/${repository}-${date}.git.tgz"
fi
if [ $? -ne 0 ]; then
echo "Error uploading ${repository} to Backblaze B2"
exit 1
fi
#delete tar file and checked out folder
/bin/rm "${temp_dir}/${repository}-${date}.git.tgz"
/bin/rm -rf "${temp_dir}/${repository}-${date}.git"
if [ $? -ne 0 ]; then
echo "Error removing temp files"
exit 1
fi
popd > /dev/null
echo "Succesfully backed up ${repository}"

134
bash3boilerplate/CHANGELOG.md Normal file
View file

@ -0,0 +1,134 @@
# Changelog
Here's is a combined todo/done list. You can see what todos are planned for the upcoming release, as well as ideas that may/may not make into a release in `Ideas`.
## Ideas
Unplanned.
- [ ] Better style guide checking (#84)
## master
Released: TBA.
[Diff](https://github.com/kvz/bash3boilerplate/compare/v2.3.0...master).
- [ ] Upgrade to `lanyon@0.0.55`
- [ ] Allow counting how many times an argument is used
- [x] Fix typos in megamount (thanks @gsaponaro)
- [x] Enable color in screen or tmux (#92, @gmasse)
- [x] Change `egrep` to `grep -E` in test and lib scripts to comply with ShellCheck (#92, @gmasse)
- [x] Fix typo in FAQ (#92, @gmasse)
- [x] Fix Travis CI failure on src/templater.sh (@gmasse)
## v2.3.0
Released: 2016-12-21.
[Diff](https://github.com/kvz/bash3boilerplate/compare/v2.2.0...v2.3.0).
- [x] Add magic variable `__i_am_main_script` to distinguish if b3bp is being sourced or called directly (#45, @zbeekman)
- [x] Add style checks for tab characters and trailing whitespace (@zbeekman)
- [x] Add backtracing to help localize errors (#44, @zbeekman)
- [x] Additional FAQ entries (#47, suggested by @gdevenyi, implemented by @zbeekman)
- [x] Ensure that shifting over `--` doesn't throw an errexit error (#21, @zbeekman)
- [x] Add Pull Request template (#83)
## v2.2.0
Released: 2016-12-21.
[Diff](https://github.com/kvz/bash3boilerplate/compare/v2.1.0...v2.2.0).
- [x] README and FAQ improvements (#66, @mstreuhofer)
- [x] Add support for sourcing b3bp (#61, @mstreuhofer)
- [x] Upgrade all Node.js dependencies for development (#78)
- [x] Switch to http://lanyon.io for static site building, add a new logo
- [x] Cleanup environment variables (#58, @mstreuhofer)
- [x] Support multi-line logs (#57, @mstreuhofer)
- [x] Run shellcheck as part of the acceptance test (#79, @mstreuhofer)
- [x] Brace all variables, used `[[` instead of `[` (#33, #76, @mstreuhofer)
- [x] Add automatic usage validation for required args (#22, #65, @mstreuhofer)
- [x] Remove all usage of eval (@mstreuhofer)
- [x] Get rid of awk, sed & egrep usage (#71, @mstreuhofer)
- [x] Fix auto-color-off code (#69, #70, @mstreuhofer)
- [x] Use shellcheck to find and fix unclean code (#68, #80, @mstreuhofer)
- [x] Allow for multiline opt description in `__usage` (#7, @mstreuhofer)
- [x] Allow `__usage` and `__helptext` to be defined before sourcing `main.sh` thus makeing b3bp behave like a library (@mstreuhofer)
- [x] Add the same License text to each script header (@mstreuhofer)
## v2.1.0
Released: 2016-11-08.
[Diff](https://github.com/kvz/bash3boilerplate/compare/v2.0.0...v2.1.0).
- [x] Cleanup b3bp variables (adds prefixes across the board) (thanks @mstreuhofer)
- [x] Add multi-line logging support (thanks @mstreuhofer)
- [x] Mangle long-option names to allow dashes (thanks @zbeekman)
- [x] Remove OS detection altogether (#38, thx @zbeekman)
- [x] Offer the main template for download as http://bash3boilerplate.sh/main.sh
- [x] Better OS detection (#38, thx @moviuro)
- [x] Improve README copy (#34, thx galaktos)
- [x] Fix unquoted variable access within (#34 thx galaktos)
- [x] For delete-key-friendliness, bundle the commandline definition block along with its parser
- [x] Less verbose header comments
- [x] For delete-key-friendliness, don't crash on undeclared help vars
- [x] Introduce `errtrace`, which is on by default (BREAKING)
- [x] Add a configurable `helptext` that is left alone by the parses and allows you to have a richer help
- [x] Add a simple documentation website
- [x] Add best practice of using `__double_underscore_prefixed_vars` to indicate global variables that are solely controlled inside your script
- [x] Make license more permissive by not requiring distribution of the LICENSE file if the copyright & attribution comments are left intact
- [x] Respect `--no-color` by setting the `NO_COLOR` flag in `main.sh` (#25, thx @gdevenyi)
- [x] Split out changelog into separate file
- [x] Added a [FAQ](./FAQ.md) (#15, #14, thanks @rouson)
- [x] Fix Travis OSX testing (before, it would silently pass failures) (#10)
- [x] Enable dashes in long, GNU style options, as well as numbers (thanks @zbeekman)
## v2.0.0
Released: 2016-02-17.
[Diff](https://github.com/kvz/bash3boilerplate/compare/v1.2.1...v2.0.0).
- [x] Add tests for `templater` and follow Library export best practices
- [x] Add tests for `ini_val` and follow Library export best practices
- [x] Add tests for `parse_url` and follow Library export best practices
- [x] Add tests for `megamount` and follow Library export best practices
- [x] Remove `bump` from `src` (BREAKING)
- [x] Remove `semver` from `src` (BREAKING)
## v1.2.1
Released: 2016-02-17.
[Diff](https://github.com/kvz/bash3boilerplate/compare/v1.2.0...v1.2.1).
- [x] Add Travis CI automated testing for OSX (thanks @zbeekman)
## v1.2.0
Released: 2016-02-16.
[Diff](https://github.com/kvz/bash3boilerplate/compare/v1.1.0...v1.2.0).
- [x] Allow disabling colors via `NO_COLOR` environment variable
- [x] Enable `errexit`, `nounset` and `pipefail` options at the top of the script already
- [x] More refined colors (thanks @arathai)
- [x] Add a changelog to the README
- [x] Add `__os` magic var (limited to discovering OSX and defaulting to Linux for now)
- [x] Add `__base` magic var (`main`, if the source script is `main.sh`)
- [x] Enable long, GNU style options (thanks @zbeekman)
- [x] Add Travis CI automated testing for Linux
## v1.1.0
Released: 2015-06-29.
[Diff](https://github.com/kvz/bash3boilerplate/compare/v1.0.3...v1.1.0).
- [x] Add `ALLOW_REMAINDERS` configuration to templater
- [x] Fix typo: 'debugmdoe' to 'debugmode' (thanks @jokajak)
- [x] Use `${BASH_SOURCE[0]}` for `__file` instead of `${0}`
## v1.0.3
Released: 2014-11-02.
[Diff](https://github.com/kvz/bash3boilerplate/compare/5db569125319a89b9561b434db84e4d91faefb63...v1.0.3).
- [x] Add `ini_val`, `megamount`, `parse_url`
- [x] Add re-usable libraries in `./src`
- [x] Use npm as an additional distribution channel

196
bash3boilerplate/FAQ.md Normal file
View file

@ -0,0 +1,196 @@
[This document is formatted with GitHub-Flavored Markdown. ]:#
[For better viewing, including hyperlinks, read it online at ]:#
[https://github.com/kvz/bash3boilerplate/blob/master/FAQ.md ]:#
## Contents
* [What is a CLI](#what-is-a-cli)?
* [How do I incorporate BASH3 Boilerplate into my own project](#how-do-i-incorporate-bash3-boilerplate-into-my-own-project)?
* [How do I add a command-line flag](#how-do-i-add-a-command-line-flag)?
* [How do I access the value of a command-line argument](#how-do-i-access-the-value-of-a-command-line-argument)?
* [What is a magic variable](#what-is-a-magic-variable)?
* [How do I submit an issue report](#how-do-i-submit-an-issue-report)?
* [How can I contribute to this project](#how-can-i-contribute-to-this-project)?
* [Why are you typing BASH in all caps](#why-are-you-typing-bash-in-all-caps)?
* [How can I locally develop and preview the b3bp website](#how-can-i-locally-develop-and-preview-the-b3bp-website)?
* [You are saying you are portable, but why won't b3bp code run in dash / busybox / posh / ksh / mksh / zsh](#you-are-saying-you-are-portable-but-why-wont-b3bp-code-run-in-dash--busybox--posh--ksh--mksh--zsh)?
* [How do I do Operating System detection](#how-do-i-do-operating-system-detection)?
* [How do I access a potentially unset (environment) variable](#how-do-i-access-a-potentially-unset-environment-variable)?
* [How can I detect or trap CTRL-C and other signals](#how-can-i-detect-or-trap-ctrl-c-and-other-signals)?
* [How can I get the PID of my running script](how-can-i-get-the-pid-of-my-running-script)?
<!--more-->
# Frequently Asked Questions
## What is a CLI?
A "CLI" is a [command-line interface](https://en.wikipedia.org/wiki/Command-line_interface).
## How do I incorporate BASH3 Boilerplate into my own project?
You can incorporate BASH3 Boilerplate into your project in one of two ways:
1. Copy the desired portions of [`main.sh`](http://bash3boilerplate.sh/main.sh) into your own script.
1. Download [`main.sh`](http://bash3boilerplate.sh/main.sh) and start pressing the delete-key to remove unwanted things
Once the `main.sh` has been tailor-made for your project, you can either append your own script in the same file, or source it in the following ways:
1. Copy [`main.sh`](http://bash3boilerplate.sh/main.sh) into the same directory as your script and then edit and embed it into your script using Bash's `source` include feature, e.g.:
```bash
#!/usr/bin/env bash
source main.sh
```
1. Source [`main.sh`](http://bash3boilerplate.sh/main.sh) in your script or at the command line:
```bash
#!/usr/bin/env bash
source main.sh
```
## How do I add a command-line flag?
1. Copy the line from the `main.sh` [read block](https://github.com/kvz/bash3boilerplate/blob/v2.1.0/main.sh#L109-L115) that most resembles the desired behavior and paste the line into the same block.
1. Edit the single-character (e.g., `-d`) and, if present, the multi-character (e.g., `--debug`) versions of the flag in the copied line.
1. Omit the `[arg]` text in the copied line, if the desired flag takes no arguments.
1. Omit or edit the text after `Default=` to set or not set default values, respectively.
1. Omit the `Required.` text, if the flag is optional.
## How do I access the value of a command-line argument?
To find out the value of an argument, append the corresponding single-character flag to the text `$arg_`. For example, if the [read block]
contains the line
```bash
-t --temp [arg] Location of tempfile. Default="/tmp/bar"
```
then you can evaluate the corresponding argument and assign it to a variable as follows:
```bash
__temp_file_name="${arg_t}"
```
## What is a magic variable?
The [magic variables](https://github.com/kvz/bash3boilerplate/blob/v2.1.0/main.sh#L26-L28) in `main.sh` are special in that they have a different value, depending on your environment. You can use `${__file}` to get a reference to your current script, and `${__dir}` to get a reference to the directory it lives in. This is not to be confused with the location of the calling script that might be sourcing the `${__file}`, which is accessible via `${0}`, or the current directory of the administrator running the script, accessible via `$(pwd)`.
## How do I submit an issue report?
Please visit our [Issues](https://github.com/kvz/bash3boilerplate/issues) page.
## How can I contribute to this project?
Please fork this repository. After that, create a branch containing your suggested changes and submit a pull request based on the master branch
of <https://github.com/kvz/bash3boilerplate/>. We are always more than happy to accept your contributions!
## Why are you typing BASH in all caps?
As an acronym, Bash stands for Bourne-again shell, and is usually written with one uppercase.
This project's name, however, is "BASH3 Boilerplate". It is a reference to
"[HTML5 Boilerplate](https://html5boilerplate.com/)", which was founded to serve a similar purpose,
only for crafting webpages.
Somewhat inconsistent – but true to Unix ancestry – the abbreviation for our project is "b3bp".
## How can I locally develop and preview the b3bp website?
You should have a working Node.js >=10, Ruby >=2 and [YARN](https://yarnpkg.com) install on your workstation. When that is the case, you can run:
```bash
yarn install
npm run start
```
This will install and start all required services and automatically open a webbrowser that reloads as soon as you make any changes to the source.
The source mainly consists of:
- `./README.md` (Front page)
- `./FAQ.md` (FAQ page)
- `./CHANGELOG.md` (changelog page)
- `./website/_layouts/default.html` (the design in which all pages are rendered)
- `./website/assets/app.js` (main JS file)
- `./website/assets/style.css` (main CSS file)
The rest is dark magic from which you should probably steer clear. : )
Any changes should be proposed as PRs. Anything added to `master` is automatically deployed using a combination of Travis CI and GitHub Pages.
## You are saying you are portable, but why won't b3bp code run in dash / busybox / posh / ksh / mksh / zsh?
When we say _portable_, we mean across Bash versions. Bash is widespread and most systems
offer at least version 3 of it. Make sure you have that available and b3bp will work for you.
We run automated tests to make sure that it will. Here is some proof for the following platforms:
- [Linux](https://travis-ci.org/kvz/bash3boilerplate/jobs/109804166#L91-L94) `GNU bash, version 4.2.25(1)-release (x86_64-pc-linux-gnu)`
- [OSX](https://travis-ci.org/kvz/bash3boilerplate/jobs/109804167#L2453-L2455) `GNU bash, version 3.2.51(1)-release (x86_64-apple-darwin13)`
This portability, however, does not mean that we try to be compatible with
KornShell, Zsh, posh, yash, dash, or other shells. We allow syntax that would explode if
you pasted it in anything but Bash 3 and up.
## How do I do Operating System detection?
We used to offer a magic `__os` variable, but we quickly [discovered](https://github.com/kvz/bash3boilerplate/issues/38) that it would be hard
to create a satisfactory abstraction that is not only correct, but also covers enough use-cases,
while still having a relatively small footprint in `main.sh`.
For simple OS detection, we recommend using the `${OSTYPE}` variable available in Bash as
is demoed in [this stackoverflow post](http://stackoverflow.com/a/8597411/151666):
```bash
if [[ "${OSTYPE}" = "linux-gnu" ]]; then
echo "GNU Linux"
elif [[ "${OSTYPE}" = "darwin"* ]]; then
echo "Mac OSX"
elif [[ "${OSTYPE}" = "cygwin" ]]; then
echo "POSIX compatibility layer and Linux environment emulation for Windows"
elif [[ "${OSTYPE}" = "msys" ]]; then
echo "Lightweight shell and GNU utilities compiled for Windows (part of MinGW)"
elif [[ "${OSTYPE}" = "win32" ]]; then
echo "I'm not sure this can happen."
elif [[ "${OSTYPE}" = "freebsd"* ]]; then
echo "..."
else
echo "Unknown."
fi
```
## How do I access a potentially unset (environment) variable?
The set -o nounset line in `main.sh` causes error termination when an unset environment variables is detected as unbound. There are multiple ways to avoid this.
Some code to illustrate:
```bash
# method 1
echo ${NAME1:-Damian} # echos Damian, $NAME1 is still unset
# method 2
echo ${NAME2:=Damian} # echos Damian, $NAME2 is set to Damian
# method 3
NAME3=${NAME3:-Damian}; echo ${NAME3} # echos Damian, $NAME3 is set to Damian
```
This subject is briefly touched on as well in the [Safety and Portability section under point 5](README.md#safety-and-portability). b3bp currently uses [method 1](https://github.com/kvz/bash3boilerplate/blob/v2.1.0/main.sh#L252) when we want to access a variable that could be undeclared, and [method 3](https://github.com/kvz/bash3boilerplate/blob/v2.1.0/main.sh#L31) when we also want to set a default to an undeclared variable, because we feel it is more readable than method 2. We feel `:=` is easily overlooked, and not very beginner friendly. Method 3 seems more explicit in that regard in our humble opinion.
## How can I detect or trap Ctrl-C and other signals?
You can trap [Unix signals](https://en.wikipedia.org/wiki/Unix_signal) like [Ctrl-C](https://en.wikipedia.org/wiki/Control-C) with code similar to:
```bash
# trap ctrl-c and call ctrl_c()
trap ctrl_c INT
function ctrl_c() {
echo "** Trapped CTRL-C"
}
```
See http://mywiki.wooledge.org/SignalTrap for a list of signals, examples, and an in depth discussion.
## How can I get the PID of my running script?
The PID of a running script is contained in the `${$}` variable. This is *not* the pid of any subshells. With Bash 4 you can get the PID of your subshell with `${BASHPID}`. For a comprehensive list of Bash built in variables see, e.g., http://www.tldp.org/LDP/abs/html/internalvariables.html

21
bash3boilerplate/LICENSE Normal file
View file

@ -0,0 +1,21 @@
The MIT License (MIT)
Copyright (c) 2013 Kevin van Zonneveld and contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

174
bash3boilerplate/README.md Normal file
View file

@ -0,0 +1,174 @@
[![Build Status](https://travis-ci.org/kvz/bash3boilerplate.svg?branch=master)](https://travis-ci.org/kvz/bash3boilerplate)
[This document is formatted with GitHub-Flavored Markdown. ]:#
[For better viewing, including hyperlinks, read it online at ]:#
[https://github.com/kvz/bash3boilerplate/blob/master/README.md]:#
* [Overview](#overview)
* [Goals](#goals)
* [Features](#features)
* [Installation](#installation)
* [Changelog](#changelog)
* [Frequently Asked Questions](#frequently-asked-questions)
* [Best Practices](#best-practices)
* [Who uses b3bp](#who-uses-b3bp)
* [Authors](#authors)
* [License](#license)
## Overview
<!--more-->
When hacking up Bash scripts, there are often things such as logging or command-line argument parsing that:
- You need every time
- Come with a number of pitfalls you want to avoid
- Keep you from your actual work
Here's an attempt to bundle those things in a generalized way so that
they are reusable as-is in most scripts.
We call it "BASH3 Boilerplate" or b3bp for short.
## Goals
Delete-Key-**Friendly**. Instead of introducing packages, includes, compilers, etc., we propose using [`main.sh`](http://bash3boilerplate.sh/main.sh) as a base and removing the parts you don't need.
While this may feel a bit archaic at first, it is exactly the strength of Bash scripts that we should want to embrace.
**Portable**. We are targeting Bash 3 (OSX still ships
with 3, for instance). If you are going to ask people to install
Bash 4 first, you might as well pick a more advanced language as a
dependency.
## Features
- Conventions that will make sure that all your scripts follow the same, battle-tested structure
- Safe by default (break on error, pipefail, etc.)
- Configuration by environment variables
- Simple command-line argument parsing that requires no external dependencies. Definitions are parsed from help info, ensuring there will be no duplication
- Helpful magic variables like `__file` and `__dir`
- Logging that supports colors and is compatible with [Syslog Severity levels](http://en.wikipedia.org/wiki/Syslog#Severity_levels), as well as the [twelve-factor](http://12factor.net/) guidelines
## Installation
There are three different ways to install b3bp:
### Option 1: Download the main template
Use curl or wget to download the source and save it as your script. Then you can start deleting the unwanted bits, and adding your own logic.
```bash
wget http://bash3boilerplate.sh/main.sh
vim main.sh
```
### Option 2: Clone the entire project
Besides `main.sh`, this will also get you the entire b3bp repository. This includes a few extra functions that we keep in the `./src` directory.
```bash
git clone git@github.com:kvz/bash3boilerplate.git
```
### Option 3: Require via npm
As of `v1.0.3`, b3bp can also be installed as a Node module, meaning you can define it as a dependency in `package.json` via:
```bash
npm init
npm install --save --save-exact bash3boilerplate
```
Even though this option introduces a Node.js dependency, it does allow for easy version pinning and distribution in environments that already have this prerequisite. This is, however, entirely optional and nothing prevents you from ignoring this possibility.
## Changelog
Please see the [CHANGELOG.md](./CHANGELOG.md) file.
## Frequently Asked Questions
Please see the [FAQ.md](./FAQ.md) file.
## Best practices
As of `v1.0.3`, b3bp offers some nice re-usable libraries in `./src`. In order to make the snippets in `./src` more useful, we recommend the following guidelines.
### Function packaging
It is nice to have a Bash package that can not only be used in the terminal, but also invoked as a command line function. In order to achieve this, the exporting of your functionality *should* follow this pattern:
```bash
if [ "${BASH_SOURCE[0]}" != "${0}" ]; then
export -f my_script
else
my_script "${@}"
exit $?
fi
```
This allows a user to `source` your script or invoke it as a script.
```bash
# Running as a script
$ ./my_script.sh some args --blah
# Sourcing the script
$ source my_script.sh
$ my_script some more args --blah
```
(taken from the [bpkg](https://raw.githubusercontent.com/bpkg/bpkg/master/README.md) project)
### Scoping
1. In functions, use `local` before every variable declaration.
1. Use `UPPERCASE_VARS` to indicate environment variables that can be controlled outside your script.
1. Use `__double_underscore_prefixed_vars` to indicate global variables that are solely controlled inside your script, with the exception of arguments that are already prefixed with `arg_`, as well as functions, over which b3bp poses no restrictions.
### Coding style
1. Use two spaces for tabs, do not use tab characters.
1. Do not introduce whitespace at the end of lines or on blank lines as they obfuscate version control diffs.
1. Use long options (`logger --priority` vs `logger -p`). If you are on the CLI, abbreviations make sense for efficiency. Nevertheless, when you are writing reusable scripts, a few extra keystrokes will pay off in readability and avoid ventures into man pages in the future, either by you or your collaborators. Similarly, we prefer `set -o nounset` over `set -u`.
1. Use a single equal sign when checking `if [[ "${NAME}" = "Kevin" ]]`; double or triple signs are not needed.
1. Use the new bash builtin test operator (`[[ ... ]]`) rather than the old single square bracket test operator or explicit call to `test`.
### Safety and Portability
1. Use `{}` to enclose your variables. Otherwise, Bash will try to access the `$ENVIRONMENT_app` variable in `/srv/$ENVIRONMENT_app`, whereas you probably intended `/srv/${ENVIRONMENT}_app`. Since it is easy to miss cases like this, we recommend that you make enclosing a habit.
1. Use `set`, rather than relying on a shebang like `#!/usr/bin/env bash -e`, since that is neutralized when someone runs your script as `bash yourscript.sh`.
1. Use `#!/usr/bin/env bash`, as it is more portable than `#!/bin/bash`.
1. Use `${BASH_SOURCE[0]}` if you refer to current file, even if it is sourced by a parent script. In other cases, use `${0}`.
1. Use `:-` if you want to test variables that could be undeclared. For instance, with `if [ "${NAME:-}" = "Kevin" ]`, `$NAME` will evaluate to `Kevin` if the variable is empty. The variable itself will remain unchanged. The syntax to assign a default value is `${NAME:=Kevin}`.
## Who uses b3bp?
- [Transloadit](https://transloadit.com)
- [OpenCoarrays](http://www.opencoarrays.org)
- [Sourcery Institute](http://www.sourceryinstitute.org)
- [Computational Brain Anatomy Laboratory](http://cobralab.ca/)
We are looking for endorsements! Are you also using b3bp? [Let us know](https://github.com/kvz/bash3boilerplate/issues/new?title=I%20use%20b3bp) and get listed.
## Authors
- [Kevin van Zonneveld](http://kvz.io)
- [Izaak Beekman](https://izaakbeekman.com/)
- [Manuel Streuhofer](https://github.com/mstreuhofer)
- [Alexander Rathai](mailto:Alexander.Rathai@gmail.com)
- [Dr. Damian Rouson](http://www.sourceryinstitute.org/) (documentation, feedback)
- [@jokajak](https://github.com/jokajak) (documentation)
- [Gabriel A. Devenyi](http://staticwave.ca/) (feedback)
- [@bravo-kernel](https://github.com/bravo-kernel) (feedback)
- [@skanga](https://github.com/skanga) (feedback)
- [galaktos](https://www.reddit.com/user/galaktos) (feedback)
- [@moviuro](https://github.com/moviuro) (feedback)
- [Giovanni Saponaro](https://github.com/gsaponaro) (feedback)
- [Germain Masse](https://github.com/gmasse)
- [A. G. Madi](https://github.com/warpengineer)
## License
Copyright (c) 2013 Kevin van Zonneveld and [contributors](https://github.com/kvz/bash3boilerplate#authors).
Licensed under [MIT](https://raw.githubusercontent.com/kvz/bash3boilerplate/master/LICENSE).
You are not obligated to bundle the LICENSE file with your b3bp projects as long
as you leave these references intact in the header comments of your source files.

393
bash3boilerplate/main.sh Executable file
View file

@ -0,0 +1,393 @@
#!/usr/bin/env bash
# This file:
#
# - Demos BASH3 Boilerplate (change this for your script)
#
# Usage:
#
# LOG_LEVEL=7 ./main.sh -f /tmp/x -d (change this for your script)
#
# Based on a template by BASH3 Boilerplate v2.3.0
# http://bash3boilerplate.sh/#authors
#
# The MIT License (MIT)
# Copyright (c) 2013 Kevin van Zonneveld and contributors
# You are not obligated to bundle the LICENSE file with your b3bp projects as long
# as you leave these references intact in the header comments of your source files.
# Exit on error. Append "|| true" if you expect an error.
set -o errexit
# Exit on error inside any functions or subshells.
set -o errtrace
# Do not allow use of undefined vars. Use ${VAR:-} to use an undefined VAR
set -o nounset
# Catch the error in case mysqldump fails (but gzip succeeds) in `mysqldump |gzip`
set -o pipefail
# Turn on traces, useful while debugging but commented out by default
# set -o xtrace
if [[ "${BASH_SOURCE[0]}" != "${0}" ]]; then
__i_am_main_script="0" # false
if [[ "${__usage+x}" ]]; then
if [[ "${BASH_SOURCE[1]}" = "${0}" ]]; then
__i_am_main_script="1" # true
fi
__b3bp_external_usage="true"
__b3bp_tmp_source_idx=1
fi
else
__i_am_main_script="1" # true
[[ "${__usage+x}" ]] && unset -v __usage
[[ "${__helptext+x}" ]] && unset -v __helptext
fi
# Set magic variables for current file, directory, os, etc.
__dir="$(cd "$(dirname "${BASH_SOURCE[${__b3bp_tmp_source_idx:-0}]}")" && pwd)"
__file="${__dir}/$(basename "${BASH_SOURCE[${__b3bp_tmp_source_idx:-0}]}")"
__base="$(basename "${__file}" .sh)"
# Define the environment variables (and their defaults) that this script depends on
LOG_LEVEL="${LOG_LEVEL:-6}" # 7 = debug -> 0 = emergency
NO_COLOR="${NO_COLOR:-}" # true = disable color. otherwise autodetected
### Functions
##############################################################################
function __b3bp_log () {
local log_level="${1}"
shift
# shellcheck disable=SC2034
local color_debug="\\x1b[35m"
# shellcheck disable=SC2034
local color_info="\\x1b[32m"
# shellcheck disable=SC2034
local color_notice="\\x1b[34m"
# shellcheck disable=SC2034
local color_warning="\\x1b[33m"
# shellcheck disable=SC2034
local color_error="\\x1b[31m"
# shellcheck disable=SC2034
local color_critical="\\x1b[1;31m"
# shellcheck disable=SC2034
local color_alert="\\x1b[1;33;41m"
# shellcheck disable=SC2034
local color_emergency="\\x1b[1;4;5;33;41m"
local colorvar="color_${log_level}"
local color="${!colorvar:-${color_error}}"
local color_reset="\\x1b[0m"
if [[ "${NO_COLOR:-}" = "true" ]] || ( [[ "${TERM:-}" != "xterm"* ]] && [[ "${TERM:-}" != "screen"* ]] ) || [[ ! -t 2 ]]; then
if [[ "${NO_COLOR:-}" != "false" ]]; then
# Don't use colors on pipes or non-recognized terminals
color=""; color_reset=""
fi
fi
# all remaining arguments are to be printed
local log_line=""
while IFS=$'\n' read -r log_line; do
echo -e "$(date -u +"%Y-%m-%d %H:%M:%S UTC") ${color}$(printf "[%9s]" "${log_level}")${color_reset} ${log_line}" 1>&2
done <<< "${@:-}"
}
function emergency () { __b3bp_log emergency "${@}"; exit 1; }
function alert () { [[ "${LOG_LEVEL:-0}" -ge 1 ]] && __b3bp_log alert "${@}"; true; }
function critical () { [[ "${LOG_LEVEL:-0}" -ge 2 ]] && __b3bp_log critical "${@}"; true; }
function error () { [[ "${LOG_LEVEL:-0}" -ge 3 ]] && __b3bp_log error "${@}"; true; }
function warning () { [[ "${LOG_LEVEL:-0}" -ge 4 ]] && __b3bp_log warning "${@}"; true; }
function notice () { [[ "${LOG_LEVEL:-0}" -ge 5 ]] && __b3bp_log notice "${@}"; true; }
function info () { [[ "${LOG_LEVEL:-0}" -ge 6 ]] && __b3bp_log info "${@}"; true; }
function debug () { [[ "${LOG_LEVEL:-0}" -ge 7 ]] && __b3bp_log debug "${@}"; true; }
function help () {
echo "" 1>&2
echo " ${*}" 1>&2
echo "" 1>&2
echo " ${__usage:-No usage available}" 1>&2
echo "" 1>&2
if [[ "${__helptext:-}" ]]; then
echo " ${__helptext}" 1>&2
echo "" 1>&2
fi
exit 1
}
### Parse commandline options
##############################################################################
# Commandline options. This defines the usage page, and is used to parse cli
# opts & defaults from. The parsing is unforgiving so be precise in your syntax
# - A short option must be preset for every long option; but every short option
# need not have a long option
# - `--` is respected as the separator between options and arguments
# - We do not bash-expand defaults, so setting '~/app' as a default will not resolve to ${HOME}.
# you can use bash variables to work around this (so use ${HOME} instead)
# shellcheck disable=SC2015
[[ "${__usage+x}" ]] || read -r -d '' __usage <<-'EOF' || true # exits non-zero when EOF encountered
-f --file [arg] Filename to process. Required.
-t --temp [arg] Location of tempfile. Default="/tmp/bar"
-v Enable verbose mode, print script as it is executed
-d --debug Enables debug mode
-h --help This page
-n --no-color Disable color output
-1 --one Do just one thing
EOF
# shellcheck disable=SC2015
[[ "${__helptext+x}" ]] || read -r -d '' __helptext <<-'EOF' || true # exits non-zero when EOF encountered
This is Bash3 Boilerplate's help text. Feel free to add any description of your
program or elaborate more on command-line arguments. This section is not
parsed and will be added as-is to the help.
EOF
# Translate usage string -> getopts arguments, and set $arg_<flag> defaults
while read -r __b3bp_tmp_line; do
if [[ "${__b3bp_tmp_line}" =~ ^- ]]; then
# fetch single character version of option string
__b3bp_tmp_opt="${__b3bp_tmp_line%% *}"
__b3bp_tmp_opt="${__b3bp_tmp_opt:1}"
# fetch long version if present
__b3bp_tmp_long_opt=""
if [[ "${__b3bp_tmp_line}" = *"--"* ]]; then
__b3bp_tmp_long_opt="${__b3bp_tmp_line#*--}"
__b3bp_tmp_long_opt="${__b3bp_tmp_long_opt%% *}"
fi
# map opt long name to+from opt short name
printf -v "__b3bp_tmp_opt_long2short_${__b3bp_tmp_long_opt//-/_}" '%s' "${__b3bp_tmp_opt}"
printf -v "__b3bp_tmp_opt_short2long_${__b3bp_tmp_opt}" '%s' "${__b3bp_tmp_long_opt//-/_}"
# check if option takes an argument
if [[ "${__b3bp_tmp_line}" =~ \[.*\] ]]; then
__b3bp_tmp_opt="${__b3bp_tmp_opt}:" # add : if opt has arg
__b3bp_tmp_init="" # it has an arg. init with ""
printf -v "__b3bp_tmp_has_arg_${__b3bp_tmp_opt:0:1}" '%s' "1"
elif [[ "${__b3bp_tmp_line}" =~ \{.*\} ]]; then
__b3bp_tmp_opt="${__b3bp_tmp_opt}:" # add : if opt has arg
__b3bp_tmp_init="" # it has an arg. init with ""
# remember that this option requires an argument
printf -v "__b3bp_tmp_has_arg_${__b3bp_tmp_opt:0:1}" '%s' "2"
else
__b3bp_tmp_init="0" # it's a flag. init with 0
printf -v "__b3bp_tmp_has_arg_${__b3bp_tmp_opt:0:1}" '%s' "0"
fi
__b3bp_tmp_opts="${__b3bp_tmp_opts:-}${__b3bp_tmp_opt}"
fi
[[ "${__b3bp_tmp_opt:-}" ]] || continue
if [[ "${__b3bp_tmp_line}" =~ ^Default= ]] || [[ "${__b3bp_tmp_line}" =~ \.\ *Default= ]]; then
# ignore default value if option does not have an argument
__b3bp_tmp_varname="__b3bp_tmp_has_arg_${__b3bp_tmp_opt:0:1}"
if [[ "${!__b3bp_tmp_varname}" != "0" ]]; then
__b3bp_tmp_init="${__b3bp_tmp_line##*Default=}"
__b3bp_tmp_re='^"(.*)"$'
if [[ "${__b3bp_tmp_init}" =~ ${__b3bp_tmp_re} ]]; then
__b3bp_tmp_init="${BASH_REMATCH[1]}"
else
__b3bp_tmp_re="^'(.*)'$"
if [[ "${__b3bp_tmp_init}" =~ ${__b3bp_tmp_re} ]]; then
__b3bp_tmp_init="${BASH_REMATCH[1]}"
fi
fi
fi
fi
if [[ "${__b3bp_tmp_line}" =~ ^Required\. ]] || [[ "${__b3bp_tmp_line}" =~ \.\ *Required\. ]]; then
# remember that this option requires an argument
printf -v "__b3bp_tmp_has_arg_${__b3bp_tmp_opt:0:1}" '%s' "2"
fi
printf -v "arg_${__b3bp_tmp_opt:0:1}" '%s' "${__b3bp_tmp_init}"
done <<< "${__usage:-}"
# run getopts only if options were specified in __usage
if [[ "${__b3bp_tmp_opts:-}" ]]; then
# Allow long options like --this
__b3bp_tmp_opts="${__b3bp_tmp_opts}-:"
# Reset in case getopts has been used previously in the shell.
OPTIND=1
# start parsing command line
set +o nounset # unexpected arguments will cause unbound variables
# to be dereferenced
# Overwrite $arg_<flag> defaults with the actual CLI options
while getopts "${__b3bp_tmp_opts}" __b3bp_tmp_opt; do
[[ "${__b3bp_tmp_opt}" = "?" ]] && help "Invalid use of script: ${*} "
if [[ "${__b3bp_tmp_opt}" = "-" ]]; then
# OPTARG is long-option-name or long-option=value
if [[ "${OPTARG}" =~ .*=.* ]]; then
# --key=value format
__b3bp_tmp_long_opt=${OPTARG/=*/}
# Set opt to the short option corresponding to the long option
__b3bp_tmp_varname="__b3bp_tmp_opt_long2short_${__b3bp_tmp_long_opt//-/_}"
printf -v "__b3bp_tmp_opt" '%s' "${!__b3bp_tmp_varname}"
OPTARG=${OPTARG#*=}
else
# --key value format
# Map long name to short version of option
__b3bp_tmp_varname="__b3bp_tmp_opt_long2short_${OPTARG//-/_}"
printf -v "__b3bp_tmp_opt" '%s' "${!__b3bp_tmp_varname}"
# Only assign OPTARG if option takes an argument
__b3bp_tmp_varname="__b3bp_tmp_has_arg_${__b3bp_tmp_opt}"
printf -v "OPTARG" '%s' "${@:OPTIND:${!__b3bp_tmp_varname}}"
# shift over the argument if argument is expected
((OPTIND+=__b3bp_tmp_has_arg_${__b3bp_tmp_opt}))
fi
# we have set opt/OPTARG to the short value and the argument as OPTARG if it exists
fi
__b3bp_tmp_varname="arg_${__b3bp_tmp_opt:0:1}"
__b3bp_tmp_default="${!__b3bp_tmp_varname}"
__b3bp_tmp_value="${OPTARG}"
if [[ -z "${OPTARG}" ]]; then
__b3bp_tmp_value=$((__b3bp_tmp_default + 1))
fi
printf -v "${__b3bp_tmp_varname}" '%s' "${__b3bp_tmp_value}"
debug "cli arg ${__b3bp_tmp_varname} = (${__b3bp_tmp_default}) -> ${!__b3bp_tmp_varname}"
done
set -o nounset # no more unbound variable references expected
shift $((OPTIND-1))
if [[ "${1:-}" = "--" ]] ; then
shift
fi
fi
### Automatic validation of required option arguments
##############################################################################
for __b3bp_tmp_varname in ${!__b3bp_tmp_has_arg_*}; do
# validate only options which required an argument
[[ "${!__b3bp_tmp_varname}" = "2" ]] || continue
__b3bp_tmp_opt_short="${__b3bp_tmp_varname##*_}"
__b3bp_tmp_varname="arg_${__b3bp_tmp_opt_short}"
[[ "${!__b3bp_tmp_varname}" ]] && continue
__b3bp_tmp_varname="__b3bp_tmp_opt_short2long_${__b3bp_tmp_opt_short}"
printf -v "__b3bp_tmp_opt_long" '%s' "${!__b3bp_tmp_varname}"
[[ "${__b3bp_tmp_opt_long:-}" ]] && __b3bp_tmp_opt_long=" (--${__b3bp_tmp_opt_long//_/-})"
help "Option -${__b3bp_tmp_opt_short}${__b3bp_tmp_opt_long:-} requires an argument"
done
### Cleanup Environment variables
##############################################################################
for __tmp_varname in ${!__b3bp_tmp_*}; do
unset -v "${__tmp_varname}"
done
unset -v __tmp_varname
### Externally supplied __usage. Nothing else to do here
##############################################################################
if [[ "${__b3bp_external_usage:-}" = "true" ]]; then
unset -v __b3bp_external_usage
return
fi
### Signal trapping and backtracing
##############################################################################
function __b3bp_cleanup_before_exit () {
info "Cleaning up. Done"
}
trap __b3bp_cleanup_before_exit EXIT
# requires `set -o errtrace`
__b3bp_err_report() {
local error_code
error_code=${?}
error "Error in ${__file} in function ${1} on line ${2}"
exit ${error_code}
}
# Uncomment the following line for always providing an error backtrace
# trap '__b3bp_err_report "${FUNCNAME:-.}" ${LINENO}' ERR
### Command-line argument switches (like -d for debugmode, -h for showing helppage)
##############################################################################
# debug mode
if [[ "${arg_d:?}" = "1" ]]; then
set -o xtrace
LOG_LEVEL="7"
# Enable error backtracing
trap '__b3bp_err_report "${FUNCNAME:-.}" ${LINENO}' ERR
fi
# verbose mode
if [[ "${arg_v:?}" = "1" ]]; then
set -o verbose
fi
# no color mode
if [[ "${arg_n:?}" = "1" ]]; then
NO_COLOR="true"
fi
# help mode
if [[ "${arg_h:?}" = "1" ]]; then
# Help exists with code 1
help "Help using ${0}"
fi
### Validation. Error out if the things required for your script are not present
##############################################################################
[[ "${arg_f:-}" ]] || help "Setting a filename with -f or --file is required"
[[ "${LOG_LEVEL:-}" ]] || emergency "Cannot continue without LOG_LEVEL. "
### Runtime
##############################################################################
info "__i_am_main_script: ${__i_am_main_script}"
info "__file: ${__file}"
info "__dir: ${__dir}"
info "__base: ${__base}"
info "OSTYPE: ${OSTYPE}"
info "arg_f: ${arg_f}"
info "arg_d: ${arg_d}"
info "arg_v: ${arg_v}"
info "arg_h: ${arg_h}"
info "$(echo -e "multiple lines example - line #1\\nmultiple lines example - line #2\\nimagine logging the output of 'ls -al /path/'")"
# All of these go to STDERR, so you can use STDOUT for piping machine readable information to other software
debug "Info useful to developers for debugging the application, not useful during operations."
info "Normal operational messages - may be harvested for reporting, measuring throughput, etc. - no action required."
notice "Events that are unusual but not error conditions - might be summarized in an email to developers or admins to spot potential problems - no immediate action required."
warning "Warning messages, not an error, but indication that an error will occur if action is not taken, e.g. file system 85% full - each item must be resolved within a given time. This is a debug message"
error "Non-urgent failures, these should be relayed to developers or admins; each item must be resolved within a given time."
critical "Should be corrected immediately, but indicates failure in a primary system, an example is a loss of a backup ISP connection."
alert "Should be corrected immediately, therefore notify staff who can fix the problem. An example would be the loss of a primary ISP connection."
emergency "A \"panic\" condition usually affecting multiple apps/servers/sites. At this level it would usually notify all tech staff on call."