Upload directory recursively to Sharepoint

For the times when you have to be icky and use it, here you go. Also this method uses the application server itself and the admin credential, because I was unable to authenticate using claims auth only from Powershell for using the client-side functions.

#    Upload-Directory-To-Sharepoint is adapted from https://stackoverflow.com/questions/20237294/upload-file-to-sharepoint-document-library-using-powershell
#    with some examples from https://www.sharepointdiary.com/2016/08/sharepoint-online-create-folder-using-powershell.html

Function Upload-Directory-To-Sharepoint {
	[CmdletBinding(SupportsShouldProcess)]
	Param(
		[Parameter(Mandatory = $False)][string]$ParentDirectory = ".",
		[Parameter(Mandatory = $False)][string]$SPSite = "https://sp.example.com/admin/test",
		[Parameter(Mandatory = $False)][string]$Library = "Documents"
	)
	$web = Get-SPWeb -Identity $SPSite

	If ($web) {
		try	{
			$list = $web.Lists.TryGetList($Library)
			# Change to the requested dir, so the relative paths all start with "." which can be easily cleaned up.
			Set-Location $ParentDirectory
			$files = Get-ChildItem -Path "." -Force -Recurse
			ForEach ($dir in $files | ? { $_.PSIsContainer } ) {
				# and we depend on them being ordered from parent directory down to child directories.
				$RelativeName = (($dir | Resolve-Path -Relative ) -Replace "^\.\\" ) -Replace "\\","/"
				$newDir = "$($list.RootFolder.ServerRelativePath)/$($RelativeName)"
				If ($PsCmdlet.ShouldProcess("Mkdir $($SPSite)$($newDir)")) { $results = $web.Folders.Add($newDir) ; }
			}
			ForEach ($file in $files | ? { !($_.PSIsContainer) -And $_.Name -ne "Thumbs.db" } ) {
				$RelativeName = (($file | Resolve-Path -Relative ) -Replace "^\.\\" ) -Replace "\\","/"
				$newFile = "$($list.RootFolder.Path)/$($RelativeName)"
				$newFileUrl = "$($SPSite)/$($newFile)"
				$stream = $file.OpenRead()
				If ($PSCmdlet.ShouldProcess("Upload $($newFileUrl)")) {
					$done = $list.RootFolder.Files.Add("$($newFile)", $stream, $true)
					Write-Host """$($SPSite)$($done.ServerRelativePath)"" Uploaded into the Site"    
				}
			}
		}
		catch {
			$ErrorMessage = $_.Exception.Message
			Write-Error $ErrorMessage
		}
	}

	else { Write-Host "Site Doesn't exist" }
	$list.Update();
}

Get L2 manager in AD

If you want to use Active Directory to store the attribute for each user, of the level 2 manager (probably “director”), you can get this value programmatically. Some assumptions include that you populate the manager attribute of each user, and that the top-level (CEO) is listed as the manager of himself. The logic of course can be manipulated for whatever your situation is.

I wrote this function which returns the L2 (or any other level you want) of the user. I leave it as an exercise for the reader to then plug this in to a ldap attribute.

Function Get-Level-Down-Manager {
	[CmdletBinding()]
	Param(
		[Parameter(Mandatory=$True )][ValidateNotNullOrEmpty()]$User,
		[Parameter(Mandatory=$False)][ValidateNotNullOrEmpty()]$Level = 2,
		[Parameter(Mandatory=$False)][ValidateNotNullOrEmpty()]$Property = "Object",
		[Parameter(Mandatory=$False)][boolean]$ShowStack = $False
	)
	Begin { $Stack = @() ; }
	Process {
		Try {
			$thisUser = Get-ADUser $User -properties manager
			Write-Verbose "$($thisUser)"
			$Stack += $thisUser
		}
		Catch { Throw "Invalid user $User. Aborted" }
		$done = $False
		While (!$done) {
			If ($ShowStack) { Write-Host $Stack }
			$newUser = Get-ADUser ( Get-ADUser $thisUser -properties samaccountname,manager,userprincipalname ).manager -properties samaccountname,manager,userprincipalname
			Write-Verbose "$($newUser)"
			Try { $newUser = Get-ADUser $newUser -properties manager }
			Catch { $LevelDownManager = $newUser ; $done = $True }
			If ($newUser.userprincipalname -eq $thisUser.userprincipalname) {
				# if person is the manager for self (so defined for the CEO, at least in AD) we are done.
				$done = $True
				$LevelDownManager = $Stack[-$Level]
			} Else {
				$Stack += $newUser
				$thisUser = $newUser
			}
		}
		# so now that we are done
		If ($Property -ne "Object") {
			$LevelDownManager = ( Get-ADUser $LevelDownManager -properties $Property ).$($Property)
		}
		Return $LevelDownManager
	}
}

nexuslib.ps1 Powershell library for uploading files to Nexus raw repository

After a few days’ worth of research, I have finally settled on the way I can upload files to a Nexus repository of type raw.

This function accepts multiple ways to pass in authentication: either the prebuilt header, or a regular Powershell credentials object, or just username and password.

And yes, this function depends on credlib.ps1.

# Filename: nexuslib.ps1
# Location: \\rdputil1\e$\scripts\Functions\
# Author: bgstack15
# Startdate: 2020-06-09
# Title: Upload files to a Nexus raw repository
# Purpose: Upload arbitrary files to Nexus
# History:
#    I originally tried http://blog.majcica.com/2016/03/31/uploading-artifacts-into-nexus-via-powershell/ but it must be out of date.
# Usage:
#    . nexuslib.ps1
#    Upload-File-To-Nexus Upload-File-To-Nexus -File "C:\Users\bgstack15\Downloads\input1.zip" -Repository "raw" -Directory "/somewhere" -FileName "output.zip" -Credential $hubtestCredential
# References:
#    curl.exe -v -X POST -F "raw.directory=example/raw/dir/" -F "raw.asset1=@C:\Users\bgstack15\Downloads\example.zip" -F "raw.asset1.filename=example.zip" -u bgstack15:SOMETHINGHERE https://hubtest.example.com/service/rest/v1/components?repository=raw --trace-ascii outputfile.txt
#    https://help.sonatype.com/repomanager3/rest-and-integration-api/components-api#ComponentsAPI-Raw
#    https://stackoverflow.com/questions/22491129/how-to-send-multipart-form-data-with-powershell-invoke-restmethod/48580319#48580319
# Improve:
#    Upload maven2 files? That use case will probably not come up here.
# Documentation:
# Dependencies:
#    \\rdputil1\e$\scripts\Functions\credlib.ps1

Function Upload-File-To-Nexus {
	[CmdletBinding()]
	Param(
		[Parameter(Mandatory=$True )][ValidateNotNullOrEmpty()][string]$File, # aka File
		[Parameter(Mandatory=$False)][string]$Server = "hubtest.benefitfocus.com",
		[Parameter(Mandatory=$False)][string]$Username,
		[Parameter(Mandatory=$False)][string]$Password,
		[Parameter(Mandatory=$False)][Hashtable]$AuthHeaders,
		[Parameter(Mandatory=$False)][System.Management.Automation.PSCredential]$Credential,
		[parameter(Mandatory=$True )][ValidateNotNullOrEmpty()][string]$Repository,
		[parameter(Mandatory=$True )][ValidateNotNullOrEmpty()][string]$Directory,
		[Parameter(Mandatory=$False)][ValidateNotNullOrEmpty()][string]$FileName # destination filename, which can be derived from uploaded file
	)

	Begin {
		If (!($AuthHeaders -ne $Null)) {
			If (!($Credential -ne $Null)) {
				If ($Username -ne $null -And $Password -ne $Null -And $Username -ne "" -And $Password -ne "") {
					$secret = $password | ConvertTo-SecureString -AsPlainText -Force
					$Credential = New-Object -TypeName "System.Management.Automation.PSCredential" -ArgumentList ${username}, ${secret}
				} Else {
					Throw "Need one of: -AuthHeaders or -Credential, or both -Username and -Password. Aborted."
				}
			}
			# transform credential to authheader format
			. \\rdputil1\e$\scripts\Functions\credlib.ps1
			$AuthHeaders = Get-Basic-Base64-Auth-Headers -Credential $Credential
		}
		$Uri = "https://${Server}/service/rest/v1/components?repository=${Repository}"
		$boundary = [System.Guid]::NewGuid().ToString()
		$LF = "`r`n"
	}

	Process {
		If (!($FileName -ne $Null -And $FileName -ne "")) {
			# if filename is null, then just use the filename from the input file
			$FileName = Split-Path $File -Leaf
		}
		Try { $fileBin = [System.IO.File]::ReadAlltext($File) }
		Catch { Throw "Unable to read file $File. Aborted." }
		$bodyLines = (
			"--${boundary}",
			"Content-Disposition: form-data; name=`"raw.directory`"",
			"",
			"${Directory}",
			"--${boundary}",
			"Content-Disposition: form-data; name=`"raw.asset1`"; filename=`"${FileName}`"",
			"Content-Type: application/octet-stream",
			"",
			$fileBin,
			"",
			"--${boundary}",
			"Content-Disposition: form-data; name=`"raw.asset1.filename`"",
			"",
			"${FileName}",
			"--${boundary}--",
			""
		) -join $LF
		$Response = Invoke-WebRequest -Uri $Uri -Method "POST" -Headers $AuthHeaders -ContentType "multipart/form-data; boundary=`"$boundary`"" -Body $bodyLines
		If ($Response.StatusCode -ge 200 -And $Response.StatusCode -lt 300) {
			$Output = "https://${Server}/repository/${repository}/${directory}/${FileName}" -Replace "//","/"
			Write-Output $Output
		} Else {
			Write-Output $Response
		}
	}
}

References

Weblinks

  1. Components API | Sonatype Nexus documentation
  2. curl – How to send multipart/form-data with PowerShell Invoke-RestMethod – Stack Overflow

Alternate reading

  1. This page attempts to explain how to upload a file to Nexus, but it must be out of date or otherwise invalid. And the official Sonatype documentation even linked to it at one point somewhere! Uploading artifacts into Nexus via PowerShell – Mummy’s blog

credlib.ps1 Powershell library for credentials

My powershell library for some heavily-used credential tasks.

# Filename: \\rdputil1\scripts\Functions\credlib.ps1
# License: CC-BY-SA 4.0
# Author: bgstack15
# Startdate: 2019-12-31 15:53
# Title: Shared Credentials library
# Purpose: Provide some reusable credential functions
# History: 
# Usage:
# Reference:
#    https://foxdeploy.com/2017/01/13/adding-tab-completion-to-your-powershell-functions/
#    original implementation of storing crypted passwords \\rdputil1\e$\modules\EncryptModule\PassEncryptModule.psm1
# Improve:
# Dependencies:
#    \\rdputil1\e$\scripts\Functions\loglib.ps1

#. \\rdputil1\scripts\Functions\loglib.ps1

${global:SecurePath} = "\\rdputil1\e$\test"

Function Set-Master-Encryption-Key {
	Param(
		[Parameter(Mandatory=$true)][string]$String
	)
	$bytes = [System.Text.Encoding]::UTF8.GetBytes($String)
	If ($bytes.Length -ne 24) {
		Throw "String must be 24 characters long."
	}
	$bytes | Out-File "${global:SecurePath}\keys\EncryptionMaster.key"
}

Function Set-Shared-Password {
	Param(
		[Parameter(Mandatory=$true)][string]$Password,
		[Parameter(Mandatory=$true)][string]$PasswordCategory
	)
	$encryptedFile = "${global:SecurePath}\EncryptedData\$($PasswordCategory)"
	$key = Get-Content "${global:SecurePath}\keys\EncryptionMaster.key"
	$Password | ConvertTo-SecureString -AsPlainText -Force | ConvertFrom-SecureString -key $key | Out-File $encryptedFile
}

Function Get-Shared-Password {
	[CmdletBinding()]
	Param () # necessary to prevent error
	DynamicParam {
		$ParameterName = 'PasswordCategory'
		$RuntimeParameterDictionary = New-Object System.Management.Automation.RuntimeDefinedParameterDictionary
		$AttributeCollection = New-Object System.Collections.ObjectModel.Collection[System.Attribute]
		$ParameterAttribute = New-Object System.Management.Automation.ParameterAttribute
		$ParameterAttribute.Mandatory = $true
		#$ParameterAttribute.Position = 1 # unneccessary, but a good example
		$AttributeCollection.Add($ParameterAttribute)
		
		$arrSet = ( Get-ChildItem "$(${global:SecurePath})\EncryptedData" ).name
		$ValidateSetAttribute = New-Object System.Management.Automation.ValidateSetAttribute($arrSet)
		$AttributeCollection.Add($ValidateSetAttribute)
		
		$RuntimeParameter = New-Object System.Management.Automation.RuntimeDefinedParameter($ParameterName, [string], $AttributeCollection)
		$RuntimeParameterDictionary.Add($ParameterName, $RuntimeParameter)
		return $RuntimeParameterDictionary
	}

	Begin {	$PasswordCategory = $PsBoundParameters[$ParameterName] } # actual variable population based on input

	Process {
		$key = Get-Content "$(${global:SecurePath})\keys\EncryptionMaster.key"
		$passFile = Get-content (Get-ChildItem -Path "$(${global:SecurePath})\EncryptedData\$($PasswordCategory)") | ConvertTo-SecureString -Key $key
		if ($passFile) {
			return [Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($passFile))
		} else {
			return "ERROR: Unable to decrypt data"
		}
	}
}

# Get-Shared-Credential -User "serviceaccount@${global:domain}" -PasswordCategory "O365"
# this one wraps around the Get-Shared-Password
Function Get-Shared-Credential {
	[CmdletBinding()]
	Param (
		[Parameter(Mandatory=$true)][string] $User
	)
	DynamicParam {
		$ParameterName = 'PasswordCategory'
		$RuntimeParameterDictionary = New-Object System.Management.Automation.RuntimeDefinedParameterDictionary
		$AttributeCollection = New-Object System.Collections.ObjectModel.Collection[System.Attribute]
		$ParameterAttribute = New-Object System.Management.Automation.ParameterAttribute
		$ParameterAttribute.Mandatory = $true
		#$ParameterAttribute.Position = 1 # unneccessary, but a good example
		$AttributeCollection.Add($ParameterAttribute)
		
		$arrSet = ( Get-ChildItem "$(${global:SecurePath})\EncryptedData" ).name
		$ValidateSetAttribute = New-Object System.Management.Automation.ValidateSetAttribute($arrSet)
		$AttributeCollection.Add($ValidateSetAttribute)
		
		$RuntimeParameter = New-Object System.Management.Automation.RuntimeDefinedParameter($ParameterName, [string], $AttributeCollection)
		$RuntimeParameterDictionary.Add($ParameterName, $RuntimeParameter)
		return $RuntimeParameterDictionary
	}
	
	Begin {	$PasswordCategory = $PsBoundParameters[$ParameterName] } # actual variable population based on input

	Process {
		$password = Get-Shared-Password -PasswordCategory $PasswordCategory | ConvertTo-SecureString -AsPlainText -Force
		if ($pass -contains "ERROR") {
			If ("${global:today}" -ne "") {
				# if loglib has been loaded
				Log "ERROR: Unable to decrypt password";
				Log "ERROR: Exiting Script";
				EXIT
			} Else {
				Throw "Unable to decrypt password. Exiting script."
			}
		}
		$credential = New-Object -TypeName "System.Management.Automation.PSCredential" -ArgumentList $user, $password
		Return $credential
	}
}

Function Set-Shared-Credential {
	Param (
		[Parameter(Mandatory=$true)][string] $Password,
		[Parameter(Mandatory=$true)][string] $PasswordCategory
	)
	Set-Shared-Password -Password $Password -PasswordCategory $PasswordCategory
}

# Get-Basic-Base64-Auth-String returns "Basic 28931725917259725" which is the 
Function Get-Basic-Base64-Auth-String {
	Param (
		[Parameter(Mandatory=$true)] [System.Management.Automation.PSCredential] $Credential,
		[bool] $NoBasic = $false
	)
	$userauth = [System.Text.Encoding]::UTF8.GetBytes("$($Credential.Username):$( Get-Password-Insecurely -Credential $Credential )")
	$return = [System.Convert]::ToBase64String($userauth)
	if (!$NoBasic) { $return = "Basic $return" }
	Return "$return"
}

# Get-Password-Insecurely -Credential $Credential
# returns the password from it. Obviously this is insecure as yogurt.
Function Get-Password-Insecurely {
	Param (
		[Parameter(Mandatory=$true)] [System.Management.Automation.PSCredential] $Credential
	)
	Return [Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($Credential.Password))
}

# Get-Basic-Base64-Auth-Headers -Credential $Credential
# returns a hashtable with {"Authorization" = "Basic q192419284391724"}
Function Get-Basic-Base64-Auth-Headers {
	Param (
		[Parameter(Mandatory=$true)] [System.Management.Automation.PSCredential] $Credential
	)
	Return @{Authorization = $(Get-Basic-Base64-Auth-String -Credential $Credential) }
}

function Test-Cred {
	# Ripped straight from https://www.powershellbros.com/test-credentials-using-powershell-function/
    [CmdletBinding()]
    [OutputType([String])] 
       
    Param ( 
        [Parameter( 
            Mandatory = $false, 
            ValueFromPipeLine = $true, 
            ValueFromPipelineByPropertyName = $true
        )][Alias('PSCredential')][ValidateNotNull()][System.Management.Automation.PSCredential][System.Management.Automation.Credential()]$Credentials
    )
    $Domain = $null
    $Root = $null
    $Username = $null
    $Password = $null
      
    If($Credentials -eq $null) {
        Try { $Credentials = Get-Credential "domain\$env:username" -ErrorAction Stop ; }
        Catch {
            $ErrorMsg = $_.Exception.Message
            Write-Warning "Failed to validate credentials: $ErrorMsg "
            Pause
            Break
        }
    }
      
    # Checking module
    Try {
        # Split username and password
        $Username = $credentials.username
        $Password = $credentials.GetNetworkCredential().password
  
        # Get Domain
        $Root = "LDAP://" + ([ADSI]'').distinguishedName
        $Domain = New-Object System.DirectoryServices.DirectoryEntry($Root,$UserName,$Password)
    }
    Catch { $_.Exception.Message ; Continue ; }

    If(!$domain) { Write-Warning "Something went wrong" }
    Else {
        If ($domain.name -ne $null)
			 { return "Authenticated" }
        Else { return "Not authenticated" }
    }
}

The “Shared” functions depend on a master encryption key. You really only need the Set-Master-Encryption key once. From then on, you use Get/Set-Shared-Password, and Get/Set-Shared-Credential.

The Set-Shared-Password stores an encrypted string that is retrievable with the master encryption key. The Get-Shared-Credential builds a PSCredential object.

Some of these functions contain the DynamicParam components necessary for autocompletion. This is a nice feature for when you have a large amount of credentials stored and you want to see what’s available.

Lots of functions depend on Get-Basic-Base64-Auth-Headers, which returns a hashtable of “Authorization: Basic 129834712743192742” needed for many http calls.

The Test-Cred function is ripped off straight from Test credentials using PowerShell function – Powershellbros.com and coincidentally shows a much more convenient way to take a password out of a PSCredential object!

Many of my other Powershell libraries depend on credlib, including:
Powershell library for Bitbucket Cloud and Server

Powershell library for Bitbucket Cloud and Server

Here is a small library I wrote to help my Bitbucket Cloud to Bitbucket Server migration. I have no sharable notes about what order to use these functions in because I wrote them as I needed them or needed to alter them. I was also building on my credlib.ps1 which is not shared yet, but will come someday. You can also see this script in my gitlab: https://gitlab.com/bgstack15/former-gists/-/blob/master/bitbucketlib.ps1/bitbucketlib.ps1

# Startdate: 2020-05-18
# Title: Library for Bitbucket functions

# References:
#    https://developer.atlassian.com/bitbucket/api/2/reference/meta/authentication
#    https://developer.atlassian.com/server/bitbucket/how-tos/command-line-rest/
#    https://github.com/AtlassianPS/BitbucketPS
#    https://stackoverflow.com/questions/9825060/powershell-filtering-for-unique-values/9825218#9825218

# Useful starting values in procedural function at end.

# call: $CloudHeaders = Get-Bitbucket-Auth-Headers
Function Get-Bitbucket-Auth-Headers {
	<# .Synopsis Either username and password, or credential are required. #>
	param(
		[Parameter(Mandatory = $false)][string]$Username,
		[Parameter(Mandatory = $false)][string]$Password,
		[Parameter(Mandatory = $false)][System.Management.Automation.PSCredential]$Credential
	)
	
	. '\\rdputils1\e$\scripts\Functions\credlib.ps1'
	$internal_continue=$True
	$Headers = @{}
	
	if ($Username -ne $null -And $Password -ne $Null -And $Username -ne "" -And $Password -ne "") {
		$secret = $password | ConvertTo-SecureString -AsPlainText -Force
		$Credential = New-Object -TypeName "System.Management.Automation.PSCredential" -ArgumentList ${username}, ${secret}
	} elseif ($Credential -ne $null) {
		# pass through safely
	} else {
		if ($Username.tolower() -eq "prompt") {
			$Credential = Get-Credential
		} else {
			Write-Error "Provide -Username and -Password, or -Credential."
			$internal_continue = $False
			break
		}
	}
	
	if ($internal_continue) {
		$Headers += Get-Basic-Base64-Auth-Headers -Credential $credential
		$Headers
	}
}

Function _Iterate-Values-From-BB-API {
	[CmdletBinding()]
	param(
		[Parameter(Mandatory = $False)][Hashtable]$AuthHeaders = ${global:CloudHeaders},
		[Parameter(Mandatory = $True )][string]$StartUri,
		[Parameter(Mandatory = $False)][string]$Location = "cloud", # "cloud" or "local" which sets page defaults
		[Parameter(Mandatory = $False)][string]$Descriptor = "item",
		[Parameter(Mandatory = $False)][boolean]$SkipCount = $False
	)
	# StartUri could be any of these, for example:
	# https://api.bitbucket.org/2.0/repositories/${org}
	# https://api.bitbucket.org/2.0/workspaces/${org}/projects
	# https://git.example.com/rest/api/1.0/repos
	# https://git.example.com/rest/api/1.0/projects
	# https://api.bitbucket.org/2.0/teams/${org}/permissions/repositories?q=permission%3D%22write%22
	Begin {
		$done = $False
		$output = @()
		$size = -1
		$count = 0
		$Baseuri = $StartUri
		$keepcounting = $false
		
		# All this just to get object count for the local Bitbucket, because its API does not list total object count on each return page.
		if (!($SkipCount)) {
			if ($Location -like "*local*" ) {
				$pageSize = 250
				$CountUri = "$($Baseuri)?limit=$($pageSize)"
				$localcount = 0
				While (!$done) {
					Write-Progress -Activity "Counting objects" -Status "$($localcount)"
					$interim = Invoke-RestMethod $CountUri -Method "GET" -Headers $AuthHeaders
					if ( !($interim.isLastPage) ) {
						$localcount += $pageSize
						$CountUri = "$($Baseuri)?limit=$($pageSize)&start=$($localcount)"
					} else {
						$done = $True
						$localcount += $interim.size
					}
				}
				$done = $False
				$size = $localcount
			} Elseif ($Location -like "*cloud*" ) {
				$CountUri = "$($Baseuri)"
				$cloudcount = 0
				While (!$done) {
					Write-Progress -Activity "Counting $($Descriptor)s" -Status "$($cloudcount)"
					$interim = Invoke-RestMethod $CountUri -Method "GET" -Headers $AuthHeaders
					# short circuit if size is provided
					If ($interim.size -ne $null) {
						$cloudcount = $interim.size
						$done = $True
					} Elseif ($interim.next -eq $null) {
						$cloudcount += $interim.pagelen
						$done = $True
					} Else {
						$cloudcount += $interim.pagelen
						$Counturi = $interim.next
					}
				}
				$done = $False
				$size = $cloudcount
			}
		} Else {
			# skip the count!
			$size = 10
		}
	}

	Process {
		Write-Verbose "Will look for $($size) $($descriptor)s"
		$Uri = $StartUri
		While (!$done) {
			$interim = Invoke-RestMethod -Uri $Uri -Method "GET" -Headers $AuthHeaders
			if (!($SkipCount) -And $size -eq -1) { # only run once because it will always be the same
				if ($interim.size -ne $null) { $size = $interim.size }
					Else { $keepcounting = $True; $size += $interim.values.count }
			}
			if ($keepcounting) { $size += $interim.values.count }
			$interim.values | % {
				$output += $_ ;
				$count += 1 ;
				[int]$percent = ($count/$size)*100
				$percent = (@($percent, 100) | Measure -Minimum).Minimum
				$percent = (@($percent, 0) | Measure -Maximum).Maximum
				Write-Progress -Activity "Listing $($descriptor)" -Status "$count/$size" -PercentComplete $percent
			}
			# Bitbucket Cloud uses property "next" but on-prem server uses "nextPageStart"
			If ( $interim.next -ne $Null ) {
				$Uri = $interim.next
			} Elseif ( $interim.nextPageStart -ne $Null -And (!($interim.isLastPage)) ) {
				$Uri = "$($Baseuri)?start=$($interim.nextPageStart)"
			} else { $done = $True }
		}
	}

	End { $output }
}

Function List-All-Cloud-Projects {
	[CmdletBinding()]
	param(
		[Parameter(Mandatory = $True )][Hashtable]$CloudHeaders,
		[Parameter(Mandatory = $False)][string]$Server = "api.bitbucket.org",
		[Parameter(Mandatory = $False)][string]$Instance = "example"
	)
	_Iterate-Values-From-BB-API -AuthHeaders $CloudHeaders -StartUri "https://${Server}/2.0/workspaces/${Instance}/projects" -Descriptor "project"
}

Function List-All-Cloud-Repos {
	[CmdletBinding()]
	param(
		[Parameter(Mandatory = $True )][Hashtable]$CloudHeaders,
		[Parameter(Mandatory = $False)][string]$Server = "api.bitbucket.org",
		[Parameter(Mandatory = $False)][string]$Instance = "example"
	)
	_Iterate-Values-From-BB-API -AuthHeaders $CloudHeaders -StartUri "https://${Server}/2.0/repositories/${Instance}" -Descriptor "repo"
}

Function List-All-Local-Projects {
	[CmdletBinding()]
	param(
		[Parameter(Mandatory = $True )][Hashtable]$LocalHeaders,
		[Parameter(Mandatory = $False)][string]$Server = "gittest.example.com"
	)
	_Iterate-Values-From-BB-API -AuthHeaders $LocalHeaders -StartUri "https://${Server}/rest/api/1.0/projects" -Descriptor "project" -Location "local"
}

Function List-All-Local-Repos {
	[CmdletBinding()]
	param(
		[Parameter(Mandatory = $True )][Hashtable]$LocalHeaders,
		[Parameter(Mandatory = $False)][string]$Server = "gittest.example.com",
		[Parameter(Mandatory = $False)][string]$Instance = "example"
	)
	_Iterate-Values-From-BB-API -AuthHeaders $LocalHeaders -StartUri "https://${Server}/rest/api/1.0/repos" -Descriptor "repo" -Location "local"
}

# idea: $foo = List-All-Cloud-Repos -CloudHeaders $CloudHeaders
#       $out1 = $foo | Transform-Cloud-Repos-To-Useful
#       MapFile is CSV with columns Old,New to transform project names
#       Good mapfile is "U:\2020\05\project-cloud-to-onprem.csv"
Function Transform-Cloud-Repos-To-Useful {
	<# .Synopsis add relevant columns #>
	[CmdletBinding()]
	param(
		[Parameter(Mandatory = $False,ValueFromPipeLine = $true,ValueFromPipelineByPropertyName = $true)]$repo,
		[Parameter(Mandatory = $True)]$MapFile
	)
	Begin {
		$output = @() ;
		If (!(Test-Path $MapFile)) {
			Write-Error "MapFile must be a valid CSV file with columns named Old and New."
			break
		}
		$newDict = @{} ; Import-Csv $MapFile | % { $newDict[$_.Old]=$_.New }
	}
	
	Process {
		# a shallow copy is good enough for this object type. We do not have links to other objects in here, that we will care about or change or use.
		# https://stackoverflow.com/questions/9581568/how-to-create-new-clone-instance-of-psobject-object/13275775#13275775
		$newitem = $_.PsObject.Copy()
		$newitem | Add-Member -NotePropertyName projectkey -NotePropertyValue $_.project.key
		$newitem | Add-Member -NotePropertyName projectname -NotePropertyValue $_.project.name
		$newitem | Add-Member -NotePropertyName projecttype -NotePropertyValue $_.project.type
		$newitem | Add-Member -NotePropertyName NewProjectKey `
			-NotePropertyValue $newDict[$newitem.projectkey]
		$output += $newitem
	}
	End { $output ; }
}

# Useful for pulling columns from cloud repo definitions. Use after Transform-Repos-To-Useful
Function Select-Useful-Cloud-Repo-Columns {
	[CmdletBinding()]
	param(
		[Parameter(Mandatory = $False,ValueFromPipeLine = $true,ValueFromPipelineByPropertyName = $true)]$repo
	)
	
	Process { $repo | Select-Object updated_on,size,slug,description, `
		projectkey,projectname,NewProjectKey,fork_policy,uuid,full_name, `
		name,language,created_on,has_issues }

}

Function Select-Useful-Local-Repo-Columns {
	[CmdletBinding()]
	param(
		[Parameter(Mandatory = $False,ValueFromPipeLine = $true,ValueFromPipelineByPropertyName = $true)]$repo
	)
	
	Process { $repo | Select-Object slug,name,forkable,
		@{L='ProjectKey';E={($_.Project.key)};},
		@{L='ProjectName';E={($_.project.name)};},
		@{L='fullName';E={("$($_.project.key)/$($_.slug)")};}
		}
}

# low-level function for making a single repo with values
Function Act-Repo {
	[CmdletBinding()]
	param(
		[Parameter(Mandatory = $False)][string]$Server = "gittest.example.com",
		[Parameter(Mandatory = $True )][string]$Name,
		[Parameter(Mandatory = $True )][string]$ProjectSlug,
		[Parameter(Mandatory = $True )][string]$Action, # either Create or Delete
		[Parameter(Mandatory = $False)][Hashtable]$LocalHeaders = ${global:LocalHeaders}
	)
	if (!($Action -in @("Create","Delete"))) {
		Throw "Action must be either Create or Delete"
		$null
	}
	if ($Action -eq "Create") {
		$data = @{
			name = $Name
			scmID = "git"
		} | ConvertTo-Json -Depth 3
		$Uri = "https://$($Server)/rest/api/1.0/projects/$($ProjectSlug)/repos"
		$Method = "POST"
	} elseif ($Action -eq "Delete") {
		$Uri = "https://$($Server)/rest/api/1.0/projects/$($ProjectSlug)/repos/$($Name)"
		$Method = "DELETE"
	}
	try {
		Write-Host "Trying to $Action repo $ProjectSlug/$Name"
		$RestParams = @{
			Uri = $Uri
			Method = $Method
			Headers = $LocalHeaders
			ContentType = "application/json"
		}		
		if ($data -ne $Null) { $RestParams += @{ Body = $Data } }
		$response = Invoke-RestMethod @RestParams
	}
	<# catch [System.InvalidOperationException] { # we will get an invalid operation if the project already exists! Write-Warning "Repo $($ProjectSlug)/$($Name) already exists. Resuming..." Continue } #>
	catch {
		Write-Error $PSItem
	}
	$response
}

# Goal: create repositories based on CSV fields
# Act-All-Repos -MapFile "U:\2020\05\repos-testfull.csv" -Action "Create"
Function Act-All-Repos {
	[CmdletBinding()]
	param(
		[Parameter(Mandatory = $False)][string]$Server = "gittest.example.com",
		[Parameter(Mandatory = $True )][string]$MapFile,
		[Parameter(Mandatory = $True )][string]$Action, # either Create or Delete
		[Parameter(Mandatory = $False)][Hashtable]$LocalHeaders = ${global:LocalHeaders}
	)
	
	# DELETE is not implemented yet.
	Begin {
		If (!(Test-Path $MapFile)) {
			Write-Error "MapFile must be a valid CSV file with columns named slug,NewProjectKey."
			break
		}
		if (!($Action -in @("Create","Delete"))) {
			Throw "Action must be either Create or Delete"
		}
		$map = Import-Csv $MapFile
	}
	
	Process {
		# for every item in $map
		$map | % {
			Write-Host "Act-Repo -Name $($_.slug) -ProjectSlug $($_.NewProjectKey) -Action $Action -LocalHeaders LocalHeaders"
			Write-Progress "Trying to $Action repo $($_.NewProjectKey)/$($_.slug)"
			Act-Repo -Name "$($_.slug)" -ProjectSlug "$($_.NewProjectKey)" -Action $Action -LocalHeaders $LocalHeaders
		}
	}
	End {
	}
}

Function Act-Group {
	[CmdletBinding(SupportsShouldProcess)]
	param(
		[Parameter(Mandatory = $False)][string]$Server = "gittest.example.com",
		[Parameter(Mandatory = $True )][string]$Name,
		[Parameter(Mandatory = $True )][string]$Action, # either Create or Delete
		[Parameter(Mandatory = $False)][Hashtable]$LocalHeaders = ${global:LocalHeaders}
	)
	
	if (!($Action -in @("Create","Delete"))) {
		Throw "Action must be either Create or Delete"
	}

	$Uri = "https://${Server}/rest/api/1.0/admin/groups?name=${Name}"
	$Method = "POST" ; if ($Action -eq "Delete") { $Method = "DELETE" }
	Try {
		If ($PsCmdlet.ShouldProcess("$($Action) group $($Name)")) {
			$response = Invoke-RestMethod -Uri $Uri -Method $Method -Headers $LocalHeaders -ContentType "application/json" -Verbose:$False
		}
	} Catch {
		if ($_.Exception.Response.StatusCode.value__ -eq 409) {
			# just be silent when trying to create an extant group. 409: conflict.
			#Write-Warning "Group ""${name}"" already exists, continuing."
		} else {
			Write-Error $_
		}
	}
	# Bitbucket will return .name and .deletable if the object was successfully created.
}

Function Act-User-In-Group {
	[CmdletBinding(SupportsShouldProcess)]
	param(
		[Parameter(Mandatory = $False)][string]$Server = "gittest.example.com",
		[Parameter(Mandatory = $False)][Hashtable]$AuthHeaders = ${global:CloudHeaders},
		[Parameter(Mandatory = $True )][string]$GroupName,
		[Parameter(Mandatory = $True )][string]$UserName, # safer to pass in uuid when location=cloud
		[Parameter(Mandatory = $False)][string]$Instance = "example",
		[Parameter(Mandatory = $True )][string]$Action, # either Add or Remove
		[Parameter(Mandatory = $False)][string]$Location = "local" # "cloud" or "local"
	)
	Begin {
		If (!($Action -in @("Add","Remove"))) {
			Write-Error "Action must be in set @(""Add"",""Remove""). Aborted."
			break
		}
		If (!($Location -in @("local","cloud"))) {
			Write-Error "Location must be in set @(""local"",""cloud""). Aborted."
			break
		}
	}
	Process {
		If ($Action -eq "Add") {
			If ($Location -eq "local") {
				$data = @{
					user = $UserName
					groups = @(	$GroupName )
				} | ConvertTo-Json -Depth 5
				$Restparams = @{
					Uri = "https://${Server}/rest/api/1.0/admin/users/add-groups"
					Method = "POST"
				}
			} ElseIf ($Location -eq "cloud") {
				# Reference: https://confluence.atlassian.com/bitbucket/groups-endpoint-296093143.html
				# WORKS: Invoke-RestMethod -ContentType "application/json" -Uri "https://api.bitbucket.org/1.0/groups/example/devops/members/exampleteam/" -Body "{}" -method "PUT" -Headers $CloudHeaders
				$data = "{}"
				$Restparams = @{
					Uri = "https://${Server}/1.0/groups/${Instance}/${GroupName}/members/${UserName}/"
					Method = "PUT"
				}
			}
		} Elseif ($Action -eq "Remove") {
			If ($Location -eq "local") {
				$data = @{
					context = $UserName
					itemName = $GroupName
				} | ConvertTo-Json -Depth 5
				$Restparams = @{
					Uri = "https://${Server}/rest/api/1.0/admin/users/remove-group"
					Method = "POST"
				}
			} ElseIf ($Location -eq "cloud") {
				$data = "{}"
				$Restparams = @{
					Uri = "https://${Server}/1.0/groups/${Instance}/${GroupName}/members/${UserName}/"
					Method = "DELETE"
				}
			}
		}

		# Generic headers, and the invocation
		$Restparams += @{ ContentType = "application/json" }
		$Restparams += @{ Body = $data }
		$Restparams += @{ Headers = $AuthHeaders }
		#$Restparams
		#$Restparams.body
		#$response = Invoke-RestMethod -Method "POST" "https://${Server}/rest/api/1.0/admin/users/add-groups" -Body $data -ContentType "application/json" -Headers $AuthHeaders
		Try {
			#Write-Verbose "$($Action) user $($Username) to group $($GroupName)"
			#$Restparams.Uri
			# WORKHERE
			If ($PsCmdlet.ShouldProcess("$($Action) user $($UserName) to group $($GroupName)")) {
				$response = Invoke-RestMethod @Restparams -Verbose:$False
			}
		}
		Catch {
			If ($Action -eq "Remove" -And $_.Exception.Response.StatusCode.value__ -eq 404) {
				# just be silent when trying to delete a non-existent group.
				#Write-Warning "Group ""${name}"" already exists, continuing."
			} ElseIf ($Action -eq "Add" -And $_.Exception.Response.StatusCode.value__ -eq 409) {
				# be silent when trying to add a user to a group he is already in
			} Else {
				Write-Error $_
				Write-Error $data
			}
		}
	}
}

Function WritePermissionsCsv_Message {
	Write-Error "WritePermissionsCsv must be a valid CSV file with columns named Repo, Name. Probably use useful-write-permissions.csv from Build-Useful-Write-Permissions-List."
}

Function ReposCsv_Message {
	Write-Error "ReposCsv must be a valid CSV with columns slug, NewProjectKey. Probably use bbcloudrepos.csv for real migration, or gittest-importtest-repos.csv."
}

# This is the individual step for making a $REPONAME-write group
Function Act-RepoPermission-Group {
	[CmdletBinding(SupportsShouldProcess)]
	param(
		[Parameter(Mandatory = $False)][string]$Server = "gittest.example.com",
		[Parameter(Mandatory = $True )][string]$Name,
		[Parameter(Mandatory = $True )][string]$ProjectSlug,
		[Parameter(Mandatory = $True )][string]$Action, # either Grant or Revoke
		[Parameter(Mandatory = $False)][Hashtable]$LocalHeaders = ${global:LocalHeaders},
		[Parameter(Mandatory = $False)][boolean]$Populate = $false, # if True, then add users based on PermissionsCsv
		[Parameter(Mandatory = $False)][string]$WritePermissionsCsv #
	)
	# assumptions: repo already exists.
	
	Begin {
		If ($Populate) {
			Try {
				If (!(Test-Path $WritePermissionsCsv)) { WritePermissionsCsv_Message ; break ; }
				$WritePermissions = Import-Csv $WritePermissionsCsv
				$ColumnNames = $WritePermissions[0].PsObject.Properties.Name
				If (!("Repo" -in $ColumnNames) -Or !("Name" -In $ColumnNames)) { WritePermissionsCsv_Message ; break ; }
			}
			Catch { WritePermissionsCsv_Message ; break ; }
		}
		If (!($Action -in @("Grant","Revoke"))) {
			Throw "Action must be either Grant or Revoke"
		}
	}
	
	Process {
		$Uri = "https://$($Server)/rest/api/1.0/projects/$($ProjectSlug)/repos/$($Name)/permissions/groups"
		$WriteGroup = "$($Name)-write"
		$WritePermissionName = "REPO_WRITE"
		
		If ($Action -eq "Grant") {
			# flow: create group and grant it permission
			#If ($PsCmdlet.ShouldProcess("Create group $($WriteGroup)")) { Act-Group -Action "Create" -Name "$($WriteGroup)" -LocalHeaders $LocalHeaders -Server $Server ; }
			Act-Group -Action "Create" -Name "$($WriteGroup)" -LocalHeaders $LocalHeaders -Server $Server
			$Restparams = @{
				Uri = "$($Uri)?permission=$($WritePermissionName)&name=$($WriteGroup)"
				Method = "PUT"
				Headers = $LocalHeaders
				ContentType = "application/json"
			}
			#$RestParams.Uri
			If ($PsCmdlet.ShouldProcess("$($Action) group $($WriteGroup) write perms to repo $($Name)")) {
				Invoke-RestMethod @RestParams -Verbose:$False
			}
			# And now do the populate step.
			If ($Populate -And $PsCmdlet.ShouldProcess("Populate group $($WriteGroup)")) {
				# For every permission entry where Repo column = $Name parameter for this function,
				$countUsersAdded = 0
				$WritePermissions | ? { $_.Repo -eq "$($Name)" } | % {
					# add user to the group
					#Write-Host "Please add user $($_.name) to $WriteGroup"
					If ( "$($_.name)" -ne "") {
						Try {
							Act-User-In-Group -AuthHeaders $LocalHeaders -Server $Server -GroupName $WriteGroup -UserName $_.name -Action "Add" -Location "local" -WhatIf:$False -Confirm:$False
							$countUsersAdded += 1
						}
						Catch {
							Write-Error $PSItem
							$countUsersAdded -= 1
						}
					} Else { # samaccountname is blank
						Write-Warning "Unable to add user $_"
					}
				}
				Write-Verbose "Added $countUsersAdded to group ""$WriteGroup"""
			}
		} elseif ($Action -eq "Revoke") {
			# flow: remove permission to group and delete it
			$Restparams = @{
				Uri = "$($Uri)?name=$($WriteGroup)"
				Method = "DELETE"
				Headers = $LocalHeaders
				ContentType = "application/json"
			}
			If ($PsCmdlet.ShouldProcess("$($Action) group $($WriteGroup) write perms to repo $($Name)")) { Invoke-RestMethod @RestParams -Verbose:$False ; }
			Act-Group -Action "Delete" -Name "$($WriteGroup)" -LocalHeaders $LocalHeaders -Server $Server
		}
	}
}

# goal: given the input file $WritePermissionsCsv, call Act-RepoPermission-Group for each repo.
Function Act-All-RepoPermission-Groups {
	[CmdletBinding(SupportsShouldProcess)]
	param(
		[Parameter(Mandatory = $False)][string]$Server = "gittest.example.com",
		[Parameter(Mandatory = $True )][string]$Action, # either Grant or Revoke
		[Parameter(Mandatory = $False)][Hashtable]$LocalHeaders = ${global:LocalHeaders},
		[Parameter(Mandatory = $False)][boolean]$Populate = $false, # if True, then add users based on PermissionsCsv
		[Parameter(Mandatory = $False)][string]$WritePermissionsCsv,
		[Parameter(Mandatory = $True )][string]$ReposCsv # columns slug, NewProjectKey, probably gittest-importest-repos.csv or bbcloudrepos.csv
	)
	Begin {
		Try {
			If (!(Test-Path $WritePermissionsCsv)) { WritePermissionsCsv_Message ; break ; }
			$WritePermissions = Import-Csv $WritePermissionsCsv
			$ColumnNames = $WritePermissions[0].PsObject.Properties.Name
			If (!("Repo" -in $ColumnNames) -Or !("Name" -In $ColumnNames)) { WritePermissionsCsv_Message ; break ; }
		}
		Catch { WritePermissionsCsv_Message ; break ; }

		Try {
			If (!(Test-Path $ReposCsv)) { ReposCsv_Message ; break ; }
			$Repos = Import-Csv $ReposCsv
			$ColumnNames = $Repos[0].PsObject.Properties.Name
			If (!("slug" -in $ColumnNames) -Or !("NewProjectKey" -In $ColumnNames)) { ReposCsv_Message ; break ; }
		}
		Catch { ReposCsv_Message ; break ; }

		If (!($Action -in @("Grant","Revoke"))) {
			Throw "Action must be either Grant or Revoke"
		}
		$count = 0
		$size = ($Repos | Group-Object { $_.Repo }).count
	}

	Process {
		$Repos | % {
			# for each repository listing
			Try {
				Act-RepoPermission-Group -Name $_.slug -ProjectSlug $_.NewProjectKey -Action $Action -LocalHeaders $Localheaders -Server $Server -Populate $Populate -WritePermissionsCsv $WritePermissionsCsv
				$count += 1 ;
				[int]$percent = ($count/$size)*100
				$percent = (@($percent, 100) | Measure -Minimum).Minimum
				$percent = (@($percent, 0) | Measure -Maximum).Maximum
				Write-Progress -Activity "$($Action) for repo" -Status "$count/$size" -PercentComplete $percent

			}
			Catch {
				$RepoCount -= 1
			}
		}
	}
}

# List users who have write permission on this one repo.
Function List-Write-Users-For-Cloud-Repo {
	[CmdletBinding()]
	param(
		[Parameter(Mandatory = $True )][Hashtable]$CloudHeaders,
		[Parameter(Mandatory = $False)][string]$Server = "api.bitbucket.org",
		[Parameter(Mandatory = $False)][string]$Instance = "example",
		[Parameter(Mandatory = $True )][string]$Name,
		[Parameter(Mandatory = $False)][bool]$SkipCount = $False,
		[Parameter(Mandatory = $False)][string]$Property = "uuid"
	)
	$interim = _Iterate-Values-From-BB-API -StartUri "https://${Server}/2.0/teams/${Instance}/permissions/repositories/${Name}?q=permission%3D%22write%22" -Location "cloud" -SkipCount $SkipCount -Descriptor "for repo ""$Name"" write permission"
	$output = @()
	$interim | % {
		<# # In case I want to have a multi-field object in the array. $output += @{ username = $_.user.display_name repo = $_.repository.full_name }#>
		#$output += $_.user.display_name
		$output += $_.user.$($Property)
	}
	$output
}

# Return array of @{ Repo = "example/foobar", User = "bgstack15"} objects
# Usage:
#    $cloudRepos = List-All-Cloud-Repos -CloudHeaders $CloudHeaders -Verbose
#    $allWritePermissions = $cloudRepos | List-Write-Users-For-All-Cloud-Repos -CloudHeaders $CloudHeaders
#    $allWritePermissions | Export-Csv U:\2002\05\all-write-permissions.csv -NoTypeInformation
# Next steps:
#    use a function that makes ${Repo}-write group and adds the users
Function List-Write-Users-For-All-Cloud-Repos {
	[CmdletBinding()]
	param(
		[Parameter(Mandatory = $True )][Hashtable]$CloudHeaders,
		[Parameter(Mandatory = $False)][string]$Server = "api.bitbucket.org",
		[Parameter(Mandatory = $False)][string]$Instance = "example",
		[Parameter(Mandatory = $False,ValueFromPipeLine = $true,ValueFromPipelineByPropertyName = $true)]$Repos
	)
	Begin {
		# list all repos
		#$Repos.count
		$repoCount = 0

		#if ($Repos -eq $Null) { $Repos = List-All-Cloud-Repos -CloudHeaders $CloudHeaders }
	}
	Process {
		# iterate through all repos
		#$repoCount = 0
		$_ | % {
			$repoCount += 1
			#$_.name
			$thisRepo = $_.name
			$interim = List-Write-Users-For-Cloud-Repo -CloudHeaders $CloudHeaders -Server $Server -Instance $Instance -Name $_.name -SkipCount $True
			Write-Host "Working on repo #$repoCount ""$thisRepo"""
			#$interim[3]
			$new = @()
			ForEach ($item in $interim) {
				#[psCustomObject]$newitem = $item.PsObject.Copy() # shallow copy is good enough
				$newItem = New-Object -TypeName psobject
				$newItem | Add-Member -MemberType NoteProperty -Name "Repo" -Value $thisRepo
				$newItem | Add-Member -MemberType NoteProperty -Name "User" -Value $item
				$new += $newitem
			}
			$new
		}
	}
}

# take a list of user ids from pipeline and convert to usable property
# I don't know where I was going with this.
Function Convert-User-To-Property-STUB {
	[CmdletBinding()]
	param(
		[Parameter(Mandatory = $True )][Hashtable]$CloudHeaders,
		[Parameter(Mandatory = $False)][string]$Server = "api.bitbucket.org",
		[Parameter(Mandatory = $False)][string]$Instance = "example",
		[Parameter(Mandatory = $False,ValueFromPipeLine = $true,ValueFromPipelineByPropertyName = $true)]$UserIds
	)
	Begin {
		$userCount = 0
		Write-Error "STUB! I cannot remember where I was going with this but it looks cool."
	}

	Process {
		$_ | %{ 
			$userCount += 1
			
		}
	}

}

Function List-Groups {
	[CmdletBinding()]
	param(
		[Parameter(Mandatory = $False)][string]$Server = "gittest.example.com",
		[Parameter(Mandatory = $False)][Hashtable]$AuthHeaders = ${global:CloudHeaders},
		[Parameter(Mandatory = $False)][string]$Instance = "example",
		[Parameter(Mandatory = $False)][string]$Location = "local", # "cloud" or "local"
		[Parameter(Mandatory = $False)][boolean]$SkipCount = $False
	)
	Begin {
		If (!($Location -in @("local","cloud"))) {
			Write-Error "Location must be in set @(""local"",""cloud""). Aborted."
			break
		}
	}
	Process {
		If ($Location -eq "cloud") {
			# we can cheat here because I know there are only 18 groups defined in bitbucket cloud, so no concerns about pagination. Plus this api endpoint is very poorly documented because it's a deprecated one despite the fact the 2.0 groups endpoint does not exist; so pagination methodology is undetermined.
			$Uri = "https://${Server}/1.0/groups/${Instance}/"
			Invoke-RestMethod -ContentType "application/json" -Uri $Uri -method "GET" -Headers $AuthHeaders
		} ElseIf ($Location -eq "local") {
			$Uri = "https://${Server}/rest/api/1.0/admin/groups"
			_Iterate-Values-From-BB-API -AuthHeaders $AuthHeaders -StartUri $Uri -Location $Location -Descriptor $Group -SkipCount $SkipCount

		}
	}
}

# Usage:
#    $cloudgroups = List-Groups -Location "cloud" -Server "api.bitbucket.org"
#    $cloudmemberships = $cloudgroups | List-Memberships-From-Cloud-Groups
#    $cloudmemberships | Export-Csv -NoTypeInformation "U:\2020\05\cloud-memberships.csv"
Function List-Memberships-From-Cloud-Groups {
	# $cloudgroups = List-Groups -Location "cloud" -Server "api.bitbucket.org"
	# $cloudmemberships = $cloudgroups | Select-Object name,members,slug | Select-Object -Property slug,name -ExpandProperty members | Select-Object -Property display_name,account_id,uuid,nickname,@{l="groupname";e={$_.name}},@{l="groupslug";e={$_.slug}}
	[CmdletBinding()]
	Param(
		[Parameter(Mandatory = $False,ValueFromPipeLine = $true,ValueFromPipelineByPropertyName = $true)]$groups
	)
	Begin {
	}
	Process {
		$_ | Select-Object name,members,slug | Select-Object -Property name,slug -ExpandProperty members -Erroraction "silentlycontinue" | Select-Object -Property display_name,account_id,uuid,nickname,@{l="groupname";e={$_.name}},@{l="groupslug";e={$_.slug}}
	}
}

# goal: take $cloudmemberships (with attributes uuid, groupname) and take $Action on them
Function Act-All-Cloud-Memberships-From-List {
	[CmdletBinding()]
	param(
		[Parameter(Mandatory = $False)][string]$Server = "api.bitbucket.org",
		[Parameter(Mandatory = $False)][Hashtable]$AuthHeaders = ${global:CloudHeaders},
		[Parameter(Mandatory = $False)][string]$Instance = "example",
		[Parameter(Mandatory = $True )][string]$Action, # either Add or Remove
		[Parameter(Mandatory = $False)][string]$Location = "cloud", # "cloud" or "local"
		[Parameter(Mandatory = $False,ValueFromPipeLine = $true)]$MembershipsPipeLine,
		[Parameter(Mandatory = $False)][string]$MembershipsCsv,
		[Parameter(Mandatory = $False)][PSObject]$Memberships,
		[Parameter(Mandatory = $False)][string]$GroupMatchProperty = "groupname"
		# WORKHERE
	)
	Begin {
		If (!($Action -in @("Add","Remove"))) {
			Write-Error "Action must be in set @(""Add"",""Remove""). Aborted."
			break
		}
		If (!($Location -in @("local","cloud"))) {
			Write-Error "Location must be in set @(""local"",""cloud""). Aborted."
			break
		}
		If (!($GroupMatchProperty -in @("groupname","groupslug"))) {
			Write-Error "GroupMatchProperty must be in set @(""groupname"",""groupslug""). Aborted."
			break
		}
	}
	Process {
		If ($MembershipsPipeLine -eq $Null) {
			If ($Memberships -eq $Null) {
				If ("$($MembershipsCsv)" -eq "" -Or !(Test-Path $MembershipsCsv)) {
					Write-Error "Either pipe in or provide -Memberships with uuid,display_name,groupname values or provide MembershipsCsv file with such-named columns. Aborted."
					break
				}
				Write-Warning "using csv"
				$Memberships = Import-Csv $MembershipsCsv
			} Else {
				Write-Warning "using memberships var directly"
			}
		} Else {
			#Write-Warning "using pipeline"
			$Memberships = $MembershipsPipeLine
		}
		$Memberships | ? { $_.groupname -ne "Administrators" } | % { 
			Write-Verbose "$($Action) user $($_.display_name) ( uuid $($_.uuid) ) to group $($_.$($GroupMatchProperty))"
			Act-User-In-Group -Server $Server -AuthHeaders $AuthHeaders -GroupName "$($_.$($GroupMatchProperty))" -UserName $_.uuid -Instance $Instance -Action $Action -Location $Location
		}
	}
}

# Goal: take Repo,User list and the display_name,email list and output a repo,samaccountname,user,email table.
# The NamesToEmailsCsv is a manually curated list that is boiled down from all the Users listed from the WritePermissionsCsv ( | sort | uniq ) with email address for each user.
# Usage:
#    $list = Build-Useful-Write-Permissions-List -WritePermissionsCsv "U:\2020\05\all-write-permissions.csv" -NamesToEmailsCsv "U:\2020\05\bitbucket-emails.csv"
#    $list | Export-Csv -NoTypeInformation "U:\2020\05\useful-write-permissions.csv"
Function Build-Useful-Write-Permissions-List {
	[CmdletBinding()]
	param(
		[Parameter(Mandatory = $True )][string]$WritePermissionsCsv,
		[Parameter(Mandatory = $True )][string]$NamesToEmailsCsv,
		[Parameter(Mandatory = $False)][string]$DefaultDomain = "example.com"
	)
	Begin {
		If (!(Test-Path $WritePermissionsCsv)) {
			Write-Error "WritePermissionsCsv must be a valid CSV file with columns named Repo, User."
			break
		}
		If (!(Test-Path $NamesToEmailsCsv)) {
			Write-Error "NamesToEmailsCsv must be a valid CSV file with columns named display_name,email."
			break
		}
		$Output = @()
		$count = 0
		$WritePermissions = Import-Csv $WritePermissionsCsv
		$NamesToEmails = Import-Csv $NamesToEmailsCsv
		$size = $WritePermissions.count
	}

	Process {
		ForEach ($wpitem in $WritePermissions) {
			if (!($wpitem.User -In $NamesToEmails.display_name)) {
				# just be silent
				#Write-Error "Cannot find $($wpitem.User) in NamesToEmails"
			} Else {
				$count += 1 ;
				[int]$percent = ($count/$size)*100
				$percent = (@($percent, 100) | Measure -Minimum).Minimum
				$percent = (@($percent, 0) | Measure -Maximum).Maximum
				Write-Progress -Activity "Listing perm" -Status "$count/$size" -PercentComplete $percent

				$NamesToEmails | % {
					$NameToEmailitem = $_
					if ($NameToEmailItem.display_name -eq $wpitem.User) {
						If ("$($NameToEmailItem.email)" -ne "") {
							$thisADUser = Get-ADUser -Filter "mail -eq '$($NameToEmailItem.email)'" -Properties mail
							if (!($thisADUser -ne $null)) {
								# if it is still empty, treat the stored email as a upn
								$thisADUser = Get-ADUser -Filter "userprincipalname -eq '$($NameToEmailItem.email)'" -Properties mail
							}
							if (!($thisADUser -ne $null)) {
								# if still empty, try it as sammaccountname@domain
								$matchString = $($NameToEmailItem.email.split("@")[0])
								$thisADUser = Get-ADUser -Filter "samaccountname -eq '$matchString'" -Properties mail
							}
							if (!($thisADUser -ne $null)) {
								Write-Error "Unable to match $($NameToEmailItem.email) to an AD user!"
							} else {
								$newItem = New-Object -TypeName psobject
								$newItem | Add-Member -MemberType NoteProperty -Name "Repo" -Value $wpitem.Repo
								$newItem | Add-Member -MemberType NoteProperty -Name "Displayname" -Value $wpitem.User
								$newItem | Add-Member -MemberType NoteProperty -Name "Mail" -Value $thisADUser.mail
								$newItem | Add-Member -MemberType NoteProperty -Name "Name" -Value $thisADUser.samaccountname
								$Output += $newItem
								Write-Verbose $newItem
							}
						} Else { # so if there is no email address listed for this user
							# well apparently it lists the non-email-mapped ones a lot, so just be quiet.
							#Write-Error "No email for user $($NameToEmailItem.display_name)"
						}
					} 
				}
			}
		}
	}
	End { $Output ; }
}

# Procedural section
if ($LoadVars) {
	. '\\rdputils1\e$\scripts\Functions\credlib.ps1'
	$cloudcred = Get-Shared-Credential -User "exampleteam" -PasswordCategory exampleteam
	$CloudHeaders = Get-Bitbucket-Auth-Headers -Credential $cloudcred
	$bbprodcred = Get-Shared-Credential -User "serviceaccount"-PasswordCategory "serviceaccount"
	$bbprodHeaders = Get-Bitbucket-Auth-Headers -Credential $bbprodcred
	$localcred = $bbprodcred
	$localHeaders = Get-Bitbucket-Auth-headers -Credential $localcred
	Remove-Variable LoadVars
}

Some thoughts that might guide users

The auth-headers functions depend on the aforementioned credlib library which is not here, but the Get-Basic-Base64-Auth-Headers function basically returns a hashtable of Authorization: Basic aabbbxx== which is just the base64 encoding of the “username:password” syntax. I should document it better, but I’ll save it for the credlib post.

Some of the best work is in the “private” function _Iterate-Values-From-BB-API. It attempts to count the number of objects being counted, and display the percentage, unless -SkipCount $True. This one function is currently written for Bitbucket Cloud as well as Bitbucket Server 5.16.0 and above. The pagination across API versions is a little different, so the function is a little complex in how it collects the values.

If a function uses -LocalHeaders, then I tested it only against the Bitbucket Server, and the same for -CloudHeaders for Bitbucket Cloud. I used -AuthHeaders when a function was tested against both.

The functions tend to get fancier towards the bottom, including even from-pipeline values or by parameter, or from -CsvFile options!

Get email addresses for Bitbucket Cloud users

For some convoluted logic, the admins of a paid space in Bitbucket Cloud cannot programmatically retrieve user email addresses. This is ridiculous because one adds new users via email address!

If your organization also uses Jira Cloud, there is a way to mash the data together and retrieve some email addresses for Bitbucket users. The user uuids are not always the same across the products, but this will give your list a good head start.

# correlating Bitbucket accounts with email addresses
# References:
#    https://community.atlassian.com/t5/Bitbucket-questions/How-to-get-user-email-address-using-BitBucket-API/qaq-p/123783#M54771

# Bitbucket creds
$org = "example"
$username = "bgstack15@example.com"
$password = "ASTERISKS"
$CloudHeaders = @{ Authorization = "Basic " + [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("$($username):$($password)")) }

# jira creds
$jiraCloudHeaders = @{ Authorization = "Basic " + [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("$($username):$($password)")) }

# Get jira users
$jira0 = Invoke-RestMethod -Headers $jiraCloudHeaders -Uri "https://example.atlassian.net/rest/api/3/users/search?startAt=0&maxResults=1000"
$jira1 = Invoke-RestMethod -Headers $jiraCloudHeaders -Uri "https://example.atlassian.net/rest/api/3/users/search?startAt=1000&maxResults=1000"
$jira2 = Invoke-RestMethod -Headers $jiraCloudHeaders -Uri "https://example.atlassian.net/rest/api/3/users/search?startAt=2000&maxResults=1000"
$jira3 = Invoke-RestMethod -Headers $jiraCloudHeaders -Uri "https://example.atlassian.net/rest/api/3/users/search?startAt=3000&maxResults=1000"
$jira = $jira0 + $jira1 +$jira2 + $jira3
$jiranew = $jira | Group-Object accountId -AsHashTable

# Get bitbucket users
$bitbucket = Iterate-Values-From-BB-API -StartUri "https://api.bitbucket.org/2.0/workspaces/${org}/members" -AuthHeaders $CloudHeaders -SkipCount $True

# Combine them.
$bitbucket_expanded = $bitbucket | select-object -ExpandProperty user | Select-Object display_name, nickname, account_id
$bitbucket_expanded | Select-Object display_name, @{n='email';e={$jiranew[$_.account_id][0].emailAddress }} | Export-Csv -NoTypeInformation "bitbucket-emails.csv"

So you probably noticed the Iterate-Value-From-BB-API function. I wrote this myself, because I use it in a soon-to-be-published Powershell library for interacting with Bitbucket Cloud and Server instances.

Function _Iterate-Values-From-BB-API {
	[CmdletBinding()]
	param(
		[Parameter(Mandatory = $False)][Hashtable]$AuthHeaders = ${global:CloudHeaders},
		[Parameter(Mandatory = $True )][string]$StartUri,
		[Parameter(Mandatory = $False)][string]$Location = "cloud", # "cloud" or "local" which sets page defaults
		[Parameter(Mandatory = $False)][string]$Descriptor = "item",
		[Parameter(Mandatory = $False)][boolean]$SkipCount = $False
	)
	# StartUri could be any of these, for example:
	# https://api.bitbucket.org/2.0/repositories/${org}
	# https://api.bitbucket.org/2.0/workspaces/${org}/projects
	# https://git.example.com/rest/api/1.0/repos
	# https://git.example.com/rest/api/1.0/projects
	# https://api.bitbucket.org/2.0/teams/${org}/permissions/repositories?q=permission%3D%22write%22
	Begin {
		$done = $False
		$output = @()
		$size = -1
		$count = 0
		$Baseuri = $StartUri
		$keepcounting = $false

		# All this just to get object count for the local Bitbucket, because its API does not list total object count on each return page.
		if (!($SkipCount)) {
			if ($Location -like "*local*" ) {
				$pageSize = 250
				$CountUri = "$($Baseuri)?limit=$($pageSize)"
				$localcount = 0
				While (!$done) {
					Write-Progress -Activity "Counting objects" -Status "$($localcount)"
					$interim = Invoke-RestMethod $CountUri -Method "GET" -Headers $AuthHeaders
					if ( !($interim.isLastPage) ) {
						$localcount += $pageSize
						$CountUri = "$($Baseuri)?limit=$($pageSize)&start=$($localcount)"
					} else {
						$done = $True
						$localcount += $interim.size
					}
				}
				$done = $False
				$size = $localcount
			} Elseif ($Location -like "*cloud*" ) {
				$CountUri = "$($Baseuri)"
				$cloudcount = 0
				While (!$done) {
					Write-Progress -Activity "Counting $($Descriptor)s" -Status "$($cloudcount)"
					$interim = Invoke-RestMethod $CountUri -Method "GET" -Headers $AuthHeaders
					# short circuit if size is provided
					If ($interim.size -ne $null) {
						$cloudcount = $interim.size
						$done = $True
					} Elseif ($interim.next -eq $null) {
						$cloudcount += $interim.pagelen
						$done = $True
					} Else {
						$cloudcount += $interim.pagelen
						$Counturi = $interim.next
					}
				}
				$done = $False
				$size = $cloudcount
			}
		} Else {
			# skip the count!
			$size = 10
		}
	}

	Process {
		Write-Verbose "Will look for $($size) $($descriptor)s"
		$Uri = $StartUri
		While (!$done) {
			$interim = Invoke-RestMethod -Uri $Uri -Method "GET" -Headers $AuthHeaders
			if (!($SkipCount) -And $size -eq -1) { # only run once because it will always be the same
				if ($interim.size -ne $null) { $size = $interim.size }
					Else { $keepcounting = $True; $size += $interim.values.count }
			}
			if ($keepcounting) { $size += $interim.values.count }
			$interim.values | % {
				$output += $_ ;
				$count += 1 ;
				[int]$percent = ($count/$size)*100
				$percent = (@($percent, 100) | Measure -Minimum).Minimum
				$percent = (@($percent, 0) | Measure -Maximum).Maximum
				Write-Progress -Activity "Listing $($descriptor)" -Status "$count/$size" -PercentComplete $percent
			}
			# Bitbucket Cloud uses property "next" but on-prem server uses "nextPageStart"
			If ( $interim.next -ne $Null ) {
				$Uri = $interim.next
			} Elseif ( $interim.nextPageStart -ne $Null -And (!($interim.isLastPage)) ) {
				$Uri = "$($Baseuri)?start=$($interim.nextPageStart)"
			} else { $done = $True }
		}
	}

	End { $output }
}

References

Web links

Ripped off and enhanced from a user in the Atlassian community: How to get user email address using BitBucket API?

Load nfs-mounted ssh keys at login automatically

I use multiple ssh keys across multiple systems. Some systems need to have the same ssh key loaded.

My solution is to store the generic ones on my nfs mount, accessible only to my user, and to use a function in .bashrc:

load_ssh_key() {
   test -n "${1}" && SSHKEY="${1}" ;
   test -z "${SSHKEY}" && SSHKEY=/mnt/bgstack15/.ssh/bgstack15_devuan.key
   if test -e "${SSHKEY}" ;
   then
      test -z "${SSH_AGENT_PID}" && eval $( ssh-agent ) | grep -viE 'Agent pid'
      ssh-add "${SSHKEY}" 2>&1 | grep -viE "Identity added:" 1>&2
   else
      echo "Unable to get to private key!" 1>&2
   fi
   unset SSHKEY
}

load_ssh_key

Also, now, the function is generally available for invoking with a filename to load that ssh key. I realize ssh-add is pretty trivial, but I want the function to fail silently for when I’m off-network (when I won’t be doing any ssh work anyway).

Samba share with AD auth, 2020 May edition

Overview

I wrote about this topic almost 4 years ago: Samba share with AD authentication
This article is the updated version. It has a different environment and purpose, as well as a new version of samba that requires a workaround.
The goal today is just get a quick home directories share.

Prequisites

  • Server is joined to the domain
  • Working on CentOS 7. The previous article included Ubuntu commands for the package manager and firewall.

Setting up Samba

Install the packages, including the server package.

yum -y install samba

Open the firewall.

firewall-cmd --permanent --add-service=samba
systemctl restart firewalld.service

Configure Samba.

cat <<EOFSMB > /etc/samba/smb.conf
[global]
   workgroup = EXAMPLE
   security = ads
   realm = EXAMPLE.COM
   kerberos method = system keytab
   netbios name = $( hostname -s )
   server string = Description here
   log file = /var/log/samba/log.%m
   max log size = 50
   dns proxy = no
   encrypt passwords = yes
   passdb backend = tdbsam
   printcap name = /dev/null
   load printers = no

[homes]
   comment = Home Directories
   valid users = user1, user2, @group1
   browseable = No
   read only = No
   inherit acls = Yes
   guest only = no
EOFSMB

Starting with Samba 4.9.1, a workaround is needed for Samba to work when the id mapping is not set up thoroughly. This example does not do any id mapping, so use this quick and dirty fix.

net -s /dev/null groupmap add sid=S-1-5-32-546 unixgroup=nobody type=builtin

You can see the custom mapping for the guest user with:

$ net -s /dev/null groupmap list
nobody (S-1-5-32-546) -> nobody
Reference: 1648399 – Samba 4.9.1: smb.service fails with ERROR: failed to setup guest info (RHBZ)

And enable and start the services.

systemctl enable --now smb nmb

This command enables (sets to run at system startup) and starts immediately, these two services. NMB is the NetBIOS name server. It helps the main Samba daemon in ways deeper than I care to research.

Configuring SELinux

Set a few SE booleans.

for word in samba_export_all_rw samba_create_home_dirs ; do setsebool -P "${word}" 1 ; done

Share your browser prefs.js!

The advances users of Mozilla-based web browsers tend to have a large set of preferences. Let’s start a trend of sharing them!

I throw all my settings into one prefs.js that gets distributed through my various means to the system directory. That’s why you see pref() here instead of user_pref(). Not all of these options apply to each browser, but the extra ones do not hurt. So these could affect Waterfox, Palemoon, or Firefox web browsers.

// file: /usr/lib/waterfox/browser/defaults/preferences/bgstack15-waterfox-prefs.js
// last modified 2020-04-15
// reference:
//    https://support.mozilla.org/en-US/kb/customizing-firefox-using-autoconfig
// Turn off updates. I use my package manager for browser updates.
pref("app.update.auto",                     false);
pref("app.update.autoInstallEnabled",       false);
pref("app.update.enabled",                  false);
pref("extensions.update.autoUpdateDefault", false);
// Disable previews of tabs. I just do not like the feature
pref("browser.allTabs.previews", false);
pref("browser.ctrlTab.previews", false);
pref("browser.ctrlTab.recentlyUsedOrder", false);
// Old-style backspace action to navigate backwards through browsing history.
pref("browser.backspace_action", 0);
// Do not prompt for download location. Just use ~/Downloads.
pref("browser.download.useDownloadDir", true);
// Show blank page on a new tab.
pref("browser.newtab.choice", 1);
pref("browser.newtabpage.enabled", false);
pref("browser.newtabpage.storageVersion", 1);
// Do not suggest similar searches when typing into the bar. 
pref("browser.search.suggest.enabled", false);
// Disable these by choice.
pref("browser.safebrowsing.malware.enabled", false);
pref("browser.safebrowsing.phishing.enabled", false);
// Hide this search plugin.
pref("browser.search.hiddenOneOffs", "DuckDuckGo");
// Do not automatically check for updates to search plugins.
pref("browser.search.update", false);
// Do not show separate widget in navigation bar.
pref("browser.search.widget.inNavBar", false);
// Load all tabs when starting browser. I hate load-on-demand, which is when it loads only when you switch to that tab.
pref("browser.sessionstore.restore_on_demand", false);
// Do not check if this is the OS default browser.
pref("browser.shell.checkDefaultBrowser",   false);
// Set my home page.
pref("browser.startup.homepage",            "data:text/plain,browser.startup.homepage=https://start.duckduckgo.com/");
pref("browser.startup.page", 3);
// Closing the last tab does not close the browser window.
pref("browser.tabs.closeWindowWithLastTab", false);
// Obviously I did not type this one myself. This controls the layout of the buttons on the navigation bar.
pref("browser.uiCustomization.state", "{\"placements\":{\"PanelUI-contents\":[\"edit-controls\",\"zoom-controls\",\"new-window-button\",\"e10s-button\",\"privatebrowsing-button\",\"save-page-button\",\"print-button\",\"history-panelmenu\",\"fullscreen-button\",\"find-button\",\"preferences-button\",\"add-ons-button\",\"developer-button\",\"sync-button\"],\"addon-bar\":[\"addonbar-closebutton\",\"status-bar\"],\"PersonalToolbar\":[\"personal-bookmarks\"],\"nav-bar\":[\"urlbar-container\",\"bookmarks-menu-button\",\"downloads-button\",\"home-button\",\"jid1-n8wh2cbfc2qauj_jetpack-browser-action\",\"ublock0_raymondhill_net-browser-action\",\"_f73df109-8fb4-453e-8373-f59e61ca4da3_-browser-action\"],\"TabsToolbar\":[\"tabbrowser-tabs\",\"new-tab-button\",\"alltabs-button\"],\"toolbar-menubar\":[\"menubar-items\"]},\"seen\":[\"jid1-n8wh2cbfc2qauj_jetpack-browser-action\",\"ublock0_raymondhill_net-browser-action\",\"_f73df109-8fb4-453e-8373-f59e61ca4da3_-browser-action\",\"developer-button\"],\"dirtyAreaCache\":[\"PersonalToolbar\",\"nav-bar\",\"TabsToolbar\",\"toolbar-menubar\",\"PanelUI-contents\",\"addon-bar\"],\"currentVersion\":6,\"newElementCount\":0}");
// Use dense view.
pref("browser.uidensity", 1);
// Do not hide the http and colon-slash-slash elements of a URL in the url bar.
pref("browser.urlbar.trimURLs", false);
// Disable enlarged-upon-selected url bar.
pref("browser.urlbar.update1", false);
// Allow me to see ssl error messages (and the ability to continue past them).
pref("browser.xul.error_pages.enabled", false);
// Do not use webcam for this feature.
pref("camera.control.face_detection.enabled", false);
// Null-route these URLs
pref("captivedetect.canonicalURL", "http://127.0.0.1:9999/");
pref("devtools.devedition.promo.url", "https://127.0.0.1:9999/");
pref("dom.push.serverURL", "wss://127.0.0.1:9999/");
pref("security.ssl.errorReporting.url", "http://127.0.0.1:9999/");
pref("services.settings.server", "http://127.0.0.1:9999/");
pref("webextensions.storage.sync.serverURL", "http://127.0.0.1:9999/");
// Metadata that is not very important but it ended up in my copy-paste work.
pref("distribution.stackrpms.bookmarksProcessed", true);
// Disable Mozilla experiments.
pref("experiments.activeExperiment", false);
// This plugin is probably uBlock origin.
pref("extensions.enabledAddons", "%7B972ce4c6-7e08-4474-a285-3208198ce6fd%7D:28.3.0");
// Tell the browser that we already showed the user the page for "Select your addons" so it does not bother the user.
pref("extensions.shownSelectionUI", true);
// Already assume these plugins are enabled. This can help suppress the warning, "An admin added these plugins. Please choose to enable them or not."
pref("extensions.webextensions.uuids", "{\"uBlock0@raymondhill.net\":\"7f64930e-0e43-4813-97c3-6fcb8a82e63b\",\"jid1-n8wH2cBfc2QaUj@jetpack\":\"5b1c5018-34cd-4778-902b-08741e3d0002\",\"{f73df109-8fb4-453e-8373-f59e61ca4da3}\":\"b7ece467-f6eb-4254-a815-1029330a9793\"}");
// Select "Highlight all" for the find function.
pref("findbar.highlightAll", true);
// Miscellaneous
pref("gecko.handlerService.migrated", true);
pref("marionette.prefs.recommended", false);
pref("network.cookie.prefsMigrated", true);
pref("privacy.sanitize.migrateFx3Prefs", true);
// Do not warn me when entering about:config
pref("general.warnOnAboutConfig", false);
// Disable geolocation functions
pref("geo.enabled", false);
// Trust my domain for Kerberos authentication
pref("network.automatic-ntlm-auth.trusted-uris", ".ipa.example.com");
pref("network.negotiate-auth.trusted-uris", ".ipa.example.com");
// Disable the captive portal detection logic.
pref("network.captive-portal-service.enabled", false);
// Disable dns prefetching (exactly what it sounds like).
pref("network.dns.disablePrefetch", true);
// Disable whatever these are.
pref("network.predictor.enabled", false);
pref("network.prefetch-next", false);
// Disable requiring HSTS. Use at my own risk!
pref("network.stricttransportsecurity.preloadlist", false);
// Disable Reader mode.
pref("reader.parse-on-load.enabled", false);
// More personal ssl choices. Use at my own risk!
pref("security.cert_pinning.enforcement_level", 0);
// Hide these search plugins. Somehow the Debian package search keeps getting re-enabled so I need to work on this one.
pref("services.sync.declinedEngines", "");
// Startup home page (if I were to choose the option "Load these pages at startup," which I did not)
pref("startup.homepage_override_url",       "");
pref("startup.homepage_override_url", "");
// Hm, there should be more of these, particularly toolkit.telemetry.enabled = false
pref("toolkit.telemetry.reportingpolicy.firstRun", false);
// Trust these domain names for installing extensions: none!
pref("xpinstall.whitelist.add", "");
// Control DNS over HTTPS (DoH) and Trusted Recursive Resolver (TRR).
// More about DoH: https://github.com/bambenek/block-doh
// https://blog.nightly.mozilla.org/2018/06/01/improving-dns-privacy-in-firefox/
// https://support.mozilla.org/en-US/kb/configuring-networks-disable-dns-over-https
// https://wiki.mozilla.org/Trusted_Recursive_Resolver
// 0: Off by default, 1: Firefox chooses faster, 2: TRR default w/DNS fallback,
// 3: TRR only mode, 4: Use DNS and shadow TRR for timings, 5: Disabled.
pref("network.trr.mode", 0);
// Disable Pocket and null-route the URLs.
pref("extensions.pocket.enabled", false);
pref("extensions.pocket.api", "http://localhost:9980");
pref("extensions.pocket.site", "http://localhost:9980");

txt2man wrapper which adds man page headings support in text file

txt2man is great. It depends on only GNU awk, which I can certainly live with. I previously used go-md2man which is also fine, but I just don’t want to depend on golang.

txt2man has an area where I wanted to improve it, but I didn’t know how to contribute to upstream in a way that I expect they would want. So I wrote txt2man-wrapper! Txt2man adds the page titles, section number, etc., from parameters on the command line invocation, but I wanted to add the ability to store these values in the text file.

#!/bin/sh
# File: txt2man-wrapper
# Location: /usr/bin
# License: CC-BY-SA 4.0
# Author: bgstack15
# Startdate: 2020-04-15 17:35
# Title: Txt2man wrapper that adds title headings support
# Purpose: add title headings support to txt2man
# History:
# Usage:
# Custom layout for input text file, where first lines include "section 3", "title txt2man-wrapper", "project bgscripts", "volume Linux command reference" in no particular order, followed by "====="
# Reference:
# txt2man
# Improve:
# Dependencies:
# dep-raw: /usr/bin/awk, /usr/bin/txt2man
# rec-devuan: gawk | mawk, txt2man
# rec-fedora: coreutils, txt2man
test -z "${INFILE}" && test -n "${1}" && export INFILE="${1}"
if test -z "${INFILE}" || test "${INFILE}" = "-" ; then USE_STDIN=1 ; fi

if test "${USE_STDIN}" = "1" ;
then
input="$( cat )"
else
input="$( cat "${INFILE}" )"
fi

head="$( echo "${input}" | awk 'BEGIN{a=0} /^=====/{a=1} {if(a==0)print}' )"
short="$( echo "${input}" | awk '{if(a)print} /^=====/{a=1}' )"

ul="$( echo "${head}" | awk '$1=="title" {$1="";print}' | sed -r -e 's/^\s*//;' )"
uln="$( echo "${head}" | awk '$1=="section" {$1="";print}' | sed -r -e 's/^\s*//;' )"
dc="$( echo "${head}" | awk '$1=="date" {$1="";print}' | sed -r -e 's/^\s*//;' )"
ur="$( echo "${head}" | awk '$1=="project" {$1="";print}' | sed -r -e 's/^\s*//;' )"
uc="$( echo "${head}" | awk '$1=="volume" {$1="";print}' | sed -r -e 's/^\s*//;' )"
bolds1="$( echo '`__xxNONExx__`' "${short}" | tr -d '\r\n' | tr '`' '\n' | awk 'NR/2==int(NR/2){print}' | xargs -n1 printf '%s %s ' '-B' )"
bolds2="$( echo "*__xxNANExx__*" "${short}" | tr -d '\r\n' | tr '*' '\n' | awk 'NR/2==int(NR/2){print}' | xargs -n1 printf '%s %s ' '-B' )"
ital1="$( echo "${short}" "**__xxNBNExx__**" | tr -d '\r\n' | sed -r -e 's/\*\*/\n/g;' | awk 'NR/2==int(NR/2){print}' | xargs -n1 printf '%s %s ' '-I' )"
test -n "${DEBUG}" && echo "${bolds1}" "${bolds2}" "${ital1}" 1>&2
echo "${short}" | txt2man -t "${ul}" -r "${ur}" -s "${uln}" -v "${uc}" -d "${dc}" ${bolds1} ${bolds2} ${ital1}

Using my handy txt2man-wrapper above, or found in my bgscripts package, you can have a text file suitable for txt2man, with some extra header lines at the top.

title sizer
section 1
project bgscripts-core
volume General Commands Manual
date April 2020
=====
NAME
  sizer - summarize directory contents by file extension

And invoke txt2man-wrapper like so:

txt2man-wrapper < sizer.1.txt | gzip > sizer.1.gz

In a Makefile, you can use something similar to:

MAN_TXT:=$(wildcard usr/share/man/man*/*.txt)
MAN_GZ:= $(subst .txt,.gz,$(MAN_TXT))

build_man: $(MAN_GZ)

$(MAN_GZ): %.gz: %.txt
   txt2man-wrapper - < $< | ${gzipbin} > $@