PowerShell Sftp Retrieve multiple Files (and Remove Source files)

/ PowerShell Scripting / By / Leave a Comment

The PowerShell script below retrieves multiple files from a SFTP server source using the native windows SFTP-client in windows After retrieving each individual file it removes the file from the SFTP-server. This specific script makes us of a (preconfigured) ssh-key authentication configuration, but it can easily been rewritten for password authentication.

<#
    Description    : Retrieve (get) and remove files from a sftp source... 
    Keywords       : Sftp, Get and Remove Original, Move

    Majorversion   : 1
    Scriptserver   : LOCALHOST
    Serviceaccount : -

    Author         : Tim van Kooten Niekerk
    Date           : 2023-03-14
#>

<#      CHANGELOG:
        2023-03-14 => Initieel (Tim van Kooten Niekerk)

#>


function fnMoveSftpFiles()
{
  param(
    [string]$sSftpServerHost,
    [string]$sSftpUser,
    [string]$sSftpServerPort = "22",
    [string]$sSftpRemoteFilePath,
    [string]$sSftpRemoteFileMask,
    [string]$sSftpLocalFilePath,
    [string]$sSftpScriptRetrieveClose = "quit"
  )

  # System variables...
  [string]$_sSftpScriptLogFile = $sSftpLocalFilePath + "\get-SftpFiles_" + $(get-date -f yyyy-MM-dd) + '.log'
  [string]$_sSftpScriptRetrieveFiles = "lcd " + $sSftpLocalFilePath + "`ncd " + $sSftpRemoteFilePath + "`n" + "ls -1 " + [string]$sSftpRemoteFileMask
  [string]$_sSftpScriptRetrieveOpen = "lcd " + $sSftpLocalFilePath + "`ncd " + $sSftpRemoteFilePath + "`n"
  [string]$_sSftpScriptRetrieveBody = $_sSftpScriptRetrieveOpen
  [string]$_CurrentTimeStamp = $(get-date -f 'yyyy-MM-dd HH:mm:ss')
  [string]$_InfoDoneWithFiles = $_CurrentTimeStamp + " | INFO | Commands executed: "
  [string]$_InfoDoneWithoutFiles = $_CurrentTimeStamp + " | INFO | No remote files found"
  [string]$_Return = $_CurrentTimeStamp + " | ERROR | Unknown Error"

  # Retrieve file list...
  [array]$aFiles = (write-output $_sSftpScriptRetrieveFiles) | sftp -b - -o Port=$sSftpServerPort $sSftpUser@$sSftpServerHost | select-string -notmatch "sftp>"

  # Create script for retrieving files...
  if ($aFiles.length -gt 0) {
    # Create get and rm script lines for every file in the listing...
    for (($n=0); $n -lt $aFiles.length; $n++) { 
      if ($aFiles[$n].line.length -gt 0)
      { 
        $_sSftpScriptRetrieveBody = $_sSftpScriptRetrieveBody + "get " + $aFiles[$n].Line + "`n"
        $_sSftpScriptRetrieveBody = $_sSftpScriptRetrieveBody + "rm " + $aFiles[$n].Line + "`n"
      }
    }

    # Close script file...
    $_sSftpScriptRetrieveBody = $_sSftpScriptRetrieveBody + $sSftpScriptRetrieveClose

    # Process script...
    [array]$aProcess = (write-output $_sSftpScriptRetrieveBody) | sftp -b - -o Port=$sSftpServerPort $sSftpUser@$sSftpServerHost
  
    # Audit log - Commands executed...
    $_InfoDoneWithFiles + $aProcess | Out-File $_sSftpScriptLogFile -Append
    $_Return = $_InfoDoneWithFiles + $aProcess
    [array]$aProcess = ""
  
  } else {
    # Audit log - No remote files...
    $_InfoDoneWithoutFiles | Out-File $_sSftpScriptLogFile -Append
    $_Return = $_InfoDoneWithoutFiles

  }
  return $_Return
}

PS> fnMoveSftpFiles -sSftpServerHost “example.com” -sSftpUser “user” -sSftpRemoteFilePath “/test” -sSftpRemoteFileMask “*.txt” -sSftpLocalFilePath “D:\Tim\Test” [-sSftpServerPort “22“]

About & Search

/ About / By

TIMVKN.NL/TECH hosts a repository of scripts and patterns I have created over the course of time.

My Frequently Used CLI Commands / Admin Tools (QuickRef)Description
dsa.msc / adsiedit.msc / virtmgmt.msc Active Directory Users & Computers / ADSI Edit / Hyper-V Manager
start-process powershell -verb runasStart PowerShell with Elevated Privileges
rundll32 sysdm.cpl,EditEnvironmentVariablesEdit system environment variables
cacls “C:\Path\To\File.TXT” /E /G username@example.com:CQuick way to add write permission to a filesystem resource
mvn clean package -DskipTestsMavin compile (Java) code and create local package
mvn test -Dtest=JavaClassTest#specificTestRun a specific unit test from the command line
java -Djavax.net.debug=ssl:handshake -jar target/packagename-1.0.0-SNAPSHOT.jarStart package with TLS handshake debugging enabled
oc create secret generic gateway1 –from-file=server.p12Create an OpenShift secret from the command line
touch -t 202301251415.00 FilenameForce file timestamp to a specific datetime
oc get pods | grep Evicted | awk ‘{print $1}’ | xargs oc delete podDelete evicted pods from current OpenShift project

GNU/Linux Telegram/Bot Mail Alert Script (Procmail/Python)

/ GNU/Linux / By

Small script to send a telegram chat message to a user when new mail arrives. Script uses procmail and a small python script to process the message and to send the alert alert message through a telegram bot api.

.procmailrc file:

#LOGFILE=/path/to/pmlog
#VERBOSE=yes
#LOGABSTRACT=all
INCLUDERC=/etc/procmail-rc/filter-potentialproblem-rc
INCLUDERC=/etc/procmail-rc/filter-potentialproblem2-rc
INCLUDERC=/etc/procmail-rc/filter-spamassassin08-rc

:0Wc:
| env python ./.pmscript.py

:0
.mail/

.pmscript.py file:

import sys
import email
import httplib
import urllib

# Contstants...
email_addresses = ['test@example.com', 'info@example.com']
key = "telegram_botkey"
userid = "telegram_userid"
headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain"}
hostname = "api.telegram.org"
uri = "/bot" + key + "/sendMessage"

# Read from stdin...
full_msg = sys.stdin.read()
msg = email.message_from_string(full_msg)

# Get info from message...
mto = msg['to']
msubject = msg['subject']
mfrom = msg['from']

# Check if e-mail in list...
for mailaddr in email_addresses:
  if mailaddr in mto:
    mrow = "[S6:NewMailAlert] \nFrom: " + mfrom + " \nSubject: " + msubject
    params = urllib.urlencode({'chat_id': userid, 'disable_web_page_preview': '1', 'text': mrow})
    connectie = httplib.HTTPSConnection(host=hostname, port=443)
    connectie.request("POST", uri, params, headers)
    resultaat = connectie.getresponse()
    if resultaat.status <> 200:
      print resultaat.status, resultaat.reason

PowerShell Validate XML Message

/ PowerShell Scripting / By

Quick (raw) guide to validate a XML message using Powershell.

# Declare schemas opbject…
$schemas = New-Object System.Xml.Schema.XmlSchemaSet

# Read schema from file…
$schemaItem = Get-Item File.xsd
$fileStream = $schemaItem.OpenRead()
$xsd = [Xml.Schema.XmlSchema]::Read($fileStream, $null)
$fileStream.Close()

# Add XSD to schemas &amp; compile…
$schemas.Add($xsd)
$schemas.Compile()

# Read &amp; validate XML…
[xml]$xml = Get-Content .\File.xml
$xml.Schemas = $schemas
$xml.Validate($null)

Powershell command to generate a new GUID.

[guid]::NewGuid().Guid