Export Selected Items from RSS Reader using Microsoft PowerShell

Introduction

Do you want to export your favorite items from RSS reader? Such a task is easy to automate with reader's API.

Background

Once upon a time, there was Google RSS Reader, but its management decided to destroy it. Fortunately, after its passing services like The Old Reader are still alive.
As its successors, they like to inherit its API style, that's why you will find strings with 'google' inside some identifications.

This article expects that its reader is an active user of one of the mentioned RSS servers: has its own account and knows what is meant by RSS or how to star item in user interface.

PowerShell Basic Introduction

Let's start with the truth: I'm a C++ programmer with mostly Windows experience. Thus, I have no problem to read C-like computing languages, even sometimes write something in C#.

But having no compiler installed at home, for private use, I started to write scripts in PowerShell. All you need is Microsoft Windows (this article's example requires PowerShell 3.0 or newer, what is available for Windows 7 and newer) and any simple text editor (Notepad or PoweShell's Integrated Scripting Environment).

OK, to be more exact, any PowerShell is also available in Microsoft Azure's Functions (functions are alternative of Amazon Web Services' Lambda) and should be already on Linux, but I never tried its limits there.

Check your system first. Press Win+R, type 'powershell' (without apostrophes) and press Enter (or push OK button). If everything goes well, it will display to you Windows PowerShell console with blinking cursor ready for your input.

Type 'dir' and press Enter. You will see something very similar with output of system's native dir command.
Now type 'cd ..' and press Enter and I hope you start to feel at home.

Naturally comes the experiment with 'ver' which brings the first disappointment, anyway, try it to see the result.

To display PowerShell's version, you need to call '$PSVersionTable' (pasting from clipboard is available by mouse right-click).
As you can see, $PSVersionTable is structure. What we need is its PSVersion. Also try '$PSVersionTable.PSVersion' and you will see even more details. What we really need is
'$PSVersionTable.PSVersion.Major', which in our case must be 3 or more, otherwise you're going to download and install, maybe with newer corresponding version of .NET Framework first (later, we will call Now.ToUnixTimeSeconds() that requires .NET 4.6 or newer).

Having required PowerShell started, it's time for the classic Hello world! example, which in PoweShell is 'write-output "Hello world!"'. Try also 'write-output 1+1' and 'write-output (1+1)' to see that there is something more behind.

Run 'exit' and your first PowerShell lesson is over.

Now you just need to know that you can write PowerShell commands into a text file (common extension is ps1) and run them as a script.
By default, Microsoft decided that allowing 'PowerShell.exe -File myscript.ps1' is too dangerous. To avoid problems with UnauthorizedAccess, simply use 'PowerShell.exe -ExecutionPolicy Bypass -File myscript.ps1'. I'm a bit afraid that one day any dangerous code will be able to find and understand this paragraph.

Export from The Old Reader

Here is the PowerShell script which reads from The Old Reader's account list of liked items which are not older than seven days. You just need to enter your real The Old Reader's account id ($Email) and password ($Passwd) and choose own name of your client ($ClientName) by which it will be identifiable for The Old Reader's statistics.

The main structure of the script is simple:

1. With previously mentioned parameters, do client login ($ClientLoginURL)
2. With received login authentication code ($ClientLoginAuth), ask for list of recent ($LastNDays) liked items ($LikesURL)
3. For every item, get its parameters ($LikeURL), write its title ($LikeTitle) and url ($LikeHref) to output file ($OutFileName), with defined file header ($LikeBeginString) and footer ($LikeEndString).

If you want to experiment with API's output, you can enter URLs directly to your web browser. Just login to server traditional way and thanks to cookies, you do not need to address authentication code.

Technical Notes

If your email (or password) contains plus character (+), then in URL, it must be replaced by %2B.

I left $LikeBeginString and $LikeEndString contents with examples of some non-English characters coding (output will contain "Decembrové cítanie" or "Tibor Blažko"), PowerShell's new-line (`n) and quotation marks (script's double "" changes to ").

Script is using invoke-webrequest cmdlet. It means that it needs PowerShell 3.0 or newer (tested with 4.0).

Result of web request is Microsoft.PowerShell.Commands.HtmlWebResponseObject which has Content property. Note that for non-English characters, output of its ToString() method should be different to Content.

To get correct Unicode coding of item's parameters returned by The Old Reader, is used [Regex]::Unescape. If you want to see difference of before and after, add to script any 'write-output $LikeString'.

Script always asks server for output in JSON format (output=json) because it is more simple than alternative XML (atom). It is even possible to use any advanced JSON (or XML) parser, because of simplicity this script uses just simple string search in hopefully forever fixed format.

Also, script's error handling is minimal. For example, for successful login, it is enough that output contains "Auth=" substring.

Except communication with server, script contains also (not mandatory) parts which improve its output.

The first one (see #if href is to be redirected) changes to-be-redirected links (like http://feedproxy.google.com/~r/chinadigitaltimes/bKzO/~3/FIifoUE0ihM/) to redirected (http://chinadigitaltimes.net/2016/12/translation-game-developers-lament-censors-demands/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+chinadigitaltimes%2FbKzO+%28China+Digital+Times+%28CDT%29%29).

The second part (see #remove some statistics parameters) thanks to predefined list of known cases removes unnecessary parts of the link, so our example link changes to http://chinadigitaltimes.net/2016/12/translation-game-developers-lament-censors-demands/.

There are web servers which for accessing their contents need log-in. For script, it could mean that valid link is redirected to login page and web request will end with access error. Because script's request contains '-ErrorAction Ignore', the action ends just with few red lines of console output and keeping original, not-redirected link.

When list of items returned by server ($LikesString) contains 'continuation', it means that not all required items are listed. Script just displays WARNING, it does not iterate further, hoping that in real life, it is hard to exceed 1000 items (n=1000) in 7 days.

#like.ps1

$Email = "tibor%2Btheoldreader@gmail.com";
$Passwd = "12345";
$LastNDays = 7;
$OutFileName = "LikesList.txt"

$ClientName = "tblazkoWordpressComExport";

$ClientLoginURL = "https://theoldreader.com/reader/api/0/accounts/ClientLogin?client=" + 
$ClientName + "&accountType=HOSTED_OR_GOOGLE&service=reader&Email=" + $Email + "&Passwd=" + $Passwd;
$LikesURL = "https://theoldreader.com/reader/api/0/stream/items/ids?output=json&s=user/-/
             state/com.google/like&n=1000&ot=";
$LikeURL = "https://theoldreader.com/reader/api/0/stream/items/contents?output=json&i=";
$UnescapeLikeString = 1;

#login to get authentication code necessary for the next requests
$ClientLogin = Invoke-WebRequest -Uri $ClientLoginURL -Method post;
$ClientLoginString = $ClientLogin.Content;
$ClientLoginAuthIndex = $ClientLoginString.IndexOf("Auth=", 0);
if($ClientLoginAuthIndex -lt 0)
{
    #something failed
    write-Output $ClientLoginString; 
}
else
{
    #this is authentication code
    $ClientLoginAuth = $ClientLoginString.Substring($ClientLoginAuthIndex + 5);
    $ClientLoginAuth = "GoogleLogin auth=" + 
    $ClientLoginAuth.Substring(0, $ClientLoginAuth.IndexOf(10));

    #ask for list of liked items ids younger than $LastNDays
    $LastNDaysUnixTimeSeconds = [DateTimeOffset]::Now.ToUnixTimeSeconds() - $LastNDays*24*60*60;
    $LikesURL = $LikesURL + $LastNDaysUnixTimeSeconds.ToString();
    $Likes = Invoke-WebRequest -Uri $LikesURL 
    -Method get -Headers @{"Authorization" = $ClientLoginAuth};
    $LikesString = $Likes.Content;
    if($LikesString.IndexOf("""itemRefs"":[", 0) -lt 0)
    {
        #something failed
        write-Output $LikesString; 
    }
    else
    {
        #write output file header
        $LikeBeginString = "Decembrov$([char]0xe9) 
        $([char]0x10d)$([char]0xed)tanie`nEnglish needed`n<!--more-->`n"
        Out-File -FilePath $OutFileName -InputObject $LikeBeginString;

        #find item ids in the list
        for($LikeIdIndex = $LikesString.IndexOf
        ("""id"":""", 0);$LikeIdIndex -gt 0;$LikeIdIndex = $LikesString.IndexOf("""id"":""", 0))
        {
            $LikesString = $LikesString.Substring($LikeIdIndex + 6);
            $LikeId = $LikesString.Substring
            (0, $LikesString.IndexOf("""", 0));

            #get item contents
            $LikeIdURL = $LikeURL + $LikeId;
            $Like = Invoke-WebRequest -Uri $LikeIdURL 
            -Method get -Headers @{"Authorization" = $ClientLoginAuth};
            $LikeString = $Like.Content;
            if($UnescapeLikeString)
            {
                $LikeString = [Regex]::Unescape($LikeString);
            }

            #find item's title
            $LikeTitleBegin = $LikeString.IndexOf("],
            ""title"":""", 0) + 11;
            $LikeTitleEnd = $LikeString.IndexOf
            (""",""published"":", $LikeTitleBegin);
            $LikeTitle = $LikeString.Substring($LikeTitleBegin, $LikeTitleEnd - $LikeTitleBegin);

            #write item's title to console and output file
            write-Output $LikeTitle;
            Out-File -FilePath $OutFileName -Append -InputObject $LikeTitle;

            #find item's href
            $LikeHrefBegin = $LikeString.IndexOf("""canonical"":
            [{""href"":""", 0) + 22;
            $LikeHrefEnd = $LikeString.IndexOf("""}],", $LikeHrefBegin);
            $LikeHref = $LikeString.Substring($LikeHrefBegin, $LikeHrefEnd - $LikeHrefBegin);

            if($LikeHref.IndexOf("ft.com/", 0) -lt 0)
            {
                #if href is to be redirected switch to redirected one
                #note that this request sometimes fails f.e. 
                #because of access rights but we can ignore it
                $Redirection = Invoke-WebRequest -Uri $LikeHref 
                -MaximumRedirection 0 -ErrorAction Ignore
                if($Redirection.StatusCode -ge 300 -and $Redirection.StatusCode -lt 400)
                {
                    $LikeHref = $Redirection.Headers.Location;
                }
            }

            #remove some statistics parameters
            if($LikeHref.IndexOf("economist.com/", 0) -gt 0)
            {
                $EconomistEnd = $LikeHref.IndexOf("?fsrc=rss", 0);
                if($EconomistEnd -gt 0)
                {
                    $LikeHref = $LikeHref.Substring(0, $EconomistEnd);
                }
            }
            if($LikeHref.IndexOf("blogs.wsj.com/", 0) -gt 0)
            {
                $WsjEnd = $LikeHref.IndexOf("?mod=WSJBlog", 0);
                if($WsjEnd -gt 0)
                {
                    $LikeHref = $LikeHref.Substring(0, $WsjEnd);
                }
            }
            if($LikeHref.IndexOf("chinadigitaltimes.net/", 0) -gt 0)
            {
                $CdtEnd = $LikeHref.IndexOf("?utm_source=feedburner", 0);
                if($CdtEnd -gt 0)
                {
                    $LikeHref = $LikeHref.Substring(0, $CdtEnd);
                }
            }
            if($LikeHref.IndexOf("sinopsis.cz/", 0) -gt 0)
            {
                $SczEnd = $LikeHref.IndexOf("?utm_source=rssfeed", 0);
                if($SczEnd -gt 0)
                {
                    $LikeHref = $LikeHref.Substring(0, $SczEnd);
                }
            }

            #write item's href to console and output file
            write-Output $LikeHref;
            $LikeHrefFull = "<a href=""" + 
            $LikeHref + """>" + $LikeHref + "</a>`n";
            Out-File -FilePath $OutFileName -Append -InputObject $LikeHrefFull;
        }

        $ContinuationIdIndex = $LikesString.IndexOf("""continuation"":""", 0);
        if($ContinuationIdIndex -gt 0)
        {
            $NotAllString = "WARNING: Not all items are listed!!!";
            write-Output $NotAllString;
            Out-File -FilePath $OutFileName -Append -InputObject $NotAllString;
        }

        #write output file footer
        $LikeEndString = "Zozbieral: Tibor Bla$([char]0x17e)ko`nKv$([char]0x20f)
        li spamu je diskusia uzavret$([char]0xe1). Pr$([char]0xed)spevky posielajte cez 
        <a href=""http://www.paypal.com"">tblazko@gmail.com</a>."
        Out-File -FilePath $OutFileName -Append -InputObject $LikeEndString;
    }
}

The Old Reader's API documentation is available here.

Export from Inoreader

Script for Inoreader is nearly the same, just replace begin by:

#like.ps1

$Email = "tibor%2Binoreader@gmail.com";
$Passwd = "12345";
$LastNDays = 7;
$OutFileName = "LikesList.txt"

$AppId = "1000000001"
$AppKey = "12345678890abcdefghijklmnopqrtuv";

$AppIdKey = "AppId=" + $AppId + "&AppKey=" + $AppKey;
$ClientLoginURL = "https://www.inoreader.com/accounts/ClientLogin?" + 
$AppIdKey + "&Email=" + $Email + "&Passwd=" + $Passwd;
$LikesURL = "https://www.inoreader.com/reader/api/0/stream/items/ids?" + 
$AppIdKey + "&output=json&s=user/-/state/com.google/starred&ot="
$LikeURL = "https://www.inoreader.com/reader/api/0/stream/items/contents?" + 
$AppIdKey + "&output=json&i=";
$UnescapeLikeString = 1;

One big difference is script identification by Inoreader's server, in this case, it is by AppId and AppKey. You can create them in your Inoreader's web interface under Preferences/Developer/Create new application.
There is a possibility to move script identification from URL to request's header, but for simplicity, I keep it there.

Because Inoreader does not support liked items, script displays starred ones (see $LikesURL). The Old Reader supports both, but comparing to Inoreader with one more click in its web interface.

Inoreader's API documentation is available here.