Fun with YouTube Data API v3 (Public Data) and Kigs Framework

• Download Source Code - 61 KB

Table of Contents

• Introduction
• Background
• Google API Key
• Building the Code
• Executing the Code
• Dive into the Code
  • Manipulating a JSON File
  • HTTP GET Request
  • Displaying Results
• Conclusion
• History

Introduction

Kigs framework is an open-source (MIT Licence), free, C++, multi-purpose and cross-platform framework that you can find at https://github.com/Kigs-framework/kigs.

The application presented here displays statistics on Youtube Channels subscribed by users subscribing to a given Channel.

To do this, the application first retrieves videos of a given YouTube Channel, then retrieves comments from each video, and for each comment's writer tests if the writer's subscriptions are public.

If yes, the application checks if the writer subscribes to the given Youtube Channel and if yes again, adds other writer's subscribed channels to the statistics.

In the following image, we can see that for about 50 Kurzgesagt subscribers found, 34% subscribes to TED Ed,  30% to Mark Rober and 30% to VSauce.

Background

A series of articles was started here on Code Project. You can have a look at the first article here:

• https://www.codeproject.com/Articles/5253209/Kigs-Framework-Introduction-1-8

Of course, you can read the present article without first reading the Kigs framework introduction series.

We would be happy to be joined by people wishing to participate in the use and improvement of the Kigs framework.

Google API Key

To execute the code, you will need to create a Google API Key. You can freely create and use a (limited) API Key.

Register with a Google account at https://console.developers.google.com.

Create a new project and on the Credentials page, click Create credentials > API key.

Then create an API key for "Youtube Data API v3", for the platform, choose "other platform..." and public data.

Building the Code

To build the code, you will have to clone the framework from GitHub and set up the development environment for Windows platform (Visual C++, CMake). Check https://github.com/Kigs-framework/kigs/wiki/Getting-Started.

You can then clone publicKigsProjects repository ( https://github.com/Kigs-framework/publicKigsProjects ) so that kigs and publicKigsProjects folders are at the same level.

Then execute (double click) kigs\scripts\generateWinCMake.bat.

Browse to generated Build\solutionWinCMake\publicKigsProjects\projects\YoutubeAnalyser folder.

And double click on YoutubeAnalyser.sln to open the solution in Visual Studio.

In Visual Studio, select YoutubeAnalyser as your startup project. Select StaticDebug or StaticRelease configuration and build.

Executing the Code

Before executing the code, you will have to edit configuration file at kigs\projects\YoutubeAnalyser\Data\launchParams.json.

Open it in a text editor. You will find this:

{
   "GoogleKey" : "INSERT_YOUR_GOOGLE_KEY_HERE",
   "ChannelID" : "INSERT_CHANNEL_ID_HERE",
   "ValidUserCount" : 100,
   "MaxChannelCount" : 40,
   "ValidChannelPercent" : 0.02,
}

Replace INSERT_YOUR_GOOGLE_KEY_HERE by your previously generated Key.

Then go to YouTube, choose one of your favorite channels (in this article, I will use "Kurzgesagt – In a Nutshell") and look at the URL, https://www.youtube.com/channel/UCsXVk37bltHxD1rDPwtNM8Q.

If URL doesn't contains channel ID, you can also look at the source code of the page and search for something starting with "canonical" and looking like a channel ID (starting with "UC" and followed by alphanumeric characters).

Copy the channel ID UCsXVk37bltHxD1rDPwtNM8Q and replace INSERT_CHANNEL_ID_HERE by the wanted channel ID.

Save and close launchParams.json. You can now execute the project in Visual C++.

Here is the percentage view, showing the highest other channel scores.

Here is the attraction/repulsion view.

Dive Into the Code

We will now see how the Kigs framework can be used to do several things.

Manipulating a JSON File

YouTubeAnalyser class has the following members:

// general application parameters

std::string                     mKey = "";
std::string                     mChannelName;
unsigned int                    mSubscribedUserCount = 100;
unsigned int                    mMaxChannelCount = 16;
int                             mMaxUserPerVideo = 0;
float                           mValidChannelPercent = 0.02f;

And at application start, we want to retrieve the values stored in launchParams.json.

This is done with the following code in YoutubeAnalyser.cpp (around line 70):

JSonFileParser L_JsonParser;
CoreItemSP initP = L_JsonParser.Get_JsonDictionary("launchParams.json");

// retrieve parameters
mKey = "&key=" + (std::string)initP["GoogleKey"];
mChannelName = initP["ChannelID"];



auto SetMemberFromParam = [&](auto& x, const char* id) {
    if (!initP[id].isNil()) x = initP[id];
};

SetMemberFromParam(mSubscribedUserCount, "ValidUserCount");
SetMemberFromParam(mMaxChannelCount, "MaxChannelCount");
SetMemberFromParam(mValidChannelPercent, "ValidChannelPercent");
SetMemberFromParam(mMaxUserPerVideo, "MaxUserPerVideo");
SetMemberFromParam(mUseKeyword, "UseKeyword");

As you can see, the process is really simple: create a JSonFileParser instance, call Get_JsonDictionary with the filename as parameter and store the result in CoreItemSP initP instance.

Then you can search if values are available on a CoreItemSP instance using isNil() method, accessing them with initP["searched object"].

More complex JSON data access is done in other parts of the code (around line 1430 for example):

for (int i = 0; i < commentThreads->size(); i++)
{
   CoreItemSP topComment = commentThreads[i]["snippet"]["topLevelComment"]["snippet"];
   if (!topComment.isNil())
   {
        std::string authorID = topComment["authorChannelId"]["value"];
        // ...
   }
}

Creating a JSON like object and saving it is also easy (around line 1170):

JSonFileParser L_JsonParser;

std::string filename = "Cache/" + mChannelName + "/videos/";
filename += videoID + "_videos.json";

CoreItemSP initP= CoreItemSP::getCoreMap();
CoreItemSP v = CoreItemSP::getCoreVector();
for (const auto& auth : mAuthorListToProcess)
{
   v->set("", CoreItemSP::getCoreValue(auth));
}

initP->set("authors", v);

L_JsonParser.Export((CoreMap<std::string>*)initP.get(), filename);

HTTP GET Request

Public YouTube data are accessed by doing GET HTTP requests (HTTPS in fact).

With the Kigs framework, we first need to create the HTTPRequest Module in the application initialization method, YoutubeAnalyser::ProtectedInit().

CoreCreateModule(HTTPRequestModule, 0);

Then a HTTPConnect instance is created and initialized:

// ask for an instance of class HTTPConnect 
mGoogleConnect = KigsCore::GetInstanceOf("googleConnect", "HTTPConnect");
// set HostName
mGoogleConnect->setValue("HostName", "www.googleapis.com");
// set connection type ( HTTP or HTTPS )
mGoogleConnect->setValue("Type", "HTTPS");
// set Port to default HTTPS port
mGoogleConnect->setValue("Port", "443");
// initialize instance with given parameters
mGoogleConnect->Init();

Then each time an HTTP GET request is needed, an instance of HTTPRequest is created:

// encode request in URL
std::string url = "/youtube/v3/channels?part=snippet&id=";
std::string request = url + mChannelName + mKey;
// then ask HTTPConnect to create a HTTPRequest instance
// parameters are:
// - the request itself
// - the callback to call when results are received
// - the object on which callback is called
mAnswer = mGoogleConnect->retreiveGetAsyncRequest(request.c_str(), "getChannelID", this);
// initialize the instance to really send the request
mAnswer->Init();

Kigs framework callbacks are declared like this (here in YoutubeAnalyser.h) :

DECLARE_METHOD(getChannelID);
DECLARE_METHOD(getChannelPage);
DECLARE_METHOD(getVideoList);
// ...
COREMODIFIABLE_METHODS(getChannelID, getVideoList, getChannelPage);

Then a callback is defined like this (here in YoutubeAnalyser.cpp around line 1060):

DEFINE_METHOD(YoutubeAnalyser, getChannelID)
{
   auto json=RetrieveJSON(sender);
   if (!json.isNil())
   {
      CoreItemSP IDItem = json["items"][0]["id"];
      // ... 

Displaying Results

The main screen is defined with the XML file, YoutubeAnalyser\Data\assets\Screen_Main.xml.

This file starts like this:

<?xml version="1.0" encoding="iso-8859-1"?>
<Inst N="sequencemain" T="DataDrivenSequence">

Each time a new sequence is initialized in a Data-Driven Kigs framework application, the method ProtectedInitSequence is called with the sequence name as a parameter.

In YoutubeAnalyser.cpp (around line 890), the method is defined like this:

void    YoutubeAnalyser::ProtectedInitSequence(const kstl::string& sequence)
{
    if (sequence == "sequencemain")
    {
        mMainInterface = GetFirstInstanceByName("UIItem", "Interface");
    }
}

The sequence name, "sequencemain", is defined in the Screen_Main.xml as seen previously. So here, we will search for an instance of class UIItem (base class for all 2D elements) named "Interface" in all the framework instanced objects and store it in member variable mMainInterface.

In the same Screen_Main.xml, the Interface UIItem has (among others) a UIImage son named "thumbnail", itself having a UIText son named "ChannelName":

<Inst N="Interface" T="UIItem">
   <Attr N="SizeY" V="800"/>
   <Attr N="SizeX" V="1280"/>
   <Inst N="thumbnail" T="UIImage">
      <Attr N="Priority" V="48"/>
      <Attr N="Dock" V="{0.5,0.5}"/>
      <Attr N="Anchor" V="{0.5,0.5}"/>
      <Inst N="ChannelName" T="UIText">
         <Attr N="Priority" V="47"/>
         <Attr N="Text" V="channelName"/>
         <Attr N="Dock" V="{0.5,1.0}"/>
         <Attr N="Anchor" V="{0.5,0.0}"/>
         <Attr N="SizeX" V="-1"/>
         <Attr N="SizeY" V="-1"/>
         <Attr N="Font" V="Calibri.ttf"/>
         <Attr N="FontSize" V="20"/>
         <Attr N="MaxWidth" V="200"/>        
      </Inst>
   </Inst>
</Inst>

In YoutubeAnalyser.cpp (around line 600), we will set the texture and name of the channel:

if (mMainInterface)
{
   // check if Channel texture was loaded
   if (mChannelInfos.mThumb.mTexture && mMainInterface["thumbnail"])
   {
      // cast to UIImage smartpointer reference
      const SP<UIImage>& tmp = mMainInterface["thumbnail"];

      // texture was not already set ?
      if (!tmp->HasTexture())
      {
         // add texture
         tmp->addItem(mChannelInfos.mThumb.mTexture);
         // set "Text" value on "ChannelName" instance son of "thumbnail" instance
         // son of mMainInterface instance 
         mMainInterface["thumbnail"]["ChannelName"]("Text") = mChannelInfos.mName;
      }
   }
}

Conclusion

The free Google API Key is limited so that our application can only do a limited amount of requests per day. But all requests are cached in files, so when the limit is reached, you can close the app and execute it again the next day to go further.

In this article, we have seen some of the Kigs framework features:

• Manipulate JSON files or objects
• Create and manipulate instances of the framework's classes
• XML serialization, 2D display
• ...

If you liked this article, feel free to rate it, read Kigs framework Introduction series, and join us using the Kigs framework and help us to make it live and grow.

History

• 13^th April, 2020: First release
• 1^st May, 2020: Kigs-framework GitHub repo was moved
• 9^th July, 2020: Fixed some bugs, better error management
• 1^st September, 2020: Updated source code after Kigs repository member variable renaming
• 14^th September, 2020: Updated source code, line numbers in article and publicKigsProjects repository
• 24^th February, 2021: Updated source code : round thumbails, new visualizations