{"id":9414,"date":"2020-04-05T15:13:16","date_gmt":"2020-04-05T05:13:16","guid":{"rendered":"https:\/\/nsrd.info\/blog\/?p=9414"},"modified":"2020-04-11T10:03:35","modified_gmt":"2020-04-11T00:03:35","slug":"networker-basics-a-simple-prototype-web-based-nsrwatch","status":"publish","type":"post","link":"https:\/\/nsrd.info\/blog\/2020\/04\/05\/networker-basics-a-simple-prototype-web-based-nsrwatch\/","title":{"rendered":"NetWorker Basics \u2013\u00a0A Simple Prototype Web-Based nsrwatch"},"content":{"rendered":"\n<p>There&#8217;s a misconception at the moment that we&#8217;ve all naturally got tonnes of spare time on our hands because a significant population of the world is staying indoors to keep away from COVID19. I say it&#8217;s a <em>misconception<\/em> because there&#8217;s still a bunch of things many of us have to do on a daily basis.<\/p>\n\n\n\n<p>I did have some spare time this weekend, and since my RSI is making Minecraft a little too painful, I decided the next best entertainment would be to delve further into NetWorker REST API programming. It started over my third or eight Saturday morning coffee\u2014I can&#8217;t recall which, but it couldn&#8217;t have been too many since I still couldn&#8217;t see ultraviolet\u2014and I wondered <em>could I write a basic web-based nsrwatch using the NetWorker REST API?<\/em><\/p>\n\n\n\n<p>To avoid a long, drawn-out story with too much suspense, the TL;DR for the question is, yes:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"3286\" height=\"2196\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/Webwatch-Final.png\" alt=\"\" class=\"wp-image-9416\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/Webwatch-Final.png 3286w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/Webwatch-Final-300x200.png 300w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/Webwatch-Final-1024x684.png 1024w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/Webwatch-Final-768x513.png 768w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/Webwatch-Final-1536x1026.png 1536w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/Webwatch-Final-2048x1369.png 2048w\" sizes=\"auto, (max-width: 3286px) 100vw, 3286px\" \/><figcaption>Prototype NSR Webwatch<\/figcaption><\/figure>\n\n\n\n<p>Now, that&#8217;s not an exact recreation of <em>nsrwatch<\/em> \u2013&nbsp;I figured I&#8217;d start by focusing on a few key things: sessions, devices and volumes. (In retrospect, volumes may not have been the most dynamic of choices, but I stuck with it.)<\/p>\n\n\n\n<p>Let&#8217;s start with the architecture I came up with. My first thought was that I didn&#8217;t want to assume the &#8216;service&#8217;, if you&#8217;d call the prototype, would run on the NetWorker server itself. So I decided on a simple communication strategy that&#8217;s one step removed from a sneakernet: there&#8217;d be a service that would probe the NetWorker server via the REST API, writing data files, and then there&#8217;d be a web front-end that would read those files to work out what to display. The system flow, therefore, ended up looking like the following:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"965\" height=\"1009\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/webwatch-flow.png\" alt=\"\" class=\"wp-image-9417\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/webwatch-flow.png 965w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/webwatch-flow-287x300.png 287w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/webwatch-flow-768x803.png 768w\" sizes=\"auto, (max-width: 965px) 100vw, 965px\" \/><figcaption>webwatch system flow<\/figcaption><\/figure>\n\n\n\n<p>Unlike the <a aria-label=\"previous REST API example (opens in a new tab)\" href=\"https:\/\/nsrd.info\/blog\/2020\/03\/30\/basics-finding-unprotected-virtual-machines-with-networker\/\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"aioseop-link\"><strong>previous REST API example<\/strong><\/a> where the script would prompt for the server, username and password, this time I&#8217;ve hard-coded them. I know, not a good idea for security, but I wanted to keep it simple. So I configured a new user account in NMC and NetWorker called <em>watcher<\/em> which was mapped only to having NetWorker monitoring privileges:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"805\" height=\"1024\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/nmc_user-805x1024.png\" alt=\"\" class=\"wp-image-9418\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/nmc_user-805x1024.png 805w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/nmc_user-236x300.png 236w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/nmc_user-768x977.png 768w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/nmc_user.png 976w\" sizes=\"auto, (max-width: 805px) 100vw, 805px\" \/><figcaption>Creating a &#8220;watcher&#8221; user (part 1 of 2)<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"646\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/networker_user_monitor-1024x646.png\" alt=\"\" class=\"wp-image-9419\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/networker_user_monitor-1024x646.png 1024w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/networker_user_monitor-300x189.png 300w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/networker_user_monitor-768x484.png 768w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/networker_user_monitor-1536x968.png 1536w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/networker_user_monitor.png 1564w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>Creating a &#8220;watcher&#8221; user (part 2 of 2)<\/figcaption><\/figure>\n\n\n\n<p>With that user created, I was happy enough to create the <em>webwatch.pl<\/em> script. As you may imagine by the extension, this is another Perl script. I know, everyone says that to be a fancy-pants REST API programmer you should use Python, but I&#8217;ve decided to stick with Perl for a while longer yet. To create the script, I had to install the following additional modules:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>JSON (&#8220;cpan install JSON&#8221;)<\/li><li>REST::Client (&#8220;cpan install REST::Client&#8221;)<\/li><\/ul>\n\n\n\n<p>In this prototype, the webwatch.pl program runs in the foreground, and generates output such as the following:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">[Sun Apr 05 14:20:54]\n[\u2022 ~\/projects\/util-Perl \u2022]\npmdg@hala\n$ <strong>.\/webwatch.pl<\/strong>\nCurrent time: Sun Apr 5 14:20:59 2020\n        Processing 7 devices\n        Processing 749 recorded sessions\n                No active sessions.\n        Processing 43 volumes\n        (Sleep 60)\n\nCurrent time: Sun Apr 5 14:22:00 2020\n        Processing 7 devices\n        Processing 753 recorded sessions\n                4 active sessions\n        Skip volume poll\n        (Sleep 60)<\/pre>\n\n\n\n<p>While you can change the polling time in the script, I configured the script to only poll the NetWorker server once every 60 seconds for new data. What you may notice in the second poll above is <em>Skip volume poll<\/em>; the script only updates its information about NetWorker volumes once every 5 polls. Since the NetWorker REST API returns a list of all the SSIDs on a volume, I wanted to only pull this data down periodically compared to the rest of the information.<\/p>\n\n\n\n<p>The back-end data-gatherer writes three data files to the nominated data directory: volumes.data, sessions.data and devices.data. (It also uses a sentinel file to indicate when it&#8217;s writing, and the web front-end tries to wait for the sentinel file to be removed before it goes on to read. The reverse is something I&#8217;ll cover off in an update.)<\/p>\n\n\n\n<p>To run the web front-end, you&#8217;ll need a web server running on the same host that <em>webwatch.pl<\/em> runs on, and that web server will need CGI enabled. You&#8217;ll need to copy the <em>webwatch-fe.cgi<\/em> into an appropriate CGI directory, and the webwatch.css file into the web server&#8217;s document root directory. (You can change these of course, but you&#8217;d subsequently need to edit the Refresh and CSS variables in <em>webwatch-fe.cgi<\/em>.)<\/p>\n\n\n\n<p>When you access the CGI script in your browser, it&#8217;ll configure itself to refresh every 20 seconds, and it&#8217;ll look for the <em>sessions.data<\/em>, <em>volumes.data<\/em> and <em>devices.data<\/em> files in the nominated data directory before constructing the basic web page:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"3282\" height=\"2112\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/Webwatch-02.png\" alt=\"\" class=\"wp-image-9421\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/Webwatch-02.png 3282w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/Webwatch-02-300x193.png 300w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/Webwatch-02-1024x659.png 1024w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/Webwatch-02-768x494.png 768w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/Webwatch-02-1536x988.png 1536w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/Webwatch-02-2048x1318.png 2048w\" sizes=\"auto, (max-width: 3282px) 100vw, 3282px\" \/><figcaption>webwatch in action<\/figcaption><\/figure>\n\n\n\n<p>(The above screen-capture was grabbed before I thought to modify the title text to include the refresh time.)<\/p>\n\n\n\n<p>This basic webwatch breaks the page up into three scrollable sections:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Sessions<\/strong>: Either a list of currently active sessions, or the text &#8220;No active sessions&#8221; is shown under the heading.<\/li><li><strong>Devices<\/strong>: The configured NetWorker devices. These are sorted by device name, so unless you frequently add devices, the list should remain in a consistent order.<\/li><li><strong>Volumes<\/strong>: All the NetWorker volumes in the media database, with basic details regarding written, recovered, etc. In a future version, I can imagine grouping before sorting, and putting disk-based volumes at the top of the list.<\/li><\/ul>\n\n\n\n<p>Being standard HTML, it&#8217;s not just accessible on a conventional browser, but also on tablets, etc. For example:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"2560\" height=\"1595\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/webwatch-tablet-scaled.jpg\" alt=\"\" class=\"wp-image-9422\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/webwatch-tablet-scaled.jpg 2560w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/webwatch-tablet-300x187.jpg 300w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/webwatch-tablet-1024x638.jpg 1024w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/webwatch-tablet-768x479.jpg 768w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/webwatch-tablet-1536x957.jpg 1536w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/webwatch-tablet-2048x1276.jpg 2048w\" sizes=\"auto, (max-width: 2560px) 100vw, 2560px\" \/><figcaption>webwatch via a tablet<\/figcaption><\/figure>\n\n\n\n<p>I&#8217;ve made a <strong><a href=\"https:\/\/nsrd.info\/scripts\/webwatch-distro.zip\" class=\"aioseop-link\">zip file available here<\/a><\/strong> which contains:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>webwatch.pl<\/li><li>webwatch-fe.cgi<\/li><li>webwatch.css<\/li><li>README.txt<\/li><\/ul>\n\n\n\n<p>The README.txt file summarises the key steps you need to follow to get the prototype working, so be sure to review that if you&#8217;re going to download and play around with the prototype.<\/p>\n\n\n\n<p>All in all, this was pretty straight-forward and fun, and if you&#8217;re using NetWorker, I hope you find it useful to experiment with.<\/p>\n\n\n\n<p><strong><em>[Edit \u2013 2020-04-11]<\/em><\/strong><\/p>\n\n\n\n<p>I&#8217;ve now written an updated version that shows messages and alerts instead of volumes. <strong><a href=\"https:\/\/nsrd.info\/blog\/2020\/04\/11\/networker-basics-enhanced-webwatch\/\" class=\"aioseop-link\">You can check it out, here<\/a><\/strong>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>There&#8217;s a misconception at the moment that we&#8217;ve all naturally got tonnes of spare time on our hands because a&hellip;<\/p>\n","protected":false},"author":1,"featured_media":9415,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[6,16,20],"tags":[1249,1317],"class_list":["post-9414","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-basics","category-networker","category-scripting","tag-networker","tag-rest-api"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2020\/04\/bigStock-Web-Development.jpg","jetpack_shortlink":"https:\/\/wp.me\/pKpIN-2rQ","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/9414","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/comments?post=9414"}],"version-history":[{"count":5,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/9414\/revisions"}],"predecessor-version":[{"id":9437,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/9414\/revisions\/9437"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/media\/9415"}],"wp:attachment":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/media?parent=9414"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/categories?post=9414"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/tags?post=9414"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}