Details

Published: 11 October 2019

• nodejs
• git

Dotenv is a zero-dependency module that loads environment variables from a .env file into process.env. Storing configuration in the environment separate from code is based on The Twelve-Factor App methodology.
Reference: https://github.com/motdotla/dotenv

While you can use NodeJS Dotenv to manage configurations per environment,
another approach is to create a configuration per environment, and use an environment based file to distinguish which configuration to use. 

Why? 

It is useful to actually version control your configuration, so every developer and every instance of your application has the same configuration keys and values.

An example of how to setup a production and staging environment follows. 

Create the environment file as root to minimize the odds of the environment being removed or changed

> cd /home/yuourapp/
> sudo touch env-prod 

> cd /home/yuourappstg/
> sudo touch env-stg 

Helper scripts to start and restart your NodeJS Forever service
Note: [ -f "env-stg" ] returns true if the file exists 

> start-yourapp.sh
#!/bin/bash

if [ -f "env-stg" ]; then
    forever start -a --minUptime 1000 --spinSleepTime 2000 --uid yourapp-stg yourapp.js
else
    forever start -a --minUptime 1000 --spinSleepTime 2000 --uid yourapp yourapp.js
fi

> restart-yourapp.js
#!/bin/bash

if [ -f "env-stg" ]; then
    forever restart yourapp-stg
else
    forever restart yourapp
fi 

Create a configuration file per environment, ensuring that each configuration has the same keys, and varying the values as appropriate.

> config/config-stg.js
module.exports = {
  port : 9011,
  log : {
    console : { level : 'silly' }
  }
};

> config/config-prod.js
module.exports = {
  port : 9001,
  log : {
    console : { level : 'error' }
  }
}; 

Create a base configuration script to read in the appropriate configuration file 

An example for a NodeJS process
> config/config.js
const path = require('path');
const fs = require('fs');

let env;
// check for env-stg or env-prod file
if (fs.existsSync('env-stg')) {
  env = 'stg';
} else {
  env = 'prod';
}

const configPath = path.resolve(process.cwd(), `config/config-${env}`);
const config = require(configPath);
// visual validation of correct env
console.log('Using config ' + configPath);
module.exports = config;

An example for a VueJS/Nuxt process 

Due to VueJS/Nuxt being a browser based solution,
to avoid warnings and errors, create another env file to be required 

> sudo vi env
module.exports = {
    env: 'stg'
}

Add configuration as needed

> nuxt.config-stg.js
module.exports = {
  server: { port: 9013 },
};

> nuxt.config-prod.js
module.exports = {
  server: { port: 9003 },
}; 

Create a base configuration script to read in the appropriate configuration file.

> nuxt.config.js
const path = require('path');

// check for env file
let env;
try {
  // (base)/
  // require ('./env-stg');
  // using exports, as when require from .vue, build causes warning 'Module not found: Error: Can't resolve './env-stg''
  env = require ('./env');
  env = env.env;
} catch (e) {
  // default prod
  env = 'prod';
}

// check for env based nuxt config when called from different relative paths
let configPath;
let config;
try {
  // (base)/
  configPath = `./nuxt.config-${env}.js`;
  // config = require(configPath); // on build, results in warning 'Critical dependency: the request of a dependency is an expression'
  config = require(`./nuxt.config-${env}.js`);
} catch (e) {
  try {
    // (base)/server/
    configPath = `../nuxt.config-${env}.js`;
    config = require(`../nuxt.config-${env}.js`);
  } catch (e) {
    // (base)/pages/dir/
    configPath = `../../nuxt.config-${env}.js`;
    config = require(`../../nuxt.config-${env}.js`);
  }
}

// visual validation of correct env
console.log('Building nuxt using ' + configPath);
module.exports = config;

Now you can check in the configuration files, but do not check in the .env and env-stg, env-prod files (add them to .gitignore), as those should vary based on the deployed environment.

Details

Published: 13 September 2019

• nodejs

The purpose of NodeJS Forever is to keep a child process (such as your node.js web server) running continuously and automatically restart it when it exits unexpectedly. Forever basically allows you to run your NodeJS application as a process.
Reference: https://stackoverflow.com/a/32944853

A simple CLI tool for ensuring that a given script runs continuously (i.e. forever)
https://github.com/foreversd/forever#readme

A simple example to start and manage Forever
> forever start -a --minUptime 1000 --spinSleepTime 2000 --uid yourapp-stg yourapp.js

-a                        append to logs
--minUptime       1000     wait a second before considering restart
--spinSleepTime 2000    wait two seconds before restarting
--uid                   name the forever process

List all running Forever processes
> forever list

info:    Forever processes running
data:    uid   command       script                          forever pid   id          logfile                            uptime
data:    [0]   yourapp-stg   /usr/bin/node start.js   1668             23197   /home/yourapp/.forever/yourapp.log     0:1:20:14.94

You can restart and stop by name or uid.
Note: uid is incremental, so it may not always be the same number.

> forever restart yourapp-stg
> forever restart 0

> forever stop yourapp-stg
> forever stop 0 

And since you may not want to type or remember all these options, create some helper shell scripts

> start-yourapp.sh
#!/bin/bash
forever start -a --minUptime 1000 --spinSleepTime 2000 --uid yourapp-stg yourapp.js 

> restart-yourapp.sh
#!/bin/bash
forever restart yourapp-stg 

While forever will keep your NodeJS process running, it will not start on reboot.
One simple method to ensure your NodeJS runs on reboot, is to add crontab entry to your forever process. 

Create a crontab entry as the user your app runs as
> crontab -e
@reboot /bin/sh /home/yourapp/crontab-reboot.sh 

And create the reboot script
> crontab-reboot.sh
#!/bin/bash

# export path to NodeJS, Forever
export PATH=/usr/local/bin:$PATH

# cd to location of script
cd /home/yourapp || exit 

# run script, in this case Forever
forever start -a --minUptime 1000 --spinSleepTime 2000 --uid yourapp start.js

So now you application will run .. Forever .. yup.

 

Details

Published: 27 July 2019

• ubuntu
• mongodb

MongoDB is a cross-platform document-oriented database program. 
Classified as a NoSQL database program, MongoDB uses JSON-like documents with schema. 
Source: Wikipedia

MongoDB 4.0 was released on 2018-08-06

New 'wow' features of MongoDB 4.0: 

• multi-document ACID transactions
• data type conversions
• 40% faster shard migrations
• non-blocking secondary replica reads 

While the new features in MongoDB 4.0 are great, 
the latest Ubuntu version 18.04 official repository still installs MongoDB version 3.6

To get the current version of MongoDB

> mongo --version 

To install MongoDB version 4.0, you need to install from MongoDB's repository.

Instructions to install MongoDB 4.0 and some hurdles I encountered follow:

1) Add the MongoDB repo
> sudo vi /etc/apt/sources.list.d/mongodb-org-4.0.list 
deb [arch=amd64] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0 multiverse  

2) Add MongoDB the repo key
> sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4

3) Update your system
> sudo apt-get update

4) Install MongoDB 4.0
> sudo apt-get install mongodb-org

5) Status and restart MongoDB
> sudo systemctl status mongod
> sudo systemctl restart mongod

Hurdles:
If MongoDB does not start, there may be some issues with removing the prior MongoDB version.

Errors I encountered:
error processing archive /var/cache/apt/archives/mongodb-org-server_4.0.10_amd64.deb (--unpack):

error trying to overwrite '/usr/bin/mongod', which is also in package mongodb-server-core 1:3.6.3-0ubuntu1.1
error trying to overwrite '/usr/bin/mongos', which is also in package mongodb-server-core 1:3.6.3-0ubuntu1.1
error trying to overwrite '/usr/bin/bsondump', which is also in package mongo-tools 3.6.3-0ubuntu1

Some potential fixes
> sudo apt --fix-broken install
This by it self did not help

Remove prior MongoDB and other unused packages
> sudo apt autoremove
This did fix the issue and allow me to run MongoDB 4.0

To get version of MongoDB
> mongo --version

Also, if you accidentally tried to get the version from the daemon
> mongodb --version  
MongoDB will start as your user, often sudo/root,
which may cause some MongoDB files to be created as root.

You may have to reset user/group permissions
> sudo chown -R mongodb:mongodb /data/mongodb/

 -End of Document-
Thanks for reading

Details

Published: 17 June 2019

• powershell

Microsoft Message Queuing or MSMQ is a message queue implementation developed by Microsoft and deployed in its Windows Server operating systems.

MSMQ is essentially a messaging protocol that allows applications running on separate servers/processes to communicate in a failsafe manner. A queue is a temporary storage location from which messages can be sent and received reliably, as and when conditions permit. This enables communication across networks and between computers, running Windows, which may not always be connected. By contrast, sockets and other network protocols assume that direct connections always exist.

Source: Wikipedia

If everything is working, messages are added to the queue by one application, and then typically read and removed from the queue by another application.

However, sometimes messages will get 'stuck' in the queue. While this is sometimes due to networking or application changes, the queue may back up if the receiving application is not running or running slower than normal.

Once the underlying problem has been fixed, depending on your application, the receiving application may never be able to catch up with the quantity of messages currently in the queue plus those still being added.

While you can purge the full queue:
Computer Management -> Services and Applications -> Message Queuing
Select your private queue, right click, All Tasks -> Purge

You may want to only purge the "old" messages.
Depending on your application, "old" could be seconds, minutes or hours.

The following PowerShell script will iterate over your private queues and remove "old" messages.
You can set the definition of "old" and you can preview the messages without removing.

###

# remove old messages for all local msmq queues

# msmq = microsoft messaging queuing

#

# 20190530 122800

#   add options $showMsg and $dryRun

# 20190521 122800

#   initial

#

#

# config

#

[System.Reflection.Assembly]::LoadWithPartialName("System.Messaging") | Out-Null

$utf8 = new-object System.Text.UTF8Encoding

$now = Get-Date

# remove messages older than

# $old = $now.AddDays(-1)

$old = $now.AddMinutes(-10)

# show details of message to be removed

$showMsg = 1

# run without removing

$dryRun = 1

#

# run

#

# get queues

$queuePaths = Get-MsmqQueue -QueueType Private | Select -Property QueueName;

if ($dryRun)

{

    echo "dry run; would be "

}

echo "removing old messages for all local msmq queues"

echo ""

echo "nbr queues: $($queuePaths.Length); checking messages older than $($old)"

$queueCounts = Get-MsmqQueue -QueueType Private | Format-Table -Property QueueName,MessageCount;

echo $queueCounts

echo ""

pause

foreach ($queuePath in $queuePaths)

{

    # for proper permissions, prepend .\

    $localQueuePath = ".\$($queuePath.QueueName)"

    echo "queue: $localQueuePath"

   $queue = New-Object System.Messaging.MessageQueue $localQueuePath

   

    # to read ArrivedTime property

    $queue.MessageReadPropertyFilter.SetAll()

   

    # get snapshot of all messages, but uses memory, and slower

   # $msgs = $queue.GetAllMessages()

    # echo "  $($msgs.Length) messages"

    # get cursor to messages in queue

    $msgs = $queue.GetMessageEnumerator2()

   

    # add a message so can test

    # $queue.Send("<test body>", "test label")

    # pause

    $removed = 0

   # foreach ($msg in $msgs)

    while ($msgs.MoveNext([timespan]::FromSeconds(1)))

   {          

       $msg = $msgs.Current

      if ($msg.ArrivedTime -and $msg.ArrivedTime -lt $old)      

      {

           if ($showMsg)

           {

               echo "--------------------------------------------------"

               echo "ArrivedTime: $($msg.ArrivedTime)"

               echo "BodyStream:"

               if ($msg.BodyStream)

               {

                   echo $utf8.GetString($msg.BodyStream.ToArray())

               }

               echo "Properties:"

               echo $msg | Select-Object

               echo ""

           }

         

           try {

               if (!$dryRun)

               {

                   # receive ie remove message by id from queue

                   $queue.ReceiveById($msg.Id, [timespan]::FromSeconds(1))

               }

               $removed++

           } catch [System.Messaging.MessageQueueException] {

               $errorMessage = $_.Exception.Message

               # ignore timeouts              

               if (!$errorMessage.ToLower().Contains("timeout"))

               {

                   throw $errorMessage

               }

           }

      }

   }

    if ($dryRun)

    {

       echo "dry run; would have "

    }

    echo "removed $removed messages from $localQueuePath"

    echo ""

}

pause

#

###

View on GitHub

-End of Document-

Thanks for reading