Blog

Details

GitLab is a web-based DevOps lifecycle tool that provides a Git-repository manager providing wiki, issue-tracking and CI/CD pipeline features, using an open-source license, developed by GitLab Inc.
Source: https://en.wikipedia.org/wiki/GitLab

Both Deploy Keys and Deploy Tokens allow you to checkout code without using your personal GitLab username/password.  And as their name indicates, they are best suited for deployments, usually automated/scripted. What are their features?

Deploy Keys

  • Utilize SSH keys (public and private)
  • Checkout via SSH (requires port 22 open)
  • GitLab will store the public SSH key
  • The server doing the checkout will have to have the private SSH key
  • Allows read and/or write of repository
  • More steps for setup

Deploy Tokens

  • Use a generate once password ie token
  • Check via HTTPS (requires port 443 open, more common)
  • Token can have an expiration date
  • Allows read only of repository
  • Simpler setup

How to create and use Deploy Keys (SSH)

By default, when you use SSH, it reads from ~/.ssh/id_rsa
But that is your personal SSH key

Create a SSH key for GitLab deployments with a unique name
Login to the server where you will clone/deploy your code
> ssh-keygen -t rsa -b 4096 -f ~/.ssh/gitlab_deploy_rsa

Tell ssh about the new gitlab key
> vim ~/.ssh/config
Host gitlab.yourdomain.com
  Preferredauthentications publickey
  IdentityFile ~/.ssh/gitlab_deploy_rsa

Note: for the Host, use the host GitLab references
GitLab: Repo: Clone button
This email address is being protected from spambots. You need JavaScript enabled to view it.:dir/repo.git

For security, and to allow SSH, make the config read only by you
> chmod 600 ~/.ssh/config

So GitLab knows about your new key, create a GitLab Deploy Key

GitLab: Repo: Settings -> Repository, scroll to Deploy Keys

Copy the contents of your public key ~/.ssh/gitlab_rsa.pub to the Key section

Back on your server,
Test that SSH access works
> ssh -vT This email address is being protected from spambots. You need JavaScript enabled to view it.
-T Disable pseudo-terminal allocation ie just give return result "as is"
-v verbose

If SSH times out, try a public site to ensure port 22 is open
> ssh -vT gitlab.com

If SSH gitlab.com does not work, then enable port 22 on your server, most likely via it's firewall.
If SSH gitlab.com does work, then maybe SSH is not enabled or accessible on your GitLab server. Review your GitLabs firewall ie iptables, tcp wrappers ie /etc/hosts.deny and /etc/hosts.allow, and the SSH daemon configuration ie /etc/sshd_config
Source: https://unix.stackexchange.com/a/406264/265278

Assuming
> ssh -vT This email address is being protected from spambots. You need JavaScript enabled to view it.
finally works

You should be able to checkout your code using SSH, using
> git clone This email address is being protected from spambots. You need JavaScript enabled to view it.:dir/repo.git target_dir
which is a slight shorthand for
> git clone ssh://This email address is being protected from spambots. You need JavaScript enabled to view it./dir/repo.git target_dir

Note: If you cannot get SSH working, you can try using Deploy Tokens, which uses HTTPS, which is more often accessible.

 

How to create and use Deploy Tokens (HTTPS)

Create a GitLab Deploy Token

GitLab: Repo: Settings -> Repository, scroll to Deploy Tokens

 

After creating the Deploy Token, make sure to copy and save the token

 

 From your server, you can checkout your code using
> git clone https://<username>:<token>@gitlab.yourdomain.com/dir/repo.git target_dir 

After cloning, you can simply Pull changes
> git pull

So, should you use SSH or HTTPS?
Ideally you should use SSH keys as they are a way to identify trusted computers, without involving passwords, but HTTPS is the easiest to set up on the widest range of networks and platforms.
Source: https://stackoverflow.com/questions/11041729/why-does-github-recommend-https-over-ssh

 

-End of Document-
Thanks for reading

 

 

 

 

Details

When you create a AWS RDS MySQL instance, you will be asked to also create a master username/password.  The master RDS user acts like the root user of all it's databases, with full access to SQL commands such as DROP and CREATE.  While you can use this root user within your applications, you should create an application specific database user with limited privileges, which will prevent harm from accidental and malicious use. 

Note: AWS (Amazon Web Services) RDS (Relational Database Service) is a managed service, and it doesn't provide SYS access (SUPER privileges) 

View permissions of root user

Show grants for RDS MySQL root user ie master username

SQL: 
SHOW GRANTS FOR 'professor';

Result:
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, PROCESS, REFERENCES, INDEX, ALTER, SHOW DATABASES, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER ON *.* TO `professor`@`%` WITH GRANT

Create new application user

Create a new user to be used by just the application

SQL:
CREATE USER 'express_app'@'%' IDENTIFIED BY '20charRandomPwd';

Note: The username should be related to the application, and the password does not need to be readable as it will only be used within the application configuration. 

Grant permissions

For application users, remove:
CREATE, DROP, RELOAD, PROCESS, REFERENCES, INDEX, ALTER, SHOW DATABASES, CREATE TEMPORARY TABLES, LOCK TABLES, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT

Leaving the basics:
SELECT, INSERT, UPDATE, DELETE, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, TRIGGER

Note: MySQL permissions reference

SQL:
GRANT SELECT, INSERT, UPDATE, DELETE, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, TRIGGER ON *.* TO `express_app`@`%` WITH GRANT OPTION;

Note: If using multiple databases in a RDS instance, then grant access only to the required databases ie ON `planet.*` TO `express_app`

You can now use the newly created, and restricted, database user express_app in your application.

-End of Document-
Thanks for reading

Details

What are PHP tags? 

When PHP parses a file, it looks for opening and closing tags, which are <?php and ?> which tell PHP to start and stop interpreting the code between them. Parsing in this manner allows PHP to be embedded in all sorts of different documents, as everything outside of a pair of opening and closing tags is ignored by the PHP parser. 

PHP includes a short echo tag <?= which is a short-hand to the more verbose <?php echo.
PHP also allows for short open tag <? (which is discouraged since it is only available if enabled using the short_open_tag php.ini configuration file directive, or if PHP was configured with the --enable-short-tags option). 

If a file contains only PHP code, it is preferable to omit the PHP closing tag at the end of the file. This prevents accidental whitespace or new lines being added after the PHP closing tag, which may cause unwanted effects because PHP will start output buffering when there is no intention from the programmer to send any output at that point in the script.

Source: php.net

 With PHP 7.4, per the PHP RFC: Deprecate PHP Short open tags, a deprecation noticed is issued for usage of the short-hand tag, and in PHP 8.0, the short-hand tag will be removed.
While the <?=short tag is unaffected, for completeness, using the full PHP tag should be preferred.
And using the short-hand tags generally make code less portable as it requires a non common ini change.

 

Ok, so for new projects you can start using <?php ?>.

But, maybe you inherited an old project which uses the short-hand tags, which was more common in the earlier pre PHP 7 days.  And you don't want to manually replace all the tags in 100s of files.


The following are some command line find + sed calls to replace PHP short-hand and normalize PHP tags across multiple files: 

What is going to be replaced and the sed command used:

Description

From

To

sed

new line; start of file

'<?\n'

'<?php\n'

-e 's/^<?\n/<?php\n/'

block

'<?\n'

'<?php\n'

-e 's/<?\n/<?php\n/g'

inline

'<? '

'<?php '

-e 's/<? /<?php /g'

echo shorthand

'<?='

'<?php echo '

-e 's/<?\=/<?php echo /g'

lowercase

'<?PHP'

'<?php'

-e 's/<?PHP/<?php/g'

no closing tag

'?>'

'\n'

-e 's/^?>/\n/'

 

Note: lowercase: PHP functions are case insensitive. Why? Just how PHP evolved.  "all the functions introduced essentially felt like tags in an HTML document and since HTML tags were case insensitive, he chose function names in PHP to be case insensitive."
Source: stackoverflow
But keeping everything lowercase is more normal, compared to other languages. 

Note: no closing tag: If your code or templates have mixed PHP/HTML content, then you do not want to remove the closing tags.  The intent is for files which are only PHP, you can and should remove the closing tag.


Make sure to make a backup, or using version control such as git or svn.

OK, now on to the actual commands:

First, optionally, convert window new lines to unix new lines, so the patterns in sed can match
> find . -iname '*.php' -type f -print0 |xargs -0 -I file dos2unix file 

Details: Finds and print all php files in the current ie . and sub directories
find . -iname '*.php' -type f -print0 

Details: Takes the first result and stores to 'file'
xargs -0 -I file 

Then, test the sed replacement
> find app/ -iname '*.php' -type f -print0 |xargs -0 sed -n -e 's/^<?\n/<?php\n/p' -e 's/<?\n/<?php\n/g' -e 's/<? /<?php /gp' -e 's/<?\=/<?php echo /gp' -e 's/<?PHP/<?php/gp' -e 's/^?>/\n/p'; 

Note: sed options:
-e = pattern
-i = inplace
-n = --quite
p = print current pattern

Run the replacements:

Removing closing tag:
> find app_src/ -iname '*.php' -type f -print0 |xargs -0 sed -i -e 's/^<?\n/<?php\n/' -e 's/<?\n/<?php\n/g' -e 's/<? /<?php /g' -e 's/<?\=/<?php echo /g' -e 's/<?PHP/<?php/g' -e 's/^?>/\n/';

Keeping closing tag due to mixed PHP/HTML:
> find html_templates/ -iname '*.php' -type f -print0 |xargs -0 sed -i -e 's/^<?\n/<?php\n/' -e 's/<?\n/<?php\n/g' -e 's/<? /<?php /g' -e 's/<?\=/<?php echo /g' -e 's/<?PHP/<?php/g';

 

Now all your PHP short-hand tags should be replaced and normalized.

Of course, verify using version control diffs and test your code.

 

-End of Document-
Thanks for reading

Details

After creating and starting an AWS EC2 instance, and installing nginx and the latest php, 
you will probably want to configure the application environments, deployment user, permissions, and re-configure nginx and php.

Goal

Create a user to be the owner of the deployed code (application abbreviation, internal code, etc)
               professor

Create environment-based directories for the application (/data could be an external volume)
       /data/prod/fry/www/html
       /data/dev/fry/www/html
       /data/qa/fry/www/html

Create environment-based urls for the application
       fry.domain.com
       fry-qa.domain.com
       fry-dev.domain.com

Note: while this should allow multiple apps per EC2, it may be better and simpler to have one app per EC2, in which case, there would not necessarily be a need for the /fry directory, although it may add clarity when view logs and debugging deployments.

 

Setup

Note: To facilitate getting stuff done, and to minimize permission problems, disable selinux.
If you have utilized and configured used services with selinux successfully before, then keep enabled and configure it appropriately.
https://serverfault.com/questions/30796/reasons-to-disable-enable-selinux

Temporarily disable selinux
> # sudo setenforce 0 

Permanently disable selinux
> sudo vi /etc/selinux/config
SELINUX=disabled

Create or obtain your ssh key pairs
You can create key pairs from AWS EC2, ssh-keygen, or putty
Note: Be sure to securely store/backup your private key and distribute the public key as needed

Add a user to be used for deploying code
Note, there could be a user per app too, but for now, one user, 
which could be based on company name, or something generic
> sudo adduser professor

Change to the app user
> sudo su - professor

Enable ssh

Enable ssh access using key pairs

Make sure in /home/professor
> pwd

Create the file to store the public key
> mkdir .ssh
> chmod 700 .ssh
> touch .ssh/authorized_keys
> chmod 600 .ssh/authorized_keys

Copy in the public key (pem) for this user
> vi .ssh/authorized_keys
> # cat >> .ssh/authorized_keys # append pasted in text, ctrl c

Note: The public key should be in the format
ssh-rsa ABC…123== rsa-key-20200110

Putty on Windows will store the format as
---- BEGIN SSH2 PUBLIC KEY ----
Comment: "rsa-key-20200110"
ABC…123
---- END SSH2 PUBLIC KEY ----

You can use PuttyGen, open the private key, and view the proper format
or do some simple text editing of the surrounding delimiters.

From another shell, you should now be able to
> ssh -i professor.pem ec2host

Permissions

Add professor to nginx group, to view logs etc
> sudo usermod -a -G nginx professor

Add nginx to professor group as app dirs will be owned by professor
> sudo usermod -a -G professor nginx

Make app dirs (-p = recursive)
The application directory can be named for the application, an abbreviation, internal code/name, etc
> sudo mkdir -p /data/prod/fry/www/html
> sudo mkdir -p /data/dev/fry/www/html
> sudo mkdir -p /data/qa/fry/www/html

Change perms to professor (nginx) (-R recursive)
> sudo chown -R professor:professor /data/prod/fry
> sudo chown -R professor:professor /data/dev/fry
> sudo chown -R professor:professor /data/qa/fry

Keep data and prod/dev/qa owned by root, but accessible to professor
> sudo chown root:professor /data/prod
> sudo chown root:professor /data/dev
> sudo chown root:professor /data/qa
> sudo chown root:professor /data

Configure

Organize nginx sites in a new dir sites.d
Note: Inspired by Debian configuration
> sudo mkdir /etc/nginx/sites.d

Add sites.d to nginx conf
> sudo vi /etc/nginx/nginx.conf
    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites.d/*.conf;

Add sites conf to sites.d
Configure prod/dev/qa in same file, or separate files
Note: This is just a modification of the default nginx config, you may have to adjust it per your framework or application
> sudo vi /etc/nginx/sites.d/fry.domain.com.conf
server {
        listen 80;
        server_name fry.domain.com;
        root   /data/prod/fry/www/html/;  
        access_log /var/log/nginx/fry.domain.com_access_log;
        error_log  /var/log/nginx/fry.domain.com_error_log error;
        location / {
               # modify based on framework
               try_files $uri $uri/ /index.php?$query_string;
        }
        # include php; replace if framework requires
        # index index.php index.html index.htm;
        # location ~ \.php$ { }
        include /etc/nginx/default.d/php.conf;      
}

server {
        listen 80;
        server_name fry-dev.domain.com;
        root   /data/dev/fry/www/html/; 
        access_log /var/log/nginx/fry-dev.domain.com_access_log;
        error_log  /var/log/nginx/fry-dev.domain.com_error_log error;
        location / {
               # modify based on framework
               try_files $uri $uri/ /index.php?$query_string;
        }
        # include php; replace if framework requires
        # index index.php index.html index.htm;
        # location ~ \.php$ { }
        include /etc/nginx/default.d/php.conf;      
}

server {
        listen 80;
        server_name fry-qa.domain.com;
        root   /data/qa/fry/www/html/;   
        access_log /var/log/nginx/fry-qa.domain.com_access_log;
        error_log  /var/log/nginx/fry-qa.domain.com_error_log error;
        location / {
               # modify based on framework
               try_files $uri $uri/ /index.php?$query_string;
        }
        # include php; replace if framework requires
        # index index.php index.html index.htm;
        # location ~ \.php$ { }
        include /etc/nginx/default.d/php.conf;      
}

Configure default ec2 to go to a dev site
> sudo vi /etc/nginx/sites.d/ec2.conf

server {
        listen 80;
        server_name ec2-1-2-3-4.us-east-9.compute.amazonaws.com;
        # point aws ec2 to a dev location
        root   /data/dev/fry/www/html/; 
        access_log /var/log/nginx/fry-dev.domain.com_access_log;
        error_log  /var/log/nginx/fry-dev.domain.com_error_log error;
        location / {
               # modify based on framework
               try_files $uri $uri/ /index.php?$query_string;
        }
        # include php; replace if framework requires
        # index index.php index.html index.htm;
        # location ~ \.php$ { }
        include /etc/nginx/default.d/php.conf;      
}

Support long AWS EC2 server names, using a new config
> sudo vi /etc/nginx/conf.d/http.conf
server_names_hash_bucket_size  128;

Validate config before restart
> sudo nginx -t

Restart nginx
> sudo systemctl restart nginx

Remove prior test page, if any
> sudo rm /usr/share/nginx/html/info.php

Create a test php page
Assuming the default AWS EC2 page goes to the development dir (ec2.conf)
> sudo vi /data/dev/fry/www/html/info.php
<?php

echo date(DATE_RFC2822);
phpinfo();

Verify
http://ec2-1-2-3-4.us-east-9.compute.amazonaws.com/info.php

Application code can be deployed to separate environment-based directories as professor


-End of Document-
Thanks for reading

  1. AWS RedHat install nginx and php
  2. AWS EC2 resize partition
  • You are here:  
  • Home