Gitolite and gitweb on Synology DS415+ with standard packages

Recently I put my hands on a Synology DS415+ Network-Attached Storage-unit. This device does a lot more than just storage. A whole bunch of services and applications can be installed via a web-interface. The device itself is running, by default, a “lightweight” Linux based on busybox and is filled up with custom builds of services and applications.

Synology DS415+

When I was asked to install a git-server/-service on it, there were already a lot of other services running. There are two git-servers provided via the Synology-package-manager: Gitlab and a “git-server”. I tried Gitlab which comes in a Docker-container. This was too much for the DS415+ – CPU and Memory consumption was going up immediately and finally crashing the whole system. I de-installed everything after a reboot.

The “git-server” was very simplistic and not exactly what the team was looking for. Installing git-repos on a per-user basis was too simple compared to the needs expressed by the team. However, git-server has to be installed and activated in order to have a git-executable available.

I’m always opting for gitolite in this kind of situations. I looked into how the internet was suggesting to install it on such a device. All of the How-Tos I found were suggesting to bootstrap the installation with ipkg to have the possibility to install much more packages from the nslu2-project.

The team wanted to keep the possibility of updating the DS415+ with the Synology-update process and was thus hesitant to add this package-source. Especially as the bootstrapping description mentioned explicitly that the “official” update-process might suffer after installation.

I finally succeeded to install gitolite and a gitweb-clone (gitphp) on the DS415+ without spoiling the system with external package-systems. Here is what I did:

1- Create user ‘git’ via the standard user-interface (have it creating a home-dir). Login with ssh as root to and  change to /sbin/no-login to /bin/sh for ‘git’ (/etc/passwd)
2- Login as user ‘git’ via ssh and
chmod 755 $PWD

This is necessary to make ssh-logins work with the public-private-key-method.

3- Create a directory for the user git to keep executables and add this path it to the user’s PATH variable.
mkdir ~/bin
echo 'export PATH=$PWD/bin:$PATH' >> .profile
4- Install mktemp for synology-intel-platform

Get the mktemp-package file (ipkg), unpack it (on a PC preferably) and copy /opt/bin/mktemp-mktemp from the archive to /usr/local/bin . This is actually a kludge to get mktemp which is not enabled by default. Installing it to /usr/local/bin should make it survived updates in the future.

5- Install Perl via the standard Synology package-manager.
6- Put your ssh-public-key to git’s directory as <your-future-gitolite-username>.pub
7- Logged in a ‘git’ install gitolite. I simply followed this guide.
gitolite/install -ln
gitolite setup -pk your-name.pub
8- Everything is readly now and gitolite-admin can be cloned (on your host as the user who owns the corresponding private key).
git clone git@host:gitolite-admin

The rest is standard gitolite administration stuff. Read http://gitolite.com/gitolite/gitolite.html#basic-admin .

9- Gitweb with Gitphp

I was unable to master the perl/cgi-setup of the DS415+ and gave up quickly when trying to make run gitweb. I found Gitphp which is a Gitweb-clone written in PHP. It was quite simple to set it up.

I had to enable the “Web Service” via the admin-interface. I also ticked ‘Enable personal website’. This way I could install gitphp directly to ~/www of the git-user.

mkdir ~/www
cd ~/www
git clone https://github.com/xiphux/gitphp.git .

I started with a simple configuration by adjusting some self-explaining variables in ~/www/config/gitphp.conf.php
after having it copied from gitphp.conf.php.example. More configuration remains to be done.

gitphp

Create an OpenCV 2 matrix from a JPEG-image in a buffer #2

Even before publishing my previous article about OpenCV and JPEG-images from buffers, I found the cv::imdecode()-function. But it was Peter’s comment which made me really take a look and compare the two methods.

Here’s the code I’m using to create a cv::Mat from a JPEG-image in a buffer using cv::imdecode().

void handleImage_o(const uint8_t *buffer, size_t size)
{
 std::vector<uint8_t> v;
 v.assign(buffer, buffer+size);
 cv::Mat img = cv::imdecode(v, CV_LOAD_IMAGE_COLOR);
}

Yes, it is only 3 lines compared to the 20+ lines using libjpeg… My initial doubt about imdecode() was the speed of decoding. Will it be as fast as my manual approach? Well, almost!:

This quick and dirty benchmark is of course subjective and not usable for reference as it lacks spread, but for my use case it is enough to help me make a decision.

I decode 100 times the same image (4288×2848 pixels) and just measure the time(1) it took. Here are the results with imdecode()

real 0m14.600s
user 0m13.652s
sys 0m0.956s

versus libjpeg-turbo.

real 0m12.476s
user 0m11.736s
sys 0m0.760s

My version using libjpeg-turbo is roughly 14% faster than imdecode() from OpenCV 2.4.9.1 .

Is 14% justifying the complexity of the libjpeg-based code? That’s up to you to decide.

Create an OpenCV 2 matrix from a JPEG-image in a buffer

I recently started to take a look at OpenCV for doing some (programmatic) image processing for a small project I’ll maybe talk about later on.

My problem: in my program I receive JPEG-images in a buffer over a network connection and not by opening a file. Now my question was: how to create an OpenCV Mat(rix) from this buffer? Normally should not fill a whole post, but it took me too much time to develop to not document it now.

Strange enough, even on Stackoverflow I only found partial answers. I did a half-hearted web-search and found nothing really complete. Here are the facts I gathered (no guarantee for their correctness, but this is my current state of understanding)

  1. OpenCV does not directly support the importation of JPEGs from a buffer (but it does support the reading of a file).
  2. You need to use a libjpeg-variant to create an uncompressed image which then can be imported into the Matrix
  3. OpenCV  needs images in BGR-colorspaces to be processed further on, by default images are in RGB-colorspace

When doing this kind of  processing I want to limit copies and processing time. Here’s the code I came up with:

class ImageProcessing
{
  struct jpeg_decompress_struct cinfo;
  struct jpeg_error_mgr jerr;

public:
  ImageProcessing()
  {
    cinfo.err = jpeg_std_error(&jerr);
    jpeg_create_decompress(&cinfo);
  }

  ~ImageProcessing()
  {
    jpeg_destroy_decompress(&cinfo);
  }

  void handleImage(uint8_t *buffer, size_t size)
  {
    jpeg_mem_src(&cinfo, buffer, size);

    switch (jpeg_read_header(&cinfo, TRUE)) {
    case JPEG_SUSPENDED:
    case JPEG_HEADER_TABLES_ONLY:
      return;
    case JPEG_HEADER_OK:
      break;
    }

    cinfo.out_color_space = JCS_EXT_BGR;

    jpeg_start_decompress(&cinfo);

    cv::Mat src = cv::Mat(
                      cv::Size(cinfo.output_width, cinfo.output_height),
                      CV_8UC3);

    while (cinfo.output_scanline < cinfo.output_height) {
      JSAMPLE *row = src.ptr(cinfo.output_scanline);
      jpeg_read_scanlines(&cinfo, &row, 1);
    }

    jpeg_finish_decompress(&cinfo);

    cv::imshow("test", src);
    cv::waitKey(0);
  }
};

Summary: I decode the JPEG-buffer using libpjeg-turbo (pre-installed) directly into the buffer allocated by cv::Mat, using the BGR-colorspace as expected by OpenCV.

Line 30: is where I’m telling libjpeg to decode directly to BGR-colorspace
Line 39-40: does the decoding line-by-line directly into the cv::Mat-buffer correspond to the line which can be easily retrieved by the cv::Mat::ptr()-method.

Careful, this code is just a snippet showing how I did. It is neither complete nor self-standing.

‘Gallery Carousel Without JetPack’-plugin and htaccess-protected wp-admin.

I maintain several WordPress-installations and I received the request to protect the wp-admin-subfolder with a second level of password protection using .htpasswd and .htaccess. Which makes a second user/password-query appear on the browser when trying to access the admin-folder. It works fine, no problem with that.

However, I’m also using the ‘Gallery Carousel Without JetPack‘-plugin to enable simple and nice full-screen galleries. It turned out, that this plugin is requiring admin-ajax.php to request comments (via JQuery/Ajax) which are displayed for each image. As this file is located in the wp-admin-folder all anonymous users (so all site-visitors) were prompted for username and password when opening any gallery.

I don’t know whether there is another/better way for plugins to fetch comments with Ajax, but to fix this problem on my site, I excluded the admin-ajax.php from .htaccess-protection by adding

<Files "admin-ajax.php">
    Allow from all
    Satisfy any
</Files>

on the top of my wp-admin/.htaccess– file. Brett Batie has made a nice short post about .htaccess-exclusions – though he forget to add the closing </Files> to his single-file-example.

This is my complete .htacces-file now:

<Files "admin-ajax.php">
    Allow from all
    Satisfy any
</Files>

AuthType Basic
AuthName "Secure area"
AuthUserFile /<absolute-path-to-wordpress-on-the-server>/wp-admin/.htpasswd
AuthGroupFile /dev/null
require valid-user

Python – my personal FAQ with StackOverflow

I’m not ashamed, no, not all … really! ;-): I wrote my first, worth-mentioning Python-script only in November 2014. I’m not new to script-languages, but so far I haven’t had the need nor really the opportunity to write a Python-based one myself from scratch.

The task was to write a small text-processor (others might call it a parser) for a proprietary and simple, linear (no branching), variable-argument-count, command-based language – to debug scripts written in this language.

When advancing, and thus learning how to write Python, I extensively used StackOverflow.com for finding answers to my questions. In retrospective, I figured out that in fact all the answers I needed were from SO. Reading my profile became a quite an interesting collection of Qs and As – so I thought, why not making a post about it, not the least for my own reference. My script is written in Python 3 (so print is now a function).

Python – language in general

Sometimes, while coding, I sometimes don’t want to fill an ifelse-block. How about ’empty if statement’ in pythonpass is an empy block, like {} is in C

When writing the grammer for my parser I needed to Determine the type of an object?, or in reality the type of a variable. type() does that.

Referencing to function/method-arguments in the callee as a list or dict is answered here: What does ** (double star) and * (star) do for Python parameters? This allows variable number of arguments for callers.

What does `if __name__ == “__main__”:` do? – good question, seems to be the main-function in Python. The answer is good.

Ever wondered why regex-strings are prefixed with rWhat exactly do “u” and “r” string flags do in Python, and what are raw string literals?

That all values are passed by reference is explained here: How do I pass a variable by reference?

I haven’t understood all quirks for Unicode yet, but having set LANG to a UTF-8 based language is important. I had to passthru UTF-8-files to my output and my system was LANG=C I had UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xef in position 1.

Why do people write #!/usr/bin/env python on the first line of a Python script? The shebang-line in a Python-script should use env.

Argparse optional positional arguments? Positional arguments are args without a switch.

How to calculate a mod b in python? Like in Perl and C: with %.

When sourcing files which are located next to your script, it is nice to know how to get the path of script.

Tokens, Strings

When parsing you need to tokenize. Search engines gave me this answer to ‘tokenizer python’: Pythonic way to implement a tokenizer: re.Scanner – an undocumented class, utterly useful for this task.

How to check whether a string is emtpy in Python: Most elegant way to check if the string is empty in Python?: An empty string is False.

Display number with leading zeroszfill will fill up (from the left) with zeros.

How to print in Python without newline or space? Printing without a line-break in Python3 is done with arguments to print.

Dicts and lists

What is the preferred syntax for initializing a dict: curly brace literals {} or the dict() function?

How to Add to a dictionary in Python? Dead simple. Delete an element from a dictionary intuitive.

Use len() to find out How to get the size of a list. And how to iterate over dictionary sorted by key?

Almost everything is copied or passed by reference in Python when you assign or return a variable. Modifying a list while iterating over it, requires to know How to clone or copy a list in Python? Copying lists while keeping the references to its elements is done with slicing.

Map/combine two lists into a dictionary in Python where one list is the keys and one the values is done with zipHow can I merge two Python dictionaries in a single expression? for merging two dicts. Updating a dict with a dict Python append dictionary to dictionary replaces elements in target dict if elements with the same key exists.