Inotify limit reached under PT Magic

Today I ran into a problem with PT Magic (PTM) that the inotify limit was reached on Ubuntu. Inotify (inode notify) is a Linux kernel subsystem that acts to extend filesystems to notice changes to the filesystem, and report those changes to applications. Just as a future reference for myself and as a reference to others a very quick write up how to solve this.

You can get your current inotify file watch limit by executing:

$ cat /proc/sys/fs/inotify/max_user_watches

You can set a temporary new limit with:

$ sudo sysctl -w fs.inotify.max_user_watches=16384

If you like to make your limit permanent use:

$ echo fs.inotify.max_user_watches=16384 | sudo tee -a /etc/sysctl.conf
$ sudo sysctl -p

That’s it. The problem should be solved. At least this is the workaround. I have submitted an issue to TP Magic so that they can fix it properly. We will see.

Advertisements

Validating a json file from the command line in Linux

Today I needed to validate a json file on one of my servers. It turns out there is a simple nodejs program or actually a linter to do this with.

Just execute the following line and you are in business.

sudo npm install jsonlint -g

This assumes nodejs is already installed. If not, execute the following line.
sudo apt-get install -y nodejs npm

To validate a json file run the following command.

jsonlint -qc settings.analyzer.json

When running the program the first time I ran into the problem that the json file contains comments. Yes, I know this is not in line with the spec, but it is damn handy for understanding the file. So I needed to strip the comments to valiate the file. After googeling around a bit I found an awk command to do just this. Try the following command.

awk '{sub(/\/.*$/,"")}1' settings.analyzer.json > settings.analyzer.json

Now run jsonlint again and it should work.

P.s. I know the title of this post says Linux, but jsonlint will also work under any other system that nodejs runs on.

Hope this saves someone some time.

Using the hosts file on Windows with ports

As you might or might not know I’m running Profit Trailer (PT) on a VPS. To be safe the GUI is not accessible to the outside world, but only through an SSH tunnel. It has been irritating me for a while that the URLs are not readable, e.g. localhost:8081. I wanted readable URLs without ports in them. Today I finally figured out how to do this using a combination of host file and the Windows networking tool netsh.

Continue reading

Inspecting and replaying requests with ngrok

A very cool feature of ngrok is the web interface. With this interface you can inspect requests and replay them. Very handy for debugging purposes.

The web interface can be reached on http://localhost:4040.

In the web interface you can view both the request received and response sent. The default view of the request and response is a summary of the post data, but it can be viewed in raw format or binary format. This is really useful when debugging webhooks.

Another great feature of this web interface is the replay functionality. Instead of having to run through the code again to get to the problem, you can just hit the resend button and the request will be resent.

Flask error loading config file in instance directory

I have been dealing with a really frustrating problem today. Yesterday my code was working and today it was failed without any code changes.

This is the piece of code causing the problem.

app = Flask(__name__, instance_relative_config=True)
app.config.from_object('config')
app.config.from_pyfile('config.cfg')

And this is the error.

Traceback (most recent call last):
  File "run.py", line 1, in 
    from project import app
  File "C:\Python27\env\vod_test\src\project\__init__.py", line 37, in 
    load_config()
  File "C:\Python27\env\vod_test\src\project\__init__.py", line 33, in load_config
    app.config.from_pyfile('config.cfg')
  File "C:\Python27\env\vod_test\lib\site-packages\flask\config.py", line 129, in from_pyfile
    with open(filename) as config_file:
IOError: [Errno 2] Unable to load configuration file (No such file or directory): 'C:\\Python27\\env\\vod_test\\var\\project-instance\\config.cfg'

Flask is trying to load the config file from C:\\Python27\\env\\vod_test\\var\\project-instance\\config.cfg instead of from C:\\Python27\\env\\vod_test\\project\\instance\\config.cfg. Where is the var coming from? And why is there a hyphen between project and instance?

The strange thing was that the problem only occurred when I was running my test cases from the command line. In PyCharm the code was working. How frustrating. It turned out that PyCharm was using a different virtualenv than the command line. After comparing the output of ‘pip freeze’ I discovered that some libraries had different versions.

To solve the problem I added the following code.

import os
from flask import Flask


def load_instance_config_fix(filename):
    config = {}
    with app.open_instance_resource(filename) as f:
        for line in f:
            if line[0] == "#":
                continue
            name, value = line.partition("=")[::2]
            config[name.strip()] = value.strip()
    return config


def handle_incorrect_instance_path(f):
    def handle_load_instance_config():
        try:
            f()
        except IOError:
            app.instance_path = os.path.join(os.path.abspath(os.curdir), 'instance')
            config = load_instance_config_fix('config.cfg')
            app.config.update(config)

    return handle_load_instance_config


@handle_incorrect_instance_path
def load_config():
    app.config.from_object('config')
    app.config.from_pyfile('config.cfg')


app = Flask(__name__, instance_relative_config=True)

load_config()


@app.route('/')
def hello_world():
    return 'Hello, World!'

Just some nice syntactic sugar with a decorator to keep the code nice and concise.

Another way to solve the problem is by setting the instance_path when initializing Flask. What fun would that be 🙂

import os
from flask import Flask


app = Flask(__name__, instance_path=os.path.join(os.path.abspath(os.curdir), 'instance'), instance_relative_config=True)
app.config.from_object('config')
app.config.from_pyfile('config.cfg')


@app.route('/')
def hello_world():
    return 'Hello, World!'

Back to real coding 🙂

Git quickstart

Just some short notes to remember how to get started with Git.

Check version

$ git --version

Setting-up

Set the following configuration items if using Git for the first time, as these values are used for each commit made.

$ git config --global user.name "Your Name"
$ git config --global user.email "[email protected]"

Initializing a new Git repository

Make sure you are in the top-level folder of your project and then run the following command.

$ git init

Create the .gitignore file

Git has a special file called ‘.gitignore’ that allows you to specify which files to NOT include in your repository.  The following file is based on the recommended .gitignore file for Python from GitHub.

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
 
# PyCharm files
.idea/
 
# Instance Folder - used for sensitive configuration parameters
/instance

Setting remote repository
Make sure you are in the top-level folder of your project and set up the remote repository.

$ git remote add origin [email protected]:nidkil/name-of-repository.git

Check the remote repository has been set correctly.

$ git remote -v

Staging files
The standard flow for adding files to your git repository is to create/edit the files, add them to the staging area, and then commit them to your repository.

$ git status
$ git add .
$ git status

Initial commit
The following command makes the first commit to the repository, including a message describing the commit.

git commit -m "Initial version"

You can confirm that the commit was successful by checking the log of the repository.

git log

Push to the remote repository
To push the local Git repository to the remote Git repository.

$ git push -u origin master

Working with a feature branch
A common method of developing features is to create a feature branch where you implement specific functionality and then merge the changes back into the master or development branch. Follow the process: create a new feature branch, make changes and merge changes back into the ‘master’ branch.

Create a new feature branch.

$ git checkout -b name_of_feature_branch

Check new feature branch was created (the star next to ‘name_of_feature_branch’ branch indicates it is the current branch that we are working in).

$ git branch

The following commands stage, commit and merge the changes into the local repository.

$ git add .
$ git status
$ git commit -m "Added use of templates and Bootstrap"
$ git checkout master
$ git merge add_templates

Delete the branch.

$ git branch -d add_templates

Add a version tag to your update.

$ git tag -a v0.1 -m "version 0.1"

You can view the changes of a version tag.

$ git show v0.1

You can list tags.

$ git tag

Push the changes into the remote repository.

$ git push -u origin master

Check difference between local and remote repository.

$ git diff master origin/master

Overwrite local master with remote master.

$ git fetch --all
$ git reset --hard origin/master

Create a branch of the local master before overwriting it with the remote master.

$ git checkout master
$ git branch 
$ git fetch origin master
$ git reset --hard origin/master

After this, all of the old commits will be kept in the . However, uncommitted changes (even staged) will be lost. Make sure to stash and commit anything you need.

Oops working in master instead of a branch

Happens to me a lot I start working on a change and after a while realize I am working on my local master instead of a feature branch. Continue working or through away your changes and start again in a new feature branch. Neh, just checkout the new feature branch and the changes will automatically move to it.

$ git checkout -b name_of_feature_branch

Just do a git branch and git status to see that it actually worked 🙂

Cool isn’t it? Who doesn’t love git?