Inotify limit reached under PT Magic

Today I ran into a problem with PT Magic (PTM) that the inotify limit was reached on Ubuntu. Inotify (inode notify) is a Linux kernel subsystem that acts to extend filesystems to notice changes to the filesystem, and report those changes to applications. Just as a future reference for myself and as a reference to others a very quick write up how to solve this.

You can get your current inotify file watch limit by executing:

$ cat /proc/sys/fs/inotify/max_user_watches

You can set a temporary new limit with:

$ sudo sysctl -w fs.inotify.max_user_watches=16384

If you like to make your limit permanent use:

$ echo fs.inotify.max_user_watches=16384 | sudo tee -a /etc/sysctl.conf
$ sudo sysctl -p

That’s it. The problem should be solved. At least this is the workaround. I have submitted an issue to TP Magic so that they can fix it properly. We will see.

Advertisements

Validating a json file from the command line in Linux

Today I needed to validate a json file on one of my servers. It turns out there is a simple nodejs program or actually a linter to do this with.

Just execute the following line and you are in business.

sudo npm install jsonlint -g

This assumes nodejs is already installed. If not, execute the following line.
sudo apt-get install -y nodejs npm

To validate a json file run the following command.

jsonlint -qc settings.analyzer.json

When running the program the first time I ran into the problem that the json file contains comments. Yes, I know this is not in line with the spec, but it is damn handy for understanding the file. So I needed to strip the comments to valiate the file. After googeling around a bit I found an awk command to do just this. Try the following command.

awk '{sub(/\/.*$/,"")}1' settings.analyzer.json > settings.analyzer.json

Now run jsonlint again and it should work.

P.s. I know the title of this post says Linux, but jsonlint will also work under any other system that nodejs runs on.

Hope this saves someone some time.

Using the hosts file on Windows with ports

As you might or might not know I’m running Profit Trailer (PT) on a VPS. To be safe the GUI is not accessible to the outside world, but only through an SSH tunnel. It has been irritating me for a while that the URLs are not readable, e.g. localhost:8081. I wanted readable URLs without ports in them. Today I finally figured out how to do this using a combination of host file and the Windows networking tool netsh.

Continue reading

Playing with W3C Annotation AKA Web Annotation

Today I was playing around with W3C Annotation AKA Web Annotation. I needed to convert the JSON-LD examples to RDF XML and test XPath expressions. I justed a couple of online tools. Just some quick notes about these tools.

  • Convert JSON-LD to RDF XML: Select JSON-LD as input format and Pretty RDF/XML as output format.
  • Test XPath expressions: Copy and past the XML into the input field. Type the XPath into the XPath input field. Hit Test XPath button and you should see the result.
  • XML Formatter: A XML formatter that lets you specify the number of leading spaces.

You can find the latest information about Web Annotation over here and example code on Github.

Flask application factory pattern and testing

This is a project I setup to show the use the application factory pattern in Flask. I set it up because I was running was stuck while refactoring one of my projects to use the factory pattern. The code would run normally, but my test cases where failing with the following two errors:

RuntimeError: application not registered on db instance and no application bound to current context
RuntimeError: working outside of application context

Most Flask examples don’t use the factory pattern so I spent a lot of time searching around to solve the problem. So I thought I would work it out and share it. Hopefully it saves someone else time.

The problem

Once your project starts to grow, code organization is everything. Flask provides a number of mechanisms for code organization. One of these mechanism’s is blueprints. Combined with the factory pattern provides a nice way to structure and organise code.

Another problem that the factory pattern helps solve is circular dependencies.

Getting the factory pattern to work isn’t hard. Getting it to work correctly, it turned out, was a little harder. The problem I had was caused in the testing code. In the following section I will briefly explain how to setup and use the factory pattern correctly.

Lessons learned

I have added more code than strictly necessary to show the concept of the factory pattern working as a realistic example.

The structure and contents of this example project is:

src
│ .gitignore
│ readme.md
│ manage.py
│ requirements.txt
├── instance
│   sensitive.cfg
├── test_app_factory
│   │ __init__.py
│   │   appliction.py
│   │   config.py
│   │   extensions.py
│   │   models.py
│   ├── helpers
│   │   __init__.py
│   │   misc.py
│   ├── module
│   │   __init__.py
│   │   viws.py
│   ├── static
│   │   favicon-16x16.png
│   │   favicon-32x32.png
│   └── templates
│   index.html
└── tests
    test_basics.py

Okay, what’s important to point out here?

The core of the factory pattern is setup in application.py and extensions.py. All extensions are initialized in extensions.py. If you add additional extensions make sure to add them to the import statement in test_app_factory/__init__.py. This is a convent way to shorten import statements.

The actual heavy lifting is done in application.py. Each part of the application initialization is a separate function, which are called by the main function app_factory. This function takes a string, which specifies the environment the configuration should be loaded for. The configuration is defined in config.py.

The factory pattern in application.py looks like this:

def app_factory(config, name):
app = Flask(...)

...

return app

The function calls a number of functions that load the configuration settings, extensions, blueprints, etc.

Using the factory is really easy, just use the following call:

app = app_factory('TST')

To access the app object in modules after the application has been initialized is done using the proxy provided by Flask:

from flask import current_app as app

Now for the part that was driving met crazy, the testing code. I still do not understand fully why it is the only place in my code that was causing a problem, probably has to do with the way unittest works. Anyway, to get the factory pattern to work you need to add app_context to specific statements. Here is an example.

class TestCase(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        cls.app = app_factory('TST')

    def setUp(self):
        with self.app.app_context():
            self.client = app.test_client()
            db.create_all()

    def tearDown(self):
        with self.app.app_context():
            db.session.remove()
            db.drop_all()

    def test_add_user(self):
        with self.app.app_context():
            db.session.add(User(name='test user', email='[email protected]'))
            db.session.commit()

Conclusion

Finding good examples isn’t always easy. The factory pattern can really help to organize the code and make it more readable and maintainable.

Any suggestions how I can further improve the code? I would love to hear from you!

Inspecting and replaying requests with ngrok

A very cool feature of ngrok is the web interface. With this interface you can inspect requests and replay them. Very handy for debugging purposes.

The web interface can be reached on http://localhost:4040.

In the web interface you can view both the request received and response sent. The default view of the request and response is a summary of the post data, but it can be viewed in raw format or binary format. This is really useful when debugging webhooks.

Another great feature of this web interface is the replay functionality. Instead of having to run through the code again to get to the problem, you can just hit the resend button and the request will be resent.

Running a specific test in nose2

Today I needed to run a specific nose2 test case to fix an error. This can be achieved the following ways.

My test cases are located in project/tests.

To run all the test cases in a test.

nose2 project.tests.test_file

To run a specific test case in a test.

nose2 project.tests.test_file.TestCase.test_method

That’s it. Happy coding.