Flask error loading config file in instance directory

I have been dealing with a really frustrating problem today. Yesterday my code was working and today it was failed without any code changes.

This is the piece of code causing the problem.

app = Flask(__name__, instance_relative_config=True)
app.config.from_object('config')
app.config.from_pyfile('config.cfg')

And this is the error.

Traceback (most recent call last):
  File "run.py", line 1, in 
    from project import app
  File "C:\Python27\env\vod_test\src\project\__init__.py", line 37, in 
    load_config()
  File "C:\Python27\env\vod_test\src\project\__init__.py", line 33, in load_config
    app.config.from_pyfile('config.cfg')
  File "C:\Python27\env\vod_test\lib\site-packages\flask\config.py", line 129, in from_pyfile
    with open(filename) as config_file:
IOError: [Errno 2] Unable to load configuration file (No such file or directory): 'C:\\Python27\\env\\vod_test\\var\\project-instance\\config.cfg'

Flask is trying to load the config file from C:\\Python27\\env\\vod_test\\var\\project-instance\\config.cfg instead of from C:\\Python27\\env\\vod_test\\project\\instance\\config.cfg. Where is the var coming from? And why is there a hyphen between project and instance?

The strange thing was that the problem only occurred when I was running my test cases from the command line. In PyCharm the code was working. How frustrating. It turned out that PyCharm was using a different virtualenv than the command line. After comparing the output of ‘pip freeze’ I discovered that some libraries had different versions.

To solve the problem I added the following code.

import os
from flask import Flask


def load_instance_config_fix(filename):
    config = {}
    with app.open_instance_resource(filename) as f:
        for line in f:
            if line[0] == "#":
                continue
            name, value = line.partition("=")[::2]
            config[name.strip()] = value.strip()
    return config


def handle_incorrect_instance_path(f):
    def handle_load_instance_config():
        try:
            f()
        except IOError:
            app.instance_path = os.path.join(os.path.abspath(os.curdir), 'instance')
            config = load_instance_config_fix('config.cfg')
            app.config.update(config)

    return handle_load_instance_config


@handle_incorrect_instance_path
def load_config():
    app.config.from_object('config')
    app.config.from_pyfile('config.cfg')


app = Flask(__name__, instance_relative_config=True)

load_config()


@app.route('/')
def hello_world():
    return 'Hello, World!'

Just some nice syntactic sugar with a decorator to keep the code nice and concise.

Another way to solve the problem is by setting the instance_path when initializing Flask. What fun would that be 🙂

import os
from flask import Flask


app = Flask(__name__, instance_path=os.path.join(os.path.abspath(os.curdir), 'instance'), instance_relative_config=True)
app.config.from_object('config')
app.config.from_pyfile('config.cfg')


@app.route('/')
def hello_world():
    return 'Hello, World!'

Back to real coding 🙂

Advertisements

Git quickstart

Just some short notes to remember how to get started with Git.

Check version

$ git --version

Setting-up

Set the following configuration items if using Git for the first time, as these values are used for each commit made.

$ git config --global user.name "Your Name"
$ git config --global user.email "[email protected]"

Initializing a new Git repository

Make sure you are in the top-level folder of your project and then run the following command.

$ git init

Create the .gitignore file

Git has a special file called ‘.gitignore’ that allows you to specify which files to NOT include in your repository.  The following file is based on the recommended .gitignore file for Python from GitHub.

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
 
# PyCharm files
.idea/
 
# Instance Folder - used for sensitive configuration parameters
/instance

Setting remote repository
Make sure you are in the top-level folder of your project and set up the remote repository.

$ git remote add origin [email protected]:nidkil/name-of-repository.git

Check the remote repository has been set correctly.

$ git remote -v

Staging files
The standard flow for adding files to your git repository is to create/edit the files, add them to the staging area, and then commit them to your repository.

$ git status
$ git add .
$ git status

Initial commit
The following command makes the first commit to the repository, including a message describing the commit.

git commit -m "Initial version"

You can confirm that the commit was successful by checking the log of the repository.

git log

Push to the remote repository
To push the local Git repository to the remote Git repository.

$ git push -u origin master

Working with a feature branch
A common method of developing features is to create a feature branch where you implement specific functionality and then merge the changes back into the master or development branch. Follow the process: create a new feature branch, make changes and merge changes back into the ‘master’ branch.

Create a new feature branch.

$ git checkout -b name_of_feature_branch

Check new feature branch was created (the star next to ‘name_of_feature_branch’ branch indicates it is the current branch that we are working in).

$ git branch

The following commands stage, commit and merge the changes into the local repository.

$ git add .
$ git status
$ git commit -m “Added use of templates and Bootstrap”
$ git checkout master
$ git merge add_templates

Delete the branch.

$ git branch -d add_templates

Push the changes into the remote repository.

$ git push -u origin master

Test API webhooks from localhost

To test the Mollie payment API on my localhost I needed a way to receive the webhook callback on my localhost. To be able to do this I needed a service to tunnel a public URL to localhost (my local machine, as opposed to a publicly accessible URL). Here are few examples of services that provide this service:

ngrok (free plan)

localtunnel (free)

pagekite

forwardhq

These types of tools give you a publicly accessible URL that forwards the request to the localhost port, which can be used for the API callback URL while testing.

Create a new project in PyCharm based on an existing virtualenv

Today I cloned a project from Github and needed to setup a project in PyCharm for it. These are some quick notes how to setup a new project in PyCharm based on an existing virutalenv.

  1. Clone the project with: git clone <URI>.
  2. Open PyCharm.
  3. Create a new project: File -> New project…
  4. Select project type: python
  5. Click create.
  6. Select the root directory of the virtualenv.
  7. Select local type by clicking the button behind the source field.
  8. From the Scripts directory of the virtualenv select python.exe.
  9. Clicke create.
  10. PyCharm will display a message that the directory is not empty (“The directory <…> is not empty. Would you like to create a project from the existing source instead?”).
  11. Click Yes.
  12. Select File -> Settings…
  13. Select Project […] -> Project Structure.
  14. Delete current source directory.
  15. Add the source directory in the virtualenv.
  16. Click OK.

You’re ready to rock. Have fun!

 

Quick test Flask, Flask-Restplus and Flask-Marshmallow

I needed to quickly test a few things today using Flask, Flask-Restplus and Flask-Marshmallow. I created the following boilerplate code that can easily be modified for further quick tests.

from flask import Flask, Blueprint, url_for, jsonify
from flask_restplus import Api, Resource
from flask_marshmallow import Marshmallow, base_fields
from marshmallow import post_dump

# Setup Flask app

url_prefix = '/api/v4'
doc = '/apidoc/'
app = Flask(__name__)
app.config['JSON_SORT_KEYS'] = False
ma = Marshmallow()
blueprint = Blueprint('api', __name__, url_prefix=url_prefix)
api = Api(blueprint, doc=doc, version='3.0')

# Models

class Bunch(dict):
    def __init__(self, *args, **kwargs):
        super(Bunch, self).__init__(*args, **kwargs)
        self.__dict__ = self

class Author(Bunch):
    pass

def mock_author():
    author = Author(id=123, name='Fred Douglass')
    return author

def mock_author_list():
    a1 = Author(id=1, name="Alice")
    a2 = Author(id=2, name="Bob")
    a3 = Author(id=3, name="Carol")
    return [a1, a2, a3]

# Schemas

class AuthorSchema(ma.Schema):
    id = base_fields.Int(dump_only=True)

    absolute_url = ma.AbsoluteURLFor('api.author', id='<id>')

    links = ma.Hyperlinks({
        'self': ma.URLFor('api.author', id='<id>'),
        'collection': ma.URLFor('api.authors')
    })

    @post_dump(pass_many=True)
    def wrap(self, data, many):
        key = 'authors' if many else 'author'
        return {
            key: data
        }

    class Meta:
        fields = ('id', 'name', 'links', 'absolute_url')
        ordered = True

# Setup operations

ns = api.namespace('authors', description='desc')
ep_1 = 'authors'
ep_2 = 'author'

@ns.route('/', endpoint=ep_1)
class AuthorsCollection(Resource):
    def get(self):
        print " ---- GET - result=" + url_for('api.' + ep_1)
        s = AuthorSchema(many=True)
        result = s.dump(mock_author_list())
        return jsonify(result.data)

    def post(self):
        print " ---- POST - result=" + url_for('api.' + ep_1)
        return None, 201

@ns.route('/<int:id>', endpoint=ep_2)
class AuthorsDetail(Resource):
    def get(self, id):
        print " ---- GET - result=" + url_for('api.' + ep_2, id=id)
        print " ---- GET - result=" + url_for('api.' + ep_1)
        s = AuthorSchema()
        result = s.dump(mock_author())
        return jsonify(result.data)

    def put(self, id):
        print " ---- PUT - result=" + url_for('api.' + ep_2, id=id)
        print " ---- PUT - result=" + url_for('api.' + ep_1)
        return None, 204

    def delete(self, id):
        print " ---- DELETE - result=" + url_for('api.' + ep_2, id=id)
        print " ---- DELETE - result=" + url_for('api.' + ep_1)
        return None, 204

app.register_blueprint(blueprint)

if __name__ == '__main__':
    print '>>>>> Starting server at http://localhost:8080{url_prefix}{doc}'.\
        format(url_prefix=url_prefix, doc=doc)
    app.run(port=8080, debug=True)

Marshmallow not respecting ordered=True

I had a little fight with Marshmallow and the ordered=True in class Meta. When ordered=True is set the fields should be outputed in the order specified in the property fields of the class Meta.

Example:

class AuthorSchema(ma.Schema):
    id = base_fields.Int(dump_only=True)

    absolute_url = ma.AbsoluteURLFor('api.author', id='<id>')

    links = ma.Hyperlinks({
        'self': ma.URLFor('api.author', id='<id>'),
        'collection': ma.URLFor('api.authors')
    })

    @post_dump(pass_many=True)
    def wrap(self, data, many):
        key = 'authors' if many else 'author'
        return {
            key: data
        }

    class Meta:
        fields = (
            'id',
            'name',
            'links',
            'absolute_url',
        )
        ordered = True

This was the output.

{
  "author": {
    "absolute_url": "http://localhost:8080/api/v4/authors/123",
    "id": 123,
    "links": {
      "collection": "/api/v4/authors/",
      "self": "/api/v4/authors/123"
    },
    "name": "Fred Douglass"
  }
}

After some Googeling I found out that the culprit was Flask.jsonify. It orders output alphabetically by default. The following setting switches this off.

app.config['JSON_SORT_KEYS'] = False

After this the output was in the specified order.

{
  "author": {
    "id": 123,
    "name": "Fred Douglass",
    "links": {
      "self": "/api/v4/authors/123",
      "collection": "/api/v4/authors/"
    },
    "absolute_url": "http://localhost:8080/api/v4/authors/123"
  }
}

Hope this saves someone else time.