Sunday, December 14, 2014

From 2 to 3

Repeating problem (task) that I encounter during migration from Python 2 to Python 3:

* class advice impossible in python3. use the @implementer class decorator instead...
More or less similar with this issue:
zope.interface has had a feature for 10 years under Python2 which can't be made to work portably under Python3. It is spelled like:
from zope.interface import Implements

class Foo(object):
    implements(IFoo)
There is a newer, Python3-compatible spelling, which the pyramid trunk now uses (as of today):
from zope.interface import implementer

@implementer(IFoo)
class Foo(object):
    pass
(That spelling only works under Python >= 2.6)
* ImportError: No module named 'urlparse'
simply refer to this page:

urlparse is part of the standard Python 2 library. It's shipped as part of Python; it isn't packaged separately on PyPI et al. urlparse.urlparse (the function) was renamed in Python 3 to urllib.parse.

If you need to write code which is Python2 and Python3 compatible you can use the following import
try:
    from urllib.parse import urlparse
except ImportError:
    from urlparse import urlparse
from urllib.parse import urlparse work for me

* ImportError: No module named 'ConfigParser'

What I normally do:
import configparser as ConfigParser
ImportError: No module named 'StringIO' 
The StringIO and cStringIO modules are gone. Instead, import the io module and use io.StringIO or io.BytesIO for text and data respectively.
try:
    from StringIO import StringIO
except ImportError:
    from io import StringIO
ImportError: No module named 'HTMLParser' 
Solution:
from html.parser import HTMLParser
Sample:
from html.parser import HTMLParser

class MyHTMLParser(HTMLParser):
    def handle_starttag(self, tag, attrs):
        print("Encountered a start tag:", tag)
    def handle_endtag(self, tag):
        print("Encountered an end tag :", tag)
    def handle_data(self, data):
        print("Encountered some data  :", data)

parser = MyHTMLParser()
parser.feed('<html><head><title>Test</title></head>'
            '<body><h1>Parse me!</h1></body></html>')
ImportError: No module named httplib
In Python 3, the module has been renamed to http.client
import http.client
httpRequest = ""
conn = http.client.HTTPConnection("localhost",8080)
conn.request("GET","/file.html",httpRequest)
response = conn.getresponse()
print(response.status,response.reason)
conn.close();
ImportError: No module named urllib2
In python 3 urllib2 was merged into urllib. To make Python 2 code work in Python 3:
try:
    import urllib.request as urllib2
except ImportError:
    import urllib2
import urllib.request
wp = urllib.request.urlopen("http://goggle.com")
pw = wp.read()
print(pw)
ImportError: No module named 'BaseHTTPServer'
BaseHTTPServer, SimpleHTTPServer modules in Python 2 have been merged into http.server module in Python 3. Sample:
import http.server
import ssl

httpd = http.server.HTTPServer(('localhost', 4443), http.server.SimpleHTTPRequestHandler)
httpd.socket = ssl.wrap_socket(httpd.socket, certfile='server.pem', server_side=True)
httpd.serve_forever()
* TypeError: the JSON object must be str, not 'bytes'
In short; decode:
result = json.loads(response.readall().decode('utf-8'))
* TypeError: Unicode-objects must be encoded before hashing
Refer to this similar issue:
line.encode('utf-8')
or:
str(line)
* NameError: global name 'unicode' is not defined
Yes, again this is Python 3. Python 3 renamed the unicode type to str, the old str type has been replaced by bytes.
if isinstance(unicode_or_str, str):
    text = unicode_or_str
    decoded = False
else:
    text = unicode_or_str.decode(encoding)
    decoded = True
In short; use str(*)

Monday, December 1, 2014

Field ??? not expected

Stack:
  • Python 3.4 (using virtualenv)
  • Pyramid 1.5.2
  • Jinja2 (as Template Engine)
  • SimpleForm (hacked for compability with Python 3)
  • FormEncode
So I encounter the error which from the stack trace I almost cannot conclude anything. By debugging SimpleForm (yes, you read it correctly... debugging) finally I able to capture the real error messages which is like

field submit not expected
or
field _csrf not expected

With the real error message in hand, the quick googling deliver me to this link: https://github.com/Pylons/pyramid_simpleform/issues/6

Oh ok, seems like I need to add something to my schema
class MySchema(Schema):
    allow_extra_fields = True
    filter_extra_fields = True

    name = validators.UnicodeString(min=5)
    value = validators.Int()
Yes, it's because of allow_extra_fields and filter_extra_fields, and the curiosity of "why" lead me to this page: https://github.com/Pylons/pyramid_simpleform/blob/master/docs/index.rst
which stated:
Note the use of the allow_extra_fields and filter_extra_fields attributes. These are recommended in order to remove unneeded fields (such as the CSRF) and also to prevent extra field values being passed through.
Oh Desson, please RT*M!

Wednesday, November 26, 2014

Change SQLLite to PyMySQL

Another task is to change the database from SqlLite to MySQL (Hey, we need to use real database here). And since I'm using SQLAlchemy
Changes in development.ini

Change
sqlalchemy.url = sqlite:///%(here)s/xyzdb.db
To
sqlalchemy.url = mysql+pymysql://root:system@localhost/xyz?charset=utf8
The charset=utf8 is important for unicode

And for sure, you need to install pymysql before

% easy_install pymysql
Why I use PyMySql? Long story make short, since it's pure Python MySQL client. For another DB client library (with SQLAlchemy), check http://docs.sqlalchemy.org/en/latest/dialects/mysql.html

test-connection.py :
import sqlalchemy
from sqlalchemy.sql import select
from sqlalchemy import Table, MetaData


def init():
    try:
        server = 'localhost'
        db = 'pyramid'
        login = 'root'
        passwd = 'toordrowssap'
        engine_str = 'mysql+pymysql://{}:{}@{}/{}'.format(login, passwd, server, db)
        engine = sqlalchemy.create_engine(engine_str, echo=False, encoding='utf-8')
        connection = engine.connect()
        metadata = MetaData()
        t_servers = Table('auth_permission', metadata, autoload=True, autoload_with=engine)
        s = select([t_servers])
        result = connection.execute(s)
        for row in result:
            print(row['name'])
    except Exception:
        raise
    finally:
        connection.close()
if __name__ == '__main__':
    init()

Tuesday, November 25, 2014

Upgrading Pyramid Application

Environment:

  • Python 3.4 (using virtualenv)
  • Pyramid 1.5.2
During an upgrade of a project developed in previous release of Pyramid (and Python 2.x), In encountered error(s) caused by Paste. Luckily, there is another person who encounter the same issue and "document" it very well here. My steps for troubleshoot this issue more or less is the same:

Changes in development.ini

Remove
[pipeline:main]
pipeline =
    egg:WebError#evalerror
    {{myproject}}
Change
[app:{{project}}]
To
[app:main]

Changed server from Paste
[server:main]
use = egg:Paste#http
host = 0.0.0.0
port = 6543
 To pyramid
[server:main]
use = egg:pyramid#wsgiref
host = 0.0.0.0
port = 6543
Changes in setup.py changed requires from
requires = [
    'pyramid',
    'pyramid_beaker',
    'SQLAlchemy',
    'transaction',
    'zope.sqlalchemy',
    'WebError',
]
to
requires = [
    'pyramid',
    'pyramid_beaker',
    'SQLAlchemy',
    'transaction',
    'zope.sqlalchemy',
]
It's important to remove WebError

[Update 4/3/2015]
Important:
It's important to remove "paste" from your system, since somehow pyramid will encounter error when this package exists in your system (virtualenv). "pastedeploy" is acceptable

Friday, November 21, 2014

Nosetests Output to a File

If you familiar with python, I'm sure you also familiar with nosetests. Running nosetests is a piece of cake (although writing the test cases is sadly... not). But how to output the test result(s) to a file? I use the normal ">" method:
nosetests > result.txt
Surprisingly, it's not working :(
Looking on the internet (yes, google), I found 2 ways to do it:

1. Indeed by using ">", but the "complete" command is:
nosetests > result.txt 2>&1
2. And by using nosetests option:
nosetests --with-xunit --xunit-file=result.xml
Will produce a xml format file

I prefer the 1st option, txt file, cleaner.

Sunday, November 16, 2014

lxml and Installing a Python Package with a .whl Extension

So, story short: I learn about Ringo as I try to get some example of Pyramid Framework. After download the project, when running the setup, one of the library needed is lxml - and it's fail to install with easy_install.
On installation guide, I found out that on Windows (too bad right, I'm using Windows), the installer expecting a build-tools (which normally is VC++), which indicated in:
For MS Windows, recent lxml releases feature community donated binary distributions, although you might still want to take a look at the related FAQ entry. If you fail to build lxml on your MS Windows system from the signed and tested sources that we release, consider using the binary builds from PyPI or the unofficial Windows binaries that Christoph Gohlke generously provides.
Since no-way to install those crap in my machine, my only choice is "unofficial Windows binaries" that mentioned in above, go to the link, and download-it. But wait, a .whl file?
OK, it's a wheel file, but how I install it?

First use easy_install (or pip) to install wheel
easy_install wheel

Then install the package using pip, somehow easy_install cannot work (correct me if I wrong)
pip install package-name.whl

And ta-da, the package installed.
And after that, I found out that Ringo is "too-huge-and-not-easily-customize", remind me about Django. Seems like I love to make my hands dirty :(

Saturday, November 8, 2014

Preparing Python Environment

Yes, I also using Python....
And since I use it in windows, preparing the environment it's a bit different with if its on Linux
The good thing is on python version is 3.x, the virtualenv is already part of the python basic installer.

To create new virtualenv in d:\python\venv\pylons:
C:\Users\Desson>python -m venv d:\python\venv\pylons
To activate virtualenv:
C:\Users\Desson>d:\python\venv\pylons\Scripts\activate.bat
And now, the virtuanenv is active:
(pylons) C:\Users\Desson>
Now you can work with this 'environment' and install 'anything you need' via easy_install (or pip)
Add virtualenv to Pycharm
I'm using Pycharm as my main IDE (Beside Notepad+, for small tasks), since we have new virtualenv, we need to bind this virtualenv. Jetbrains already documented the process in https://www.jetbrains.com/pycharm/help/adding-existing-virtual-environment.html, I only copied here (with my own words):
  1. Go to Project Settings, find Project Interpreter page, click the 'gear-wheel' button on the right.
  2. In the drop-down list, choose Add local 
  3. In the Select Python Interpreter dialog box that opens, choose the desired Python executable (in our virtualenv), and click OK.
The result should be like this:
And after that? Happy coding...


Monday, September 29, 2014

Configure Pip and Npm Behind a Proxy

In Linux;

It's a must to set your HTTP_PROXY and HTTPS_PROXY env
export HTTP_PROXY=http://proxy.mycompany.com:3333
export HTTPS_PROXY=http://proxy.mycompany.com:3333
It's also a must for Windows environment to set those variables

for pip
pip install package
Is enough, although
pip --proxy http://proxy.mycompany.com:3333 install package
also can do the trick

For npm

Nope, the variables can't work. Npm use a configuration file and it can be added via command line npm config set
npm config set proxy http://proxy.mycompany.com:3333
npm config set https-proxy http://proxy.mycompany.com:3333

and do the job
npm install package

They say that NodeJS faster than Python?

Thursday, September 18, 2014

Action Report: poc-swagger-springmvc

Upgrading swagger-springmvc from version 0.65 to 0.8.8 is not a smooth upgrade in eGospel project. So, I decided to create separated Proof of Concept (POC) to make sure if we removed some complexity in the project, the investigation will be lot easier. My (stupid) first mistake is, I forgot to add
<bean class="com.mangofactory.swagger.configuration.SpringSwaggerConfig" />
in my springmvc-context.xml. After 1-2 hour, I figured out what happen, but till now still lamented for 2 hours wasted because of my stupidity. Moral of the story is, don't do any coding, when you are tired and sleepy, just rest!

Then I add (and replaced) the latest swagger-ui from "https://github.com/wordnik/swagger-ui" dist folder, and add their index.html as my "welcome-file" in web.xml
The controller I built is very simple HelloController, with one function returning Hello object and another function returning String. I also add HelloControllerTestIT for integration test, and setup all maven-plugin as in my eGospel project. And testing phase is success.

But problem now happen when I deploy my poc in application server (in my case, Tomcat) and browse it in browser. Yes, the swagger-ui can list down the controllers (which the swagger is working), but when I "Tryit out!" I got http 404; not found error. It's strange since the integration test can find the service.

I do more readings (by browsing), and guess it caused by relative path in the URL. I read several sources:
https://github.com/wordnik/swagger-ui/issues/114
https://github.com/wordnik/swagger-ui/issues/355
https://github.com/wordnik/swagger-ui/issues/337
http://stackoverflow.com/questions/14262660/rest-api-swagger-java-jersey-wrong-listing-path-and-additional-apis

Although not directly helping me, in my problem, I know what I need to do. Change my basepath into absolute path!
Since I don't want to make my hand dirty by writing my own CustomizeSwaggerConfiguration, what I need to do is to extend SpringSwaggerConfig into my new class. here comes SpringSwaggerConfig. 
package com.dariawan.poc.config;

import com.mangofactory.swagger.configuration.SpringSwaggerConfig;
import com.mangofactory.swagger.models.configuration.SwaggerModelsConfiguration;
import com.mangofactory.swagger.paths.AbsoluteSwaggerPathProvider;
import com.mangofactory.swagger.paths.SwaggerPathProvider;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Import;

/**
 *
 * @author Desson Ariawan <teodesson@yahoo.com>
 */
@Configuration
//@ComponentScan(basePackages = {"com.mangofactory.swagger.controllers"})
@Import(SwaggerModelsConfiguration.class)
public class MySpringSwaggerConfig extends SpringSwaggerConfig {

    @Bean
    @Override
    public SwaggerPathProvider defaultSwaggerPathProvider() {
        return new AbsoluteSwaggerPathProvider();
    }
}

Yes, what I do is only overriding  defaultSwaggerPathProvider() function. Replace bean in springmvc-context.xml, now using MySpringSwaggerConfig.
<bean class="com.dariawan.poc.config.MySpringSwaggerConfig" />

And voila! Integration Test now passed, and swagger-ui also showing what I'm expecting.



You can check complete project in https://github.com/teodesson/poc-swagger-springmvc
Now I need to sleep.

Update: add pictures, and later on: added spring-security into the project, no functional impact, swagger and spring-mvc works as expected

Monday, September 15, 2014

POC: Swagger - Spring MVC (And Swagger UI)

My eGospel project using SpringMVC in the backend, and produces JSON RESTful web services via it's controllers as JSON objects. To quickly see the list of the services, and do simple check, I use swagger-ui (which means I also need to use swagger, in this case swagger-springmvc). Previously, I'm using swagger-springmvc ver 0.6.5. But when I'm upgrading to ver 0.8.8, it start to have issues.

So, I decided to create another project, as showcase or working example of the framework. I will write my finding on another post. Please check the project on github, and please give me some direction.

Introduction to Spring Web MVC framework

The Spring Web model-view-controller (MVC) framework is designed around a DispatcherServlet that dispatches requests to handlers, with configurable handler mappings, view resolution, locale, time zone and theme resolution as well as support for uploading files. The default handler is based on the @Controller and@RequestMapping annotations, offering a wide range of flexible handling methods. With the introduction of Spring 3.0, the @Controller mechanism also allows you to create RESTful Web sites and applications, through the @PathVariable annotation and other features.

What's Swagger?

The goal of Swagger™ is to define a standard, language-agnostic interface to REST APIs which allows both humans and computers to discover and understand the capabilities of the service without access to source code, documentation, or through network traffic inspection. When properly defined via Swagger, a consumer can understand and interact with the remote service with a minimal amount of implementation logic. Similar to what interfaces have done for lower-level programming, Swager removes the guesswork in calling the service.
Check out Swagger-Spec for additional information about the Swagger project, including additional libraries with support for other languages and more.

Swagger UI

Swagger UI is part of Swagger project. The Swagger project allows you to produce, visualize and consume your OWN RESTful services. No proxy or 3rd party services required. Do it your own way.
Swagger UI is a dependency-free collection of HTML, Javascript, and CSS assets that dynamically generate beautiful documentation and sandbox from a Swagger-compliant API. Because Swagger UI has no dependencies, you can host it in any server environment, or on your local machine.

Sunday, September 7, 2014

eGospel in GitHub

I copied my eGospel project from a private repository in Bitbucket to GitHub. If you interested, you can check the project in this repo.

So, what is eGospel?
in short; eGospel is a simple church dynamic website project, running on Java.

For Techies, Geeks 'n Coders,
this is the summary of technologies adopted in this project:
  • Java (for sure, now I'm using JDK 1.7)
  • Spring Framework, including Spring Security and Spring Data JPA
  • Hibernate
  • AngularJS
  • Twitter Bootstrap
  • And a quite number of Javascript libraries (jQuery, Underscore JS, etc)
I decided to skip Coffeescript this time. I think I need more hands-on on AngularJS to get the essence of this framework, before (if needed) later on encapsulate it with another framework.

Saturday, August 30, 2014

To Coffeescript or Not?

I must admit if I'm a framework addict. I love to learn new 'hot' technology out there which can benefit my projects. I already put Coffeescript in my radar for quite sometimes. Since currently I'm working in AngularJS, I'm interested if I can use Coffeescript to speedup my development (hey, I'm never mastered Javascript... in fact I'm more to back-end person than front-end person)

Searching in Google, I come accross these articles:
Conclusion: As you can see, CoffeeScript is a great companion to AngularJS. Whilst not for everyone, the ease of writing CoffeeScript far outweighs the downsides.

In short: looks like AngularJS and Coffeescript is a great combination but I need to dig further. Any input or opinion?

Sunday, July 13, 2014

Processor Performance: Clock Frequency vs Multi Core

I still remember the era when we remember the processor with their version: Pentium I, Pentium II, Pentium III, and Pentium IV. With the increase of version, processor performance also increase in terms of processor clock frequency.
But beyond that, there are another level: Core i3, i5, and i7, and Xeon. And the performance is no longer based on the clock frequency but the number of processor cores. From dual-core, then quad-core, then 6-core, 8-core, and so on. The clock frequency doesn't increase much, but number of core increased. Same things happens to the processor in our gadget or smartphone.


Clock frequency indeed determine the speed at which the computer works. The higher it is, the faster it works. But there are side effects for increasing the frequency: the processor temperature getting hotter and hotter.  And processor manufacturers aware of this, that silicon (which made a processor) has limits in terms of heat it can hold. So manufacturers must change the way on how to improve computer performance.
The solution is to increase the number of processor cores in one chip to do what is called multiprocessing. In multiprocessing, the work that the computer has to do is divided into several processor cores.
Similar with our daily lives, work done by two people is usually faster than just one person. Thus increasing a performance that two person can achieve. Same things also works for processor, more core, more performance it's achieved. Exactly as expected.
However, the increase in the number of processor cores must be followed by software program that able to split the processing into different processor cores. Normally, this depends on the operating systems. If the program execution turns out cannot be shared between processor's cores, then the program just use one core at the time, and we not get advantage from multiprocessing.
And now, here I'm... writing this blog with a laptop powered by Intel Core i7-3630QM 2.4GHz which is a fast quad-core processor

Saturday, July 12, 2014

VirtualBox


Oracle VM VirtualBox (formerly Sun VirtualBox, Sun xVM VirtualBox and Innotek VirtualBox) is a free and open-source hosted hypervisor for x86 virtualization, developed by Oracle Corporation. Created by Innotek GmbH, it was acquired by Sun Microsystems in 2008, which was, in turn, acquired by Oracle in 2010.(wikipedia)

VirtualBox is a powerful x86 and AMD64/Intel64 virtualization product for enterprise as well as home use. Not only is VirtualBox an extremely feature rich, high performance product for enterprise customers, it is also the only professional solution that is freely available as Open Source Software under the terms of the GNU General Public License (GPL) version 2. (VirtualBox)

Friday, June 20, 2014

Configure TortoiseHg to Remember Your Credential

I should write about this two years ago...
I'm migrating my project to another repository. And if you are working with Mercurial and using TortoiseHg, you must be know this annoying prompt:


What you need to do, is to save your credential:

1) Right click on your repository and click Settings

2) Edit File

3) Write your credential

Yes, like this:
[paths]
bitbucket = https://bitbucket.org/teodesson/dallanube-sims

[auth]
bitbucket.prefix = bitbucket.org
bitbucket.username = teodesson
bitbucket.password = <password>

Save, and ta-da! 

Update:
I got this prompt when trying to detect/push outgoing changes


Solve it by changes "bitbucket" to default
[paths]
default = https://bitbucket.org/teodesson/dallanube-sims

[auth]
default.prefix = bitbucket.org
default.username = teodesson
default.password = <password>

Again save, and another ta-da!

Reference:
https://www.swiftsoftwaregroup.com/how-to-configure-tortoisehg-to-remember-your-username-and-password/