The http module was migrated from urllib to Requests, this solved the memory leaks issue and let me build a new structure for this module, now it only cares for GET and POST requests (as should be) and can be tested separately, outside a libturpial environment. After came the adaptation of protocol module, this means integration with the new http module and update the API version (v1.1) for the Twitter implementation. Finally, yesterday I was working with the account module, this integrates all the configuration part and now we can easily load the existing profiles of libturpial into a shell and make tweets, fetch the timeline, see our mentions and further.

Modules that need to be improved yet are the account manager and the final integration into the core. With this I could make some unit test and start an official testing phase for a next stable release.

That’s all for now.

EOF

People have been wondering if Turpial is dead. The short answer is: No.

Now let’s go with the long anwser.

I’ve been working on the migration of Turpial from Webkit to Gtk 3 and it hasn’t been easy, a lot of work trying to implement the same we got with HTML and sadly our friends Carlos Guerrero and Andrea Stagi have been really busy lately so they haven’t had a chance to collaborate as they wanted.

Webkit interface

Technically speaking, Gtk 3 has almost the same limitations than Gtk 2. Gtk.Treeview is still based on Cairo so you can’t render clickable elements (as consequence there are no links, hashtags, mentions, etc), neither draw little clickable icons on top of tweets and no fancy stuffs like we did on Webkit. This made the migration much harder. It took me a lot of time to hack some widgets or create custom ones in order to get the same look & feel, and this brought a side effect: memory leaks.

Yes, memory leaks. Again. Fortunately not so critical as in Webkit but as harmful as any memory leak, hitting our performance like a big and heavy rock.

Gtk 3 interface

Basically that was what I’ve been doing so far, struggling with Gtk 3 and trying to get a really nice and light interface. However, nothing is perfect. GNOME has taken some decisions about the future of their platform, talking about it as the “brand” and making things hard for developers, all that sh*t and the proper limitations of Gtk 3 made me step aside. That’s it Gtk. Welcome Qt. Now I’m studying about Qt and QML to see if I finally can build the interface that I always have wanted for Turpial.

I have no plans of keeping Turpial 1.6 until I finish this research, Turpial 1.6 has been on service for a long time and it’s time to give it the rest it deserves. Instead I’m going to take advantage of all the progress I did with Gtk 3 and I will release a Gtk 3 version with all the limitations that I talked before but on top of libturpial, this mean the same interface than 1.6 but with multi-account, multi-protocol and multi-column support. It won’t be so bad anyway.

This limited version will be the 2.0 (as planned) and then I will keep working on Qt version to make the 2.5 (or 3.0) version with all the desired features. I hope to solve lots of issues with this decision and provide a decent update of our beloved bird.

That’s all for now.

EOF

It’s in our plans to continue improving the integration, not only on this main distros else with Canaima, ArchLinux and Fedora too. Our primary goal is to offer a great user experience no matter which operating system you use.


We should notice too that 40% of current users would like to use it Turpial in Mac OS and this is a pretty good sign about the need of port Turpial to other platforms.

I’ve decided that we are going to offer official support for Qt and this way Turpial can be easily ported to Windows and Mac OS but we need more hands to code faster. If you have some PyQt knowledge and want to contribute with some lines of code, please let us know (you could help us to develop a native interface for Turpial in Mac OS too).

That’s all for now folks, thanks to everyone who participated on the beginning of this master plan and remember that we will keep working to make an awesome and faster microblogging client.

Yes, Webkit let you do anything you want, imagination is the limit, but as the uncle Ben said: “With great power comes great responsabilities”. And that quote never had such a big meaning for me as now. This apparently unlimited power requires a extremely solid base of development and in our case, we hadn’t.

The unknown path

Technically speaking, there are no tools or frameworks to do what we wanted the way we wanted, it was: integrate HTML and Javascript into Python (desktop) applications using just a Webkit container. Well, actually Pyjamas and PyQt does something like that, but the first it’s traumatizing and with the second we should marry with Qt toolkit and this was not the idea. We wanted to deliver the same application with pretty much the same code in any graphic toolkit with webkit support: Gtk, Qt, WxWidget, et al and pure Python. That was our goal.

First approach

First thing I had to deal with was Webkit and threads. Threads? Yes sir, threads. Threading is the only way you can do a long task without freezing the whole UI, it’s not very pleasant to get a gray and useless window while your app is fetching your timelime, for example.

At he beginning was very frustrating and I got “solutions” like: “You have to enclose all the methods that access the UI from the thread with this”:

gtk.threads_enter()
my_method()
gtk.threads_leave()

Can you imagine doing so for every single method that should access to UI widgets? It’s at least overwhelming.

I continued my research and after like 2 weeks of frantic reading I finally got a reasonable explanation: Gtk is thread-aware not thread-safe, so you just need to start the threads support and enclose the main loop with those instructions and Gtk will know what to do. Ok the code was simple, at the beginning of your file do this:

gtk.threads_init()

And then enclose your main loop with this:

gtk.threads_enter()
gtk.main()
gtk.threads_leave()

That was one of happiest moments of my life, Webkit was running smoothly whilst the app was executing a lot of task on background. If you are curious about the threading implementation checkout the code of the worker in our legacy branch.

Passed this obstacle I started to work in a HTML Render Engine for Turpial, based on Python, Webkit and Jinja2. It worked like a charm, Jinja2 is a really amazing solution for template render but there was something else, Javascript integration. We needed to find a way to call Python functions from Javascript code.

After a little research we did it. We intercepted the click event generated by Webkit and changed the protocol for the URL according our needs. Regular links started with the common http:// and they trigger a new tab on the default browser whilst links to call Python functions started with cmd://. This did the trick.

Passing arguments

Intercepting clicks wasn’t the end, it was just the beginning. Despite the idea of calling Python methods from URLs in HTML seems motivating, we faced a new challenge: Passing arguments to those Python methods. I say a “challenge” because we only could build links, so arguments should be passed as a part of the link and then parsed out. After lot of testing I decided to implement this schema:

cmd://method_name-%&%-argument1-%&%-argument2...

Where -%&%- acts as arguments separator. “Why such a complicated characters sequence?” You may ask. Well, because I thought that making a hard sequence is less probable that parsing fails by misunderstanding the separator with the content of one argument. I tried with simpler separators and base64 encoding but we needed to pass other params (like status messages) that have a lot of non-ASCII chars and base64 is not very good for that. For those arguments I ended using URL encoding.

At the moment of writing this I think that maybe a better solution could be making RESTful links, this means to use slashes (/) as separators and URL encoding each argument. Like this:

cmd://method_name/url_encoded_argument1/url_encoded_argument2/url_encoded_argument3/…

But, hey! it’s done. Now I’m relaxed, working on GTK3 migration and probably my brain is not under stress. Sh*t happens.

Layout

One of the most annoying things to deal with was the layout. HTML let you build anything, but if you’ve worked a little bit with HTML you should know how frustrating could be sometimes build a decent layout. Multiplies that per 100 and then you will have an idea of how frustrating becomes to do it for a desktop application. Just a bad css-margin or a wrong floating div is enough to send your layout to hell. Turpial is a desktop application and, as such, it should be flexible, resizable, movable and everything that ends with “able”.

I had to use a table for layout (yes, shame on me) because it was the only way I could guarantee that layout won’t break my app into an unusable window just for 1px extra in a margin. Trust me, I tested A LOT and everything failed, so I used one table.

Another need was to show modal dialogs, and this is kind of annoying with pure HTML and Javascript. You probably would be thinking “But dude, there are lots of Javascript libraries that even cook the breakfast for you” and you’re right, but this webkit implementation had another limitation. With PyWebkitGtk you can render HTML by loading the URL or just passing a HTML string. Obviuosly, I used the second one for Turpial. The issue is that when you pass a HTML string, you must pass a relative path for the resources (images, CSS, etc) and none of the most used forms was valid. I tested with file:///my/path/, /my/path/, path/ and other possible values, even with and without trailing slashes and nothing worked. Resources weren’t loaded correctly.

One solution was to host those resource in the cloud but that would make Turpial relies in other services/platform and that wasn’t an option for me. The best shot was to embed the whole code (HTML, Javascript and CSS) in a single file and pass it as string. Can you imagine now how hard it was to add lots of javascript libraries? And now try to imagine how hard could be debugging?

It was more than a need to keep Turpial with the smallest amount of Javascript libraries. For me this meant one thing: No more libraries than JQuery.

Performance

Keeping only one Javascript library for Turpial wasn’t only matter of comfort, it was related with performance too. Rendering a HTML page with thousands and thousands lines of code increase the memory consumption of webkit and becomes Turpial into a fat and slow little bird.

Speaking of performance, the minimum webkit consumption was 50MB (with just a Hello World page) plus the 40MB of GTK consumption, 90MB just to open Turpial. We were flushing our lightness through the toilet but maybe we could live with it, we just need to make it worth. We started to work in some javascript methods to optimize the status rendering, for example, we didn’t render the whole column (of statuses), we just added new tweets and deleted the old ones keeping a fixed number of tweets on the column (no more than 200 per column). I used that technique in Gtk and it worked, even in Webkit seems to work, but if you used Turpial for a long time you could see how memory increased until cause segmentation fault. Yeap, your operating system had to act to avoid Turpial ate the whole memory and this was tested by Willicab who saw 1.8GB of RAM used by the bird. We had memory leaks.

After digging a lot in the source code, testing and profiling memory I realized that webkit had a memory leak when you use the execute_script method to update the DOM of the page. Everytime you use that method your memory increases a little bit. Tweets were keeping in memory (in an array) and in HTML view ad infinitum, making the memory grows and grows until the OS decided to close the application.

Turpial 1.6.x used like 50~60MB of RAM but Turpial 2 was using almost 100MB of RAM while applications like Banshee were using 60MB and gnome-shell 120MB. For my this was unacceptable for a “lightweight” application.

This issue taugh me that PyWebkitGtk implementation was not so good as I expected, it lacks of documentation (the only available is the C++ documentation) and that if you want to develop complex applications on top of it you should think twice.

Conclusion

It was a really good experience (in terms of learning) but all the issues explained above make me take the decision of drop the webkit support and back to the limited but stable-and-well-known GTK3 in order to deliver a new Turpial version as soon as possible. Turpial 1.6.9 has been out there for more than 1 year and it’s time to release a fresh version of the bird.

Webkit is a really powerful tool, I have no doubt about it. You can build anything on top of HTML and present the same look & feel for all platforms, besides using CSS you can implement a theme engine easily. However the complexity of the integration, the frustration you have to deal with everytime you want to make changes on your layout and it breaks and the performance issues put Webkit out of my list of tools for Turpial. Probably I could use it, but not for Turpial. For using Webkit as we want we need a framework, limited and bounded that minimize all the suffering and offer you a stable tool for development.

That’s all folks, I hope this article can be helpful for someone else.

Time of death: October 10, 2012

If you are interested on this topic you can read more here:

• First Turpial 2.0 official announcement
• First 2.0 alpha release/
• Alpha release delayed
• Mailing list thread about performance issues
• Mailing list thread about Gtk3 migration
• Official announcement of Gtk3 migration
• Interesting article about Web interfaces with Python
• GNOME explanation about Gtk threading awareness

As part of an activity started in Venezuela by friends of the Python community, I was invited to do a Google Hangout this saturday (Oct 20) to talk about libturpial. This initiative is been called Pygout (for Python-Hangout) and it main goals is to share knowledge.

Pygouts are broadcasted live and besides they are recorded and uploaded to YouTube, so don’t worry if you miss it. You can see the whole video of Pygout: Hacking libturpial (in spanish) and if you like it, spread the word and keep track of next hangouts.

Update: As result of this hacktivity we got awesome comments and suggestions:

• Deprecation Warning for obsolete methods on libturpial
• Use Requests library to handle HTTP request and OAuth
• Service for API keys and usage statistics
• Improve proxy support
• Unit test for libturpial
• Improve setup.py to recommend dependencies to install (with OS instructions)

I want to thank and congratulate to our friends of the Python Community for this awesome initiative and encourage the people to participate and thanks to all the people who participated in this great pygout.

Keep hacking!

Today is a great day to join us in our plan to rule the world. Let’s do it!

"I'm Going Through Changes, eh eh"

[Eminem on "Going Through Changes" from his album Recovery]

That’s right everybody, the Bird‘s going to take a new direction back to its roots, back to where it all began, so i guess it isn’t quite a new direction after all.

Have you ever gone to the gym and tried out every single trasformer machine they got there in the hopes of getting the exercise you need to build your muscles properly? Yeah, me neither And then figured out that what you continue to do in that spensive gym is jog, run and weights? All of which you actually were pretty much doing all by yourself, effortlessly,  cheap AND you were getting results! The chicks were staring at you, when you went for morning/afternoon/night jogs and workouts in the park, instead of going to that four-walled torture chamber they so call gymnasium.

Well, something like that happened to the Bird guys, we started heading to a path with great hopes of achieving new ways to port, use and implement Turpial in it’s newest forms but then it came to our attention that we’ve been over working ourselves for no reason, and that everything we’ve come to so far was resulting on too much power useage and not enought lightness and fastness from our beloved Turpial.

So we had to make a choice, an adjustment to benefit you and The Bird in the way it was first planed: Light, Fast and Native. That’s why, we have decided to drop Webkit development to focus on Qt, Python and Gtk3. This was the way things worked back in 2009-’10 and it’s time we go back to our roots, back to where it all started. These adjustment will take a bit of time to come out to the public, but we are hard working on it to make it the best solutions for you, Twitter, Identi.ca and for all our development team.

Thanks for being with us through all of our story and for allowing us to grow with you. Remember, we are all always

"Going through changes..."

If you need more info, please read: To webkit or not to webkit

We have created a new article in our wiki that explains the basic flow to interact and write code for Turpial.

We assume that you already have a minimum understanding of Git and that you feel comfortable with shell commands. If you have any doubt or suggestion just pass by to #turpial channel on IRC or join to our mailing list and ask your questions, we’ll be glad to hear you.

Check it out and tell us what do you think.

This procedure will erase your current settings and all registered accounts but is completely necessary, otherwise Turpial won’t run after update because a conflict with the old configuration files.

After a long time, and thanks to the efforts of our friend Efrain Valles, Turpial’s PPA finally got updated with the lastest stable version of Turpial (1.6.9).

If you’re using a previous version we encourage you to update as soon as posible. For more information visit our downloads page

.