Dependency-driven development: forced OSS contributions

It is such a relief when you app just works. Moreover, when it is open. My pet project Xpiks is not only an open-source project itself, but it also uses a lot of the other open-source technologies inside. Qt framework, zlib, hunspell – to name just a few. A big deal is to make them work together. A much bigger deal to make them work together across different platforms (Xpiks is announced as cross-platform for Windows, OS X and Linux). The least problems you can expect – is a tricky build process or somebody’s typo in the Makefile which breaks the-other-system’s build.

More often what you’ll encounter – is people building a huge pile of code working only for their needs. Only for their server. Only for their version if libcurl. Only for x86 operating system. And then they open-source it to GitHub – much like a cemetery for projects with 1 star and 0 forks, decaying there until forgotten forever.

This is how the initial joy of finding an open-source technology you needed is being replaced by a constant frustration of not just a need to slightly tweak some header file or Makefile, but to go the sources, read them, understand everything inside and fix. This is what I have encountered many times and what I did as well.

(more…)

Unicode support for avformat_open_input in Windows

For those of us ever writing cross-platform application there has always been enough quires and quests to accomplish. Typical one is to correctly handle multibyte/unicode filepaths in Windows. And though they are handled pretty good in Qt, when you write your own library you have to do it yourself.

Another level of quests is using third-party libraries which were not designed for cross-platform usage. For example if you wanted to use ffmpeg / libav libraries in Windows, you have to deal with lack of support of std::wstring parameters in the API. One way to deal with it – arrange a custom IO using AVFormatContext and handle file paths by yourself. I have found a wonderful article and code example of how to do it in the blog of Marika Wei. Slightly adapted, the solution will handle all Windows paths


struct {
#ifdef _WIN32
    std::wstring m_FilePath;
#else
    std::string m_FilePath;
#endif
    AVIOContext *m_IOCtx;
    uint8_t *m_Buffer; // internal buffer for ffmpeg
    int m_BufferSize;
    FILE *m_File;
}

#ifdef _WIN32
    m_File = _wfopen(m_FilePath.c_str(), L"rb");
#else
    m_File = fopen(m_FilePath.c_str(), "rb");
#endif

m_IOContext = avio_alloc_context(
    m_Buffer, m_BufferSize, // internal buffer and its size
    0, // write flag (1=true, 0=false)
    (void*)this, // user data, will be passed to our callback functions
    IOReadFunc,
    0, // no writing
    IOSeekFunc
);

Check out the full code at GitHub.

Replacing QNetworkAccessManager for the great good

Everybody using Qt for networking for small tasks will sometimes face oddities of QNetworkAccessManager. This class aims to be useful and convenient while having few quite sensible drawbacks. First one of couse is inability to use it in blocking way. What you should do instead is to create instance of QEventLoop and connect it’s quit() signal with network manager.

QNetworkAccessManager networkManager;
QEventLoop loop;
QNetworkReply *netReply = networkManager.get(resource);
connect(netReply, SIGNAL(finished()), &loop, SLOT(quit()));
loop.exec();    

This is overkill and overengineering of course. This inconveniency strikes also when you try to use it from background thread for downloading something – QNetworkAccessManager needs an event loop and it will launch one more thread – it’s own to do all the operations required.

Also it has a lot of data, methods and abilities not needed for “everyday simple network operations” like querying some API or downloading files. I don’t know anybody who wasn’t looking for a substitude for it at least once. But fortunately the solution exists.

(more…)

Resources to learn and understand parallel programming. The hard way

There’s no way other than the hard way. (c)

Parallel programming is considered as not easy or even advanced topic by many programmers. It’s the starting point for even more advanced stuff like distributed computations, reliability, CAP theorem, consensus problems and much more. Besides, deep understanding of how CPU and operating system works can help you to write less buggy software and parallel programming can help you with that too.

In this post I will focus on books describing parallel programming using 1 computer and 1 CPU using classical approaches. Neither they contain SSE instructions guides nor you will find matterials on CUDA or OpenCL. Similary you will find no resourced about Hadoop and/or MapReduce technologies and nothing about technologies supporting parallel programming out of the box like Go or Erlang.

So I will go now through all the resources which I find more or less useful. I’m not going to stick to any technology in general – the point is to understand the topic from different perspectives. The materials I’m refering to in general should not be considered as entry-level –  they require fair amount of knowledge, but nevertheless, list goes sorted starting from “easier” things.

(more…)

Implementing spellchecking in desktop application in C++

When user is supposed to enter significant amount of text in your application, it’s better to help him/her to control it with checking spelling. Basically, to check spelling you need a dictionary with words and algorithm to order these words. Also it might be useful to provide user with possible corrections for any spelling error. Here where Hunspell comes handy. It’s an open source library built on top of MySpell library and used in a significant number of projects varying from open source projects like Firefox to proprietary like OS X. It contains bindings to a number of platforms (.NET, Ruby etc.) and should be fairly easy to integrate to your project. In this post I’ll discuss how to integrate it to C++/Qt project.

(more…)

Classic Producer-Consumer in Qt/C++

Producer-Consumer is a classic pattern of interaction between two or more threads which share common tasks queue and workers who process that queue. When I came to similar task first I googled for standard approaches in Qt to solve this problem, but they were based on signals/slots plus synchronization primitives while I wanted simple and clear solution. Of course, in the end I’ve invented my own wheel and I invite you to take a look at it.

For the synchronization in Producer-Consumer it’s useful to use Mutex and some kind of WaitingEvent for synchronous waiting until mutex is acquired. In Qt you have QMutex and QWaitCondition which are all that we need.

Let’s suppose we have following data structures:

        QWaitCondition m_WaitAnyItem;
        QMutex m_QueueMutex;
        QVector<T*> m_Queue;

where T is type of messages we’re producing/consuming. So we have queue of elements being processed, mutex to secure access to the queue and wait condition to wait if the queue is empty.

For Producer-Consumer usually we need methods produce() and consume(). Let’s see how we can implement them.

(more…)

Implementing autocomplete for English in C++

When it comes to implementing autocompletion in C++ in some type of input field, the question is which algorithm to choose and where to get the source for completion. In this post I’ll try to answer both questions.

As for the algorithm, SO gives us hints about tries, segment trees and others. You can find good article about them. Author has implemented some of them in a repository called FACE (fastest auto-complete in the east). You can easily find it on GitHub. This solution is used for the autocompletion in search engine Duck-Duck-Go which should tell you how good it is. Unfortunately their solution requires dependencies on libuv and joyent http-parser, which is not good in case you need just to integrate autocompletion functionality into your C++ application, but not build auto-complete server and send queries to it. Another drawback – libuv and cpp-libface itself fails to compile in Windows which is bad in case you’re building cross-platform solution.

You can find out how to built FACE into your cross-platform C++ application below.

(more…)

Handling drag’n’drop of files in Qt under OS X

If you ever tried to handle drag’n’drop files in your Qt application, you would usually come up with the code like the following.
First of all you will need a Drop Area somewhere in your application, which will handle drops

DropArea {
  anchors.fill: parent
  onDropped: {
    if (drop.hasUrls) {
      var filesCount = yourCppModel.dropFiles(drop.urls)
      console.log(filesCount + ' files added via drag&drop')
    }
 }
}

Where yourCppModel is a model exposed to Qml in main.cpp or wherever like this:

QQmlContext *rootContext = engine.rootContext();
rootContext->setContextProperty("yourCppModel", &myCppModel);

and int dropFiles(const QList<QUrl> &urls) is just an ordinary method exposed to QML via Q_INVOKABLE attribute.

You will sure notice everything works fine unless you’re working under OS X. In OS X instead of QUrls to local files you will get something like this: file:///.file/id=6571367.2773272/. There’s a bug in Qt for that and it even looks closed, but it still doesn’t work for me that’s why I’ve implemented my own helper using mixing of Objective-C and Qt-C++ code.

(more…)

Tips and tricks to improve performance of your ACM solution

Here I gathered system-programming tricks that can improve performance of your solution in C++ dramatically!

  • Use scanf/printf functions for standard IO instead of cin/cout
  • Memory-align buffers and structures to WORD size of your architecture (4 bytes for 32-bit and 8 bytes for 64-bit)
  • Use arrays instead of linked lists (to use memory block caching)
  • Avoid “if” stamements in loops
  • If-close should contain code, which is more likely to execute (if-condition == true)
  • Use inlining for short functions
  • Use objects allocated on stack but not on heap (local objects for functions instead of allocated with malloc/new)
  • Use pre-calculated hardcoded data (e.g. you can store first N prime numbers or first N Fibonacci numbers in order not to calculate them every time you need one)

Interesting issues and features of Qt programming

In this post I enlist all interesting facts and issues I’ve experienced while developing my first project in Qt (from 5.3 to 5.4 5.6)

18.02 – QSettings interface in Qml transformed bool to string and was always true on deserialization

18.02 – QByteArray returned from local scope crashed with heap corruption on return of function (destructor of QByteArray)

Qt can delete your object in it’s gui loop. QObject should have CppManaged attribute and it should be set before returning object to UI code.

26.02 QTimer can be only started when EventLoop is already running (e.g. with app.exec()) otherwise you get an error “QTimer can be started in QThread”

Drag and Drop files in OS X inserts NSUrl instead of QUrl and you have to convert it using Objective C to real filepath. You can add .m file to Qt project and write C++/Objective-C code.

23.03 QtConcurrent::mapped can accept a struct with operator() but only with inner typedef T return_type; where T is mapped type for correct QFuture<T> conversion. Also, QtConcurrent::mapped cannot be cancelled.

30.05 Qt Column allows to do animations, but has issues with stretching and ColumnLayout has no issues with stretching, but doesn’t allow you to do animations. I had to use simple anchors layout and States with Transitions to animate properties I wanted

1.06 Qt ListView has Transitions add/remove/removeDisplaced/displaced/etc, which allow to create nice animations for adding/deleting items from listview and to make UI really nice

6.06 Qt lacks standard Zip/Unzip functionality. QuaZip makes life easier, but has minor issues with compilation (linking etc).

22.07 TextInput’s EditingFinished signal fires twice in OS X

6.09 Windeployqt does not pack qml dlls into the bundle, you have to do it manually from your <Qt-bin-dir>/qml. You need to copy QtQuick and QtQuick2 directories and others which you use

19.09 Tab control is a loader and you can’t access it’s child object by Id. You have to create a property of Tab and do double binding inside and outside Tab to that property.

25.10 Repeater in GridLayout does not respect QAbstractItemModel changes, but does respect once put into Flow

23.11 Always initialize boolean fields, volatile or not. Always do that because in other case you will get tons of unpredictable behavior.

28.11 Use QVector instead of QList for most types. If you have released public version with wrong type and users depend on it, you’re screwed

12.12 QML ListView fails to update itself after sophisticated filter/remove operations. You have to manually positionViewAtBeginning() stuff

20.03 QFile in Windows does not respect flag QIODevice::Unbuffered because of “lack of native support in Windows” (bullshit) as of Qt 5.6

21.03 QProcess has super strange problems. Exiftool does not work with unicode from QProcess but works when launched via cmd.exe

2.05 QWaitCondition destructor produced a warning that it’s destroyed while inner Mutex is still locked. Was able to debug that using export QT_FATAL_WARNINGS=true in the running environment

20.07 QString::fromLatin1() truncated buffer of QByteArray parameter if the latter contained some eol/nl characters

16.12 virtual inheritance from the QObject is not supported.. Need to have stubs for calling signals

17.03 Timer in QML can have very high CPU utilization. Need to use with care