Posted Fri 17 Mar 2017 08:43:55 AM BRT
No próximo sábado, dia 18 de Março de 2017, vou participar de uma oficina de Debian em gadgets, incluindo smartphones. Com esse post e outros por vir, gostaria de ajudar a preparar outras oficinas do tipo, e documentar algumas das atividades propostas e realizadas durante a oficina.
Você pode ver os demais posts aqui.
Para se preparar para a oficina, um aviso é muito importante: você pode perder seu aparelho e seus dados. Faça um backup dos seus dados antes de vir para a oficina. E caso opte por tentar alguns dos procedimentos descritos, esteja ciente dos riscos do seu aparelho não ligar mais, ou de você perder seus dados no processo.
Além disso, se possível, traga consigo um cabo USB de dados, um cartão SD compatível com seu aparelho, e um notebook com os seguintes pacotes instalados: abootimg, adb, fastboot, heimdall-flash. Traga o máximo que puder. Se for capaz de obter root em seu aparelho, melhor ainda.
Outros preparativos também são bem-vindos para algumas versões mais avançadas da oficina, o que vou tentar abordar durante a Mini Debconf Brasil 2017. Lembre-se de fazer a sua inscrição gratuita no site do evento. Ela é necessária para a entrada na UTFPR, local do evento.
Declarando IRPF com software livre
Comecei a escrever o código utilizando Python. Mas como não sou proficiente na linguagem e outras mudanças exigiriam alterar o código em Java do IRPF Livre, o processo de desenvolvimento travou.
Então me veio a idéia de utilizar comandos que serviriam como um formato de entrada em simples texto para dar maior usabilidade comparada à edição de XML. Aproveitei a idéia e comecei a escrever em C. No entanto, ao invés de gerar como saída o XML que serviria de entrada ao IRPF Livre, decidi gerar o arquivo em formato DEC de uma vez. Este formato é o formato final utilizado para envio à Receita Federal.
Com a minha experiência anterior com o rnetclient, ajuda do código do IRPF Livre, um arquivo que descreve os campos de cada linha do arquivo DEC e um pouco mais de engenharia reversa, tive sucesso em emitir minhas declarações nos últimos dois anos.
Mas o programa ainda não é tão amigável quanto eu gostaria. Os comandos não são documentados, os erros nem sempre são muito claros, e alguns usuários ainda prefeririam uma interface de formulários.
Para mudar isso, adicionei ao código, recentemente, uma interface de linha de comandos e um comando de ajuda, além do código necessário para emitir melhores mensagens de erro. Agora, as mensagens de erro precisam ser escritas, assim como os textos de ajuda.
Outra idéia que tive foi a de criar um modo tutorial, já que a interface de linha de comando foi adicionada. De alguma maneira, tal modo tutorial poderia indicar pontos de melhoria no programa para permitir novas interfaces no futuro.
I am typing this on an editor running on Debian on my Samsung Galaxy S Relay 4G, aka apexqtmo. Sorry if I got you the wrong image. I am not using its touchscreen or keyboard for input or its screen for output. And the editor is not graphical. I am using a console editor over an SSH session.
I will try to summarize the steps needed to get there, what else I got, the challenges that I had to overcome, and some of the challenges ahead.
My first challenge was to get some output so that I could debug any problems. If I had a terminal, even better. So, this device has a keyboard, which makes things easier for input. I once had it running with a small initrd with GNU bash and got output on the framebuffer, though a little garbled. I realize there are some problems with the framebuffer driver after I have done some tests, which would explain that. For some reason, that didn't work again for me for some time.
So, in the effort to get some debug output, I have spent a lot of time trying to get a Linux binary that would manage to output to a framebuffer console, but without no luck. Then, I realized that besides ADB, I could just configure the USB gadget to act like a serial ACM device, and have a serial console. After that, I have made a lot of progress.
So, I had a small init which would do the configuration, then mount an SD card partition as the root filesystem and have bash for me. Some of the first challenges here were the fact that this old kernel didn't support metadata checksum on ext4, so I had to tune or recreate the filesystem.
After a success with bash, I went for init. As I had a debootstrapped Debian Jessie, that would be systemd, which failed for no known reason yet, but that was expected. Though this was a 3.4 Linux, there are many missing features in the binary which would explain it. So, I went for System V, and after setting it up to have a getty at ttyGS0, I had a login console.
So, another problem is that udev initscript requires devtmpfs, which this Linux binary didn't have enabled. I decided to ignore that and skip udev. I had created some static nodes at /dev/, which were good enough for me.
Then, for the network, I noticed this kernel didn't have wireless as well, so I booted with a different one, but it looks like it requires some extra initialization, maybe some userspace loading firmware, so I skipped that as well. Now, as for a network connection, I then thought I could use the same USB strategy and have it as a multiple interface gadget, with both serial ACM and network functions. That worked wonders.
Now, I could SSH into the smartphone and start running apt, and install packages. You know what's the next step? Get that framebuffer working and have X.org running. More work getting back to a kernel where framebuffer worked. Lots of headaches for that. Though there was a framebuffer driver, it wouldn't work for the kernel that is used for CyanogenMod. It seems it requires OpenGL to work. Or some fiddling with sysfs that I couldn't mimick in order for the display to turn on. So, back to a different kernel, where things seemed to work somehow.
Now, as udev was not running, X.org would give me something at the framebuffer, but no input would work. After trying to configure it manually, change some settings here and there, I found out that only the init scripts checked for the presence of devtmpfs. So, I removed some part of that, and udev was on. But that was after an upgrade to sid, so that might have been necessary, though I can't tell for sure.
For some reason, some X clients didn't like when I used nodm as a display manager, so I went for xinit. And the X server crashed when I used certain clients. I have a hunch this is caused by a framebuffer driver bug in the kernel, which would be a very sad fact if true. So, I got xterm and matchbox-keyboard running. The touchscreen and keyboard worked fine, so now I had something to show off. Though I was excited already when I had "Hello, World!" at my USB serial console.
So, now we realize what a problem we have! These binary kernels, even the ones built by a community like CyanogenMod, require a userspace in order to initialize drivers, including OpenGL and wireless. They don't have basic features turned on, like devtmpfs. It's a daunting task to run a different userspace, that might require these features or not have any code to do those required initializations.
It's still doable to some point, but maybe not sustainable for long. We would either have to work on an alternative userspace, one that does not require or can work without some kernel features, and that can do some of the initialization or workaround some of the bugs we find in the drivers, like the framebuffer one.
The best alternative is to get those devices supported upstream. And, for that, the first step is to get some source code that builds and runs on those things. I didn't manage to get there yet. After some failures with what is supposed to be used for CyanogenMod, I tried getting source code directly from Samsung. As I couldn't find my device in their opensource site, I asked them for the source code. Four days later, they sent me a message that the source was now available in their site. That was a nice surprise. I didn't expect them to answer, and that was quick enough for me to try it before I was about to present this topic at Latinoware. It turns out I couldn't boot that yet either, and it is a 3.0 Linux, so old enough not to be supported by the latest Debian.
Next I am going to write how to try the same thing for your own Samsung gadget at your own risk, no warranties.
Some time ago, I built a static program that I wanted to run on an Android tablet. What was my surprise when I saw a message saying "FATAL: kernel too old".
After some investigation, it turns out that GNU libc may assume some Linux features are present during build time. This means that given a minimum Linux version, that built libc might only work on that version or newer.
Since 2014, GNU libc itself requires 2.6.32 as the minimum. Previously, it was 2.6.16, changed in 2012.
Debian, as of 2015, builds it with a required minimum Linux version of 3.2.
To give an idea about the history of these kernel releases, we have:
- December 2009, Linus releases Linux 2.6.32.
- November 2010, Red Hat releases RHEL 6.0, Linux 2.6.32.
- February 2011, Debian 6.0 Squeeze, Linux 2.6.32.
- January 2012, Linus releases Linux 3.2.
- April 2014, GNU Libc requires Linux 2.6.32.
- December 2015, Debian GNU Libc uploaded to unstable requires Linux 3.2.
So, at least for GNU libc upstream, it would appear that not many devices would stop being supported, though the situation would not be as good for binary versions of Debian. However, I have a small list of devices that might show otherwise.
- Nokia N900 shipped with Linux 2.6.28.
- Android emulator, a platform called Goldfish, uses Linux 2.6.29.
- The Wii port most widely tested is based on Linux 2.6.32.
Many Android devices have been shipped with Linux 3.4, but I encountered at least one using Linux 3.0.
So, even though many new devices ship with newer versions of Linux, it's still possible to find some new and older devices using versions older than 3.2, and even versions older than 2.6.32 may be found.
Another interesting note is that, without a few patches, it's not possible to build Linux 2.6.32 with GCC 5 and newer. For that and many other reasons, it's important that we update. For bug fixes, and so we can make progress and use better software are some of the other reasons.
So, it's imperative that we have devices support upstream. Otherwise, the task of doing updates with forward porting becomes daunting. And that's the current state for many devices, which is why I have been trying to use new software with older versions of Linux. But as time passes, I realize how hard a task this is, as most new software these days, even a building block like GNU libc, requires ever new versions of Linux.
For now, most gadgets I have support Linux 3.4 or newer. But not long ahead, that support will be dropped as well. And that means there will be no more updates for those devices. It's a consequence of both targeting time-to-market and programmed obsolescence as business practices. Upstream support is no priority, and maintenance is only that required until the next model is available on the stores.
This is one more reason why we need to have more operating systems available for those devices. Systems that are designed to last more than a couple of years. As I said, upstream support is imperative, but as a step forward, I still believe we can provide the GNU experience to lots of devices out there.
Acesso a bancos sem software livre
Há muitos anos, usuários de grandes bancos no Brasil são obrigados a utilizar software não livre em seus computadores para fazerem consultas e operações via Internet.
Em muitos desses casos, era comum a utilização de um applet Java, muitas vezes funcionando apenas com um plugin e um JRE não livre, fornecido pela Sun, hoje Oracle. Tal JRE utilizava uma EULA draconiana, que me recordo interpretar como impedindo aquele que a aceitasse de realizar engenharia reversa, algo que algumas juridições permitem.
Além do mais, há a limitação da interface a um browser Web, dificultando o acesso via outras interfaces, como linha de comando, e automação de operações por parte do usuário.
Acesso a bancos com software livre
Não seria muito melhor se o software para acesso aos bancos fosse livre e permitisse o uso de diferentes interfaces e automação? Que modificações a esse software pudessem ser feitas e compartilhadas para que outras pessoas pudessem utilizá-las?
Imagine poder obter seus extratos periodicamente e alimentar um banco de dados próprio, agregando movimentações financeiras de múltiplas fontes. E fazê-lo de forma automática, conhecendo como o software funciona e o adaptando como preferir.
Esse é um projeto em que venho trabalhado há algum tempo, sem muito sucesso, mas com algum progresso que nunca documentei apropriadamente. Tal documentação é uma forma de compartilhar meu trabalho com outros, para que possam contribuir e ajudar no progresso de tal trabalho.
Meu objetivo com este post é documentar uma parte do trabalho que venho fazendo com a Caixa Econômica Federal. A CEF exigia um applet Java até alguns meses atrás, quando passou a exigir a instalação de um servidor WebSocket.
Uma primeira observação importante é que, independente do mecanismo, seja o applet ou o servidor WebSocket, após o login, cookies podem ser reutilizados e operações realizadas.
Ou seja, é possível fazer o login utilizando um computador com o software proprietário instalado, exportar os cookies da sessão de navegação, e importá-los em outra sessão em outro computador. A partir daí, as operações podem ser realizadas sem utilização e interferência do software não livre.
Antes da substituição do applet Java, escrevi um script que permitia fazer um login e obter o saldo através de requisições HTTP. Sem utilização do applet Java, uma máquina podia ser cadastrada e seu cadastro utilizado para o login. No entanto, outras operações não eram permitidas. Ainda que utilizado o cookie em um browser, até mesmo qualquer consulta que não o saldo era negada.
Talvez fosse uma falha no cadastro, ou na operação de login, mas não pude determinar antes que a substituição acontecesse.
No entanto, novamente nenhuma substituição de operações realizadas pelo applet foram necessárias. O que me levou a descobrir que o applet não estava sequer operando como esperado.
Uma nova opção é fazer a engenharia reversa da comunicação WebSocket e substituir o servidor WebSocket. Daí, tal comunicação poderia ser substituída por chamadas locais.
Ainda assim, deve ser avaliada a possibilidade de ignorar tal comunicação totalmente. No entanto, ainda não tive sucesso em realizar o login como anteriormente, sem uso de algumas chaves aparentemente geradas pelo servidor.
Também sugiro conversar na lista de e-mails software-impostos.
On recent posts, I mentioned that I have a Chromebook, and that I would like to run Debian on devices that ship with old Linux versions. The Samsung ARM Chromebook is such a device, it has Linux 3.4, and that's still what I am running on it most of the time.
After an upgrade of Debian, systemd stopped working, and it took me some time before I could look into it. In order to boot, I had to temporarily use System V init. The bug involves the use of new interfaces, bugs in such interfaces, and how versioning would not be a solution to such cases. Cases like this are not common, so it's no excuse to doing the right thing when developing software, like considering portability, API and ABI maintenance and not bundle. But they expose some of the challenges when trying to support different OSes on top of old versions of Linux, or maybe even new versions.
The new version of systemd included in Debian Jessie uses a feature of Linux of checking the mount ID of a file by inspecting the mnt_id line in the fdinfo files. fdinfo files are not that new, they have been present since 2007, Linux 2.6.22. The mnt_id line, however, is from Linux 3.15, from 2014. So, a fairly new feature. But systemd falls back to other methods if the feature is not present.
However, during the last few weeks of development of Linux 3.4, a bug was introduced to fdinfo handling, which will cause its access to fail in a way that will cause systemd to halt. It simply will refuse to mount the most basic filesystems and not proceed. The bug was fixed not much longer than two weeks after that, but that window of time was enough to have it on a shipped product, one that was not using systemd, let alone one that required a feature not yet present on Linux.
But now I wanted to run this new systemd on this old Linux version. Being too early in the boot process, systemd even ignored some of the debug options and I had to resort to other methods of debugging. As I am running a kernel I built by myself, it wasn't hard for me to have a fixed kernel up and running after I found out the problem. Unfortunately, not everyone will be able to do it by themselves. And the point of getting different OSes working on these many varied devices is scaling. If every device needs to be tested, we lose this scalability.
The other problem I found was that this old Linux version didn't load firmware files by itself, and requested help from userspace, which used to be the job of udev. Well, udev now does not support loading firmware, and the kernel must do it by itself. I managed to backport that feature too. But now, it sounds like I should get upstream support for this device.
Unfortunately, this means not working on the scalability problem, but going back to get upstream support for a lot of devices out there, which is very hard to scale without the necessary people to do the work. I am certainly going to do that, because it's the right thing to do. But not as many people will benefit from that.
But I think it has been interesting to look into the possible problems we might find when trying to make Debian and other OSes working on old Linux versions. I know even older Debian versions won't boot fine on Linux 2.6.32, one of the LTS versions of Linux. I have one other device which runs on it, and I am also working on getting better upstream support for it. Let's see if I can get some news on that soon.
Three years ago, I wanted to get a new laptop. I wanted something that could run free software, preferably without blobs, with some good amount of RAM, good battery and very light, something I could carry along with a work laptop. And I didn't want to spend too much. I don't want to make this too long, so in the end, I asked in the store anything that didn't come with Windows installed, and before I was dragged into the Macbook section, I shouted "and no Apple!". That's how I got into the Chromebook section with two options before me.
There was the Chromebook Pixel, too expensive for me, and the Samsung Chromebook, using ARM. Getting a laptop with an ARM processor was interesting for me, because I like playing with different stuff. I looked up if it would be possible to run something other than ChromeOS on it, got the sense that is, it would, and make a call. It does not have too much RAM, but it was cheap. I got an external HD to compensate for the lack of storage (only 16GB eMMC), and that was it.
Wifi does require non-free firmware to be loaded, but booting was a nice surprise. It is not perfect, but I will see if I can get to that another day.
I managed to get Fedora installed, downloading chunks of an image that I could write into the storage. After a while, I backed up home, and installed Debian using debootstrap.
Recently, after an upgrade from wheezy to jessie, things stopped working. systemd would not mount the most basic partitions and would simply stop very early in the boot process. That's a story on my backlog as well, that I plan to tell soon, since I believe this connects with supporting Debian on mobile devices.
After fixing some things, I decided to try libinput instead of synaptics for the Trackpad. The Chromebook uses a Cypress APA Trackpad. The driver was upstreamed in Linux 3.9. The Chrome OS ships with Linux 3.4, but had the driver in its branch.
After changing to libinput, I realized clicking did not work. Neither did tapping. I moved back to synaptics, and was reminded things didn't work too well with that either. I always had to enable tapping.
I have some experience with input devices. I wrote drivers, small applications reacting to some events, and some uinput userspace drivers as well. I like playing with that subsystem a lot. But I don't have too much experience with multitouch and libinput is kind of new for me too.
I got my hands on the code and found out there is libinput-debug-events. It will show you how libinput translates evdev events. I clicked on the Trackpad and got nothing but some pointer movements. I tried evtest and there were some multitouch events I didn't understand too well, but it looked like there were important events there that I thought libinput should have recognized.
I tried reading some of libinput code, but didn't get too far before I tried something else. But then, I had to let this exercise for another day. Today, I decided to do it again. Now, with some fresh eyes, I looked at the driver code. It showed support for left, right and middle buttons. But maybe my device doesn't support it, because I don't remember seeing it on evtest when clicking the Trackpad. I also understood better the other multitouch events, they were just saying how many fingers there were and what was the position of which one of them. In the case of a single finger, you still get an identifier. For better understanding of all this, reading Documentation/input/event-codes.txt and Documentation/input/multi-touch-protocol.txt is recommended.
So, in trying to answer if libinput needs to handle my devices events properly, or handle my device specially, or if the driver requires changes, or what else I can do to have a better experience with this Trackpad, things were tending to the driver and device. Then, after running evtest, I noticed a BTN_LEFT event. OK, so the device and driver support it, what is libinput doing with that? Running evtest and libinput-debug-events at the same time, I found out the problem. libinput was handling BTN_LEFT correctly, but the driver was not reporting it all the time.
By going through the driver, it looks like this is either a firmware or a hardware problem. When you get the click response, sound and everything, the drivers will not always report it. It could be pressure, eletrical contact, I can't tell for sure. But the driver does not check for anything but what the firmware has reported, so it's not the driver.
A very interesting I found out is that you can read and write the firmware. I dumped it to a file, but still could not analyze what it is. There are some commands to put the driver into some bootloader state, so maybe it's possible to play with the firmware without bricking the device, though I am not sure yet. Even then, the problem might not be fixable by just changing the firmware.
So, I left with the possibility of using tapping, which was not working with libinput. Grepping at the code, I found out by libinput documentation that tapping needs to be enabled. The libinput xorg driver supports that. Just set the Tapping option to true and that's it.
So, now I am a happy libinput user, with some of the same issues I had before with synaptics, but something you get used to. And I have a new firmware in front of me that maybe we could tackle by some reverse engineering.
O rnetclient 2016.0 já está disponível aqui.
A nova versão suporta a transmissão de declarações de 2016, além de retificações de 2013 a 2015. Possivelmente, anos posteriores são suportados, desde que não haja mudanças inesperadas no protocolo da Receita Federal.
Outras melhorias incluem:
Não transmitir a declaração se não for possível gravar o arquivo de recibo. Em versões anteriores, se um arquivo de recibo já existisse, o arquivo não era sobrescrito, mas o novo recibo não era gravado em nenhum arquivo.
Teste unitário, verificando que alterações no código ainda permitem a correta identificação e leitura de declarações de anos anteriores.
Agradeço ao Gabriel F. T. Gomes pelos testes e contribuições a esta versão.
Caso tenha algum problema com o uso do rnetclient, por favor, reporte na lista software-impostos.
Some time ago, I wrote how to get GNU on a smartphone. This is going to be a long discussion on why and how we should work on more operating systems for more devices.
So, why should we bother if we already have Android, some might ask? If it's just because of some binary blobs, one could just use Replicant, right?
Well, one of the problems is that Android development is done in hiding, and pushed downstream when a new version is launched. There is no community behind that anyone can join. Replicant ends up either following it or staying behind. It could do a fork and have its own innovations. And I am all for it. But the lack of manpower for supporting devices and keeping up with the latest versions and security fixes already takes most of the time for the one or two developers involved.
Also, Android has a huge modularity problem, that I will discuss further below. But it's hard to replace many components in the system, unless you replace them all. And that also causes the problem that applications can hardly share extra components.
I would rather see Debian running on my devices and supporting good environments and frameworks for all kinds of devices, like phones, tablets, TVs, cars, etc. It's developed by a community I can join, it allows a diverse set of frameworks and environments, and it's much easier to replace single components on such a system.
On Copyleft Hardware
I get it. Hardware is not free or libre. Its design is. I love the concept of copyleft hardware, where one applies the copyleft principles to a hardware design. Also, there is the concept of Hardware that Respects Your Freedom, that is, one that allows you to use it with only free software.
Note that RYF hardware is not necessarily copyleft hardware. In fact, most of the time, the original vendor has not helped at all, and it required reverse engineering efforts by other people to be able to run free software on those systems.
My point here is that we should not prevent people from running free software on hardware that does not RYF or that is not copyleft. We should continue the efforts of reverse engineering and of pushing hardware vendors to not only ship free software for their hardware, but also release their design under free licenses. But in the meantime, we need to make free software flourish in the plethora of devices on the hands of so many people around the world.
I won't go into details on this post about two things. One topic I love is retrocomputing and how Linux supported so many devices, and how many free or quasi-free operating systems ran on so many of them in the past. I could mention ucLinux, Familiar, PDAs, Palmtops, Motorolas, etc. But I will leave it to another time and go from Openmoko and Maemo forward.
The other topic is application scalability. Even Debian with so many software available does not ship all free software there is available. And it doesn't support all third-party services out there. How can we solve that? It has to do with platforms, protocols, open protocols, etc. I will not go into that today.
Because I believe that either way, it's healthy for our society to have diversity. I believe we should have other operating systems available for our devices. Even if application developers will not develop for all of them. That is already the case today. There are other ways to fix that, when that needs fixing. Sometimes, it's sufficient that you can have your own operating system on your device, that you can customize it, enhance it and share it with friends.
And also, that would allow for innovation. It would make possible that some other operating system would have enough market share on some other niche. Other than Android and iOS, for example. But that requires that we can support that operating system on different devices.
And that is the scalability that I want to talk about. How to support more devices with less effort.
But before I go into that, let me write more about the alternatives we have out there. And some of the history around it.
Openmoko developed a set of devices that had some free design. And it has some free operating systems running on top. The community developed a lot of their own as well. Debian could (still can) run on it. There is SHR, which uses FSO, a mobile middleware based on D-Bus.
Maemo, Moblin, Meego and Tizen
I remember the announcement of Nokia N770. During FISL in 2005, I even criticized a lot, because it shipped with proprietary drivers. And applications. But it was the first time we heard of Maemo. It was based on Debian and GNOME. The GNOME Mobile initiative was born from that, I believe, but died later on.
But with the launch of N900, and later events, like the merge of Moblin with Maemo, to create Meego, we all had an operating system that was based on community developed components, that had some community participation, and was much more like the systems we were used to. You could run gcc on N900. You could install Das U-Boot and have other operating systems running there.
But Nokia has gone a different path. Intel has started Tizen with Samsung. There is so much legacy there, that could still be developed upon. I am just sad that Jolla decided to put proprietary layers on top of that, to create SailfishOS.
Not to forget, Neo900, a project to upgrade the board, in the same vein as GTA04.
FirefoxOS and Ubuntu
I won't go too much here in what I think about the Web as a platform. But I think we need a system that has more options for platforms. I haven't participated in projects directed by Mozilla either, so I can't say much about that.
Ubuntu Phone should be a system more like what we are used to. But Canonical is going to set the directions to the project, it's not a community project.
Nonetheless, I think they add to the diversity, and users should be able to try them, or their free parts or versions. But there are challenges there, that I think need to be discussed.
So, both of these systems are based on Android. They don't use Dalvik or the application framework. But they use what is below that, which means device drivers in userspaces, like RIL, for the modem drivers, and EGL, for graphics drivers. The reason they do that is to build on top of all the existing vendor support for Android. If a SOC and phone vendor already supports Android, there should be not much needed to do to support FirefoxOS or Ubuntu Phone.
In practice, this has not benefited the users or the projects. Porting should be as simple as getting a generic FirefoxOS or Ubuntu Phone image and run it on top of your Android or CyanogenMod port.
Porting usually requires doing all the same work as porting Android. Even though one should be able to take advantage of existing ports, it still requires doing a lot of integration work and building an entire image, most of the time, including the upper layers, that should be equal in all devices. This process should require at most creating a new image based on other images and loading it on the device. I will discuss more about this below.
I can't forget to mention the Matchstick TV project. It is based on FirefoxOS. I think it would have much more changes to succeed if it was easier for testers to have images available for all of their gadgets capable of running some form of Android.
And then we have Replicant. It has a lot of the problems Android has. Even so, it's a tremendously important project. First, it offers a free version of Android, removing all proprietary pieces. This is very good for the community. But more than that, it tries to free the devices beyond that.
What the project has been done is reverse engineer some of the pieces, mostly the modem drivers. That allows other projects to build upon that work, and support those modems. Without that, there is no telephony or celullar data available on any of these devices. Not without proprietary software, that is.
They also have been working on free bootloaders, which is another important step for a completely free system.
There are many challenges here, but I believe we should work on a set of steps that make it more palatable and produce intermediary results that can be used by users and other projects.
One of the goals should be good modularity. The ability to replace some pieces with the least trouble necessary. The update of a single framework library should be just a package install, instead of building the entire framework again. If there is ABI breakage, users should still be able to have access to binaries (and accompanying source code) and only need to update the library and its reverse dependencies. If one layer does not have these properties, at least it should be possible to update this big chunk without interfering with the other layers.
One example is FirefoxOS and Ubuntu Phone. Even if there is a big image with all system userspace and middleware, the porting, install and upgrade process should allow the user to keep the applications and to leverage porting already done by similar projects. Heck, these two projects should be able to share porting efforts without any trouble.
So what follows is a quick discussion on Android builds, then suggestions on how to approach some of the challenges.
On Android build
The big problem with Android build is its non-modularity. You need to build the world and more to get a single image in the end. Instead of packages that could be built independently, that generated modular package images, that could be bundled in a single system image. Better yet would be the ability to pick only those components that matter.
Certainly, portability would be just as interesting, being able to build certain components on top a GNU system with GNU libc.
At times, it seems like this is done by design, to make it difficult for "fragmentation" and competition. Basically, making it difficult to exercise the freedom to modify the software and share your modifications with others.
Imagine the possibilities of being able to:
- Build Dalvik and the framework on a GNU system in order to run Android programs on GNU systems.
- Or build a single framework library that would be useful for your Java project.
- Build only cutils, because you want to use that on your project.
- Build the HAL and be able to use Android drivers on your system.
- Build only RIL and use the modem drivers with ofono.
There are some dangers in promoting the use of Android like this. Since it promotes proprietary drivers, relying on such layers instead of better community oriented layers means giving an advantage to the first. So, the best plan would be to replace those layers, when they are used, for things like ofono and Wayland, for example.
But, in the meantime, making it easier for users to experiment with FirefoxOS, Ubuntu Phone, Matchstick TV, Replicant on other devices, without resorting to building the entire Android system, would be a very nice thing.
It is possible that there are some challenges with respect to API and ABI. That these layers are too intermingled that some changes present in one port would prevent FirefoxOS to run on top of it without changes in either of the layers. I can't confirm that is the case, but can't deny the possibility.
One of the challenges we have that may have trouble in scaling is rooting devices.
Unfortunately, most of the devices are owned by the vendor or carrier, not the user. The user is prevented from replacing software components in the system, by removing its permissions from replacing most of the files in the filesystem, or writing to block devices, or change boot parameters or write to the storage devices.
Fortunately, there are many documents, guides, howtos, and whatnots out there instructing on how to root a lot of devices. In some cases, it depends on software bugs that can be patched by the users as instructed by vendors.
Certainly, favoring devices that are built to allow rooting, hacking, etc, is a good thing. But we should still help those users out there who do not have such devices, and allow them to run free software on them.
Then, comes booting, which is a large discussion on its own, and also related to rooting, or how to allow users to replace their software.
First, we have the topic of bootloaders, which are usually not free, and embedded in the chip, not on external storage. So, there are those pieces of the bootloader which we can replace more easily and those that we are not, because they would require changing a piece of ROM, for example.
Das U-Boot would be one of the preferred options for a system. It supports a lot of hardware platforms out there, a lot of storage devices and filesystems, network booting, USB and serial booting, and a lot of other things. Porting is not an easy task, but it is possible.
Meanwhile, using the bootloaders that ship with the system, when possible, would allow a path where other pieces of the system would be free.
One of the challenges here is building single images that could boot everywhere. The lack of a single ARM boot environment is a curse and a blessing. It makes it hard to have this single boot image, but, on the other hand, it has allowed so much of the information for such systems to be free, to be embedded in copyleft software, instead of having blobs executed by our free systems, like ACPI encourages.
Device trees have pushed this portability issue from the kernel to the bootloaders. Possibily encouraging vendors to now hide and copyright this information in the form of proprietary device trees. But it has made it easier for single images.
In practice, we still need to see a lot of devices out there supporting this single kernel scenario. And this mechanism only brings us the possibility of a single kernel. We still need to ship bootable images that have pieces of bootloaders that are not as portable.
This has caused lots of operating systems out there to be built for a single board. Or to support just a small set of boards. I struggle with the concept. Why are we not able to mix device-specific pieces with generic pieces and get a resulting image that will boot on our choice of board? Why does every project need to do all the porting work again, repeating the efforts? Or have one entier ISO image for every supported board? Check how Geexbox does it, for an example.
Fortunately, I see Debian going in a good direction. Here, one can see how it instructs the user to combine a system-dependent part with a system-independent part.
Which brings us to the installation challenge. We should make this easy and also customizable by the user. Every project might have its own method, and that is part of the diversity that we should allow.
The great challenge here is handling rooting and booting for the user. But it's also possible to leave that as separate efforts, as it would be nice to have good installers as we have in the desktop and server environments.
Linux is one of the most ported kernels out there. Of course, it would be healthy to the diversity I propose that other kernels and systems out there should work on those mobile devices. NetBSD, for example, has a reputation of being very portable and ported to many platforms. And GNU runs on many of them, either together with the original systems, or as the main system on top of other kernels, as proven by the Debian GNU/kFreeBSD port, which uses GNU libc.
But though this looks like one of the easier tasks at hand, it is not. If you run a derivative version from Linux, provided by the vendor, things should go smooth. But most of the time, the vendor does not provide patches upstream, and leave their fork to rot, if you are lucky, in the earlier 3.x series.
And some times, there is not even source code available. There are vendors out there who do not respect the GPL and that is one of the reasons why GPL enforcement is important. I take this opportunity to call attention to the work of Software Freedom Conservancy and its current efforts to raise funds to continue their work.
Running an existing binary version of Linux on your device with a free userspace is part of the strategy of replacing pieces one at a time while allowing for good diversity and reaching more users.
Then we have the matter of drivers. And in this case, it's not only Linux drivers, but drivers running in userspace. Though others exist, there are two important and common cases, which are modem and graphics drivers, both of which are usually provided by Android frameworks, and that other systems try to leverage, instead of using alternatives that are more community friendly.
In the case of modem drivers, Android uses its RIL. Besides not using a Radio or Modem at all, there are two strategies for free versions. One is using free versions of the drivers for RIL. That's what Replicant does, because, well, it uses the other Android subsystems after all. The other one is using a system that is developed by the community. ofono is one that, even though it's an Intel iniative, looks more community governed or open to community participation than any Android subsystem ever was.
As for Graphics, Canonical even built its Mir project with the goal of being able to use Android GPU drivers. Luckily, there has been a lot of reverse engineering efforts out there for a lot of those GPUs shipped with ARM SoCs.
Then, we can use Mesa, Wayland and X.org. Other option, until then, is just using a framebuffer driver, possibly one that does not need any initialization, depending on one done by the bootloader, for example.
Middleware and Shells
Plasma Active, that I just found out now is Plasma Mobile, looks like a great system. Built by folks behind KDE, we can expect great freedom to come from it. Unfortunately, it suffers from what we have been discussing here, which is lack of good device support, or shipping images that run on top of single devices, without leveraging the porting that has come before of other systems. Fortunately, that is just what I want to accomplish with this effort.
FSO, that I mentioned before, is a good middleware, that we should try to run on top of these devices. Running GNOME or Unity shell, and use rpm or deb based systems, that's all part of the plan on diversity. You could use systemd, or System V init systems, whatever gives you the kicks.
It's not an easy task, since there are so many new things on these new kinds of devices. Besides touch, there is telephony, which as I mentioned, would be a good task for ofono. As for TV sets or dongles, I would love to see OpenFlint, created by the Matchstick TV guys, flourish out there and allow me to flint stuff from my mobile phone running Debian into my TV dongle running Debian.
So, is there a project out there you can start contributing to? Well, I pointed out a lot of them. All of them make part of this plan. Contributing to reverse engineering GPUs, or to Plasma Mobile, ofono, GNOME, Linux-libre, bootloaders, Debian, and so many other projects.
My plans are to contribute in the scope of Debian and make sure it works well on top of semi-liberated devices, and make sure there is a nice user interface and good applications when using either GNOME or Plasma. Integrating ofono is a good next step, but I am running ahead of myself.
Right now, I don't think there is need for an integrated effort. If you think otherwise, please let me know. If you are doing something in this direction, I would love to know.
Paul Wise reminded me to point out to the Debian Mobile, where I will probably contribute any documentation and other relevant results.
There are some many people to thank, but I would like to remember Ian Murdock, who created Debian, one of the projects who inspires me the most. I think the best way to handle his passing is to celebrate what he has helped create, and move forward, taking more free software to more people in an easy and modular way.
I have been wanting to write something like this for a long time, so I thank Chris Webber for inspiring me on doing it.