2013-05-31

The state of hardware acceleration on ODROID (xf86-video-sunxi, xf86-video-mali and kernel 3.8)

Yesterday I tried xf86-video-sunxi (thanks to ssvb, rz2k and xjuan for the support!) and I have to say that this driver bears promise, in particular regarding G2D support, that hopefully in future will bring us 2D hardware acceleration.

The rotating horse is always fun, even when you rotate it.
I have run tests with the following combinations (some old test logs available here):

Kernel X11 Driver Resolution es2gears glmark2-es2 glmark2-es2 --fullscreen glxgears
3.0.79 ARM mali (31 may) 720p ~ 345  fps 160 96 125 fps
3.0.79 ARM mali 720p ~ 345 fps 160 96 125 fps
3.0.79 ssvb sunxi 720p ~ 261 fps 153 89 125 fps
3.8.13 ARM mali (31 may) 720p ~ 226 fps 61 37 ~ 140 fps
3.8.13 ARM mali 720p ~ 226 fps 61 37 ~ 140 fps
3.8.13 ssvb sunxi 720p ~ 226 fps 66 42 ~ 144 fps

Update 8 June 2013: see 2nd round of tests

The compiled sunxi driver and glmark2-es2 are included in version 0.0.7 of the odroid-wheezy-retro SD image.

RetroArch

With both kernels and with both X11 drivers RetroArch performs well in windowed mode (50FPS with rare frame drops with the 3.8 kernel).

Conclusions

  • there is currently a performance issue on kernel 3.8, our beloved mdrjr is working on it
  • glxgears is always software-rendered (see HardwareAcceleration), it's included only for reference
  • 1080p resolution is slow with all configurations, thus I have not targeted it for testing
  • the new Mali drivers (r3p2 released on 31st May) are the same as previous
  • xf86-video-sunxi starts to perform better on kernel 3.8, probably because of the NEON bug that affects 3.0
For now I will keep using kernel 3.0 and ARM mali drivers in odroid-wheezy-retro SD image, however I expect great improvements on both fronts (3.8 kernel and sunxi driver).

2013-05-28

A couple of updates for odroid-wheezy-retro (kernel 3.0.79 and 3.8.13)

I have released 2 scripts to upgrade your odroid-wheezy-retro image to a more recent kernel:


  • upgrade-3.0.75-to-3.0.79.sh, this will upgrade to the most recent 3.0.y kernel. This is currently stable & supported (by me) and will be in next SD image release (0.0.7); also contains a patch to silence the fan message spam in kern.log
  • upgrade-3.0.y-to-3.8.13.sh, this is experimental and subject to change without notice. This script will upgrade to a recent (consider it a nightly build) 3.8 kernel, complete with modules
NOTE: it does not work perfectly with X/X2 e.g. you will have to fix some issues manually
I have not yet decided when to put 3.8 on the odroid-wheezy-retro image - I am going to do some testing before that.

If you are installing 3.8 remember to change your device from /dev/fb6 to /dev/fb1 in /etc/X11/xorg.conf

Official ODROID forum discussion here.

2013-05-27

PCSX-ReARMed on ODROID-U2 (through RetroArch)


These notes are useful to get PCSX-ReARMed (by notaz) running on an ODROID-U2, with full ARM optimizations.

Due to some bug in the current configure/Makefiles, you will not be able to compile&run it succesffully. I got errors like this:
RetroArch [ERROR] :: dylib_load() failed: "pcsx_rearmed_libretro.so: undefined symbol: gteNCLIP_arm".
RetroArch [ERROR] :: Failed to open dynamic library: "pcsx_rearmed_libretro.so"
RetroArch [ERROR] :: Fatal error received in: "load_dynamic()"
And this:
RetroArch [ERROR] :: dylib_load() failed: "pcsx_rearmed_libretro.so: undefined symbol: gteRTPT_neon".
RetroArch [ERROR] :: Failed to open dynamic library: "pcsx_rearmed_libretro.so"
RetroArch [ERROR] :: Fatal error received in: "load_dynamic()"
 
So here you are some instructions to get it running.
First thing, get the libretro port:
git clone https://github.com/libretro/pcsx_rearmed

Apply this patch:

diff --git a/Makefile b/Makefile
index a472492..ed95702 100644
--- a/Makefile
+++ b/Makefile
@@ -10,6 +10,9 @@ CXXFLAGS += $(CFLAGS)
 #DRC_DBG = 1
 #PCNT = 1

+ARCH = arm
+HAVE_NEON=1
+
 all: config.mak target_ plugins_

 ifndef NO_CONFIG_MAK
diff --git a/Makefile.libretro b/Makefile.libretro
index 9436c8a..8a0d67a 100644
--- a/Makefile.libretro
+++ b/Makefile.libretro
@@ -97,8 +97,6 @@ else ifeq ($(platform), arm)
    DRC_CACHE_BASE = 0
    BUILTIN_GPU = neon
    ARCH = arm
-   CFLAGS += -marm -mcpu=cortex-a8 -mfpu=neon
-   ASFLAGS += -mcpu=cortex-a8 -mfpu=neon
 else
    TARGET := $(TARGET_NAME)_retro.dll
    CC = gcc

Now you're (almost) set. Run this script:

export CFLAGS='-marm -mcpu=cortex-a9 -mtune=cortex-a9 -mfpu=neon -mfloat-abi=hard'
export CXXFLAGS='-marm -mcpu=cortex-a9 -mtune=cortex-a9 -mfpu=neon -mfloat-abi=hard'
export ASFLAGS='-marm -mcpu=cortex-a9 -mtune=cortex-a9 -mfpu=neon -mfloat-abi=hard'
./configure --enable-neon

make -j5 -f Makefile.libretro platform=arm

And enjoy PCSX-ReARMed on your ODROID :)

Issues

It is very slow! It's not performing very well, although all possible optimizations have been enabled. Update 26 June 2013: It's working very fast! Thanks to Squarepusher for the tip of using platform=arm :)

Furthermore, CSO/ISO are not working. Update 26 June 2013: I don't yet know how to work-around this, will publish a new post when I do

2013-05-26

65c816: from assembly to C: a quest for die hard retroengineers!

Some things are impossible to accomplish

For the rest, it's just that you did not try hard enough.

a skeptic (sketchic?) Ray Arnold
This simple fact is behind my interest in 65c816 and the old games that were developed with it: challenge.

Although I had some Intel assembly experience, I clearly overestimated it - deep diving into 6502/65c816 assembly taught me (the hard way) how things work under the hood, and it's a fashioning digital mechanical world.

On reverse engineering

Here be dragons! read more at Wikipedia and here, talking about ethics in computing.

What will happen when there will be a law to regulate everything? You will cease to be free.

This information is provided for learning purposes, not to reverse engineer some company algorithm to steal the precious industrial secrets.
So follow my advice and use reverse engineering for ethical purposes :)

Learning material

A good collection of reference material is essential to succeed. These are the sources I used most often:
In general the other SNES resources at wikibooks are of good level, you can start from there.

As a general advice: DO NOT use badly written manuals/guides (that is the norm nowadays), or anything that leaves you a doubt. Those doubts will be accumulating even if you don't notice it, and you will waste your time in frustration.
When dealing with assembly you really need to reduce number of variables you are dealing with! Make sure you have a grasp of each conditional branching (vs CPU flags) and how ROL,ADC,SBC work - tip: carry is not trivial! ;)

Other interesting pages


What I have learnt:

  • most past games ('90s era) were written directly in assembly (with macro aid, at best)
  • you had to be very creative to fit the limited resources of the consoles
  • dead code and mistakes are commonplace in these games
  • reverse-engineering without tools is hardly possible - I estimate that with double brain size I could probably have a better shot at it

Context limitation

We are human beings, thus we work best when we can focus on a few related abstractions rather than on a wide context with little correlation.

I am going to release a tool, asmdecor, that I developed as an aid for tool-assisted reverse engineering, hopefully they will be of help to other retroengineers as well. 

Labels are your friends!

Doesn't matter if correct - we need labels to reference abstractions. Absolute/Relative memory addresses are not easy to remember.

NOTE: in assembly world there is little difference between a label and a variable, since they both point to a memory location (although in different banks). Here I use them interchangeably for this reason.

I have come up with these rules to assign labels:
  • assign a label to the variables you are currently focusing on
  • remember to change the label once you have a better insight into its usage
  • if you did not succeed, remove the label - next time you will want to start from scratch instead of being conditioned by your previous (inconcludent) ideas
  • if some variable turns out to be too much generically described, drop it
  • make a smart usage of "copyOf", "previous", "counter" etc. basically you should favour relation-rich names
The tool I developed, asmdecor, helps with:
  • variables used only inside a specific procedure, prefixed as "tmp"
  • variables used only for writing, prefixed as "writeOut"
  • variables used only for reading, prefixed as "const"
  • read/write variables, prefixed with "var"
  • procedures and branching point recognition
All variables also come with reference count in their name, to identify "high traffic"/"high value" buffers.

Tracing is your friend!

When trying to understand a piece of code you will find yourself countless times debugging/tracing (step by step) the code and comparing the values with your own version of the algorithm.

The more complex is the piece of code you are trying to understand, the worse it will be - and you will quickly find out that you are doing it ineffectively.

But this was a quest for die hard retroengineers, remember? So I will share with you how I approach this problem - read on!

Vertical tracing


What I have found useful is a technique that I call "vertical tracing" (or "core sampling"), effective on loops and state machines (decompression etc).

This technique consists in creating two indexed stripes, one from your original 65c816 code (master stripe) and one from your reverse-engineered code (carbon copy stripe).
Each of these stripes is made up of multiple (key, value) tuples, created as follows:
  1. normally you want to sample at each iteration, so you will "tether" the loop by using the counter variable as the index of your stripe
  2. you define a "collection spot" for your loop, e.g. a specific variable in your code that you want to monitor in the master stripe
  3. create the corresponding collection spot in your code
  4. compare the master and carbon copy stripes
At step (4) it is important that you throw an exception or otherwise crash when your program is not matching the master stripe.

Example of stripe

1: 0x6300
2: 0x64C0
3: 0x6A00
4: 0x62B0
5: 0x710A
6: 0x712A

In a more compact form:
const char *masterStripe = "0x6300, 0x64C0, 0x6A00, 0x62B0, 0x710A, 0x712A";

By using the compact form you can string-compare it to your carbon copy stripe and thus easily verify behaviour from within your code.

Note that you can also choose to collect the decision taken at a specific branch instead of the value of some variable.

How to generate the master stripe


The most convenient way is to create a trace of the unit of logic you want to reproduce, then use grep/bash scripts to extract the stripe.

Summary: how to write C code from 65c816 assembly

No, there is no magic tool to convert from assembly to C*.

"Isn't tracing a bit like DNA extraction?",
the wise raptor asked.



Tracing, variable context heuristics and procedure identification are the key ingredients.
asmdecor will help you with the latter two, but you still have to analyze it with your mind (and that's the fun part ;) ).

Having advised with all the above, I am left with just a few tips to give you:
  1. identify the procedures, and do not try to study too many procedures at once; ideally you should start with the minimum amount of procedures that are needed
  2. put comments near the branch points with the original addresses, they will be later on helpful for lookup when you are tracing or doing step-by-step debugging
  3. use a original dumps to initialize your buffers, like sections of WRAM and ROM
That's all folks, I hope this read was useful - send me your feedback!




* = It could be an interesting project, however the generated code would be little readable and you still would need to understand it

2013-05-18

odroid-wheezy-retro 0.6 released: you get kernel 3.0.75, 24bit HDMI output and wifi support!

As per title I have just released version 0.0.6 of the image that contains:
  • Hardkernel's linux kernel version 3.0.75, that comes with forced 24bit HDMI output (thanks to libv for the patch)
  • extra wifi drivers :)

If you have kernel 3.0.63 and want to upgrade just use the upgrade script from repository.

I am planning to release an even more streamlined release, so keep tuned! :)
Official ODROID forum discussion here.

2013-05-14

Recipe: compiling BSNES debugger (laevateinn) on Linux

BSNES is the best emulator around, there's little to say about that. Unfortunately the debugger is no more available in current higan because of refactoring and othe changes the author is making.

Let's get the last working bsnes debugger, laevateinn, compiling & working (spoiler: it.does.not.work) on Linux:

git clone git://gitorious.org/bsnes/bsnes.git
cd bsnes
git checkout v089
cd bsnes
nano nall/Makefile # replace "gcc-4.7" with "gcc"
make -j3 target=laevateinn

Finally the debugger will be found in out/laevateinn, but we are not yet there! It can only work with "cartridge folders".

So let's compile purify as well, and run it:

cd ..
cd purify
nano nall/Makefile # here replace "gcc-4.6" with "gcc"
make -j3
./purify output your-rom-path/ target-directory/


Now we are ready to run laevateinn:

cd ..
bsnes/out/laevateinn target-directory/

However, I could not use it because it shows me a non-sexy black screen :(
Possibly you might pick up the task from there.

Meanwhile I am using T.Geiger's snes9x debugger and I have contacted the author about sharing the sources (fingers crossed!)

Update 23 July 2013: instructions for Ubuntu 13.04 provided by user Jay Oster
NOTE: bsnes looks for its system ROMs in ~/.config/laevateinn/

Dependencies

$ sudo apt-get update
$ sudo apt-get install git build-essential libgtk2.0-dev libasound2-dev
# FOR PURIFY
$ sudo apt-get install libx11-dev

Build Laevateinn

$ git clone git://gitorious.org/bsnes/bsnes.git
$ cd bsnes
$ git checkout d418eda9
$ vim bsnes/nall/Makefile # replace "gcc-4.7" with "gcc"
$ make -C bsnes -j 8 target=laevateinn install

Build Purify

$ vim purify/nall/Makefile # replace "gcc-4.6" with "gcc"
$ vim purify/Makefile # add "link += `pkg-config --libs x11`"
$ make -C purify -j 8

PUTRIFY your ROM library

$ ./purify/purify output your-rom-path/ target-directory/

2013-05-11

odroid-wheezy-retro: updated wiki & release

I have started to populate and mantain some wiki pages for odroid-wheezy-retro


ODROID Wheezy Retro Wiki

It's hosted within the Google Code project's wiki, and it fits the purpose quite well so far.

Current release is 0.0.5, I have fixed the nasty issue with Debian automatic network udev rules, thanks to user be.lietaus for reporting it.

Many thanks to mdrjr for providing a backup location for the release files, that you can find here.

Are you an ODROID forum member? Please vote for me on this thread! Well, only if you like this release and want to see more about it coming  :)


2013-05-08

Neagix' excellent (and unofficial) Debian Wheezy Development&Emulation SD image


I have packaged a minimal Debian Wheezy downloadable SD image which achieves the same goals of my previous tutorial (and adds a bit more).
What You See Is What You Will Get 1/3

What You See Is What You Will Get 2/3
What You See Is What You Will Get 3/3

Features of this release

  • very small (187M packed), based on odroid.us base rootfs image
  • using stable kernel 3.0.68 released by Hardkernel
  • X11 hardware accelerated EGL, GLESv1 and GLESv2 (through Hardkernel's proprietary drivers)
  • boots automatically into Xorg 
  • it will not automatically turn off HDMI output when idle (so be careful with your LCDs!)
  • basic development packages already installed (gcc, g++, make, autoconf, automake etc)
  • scripts to fetch and build my fork of retroarch (+ related libretro cores)
  • retroarch (with snes9x-next core) and retroarch-joyconfig ready-to-run binaries included (these versions have my patches)
  • testing scripts for sound and 3D

Coming soon features

  • 3.0.75 kernel (I am testing it before adding to next release, but if you prefer you can upgrade using this script)
  • 24bit HDMI output support (I am not sure about this)
The first time you boot the image it will automatically re-generate SSH keys and assign a random MAC address (see /root/bin/bootstrap.sh for reference), so it might take a few seconds longer.

See project home page and instructions to flash the SD image.



Many thanks to mdrjr and all the other nice IRC people for the help provided since the beginning!

DISCLAIMER: this SD image is not officially supported by Hardkernel, nor I assume any responsability for its use and misuse

Previous posts on this subject:
http://neagix.blogspot.com/2013/04/retroarch-on-odroid-u2-with-hardware.html
http://neagix.blogspot.com/2013/05/retroarch-on-odroid-u2-revisited.html

2013-05-05

RetroArch on ODROID-U2, revisited

After my previous blog post I am considering creating a stand-alone, auto-updating modified Linaro image for RetroArch on ODROID-U2, but don't hold your breath for this now :)


Tip: listen to this great song while reading and hacking your ODROID+RetroArch setup

Update 7 May 2013: I released neagix' excellent (and unofficial) Debian Wheezy for Development&Emulation (with X11 hardware acceleration), get it while it's hot!

Some general advices for people approaching console emulation on ODROID-U2:
  • there is currently no known way to natively exploit the ODROID framebuffer (Exynos 4), thus we have to go through Xorg to run any emulator
  • composing desktop environments are bad for you, so no Unity/Compiz/you-name-it, use instead LXDE or XFCE
  • I have experienced kernel panic/oops/alignment trap issues when using my own compiled version of the kernel, thus I advice to stick to kernel 3.0.63 for now (this has proven to be stable so far) issue was due to PSU
  • remove xscreensaver and other screensaver packages, then modify xorg.conf to not automatically turn off DPMS and other screen blank settings
  • RetroArch runs fine with GL video driver and SDL input
  • be kind with your MCC/SD support, it's not like a regular hard disk and wearout factor is present. Nothing to be scared of, but consider moving write-often directories to external mountpoints
  • create a backup script and run it regularly. You don't know when, but your SD/MMC will one day wake up and tell you "So long, thanks for all the fish..."
  • use nobootwait in your fstab mountpoints, and if you are using NFS mountpoints specify an option for short timeout, like timeo=5
  • disable the long timeouts in /etc/init/failsafe.conf, and feel free to comment lines 34-35
  • if you are using lxdm as login manager, consider enabling autologin by uncommenting autologin=username in /etc/lxdm/lxdm.conf (username shall be your actual username)
  • if you are using LXDE, edit/create ~/.config/lxsession/LXDE/autostart and add a single line @retroarch (retroarch binary must be in your $PATH for this to work), this way once you boot ODROID you will be presented with retroarch
Suggested settings block for  ~/.retroarch.cfg:

video_driver = "gl"
video_fullscreen = "true"
video_windowed_fullscreen = "true"
video_aspect_ratio = "1.33333333333333"
audio_enable = "true"
audio_driver = "alsa"
audio_sync = "true"
input_driver = "sdl"
input_joypad_driver = "sdl"


Please note that only the first definition in the configuration file will be read, so I advise to remove duplicate definitions to reduce confusion.

Some patches for RetroArch

After my unsuccessful attempts at contributing RetroArch I've decided to publish these patches in my retroarch fork, hopefully they will be useful by people affected by my same problems.
Issue: when using RGUI and loading a saved state the button release is passed to the underlying core, causing all weird undesired actions

Fix: wait for all input to be released before starting the core, commit patch here

Issue: rather difficult to troubleshoot input issues might be related to wrong joystick configuration, that is also difficult to detect without a jstest-like program

Fix: enable logging of input events directly in retroarch, commit patch here
Feature: halt the system from RetroArch menu, commit patch here

Leaderitis acutis in open source

Leaderitis acutis

In the last days I wanted to contribute to an open source project with some feedback and bugfix/enhancement patches. Although I fully recognize my non-excellent C++ skills, I found the project to be affected by one of the most common diseases in the FOSS world: leaderitis acutis.

This is a purely fictional disease that I just invented and that I have been thinking about recently (I later found out it's used in american politics jargon, although I don't know how closely matches by what I describe here).

I have always been interested in open source philosophy (and also in free software to a lesser extent), thus I always find it appalling deludent when open source projects do not meet the essential expectations (that is very often for the smaller projects around these days).

Symptoms

  1. Project has very few developers and it has not been exposed to the beneficial strength of the "many eyes"
  2. The most prominent developers use the project to foster all their favourite code smells and anti-patterns (lack of comments, cryptic/mangled code, Baklava code, you-name-it)
  3. Issues, patches and any feedback is negatively considered as a critic or even as an offence (although this might have roots deep into cursed developers' personality)
    Sketch by 3-3 studios
  4. The attitude towards any form of feedback that is not a plain compliment is "Sheesh! This is my little precious, you won't touch it!", missing the point that open source is not your own thing, not even when you start it or are the main contributor
  5. As a last resort solution to any civil and logic argument, developers/mantainers will assert that you have no right to express your opinion, or that your opinion is not supposed to be worth anything in first place. To justify this assertion and enforce the paradigm that they should be trusted as superior, God-gifted owners of truth, they will either:
  • specify that your total contributions amount to 0% and their contributions to 99%, cheerfully living in a chicken-and-egg paradox (or simply barring entry to potential contributors to mantain their sphere of influence untouched)
  • possibly try with any form of personal attack like "you make mistakes, thus your opinions are not trustable".

Cure

There is no known cure for leaderitis acutis, although progress has been verified with personal growth of the involved developers and mantainers that generally happens when time tells them that open source is not about going solo/berserk.