Pour mieux comprendre l’Archos Fusion Storage
Voici de la main de Marc de Courville – le grand Manitou de la r&D d’Archos – l’explication de cette Archos Fusion Storage (AFS) annoncé ce matin !
La R&D d’archos frappe toujours lorsque l’on ne l’attends pas, Mapping pour la GamePad, Archos Video Player (un must), Gestion du Bluetooth pour l’Archos Smart Home….
Cette nouvelle création est une grande avancée dans la gestion de la mémoire interne. De plus, si Archos se permet de mettre cette évolution en place, c’est avec l’accord de Google qui est frileux sur les modifications de son O.S.. Archos ne peux pas se permettre de modifier Android sans cette accors sinon les produits ne seraient plus l’imprimatur d’Android et donc plus accés en natif aux Googles Applciations .
« Marketing short description »
ARCHOS Fusion Storage
The exclusive ARCHOS Fusion Storage system seamlessly combines the memory of any MicroSD card* with the internal storage on ARCHOS products.
Giving you even more space for your apps, games and multimedia, anytime your device’s internal memory is running low, this smart storage system automatically migrates data, optimizes memory balance and splits and combines extra-large files so you never have to worry about storage space or transferring files again.
« Technical executive summary »
Archos Fusion Storage consists in a user friendly expansion of Android phone/tablet storage space proposed at external SDcard insertion featuring:
– transparent fusion of internal storage and external sdcard: you can do it anytime and seamlessly (as opposed to hard internal storage/external SDcard switch)
– optimization of data/media and application storage: a migration process of internal data to external storage is proposed
– no file size limit even with FAT32 sdcard without reformatting: transparent to Android file splitting
– smart storage management, user do not need to keep track of the location of a file: internal/external storage load balancing
Technology under the hood: it relies on fuse kernel module that enables to unify storage at operating system level in order to be transparent to Android framework. Automatic large file split and storage balancing are Archos proprietary extensions to unionfs fuse module.
Archos is announcing a new way to handle uSD cards in its Android devices.
From the user’s point of view, it’s an option to unify external and internal storage.
Here are some explanations of how Android handles storage, and what Archos did to get around Android’s limits.
To fully grasp the concept you need to know the 4 essential folders where applications and data get stored in Android:
– /data/app: this is where the application apk goes. Once the apk is installed, this part never changes;
– /data/data: this is where the application stores settings and cache, some optimizations (odexes) done by Android at the installation.
Some games (or applications) even store uncompressed data here.
Often /data/data for an application is larger than the /data/app space itself (e.g. mail apps)
Implicitly /data/data refers to the total size excluding the apk of the app (i.e. the sum of /data/data, /data/dalvik-cache, /data/app-lib/ and many other folders size)
– /sdcard which is a « shared storage among many applications » zone;
– /extsdcard – the physical external sdcard file system (note that this name may change depending on the manufacturer’s implementation).
Historically, only /data (so /data/app/ and /data/data/) existed. Apps were isolated and no file sharing between them was (easily) possible.
Then along came Camera applications that needed to share pictures across many applications. The solution was a shared storage space expandable in size through the use of a uSD: the traditional /sdcard zone!
This uSD used a FAT file system, which inherently could not handle file permissions and thus allowed cross application file sharing.
This shared folder was thus called /sdcard (or ExternalStorage from Android API ) and this name is still there nowadays on Android to preserve legacy compatibility.
Surprisingly, some recent devices such as the Android One still require a physical uSD card to take camera pictures (like the old G1).
At one point internal storage space was not big enough to install loads of heavy applications and Google introduced the App2SD feature in Android Froyo allowing users move /data/app to /sdcard.
Note that /data/data was not moved but in most cases contains the larger part of an applications storage use…
Thus some heavy games started to be split in two parts: the APK itself, and the game data (e.g. textures, maps, etc).
Usually the APK downloads the data by itself, and put the result in the sdcard so that user can install more games.
Then, some manufacturers wanted to be able to take photos/save videos without adding a uSD, so they added a partition to the internal storage (eMMC or NAND flash) which behaved like a sdcard.
Since all applications were expecting the sdcard to be in /sdcard, this new internal partition has been mounted on /sdcard as well.
From this moment /sdcard stopped referring to an existing external, physical uSD creating a little (read: a lot) of confusion for end users.
App2SD in this case was only used to move an apk between two partitions of the same physical internal storage. Which is not really that much help.
The main issue with this approach is that the partition is fixed in size during the production process providing no flexibility in the balance between storage allocated to applications or multimedia files.
To get around this hard partion split, when Android 3.0 « Honeycomb » came around Google created a virtual filesystem called « sdcard » which fakes the same permission behavior as a standard FAT uSD, and in reality stores the multimedia files in /data/media (so in the /data partition). Though software shielded, the two legacy storage spaces are residing on the same physical partition.
At this point, a fuse daemon called sdcard is present with the unique aim of emulating the legacy /sdcard and the external ExternalStorage variable refers to an internal storage… Which isn’t confusing at all.
After that change, in Jellybean, Google started to implement the multi-user system on Android. One thing that appeared, is that games on multiple accounts would be downloaded multiple times.
In order to overcome this problem, games were recommended to move their data to /sdcard/Android/obb, because this folder could be shared among all the device users.
Before Kitkat, there was simply no official way for an application to know if a second shared storage was available (either an actual uSD, USB storage, etc.).
Some applications (mostly File Managers, or downloaders) manage to handle secondary storage using non-standard ways.
Very few applications have started to use the new Android 4.4 API to handle secondary storage (only Google Music is a big one that actually does it).
So, basically, most applications are still storing their data in /sdcard blindly.
After this short introduction into the wonderful (and not at all confusing) world of storage on Android here comes the changes Archos that made!
The user now has the possibility to have /sdcard being a virtual filesystem, which consists in a fusion of /data/media (as previous sdcard emulation) and the external sdcard.
This is made possible by a fuse FS. It is a fork of unionfs-fuse, a fuse meant to have a virtual filesystem which merges one writable file system, and one which is read-only.
The source filesystems are called branches. The first branch is the external sdcard, the second branch is /data/media.
Here are the features of this unionfs fork:
– No file permissions implemented, to behave as a FAT. The fuse FS is still backed by the original android « sdcard » daemon to handle permissions properly.
– The files can be written on any branch. If a file already exists on one of the branch, it uses this branch.
– The branch used for new files is the external sdcard, unless it has less than 2GB available, in which case it goes to the storage.
– The files are automatically split at 2GB directly on write/read calls.
To be able to use this fuse FS, some other modifications were required:
– There is an init script, which on boots, determines if the current uSD is the same as the previous boot.
If it is different, it disables the fusion, and asks the user if he wants to activate it.
– There is another init script, which checks if the fusion is enabled, and if it is enabled the unionfs is activated with the sdcard daemon on top of it.
If it is disabled, the sdcard daemon is just started with standard /data/media
The intermediate mountpoints (/sd/external for the sdcard, and /sd/emulated for the unionfs) are located in /sd, which is accessible only by media_rw:media_rw, thus only system apps can access it.
This fusion only requires a reboot to be activated. At this point, old files still remain on the internal storage, and new files are created on the sdcard.
So there is another optional step available to optimize the storage space for applications, which consists in migrating the files from /data/media to the /sd/external.
There is no cache in the fuse FS thus we can move files on the fly without any specific safe guards.
By the way, Android API mentions that /sdcard should be considered as a non-safe storage, so all problems like files disappearing because the sdcard has been removed should already be handled by the applications.
Indeed, one can notice that most applications downloads again their data when detecting that they are missing providing inherently a cold sdcard removal resilience.
Pour faire simple et en attendant l’update.zip :
– D’unifier la SD Interne avec une SD externe (exemple 16 +128go = 134 go) et cette fusion est reconnue comme une SD INTERNE,
– De pourvoir recopier des fichiers de plus de 4Go (l’AFS le saucissonne en interne en plusieurs « petits fichiers »),
– De pouvoir mieux gérer la nouvelle SD INTERNE en évitant les soucis de création/suppression sur la SD Externe du fait des nouvelles versions d’Android.