User blog:Nefere/The list of VL Wikia songs

I spend few evenings with working on something new and useful for Vocaloid Lyrics Wikia community, and here it is.

...I'm not sure how to name it, but it's a table with (almost) all songs on the Wikia and each song comes with some info. I thought about a whole project as an extention of Untranslated Songs List, but it started to become bigger than I expected: it not only shows pages with no translation, but also Vocaloids and producers of songs.

It not only can be used to check which songs needs to be translated, but also


 * to find songs without proper categories (or with typos, like this one here),


 * producers with only few songs,


 * or songs of favourite Vocaloid groups (I haven't know there's ~150 Kagamine duet entries).

I gathered all needed info from category pages by API - they're are neatly organised here - and song entries are based on them. Contrary to my last project, the only thing to get results is to load the page :) It's done by sending several requests to Wikia to get the lists of songs by language, by Vocaloid etc. To get all info about every untranslated song, only one request is needed, about language - a dozen or so requests, and about Vocaloids - few tens of requests. The can be handled by most browsers while loading the page, but when I got to producers... 1,8k extra requests to get each producer song list freezed my Firefox for long few minutes. So I set the way to integrate all data into one file and to download the only file, without checking the Wikia. And I make smaller data size to download - from first 10~20 MB to 2.51 MB.

The project has some flaws I'm aware and I wold like to do something with them if I have time and know how to:
 * the data aren't fresh - they came from the txt file prepared by me which I want to update once in few days. Even if I'm making it, the data on Wikia are cached as well - Wikia refreshes every few hours. It means that new pages would got false info, for example lack of language or producer.
 * filtering and searching are disabled (only sorting works) - JS Grid I use to show the data in table simplifies the filtering to "ask the server to do the dirty work!" but I don't have a server :/ I would like to find another table framework or just write work-arounds to make it work in-browser.
 * cover songs don't have language info - it's kept in different categories but I'm too lazy and sleepy to fix that now.
 * the page is barely designed - the're only the table default skin.

If anyone have some suggestions or issues - just write about them in comments, I would be more than happy to read them.