Citadel on the Move הוא פרוייקט מחקר בעלות 4 מיליון יורו של האיחוד האירופי שמתחקה אחרי פרקטיקות שימוש וניצול מידע פתוח ב-140 ערים מסביב לעולם. הם זיהו 77 פורמטים שונים כשהנפוץ מביניהם הוא CSV ואחריו XML ו-XLS
מעניין שבימיו הראשונים של מידע פתוח ה-Citadel עצמם המליצו על שימוש בטיפוס הנתונים המורכב LOD. אח"כ הם עברו ל-XML וכיום CSV הוא ההמלצה החמה.
מעניין גם שישראל לא משתתפת במחקר. מישהו רוצה לרתום את העירייה שלו?..
I’ve mentioned it before here in a different context, but data as CSV is so much more important than APIs, and it is great to see that this is the result of their research.
Many organizations are working on tooling around CSV for open data, including OK and most of the work I do there.
From what I’ve seen in the UK, CSV as a target format also ties in extremely well with developing a culture around open data - the people who are producing and processing reports are using CSV and tooling right at the point of data production, and therefore they are aware of transparency, accessibility and other reasons for publishing data in certain formats.
API-first, and other non-human-readable approaches do not gain this benefit - they position the issue as a technical one rather than a socio-cultural one.
אם היו עושים את הטבלה הזאת על מדינת ישראל, היו רק 4 פורמטים שהעייקרי בהם הוא PDF.
CSV אפילו לא בפוקוס…
הכל טוב ויפה, ואכן CSV הוא שיפור לעומת PDFים (או SOAP), אבל בינינו זה פורמט ממש גרוע.
אם pdf הוא פורמט נעים לאנשים ונורא למכונות, אז csv הוא פורמט מעצבן ברובו לאנשים ומכונות כאחד.
פתרונות אפשריים אפשר למצוא בין השאר בעבודה ש- @pwalsh עושה ב-OKF עם json table scheme.
אדם
Hey @aaa,
I definitely don’t agree on CSV, but I admit I’m biased :).
Let’s not forget, though, why CSV is so good as a baseline format for open data:
@pwalsh, as I said, CSV is not the worst format you could use.
However, it’s as bad format nonetheless.
The de-facto baseline for “what’s good for humans” is Microsoft Excel - you might not like it but it’s a fact.
@aaa,
With the example file, Good Tables finds it structurally valid, and looking at it manually that seems about right. I guess when they wrote CSVLint, they didn’t consider unicode support in their code, but in GoodTables, I did :). So - this is an issue with the implementation, not the file format itself.
About Excel - to be totally clear, I have no problem with Excel whatsoever. I do not believe that it is right or reasonable to expect data to be produced outside of Excel in the vast majority of cases for many, many years to come. GoodTables actually supports all Excel formats in addition to CSV, but, as you may know, Excel is not exactly a walk in the park due to the date format and other quirks.
The point is though, that Excel, the program, reads and writes CSV - this is how spend data of UK ministries is produced, in Excel, and published as both Excel and CSV.
About formats, firmer foundations and so on. It is kind of like the arguments around Javascript in comparison to other programming languages: Let’s not let best be the enemy of good.
I think CSV to open data will basically for the foreseeable future be like Javascript for web programming: sure, it is not perfect, it is even weird sometimes, missing (supposedly) fundamental data types and so on - yet, it basically has the minimum you need to get stuff done, and a growing ecosystem to fill the gaps.
The problem with the sample file is that someone decided to put a single " character inside a text field, which is illegal (although it may seem mostly valid, it’s not)
As for Excel - the requirement should be to be able to open the published files.
Right now this is is only possible in English speaking countries, as excel doesn’t support UTF8 in CSV.
And as for best being the enemy of good - the same argument could have been used in the discussion of PDF vs CSV, or paper vs PDF etc. We’re not talking about developing something new or cutting-edge - there are better formats which are being widely used and industry accepted. Using CSV might have been nice when there were no such alternatives, now it’s just backwards.
I think that instead of going the extra work and linting their CSVs, they can simply use any CSV2JSON 
converter (such as these) and save us all the trouble.
But, CSV to JSON doesn’t solve any problem whatsoever.
It will just give you json objects of strings, which is not useful, or, try to guess types, which is barely useful.
There are no consumer grade desktop apps that would let a user work with JSON, and streaming JSON is more difficult - there is newline-delimited JSON but try getting humans to edit one of those.
Can you send me an example of an Excel file that you think cannot be opened? UTF-8 is one thing, but we can obviously pass specific encodings to a parser, or, use character detection libraries to guess.
For best vs good - there are all sorts of formats - but a hell of a lot of open data is converging around csv now - for better or worse.
Oh, and about the quote symbol - it is legal in CSV. This file does not use quoting for enclosing fields, so that means that it is fine to use quote symbols in the field data.
This is the best reference for CSV formats, for what it is worth: http://tools.ietf.org/html/rfc4180
I feel we’re spamming this thread, but let me just say that JSON has full unicode support and a single standard way to encode and validate - which is already a huge improvement over CSV. It also has streaming support, online editors and whatnot. It’s probably better supported by almost all programming languages.
Anyway, try to open this file in excel and lmk how it goes:
http://www.obudget.org/api/budget/00?o=csv
We can keep discussing this in private…
Hi,
Ok, sorry you think this is spamming. It is probably too technical now, but it does provide some context into why CSV is becoming the defacto for distribution of open data.
Especially because this is not the case in Israel, not in government and not in open data orgs like HaSadna, I thought it was relevant.
So, we can talk in private, but the last points I want to make here are:
you’re not spamming
reading carefully and enjoying it 
plus that’s what we’re here for - technical expertise