{"html_url": "https://github.com/simonw/sqlite-utils/issues/86#issuecomment-591770623", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/86", "id": 591770623, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MTc3MDYyMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-27T04:12:39Z", "updated_at": "2020-02-27T04:12:39Z", "author_association": "OWNER", "body": "I pushed a branch with my experiment in it, but I'm going to fix this by throwing an error on `[` or `]` in a column name instead - I won't implement the changes from that branch.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 564579430, "label": "Problem with square bracket in CSV column name"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586729798", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/86", "id": 586729798, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjcyOTc5OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-16T17:11:02Z", "updated_at": "2020-02-16T17:11:02Z", "author_association": "OWNER", "body": "I filed a bug in the Python issue tracker here: https://bugs.python.org/issue39652", "reactions": "{\"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 564579430, "label": "Problem with square bracket in CSV column name"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586683572", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/86", "id": 586683572, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjY4MzU3Mg==", "user": {"value": 8149512, "label": "foscoj"}, "created_at": "2020-02-16T09:03:54Z", "updated_at": "2020-02-16T09:03:54Z", "author_association": "NONE", "body": "Probably the best option to just throw the error.\r\nIs there any active dev chan where we could post the issue to python sqlite3?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 564579430, "label": "Problem with square bracket in CSV column name"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586676856", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/86", "id": 586676856, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjY3Njg1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-16T07:20:34Z", "updated_at": "2020-02-16T07:20:34Z", "author_association": "OWNER", "body": "I'm not sure what to do about this one.\r\n\r\nI can't fix it: this bug in Python's `sqlite3` module means that even if I write a database out with column names that include `[]` I won't be able to read them back again.\r\n\r\nSo... I could do one of the following:\r\n\r\n- Throw an error if a column name includes those characters. That's my preferred option I think.\r\n- Automatically replace `[` in column names with `(` and `]` with `)`\r\n- Do the automatic replacement but show a user-visible warning when I do it\r\n- Throw an error, but give the user an option to run with e.g. `--fix-column-names` which applies that automatic fix.\r\n\r\nSince this is likely to be an incredibly rare edge-case I think I'd rather minimize the amount of code that deals with it, so my preferred option is to just throw that error and stop.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 564579430, "label": "Problem with square bracket in CSV column name"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586676640", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/86", "id": 586676640, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjY3NjY0MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-16T07:16:31Z", "updated_at": "2020-02-16T07:16:31Z", "author_association": "OWNER", "body": "There's something weird about this. I created a test database file like so:\r\n```\r\nsqlite3 /tmp/demo.db <<EOF\r\n    BEGIN TRANSACTION;\r\n    CREATE TABLE \"data\" (\r\n        \"MTU (CET)\" TEXT,\r\n        \"Day-ahead Price [EUR/MWh]\" TEXT\r\n    );\r\n    INSERT INTO \"data\" VALUES('01.01.2016 00:00 - 01.01.2016 01:00','23.86');\r\n    COMMIT;\r\nEOF\r\n```\r\nThen confirmed that it works as expected in SQLite:\r\n```\r\n $ sqlite3 /tmp/demo.db \r\nSQLite version 3.24.0 2018-06-04 14:10:15\r\nEnter \".help\" for usage hints.\r\nsqlite> .schema\r\nCREATE TABLE IF NOT EXISTS \"data\" (\r\n        \"MTU (CET)\" TEXT,\r\n        \"Day-ahead Price [EUR/MWh]\" TEXT\r\n    );\r\nsqlite> .headers on\r\nsqlite> select * from data;\r\nMTU (CET)|Day-ahead Price [EUR/MWh]\r\n01.01.2016 00:00 - 01.01.2016 01:00|23.86\r\nsqlite> \r\n```\r\nBUT... if I open the same database in Python, something weird happens:\r\n```\r\nIn [1]: import sqlite3                                                                                                         \r\nIn [2]: conn = sqlite3.connect(\"/tmp/demo.db\")                                                                                 \r\nIn [3]: cursor = conn.cursor()                                                                                                 \r\nIn [4]: cursor.execute(\"select * from data\")                                                                                   \r\nOut[4]: <sqlite3.Cursor at 0x10c70a0a0>\r\nIn [5]: cursor.fetchall()                                                                                                      \r\nOut[5]: [('01.01.2016 00:00 - 01.01.2016 01:00', '23.86')]\r\nIn [6]: cursor.description                                                                                                     \r\nOut[6]: \r\n(('MTU (CET)', None, None, None, None, None, None),\r\n ('Day-ahead Price', None, None, None, None, None, None))\r\nIn [7]: conn.row_factory = sqlite3.Row                                                                                         \r\nIn [8]: cursor = conn.cursor()                                                                                                 \r\nIn [9]: cursor.execute(\"select * from data\")                                                                                   \r\nOut[9]: <sqlite3.Cursor at 0x10c7a8490>\r\nIn [10]: row = cursor.fetchall()                                                                                                     \r\nIn [12]: row                                                                                                                   \r\nOut[12]: <sqlite3.Row at 0x10c3fe670>\r\nIn [15]: row.keys()                                                                                                            \r\nOut[15]: ['MTU (CET)', 'Day-ahead Price']\r\n```\r\nNote that in `cursor.description` AND in `row.keys()` above the second column is displayed as `'Day-ahead Price'` - when we would expect it to be displayed as `Day-ahead Price [EUR/MWh]`\r\n\r\nSo.... it looks like there may be a bug in Python's `sqlite3` module where columns with square braces in them have that portion of the name stripped out!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 564579430, "label": "Problem with square bracket in CSV column name"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586662404", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/86", "id": 586662404, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjY2MjQwNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-16T02:43:12Z", "updated_at": "2020-02-16T02:43:12Z", "author_association": "OWNER", "body": "https://stackoverflow.com/a/22694438 looks like the answer:\r\n> When using square brackets, it is not possible to have these characters in the identifier.\r\n> \r\n> When using double quotes, you can escape them in the name by doubling them:\r\n> \r\n> `CREATE TABLE \"hello \"\"world\"\"\"(key INTEGER PRIMARY KEY);`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 564579430, "label": "Problem with square bracket in CSV column name"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586661934", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/86", "id": 586661934, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjY2MTkzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-16T02:33:07Z", "updated_at": "2020-02-16T02:33:07Z", "author_association": "OWNER", "body": "Thanks for the example file - looks like it can be trimmed down to just these two lines to replicate the bug:\r\n```csv\r\n\"MTU (CET)\",\"Day-ahead Price [EUR/MWh]\"\r\n\"01.01.2016 00:00 - 01.01.2016 01:00\",\"23.86\"\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 564579430, "label": "Problem with square bracket in CSV column name"}, "performed_via_github_app": null}