{"html_url": "https://github.com/simonw/datasette/issues/673#issuecomment-586442292", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/673", "id": 586442292, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjQ0MjI5Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-14T19:36:37Z", "updated_at": "2020-02-14T19:36:37Z", "author_association": "OWNER", "body": "This can be a function in `utils/__init__.py`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565518772, "label": "Mechanism for checking if a SQLite database file is safe to open"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/673#issuecomment-586442978", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/673", "id": 586442978, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjQ0Mjk3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-14T19:38:19Z", "updated_at": "2020-02-14T19:38:19Z", "author_association": "OWNER", "body": "Amazingly, I get 0 search results on Google for `RidList_VirtualReaderModule`! I guess no-one has reverse engineered the Apple Photos SQLite database at that level yet.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565518772, "label": "Mechanism for checking if a SQLite database file is safe to open"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/673#issuecomment-586443837", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/673", "id": 586443837, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjQ0MzgzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-14T19:40:42Z", "updated_at": "2020-02-14T19:41:56Z", "author_association": "OWNER", "body": "Here's how to test if the `rtree` virtual table is supported:\r\n```\r\n>>> import sqlite3\r\n>>> c = sqlite3.connect(\":memory:\")\r\n>>> c.execute(\"create virtual table blah using rtree (a, b, c)\")\r\n\r\n>>> c.execute(\"create virtual table blah2 using rtree2 (a, b, c)\")\r\nTraceback (most recent call last):\r\n File \"\", line 1, in \r\nsqlite3.OperationalError: table blah already exists\r\n```\r\nAlso:\r\n```\r\n>>> c.execute('''CREATE VIRTUAL TABLE SpatialIndex USING VirtualSpatialIndex()''')\r\nTraceback (most recent call last):\r\n File \"\", line 1, in \r\nsqlite3.OperationalError: no such module: VirtualSpatialIndex\r\n>>> c.enable_load_extension(\r\n... True)\r\n>>> \r\n>>> c.load_extension(\"/usr/local/lib/mod_spatialite.dylib\")\r\n>>> c.execute('''CREATE VIRTUAL TABLE SpatialIndex USING VirtualSpatialIndex()''')\r\n\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565518772, "label": "Mechanism for checking if a SQLite database file is safe to open"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/673#issuecomment-586444835", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/673", "id": 586444835, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjQ0NDgzNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-14T19:43:27Z", "updated_at": "2020-02-14T19:43:27Z", "author_association": "OWNER", "body": "I can extend this function (maybe also rename it):\r\n\r\nhttps://github.com/simonw/datasette/blob/52ba34701cdbf510236de87d35b0e6df330626d1/datasette/utils/__init__.py#L595-L610", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565518772, "label": "Mechanism for checking if a SQLite database file is safe to open"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/673#issuecomment-586444970", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/673", "id": 586444970, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjQ0NDk3MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-14T19:43:46Z", "updated_at": "2020-02-14T19:43:46Z", "author_association": "OWNER", "body": "`is_openable_sqlite` perhaps?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565518772, "label": "Mechanism for checking if a SQLite database file is safe to open"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/673#issuecomment-586445210", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/673", "id": 586445210, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjQ0NTIxMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-14T19:44:27Z", "updated_at": "2020-02-14T19:44:27Z", "author_association": "OWNER", "body": "For the unit tests I think I'm going to have to create minimal binary SQLite file examples and include them in the repo.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565518772, "label": "Mechanism for checking if a SQLite database file is safe to open"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/673#issuecomment-586448292", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/673", "id": 586448292, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjQ0ODI5Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-14T19:53:05Z", "updated_at": "2020-02-14T19:53:05Z", "author_association": "OWNER", "body": "I may be re-inventing this code at the moment:\r\nhttps://github.com/simonw/datasette/blob/3ffb8f3b98252531d11897fd431711e9b8045ace/datasette/app.py#L219-L237", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565518772, "label": "Mechanism for checking if a SQLite database file is safe to open"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/673#issuecomment-586449286", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/673", "id": 586449286, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjQ0OTI4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-14T19:56:00Z", "updated_at": "2020-02-14T19:57:17Z", "author_association": "OWNER", "body": "I tried to make the smallest SpatiaLite database file I could (to use for the tests), but it ended up over 5MB!\r\n```\r\n$ echo '{\"type\":\"Feature\",\"properties\":{\"name\":\"Hearst Castle\"},\"geometry\":{\"type\":\"Point\",\"coordinates\":[-121.1686,35.685]}}' | geojson-to-sqlite /tmp/hearst.db places - --spatialite\r\n$ ls -lah /tmp/hearst.db \r\n-rw-r--r-- 1 simonw wheel 5.3M Feb 14 11:54 /tmp/hearst.db\r\n```\r\nI imagine that's because of these tables:\r\n\"tiny\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565518772, "label": "Mechanism for checking if a SQLite database file is safe to open"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/673#issuecomment-586450571", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/673", "id": 586450571, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjQ1MDU3MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-14T19:59:41Z", "updated_at": "2020-02-14T20:01:14Z", "author_association": "OWNER", "body": "This helped:\r\n```\r\n$ sqlite3 /tmp/hearst.db \r\nSQLite version 3.24.0 2018-06-04 14:10:15\r\nEnter \".help\" for usage hints.\r\nsqlite> delete from spatial_ref_sys where srid != 4326;\r\nsqlitte> delete from spatial_ref_sys_aux where srid != 4326;\r\nsqlite> vacuum;\r\nsqlite> ^D\r\n$ ls -lah /tmp/hearst.db \r\n-rw-r--r-- 1 simonw wheel 216K Feb 14 12:01 /tmp/hearst.db\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565518772, "label": "Mechanism for checking if a SQLite database file is safe to open"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/673#issuecomment-586454371", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/673", "id": 586454371, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjQ1NDM3MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-14T20:11:02Z", "updated_at": "2020-02-14T20:11:02Z", "author_association": "OWNER", "body": "The technique from `run_sanity_checks` of running `PRAGMA table_info({})` for every table seems to work just fine. It failed for the Apple Photos database for example:\r\n```\r\nsqlite> pragma table_info(RKSceneInVersion_VirtualBufferReader);\r\nError: no such module: VirtualBufferReaderModule\r\n```\r\nSo I think the solution to this ticket is going to be moving that logic into a new utility function.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565518772, "label": "Mechanism for checking if a SQLite database file is safe to open"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/673#issuecomment-586455321", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/673", "id": 586455321, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjQ1NTMyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-14T20:13:59Z", "updated_at": "2020-02-14T20:13:59Z", "author_association": "OWNER", "body": "Closing this in favour of rethinking how sanity checks work.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565518772, "label": "Mechanism for checking if a SQLite database file is safe to open"}, "performed_via_github_app": null}