{"id": 275125561, "node_id": "MDU6SXNzdWUyNzUxMjU1NjE=", "number": 123, "title": "Datasette serve should accept paths/URLs to CSVs and other file formats", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2017-11-19T02:05:48Z", "updated_at": "2021-07-19T00:04:32Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This would remove the csvs-to-sqlite step which I end up using for almost everything.\r\n\r\nI'm hesitant to introduce pandas as a required dependency though since it require compiling numpy. Could build it so this option is only available if you have pandas installed.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/123/reactions\", \"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 275125805, "node_id": "MDU6SXNzdWUyNzUxMjU4MDU=", "number": 124, "title": "Option to open readonly but not immutable", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2017-11-19T02:11:03Z", "updated_at": "2019-06-24T06:43:46Z", "closed_at": "2019-06-24T06:43:46Z", "author_association": "OWNER", "pull_request": null, "body": "Immutable assumes no other process can modify the file. An option to open reqdonly instead would enable other processes to update the file in place.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/124/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 275159710, "node_id": "MDU6SXNzdWUyNzUxNTk3MTA=", "number": 128, "title": "Every visualization should have an \"embed\" button", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2017-11-19T13:38:13Z", "updated_at": "2019-05-13T18:33:51Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "At least for the first round of visualizations, any time you construct one using the UI the result should include an \"embed this\" button that returns source code to copy and paste\r\n\r\nThese examples should use unpkg.com (or similarl) urls with SRI hashes, eg https://www.srihash.org - and should load data from the datasette JSON API.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/128/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 275135393, "node_id": "MDU6SXNzdWUyNzUxMzUzOTM=", "number": 125, "title": "Plot rows on a map with Leaflet and Leaflet.markercluster", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2017-11-19T06:05:05Z", "updated_at": "2018-04-26T15:14:31Z", "closed_at": "2018-04-26T15:14:31Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/Leaflet/Leaflet.markercluster would allow us to paginate-load in an enormous set of rows with latitude/longitude points, e.g. https://australian-dunnies.now.sh/\r\n\r\nHere's a demo of it loading 50,000 markers: https://leaflet.github.io/Leaflet.markercluster/example/marker-clustering-realworld.50000.html - and it looks like it's easy to support progress bars for if we were iteratively loading 1,000 markers at a time using datasette pagination.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/125/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 275166078, "node_id": "MDU6SXNzdWUyNzUxNjYwNzg=", "number": 130, "title": "Rename \"datasette build\" to \"datasette inspect\"", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2017-11-19T15:08:02Z", "updated_at": "2017-12-07T16:57:58Z", "closed_at": "2017-12-07T16:57:58Z", "author_association": "OWNER", "pull_request": null, "body": "This command introspects the databases and writes out a JSON summary.\r\n\r\nI think I'd like to use `datasette build` for something more interesting, potentially duplicating functionality from https://github.com/simonw/csvs-to-sqlite\r\n\r\nSince the internal method that does this is called `ds.inspect()` that seems like a reasonable replacement name for the command.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/130/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 275179724, "node_id": "MDU6SXNzdWUyNzUxNzk3MjQ=", "number": 135, "title": "?_search=x should work if used directly against a FTS virtual table", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2949431, "label": "Custom templates edition"}, "comments": 3, "created_at": "2017-11-19T18:17:53Z", "updated_at": "2017-12-07T04:54:41Z", "closed_at": "2017-12-07T04:54:41Z", "author_association": "OWNER", "pull_request": null, "body": "e.g. https://sf-trees.now.sh/sf-trees-ebc2ad9/Street_Tree_List_fts?_search=grove should work", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/135/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 275135535, "node_id": "MDU6SXNzdWUyNzUxMzU1MzU=", "number": 126, "title": "Blog entry announcing foreign key support", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2919870, "label": "Foreign key edition"}, "comments": 1, "created_at": "2017-11-19T06:09:06Z", "updated_at": "2017-11-30T16:49:24Z", "closed_at": "2017-11-30T16:49:24Z", "author_association": "OWNER", "pull_request": null, "body": "", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/126/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 275176006, "node_id": "MDU6SXNzdWUyNzUxNzYwMDY=", "number": 133, "title": "If view is filtered, search should apply within those filtered rows", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2919870, "label": "Foreign key edition"}, "comments": 3, "created_at": "2017-11-19T17:25:36Z", "updated_at": "2017-11-24T22:30:32Z", "closed_at": "2017-11-24T22:30:15Z", "author_association": "OWNER", "pull_request": null, "body": "Eg on https://sf-trees.now.sh/sf-trees-ebc2ad9/Street_Tree_List?qSpecies=1", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/133/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 275135719, "node_id": "MDU6SXNzdWUyNzUxMzU3MTk=", "number": 127, "title": "Filtered tables should show count of all matching rows, if fast enough", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2919870, "label": "Foreign key edition"}, "comments": 2, "created_at": "2017-11-19T06:13:29Z", "updated_at": "2017-11-24T22:02:01Z", "closed_at": "2017-11-24T22:02:01Z", "author_association": "OWNER", "pull_request": null, "body": "Relates to #86. If you are viewing a filtered page e.g. https://fivethirtyeight.datasettes.com/fivethirtyeight-2628db9/bob-ross%2Felements-by-episode?CLOUDS=1 we should show the count of matching rows.\r\n\r\nSince this could be an expensive operation, we will run it with a strict time limit (maybe 50ms). If the time limit is exceeded we will display \"many\" instead, perhaps? Maybe even link to a count(*) query that would get the full 1000ms time limit which the user can click on if they like (that could even Ajax-in the result).", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/127/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 275175929, "node_id": "MDU6SXNzdWUyNzUxNzU5Mjk=", "number": 132, "title": "Row view is not currently expanding foreign keys", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2919870, "label": "Foreign key edition"}, "comments": 1, "created_at": "2017-11-19T17:24:25Z", "updated_at": "2017-11-23T21:51:51Z", "closed_at": "2017-11-23T21:51:30Z", "author_association": "OWNER", "pull_request": null, "body": "Eg https://sf-trees.now.sh/sf-trees-ebc2ad9/Street_Tree_List/1", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/132/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 275164558, "node_id": "MDU6SXNzdWUyNzUxNjQ1NTg=", "number": 129, "title": "Hide FTS-created tables by default on the database index page", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2017-11-19T14:50:42Z", "updated_at": "2017-11-22T20:22:02Z", "closed_at": "2017-11-22T20:19:04Z", "author_association": "OWNER", "pull_request": null, "body": "SQLite databases that use FTS include a number of automatically generated tables, e.g.:\r\n\r\nhttps://sf-trees-search.now.sh/sf-trees-search-a899b92\r\n\r\n<img width=\"730\" alt=\"sf-trees-search_and_sf-trees-search\" src=\"https://user-images.githubusercontent.com/9599/32991960-bf05abee-ccf9-11e7-8bd1-54dcde4ca491.png\">\r\n\r\nOf these, only the `Street_Tree_List` table is actually relevant to the user.\r\n\r\nWe can detect which tables are FTS tables by first finding the virtual tables:\r\n\r\n    sqlite> .headers on\r\n    sqlite> select * from sqlite_master where rootpage = 0;\r\n    type|name|tbl_name|rootpage|sql\r\n    table|Search|Search|0|CREATE VIRTUAL TABLE \"Street_Tree_List_fts\" USING FTS4 (\"qAddress\", \"qCaretaker\", \"qSpecies\")\r\n\r\nThen parsing the above to figure out which ones are USING FTS? - then assume that any table which starts with that `Street_Tree_List_fts` prefix was created to support search:\r\n\r\n    sqlite> select * from sqlite_master where type='table' and tbl_name like 'Street_Tree_List_fts%';\r\n    type|name|tbl_name|rootpage|sql\r\n    table|Search_content|Search_content|10355|CREATE TABLE 'Street_Tree_List_fts_content'(docid INTEGER PRIMARY KEY, 'c0qAddress', 'c1qCaretaker', 'c2qSpecies')\r\n    table|Search_segments|Search_segments|10356|CREATE TABLE 'Street_Tree_List_fts_segments'(blockid INTEGER PRIMARY KEY, block BLOB)\r\n    table|Search_segdir|Search_segdir|10357|CREATE TABLE 'Street_Tree_List_fts_segdir'(level INTEGER,idx INTEGER,start_block INTEGER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx))\r\n    table|Search_docsize|Search_docsize|10359|CREATE TABLE 'Street_Tree_List_fts_docsize'(docid INTEGER PRIMARY KEY, size BLOB)\r\n    table|Search_stat|Search_stat|10360|CREATE TABLE 'Street_Tree_List_fts_stat'(id INTEGER PRIMARY KEY, value BLOB)\r\n\r\nWe won't hide these completely - instead, we'll default the database index view to not showing them with a message that says \"5 hidden tables\" and support ?_hidden=1 to display them.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/129/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 275176094, "node_id": "MDU6SXNzdWUyNzUxNzYwOTQ=", "number": 134, "title": "Filtered table view should show a count", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2919870, "label": "Foreign key edition"}, "comments": 1, "created_at": "2017-11-19T17:26:53Z", "updated_at": "2017-11-19T18:10:49Z", "closed_at": "2017-11-19T18:10:49Z", "author_association": "OWNER", "pull_request": null, "body": "Let's do the thing where we attempt to show an accurate count if it can be done in less than 50ms", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/134/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 275166669, "node_id": "MDU6SXNzdWUyNzUxNjY2Njk=", "number": 131, "title": "UI support for running FTS searches", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2017-11-19T15:16:20Z", "updated_at": "2017-11-19T17:18:05Z", "closed_at": "2017-11-19T17:00:12Z", "author_association": "OWNER", "pull_request": null, "body": "Here's an example query that searches all FTS indexed columns in a table: https://sf-trees-search.now.sh/sf-trees-search-a899b92?sql=select+*+from+Street_Tree_List+where+rowid+in+%28select+rowid+from+Street_Tree_List_fts+where+Street_Tree_List_fts+match+%27grove+london+dpw%27%29%0D%0A\r\n\r\nAnd here's a query that searches a specific column: https://sf-trees-search.now.sh/sf-trees-search-a899b92?sql=select+*+from+Street_Tree_List+where+rowid+in+%28select+rowid+from+Street_Tree_List_fts+where+qSpecies+match+%27london%27%29%0D%0A\r\n\r\nIf we detect that a table has FTS enabled (which we can do by looking for it as a content table reference in another FTS table's create definition) we should add a search box to the table page which constructs this query - maybe using `?_search=XXX` in the query string?\r\n\r\n<s>To support search against specified columns, we can do `?_search__ qSpecies=London`.</s> - not necessary, see comment below.\r\n\r\n- [x] Detect if a table has a FTS index defined against it as a content= parameter\r\n- [x] Decide what to do if there is more than one FTS index (maybe just pick the first one?)\r\n- [x] Add the `?_search=` query string argument\r\n- [x] Add the UI", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/131/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}