{"id": 836273891, "node_id": "MDU6SXNzdWU4MzYyNzM4OTE=", "number": 1266, "title": "Documentation for Response.asgi_send(send) method", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-03-19T18:52:49Z", "updated_at": "2021-03-20T21:35:00Z", "closed_at": "2021-03-20T21:32:28Z", "author_association": "OWNER", "pull_request": null, "body": "I found myself wanting to use this method for https://github.com/simonw/datasette-auth-passwords/issues/15 - but it's not documented. It should be documented.\r\n\r\nhttps://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/utils/asgi.py#L320-L340", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1266/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 836123030, "node_id": "MDU6SXNzdWU4MzYxMjMwMzA=", "number": 1265, "title": "Support for HTTP Basic Authentication", "user": {"value": 468612, "label": "yunzheng"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-03-19T15:31:09Z", "updated_at": "2021-03-19T22:05:12Z", "closed_at": "2021-03-19T21:03:09Z", "author_association": "NONE", "pull_request": null, "body": "It would be nice if datasette could support [HTTP Basic Authentication](https://en.wikipedia.org/wiki/Basic_access_authentication).\r\n\r\nFor now I could ofcourse leverage Nginx for basic authentication, but it would be nice to have support for this in datasette by default or via a plugin like datasette-auth-github.\r\n\r\nMy main usecase is to put the whole datasette instance behind a username/password prompt via Basic Auth and not specific urls.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1265/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 836064851, "node_id": "MDExOlB1bGxSZXF1ZXN0NTk2NjI3Nzgw", "number": 18, "title": "Add datetime parsing", "user": {"value": 1234956, "label": "n8henrie"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-03-19T14:34:22Z", "updated_at": "2021-03-19T14:34:22Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/healthkit-to-sqlite/pulls/18", "body": "Parses the datetime columns so they are subsequently properly recognized as\ndatetime.\n\nFixes https://github.com/dogsheep/healthkit-to-sqlite/issues/17\n", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/18/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 836063389, "node_id": "MDU6SXNzdWU4MzYwNjMzODk=", "number": 17, "title": "Datetime columns are not properly formatted to be recognizes as datetime", "user": {"value": 1234956, "label": "n8henrie"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-03-19T14:33:04Z", "updated_at": "2021-03-19T14:33:04Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "\r\n\r\nCurrently, the datetimes are formatted in a way that is not recognized by datasette-vega for plotting with a `Date/time` type for the axis. \r\n\r\nFor example, if you have datasette running locally with `datasette-vega` installed and have a database that includes resting heart rate:\r\n\r\n```\r\nhttp://localhost:8001/healthkit/rRestingHeartRate#g.mark=line&g.x_column=startDate&g.x_type=temporal&g.y_column=value&g.y_type=quantitative\r\n```\r\n\r\nThe plot is blank unless you choose `Label` as the type for the date data.\r\n\r\nThe `startDate` (and `creationDate` and `endDate`) columns appear like: `2019-11-14 18:22:18 -0700`\r\n\r\nIf instead the format for this column is changed slightly: `2019-11-14T18:22:18-07:00` they are recognized as proper dates and the charting works as expected.\r\n\r\nI have a PR that addresses this issue, will submit shortly.", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/17/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 834602299, "node_id": "MDU6SXNzdWU4MzQ2MDIyOTk=", "number": 1262, "title": "Plugin hook that could support 'order by random()' for table view", "user": {"value": 19328961, "label": "henry501"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-03-18T10:02:01Z", "updated_at": "2021-03-18T17:55:01Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I am frequently using Datasette to quickly get a visual impression for a table without reviewing it in its entirety. Because I have some groups of similar records, the default sorting options mean that each page is very similar and not representative of the full dataset. The current interface allows sorting by columns, but random sorting is only available via custom SQL.\r\n\r\nMaybe this could be a button or link.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1262/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 830901133, "node_id": "MDExOlB1bGxSZXF1ZXN0NTkyMzY0MjU1", "number": 16, "title": "Add a fallback ID, print if no ID found", "user": {"value": 1234956, "label": "n8henrie"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-03-13T13:38:29Z", "updated_at": "2021-03-13T14:44:04Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/healthkit-to-sqlite/pulls/16", "body": "Fixes https://github.com/dogsheep/healthkit-to-sqlite/issues/14\n", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/16/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 830283447, "node_id": "MDU6SXNzdWU4MzAyODM0NDc=", "number": 34, "title": "bucket name", "user": {"value": 6213, "label": "dsisnero"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-03-12T16:40:57Z", "updated_at": "2021-03-12T16:40:57Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I followed the instructions to setup credentials but I am getting a invalid bucket name. Can you put a sample auth.json file in the base that shows the correct format for this? Thanks", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/34/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 787173276, "node_id": "MDU6SXNzdWU3ODcxNzMyNzY=", "number": 1193, "title": "Research plugin hook for alternative database backends", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-01-15T20:27:50Z", "updated_at": "2021-03-12T01:01:54Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I started exploring what Datasette would like running against PostgreSQL in #670 and @dazzag24 did some work on Parquet described in #657.\r\n\r\nI had initially thought this was WAY too much additional complexity, but I'm beginning to think that the `Database` class may be small enough that having it abstract away the details of running queries against alternative database backends could be feasible.\r\n\r\nA bigger issue is SQL generation, but I realized that most of Datasette's SQL generation code exists just in the `TableView` class that runs the table page. If this was abstracted into some kind of SQL builder that could be then customized per-database it might be reasonable to get it working.\r\n\r\nVery unlikely for this to make it into Datasette 1.0, but maybe this would be the defining feature of Datasette 2.0?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1193/reactions\", \"total_count\": 3, \"+1\": 3, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 824067604, "node_id": "MDU6SXNzdWU4MjQwNjc2MDQ=", "number": 1250, "title": "Research: Plugin hook for alternative database connections", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-03-08T00:28:15Z", "updated_at": "2021-03-12T01:01:25Z", "closed_at": "2021-03-12T01:01:17Z", "author_association": "OWNER", "pull_request": null, "body": "The `Database` class is a natural looking fit for a plugin hook to load custom database connections... potentially even databases other than SQLite. DuckDB (refs #968) could make for a great starting point, since it looks very compatible with the existing SQLite code.\r\n\r\nThe real win would be if this could lead to running Datasette against PostgreSQL. I made some initial explorations in that direction a while ago in #670.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1250/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 797649915, "node_id": "MDExOlB1bGxSZXF1ZXN0NTY0NjA4MjY0", "number": 1211, "title": "Use context manager instead of plain open", "user": {"value": 4488943, "label": "kbaikov"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-01-31T07:58:10Z", "updated_at": "2021-03-11T16:15:50Z", "closed_at": "2021-03-11T16:15:50Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/1211", "body": "Context manager with open closes the files after usage. Fixes: https://github.com/simonw/datasette/issues/1208\r\n\r\nWhen the object is already a pathlib.Path i used read_text\r\nwrite_text functions\r\n\r\nIn some cases pathlib.Path.open were used in context manager,\r\nit is basically the same as builtin open.\r\n\r\nTests are passing: 850 passed, 5 xfailed, 10 xpassed", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1211/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 794554881, "node_id": "MDU6SXNzdWU3OTQ1NTQ4ODE=", "number": 1208, "title": "A lot of open(file) functions are used without a context manager thus producing ResourceWarning: unclosed file <_io.TextIOWrapper", "user": {"value": 4488943, "label": "kbaikov"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-01-26T20:56:28Z", "updated_at": "2021-03-11T16:15:49Z", "closed_at": "2021-03-11T16:15:49Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Your code is full of open files that are never closed, especially when you deal with reading/writing json/yaml files.\r\n\r\nIf you run python with warnings enabled this problem becomes evident.\r\nThis probably contributes to some memory leaks in long running datasettes if the GC will not 'collect' those resources properly.\r\n\r\nThis is easily fixed by using a context manager instead of just using open:\r\n```python\r\nwith open('some_file', 'w') as opened_file:\r\n opened_file.write('string')\r\n```\r\n\r\nIn some newer parts of the code you use Path objects 'read_text' and 'write_text' functions which close the file properly and are prefered in some cases.\r\n\r\n\r\nIf you want I can create a PR for all places i found this pattern in.\r\n\r\n\r\nBellow is a fraction of places where i found a ResourceWarning:\r\n```python\r\n\r\nupdate-docs-help.py:\r\n 20 actual = actual.replace(\"Usage: cli \", \"Usage: datasette \")\r\n 21: open(docs_path / filename, \"w\").write(actual)\r\n 22 \r\n\r\ndatasette\\app.py:\r\n 210 ):\r\n 211: inspect_data = json.load((config_dir / \"inspect-data.json\").open())\r\n 212 if immutables is None:\r\n\r\n 266 if config_dir and (config_dir / \"settings.json\").exists() and not config:\r\n 267: config = json.load((config_dir / \"settings.json\").open())\r\n 268 self._settings = dict(DEFAULT_SETTINGS, **(config or {}))\r\n\r\n 445 self._app_css_hash = hashlib.sha1(\r\n 446: open(os.path.join(str(app_root), \"datasette/static/app.css\"))\r\n 447 .read()\r\n\r\ndatasette\\cli.py:\r\n 130 else:\r\n 131: out = open(inspect_file, \"w\")\r\n 132 loop = asyncio.get_event_loop()\r\n\r\n 459 if inspect_file:\r\n 460: inspect_data = json.load(open(inspect_file))\r\n 461 \r\n\r\n```\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1208/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 826613352, "node_id": "MDExOlB1bGxSZXF1ZXN0NTg4NjAxNjI3", "number": 1254, "title": "Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions", "user": {"value": 3200608, "label": "durkie"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2021-03-09T20:49:08Z", "updated_at": "2021-03-10T18:27:45Z", "closed_at": "2021-03-09T22:04:23Z", "author_association": "NONE", "pull_request": "simonw/datasette/pulls/1254", "body": "This requires adding the RT Topology library (Spatialite changed to RT Topology from LWGEOM between 4.4 and 5.0), as well as upgrading the GEOS version (which is the reason for switching to `python:3.7.10-slim-buster` as the base image.)\r\n\r\n`autoconf` and `libtool` are added to build RT Topology, and Spatialite is now built with `--disable-minizip` (minizip wasn't an option in 4.4 and I didn't want to add another dependency) and `--disable-dependency-tracking` which, according to Spatialite, \"speeds up one-time builds\"", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1254/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 827341657, "node_id": "MDExOlB1bGxSZXF1ZXN0NTg5MjYzMjk3", "number": 1256, "title": "Minor type in IP adress", "user": {"value": 6371750, "label": "JBPressac"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-03-10T08:28:22Z", "updated_at": "2021-03-10T18:26:46Z", "closed_at": "2021-03-10T18:26:40Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/1256", "body": "127.0.01 replaced by 127.0.0.1", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1256/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 824750134, "node_id": "MDU6SXNzdWU4MjQ3NTAxMzQ=", "number": 1251, "title": "facet option not appearing when table is big", "user": {"value": 15836677, "label": "verajosemanuel"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-03-08T16:54:04Z", "updated_at": "2021-03-08T16:54:16Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I have a big table with more than 500.000 rows.\r\nTrying to facet by one of my columns, the options are not available as for the other smaller tables.\r\n\r\nI have tried to set it in URL as:\r\n\r\n`&_facet=city_id`\r\n\r\nto no avail.\r\n\r\nis there any limit? how can I force the option \"facet\" to appear for big tables?\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1251/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 823035080, "node_id": "MDU6SXNzdWU4MjMwMzUwODA=", "number": 1248, "title": "duckdb database (very low performance in SQLite)", "user": {"value": 15836677, "label": "verajosemanuel"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-03-05T12:20:29Z", "updated_at": "2021-03-08T00:25:27Z", "closed_at": "2021-03-08T00:25:27Z", "author_association": "NONE", "pull_request": null, "body": "My sqlite is getting too big to be processed by datasette (more than 10 minutes waiting to load) so I am working with duckdb and is waaaaay faster. I think the fastest embeddable database actually.\r\n\r\nhttps://duckdb.org/\r\n\r\nTaking into account DuckDb is SQLite based it would be GREAT to use it with datasette.\r\n\r\nis that possible?\r\n\r\nRegards and thanks for a superb job", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1248/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 806918878, "node_id": "MDExOlB1bGxSZXF1ZXN0NTcyMjU0MTAz", "number": 1223, "title": "Add compile option to Dockerfile to fix failing test (fixes #696)", "user": {"value": 7476523, "label": "bobwhitelock"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-02-12T03:38:05Z", "updated_at": "2021-03-07T12:01:12Z", "closed_at": "2021-03-07T07:41:17Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/1223", "body": "This test was failing when run inside the Docker container: `test_searchable[/fixtures/searchable.json?_search=te*+AND+do*&_searchmode=raw-expected_rows3]`,\r\n\r\nwith this error:\r\n\r\n```\r\n def test_searchable(app_client, path, expected_rows):\r\n response = app_client.get(path)\r\n> assert expected_rows == response.json[\"rows\"]\r\nE AssertionError: assert [[1, 'barry c...sel', 'puma']] == []\r\nE Left contains 2 more items, first extra item: [1, 'barry cat', 'terry dog', 'panther']\r\nE Full diff:\r\nE + []\r\nE - [[1, 'barry cat', 'terry dog', 'panther'],\r\nE - [2, 'terry dog', 'sara weasel', 'puma']]\r\n```\r\n\r\nThe issue was that the version of sqlite3 built inside the Docker container was built with FTS3 and FTS4 enabled, but without the\r\n`SQLITE_ENABLE_FTS3_PARENTHESIS` compile option passed, which adds support for using `AND` and `NOT` within `match` expressions (see https://sqlite.org/fts3.html#compiling_and_enabling_fts3_and_fts4 and https://www.sqlite.org/compile.html).\r\n\r\nWithout this, the `AND` used in the search in this test was being interpreted as a literal string, and so no matches were found. Adding this compile option fixes this.\r\n\r\n---\r\n\r\nI actually ran into this issue because the same test was failing when I ran the test suite on my own machine, outside of Docker, and so I eventually tracked this down to my system sqlite3 also being compiled without this option.\r\n\r\nI wonder if this is a sign of a slightly deeper issue, that Datasette can silently behave differently based on the version and compilation of sqlite3 it is being used with. On my own system I fixed the test suite by running `pip install pysqlite3-binary`, so that this would be picked up instead of the `sqlite` package, as this seems to be compiled using this option, . Maybe using `pysqlite3-binary` could be installed/recommended by default so a more deterministic version of sqlite is used? Or there could be some feature detection done on the available sqlite version, to know what features are available and can be used/tested?", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1223/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 617323873, "node_id": "MDU6SXNzdWU2MTczMjM4NzM=", "number": 766, "title": "Enable wildcard-searches by default", "user": {"value": 2181410, "label": "clausjuhl"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-05-13T10:14:48Z", "updated_at": "2021-03-05T16:35:21Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi Simon.\r\n\r\nIt seems that datasette currently has wildcard-searches disabled by default (along with the boolean search-options, NEAR-queries and more, and despite the docs). If I try out the search-url provided in the [docs](https://datasette.readthedocs.io/en/stable/full_text_search.html#the-table-page-and-table-view-api) (https://fara.datasettes.com/fara/FARA_All_ShortForms?_search=manafort), it does not handle wildcard-searches, and I'm unable to make it work on my datasette-instance.\r\n\r\nI would argue that wildcard-searches is such a standard query, that it should be enabled by default. Requiring \"_searchmode=raw\" when using prefix-searches seems unnecessary. Plus: What happens to non-ascii searches when using \"_searchmode=raw\"? Is the \"escape_fts\"-function from datasette.utils ignored?\r\n\r\n\r\nThanks!\r\n\r\n/Claus", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/766/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 778380836, "node_id": "MDU6SXNzdWU3NzgzODA4MzY=", "number": 4, "title": "Feature Request: Gmail", "user": {"value": 203343, "label": "Btibert3"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2021-01-04T21:31:09Z", "updated_at": "2021-03-04T20:54:44Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "From takeout, I only exported my Gmail account. Ideally I could parse this into sqlite via this tool.", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 821841046, "node_id": "MDU6SXNzdWU4MjE4NDEwNDY=", "number": 6, "title": "Upgrade to latest sqlite-utils", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-03-04T07:21:54Z", "updated_at": "2021-03-04T07:22:51Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "This is pinned to v1 at the moment.", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 815955014, "node_id": "MDExOlB1bGxSZXF1ZXN0NTc5Njk3ODMz", "number": 1243, "title": "fix small typo", "user": {"value": 306240, "label": "UtahDave"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-02-25T00:22:34Z", "updated_at": "2021-03-04T05:46:10Z", "closed_at": "2021-03-04T05:46:10Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/1243", "body": "", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1243/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 818684978, "node_id": "MDU6SXNzdWU4MTg2ODQ5Nzg=", "number": 243, "title": "How can i use this utils to deal with fts on column meta of tables ?", "user": {"value": 27874014, "label": "svjack"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-03-01T09:45:05Z", "updated_at": "2021-03-01T09:45:05Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Thank you to release this bravo project.\r\nWhen i use this project on multi table db, I want to implement \r\nconvenient search on column name from different tables.\r\nI want to develop a meta table to save the meta data of different columns\r\nof different tables and search on this meta table to get rows from the\r\ndata table (which the meta table describes)\r\ndoes this project provide some simple function on it ?\r\n\r\nYou can think a have a knowledge graph about the table in the db, \r\nand i save this knowledge graph into the db with fts enabled.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/243/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 818430405, "node_id": "MDU6SXNzdWU4MTg0MzA0MDU=", "number": 1247, "title": "datasette.add_memory_database() method", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-03-01T03:48:38Z", "updated_at": "2021-03-01T04:02:26Z", "closed_at": "2021-03-01T04:02:26Z", "author_association": "OWNER", "pull_request": null, "body": "I just wrote this code:\r\n\r\nhttps://github.com/simonw/datasette/blob/47eb885cc2c3aafa03645c330c6f597bee9b3b25/tests/test_facets.py#L334-L335\r\n\r\nIt would be nice if you didn't have to separately instantiate a database object here.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1247/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 817597268, "node_id": "MDU6SXNzdWU4MTc1OTcyNjg=", "number": 1246, "title": "Suggest for ArrayFacet possibly confused by blank values", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-02-26T19:11:52Z", "updated_at": "2021-03-01T03:46:11Z", "closed_at": "2021-03-01T03:46:11Z", "author_association": "OWNER", "pull_request": null, "body": "I sometimes don't get the suggestion for facet-by-array for columns that contain arrays. I think it may be because they have empty spaces in them - or perhaps it's because the null detection doesn't actually work.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1246/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 718259202, "node_id": "MDU6SXNzdWU3MTgyNTkyMDI=", "number": 1005, "title": "Remove xfail tests when new httpx is released", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 3, "created_at": "2020-10-09T16:00:19Z", "updated_at": "2021-02-28T22:41:08Z", "closed_at": "2021-02-28T22:41:08Z", "author_association": "OWNER", "pull_request": null, "body": "> My `httpx` pull request adding `raw_path` support was just merged: https://github.com/encode/httpx/pull/1357 - but it's not in a release yet.\r\n>\r\n> I'm going to mark these tests as `xfail` so I can land this change - I'll remove that once an `httpx` release comes out that I can use to get the tests passing.\r\n>\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/pull/1000#issuecomment-706263157_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1005/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 814591962, "node_id": "MDU6SXNzdWU4MTQ1OTE5NjI=", "number": 1240, "title": "Allow facetting on custom queries", "user": {"value": 7107523, "label": "Kabouik"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-02-23T15:52:19Z", "updated_at": "2021-02-26T18:19:46Z", "closed_at": "2021-02-26T18:18:18Z", "author_association": "NONE", "pull_request": null, "body": "Facets are a tremendously useful feature, especially for people peeking at the database for the first time and still having little knowledge about the details of the data. It is of great assistance to discover interesting features to explore futher in advanced queries.\r\n\r\nYet, it seems it's impossible to use facets when running a custom SQL query, be it from the little gear icons in column names, the facet suggestions at the top (hidden when performing a custom query), or by appending a facet code to the URL. \r\n\r\nIs there a technical limitation, or is this something that could be unlocked easily?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1240/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 817528452, "node_id": "MDU6SXNzdWU4MTc1Mjg0NTI=", "number": 1244, "title": "Plugin tip: look at the examples linked from the hooks page", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-02-26T17:18:27Z", "updated_at": "2021-02-26T17:30:38Z", "closed_at": "2021-02-26T17:27:15Z", "author_association": "OWNER", "pull_request": null, "body": "Someone asked \"what are good example plugins I can look at?\" and I realized that the answer is to look through the example links on https://docs.datasette.io/en/stable/plugin_hooks.html - but that tip should be written down somewhere on the https://docs.datasette.io/en/stable/writing_plugins.html page.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1244/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 815554385, "node_id": "MDU6SXNzdWU4MTU1NTQzODU=", "number": 237, "title": "db[\"my_table\"].drop(ignore=True) parameter, plus sqlite-utils drop-table --ignore and drop-view --ignore", "user": {"value": 649467, "label": "mhalle"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-02-24T14:55:06Z", "updated_at": "2021-02-25T17:11:41Z", "closed_at": "2021-02-25T17:11:41Z", "author_association": "NONE", "pull_request": null, "body": "When I'm generating a derived table in python, I often drop the table and create it from scratch. However, the first time I generate the table, it doesn't exist, so the drop raises an exception. That means more boilerplate.\r\n\r\nI was going to submit a pull request that adds an \"if_exists\" option to the `drop` method of tables and views. \r\n\r\nHowever, for a utility like sqlite_utils, perhaps the \"IF EXISTS\" SQL semantics is what you want most of the time, and thus should be the default.\r\n\r\nWhat do you think?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/237/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 816523763, "node_id": "MDU6SXNzdWU4MTY1MjM3NjM=", "number": 238, "title": ".add_foreign_key() corrupts database if column contains a space", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-02-25T15:07:20Z", "updated_at": "2021-02-25T16:54:02Z", "closed_at": "2021-02-25T16:54:02Z", "author_association": "OWNER", "pull_request": null, "body": "I ran this:\r\n\r\n db[\"Reports\"].add_foreign_key(\"Reported by ID\", \"Reporters\", \"id\")\r\n\r\nAnd got this:\r\n\r\n```\r\n~/jupyter-venv/lib/python3.9/site-packages/sqlite_utils/db.py in add_foreign_keys(self, foreign_keys)\r\n 616 # Have to VACUUM outside the transaction to ensure .foreign_keys property\r\n 617 # can see the newly created foreign key.\r\n--> 618 self.vacuum()\r\n 619 \r\n 620 def index_foreign_keys(self):\r\n\r\n~/jupyter-venv/lib/python3.9/site-packages/sqlite_utils/db.py in vacuum(self)\r\n 629 \r\n 630 def vacuum(self):\r\n--> 631 self.execute(\"VACUUM;\")\r\n 632 \r\n 633 \r\n\r\n~/jupyter-venv/lib/python3.9/site-packages/sqlite_utils/db.py in execute(self, sql, parameters)\r\n 234 return self.conn.execute(sql, parameters)\r\n 235 else:\r\n--> 236 return self.conn.execute(sql)\r\n 237 \r\n 238 def executescript(self, sql):\r\n\r\nDatabaseError: database disk image is malformed\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/238/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 816560819, "node_id": "MDU6SXNzdWU4MTY1NjA4MTk=", "number": 240, "title": "table.pks_and_rows_where() method returning primary keys along with the rows", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2021-02-25T15:49:28Z", "updated_at": "2021-02-25T16:39:23Z", "closed_at": "2021-02-25T16:28:23Z", "author_association": "OWNER", "pull_request": null, "body": "*Original title: Easier way to update a row returned from .rows*\r\n\r\nHere's a surprisingly hard problem I ran into while trying to implement #239 - given a row returned by `db[table].rows` how can you update that row?\r\n\r\nThe problem is that the `db[table].update(...)` method requires a primary key. But if you have a row from the `db[table].rows` iterator it might not even contain the primary key - provided the table is a `rowid` table.\r\n\r\nInstead, currently, you need to introspect the table and, if `rowid` is a primary key, explicitly include that in the `select=` argument to `table.rows_where(...)` - otherwise it will not be returned.\r\n\r\nA utility mechanism to make this easier would be very welcome.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/240/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 816601354, "node_id": "MDExOlB1bGxSZXF1ZXN0NTgwMjM1NDI3", "number": 241, "title": "Extract expand - work in progress", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-02-25T16:36:38Z", "updated_at": "2021-02-25T16:36:38Z", "closed_at": null, "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/241", "body": "Refs #239. Still needs documentation and CLI implementation.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/241/reactions\", \"total_count\": 3, \"+1\": 3, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 1, "state_reason": null} {"id": 803356942, "node_id": "MDU6SXNzdWU4MDMzNTY5NDI=", "number": 1218, "title": " /usr/local/opt/python3/bin/python3.6: bad interpreter: No such file or directory", "user": {"value": 11855322, "label": "robmarkcole"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-02-08T09:07:00Z", "updated_at": "2021-02-23T12:12:17Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Error as above, however I do have python3.8 and the readme indicates this is supported.\r\n\r\n```\r\n(venv) (base) Robins-MacBook:datasette robin$ ls /usr/local/opt/python3/bin/\r\n\r\n.. pip3 python3 python3.8\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1218/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 813978858, "node_id": "MDU6SXNzdWU4MTM5Nzg4NTg=", "number": 1239, "title": "JSON filter fails if column contains spaces", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-02-23T00:18:07Z", "updated_at": "2021-02-23T00:22:53Z", "closed_at": "2021-02-23T00:22:53Z", "author_association": "OWNER", "pull_request": null, "body": "Got this exception:\r\n\r\n`ERROR: conn=, sql = 'select Address, Affiliation, County, [Has Report], [Latest report notes], [Latest report yes], Latitude, [Location Type], Longitude, Name, id, [Appointment scheduling instructions], [Availability Info], [Latest report] from locations where rowid in (\\n select locations.rowid from locations, json_each(locations.Availability Info) j\\n where j.value = :p0\\n ) and \"Latest report yes\" = :p1 order by id limit 101', params = {'p0': 'Yes: appointment required', 'p1': '1'}: near \"Info\": syntax error`", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1239/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 811505638, "node_id": "MDU6SXNzdWU4MTE1MDU2Mzg=", "number": 1234, "title": "Runtime support for ATTACHing multiple databases", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-02-18T22:06:47Z", "updated_at": "2021-02-22T21:06:28Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> The implementation in #1232 is ready to land. It's the simplest-thing-that-could-possibly-work: you can run `datasette one.db two.db three.db --crossdb` and then use the `/_memory` page to run joins across tables from multiple databases.\r\n>\r\n> It only works on the first 10 databases that were passed to the command-line. This means that if you have a Datasette instance with hundreds of attached databases (see [Datasette Library](https://github.com/simonw/datasette/issues/417)) this won't be particularly useful for you.\r\n>\r\n> So... a better, future version of this feature would be one that lets you join across databases on command - maybe by hitting `/_memory?attach=db1&attach=db2` to get a special connection.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/283#issuecomment-781665560_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1234/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 783778672, "node_id": "MDU6SXNzdWU3ODM3Nzg2NzI=", "number": 220, "title": "Better error message for *_fts methods against views", "user": {"value": 649467, "label": "mhalle"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-01-11T23:24:00Z", "updated_at": "2021-02-22T20:44:51Z", "closed_at": "2021-02-14T22:34:26Z", "author_association": "NONE", "pull_request": null, "body": "enable_fts and its related methods only work on tables, not views. \r\n\r\nCould those methods and possibly others move up to the Queryable superclass?\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/220/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 797651831, "node_id": "MDU6SXNzdWU3OTc2NTE4MzE=", "number": 1212, "title": "Tests are very slow. ", "user": {"value": 4488943, "label": "kbaikov"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-01-31T08:06:16Z", "updated_at": "2021-02-19T22:54:13Z", "closed_at": "2021-02-19T22:54:13Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Working on my PR i noticed that tests are very slow.\r\n\r\nThe plain pytest run took about 37 minutes for me.\r\nHowever i could shave of about 10 minutes from that if i used pytest-xdist to parallelize execution.\r\n`pytest -n 8` is run only in 28 minutes on my machine.\r\n\r\nI can create a PR to mention that in your documentation.\r\nThis will be a simple change to add pytest-xdist to requirements and change a command to run pytest in documentation.\r\n\r\nDoes that make sense to you?\r\n\r\nAfter a bit more investigation it looks like python-xdist is not an answer. It creates a race condition for tests that try to clead temp dir before run.\r\n\r\nProfiling shows that most time is spent on conn.executescript(TABLES) in make_app_client function. Which makes sense.\r\n\r\nPerhaps the better approach would be look at the app_client fixture which is already session scoped, but not used by all test cases.\r\nAnd/or use conn = sqlite3.connect(\":memory:\") which is much faster.\r\nAnd/or truncate tables after each TC instead of deleting the file and re-creating them.\r\n\r\nI can take a look which is the best approach if you give the go-ahead. ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1212/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 811680502, "node_id": "MDU6SXNzdWU4MTE2ODA1MDI=", "number": 236, "title": "--attach command line option for attaching extra databases", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-02-19T04:38:30Z", "updated_at": "2021-02-19T05:10:41Z", "closed_at": "2021-02-19T05:08:43Z", "author_association": "OWNER", "pull_request": null, "body": "This will enable cross-database joins, as seen in https://github.com/simonw/datasette/issues/283\r\n\r\nAlso refs #113", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/236/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 621286870, "node_id": "MDU6SXNzdWU2MjEyODY4NzA=", "number": 113, "title": "Syntactic sugar for ATTACH DATABASE", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-05-19T21:10:00Z", "updated_at": "2021-02-19T05:09:12Z", "closed_at": "2021-02-19T04:56:36Z", "author_association": "OWNER", "pull_request": null, "body": "https://www.sqlite.org/lang_attach.html\r\n\r\nMaybe something like this:\r\n```python\r\ndb.attach(\"other_db\", \"other_db.db\")\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/113/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 811589344, "node_id": "MDU6SXNzdWU4MTE1ODkzNDQ=", "number": 1235, "title": "Upgrade Python version used by official Datasette Docker image", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-02-19T00:47:40Z", "updated_at": "2021-02-19T01:48:31Z", "closed_at": "2021-02-19T01:48:30Z", "author_association": "OWNER", "pull_request": null, "body": "Currently uses 3.7.2:\r\n\r\nhttps://github.com/simonw/datasette/blob/73bed175631a79e13a521eee82f8451dd0477eb3/Dockerfile#L1\r\n\r\nThere's a security fix for Python which it would be good to ship in this image (even though I'm reasonably confident it doesn't affect Datasette): https://bugs.python.org/issue42938", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1235/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 811407131, "node_id": "MDExOlB1bGxSZXF1ZXN0NTc1OTQwMTkz", "number": 1232, "title": "--crossdb option for joining across databases", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 8, "created_at": "2021-02-18T19:48:50Z", "updated_at": "2021-02-18T22:09:13Z", "closed_at": "2021-02-18T22:09:12Z", "author_association": "OWNER", "pull_request": "simonw/datasette/pulls/1232", "body": "Refs #283. Still needs:\r\n\r\n- [x] Unit test for --crossdb queries\r\n- [x] Show warning on console if it truncates at ten databases (or on web interface)\r\n- [x] Show connected databases on the `/_memory` database page\r\n- [x] Documentation\r\n- [x] https://latest.datasette.io/ demo should demonstrate this feature", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1232/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 811458446, "node_id": "MDU6SXNzdWU4MTE0NTg0NDY=", "number": 1233, "title": "\"datasette publish cloudrun\" cannot publish files with spaces in their name", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-02-18T21:08:31Z", "updated_at": "2021-02-18T21:10:08Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Got this error:\r\n```\r\nStep 6/9 : RUN datasette inspect fixtures.db extra database.db --inspect-file inspect-data.json\r\n ---> Running in db9da0068592\r\nUsage: datasette inspect [OPTIONS] [FILES]...\r\nTry 'datasette inspect --help' for help.\r\n\r\nError: Invalid value for '[FILES]...': Path 'extra' does not exist.\r\nThe command '/bin/sh -c datasette inspect fixtures.db extra database.db --inspect-file inspect-data.json' returned a non-zero code: 2\r\nERROR\r\nERROR: build step 0 \"gcr.io/cloud-builders/docker\" failed: step exited with non-zero status: 2\r\n```\r\nWhile working on the demo for #1232, using this deploy command:\r\n```\r\nGITHUB_SHA=crossdb datasette publish cloudrun fixtures.db 'extra database.db' \\\r\n -m fixtures.json \\\r\n --plugins-dir=plugins \\\r\n --branch=$GITHUB_SHA \\\r\n --version-note=$GITHUB_SHA \\\r\n --extra-options=\"--setting template_debug 1 --crossdb\" \\\r\n --install=pysqlite3-binary \\\r\n --service=datasette-latest-crossdb\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1233/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 808843401, "node_id": "MDU6SXNzdWU4MDg4NDM0MDE=", "number": 1226, "title": "--port option should validate port is between 0 and 65535", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-02-15T22:01:33Z", "updated_at": "2021-02-18T18:41:27Z", "closed_at": "2021-02-18T18:41:27Z", "author_association": "OWNER", "pull_request": null, "body": "Currently throws an ugly error message:\r\n```\r\n(datasette-graphql) datasette-graphql % datasette fivethirtyeight.db -p 80094\r\nINFO: Started server process [45497]\r\nINFO: Waiting for application startup.\r\nINFO: Application startup complete.\r\nTraceback (most recent call last):\r\n File \"/Users/simon/.local/share/virtualenvs/datasette-graphql-n1OSJCS8/bin/datasette\", line 8, in \r\n sys.exit(cli())\r\n...\r\n server = await loop.create_server(\r\n File \"/Users/simon/.pyenv/versions/3.8.2/lib/python3.8/asyncio/base_events.py\", line 1461, in create_server\r\n sock.bind(sa)\r\nOverflowError: bind(): port must be 0-65535.\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1226/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 811054000, "node_id": "MDU6SXNzdWU4MTEwNTQwMDA=", "number": 1230, "title": "Vega charts are plotted only for rows on the visible page, cluster maps only for rows in the remaining pages", "user": {"value": 7107523, "label": "Kabouik"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-02-18T12:27:02Z", "updated_at": "2021-02-18T15:22:15Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I filtered a data set on some criteria and obtain 265 results, split over three pages (100, 100, 65), and reazlized that Vega plots are only applied to the results displayed on the current page, instead of the whole filtered data, _e.g._, 100 on page 1, 100 on page 2, 65 on page 3. Is there a way to force the graphs to consider all results instead of just the page, considering that pages rarely represent sensible information?\r\n\r\nLikewise, while the cluster map does show all results on the first page, if you go to next pages, it will show all remaining results except the previous page(s), _e.g._, 265 on page 1, 165 on page 2, 65 on page 3.\r\n\r\nIn both cases, I don't see many situations where one would like to represent the data this way, and it might even lead to interpretation errors when viewing the data. Am I missing some cases where this would be best? Perhaps a clickable option to subset visual representations according visible pages _vs._ display all search results would do?\r\n\r\n[Edit] Oh, I just saw the \"Load all\" button under the cluster map as well as the [setting to alter the max number or results](https://docs.datasette.io/en/stable/settings.html#max-returned-rows). So I guess this issue only is about the Vega charts.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": null, "draft": null, "state_reason": null} {"id": 807174161, "node_id": "MDU6SXNzdWU4MDcxNzQxNjE=", "number": 227, "title": "Error reading csv files with large column data", "user": {"value": 295329, "label": "camallen"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-02-12T11:51:47Z", "updated_at": "2021-02-16T11:48:03Z", "closed_at": "2021-02-14T21:17:19Z", "author_association": "NONE", "pull_request": null, "body": "*Feel free to close this issue - I mostly added it for reference for future folks that run into this :)*\r\n\r\nI have a CSV file with one column that has very long strings. When i try to import this file via the `insert` command I get the following error: \r\n```\r\nsqlite-utils insert database.db table_name file_with_large_column.csv\r\n\r\nTraceback (most recent call last):\r\n File \"/usr/local/bin/sqlite-utils\", line 10, in \r\n sys.exit(cli())\r\n File \"/usr/local/lib/python3.7/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/usr/local/lib/python3.7/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/usr/local/lib/python3.7/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/usr/local/lib/python3.7/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/usr/local/lib/python3.7/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/usr/local/lib/python3.7/site-packages/sqlite_utils/cli.py\", line 774, in insert\r\n default=default,\r\n File \"/usr/local/lib/python3.7/site-packages/sqlite_utils/cli.py\", line 705, in insert_upsert_implementation\r\n docs, pk=pk, batch_size=batch_size, alter=alter, **extra_kwargs\r\n File \"/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py\", line 1852, in insert_all\r\n first_record = next(records)\r\n File \"/usr/local/lib/python3.7/site-packages/sqlite_utils/cli.py\", line 703, in \r\n docs = (decode_base64_values(doc) for doc in docs)\r\n File \"/usr/local/lib/python3.7/site-packages/sqlite_utils/cli.py\", line 681, in \r\n docs = (dict(zip(headers, row)) for row in reader)\r\n_csv.Error: field larger than field limit (131072)\r\n```\r\nBuilt with the docker image `datasetteproject/datasette:0.54` with the following versions:\r\n```\r\n# sqlite-utils --version\r\nsqlite-utils, version 3.4.1\r\n\r\n# datasette --version\r\ndatasette, version 0.54\r\n```\r\nIt appears this is a [known issue](https://stackoverflow.com/a/54517228/2761423) reading in csv files in python and [doesn't look to be modifiable](https://github.com/python/cpython/blob/ea46579067fd2d4e164d6605719ffec690c4d621/Modules/_csv.c#L1685) through system / env vars (i may be very wrong on this).\r\n\r\nNoting that using sqlite3 `import` command work without error (not using the python csv reader)\r\n```\r\nsqlite3 database.db\r\nsqlite> .mode csv\r\nsqlite> .import file_with_large_column.csv table_name\r\n```\r\nSadly I couldn't see an easy way around this while using the cli as it appears this value needs to be changed in python code. FWIW I've switched to using https://datasette.io/tools/csvs-to-sqlite for importing csv data and it's working well. \r\n\r\nFinally, I'm loving https://datasette.io/ thank you very much for an amazing tool and data ecosytem \ud83d\ude47\u200d\u2640\ufe0f ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/227/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 688670158, "node_id": "MDU6SXNzdWU2ODg2NzAxNTg=", "number": 147, "title": "SQLITE_MAX_VARS maybe hard-coded too low", "user": {"value": 96218, "label": "simonwiles"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2020-08-30T07:26:45Z", "updated_at": "2021-02-15T21:27:55Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "I came across this while about to open an issue and PR against the documentation for `batch_size`, which is a bit incomplete.\r\n\r\nAs mentioned in #145, while:\r\n\r\n> [`SQLITE_MAX_VARIABLE_NUMBER`](https://www.sqlite.org/limits.html#max_variable_number) ... defaults to 999 for SQLite versions prior to 3.32.0 (2020-05-22) or 32766 for SQLite versions after 3.32.0.\r\n\r\nit is common that it is increased at compile time. Debian-based systems, for example, seem to ship with a version of sqlite compiled with SQLITE_MAX_VARIABLE_NUMBER set to 250,000, and I believe this is the case for homebrew installations too.\r\n\r\nIn working to understand what `batch_size` was actually doing and why, I realized that by setting `SQLITE_MAX_VARS` in `db.py` to match the value my sqlite was compiled with (I'm on Debian), I was able to decrease the time to `insert_all()` my test data set (~128k records across 7 tables) from ~26.5s to ~3.5s. Given that this about .05% of my total dataset, this is time I am keen to save...\r\n\r\nUnfortunately, it seems that `sqlite3` in the python standard library doesn't expose the `get_limit()` C API (even though `pysqlite` used to), so it's hard to know what value sqlite has been compiled with (note that this could mean, I suppose, that it's less than 999, and even hardcoding `SQLITE_MAX_VARS` to the conservative default might not be adequate. It can also be lowered -- but not raised -- at runtime). The best I could come up with is `echo \"\" | sqlite3 -cmd \".limits variable_number\"` (only available in `sqlite >= 2015-05-07 (3.8.10)`).\r\n\r\nObviously this couldn't be relied upon in `sqlite_utils`, but I wonder what your opinion would be about exposing `SQLITE_MAX_VARS` as a user-configurable parameter (with suitable \"here be dragons\" warnings)? I'm going to go ahead and monkey-patch it for my purposes in any event, but it seems like it might be worth considering.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/147/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 808771690, "node_id": "MDU6SXNzdWU4MDg3NzE2OTA=", "number": 1225, "title": "More flexible formatting of records with CSS grid", "user": {"value": 649467, "label": "mhalle"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-02-15T19:28:17Z", "updated_at": "2021-02-15T19:28:35Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "In several applications I've been experimenting with alternate formatting of datasette query results. Lately I've found that CSS grids work very well and seem quite general for formatting rows. In CSS I use grid templates to define the layout of each record and the regions for each field, hiding the fields I don't want. It's pretty flexible and looks good. It's also a great basis for highly responsive layout.\r\n\r\nI initially thought I'd only use this feature for record detail views, but now I use it for index views as well. \r\n\r\nHowever, there are some limitations:\r\n* With the existing table templates, it seems that you can change the `display` property on the enclosing `table`, `tbody`, and `tr` to make them be grid-like, but that seems hacky (convert `table` and `tbody` to be `display: block` and `tr` to be `display: grid`).\r\n* More significantly, it's very nice to have the column name available when rendering each record to display headers/field labels. The existing templates don't do that, so a custom `_table` template is necessary.\r\n* I don't know if any plugins are sensitive to whether data is rendered as a table or not since I'm not completely clear how plugins get their data. \r\n* Regardless, you need custom CSS to take full advantage of grids. I don't have a proposal on how to integrate them more deeply.\r\n\r\nIt would be helpful to at least have an official example or test that used a grid layout for records to make sure nothing in datasette breaks with it. ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1225/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 807817197, "node_id": "MDU6SXNzdWU4MDc4MTcxOTc=", "number": 229, "title": "Hitting `_csv.Error: field larger than field limit (131072)`", "user": {"value": 631242, "label": "frosencrantz"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-02-13T19:52:44Z", "updated_at": "2021-02-14T21:33:33Z", "closed_at": "2021-02-14T21:33:33Z", "author_association": "NONE", "pull_request": null, "body": "I have a csv file where one of the fields is so large it is throwing an exception with this error and stops loading:\r\n\t```\r\n\t_csv.Error: field larger than field limit (131072)\r\n\t```\r\n\r\nThe stack trace occurs here: https://github.com/simonw/sqlite-utils/blob/3.1/sqlite_utils/cli.py#L633\r\n\r\n\r\nThere is a way to handle this that helps:\r\nhttps://stackoverflow.com/questions/15063936/csv-error-field-larger-than-field-limit-131072\r\n\r\nOne issue I had with this problem was sqlite-utils only provides limited context as to where the problem line is.\r\nThere is the progress bar, but that is by percent rather than by line number. It would have been helpful if it could have provided a line number.\r\n\r\nAlso, it would have been useful if it had allowed the loading to continue with later lines.\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/229/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 808008305, "node_id": "MDU6SXNzdWU4MDgwMDgzMDU=", "number": 230, "title": "--sniff option for sniffing delimiters", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 8, "created_at": "2021-02-14T17:43:54Z", "updated_at": "2021-02-14T21:15:33Z", "closed_at": "2021-02-14T19:24:32Z", "author_association": "OWNER", "pull_request": null, "body": "> I just spotted that `csv.Sniffer` in the Python standard library has a `.has_header(sample)` method which detects if the first row appears to be a header or not, which is interesting. https://docs.python.org/3/library/csv.html#csv.Sniffer\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778812050_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/230/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 797159961, "node_id": "MDExOlB1bGxSZXF1ZXN0NTY0MjE1MDEx", "number": 225, "title": "fix for problem in Table.insert_all on search for columns per chunk of rows", "user": {"value": 261237, "label": "nieuwenhoven"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-01-29T20:16:07Z", "updated_at": "2021-02-14T21:04:13Z", "closed_at": "2021-02-14T21:04:13Z", "author_association": "NONE", "pull_request": "simonw/sqlite-utils/pulls/225", "body": "Hi,\r\n\r\nI ran into a problem when trying to create a database from my Apple Healthkit data using [healthkit-to-sqlite](https://github.com/dogsheep/healthkit-to-sqlite). The program crashed because of an invalid insert statement that was generated for table `rDistanceCycling`. \r\n\r\nThe actual problem turned out to be in [sqlite-utils](https://github.com/simonw/sqlite-utils). `Table.insert_all` processes the data to be inserted in chunks of rows and checks for every chunk which columns are used, and it will collect all column names in the variable `all_columns`. The collection of columns is done using a nested list comprehension that is not completely correct. \r\n\r\nI'm using a Windows machine and had to make a few adjustments to the tests in order to be able to run them because they had a posix dependency.\r\n\r\nThanks, kind regards,\r\n\r\nFrans\r\n\r\n```\r\n# this is a (condensed) chunk of data from my Apple healthkit export that caused the problem.\r\n# the 3 last items in the chunk have additional keys: metadata_HKMetadataKeySyncVersion and metadata_HKMetadataKeySyncIdentifier\r\n\r\nchunk = [{'sourceName': 'Apple\u00c2\\xa0Watch van Frans', 'sourceVersion': '7.0.1',\r\n 'device': '<, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:7.0.1>',\r\n 'unit': 'km', 'creationDate': '2020-10-10 12:29:09 +0100', 'startDate': '2020-10-10 12:29:06 +0100',\r\n 'endDate': '2020-10-10 12:29:07 +0100', 'value': '0.00518016'},\r\n {'sourceName': 'Apple\u00c2\\xa0Watch van Frans', 'sourceVersion': '7.0.1',\r\n 'device': '<, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:7.0.1>',\r\n 'unit': 'km', 'creationDate': '2020-10-10 12:29:10 +0100', 'startDate': '2020-10-10 12:29:07 +0100',\r\n 'endDate': '2020-10-10 12:29:08 +0100', 'value': '0.00544049'},\r\n {'sourceName': 'Apple\u00c2\\xa0Watch van Frans', 'sourceVersion': '6.2.6',\r\n 'device': '<, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:6.2.6>',\r\n 'unit': 'km', 'creationDate': '2020-10-14 05:54:12 +0100', 'startDate': '2020-07-15 16:40:50 +0100',\r\n 'endDate': '2020-07-15 16:42:49 +0100', 'value': '0.952092', 'metadata_HKMetadataKeySyncVersion': '1',\r\n 'metadata_HKMetadataKeySyncIdentifier': '3:674DBCDB-3FE8-40D1-9FC1-E54A2B413805:616520450.99823:616520569.99360:119'},\r\n {'sourceName': 'Apple\u00c2\\xa0Watch van Frans', 'sourceVersion': '6.2.6',\r\n 'device': '<, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:6.2.6>',\r\n 'unit': 'km', 'creationDate': '2020-10-14 05:54:12 +0100', 'startDate': '2020-07-15 16:42:49 +0100',\r\n 'endDate': '2020-07-15 16:44:51 +0100', 'value': '0.848983', 'metadata_HKMetadataKeySyncVersion': '1',\r\n 'metadata_HKMetadataKeySyncIdentifier': '3:674DBCDB-3FE8-40D1-9FC1-E54A2B413805:616520569.99360:616520691.98826:119'},\r\n {'sourceName': 'Apple\u00c2\\xa0Watch van Frans', 'sourceVersion': '6.2.6',\r\n 'device': '<, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:6.2.6>',\r\n 'unit': 'km', 'creationDate': '2020-10-14 05:54:12 +0100', 'startDate': '2020-07-15 16:44:51 +0100',\r\n 'endDate': '2020-07-15 16:46:50 +0100', 'value': '0.834403', 'metadata_HKMetadataKeySyncVersion': '1',\r\n 'metadata_HKMetadataKeySyncIdentifier': '3:674DBCDB-3FE8-40D1-9FC1-E54A2B413805:616520691.98826:616520810.98305:119'}]\r\n\r\n\r\n\r\ndef all_columns_old():\r\n all_columns = [col for col in chunk[0]]\r\n all_columns += [column for record in chunk\r\n for column in record if column not in all_columns]\r\n return all_columns\r\n\r\n\r\ndef all_columns_new():\r\n all_columns = [col for col in chunk[0]]\r\n for record in chunk:\r\n all_columns += [column for column in record if column not in all_columns]\r\n return all_columns\r\n\r\n\r\n\r\nif __name__ == '__main__':\r\n from pprint import pprint\r\n\r\n print('problem: ')\r\n pprint(all_columns_old())\r\n print('\\nfix: ')\r\n pprint(all_columns_new())\r\n\r\n```\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/225/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 808046597, "node_id": "MDU6SXNzdWU4MDgwNDY1OTc=", "number": 234, "title": ".insert_all() fails if subsequent chunks contain additional columns", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-02-14T21:01:51Z", "updated_at": "2021-02-14T21:03:40Z", "closed_at": "2021-02-14T21:03:40Z", "author_association": "OWNER", "pull_request": null, "body": "Reported by @nieuwenhoven in #225 along with a proposed fix.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/234/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 808036774, "node_id": "MDU6SXNzdWU4MDgwMzY3NzQ=", "number": 232, "title": "Run tests against Windows in GitHub Actions", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-02-14T20:09:45Z", "updated_at": "2021-02-14T20:39:55Z", "closed_at": "2021-02-14T20:39:55Z", "author_association": "OWNER", "pull_request": null, "body": "> I'm going to try and get the test suite to run in Windows on GitHub Actions.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/225#issuecomment-778834504_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/232/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 808037010, "node_id": "MDExOlB1bGxSZXF1ZXN0NTczMTQ3MTY4", "number": 233, "title": "Run tests against Ubuntu, macOS and Windows", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-02-14T20:11:02Z", "updated_at": "2021-02-14T20:39:54Z", "closed_at": "2021-02-14T20:39:54Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/233", "body": "Refs #232", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/233/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 808028757, "node_id": "MDU6SXNzdWU4MDgwMjg3NTc=", "number": 231, "title": "limit=X, offset=Y parameters for more Python methods", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-02-14T19:31:23Z", "updated_at": "2021-02-14T20:03:08Z", "closed_at": "2021-02-14T20:03:08Z", "author_association": "OWNER", "pull_request": null, "body": "> I'm going to add a `offset=` parameter to support this case. Thanks for the suggestion!\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/224#issuecomment-778828495_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/231/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 792297010, "node_id": "MDExOlB1bGxSZXF1ZXN0NTYwMjA0MzA2", "number": 224, "title": "Add fts offset docs.", "user": {"value": 37962604, "label": "polyrand"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-01-22T20:50:58Z", "updated_at": "2021-02-14T19:31:06Z", "closed_at": "2021-02-14T19:31:06Z", "author_association": "NONE", "pull_request": "simonw/sqlite-utils/pulls/224", "body": "The limit can be passed as a string to the query builder to have an offset. I have tested it using the shorthand `limit=f\"15, 30\"`, the standard syntax should work too.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/224/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 806743116, "node_id": "MDU6SXNzdWU4MDY3NDMxMTY=", "number": 1220, "title": "Installing datasette via docker: Path 'fixtures.db' does not exist", "user": {"value": 30607, "label": "aborruso"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-02-11T21:09:14Z", "updated_at": "2021-02-12T21:35:17Z", "closed_at": "2021-02-12T21:35:17Z", "author_association": "NONE", "pull_request": null, "body": "Hi,\r\nIf I run\r\n\r\n```\r\ndocker run -p 8001:8001 -v `pwd`:/mnt \\ 1 \u21b5\r\n datasetteproject/datasette \\\r\n datasette -p 8001 -h 0.0.0.0 fixtures.db\r\n```\r\n\r\nI have \r\n\r\n```\r\nError: Invalid value for '[FILES]...': Path 'fixtures.db' does not exist.\r\n```\r\n\r\nIf I run `test -f fixtures.db && echo \"it exists.\"` I have `it exists.`.\r\n\r\nWhat's my error?\r\n\r\nThank you", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1220/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 797097140, "node_id": "MDU6SXNzdWU3OTcwOTcxNDA=", "number": 60, "title": "Use Data from SQLite in other commands", "user": {"value": 22578954, "label": "daniel-butler"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-01-29T18:35:52Z", "updated_at": "2021-02-12T18:29:43Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "As a total beginner here how could you access data from the sqlite table to run other commands.\r\n\r\nWhat I am thinking is I want to get all the repos in an organization then using the repo list pull all the commit messages for each repo. \r\n\r\nI love this project by the way!", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 803338729, "node_id": "MDU6SXNzdWU4MDMzMzg3Mjk=", "number": 33, "title": "photo-to-sqlite: command not found", "user": {"value": 11855322, "label": "robmarkcole"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-02-08T08:42:57Z", "updated_at": "2021-02-12T15:00:44Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Having installed in a venv I get:\r\n```\r\n(venv) (base) Robins-MacBook:datasette robin$ photo-to-sqlite apple-photos photos.db\r\n\r\n-bash: photo-to-sqlite: command not found\r\n```", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 806861312, "node_id": "MDExOlB1bGxSZXF1ZXN0NTcyMjA5MjQz", "number": 1222, "title": "--ssl-keyfile and --ssl-certfile, refs #1221", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-02-12T00:45:58Z", "updated_at": "2021-02-12T00:52:18Z", "closed_at": "2021-02-12T00:52:17Z", "author_association": "OWNER", "pull_request": "simonw/datasette/pulls/1222", "body": "", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1222/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 770712149, "node_id": "MDExOlB1bGxSZXF1ZXN0NTQyNDA2OTEw", "number": 10, "title": "BugFix for encoding and not update info.", "user": {"value": 1277270, "label": "riverzhou"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-12-18T08:58:54Z", "updated_at": "2021-02-11T22:37:56Z", "closed_at": "2021-02-11T22:37:56Z", "author_association": "NONE", "pull_request": "dogsheep/evernote-to-sqlite/pulls/10", "body": "Bugfix 1:\r\n\r\nTraceback (most recent call last):\r\n File \"d:\\anaconda3\\lib\\runpy.py\", line 194, in _run_module_as_main\r\n return _run_code(code, main_globals, None,\r\n File \"d:\\anaconda3\\lib\\runpy.py\", line 87, in _run_code\r\n exec(code, run_globals)\r\n File \"D:\\Anaconda3\\Scripts\\evernote-to-sqlite.exe\\__main__.py\", line 7, in \r\n File \"d:\\anaconda3\\lib\\site-packages\\click\\core.py\", line 829, in __call__\r\n File \"d:\\anaconda3\\lib\\site-packages\\click\\core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"d:\\anaconda3\\lib\\site-packages\\click\\core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"d:\\anaconda3\\lib\\site-packages\\click\\core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"d:\\anaconda3\\lib\\site-packages\\evernote_to_sqlite\\cli.py\", line 30, in enex\r\n for tag, note in find_all_tags(fp, [\"note\"], progress_callback=bar.update):\r\n File \"d:\\anaconda3\\lib\\site-packages\\evernote_to_sqlite\\utils.py\", line 11, in find_all_tags\r\n chunk = fp.read(1024 * 1024)\r\nUnicodeDecodeError: 'gbk' codec can't decode byte 0xa4 in position 383: illegal multibyte sequence\r\n\r\nBugfix 2:\r\n\r\nTraceback (most recent call last):\r\n File \"D:\\Anaconda3\\Scripts\\evernote-to-sqlite-script.py\", line 33, in \r\n sys.exit(load_entry_point('evernote-to-sqlite==0.3', 'console_scripts', 'evernote-to-sqlite')())\r\n File \"D:\\Anaconda3\\lib\\site-packages\\click\\core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"D:\\Anaconda3\\lib\\site-packages\\click\\core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"D:\\Anaconda3\\lib\\site-packages\\click\\core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"D:\\Anaconda3\\lib\\site-packages\\click\\core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"D:\\Anaconda3\\lib\\site-packages\\click\\core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"D:\\Anaconda3\\lib\\site-packages\\evernote_to_sqlite-0.3-py3.8.egg\\evernote_to_sqlite\\cli.py\", line 31, in enex\r\n File \"D:\\Anaconda3\\lib\\site-packages\\evernote_to_sqlite-0.3-py3.8.egg\\evernote_to_sqlite\\utils.py\", line 28, in save_note\r\nAttributeError: 'NoneType' object has no attribute 'text'", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/10/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 748370021, "node_id": "MDExOlB1bGxSZXF1ZXN0NTI1MzcxMDI5", "number": 8, "title": "fix import error if note has no \"updated\" element", "user": {"value": 4028322, "label": "mkorosec"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-11-22T22:51:05Z", "updated_at": "2021-02-11T22:34:06Z", "closed_at": "2021-02-11T22:34:06Z", "author_association": "CONTRIBUTOR", "pull_request": "dogsheep/evernote-to-sqlite/pulls/8", "body": "I got the following error when executing evernote-to-sqlite enex evernote.db evernote.enex\r\n``` \r\n... \r\n File \"evernote_to_sqlite/cli.py\", line 31, in enex\r\n save_note(db, note)\r\n File \"evernote_to_sqlite/utils.py\", line 28, in save_note\r\n updated = note.find(\"updated\").text\r\nAttributeError: 'NoneType' object has no attribute 'text'\r\n``` \r\n\r\nSeems that in some cases the updated element is not added to the note, this is a part of the problematic note:\r\n\r\n``` \r\n20201019T074518Z\r\n\r\n web.clip7\r\n webclipper.evernote\r\n\r\n```", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/8/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 743297582, "node_id": "MDU6SXNzdWU3NDMyOTc1ODI=", "number": 7, "title": "evernote-to-sqlite on windows 10 give this error: TypeError: insert() got an unexpected keyword argument 'replace'", "user": {"value": 42387931, "label": "martinvanwieringen"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-11-15T16:57:28Z", "updated_at": "2021-02-11T22:13:17Z", "closed_at": "2021-02-11T22:13:17Z", "author_association": "NONE", "pull_request": null, "body": "running evernote-to-sqlite 0.2 on windows 10. Command:\r\n\r\nevernote-to-sqlite enex evernote.db MyNotes.enex\r\n\r\nI get the followinng error:\r\n\r\nFile \"C:\\Users\\marti\\AppData\\Roaming\\Python\\Python38\\site-packages\\evernote_to_sqlite\\utils.py\", line 46, in save_note\r\n note_id = db[\"notes\"].insert(row, hash_id=\"id\", replace=True, alter=True).last_pk\r\nTypeError: insert() got an unexpected keyword argument 'replace'\r\n\r\nRemoving replace=True,\r\n\r\nLeads to below error:\r\n\r\n note_id = db[\"notes\"].insert(row, hash_id=\"id\", alter=True).last_pk\r\n File \"C:\\Users\\marti\\AppData\\Roaming\\Python\\Python38\\site-packages\\sqlite_utils\\db.py\", line 924, in insert\r\n return self.insert_all(\r\n File \"C:\\Users\\marti\\AppData\\Roaming\\Python\\Python38\\site-packages\\sqlite_utils\\db.py\", line 1046, in insert_all\r\n result = self.db.conn.execute(sql, values)\r\nsqlite3.IntegrityError: UNIQUE constraint failed: notes.id", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/7/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 748372469, "node_id": "MDU6SXNzdWU3NDgzNzI0Njk=", "number": 9, "title": "ParseError: undefined entity š", "user": {"value": 4028322, "label": "mkorosec"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-11-22T23:04:35Z", "updated_at": "2021-02-11T22:10:55Z", "closed_at": "2021-02-11T22:10:55Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "I encountered a parse error if the enex file contained š or   \r\n\r\nRun command: \r\nevernote-to-sqlite enex evernote.db evernote.enex\r\n\r\n```\r\nTraceback (most recent call last):\r\n...\r\n File \"evernote_to_sqlite/cli.py\", line 31, in enex\r\n save_note(db, note)\r\n File \"evernote_to_sqlite/utils.py\", line 35, in save_note\r\n content = ET.tostring(ET.fromstring(content_xml)).decode(\"utf-8\")\r\n File \"/usr/lib/python3.8/xml/etree/ElementTree.py\", line 1320, in XML\r\n parser.feed(text)\r\nxml.etree.ElementTree.ParseError: undefined entity š: line 3, column 35\r\n```\r\n\r\nWorkaround:\r\n```\r\nsed -i 's/š//g' evernote.enex\r\nsed -i 's/ //g' evernote.enex\r\n```", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/9/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 792851444, "node_id": "MDU6SXNzdWU3OTI4NTE0NDQ=", "number": 11, "title": "XML parse error", "user": {"value": 3613583, "label": "dskrad"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-01-24T17:38:54Z", "updated_at": "2021-02-11T21:18:58Z", "closed_at": "2021-02-11T21:18:48Z", "author_association": "NONE", "pull_request": null, "body": "I am on Windows 10 using Windows Subsystem for Linux, Python 3.8. I installed evernote-to-sqlite via pipx (in a venv). I tried using enex files from the latest version of Evernote for Windows (10.6.9 which only lets you export 50 notes at a time) and from Legacy Evernote (6.25.2.9198 which lets you export all your notes at once). The enex file from latest evernote gives this error:\r\n\r\n File \"/usr/lib/python3.8/xml/etree/ElementTree.py\", line 1320, in XML parser.feed(text)\r\n xml.etree.ElementTree.ParseError: XML or text declaration not at start of entity: line 2, column 6\r\n\r\nThe enex file from Legacy Evernote gives this error:\r\n\r\n File \"/home/david/.local/pipx/venvs/evernote-to-sqlite/lib/python3.8/site-packages/evernote_to_sqlite/utils.py\", line 28, in save_note\r\n updated = note.find(\"updated\").text\r\n AttributeError: 'NoneType' object has no attribute 'text'", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/11/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 792890765, "node_id": "MDU6SXNzdWU3OTI4OTA3NjU=", "number": 1200, "title": "?_size=10 option for the arbitrary query page would be useful", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-01-24T20:55:35Z", "updated_at": "2021-02-11T03:13:59Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://latest.datasette.io/fixtures?sql=select+*+from+compound_three_primary_keys&_size=10 - `_size=10` does not do anything at the moment. It would be useful if it did.\r\n\r\nWould also be good if it persisted in a hidden form field.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1200/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 803929694, "node_id": "MDU6SXNzdWU4MDM5Mjk2OTQ=", "number": 1219, "title": "Try profiling Datasette using scalene", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-02-08T20:37:06Z", "updated_at": "2021-02-08T22:13:00Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://github.com/emeryberger/scalene looks like an interesting profiling tool.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1219/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 801780625, "node_id": "MDU6SXNzdWU4MDE3ODA2MjU=", "number": 9, "title": "SSL Error", "user": {"value": 12669260, "label": "jfeiwell"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-02-05T02:12:56Z", "updated_at": "2021-02-07T18:45:04Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Here's the error I get when running `pip install pocket-to-sqlite`: \r\n\r\n```\r\n Could not fetch URL https://pypi.python.org/simple/pocket-to-sqlite/: There was a problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661) - skipping\r\n Could not find a version that satisfies the requirement pocket-to-sqlite (from versions: )\r\nNo matching distribution found for pocket-to-sqlite\r\n```\r\n\r\nDoes this require python 3? ", "repo": {"value": 213286752, "label": "pocket-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/9/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 802583450, "node_id": "MDU6SXNzdWU4MDI1ODM0NTA=", "number": 226, "title": "3.4 release is broken - includes a rogue line", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-02-06T02:08:01Z", "updated_at": "2021-02-06T02:10:26Z", "closed_at": "2021-02-06T02:10:26Z", "author_association": "OWNER", "pull_request": null, "body": "I started seeing weird errors, caused by this line: https://github.com/simonw/sqlite-utils/blob/f8010ca78fed8c5fca6cde19658ec09fdd468420/sqlite_utils/cli.py#L1-L3\r\n\r\nThat was added by accident in 1b666f9315d4ea6bb332b2e75e48480c26100199\r\n\r\nI'm surprised the tests didn't catch this!", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/226/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 788527932, "node_id": "MDU6SXNzdWU3ODg1Mjc5MzI=", "number": 223, "title": "--delimiter option for CSV import", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-01-18T20:25:03Z", "updated_at": "2021-02-06T01:39:47Z", "closed_at": "2021-02-06T01:34:54Z", "author_association": "OWNER", "pull_request": null, "body": "https://bruxellesdata.opendatasoft.com/explore/dataset/dog-toilets/export/?location=12,50.85802,4.38054 says:\r\n\r\n> CSV uses semicolon (;) as a separator.\r\n\r\nWould be useful to be able to do this:\r\n\r\n sqlite-utils insert places.db places places.csv --delimiter ';'\r\n\r\n`--delimiter` could imply `--csv`", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/223/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 796234313, "node_id": "MDU6SXNzdWU3OTYyMzQzMTM=", "number": 1210, "title": "Immutable Database w/ Canned Queries", "user": {"value": 525780, "label": "heyarne"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-01-28T18:08:29Z", "updated_at": "2021-02-05T11:30:34Z", "closed_at": "2021-02-05T11:30:34Z", "author_association": "NONE", "pull_request": null, "body": "I have a database that I only want to read from; when instructing datasette to treat the database as immutable my defined canned queries disappear. Are these two features incompatible or have I hit an unintended bug? Thanks for datasette in any way, it's a joy to use!", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1210/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 796736607, "node_id": "MDU6SXNzdWU3OTY3MzY2MDc=", "number": 56, "title": "Not all quoted statuses get fetched?", "user": {"value": 42315895, "label": "gsajko"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-01-29T09:48:44Z", "updated_at": "2021-02-03T10:36:36Z", "closed_at": "2021-02-03T10:36:36Z", "author_association": "NONE", "pull_request": null, "body": "\r\n![image](https://user-images.githubusercontent.com/42315895/106259325-5f75dc80-621f-11eb-8311-db8f2fe2a257.png)\r\n\r\nIn my database I have 13300 quote tweets, but eta 3600 have `quoted_status` empty.\r\n\r\nI fetched some of them using `https://api.twitter.com/1.1/statuses/show.json?id=xx` and they did have ids of quoted tweets.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 799693777, "node_id": "MDU6SXNzdWU3OTk2OTM3Nzc=", "number": 1214, "title": "Re-submitting filter form duplicates _x querystring arguments", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-02-02T21:13:35Z", "updated_at": "2021-02-02T21:28:53Z", "closed_at": "2021-02-02T21:21:13Z", "author_association": "OWNER", "pull_request": null, "body": "Really nasty bug, caused by #1194 fix in 07e163561592c743e4117f72102fcd350a600909\r\n\r\nNavigate to this page: https://github-to-sqlite.dogsheep.net/github/labels?_search=help&_sort=id\r\n\r\nClick \"Apply\" to submit the form and the resulting URL is https://github-to-sqlite.dogsheep.net/github/labels?_search=help&_sort=id&_search=help&_sort=id\r\n\r\nThat's because the (truncated) HTML for the form looks like this:\r\n\r\n```html\r\n ... \r\n ...\r\n
\r\n \r\n
\r\n ...\r\n \r\n \r\n \r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1214/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 799663959, "node_id": "MDU6SXNzdWU3OTk2NjM5NTk=", "number": 1213, "title": "gzip support for HTML (and JSON) responses", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-02-02T20:36:28Z", "updated_at": "2021-02-02T20:41:55Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This page https://datasette-tiles-demo.datasette.io/San_Francisco/tiles is 2MB because of all of the base64 images. Gzipped it's 1.5MB.\r\n\r\nSince Datasette is usually deployed without a frontend gzipping proxy, Datasette itself needs to solve for this.\r\n\r\nGzipping everything won't work because some endpoints - the all-rows CSV endpoint and the download-database endpoint - are streaming and hence can't be buffered-and-gzipped.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1213/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 797784080, "node_id": "MDU6SXNzdWU3OTc3ODQwODA=", "number": 62, "title": "Stargazers and workflows commands always require an auth file when using GITHUB_TOKEN ", "user": {"value": 631242, "label": "frosencrantz"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-01-31T18:56:05Z", "updated_at": "2021-01-31T18:56:05Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Requested fix in https://github.com/dogsheep/github-to-sqlite/pull/59\r\n\r\nThe stargazers and workflows commands always require an auth file, even when using a `GITHUB_TOKEN`. Other commands don't require the auth file.\r\n\r\n", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/62/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 797728929, "node_id": "MDU6SXNzdWU3OTc3Mjg5Mjk=", "number": 8, "title": "QUESTION: extract full text", "user": {"value": 417363, "label": "darribas"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-01-31T14:50:10Z", "updated_at": "2021-01-31T14:50:10Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "This may be solved or a feature already, but I couldn't figure it out, is it possible to extract and store also full text from the saved pages? The same way that Pocket parses the text, it'd be amazing to be able to store (and thus make searchable later) the text.\r\n\r\nThank you very much for the project, it's such an amazing idea! ", "repo": {"value": 213286752, "label": "pocket-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/8/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 793881756, "node_id": "MDU6SXNzdWU3OTM4ODE3NTY=", "number": 1207, "title": "Document the Datasette(..., pdb=True) testing pattern", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-01-26T02:48:10Z", "updated_at": "2021-01-29T02:37:19Z", "closed_at": "2021-01-29T02:12:34Z", "author_association": "OWNER", "pull_request": null, "body": "If you're writing tests for a Datasette plugin and you get a 500 error from inside Datasette, you can cause Datasette to open a PDB session within the application server code by doing this:\r\n\r\n```python\r\nds = Datasette([db_path], pdb=True)\r\nresponse = await ds.client.get(\"/\")\r\n```\r\n\r\nYou'll need to run `pytest -s` to interact with the debugger, otherwise you'll get an error.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1207/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 795367402, "node_id": "MDU6SXNzdWU3OTUzNjc0MDI=", "number": 1209, "title": "v0.54 500 error from sql query in custom template; code worked in v0.53; found a workaround", "user": {"value": 11788561, "label": "jrdmb"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-01-27T19:08:13Z", "updated_at": "2021-01-28T23:00:27Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "v0.54 500 error in sql query template; code worked in v0.53; found a workaround\r\n\r\n**schema:** \r\nCREATE TABLE \"talks\" (\"talk\" TEXT,\"series\" INTEGER, \"talkdate\" TEXT) \r\nCREATE TABLE \"series\" (\"id\" INTEGER PRIMARY KEY, \"series\" TEXT, talks_list TEXT default '', website TEXT default '');\r\n\r\n**Live example of correctly rendered template in v.053:** https://cosmotalks-cy6xkkbezq-uw.a.run.app/cosmotalks/talks/1\r\n\r\n**Description of problem:** I needed 'sql select' code in a custom row-mydatabase-mytable.html template to lookup the series name for a foreign key integer value in the talks table. So `metadata.json` specifies the `datasette-template-sql` plugin.\r\n\r\nThe code below worked perfectly in v0.53 (just the relevant sql statement part is shown; full code is [here](https://github.com/jrdmb/cosmotalks-datasette/blob/main/templates/row-cosmotalks-talks.html)):\r\n\r\n```\r\n{# custom addition #} \r\n{% for row in display_rows %} \r\n ... \r\n {% set sname = sql(\"select series from series where id = ?\", [row.series]) %} \r\n Series name: {{ sname[0].series }} \r\n ... \r\n{% endfor %} \r\n{# End of custom addition #} \r\n```\r\n\r\n**In v0.54, that code resulted in a 500 error with a 'no such table series' message.** A second query in that template also did not work but the above is fully illustrative of the problem.\r\n\r\nAll templates were up-to-date along with datasette v0.54.\r\n\r\n**Workaround:** After fiddling around with trying different things, what worked was the syntax from [Querying a different database from the datasette-template-sql github repo](https://github.com/simonw/datasette-template-sql#querying-a-different-database) to add the database name to the sql statement:\r\n\r\n`{% set sname = sql(\"select series from series where id = ?\", [row.series], database=\"cosmotalks\") %}`\r\n\r\nThough this was found to work, it should not be necessary to add `database=\"cosmotalks\"` since per the `datasette-template-sql` README, it's only needed when querying a different database, but here it's a table within the same database.\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": null, "draft": null, "state_reason": null} {"id": 793027837, "node_id": "MDU6SXNzdWU3OTMwMjc4Mzc=", "number": 1205, "title": "Rename /:memory: to /_memory", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 3, "created_at": "2021-01-25T05:04:56Z", "updated_at": "2021-01-28T22:55:02Z", "closed_at": "2021-01-28T22:51:42Z", "author_association": "OWNER", "pull_request": null, "body": "For consistency with `/_internal` - and because then we don't need to escape the `:` characters.\r\n\r\nThis change would need to be in before Datasette 1.0. I could land it earlier and set up redirects from the old URLs though.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1205/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 770448622, "node_id": "MDU6SXNzdWU3NzA0NDg2MjI=", "number": 1151, "title": "Database class mechanism for cross-connection in-memory databases", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6346396, "label": "Datasette 0.54"}, "comments": 11, "created_at": "2020-12-17T23:25:43Z", "updated_at": "2021-01-26T19:07:44Z", "closed_at": "2020-12-18T01:01:26Z", "author_association": "OWNER", "pull_request": null, "body": "> Next challenge: figure out how to use the `Database` class from https://github.com/simonw/datasette/blob/0.53/datasette/database.py for an in-memory database which persists data for the duration of the lifetime of the server, and allows access to that in-memory database from multiple threads in a way that lets them see each other's changes.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1150#issuecomment-747768112_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1151/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 714377268, "node_id": "MDU6SXNzdWU3MTQzNzcyNjg=", "number": 991, "title": "Redesign application homepage", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2020-10-04T18:48:45Z", "updated_at": "2021-01-26T19:06:36Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Most Datasette instances only host a single database, but the current homepage design assumes that it should leave plenty of space for multiple databases:\r\n\r\n\"Datasette_Fixtures__fixtures\"\r\n\r\nReconsider this design - should the default show more information?\r\n\r\nThe Covid-19 Datasette homepage looks particularly sparse I think: https://covid-19.datasettes.com/\r\n\r\n\"COVID-19_cases__using_data_from_Johns_Hopkins_CSSE__the_New_York_Times_and_the_LA_Times__covid\"", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/991/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 793907673, "node_id": "MDExOlB1bGxSZXF1ZXN0NTYxNTEyNTAz", "number": 15, "title": "added try / except to write_records ", "user": {"value": 9857779, "label": "ryancheley"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-01-26T03:56:21Z", "updated_at": "2021-01-26T03:56:21Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/healthkit-to-sqlite/pulls/15", "body": "to keep the data write from failing if it came across an error during processing. In particular when trying to convert my HealthKit zip file (and that of my wife's) it would consistently error out with the following:\r\n\r\n```\r\ndb.py 1709 insert_chunk\r\nresult = self.db.execute(query, params)\r\n\r\ndb.py 226 execute\r\nreturn self.conn.execute(sql, parameters)\r\n\r\nsqlite3.OperationalError:\r\ntoo many SQL variables\r\n\r\n---------------------------------------------------------------------------------------------------------------------------------------------------------------------\r\ndb.py 1709 insert_chunk\r\nresult = self.db.execute(query, params)\r\n\r\ndb.py 226 execute\r\nreturn self.conn.execute(sql, parameters)\r\n\r\nsqlite3.OperationalError:\r\ntoo many SQL variables\r\n\r\n---------------------------------------------------------------------------------------------------------------------------------------------------------------------\r\ndb.py 1709 insert_chunk\r\nresult = self.db.execute(query, params)\r\n\r\ndb.py 226 execute\r\nreturn self.conn.execute(sql, parameters)\r\n\r\nsqlite3.OperationalError:\r\ntable rBodyMass has no column named metadata_HKWasUserEntered\r\n\r\n---------------------------------------------------------------------------------------------------------------------------------------------------------------------\r\nhealthkit-to-sqlite 8 \r\nsys.exit(cli())\r\n\r\ncore.py 829 __call__\r\nreturn self.main(*args, **kwargs)\r\n\r\ncore.py 782 main\r\nrv = self.invoke(ctx)\r\n\r\ncore.py 1066 invoke\r\nreturn ctx.invoke(self.callback, **ctx.params)\r\n\r\ncore.py 610 invoke\r\nreturn callback(*args, **kwargs)\r\n\r\ncli.py 57 cli\r\nconvert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)\r\n\r\nutils.py 42 convert_xml_to_sqlite\r\nwrite_records(records, db)\r\n\r\nutils.py 143 write_records\r\ndb[table].insert_all(\r\n\r\ndb.py 1899 insert_all\r\nself.insert_chunk(\r\n\r\ndb.py 1720 insert_chunk\r\nself.insert_chunk(\r\n\r\ndb.py 1720 insert_chunk\r\nself.insert_chunk(\r\n\r\ndb.py 1714 insert_chunk\r\nresult = self.db.execute(query, params)\r\n\r\ndb.py 226 execute\r\nreturn self.conn.execute(sql, parameters)\r\n\r\nsqlite3.OperationalError:\r\ntable rBodyMass has no column named metadata_HKWasUserEntered\r\n```\r\n\r\nAdding the try / except in the `write_records` seems to fix that issue. ", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/15/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 792904595, "node_id": "MDU6SXNzdWU3OTI5MDQ1OTU=", "number": 1201, "title": "Release notes for Datasette 0.54", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6346396, "label": "Datasette 0.54"}, "comments": 5, "created_at": "2021-01-24T21:22:28Z", "updated_at": "2021-01-25T17:42:21Z", "closed_at": "2021-01-25T17:42:21Z", "author_association": "OWNER", "pull_request": null, "body": "These will incorporate the release notes from the alpha, much expanded: https://github.com/simonw/datasette/releases/tag/0.54a0", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1201/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 793086333, "node_id": "MDExOlB1bGxSZXF1ZXN0NTYwODMxNjM4", "number": 1206, "title": "Release 0.54", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-01-25T06:45:47Z", "updated_at": "2021-01-25T17:33:30Z", "closed_at": "2021-01-25T17:33:29Z", "author_association": "OWNER", "pull_request": "simonw/datasette/pulls/1206", "body": "Refs #1201", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1206/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 780278550, "node_id": "MDU6SXNzdWU3ODAyNzg1NTA=", "number": 1179, "title": "Make original path available to render hooks", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 8, "created_at": "2021-01-06T08:31:45Z", "updated_at": "2021-01-25T04:44:33Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/datasette-export-notebook/blob/0.1/datasette_export_notebook/__init__.py\r\n\r\n```python\r\nasync def render_notebook(datasette, request):\r\n return Response.html(\r\n await datasette.render_template(\r\n \"export_notebook.html\",\r\n {\r\n \"csv_stream_url\": datasette.absolute_url(\r\n request,\r\n path_with_format(\r\n request=request, format=\"csv\", extra_qs={\"_stream\": \"on\"}\r\n ),\r\n ),\r\n \"json_url\": datasette.absolute_url(\r\n request,\r\n path_with_format(\r\n request=request, format=\"json\", extra_qs={\"_shape\": \"array\"}\r\n ),\r\n ),\r\n \"json\": json,\r\n },\r\n )\r\n )\r\n```\r\nThis results in https://latest-with-plugins.datasette.io/github/issue_comments.Notebook showing `http://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_format=json&_shape=array`", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1179/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 712260429, "node_id": "MDU6SXNzdWU3MTIyNjA0Mjk=", "number": 983, "title": "JavaScript plugin hooks mechanism similar to pluggy", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 47, "created_at": "2020-09-30T20:32:43Z", "updated_at": "2021-01-25T04:43:58Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> It would be neat to provide a JavaScript plugin hook that plugins can use to add their own options to this menu. No idea what that would look like though.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/981#issuecomment-701616922_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/983/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 789336592, "node_id": "MDU6SXNzdWU3ODkzMzY1OTI=", "number": 1195, "title": "view_name = \"query\" for the query page", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-01-19T20:21:36Z", "updated_at": "2021-01-25T04:40:08Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "It uses `view_name` of `database` at the moment which isn't as useful.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1195/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 712984738, "node_id": "MDU6SXNzdWU3MTI5ODQ3Mzg=", "number": 987, "title": "Documented HTML hooks for JavaScript plugin authors", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2020-10-01T16:10:14Z", "updated_at": "2021-01-25T04:00:03Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "In #981 I added `data-column=` attributes to the `` on the table page. These should become part of Datasette's documented API so JavaScript plugin authors can use them to derive things about the tables shown on a page (`datasette-cluster-map uses them as-of https://github.com/simonw/datasette-cluster-map/issues/18).", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/987/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 788447787, "node_id": "MDU6SXNzdWU3ODg0NDc3ODc=", "number": 1194, "title": "?_size= argument is not persisted by hidden form fields in the table filters", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6346396, "label": "Datasette 0.54"}, "comments": 3, "created_at": "2021-01-18T17:41:52Z", "updated_at": "2021-01-25T03:10:23Z", "closed_at": "2021-01-25T03:10:23Z", "author_association": "OWNER", "pull_request": null, "body": "Click \"Apply\" on https://covid-19.datasettes.com/covid/ny_times_us_counties?_size=1000&county__exact=San+Francisco&state__exact=California&_sort_desc=date#g.mark=line&g.x_column=date&g.x_type=temporal&g.y_column=cases&g.y_type=quantitative and the `?_size=1000` parameter from the URL will no longer apply on the reloaded page.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1194/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 777145954, "node_id": "MDU6SXNzdWU3NzcxNDU5NTQ=", "number": 1167, "title": "Add Prettier to contributing documentation", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6346396, "label": "Datasette 0.54"}, "comments": 3, "created_at": "2020-12-31T22:00:55Z", "updated_at": "2021-01-25T02:01:19Z", "closed_at": "2021-01-25T01:58:28Z", "author_association": "OWNER", "pull_request": null, "body": "Following #1166 - the docs at https://docs.datasette.io/en/stable/contributing.html should include a section about JavaScript, and it should document how to run Prettier.\r\n\r\nI run it in VS Code but it can be run on the command-line too:\r\n\r\n npx prettier 'datasette/static/*[!.min].js' --write\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1167/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 792958773, "node_id": "MDExOlB1bGxSZXF1ZXN0NTYwNzI1NzE0", "number": 1203, "title": "Easier way to run Prettier locally", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-01-25T01:39:06Z", "updated_at": "2021-01-25T01:41:46Z", "closed_at": "2021-01-25T01:41:46Z", "author_association": "OWNER", "pull_request": "simonw/datasette/pulls/1203", "body": "Refs #1167", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1203/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 771208009, "node_id": "MDU6SXNzdWU3NzEyMDgwMDk=", "number": 1154, "title": "Documentation for new _internal database and tables", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6346396, "label": "Datasette 0.54"}, "comments": 2, "created_at": "2020-12-18T22:34:52Z", "updated_at": "2021-01-25T00:09:22Z", "closed_at": "2021-01-25T00:08:41Z", "author_association": "OWNER", "pull_request": null, "body": "> Needs documentation, but I can wait to write that until I've tested out the feature a bit more.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1150#issuecomment-748352106_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1154/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 792931244, "node_id": "MDU6SXNzdWU3OTI5MzEyNDQ=", "number": 1202, "title": "Documentation convention for marking unstable APIs.", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6346396, "label": "Datasette 0.54"}, "comments": 2, "created_at": "2021-01-24T23:47:18Z", "updated_at": "2021-01-25T00:01:02Z", "closed_at": "2021-01-25T00:01:02Z", "author_association": "OWNER", "pull_request": null, "body": "> I'm going to document this but mark it as unstable, using a new documentation convention for marking unstable APIs.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1154#issuecomment-766462197_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1202/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 785588942, "node_id": "MDU6SXNzdWU3ODU1ODg5NDI=", "number": 1187, "title": "extra_body_script() support for script type=\"module\"", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6346396, "label": "Datasette 0.54"}, "comments": 1, "created_at": "2021-01-14T02:01:47Z", "updated_at": "2021-01-24T21:21:44Z", "closed_at": "2021-01-14T02:14:39Z", "author_association": "OWNER", "pull_request": null, "body": "Follows #1186. The `extra_body_script()` plugin hook should provide a mechanism for specifying that the script should use `