{"id": 1988525411, "node_id": "I_kwDOCGYnMM52hn1j", "number": 603, "title": "Pyhton 3.12 Bug report", "user": {"value": 1324252, "label": "constantinedev"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-11-10T22:57:48Z", "updated_at": "2023-12-08T05:10:31Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I start with new python3 verison 3.12.0\r\nAlso have the error where connect DataBase\r\n\r\n```\r\nTraceback (most recent call last):\r\n File \"/home/t/Development/python/FKPJ/ClinicSYS/run.py\", line 1, in \r\n import re, os, io, json, sqlite_utils, requests, pytz, logging\r\n File \"/home/t/.local/lib/python3.12/site-packages/sqlite_utils/__init__.py\", line 1, in \r\n from .db import Database\r\n File \"/home/t/.local/lib/python3.12/site-packages/sqlite_utils/db.py\", line 277, in \r\n class Database:\r\n File \"/home/t/.local/lib/python3.12/site-packages/sqlite_utils/db.py\", line 306, in Database\r\n filename_or_conn: Optional[Union[str, pathlib.Path, sqlite3.Connection]] = None,\r\n ^^^^^^^^^^^^^^^^^^\r\n```\r\nThis bug come from `sqlite-utils` since's v3.33.\r\nAnyone get the same ?\r\n\r\nAs well now of the resolved plan just keep the sqlite-utils version in python3.12 with v3.32.1 [tested]\r\nbut where are the sqlite3.Connection problem.... \r\n\r\nThis won't happen on python version down to 3.11[tested]\r\nJust the python3.12.0, I have test this error are come from the sqlite3 connection\r\nThe error say from `sqlite_utils` and with the sqlite3 Connection, what can I do.\r\n\r\nLet fix together.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/603/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 2007893839, "node_id": "I_kwDOCGYnMM53rgdP", "number": 605, "title": "Insert fails with `Error: Python int too large to convert to SQLite INTEGER`; can we use `NUMERIC` here?", "user": {"value": 12229877, "label": "Zac-HD"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-11-23T10:19:46Z", "updated_at": "2023-12-08T05:07:54Z", "closed_at": "2023-12-08T05:07:54Z", "author_association": "NONE", "pull_request": null, "body": "I'm currently working on a new feature for Hypothesis, where we can dump a tidy jsonlines table of all the test cases we tried - including arguments, outcomes, timings, coverage, etc. Exploring this seems like a perfect cases for `sqlite-utils` and `datasette`, but I pretty quickly ran into an integer overflow problem and don't want to recommend that experience to my users.\r\n\r\nI originally went to report this as a bug... and then found https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895581038 almost exactly matched my repro \ud83d\ude05 \r\n\r\nhttps://github.com/simonw/sqlite-utils/issues/110#issuecomment-626391063 suggests that using `NUMERIC` would avoid this overflow error, although \"If the TEXT value is a well-formed integer literal that is too large to fit in a 64-bit signed integer, it is converted to REAL.\" suggests that this would come at the cost of rounding to the nearest float value. Maybe I should just convert large integers to float before writing out my json?\r\n\r\nAfter a bit more hacking, \"manually cast large integers to float\" seems like a decent solution for my particular case, but having written it up I thought I might as well post this issue anyway - I hope it's useful feedback, and won't mind at all if you close as wontfix if it's not.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/605/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 2029908157, "node_id": "I_kwDOBm6k_c54_fC9", "number": 2214, "title": "CSV export fails for some `text` foreign key references", "user": {"value": 2874, "label": "precipice"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-12-07T05:04:34Z", "updated_at": "2023-12-07T07:36:34Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I'm starting this issue without a clear reproduction in case someone else has seen this behavior, and to use the issue as a notebook for research. \r\n\r\nI'm using Datasette with the [SWITRS](https://iswitrs.chp.ca.gov/) data set, which is a California Highway Patrol collection of traffic incident data from the past decade or so. I receive data from them in CSV and want to work with it in Datasette, then export it to CSV for mapping in Felt.com.\r\n\r\nTheir data makes extensive use of codes for incident column data (`1` for `Monday` and so on), some of it integer codes and some of it letter/text codes. The text codes are sometimes blank or `-`. During import, I'm creating lookup tables for foreign key references to make the Datasette UI presentation of the data easier to read.\r\n\r\nIf I import the data and set up the integer foreign keys, everything works fine, but if I set up the text foreign keys, CSV export starts to fail. \r\n\r\nThe foreign key configuration is as follows:\r\n\r\n```\r\n# Some tables use integer ids, like sensible tables do. Let's import them first\r\n# since we favor them.\r\n\r\nfor TABLE in DAY_OF_WEEK CHP_SHIFT POPULATION SPECIAL_COND BEAT_TYPE COLLISION_SEVERITY\r\ndo\r\n\tsqlite-utils create-table records.db $TABLE id integer name text --pk=id\r\n\tsqlite-utils insert records.db $TABLE lookup-tables/$TABLE.csv --csv\r\n\tsqlite-utils add-foreign-key records.db collisions $TABLE $TABLE id\r\n\tsqlite-utils create-index records.db collisions $TABLE\r\ndone\r\n\r\n# *Other* tables use letter keys, like they were raised by WOLVES. Let's put them\r\n# at the end of the import queue.\r\n\r\nfor TABLE in WEATHER_1 WEATHER_2 LOCATION_TYPE RAMP_INTERSECTION SIDE_OF_HWY \\\r\nPRIMARY_COLL_FACTOR PCF_CODE_OF_VIOL PCF_VIOL_CATEGORY TYPE_OF_COLLISION MVIW \\\r\nPED_ACTION ROAD_SURFACE ROAD_COND_1 ROAD_COND_2 LIGHTING CONTROL_DEVICE \\\r\nSTWD_VEHTYPE_AT_FAULT CHP_VEHTYPE_AT_FAULT PRIMARY_RAMP SECONDARY_RAMP\r\ndo\r\n\tsqlite-utils create-table records.db $TABLE key text name text --pk=key\r\n\tsqlite-utils insert records.db $TABLE lookup-tables/$TABLE.csv --csv\r\n\tsqlite-utils add-foreign-key records.db collisions $TABLE $TABLE key\r\n\tsqlite-utils create-index records.db collisions $TABLE\r\ndone\r\n```\r\n\r\nYou can see the full code and import script here: https://github.com/radical-bike-lobby/switrs-db\r\n\r\nIf I run this code and then hit the CSV export link in the Datasette interface (the simple link or the \"advanced\" dialog), export fails after a small number of CSV rows are written. I am not seeing any detailed error messages but this appears in the logging output:\r\n\r\n```\r\nINFO: 127.0.0.1:57885 - \"GET /records/collisions.csv?_facet=PRIMARY_RD&PRIMARY_RD=ASHBY+AV&_labels=on&_size=max HTTP/1.1\" 200 OK\r\nCaught this error: \r\n\r\n```\r\n\r\n(No other output follows `error:` other than a blank line.)\r\n\r\nI've stared at the rows directly after the error occurs and can't yet see what is causing the problem. I'm going to set up a development environment and see if I get any more detailed error output, and then stare more at some problematic lines to see if I can get a simple reproduction.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2214/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 2023057255, "node_id": "I_kwDOBm6k_c54lWdn", "number": 2212, "title": "Can't filter with numbers", "user": {"value": 605070, "label": "fzakaria"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-12-04T05:26:29Z", "updated_at": "2023-12-04T05:26:29Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I have a schema that uses numbers for a column (actually it's a boolean 1 or 0 but SQLite doesn't have Boolean).\r\nI can't seem to get the facet to work or even filtering on this column.\r\n\r\nMy guess is that Datasette is \"stringifying\" the number and it's not matching?\r\nExample: https://debian-sqlelf.fly.dev/debian/elf_symbols?_sort_desc=name&_facet=exported&exported=0", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2212/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 2019811176, "node_id": "I_kwDOBm6k_c54Y99o", "number": 2211, "title": "Unreachable exception handlers for `sqlite3.OperationalError`", "user": {"value": 1214074, "label": "mattparmett"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-12-01T00:50:22Z", "updated_at": "2023-12-01T00:50:22Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "There are several places where `sqlite3.OperationalError` is caught as part of an exception handler which catches multiple exceptions, but is then caught again immediately afterwards by a dedicated exception handler.\r\n\r\nBecause the exception will be caught by the first handler, the logic in the second handler is unreachable and will never be executed. If this is intended behavior, the second handler can be removed. If this is not intended, and the second handler should be the one that catches this exception, then `sqlite3.OperationalError` should be removed from the tuple of exceptions in the first handler.\r\n\r\nThis issue was found via a CodeQL query on the repository, and I've listed the occurrences found by the query below. There may be other instances of this issue in the code that were not surfaced by the query. I'd be happy to share the query if others would like to view or run it.\r\n\r\nOne example:\r\n\r\nhttps://github.com/simonw/datasette/blob/452a587e236ef642cbc6ae345b58767ea8420cb5/datasette/views/database.py#L534-L537\r\n\r\nOther instances:\r\n\r\nhttps://github.com/simonw/datasette/blob/main/datasette/views/base.py#L266-L270\r\nhttps://github.com/simonw/datasette/blob/main/datasette/views/base.py#L452-L456", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2211/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1994845152, "node_id": "I_kwDOBm6k_c525uvg", "number": 2207, "title": "ModuleNotFoundError: No module named 'click_default_group", "user": {"value": 283441, "label": "honzajavorek"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-11-15T14:04:32Z", "updated_at": "2023-11-15T14:04:32Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "No matter what I do, I'm getting this error:\r\n\r\n```\r\n$ datasette\r\nTraceback (most recent call last):\r\n File \"/Users/honza/Library/Caches/pypoetry/virtualenvs/juniorguru-Lgaxwd2n-py3.11/bin/datasette\", line 5, in \r\n from datasette.cli import cli\r\n File \"/Users/honza/Library/Caches/pypoetry/virtualenvs/juniorguru-Lgaxwd2n-py3.11/lib/python3.11/site-packages/datasette/cli.py\", line 6, in \r\n from click_default_group import DefaultGroup\r\nModuleNotFoundError: No module named 'click_default_group'\r\n```\r\n\r\nI have datasette in my dependencies like this:\r\n\r\n```toml\r\n[tool.poetry.group.dev.dependencies]\r\ndatasette = {version = \"1.0a7\", allow-prereleases = true}\r\n```\r\n\r\nI had the latest regular version (not pre-release) there originally, but the result was the same:\r\n\r\n```toml\r\n[tool.poetry.group.dev.dependencies]\r\ndatasette = \"0.64.5\"\r\n```\r\n\r\nFull pyproject.toml is at https://github.com/honzajavorek/junior.guru/ Previously datasette worked for me, but I guess something had to upgrade and now I can't even launch it.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2207/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1978603203, "node_id": "I_kwDOCGYnMM517xbD", "number": 602, "title": "`sqlite-utils transform` removes the `AUTOINCREMENT` keyword", "user": {"value": 4472046, "label": "ArsTapatun"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-11-06T08:48:43Z", "updated_at": "2023-11-06T08:48:43Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "### Context\r\n\r\nWe ran into this bug randomly, noticing that deleted `ROWID` would get reused after migrating the DB. Using `transform` to change any column in the table will also unexpectedly strip away the `AUTOINCREMENT` keyword from the primary key definition, even if it was not the transformation target.\r\n\r\n### Reproducible example\r\n\r\n**Original database**\r\n\r\n```sql\r\n$ sqlite3 test.db << EOF\r\nCREATE TABLE mytable (\r\n col1 INTEGER PRIMARY KEY AUTOINCREMENT,\r\n col2 TEXT NOT NULL\r\n)\r\nEOF\r\n\r\n$ sqlite3 test.db \".schema mytable\"\r\nCREATE TABLE mytable (\r\n col1 INTEGER PRIMARY KEY AUTOINCREMENT,\r\n col2 TEXT NOT NULL\r\n);\r\n```\r\n\r\n**Modified database after sqlite-utils**\r\n\r\n```sql\r\n$ sqlite-utils transform test.db mytable --rename col2 renamedcol2\r\n\r\n$ sqlite3 test.db \"SELECT sql FROM sqlite_master WHERE name = 'mytable';\"\r\nCREATE TABLE IF NOT EXISTS \"mytable\" (\r\n [col1] INTEGER PRIMARY KEY,\r\n [renamedcol2] TEXT NOT NULL\r\n);\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/602/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1977726056, "node_id": "I_kwDOBm6k_c514bRo", "number": 2203, "title": "custom plugin not seen as sql function", "user": {"value": 7113541, "label": "LyzardKing"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-11-05T10:30:19Z", "updated_at": "2023-11-05T10:30:19Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi, I'm not sure if this is the right repo for this issue.\r\n\r\nI'm using datasette with the parquet (to read a duckdb), and jellyfish plugins. Both work perfectly.\r\n\r\nNow I need to create a simple plugin that uses the python rouge package and returns a similarity score (similarly to how the jellyfish plugin works).\r\nIf I create a custom plugin, even the example hello_world one, copied directly from the tutorial, I get the following error:\r\n```duckdb.duckdb.CatalogException: Catalog Error: Scalar Function with name hello_world does not exist!```\r\n\r\nSince the jellyfish plugin doesn't do anything more complex, I'm wondering if there is some other kind of issue with my setup.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2203/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1553425465, "node_id": "I_kwDOCGYnMM5cl2Q5", "number": 522, "title": "Add COLUMN_TYPE_MAPPING for timedelta", "user": {"value": 81377, "label": "maport"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-01-23T16:49:54Z", "updated_at": "2023-11-04T00:49:51Z", "closed_at": "2023-11-04T00:49:51Z", "author_association": "NONE", "pull_request": null, "body": "Currently trying to create a column with Python type `datetime.timedelta` results in an error:\r\n\r\n```\r\n>>> from sqlite_utils import Database\r\n>>> db = Database(\"test.db\")\r\n>>> test_tbl = db['test']\r\n>>> test_tbl.insert({'col1': datetime.timedelta()})\r\nTraceback (most recent call last):\r\n File \"\", line 1, in \r\n File \"/usr/local/lib/python3.10/dist-packages/sqlite_utils/db.py\", line 2979, in insert\r\n return self.insert_all(\r\n File \"/usr/local/lib/python3.10/dist-packages/sqlite_utils/db.py\", line 3082, in insert_all\r\n self.create(\r\n File \"/usr/local/lib/python3.10/dist-packages/sqlite_utils/db.py\", line 1574, in create\r\n self.db.create_table(\r\n File \"/usr/local/lib/python3.10/dist-packages/sqlite_utils/db.py\", line 961, in create_table\r\n sql = self.create_table_sql(\r\n File \"/usr/local/lib/python3.10/dist-packages/sqlite_utils/db.py\", line 852, in create_table_sql\r\n column_type=COLUMN_TYPE_MAPPING[column_type],\r\nKeyError: \r\n```\r\n\r\nThe reason this would be useful is that `MySQLdb` uses `timedelta` for MySQL `TIME` columns:\r\n\r\n```\r\n>>> import MySQLdb\r\n>>> conn = MySQLdb.connect(host='database', user='user', passwd='pw')\r\n>>> csr = conn.cursor()\r\n>>> csr.execute(\"SELECT CAST('11:20' AS TIME)\")\r\n>>> tuple(csr)\r\n((datetime.timedelta(seconds=40800),),)\r\n```\r\n\r\nSo currently any attempt to convert a MySQL DB with a `TIME` column using `db-to-sqlite` will result in the above error.\r\n\r\nI was rather surprised that `MySQLdb` uses `timedelta` for `TIME` columns but I see that [this column type](https://dev.mysql.com/doc/refman/8.0/en/time.html) is intended for time intervals as well as the time of day so it makes sense. \r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/522/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 410384988, "node_id": "MDU6SXNzdWU0MTAzODQ5ODg=", "number": 411, "title": "How to pass named parameter into spatialite MakePoint() function", "user": {"value": 1055831, "label": "dazzag24"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2019-02-14T16:30:22Z", "updated_at": "2023-10-25T13:23:04Z", "closed_at": "2019-05-05T12:25:04Z", "author_association": "NONE", "pull_request": null, "body": "Hi,\r\ndatasette version: \"0.26.2\"\r\nextensions: \r\n spatialite: \"4.4.0-RC0\"\r\nsqlite version: \"3.22.0\"\r\n\r\nI have a table of airports with latitude and longitude columns. I've added spatialite (with KNN support). After creating the db using csvs-to-sqlit, I run these commands to setup the spatialite tables:\r\n\r\n```\r\nconn.execute('SELECT InitSpatialMetadata(1)')\r\n\r\nconn.execute(\"SELECT AddGeometryColumn('airports', 'point_geom', 4326, 'POINT', 2);\")\r\n\r\nconn.execute('''UPDATE airports SET point_geom = GeomFromText('POINT('||\"longitude\"||' '||\"latitude\"||')',4326);''')\r\n\r\nconn.execute(\"SELECT CreateSpatialIndex('airports', 'point_geom');\")\r\n```\r\n\r\nI'm attempting to create a canned query and have this in my metadata.json file:\r\n```\r\n\"find_airports_nearest_to_point\":{\r\n \"sql\":\"SELECT a.pos AS rank, b.id, b.name, b.country, b.latitude AS latitude, b.longitude AS longitude, a.distance / 1000.0 AS dist_km FROM KNN AS a JOIN airports AS b ON (b.rowid = a.fid) WHERE f_table_name = \\\"airports\\\" AND ref_geometry = MakePoint( :Long , :Lat ) AND max_items = 10;\"}\r\n```\r\nwhich doesn't seem to perform the templating of the name parameters correctly and I get no results. \r\n\r\nHave also tired:\r\n```\r\nMakePoint( || :Long || , || :Lat || )\r\n```\r\nwhich returns this error:\r\n```\r\nnear \"||\": syntax error\r\n```\r\n\r\nHowever I cannot seem to find the correct combination of named parameter syntax (:Lat) or sqlite concatenation operator to make it work. Any ideas if using named parameters inside functions is supported?\r\n\r\nThanks\r\nDarren", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/411/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1955676270, "node_id": "I_kwDOBm6k_c50kUBu", "number": 2201, "title": "Discord invite link is invalid", "user": {"value": 11708906, "label": "andrewsanchez"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-10-21T21:50:05Z", "updated_at": "2023-10-21T21:50:05Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "https://datasette.io/discord leads to https://discord.com/invite/ktd74dm5mw and returns the following:\r\n\r\n\"CleanShot\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2201/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1943259395, "node_id": "I_kwDOEhK-wc5z08kD", "number": 16, "title": " time data '2014-11-21T11:44:12.000Z' does not match format '%Y%m%dT%H%M%SZ'", "user": {"value": 3746270, "label": "linonetwo"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-10-14T13:24:39Z", "updated_at": "2023-10-14T13:24:39Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "\r\n```\r\nevernote-to-sqlite enex evernote.db ./\u6211\u7684\u7b14\u8bb0.enex\r\nImporting from ENEX [#####-------------------------------] 14%\r\nTraceback (most recent call last):\r\n File \"/usr/local/bin/evernote-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n ^^^^^\r\n File \"/usr/local/lib/python3.11/site-packages/click/core.py\", line 1157, in __call__\r\n return self.main(*args, **kwargs)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/lib/python3.11/site-packages/click/core.py\", line 1078, in main\r\n rv = self.invoke(ctx)\r\n ^^^^^^^^^^^^^^^^\r\n File \"/usr/local/lib/python3.11/site-packages/click/core.py\", line 1688, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/lib/python3.11/site-packages/click/core.py\", line 1434, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/lib/python3.11/site-packages/click/core.py\", line 783, in invoke\r\n return __callback(*args, **kwargs)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/lib/python3.11/site-packages/evernote_to_sqlite/cli.py\", line 31, in enex\r\n save_note(db, note)\r\n File \"/usr/local/lib/python3.11/site-packages/evernote_to_sqlite/utils.py\", line 46, in save_note\r\n \"created\": convert_datetime(created),\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/lib/python3.11/site-packages/evernote_to_sqlite/utils.py\", line 111, in convert_datetime\r\n return datetime.datetime.strptime(s, \"%Y%m%dT%H%M%SZ\").isoformat()\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/_strptime.py\", line 568, in _strptime_datetime\r\n tt, fraction, gmtoff_fraction = _strptime(data_string, format)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/_strptime.py\", line 349, in _strptime\r\n raise ValueError(\"time data %r does not match format %r\" %\r\nValueError: time data '2014-11-21T11:44:12.000Z' does not match format '%Y%m%dT%H%M%SZ'\r\n```\r\n\r\nenex is exported by evernote mac client ", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/16/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1910269679, "node_id": "I_kwDOBm6k_c5x3Gbv", "number": 2196, "title": "Discord invite link returns 401", "user": {"value": 1892194, "label": "Olshansk"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-09-24T15:16:54Z", "updated_at": "2023-10-13T00:07:08Z", "closed_at": "2023-10-12T21:54:54Z", "author_association": "NONE", "pull_request": null, "body": "I found the link to the datasette discord channel via [this query](https://github.com/search?q=repo%3Asimonw%2Fdatasette%20discord&type=code).\r\n\r\nThe following video should be self explanatory:\r\n\r\nhttps://github.com/simonw/datasette/assets/1892194/8cd33e88-bcaa-41f3-9818-ab4d589c3f02\r\n\r\n\r\nLink for reference: https://discord.com/invite/ktd74dm5mw", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2196/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1930008379, "node_id": "I_kwDOBm6k_c5zCZc7", "number": 2197, "title": "click-default-group-wheel dependency conflict", "user": {"value": 1176293, "label": "ar-jan"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2023-10-06T11:49:20Z", "updated_at": "2023-10-12T21:53:17Z", "closed_at": "2023-10-12T21:53:17Z", "author_association": "NONE", "pull_request": null, "body": "I upgraded my dependencies, then ran into this problem running `datasette inspect`:\r\n\r\n> env/lib/python3.9/site-packages/datasette/cli.py\", line 6, in \r\n> from click_default_group import DefaultGroup\r\n> ModuleNotFoundError: No module named 'click_default_group'\r\n\r\nTurns out the released version of datasette still depends on `click-default-group-wheel`, so `click-default-group` doesn't get installed/recognized:\r\n\r\n```\r\n$ virtualenv venv\r\n$ source venv/bin/activate\r\n$ pip install datasette\r\n$ pip list | grep click-default-group\r\nclick-default-group 1.2.4\r\nclick-default-group-wheel 1.2.3\r\n$ python -c \"from click_default_group import DefaultGroup\"\r\nTraceback (most recent call last):\r\n File \"\", line 1, in \r\nModuleNotFoundError: No module named 'click_default_group'\r\n$ pip install --force-reinstall click-default-group\r\n...\r\nERROR: pip's dependency resolver does not currently take into account all the packages that are installed.\r\nThis behaviour is the source of the following dependency conflicts.\r\ndatasette 0.64.4 requires click-default-group-wheel>=1.2.2, which is not installed.\r\nSuccessfully installed click-8.1.7 click-default-group-1.2.4\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2197/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1931794126, "node_id": "I_kwDOBm6k_c5zJNbO", "number": 2198, "title": "--load-extension=spatialite not working with Windows", "user": {"value": 363004, "label": "hcarter333"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-10-08T12:50:22Z", "updated_at": "2023-10-08T12:50:22Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Using each of\r\n`python -m datasette counties.db -m metadata.yml --load-extension=SpatiaLite`\r\n\r\nand \r\n\r\n`python -m datasette counties.db --load-extension=\"C:\\Windows\\System32\\mod_spatialite.dll\"`\r\n\r\nand\r\n\r\n`python -m datasette counties.db --load-extension=C:\\Windows\\System32\\mod_spatialite.dll`\r\n\r\nI got the error:\r\n\r\n```\r\n File \"C:\\Users\\m3n7es\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python311\\site-packages\\datasette\\database.py\", line 209, in in_thread\r\n self.ds._prepare_connection(conn, self.name)\r\n File \"C:\\Users\\m3n7es\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python311\\site-packages\\datasette\\app.py\", line 596, in _prepare_connection\r\n conn.execute(\"SELECT load_extension(?, ?)\", [path, entrypoint])\r\nsqlite3.OperationalError: The specified module could not be found.\r\n\r\n```\r\n\r\nI finally tried modifying the code in app.py to read:\r\n\r\n```\r\n def _prepare_connection(self, conn, database):\r\n conn.row_factory = sqlite3.Row\r\n conn.text_factory = lambda x: str(x, \"utf-8\", \"replace\")\r\n if self.sqlite_extensions:\r\n conn.enable_load_extension(True)\r\n for extension in self.sqlite_extensions:\r\n # \"extension\" is either a string path to the extension\r\n # or a 2-item tuple that specifies which entrypoint to load.\r\n #if isinstance(extension, tuple):\r\n # path, entrypoint = extension\r\n # conn.execute(\"SELECT load_extension(?, ?)\", [path, entrypoint])\r\n #else:\r\n conn.execute(\"SELECT load_extension('C:\\Windows\\System32\\mod_spatialite.dll')\")\r\n\r\n```\r\nAt which point the counties example worked. \r\n\r\nIs there a correct way to install/use the extension on Windows? My method will cause issues if there's a second extension to be used.\r\n\r\nOn an unrelated note, my next step is to figure out how to write a query across the two loaded databases supplied from the command line:\r\n`python -m datasette rm_toucans_23_10_07.db counties.db -m metadata.yml --load-extension=SpatiaLite`\r\n\r\n\r\n\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2198/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1920416843, "node_id": "I_kwDOCGYnMM5ydzxL", "number": 597, "title": "sqlite-utils insert-files should be able to convert fields", "user": {"value": 1737541, "label": "grimnight"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-09-30T22:20:47Z", "updated_at": "2023-09-30T22:20:47Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Currently using both `insert-files` and `convert` is needed in order to create sqlar files, it would be more convenient if it could be done with just one command.\r\n\r\n```shell\r\n~\r\n\u276f cat test.py\r\nimport os\r\n\r\nclass Example:\r\n def __init__(self, arg1, arg2):\r\n self.arg1 = arg1\r\n\r\n~\r\n\u276f sqlite-utils insert-files test.sqlar sqlar test.py -c name:name -c data:content -c mode:mode -c mtime:mtime -c sz:size --pk=name\r\n [####################################] 100%\r\n\r\n~\r\n\u276f sqlite-utils convert test.sqlar sqlar data \"zlib.compress(value)\" --import=zlib --where \"name = 'test.py'\"\r\n[####################################] 100%\r\n\r\n~\r\n\u276f cat test.py | sqlite-utils convert test.sqlar sqlar data \"zlib.compress(sys.stdin.buffer.read())\" --import=zlib --import=sys --where \"name = 'test.py'\" # Alternative way\r\n [####################################] 100%\r\n\r\n~\r\n\u276f sqlite3 test.sqlar \"SELECT hex(data) FROM sqlar WHERE name = 'test.py';\" | python3 -c \"import sys, zlib; sys.stdout.buffer.write(zlib.decompress(bytes.fromhex(sys.stdin.read())))\"\r\nimport os\r\n\r\nclass Example:\r\n def __init__(self, arg1, arg2):\r\n self.arg1 = arg1\r\n\r\n~\r\n\u276f rm test.py\r\n\r\n~\r\n\u276f sqlar -l test.sqlar\r\ntest.py\r\n\r\n~\r\n\u276f sqlar -x test.sqlar\r\n\r\n~\r\n\u276f cat test.py\r\nimport os\r\n\r\nclass Example:\r\n def __init__(self, arg1, arg2):\r\n self.arg1 = arg1\r\n\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/597/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1907281675, "node_id": "I_kwDOCGYnMM5xrs8L", "number": 595, "title": "Cascading DELETE not working with Table.delete(pk)", "user": {"value": 123451970, "label": "cycle-data"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-09-21T15:46:41Z", "updated_at": "2023-09-25T09:38:57Z", "closed_at": "2023-09-25T09:38:13Z", "author_association": "NONE", "pull_request": null, "body": "Hi !\r\nI noticed that when I am trying to use the delete method of the Table object,\r\nthe record get properly deleted from the table, but the cascading delete triggers on foreign keys do not activate.\r\n\r\n`self.db[\"contact\"].delete(contact_id)`\r\n\r\nI tried querying the database directly via DB Browser and the triggers work without any issue.\r\nLooked up the source code and behind the scene this method is just querying the database normally so I'm not exactly sure where this behavior comes from.\r\n\r\nThank you in advance for your time ! ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/595/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1825007061, "node_id": "I_kwDOBm6k_c5sx2XV", "number": 2123, "title": "datasette serve when invoked with --reload interprets the serve command as a file", "user": {"value": 79087, "label": "cadeef"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-07-27T19:07:22Z", "updated_at": "2023-09-18T13:02:46Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "When running `datasette serve` with the `--reload` flag, the serve command is picked up as a file argument:\r\n\r\n```\r\n$ datasette serve --reload test_db\r\nStarting monitor for PID 13574.\r\nError: Invalid value for '[FILES]...': Path 'serve' does not exist.\r\nPress ENTER or change a file to reload.\r\n```\r\n\r\nIf a 'serve' file is created it launches properly (albeit with an empty database called serve):\r\n\r\n```\r\n$ touch serve; datasette serve --reload test_db\r\nStarting monitor for PID 13628.\r\nINFO: Started server process [13628]\r\nINFO: Waiting for application startup.\r\nINFO: Application startup complete.\r\nINFO: Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)\r\n```\r\n\r\nVersion (running from HEAD on main):\r\n\r\n```\r\n$ datasette --version\r\ndatasette, version 1.0a2\r\n```\r\n\r\nThis issue appears to have existed for awhile as https://github.com/simonw/datasette/issues/1380#issuecomment-953366110 mentions the error in a different context.\r\n\r\nI'm happy to debug and land a patch if it's welcome.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2123/reactions\", \"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1899310542, "node_id": "I_kwDOBm6k_c5xNS3O", "number": 2187, "title": "Datasette for serving JSON only", "user": {"value": 19705106, "label": "geofinder"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-09-16T05:48:29Z", "updated_at": "2023-09-16T05:48:29Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi, is there any way to use datasette for serving json only without displaying webpage? I've tried to search about this in documentation but didn't get any information", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2187/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1010112818, "node_id": "I_kwDOBm6k_c48NRky", "number": 1479, "title": "Win32 \"used by another process\" error with datasette publish", "user": {"value": 76450761, "label": "kirajano"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2021-09-28T19:12:00Z", "updated_at": "2023-09-07T02:14:16Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I unfortunately was not successful to deploy to fly.io. Please see the details above of the three scenarios that I took. I am also new to datasette.\r\n\r\nFailed to deploy. Attaching logs:\r\n1. Tried with an app created via `flyctl apps create frosty-fog-8565` and the ran `datasette publish fly covid.db --app frosty-fog-8565` \r\n``` \r\nDeploying frosty-fog-8565\r\n==> Validating app configuration\r\n--> Validating app configuration done\r\nServices\r\nTCP 80/443 \u21e2 8080\r\n\r\nError error connecting to docker: An unknown error occured.\r\n\r\nTraceback (most recent call last):\r\n File \"c:\\users\\grott\\anaconda3\\lib\\runpy.py\", line 193, in _run_module_as_main\r\n \"__main__\", mod_spec)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\runpy.py\", line 85, in _run_code\r\n exec(code, run_globals)\r\n File \"C:\\Users\\grott\\Anaconda3\\Scripts\\datasette.exe\\__main__.py\", line 7, in \r\n File \"c:\\users\\grott\\anaconda3\\lib\\site-packages\\click\\core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\site-packages\\click\\core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\site-packages\\click\\core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"c:\\users\\grott\\anaconda3\\lib\\site-packages\\click\\core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"c:\\users\\grott\\anaconda3\\lib\\site-packages\\click\\core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\site-packages\\click\\core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\site-packages\\datasette_publish_fly\\__init__.py\", line 156, in fly\r\n \"--remote-only\",\r\n File \"c:\\users\\grott\\anaconda3\\lib\\contextlib.py\", line 119, in __exit__\r\n next(self.gen)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\site-packages\\datasette\\utils\\__init__.py\", line 451, in temporary_docker_directory\r\n tmp.cleanup()\r\n File \"c:\\users\\grott\\anaconda3\\lib\\tempfile.py\", line 811, in cleanup\r\n _shutil.rmtree(self.name)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\shutil.py\", line 516, in rmtree\r\n return _rmtree_unsafe(path, onerror)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\shutil.py\", line 395, in _rmtree_unsafe\r\n _rmtree_unsafe(fullname, onerror)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\shutil.py\", line 404, in _rmtree_unsafe\r\n onerror(os.rmdir, path, sys.exc_info())\r\n File \"c:\\users\\grott\\anaconda3\\lib\\shutil.py\", line 402, in _rmtree_unsafe\r\n os.rmdir(path)\r\nPermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\\\Users\\\\grott\\\\AppData\\\\Local\\\\Temp\\\\tmpgcm8cz66\\\\frosty-fog-8565'\r\n```\r\n\r\n2. Tried also with an app that gets autogenerate when running `flyctl launch`. This also generates the .toml file. Ran then `datasette publish fly covid.db --app dark-feather-168` **but different error now**\r\n```Deploying dark-feather-168\r\n==> Validating app configuration\r\n\r\nError not possible to validate configuration: server returned Post \"https://api.fly.io/graphql\": unexpected EOF\r\n\r\nTraceback (most recent call last):\r\n File \"c:\\users\\grott\\anaconda3\\lib\\runpy.py\", line 193, in _run_module_as_main \r\n \"__main__\", mod_spec)\r\n exec(code, run_globals)\r\n File \"C:\\Users\\grott\\Anaconda3\\Scripts\\datasette.exe\\__main__.py\", line 7, in \r\n File \"c:\\users\\grott\\anaconda3\\lib\\site-packages\\click\\core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\site-packages\\click\\core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\site-packages\\click\\core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"c:\\users\\grott\\anaconda3\\lib\\site-packages\\click\\core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"c:\\users\\grott\\anaconda3\\lib\\site-packages\\click\\core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\site-packages\\click\\core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\site-packages\\datasette_publish_fly\\__init__.py\", line 156, in fly\r\n \"--remote-only\",\r\n File \"c:\\users\\grott\\anaconda3\\lib\\contextlib.py\", line 119, in __exit__\r\n next(self.gen)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\site-packages\\datasette\\utils\\__init__.py\", line 451, in temporary_docker_directory\r\n tmp.cleanup()\r\n File \"c:\\users\\grott\\anaconda3\\lib\\tempfile.py\", line 811, in cleanup\r\n _shutil.rmtree(self.name)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\shutil.py\", line 516, in rmtree\r\n return _rmtree_unsafe(path, onerror)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\shutil.py\", line 395, in _rmtree_unsafe\r\n _rmtree_unsafe(fullname, onerror)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\shutil.py\", line 404, in _rmtree_unsafe\r\n onerror(os.rmdir, path, sys.exc_info())\r\n File \"c:\\users\\grott\\anaconda3\\lib\\shutil.py\", line 402, in _rmtree_unsafe\r\n os.rmdir(path)\r\nPermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\\\Users\\\\grott\\\\AppData\\\\Local\\\\Temp\\\\tmpnoyewcre\\\\dark-feather-168'\r\n```\r\n\r\nThese are also the contents of the generated **.toml file** in 2 scenario:\r\n\r\n```\r\n# fly.toml file generated for dark-feather-168 on 2021-09-28T20:35:44+02:00\r\n\r\napp = \"dark-feather-168\"\r\n\r\nkill_signal = \"SIGINT\"\r\nkill_timeout = 5\r\nprocesses = []\r\n\r\n[env]\r\n\r\n[experimental]\r\n allowed_public_ports = []\r\n auto_rollback = true\r\n\r\n[[services]]\r\n http_checks = []\r\n internal_port = 8080\r\n processes = [\"app\"]\r\n protocol = \"tcp\"\r\n script_checks = []\r\n\r\n [services.concurrency]\r\n hard_limit = 25\r\n soft_limit = 20\r\n type = \"connections\"\r\n\r\n [[services.ports]]\r\n handlers = [\"http\"]\r\n port = 80\r\n\r\n [[services.ports]]\r\n handlers = [\"tls\", \"http\"]\r\n port = 443\r\n\r\n [[services.tcp_checks]]\r\n grace_period = \"1s\"\r\n interval = \"15s\"\r\n restart_limit = 6\r\n timeout = \"2s\"\r\n```\r\n\r\n3. But also trying `datasette package covid.db` to create a local DOCKERFILE to later try to push it via `flyctl deploy` fails as well.\r\n\r\n```[+] Building 147.3s (11/11) FINISHED\r\n => [internal] load build definition from Dockerfile 0.2s \r\n => => transferring dockerfile: 396B 0.0s \r\n => [internal] load .dockerignore 0.1s \r\n => => transferring context: 2B 0.0s \r\n => [internal] load metadata for docker.io/library/python:3.8 4.7s \r\n => [auth] library/python:pull token for registry-1.docker.io 0.0s \r\n => [internal] load build context 0.1s \r\n => => transferring context: 82.37kB 0.0s \r\n => [1/5] FROM docker.io/library/python:3.8@sha256:530de807b46a11734e2587a784573c12c5034f2f14025f838589e6c0e3 108.3s \r\n => => resolve docker.io/library/python:3.8@sha256:530de807b46a11734e2587a784573c12c5034f2f14025f838589e6c0e3b5 0.0s \r\n => => sha256:56182bcdf4d4283aa1f46944b4ef7ac881e28b4d5526720a4e9ba03a4730846a 2.22kB / 2.22kB 0.0s \r\n => => sha256:955615a668ce169f8a1443fc6b6e6215f43fe0babfb4790712a2d3171f34d366 54.93MB / 54.93MB 21.6s \r\n => => sha256:911ea9f2bd51e53a455297e0631e18a72a86d7e2c8e1807176e80f991bde5d64 10.87MB / 10.87MB 15.5s \r\n => => sha256:530de807b46a11734e2587a784573c12c5034f2f14025f838589e6c0e3b5c5b6 1.86kB / 1.86kB 0.0s \r\n => => sha256:ff08f08727e50193dcf499afc30594c47e70cc96f6fcfd1a01240524624264d0 8.65kB / 8.65kB 0.0s \r\n => => sha256:2756ef5f69a5190f4308619e0f446d95f5515eef4a814dbad0bcebbbbc7b25a8 5.15MB / 5.15MB 6.4s \r\n => => sha256:27b0a22ee906271a6ce9ddd1754fdd7d3b59078e0b57b6cc054c7ed7ac301587 54.57MB / 54.57MB 37.7s \r\n => => sha256:8584d51a9262f9a3a436dea09ba40fa50f85802018f9bd299eee1bf538481077 196.45MB / 196.45MB 82.3s \r\n => => sha256:524774b7d3638702fe9ae0ea3fcfb81b027dfd75cc2fc14f0119e764b9543d58 6.29MB / 6.29MB 26.6s \r\n => => extracting sha256:955615a668ce169f8a1443fc6b6e6215f43fe0babfb4790712a2d3171f34d366 5.4s \r\n => => sha256:9460f6b75036e38367e2f27bb15e85777c5d6cd52ad168741c9566186415aa26 16.81MB / 16.81MB 40.5s \r\n => => extracting sha256:2756ef5f69a5190f4308619e0f446d95f5515eef4a814dbad0bcebbbbc7b25a8 0.6s \r\n => => extracting sha256:911ea9f2bd51e53a455297e0631e18a72a86d7e2c8e1807176e80f991bde5d64 0.6s \r\n => => sha256:9bc548096c181514aa1253966a330134d939496027f92f57ab376cd236eb280b 232B / 232B 40.1s \r\n => => extracting sha256:27b0a22ee906271a6ce9ddd1754fdd7d3b59078e0b57b6cc054c7ed7ac301587 5.8s \r\n => => sha256:1d87379b86b89fd3b8bb1621128f00c8f962756e6aaaed264ec38db733273543 2.35MB / 2.35MB 41.8s \r\n => => extracting sha256:8584d51a9262f9a3a436dea09ba40fa50f85802018f9bd299eee1bf538481077 18.8s \r\n => => extracting sha256:524774b7d3638702fe9ae0ea3fcfb81b027dfd75cc2fc14f0119e764b9543d58 1.2s \r\n => => extracting sha256:9460f6b75036e38367e2f27bb15e85777c5d6cd52ad168741c9566186415aa26 2.9s \r\n => => extracting sha256:9bc548096c181514aa1253966a330134d939496027f92f57ab376cd236eb280b 0.0s \r\n => => extracting sha256:1d87379b86b89fd3b8bb1621128f00c8f962756e6aaaed264ec38db733273543 0.8s \r\n => [2/5] COPY . /app 2.3s \r\n => [3/5] WORKDIR /app 0.2s \r\n => [4/5] RUN pip install -U datasette 26.9s \r\n => [5/5] RUN datasette inspect covid.db --inspect-file inspect-data.json 3.1s\r\n => exporting to image 1.2s \r\n => => exporting layers 1.2s \r\n => => writing image sha256:b5db0c205cd3454c21fbb00ecf6043f261540bcf91c2dfc36d418f1a23a75d7a 0.0s\r\n\r\nUse 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them\r\nTraceback (most recent call last):\r\n \"__main__\", mod_spec)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\runpy.py\", line 85, in _run_code\r\n exec(code, run_globals)\r\n File \"C:\\Users\\grott\\Anaconda3\\Scripts\\datasette.exe\\__main__.py\", line 7, in \r\n File \"c:\\users\\grott\\anaconda3\\lib\\site-packages\\click\\core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\site-packages\\click\\core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\site-packages\\click\\core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"c:\\users\\grott\\anaconda3\\lib\\site-packages\\click\\core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\site-packages\\click\\core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\site-packages\\datasette\\cli.py\", line 283, in package\r\n call(args)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\contextlib.py\", line 119, in __exit__\r\n next(self.gen)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\site-packages\\datasette\\utils\\__init__.py\", line 451, in temporary_docker_directory\r\n tmp.cleanup()\r\n File \"c:\\users\\grott\\anaconda3\\lib\\tempfile.py\", line 811, in cleanup\r\n _shutil.rmtree(self.name)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\shutil.py\", line 516, in rmtree\r\n return _rmtree_unsafe(path, onerror)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\shutil.py\", line 395, in _rmtree_unsafe\r\n _rmtree_unsafe(fullname, onerror)\r\n File \"c:\\users\\grott\\anaconda3\\lib\\shutil.py\", line 404, in _rmtree_unsafe\r\n onerror(os.rmdir, path, sys.exc_info())\r\n File \"c:\\users\\grott\\anaconda3\\lib\\shutil.py\", line 402, in _rmtree_unsafe\r\n os.rmdir(path)\r\nPermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\\\Users\\\\grott\\\\AppData\\\\Local\\\\Temp\\\\tmpkb27qid3\\\\datasette'```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1479/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1871935751, "node_id": "I_kwDOD079W85vk3kH", "number": 40, "title": " ImportError: cannot import name 'formatargspec' from 'inspect'", "user": {"value": 36752421, "label": "hosslikw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-08-29T15:36:31Z", "updated_at": "2023-08-31T03:18:07Z", "closed_at": "2023-08-31T03:18:06Z", "author_association": "NONE", "pull_request": null, "body": "I get the following error when running \"pip3 install dogsheep-photos\"\r\n\" from inspect import ismethod, isclass, formatargspec\r\n ImportError: cannot import name 'formatargspec' from 'inspect' (/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/inspect.py). Did you mean: 'formatargvalues'?\"\r\n \r\nPython 3.12.0rc1\r\nsqlite 3.43.0\r\ndatasette, version 0.64.3", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/40/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 787098146, "node_id": "MDU6SXNzdWU3ODcwOTgxNDY=", "number": 1190, "title": "`datasette publish upload` mechanism for uploading databases to an existing Datasette instance", "user": {"value": 1024355, "label": "tomershvueli"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-01-15T18:18:42Z", "updated_at": "2023-08-30T22:16:39Z", "closed_at": "2023-08-30T22:16:38Z", "author_association": "NONE", "pull_request": null, "body": "If I have a self-hosted instance of Datasette up and running, I'd like to be able to the use the CLI to publish databases to that instance, not only Google or Heroku. Ideally there'd be a `url` parameter or something similar to which one could point the publish command to their instance. ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1190/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 814595021, "node_id": "MDU6SXNzdWU4MTQ1OTUwMjE=", "number": 1241, "title": "Share button for copying current URL", "user": {"value": 7107523, "label": "Kabouik"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2021-02-23T15:55:40Z", "updated_at": "2023-08-24T20:09:52Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I use datasette in an `iframe` inside another HTML file that contains other ways to represent my data (mostly leaflets maps built with R on summarized data), and the datasette `iframe` is a tab in that page. \r\n\r\nThis particular use prevents users to access the full URLs of their datasette views and queries, which is a shame because the way datasette handles URLs to make every view or query easy to share is awesome. I know how to get the URL from the context menu of my browser, but I don't think many visitors would do it or even notice that datasette uses permalinks for pretty much every action they do. Would it be possible to add a \"Share link\" button to the interface, either in datasette itself or in a plugin?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1241/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1858228057, "node_id": "I_kwDOBm6k_c5uwk9Z", "number": 2147, "title": "Plugin hook for database queries that are run", "user": {"value": 18899, "label": "jackowayed"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2023-08-20T18:43:50Z", "updated_at": "2023-08-24T03:54:35Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I'm interested in making a plugin that saves every query that gets run to a table in the database. (I know about datasette-query-history but thought it would be good to have a server-side option.)\r\n\r\nAs far as I can tell reading the docs, there isn't really a hook setup to allow this.\r\n\r\nMaybe I could hack it with some of the hooks that are passed requests, but that doesn't seem good.\r\n\r\nI'm a little surprised this isn't possible, so I thought I would open an issue and see if that's a deeply considered decision or just \"haven't needed it yet.\" I'm potentially interested in implementing the hook if the latter.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2147/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1754174496, "node_id": "I_kwDOCGYnMM5ojpQg", "number": 558, "title": "Ability to define unique columns when creating a table", "user": {"value": 1910303, "label": "aguinane"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-06-13T06:56:19Z", "updated_at": "2023-08-18T01:06:03Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "When creating a new table, it would be good to have an option to set unique columns similar to how not_null is set.\r\n\r\n```python\r\nfrom sqlite_utils import Database\r\n\r\ncolumns = {\"mRID\": str, \"name\": str}\r\ndb = Database(\"example.db\")\r\ndb[\"ExampleTable\"].create(columns, pk=\"mRID\", not_null=[\"mRID\"], if_not_exists=True)\r\ndb[\"ExampleTable\"].create_index([\"mRID\"], unique=True, if_not_exists=True)\r\n```\r\n\r\nSo something like this would add the UNIQUE flag to the table definition. \r\n\r\n```python\r\ndb[\"ExampleTable\"].create(columns, pk=\"mRID\", not_null=[\"mRID\"], unique=[\"mRID\"], if_not_exists=True)\r\n```\r\n\r\n```sql\r\nCREATE TABLE ExampleTable (\r\n mRID TEXT PRIMARY KEY\r\n NOT NULL\r\n UNIQUE,\r\n name TEXT\r\n);\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/558/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1838266862, "node_id": "I_kwDOBm6k_c5tkbnu", "number": 2126, "title": "Permissions in metadata.yml / metadata.json", "user": {"value": 36199671, "label": "ctsrc"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2023-08-06T16:24:10Z", "updated_at": "2023-08-11T05:52:30Z", "closed_at": "2023-08-11T05:52:29Z", "author_association": "NONE", "pull_request": null, "body": "https://docs.datasette.io/en/latest/authentication.html#other-permissions-in-metadata says the following:\r\n\r\n> For all other permissions, you can use one or more \"permissions\" blocks in your metadata.\r\n\r\n> To grant access to the permissions debug tool to all signed in users you can grant permissions-debug to any actor with an id matching the wildcard * by adding this a the root of your metadata:\r\n\r\n```yaml\r\npermissions:\r\n debug-menu:\r\n id: '*'\r\n```\r\n\r\nI tried this.\r\n\r\nMy `metadata.yml` file looks like:\r\n\r\n```yaml\r\npermissions:\r\n debug-menu:\r\n id: '*'\r\n permissions-debug:\r\n id: '*'\r\nplugins:\r\n datasette-auth-passwords:\r\n myuser_password_hash:\r\n $env: \"PASSWORD_HASH_MYUSER\"\r\n```\r\n\r\nAnd then I run\r\n\r\n```zsh\r\ndatasette -m metadata.yml tiddlywiki.db --root\r\n```\r\n\r\nAnd I open a session for the \"root\" user of datasette with the link given.\r\n\r\nI open a private browser session and log in as \"myuser\" from http://127.0.0.1:8001/-/login\r\n\r\nThen I check http://127.0.0.1:8001/-/actor which confirms that I am logged in as the \"myuser\" actor\r\n\r\n```json\r\n{\r\n \"actor\": {\r\n \"id\": \"myuser\"\r\n }\r\n}\r\n```\r\n\r\nIn the session where I am logged in as \"myuser\" I then try to go to http://127.0.0.1:8001/-/permissions\r\n\r\nBut all I get there as the logged in user \"myuser\" is\r\n\r\n> Forbidden\r\n>\r\n> Permission denied\r\n\r\nAnd then if I check the http://127.0.0.1:8001/-/permissions as the datasette \"root\" user from another browser session, I see:\r\n\r\n> permissions-debug checked at 2023-08-06T16:22:58.997841 \u2717 (used default)\r\n>\r\n> Actor: {\"id\": \"myuser\"}\r\n\r\nIt seems that in spite of having tried to give the `permissions-debug` permission to the \"myuser\" user in my `metadata.yml` file, datasette does not agree that \"myuser\" has permission `permissions-debug`..\r\n\r\nWhat do I need to do differently so that my \"myuser\" user is able to access http://127.0.0.1:8001/-/permissions ?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2126/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 476852861, "node_id": "MDU6SXNzdWU0NzY4NTI4NjE=", "number": 568, "title": "Add database_color as a configurable option", "user": {"value": 50906992, "label": "LBHELewis"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-08-05T13:14:45Z", "updated_at": "2023-08-11T05:19:42Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "This would be really useful as it would allow us to tie in with colour schemes.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/568/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1841501975, "node_id": "I_kwDOBm6k_c5twxcX", "number": 2133, "title": "[feature request]`datasette install plugins.json` options", "user": {"value": 54462, "label": "HaveF"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2023-08-08T15:06:50Z", "updated_at": "2023-08-10T00:31:24Z", "closed_at": "2023-08-09T22:04:46Z", "author_association": "NONE", "pull_request": null, "body": "Hi, simon \u2764\ufe0f \r\n\r\n`datasette plugins --all > plugins.json` could generate all plugins info. On another machine, it would be great to install all plugins just by `datasette install plugins.json`", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2133/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1839344979, "node_id": "I_kwDOCGYnMM5toi1T", "number": 582, "title": "Handling CSV/file input that contains NUL bytes", "user": {"value": 1448859, "label": "betatim"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-08-07T12:24:14Z", "updated_at": "2023-08-07T12:24:14Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I was using sqlite-utils to create a DB from a CSV and it turns out the CSV contains a NUL byte.\r\n\r\nWhen the processing reaches the line that contains the NUL an exception is raised.\r\n\r\nI'm wondering if there is something that can be done in `sqlite-utils` to say \"skip lines with encoding errors\" or some such. I think it isn't super straightforward though as the exception comes from inside the `csv` module that does all the parsing.\r\n\r\nConcretely the file is the `KernelVersions.csv` from https://www.kaggle.com/datasets/kaggle/meta-kaggle\r\n\r\nThis is the command and output:\r\n```\r\n$ sqlite-utils insert --csv kaggle.db kaggle KernelVersions.csv\r\n [------------------------------------] 0%\r\n [#####################---------------] 60% 00:04:24Traceback (most recent call last):\r\n File \"/home/foobar/miniconda/envs/meta-kaggle/bin/sqlite-utils\", line 10, in \r\n sys.exit(cli())\r\n File \"/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/click/core.py\", line 1128, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/click/core.py\", line 1053, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/click/core.py\", line 1659, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/click/core.py\", line 1395, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/click/core.py\", line 754, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/sqlite_utils/cli.py\", line 1223, in insert\r\n insert_upsert_implementation(\r\n File \"/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/sqlite_utils/cli.py\", line 1085, in insert_upsert_implementation\r\n db[table].insert_all(\r\n File \"/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/sqlite_utils/db.py\", line 3198, in insert_all\r\n chunk = list(chunk)\r\n File \"/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/sqlite_utils/db.py\", line 3742, in fix_square_braces\r\n for record in records:\r\n File \"/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/sqlite_utils/cli.py\", line 1071, in \r\n docs = (decode_base64_values(doc) for doc in docs)\r\n File \"/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/sqlite_utils/cli.py\", line 1068, in \r\n docs = (verify_is_dict(doc) for doc in docs)\r\n File \"/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/sqlite_utils/cli.py\", line 1003, in \r\n docs = (dict(zip(headers, row)) for row in reader)\r\n_csv.Error: line contains NUL\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/582/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1824457306, "node_id": "I_kwDOBm6k_c5svwJa", "number": 2122, "title": "Parameters on canned queries: fixed or query-generated list?", "user": {"value": 1563881, "label": "meowcat"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-07-27T14:07:07Z", "updated_at": "2023-07-27T14:07:07Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi,\r\n\r\ncurrently parameters in canned queries are just text fields. It would be cool to have one of the options below. Would you accept a PR doing something in this direction? (Possibly this could even work as a plugin.)\r\n\r\n* adding facets, which would work like facets on tables or views, giving a list of selectable options (and leaving parameters as is)\r\n* making it possible to provide a query which returns selectable values for a parameter, e.g.\r\n``` \r\ncalendar_entries_current_instrument:\r\n sql: | \r\n select * from calendar_entries \r\n where \r\n DTEND_UNIX > UNIXEPOCH() and\r\n DTSTART_UNIX < UNIXEPOCH() + :days *24*60*60 and\r\n current = 1 and\r\n MACHINE = :instrument\r\n order by\r\n DTSTART_UNIX\r\n params:\r\n days: \r\n sql: \"SELECT VALUE FROM generate_series(1, 30, 1)\"\r\n # this obviously requires the corresponding sqlite extension\r\n instrument:\r\n sql: \"SELECT DISTINCT MACHINE FROM calendar_entries\"\r\n```\r\n* making it possible to provide a fixed list of parameters\r\n``` \r\ncalendar_entries_current_instrument:\r\n sql: | \r\n select * from calendar_entries \r\n where \r\n DTEND_UNIX > UNIXEPOCH() and\r\n DTSTART_UNIX < UNIXEPOCH() + :days *24*60*60 and\r\n current = 1 and\r\n MACHINE = :instrument\r\n order by\r\n DTSTART_UNIX\r\n params:\r\n days: \r\n values: [1, 2, 3, 5, 10, 20, 30]\r\n instrument:\r\n values: [supermachine, crappymachine, boringmachine]\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2122/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1822918995, "node_id": "I_kwDOCGYnMM5sp4lT", "number": 580, "title": "Add way to export to a csv file using the Python library", "user": {"value": 44324811, "label": "kevinlinxc"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-07-26T18:09:26Z", "updated_at": "2023-07-26T18:09:26Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "According to the documentation, we can make a csv output using the CLI tool, but not the Python library. Could we have the latter?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/580/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1817281557, "node_id": "I_kwDOC8SPRc5sUYQV", "number": 37, "title": "cannot use jinja filters in display?", "user": {"value": 10352819, "label": "rprimet"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-07-23T20:09:54Z", "updated_at": "2023-07-23T20:18:27Z", "closed_at": "2023-07-23T20:18:26Z", "author_association": "NONE", "pull_request": null, "body": "Hi, I'm trying to have a display function in Dogsheep's `config.yml` that includes something like this:\r\n\r\n``` \r\n

\r\n {{ display.title }}\r\n (source)\r\n

\r\n

{{ display.snippet|safe }}

\r\n```\r\n\r\nUnfortunately, rendering fails with a message 'urls is undefined'.\r\n\r\nThe same happens if I'm trying to build a row URL manually, using filters like `quote_plus` (as my keys are URLs).\r\n\r\nAny hints?\r\n\r\nThanks!", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/37/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1816830546, "node_id": "I_kwDODEm0Qs5sSqJS", "number": 73, "title": "Twitter v1 API shutdown", "user": {"value": 6341745, "label": "david-perez"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-07-22T16:57:41Z", "updated_at": "2023-07-22T16:57:41Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I've been using this project reliably over the past two years to periodically download my liked tweets, but unfortunately since 19th July I get:\r\n\r\n```\r\n[2023-07-19 21:00:04.937536] File \"/home/pi/code/liked-tweets/lib/python3.7/site-packages/twitter_to_sqlite/utils.py\", line 202, in fetch_timeline\r\n[2023-07-19 21:00:04.937606] raise Exception(str(tweets[\"errors\"]))\r\n[2023-07-19 21:00:04.937678] Exception: [{'message': 'You currently have access to a subset of Twitter API v2 endpoints and limited v1.1 endpoints (e.g. media post, oauth) only. If you need access to this endpoint, you may need a different access level. You can learn more here: https://developer.twitter.com/en/portal/product', 'code': 453}]\r\n```\r\n\r\nIt appears like Twitter has now shut down their v1 endpoints, which is rather gracious of them, considering they [announced they'd be deprecated on 29th April](https://twittercommunity.com/t/reminder-to-migrate-to-the-new-free-basic-or-enterprise-plans-of-the-twitter-api/189737).\r\n\r\nUnfortunately [retrieving likes using the v2 API](https://developer.twitter.com/en/docs/twitter-api/tweets/likes/introduction) is not part of their [free plan](https://developer.twitter.com/en/portal/products). In fact, with the free plan one can only post and delete tweets and retrieve information about oneself.\r\n\r\nSo I'm afraid this is the end of this very nice project. It was very useful, thank you!\r\n", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/73/reactions\", \"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 1}", "draft": null, "state_reason": null} {"id": 1811824307, "node_id": "I_kwDOBm6k_c5r_j6z", "number": 2105, "title": "When reverse proxying datasette with nginx an URL element gets erronously added", "user": {"value": 2235371, "label": "aki-k"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2023-07-19T12:16:53Z", "updated_at": "2023-07-21T21:17:09Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I use this nginx config:\r\n```\r\n location /datasette-llm {\r\n return 302 /datasette-llm/;\r\n }\r\n\r\n location /datasette-llm/ {\r\n proxy_set_header Upgrade $http_upgrade;\r\n proxy_set_header Connection \"Upgrade\";\r\n proxy_http_version 1.1;\r\n proxy_set_header X-Real-IP $remote_addr;\r\n proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\r\n proxy_set_header X-Forwarded-Proto https;\r\n proxy_set_header X-Forwarded-Host $http_host;\r\n proxy_set_header Host $host;\r\n proxy_max_temp_file_size 0;\r\n proxy_pass http://127.0.0.1:8001/datasette-llm/;\r\n proxy_redirect http:// https://;\r\n proxy_buffering off;\r\n proxy_request_buffering off;\r\n proxy_set_header Origin '';\r\n client_max_body_size 0;\r\n auth_basic \"datasette-llm\";\r\n auth_basic_user_file /etc/nginx/custom-userdb;\r\n }\r\n```\r\nThen I start datasette with this command:\r\n```\r\ndatasette serve --setting base_url /datasette-llm/ $(llm logs path)\r\n```\r\nEverything else works right, except the links in \"This data as json, CSV\".\r\nThey get an extra URL element \"datasette-llm\" like this:\r\n\r\nhttps://192.168.1.3:5432/datasette-llm/datasette-llm/logs.json?sql=select+*+from+_llm_migrations\r\n\r\nhttps://192.168.1.3:5432/datasette-llm/datasette-llm/logs.csv?sql=select+*+from+_llm_migrations&_size=max\r\n\r\nWhen I remove that extra \"datasette-llm\" from the URL, those links work too.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2105/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 771608692, "node_id": "MDU6SXNzdWU3NzE2MDg2OTI=", "number": 14, "title": "UNIQUE constraint failed: workouts.id", "user": {"value": 1234956, "label": "n8henrie"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2020-12-20T15:11:20Z", "updated_at": "2023-07-10T14:46:52Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I'm getting an error on my initial attempt to import data:\r\n\r\n```console\r\n$ healthkit-to-sqlite 20201119\\ healthkit\\ export.zip healthkit.db\r\nImporting from HealthKit [###################################-] 98% 00:00:01\r\nTraceback (most recent call last):\r\n File \"venv/bin/healthkit-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"venv/lib/python3.9/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"venv/lib/python3.9/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"venv/lib/python3.9/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"venv/lib/python3.9/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"venv/lib/python3.9/site-packages/healthkit_to_sqlite/cli.py\", line 57, in cli\r\n convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)\r\n File \"venv/lib/python3.9/site-packages/healthkit_to_sqlite/utils.py\", line 34, in convert_xml_to_sqlite\r\n workout_to_db(el, db, zipfile)\r\n File \"venv/lib/python3.9/site-packages/healthkit_to_sqlite/utils.py\", line 57, in workout_to_db\r\n pk = db[\"workouts\"].insert(record, alter=True, hash_id=\"id\").last_pk\r\n File \"venv/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1660, in insert\r\n return self.insert_all(\r\n File \"venv/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1778, in insert_all\r\n self.insert_chunk(\r\n File \"venv/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1588, in insert_chunk\r\n result = self.db.execute(query, params)\r\n File \"venv/lib/python3.9/site-packages/sqlite_utils/db.py\", line 213, in execute\r\n return self.conn.execute(sql, parameters)\r\nsqlite3.IntegrityError: UNIQUE constraint failed: workouts.id\r\n```", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1795219865, "node_id": "I_kwDOCGYnMM5rAOGZ", "number": 566, "title": "`--no-headers` doesn't work on most formats", "user": {"value": 33625, "label": "zellyn"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-07-09T03:43:36Z", "updated_at": "2023-07-09T04:13:35Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Version 3.33\r\n\r\n```\r\nsqlite-utils query library.db 'select asin from audible' --fmt plain --no-headers | head -3\r\nasin\r\n0062804006\r\n0062891421\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/566/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1794097871, "node_id": "I_kwDOBm6k_c5q78LP", "number": 2095, "title": "Introduce \"dark mode\" CSS", "user": {"value": 3315059, "label": "jamietanna"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-07-07T19:15:58Z", "updated_at": "2023-07-07T19:15:58Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Using [the CSS media query `prefers-color-scheme`](https://developer.mozilla.org/en-US/docs/Web/CSS/@media/prefers-color-scheme) we can provide a dark-mode version of Datasette", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2095/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 810618495, "node_id": "MDU6SXNzdWU4MTA2MTg0OTU=", "number": 235, "title": "Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified", "user": {"value": 6913891, "label": "kristomi"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 18, "created_at": "2021-02-17T23:33:23Z", "updated_at": "2023-06-26T01:47:01Z", "closed_at": "2023-06-25T23:25:53Z", "author_association": "NONE", "pull_request": null, "body": "Thanks for what seems like a truly great suite of libraries. I wanted to try out Datasette, but never got more than half way through your YouTube video with the SF tree dataset. Whenever I try to extract a column, I get a `sqlite3.OperationalError: table sqlite_master may not be modified` error from Python. This snippet reproduces the error on my system, Python 3.9.1 and sqlite-utils 3.5 on an M1 Macbook Pro running in rosetta mode:\r\n```\r\ncurl \"https://data.nasa.gov/resource/y77d-th95.json\" | \\\r\n sqlite-utils insert meteorites.db meteorites - --pk=id\r\nsqlite-utils extract meteorites.db meteorites recclass\r\n```\r\n\r\nI have tried googling the problem, but all I've found is that this *might* be a problem with the sqlite3 database running in defensive mode, but I definitely can't know for sure. Does the problem seem familiar to you?\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/235/reactions\", \"total_count\": 3, \"+1\": 3, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1762180409, "node_id": "I_kwDOBm6k_c5pCL05", "number": 2085, "title": "Interactive row selection in Datasette ", "user": {"value": 24938923, "label": "learning4life"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-06-18T08:29:45Z", "updated_at": "2023-06-18T08:31:23Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Simon did a excellent [prototype](https://til.simonwillison.net/datasette/row-selection-prototype) of an interactive row selection in Datasette.\r\n\r\nI hope this [functionality](https://camo.githubusercontent.com/3d4a0f31fb6a27fd279f809af5b53dc3b76faa63c7721e228951c5252b645a77/68747470733a2f2f7374617469632e73696d6f6e77696c6c69736f6e2e6e65742f7374617469632f323032332f6461746173657474652d7069636b65722e676966) can be turned into a Datasette plugin.\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2085/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1761613778, "node_id": "I_kwDOBm6k_c5pABfS", "number": 2084, "title": "Support facets for columns that contain timestamps", "user": {"value": 19492893, "label": "devxpy"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-06-17T03:33:54Z", "updated_at": "2023-06-17T03:33:54Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "\r\nDjango has this very nice filter for datetime fields -\r\n\r\n\"image\"\r\n\r\nIt would be nice to have something similar to facet by a field that contains a timestamp in datasette too - Which doesn't seem to do anything with timestamps right now...\r\n\r\n\"image\"\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2084/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1383646615, "node_id": "I_kwDOCGYnMM5SeMWX", "number": 491, "title": "Ability to merge databases and tables", "user": {"value": 8904453, "label": "sgraaf"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2022-09-23T11:10:55Z", "updated_at": "2023-06-14T22:14:24Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi! Let me firstly say that I am a big fan of your work -- I follow your tweets and blog posts with great interest \ud83d\ude04.\r\n\r\nNow onto the matter at hand: I think it would be great if `sqlite-utils` included a `merge` or `combine` command, with the purpose of combining different SQLite databases into a single SQLite database. This way, the newly \"merged\" database would contain all differently named tables contained in the databases to be merged as-is, as well a concatenation of all tables of the same name.\r\n\r\nThis could look something like this:\r\n\r\n```bash\r\nsqlite-utils merge cats.db dogs.db > animals.db\r\n```\r\n\r\nI imagine this is rather straightforward if all databases involved in the merge contain differently named tables (i.e. no chance of conflicts), but things get slightly more complicated if two or more of the databases to be merged contain tables with the same name. Not only do you have to \"do something\" with the primary key(s), but these tables could also simply have different schemas (and therefore be incompatible for concatenation to begin with).\r\n\r\nAnyhow, I would love your thoughts on this, and, if you are open to it, work together on the design and implementation!", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/491/reactions\", \"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1733198948, "node_id": "I_kwDOCGYnMM5nToRk", "number": 555, "title": "Filter table by a large bunch of ids", "user": {"value": 10843208, "label": "redraw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-05-31T00:29:51Z", "updated_at": "2023-06-14T22:01:57Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi! this might be a question related to both SQLite & sqlite-utils, and you might be more experienced with them.\r\n\r\nI have a large bunch of ids, and I'm wondering which is the best way to query them in terms of performance, and simplicity if possible.\r\n\r\nThe naive approach would be something like `select * from table where rowid in (?, ?, ?...)` but that wouldn't scale if ids are >1k.\r\n\r\nAnother approach might be creating a temp table, or in-memory db table, insert all ids in that table and then join with the target one.\r\n\r\nI failed to attach an in-memory db both using sqlite-utils, and plain sql's execute(), so my closest approach is something like,\r\n\r\n```python\r\ndef filter_existing_video_ids(video_ids):\r\n db = get_db() # contains a \"videos\" table\r\n db.execute(\"CREATE TEMPORARY TABLE IF NOT EXISTS tmp (video_id TEXT NOT NULL PRIMARY KEY)\")\r\n db[\"tmp\"].insert_all([{\"video_id\": video_id} for video_id in video_ids])\r\n for row in db[\"tmp\"].rows_where(\"video_id not in (select video_id from videos)\"):\r\n yield row[\"video_id\"]\r\n db[\"tmp\"].drop()\r\n```\r\n\r\nThat kinda worked, I couldn't find an option in sqlite-utils's `create_table()` to tell it's a temporary table. Also, `tmp` table is not dropped finally, neither using `.drop()` despite being created with the keyword `TEMPORARY`. I believe it should be automatically dropped after connection/session ends though I read.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/555/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1751214236, "node_id": "I_kwDOC8SPRc5oYWic", "number": 36, "title": "Getting sqlite_master may not be modified when creating dogsheep index", "user": {"value": 8711912, "label": "khushmeeet"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-06-11T03:21:53Z", "updated_at": "2023-06-11T03:21:53Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "When creating a `dogsheep` index from `config.yml` file on pocket.db (created using pocket-to-sqlite), I am getting this error\r\n\r\n```\r\nTraceback (most recent call last):\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/bin/dogsheep-beta\", line 8, in \r\n sys.exit(cli())\r\n ^^^^^\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py\", line 1130, in __call__\r\n return self.main(*args, **kwargs)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py\", line 1055, in main\r\n rv = self.invoke(ctx)\r\n ^^^^^^^^^^^^^^^^\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py\", line 1657, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py\", line 760, in invoke\r\n return __callback(*args, **kwargs)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/dogsheep_beta/cli.py\", line 36, in index\r\n run_indexer(\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/dogsheep_beta/utils.py\", line 32, in run_indexer\r\n ensure_table_and_indexes(db, tokenize)\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/dogsheep_beta/utils.py\", line 91, in ensure_table_and_indexes\r\n table.add_foreign_key(*fk)\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/sqlite_utils/db.py\", line 2155, in add_foreign_key\r\n self.db.add_foreign_keys([(self.name, column, other_table, other_column)])\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/sqlite_utils/db.py\", line 1116, in add_foreign_keys\r\n cursor.execute(\r\nsqlite3.OperationalError: table sqlite_master may not be modified\r\n```\r\n\r\nCommand I ran to get this error\r\n```\r\ndogsheep-beta index pocket.db config.yml\r\n```\r\n\r\nDogsheep version\r\n```\r\ndogsheep-beta, version 0.10.2\r\n```\r\n\r\nPython version \r\n```\r\nPython 3.11.2\r\n```", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/36/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1727478903, "node_id": "I_kwDOBm6k_c5m9zx3", "number": 2081, "title": "Update Endpoints defined in metadata throws 403 Forbidden after a while", "user": {"value": 15085007, "label": "cutmasta-kun"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-05-26T11:52:30Z", "updated_at": "2023-05-26T11:52:30Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hello. I expose an endpoint to update `tasks`:\r\n```\r\n{\r\n \"title\": \"My Datasette Instance\",\r\n \"databases\": {\r\n \"tasks\": {\r\n \"queries\": {\r\n \"update_task\": {\r\n \"sql\": \"UPDATE tasks SET status = :status, result = :result, systemMessage = :systemMessage WHERE queueID = :queueID\",\r\n \"write\": true,\r\n \"on_success_message\": \"Task updated\",\r\n \"on_success_redirect\": \"/tasks/tasks.json\",\r\n \"on_error_message\": \"Task update failed\",\r\n \"on_error_redirect\": \"/tasks.json\",\r\n \"params\": [\"queueID\", \"taskData\", \"status\", \"result\", \"systemMessage\"]\r\n }\r\n }\r\n }\r\n }\r\n}\r\n```\r\n\r\nThis works really well! But after a while, the Datasette Instanz answers with **403 Forbidden**.\r\nI have to delete the database and recreate it in order to work again.\r\n\r\nAny help here? (\u00b4\u3002\uff3f\u3002\uff40)", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2081/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1720096994, "node_id": "I_kwDOCGYnMM5mhpji", "number": 554, "title": "`IndexError` when doing `.insert(..., pk='id')` after `insert_all`", "user": {"value": 1231935, "label": "xavdid"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-05-22T17:13:02Z", "updated_at": "2023-05-22T17:18:33Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I believe this is related to https://github.com/simonw/sqlite-utils/issues/98.\r\n\r\nWhen `pk` is specified by table A's `insert` call, it throws an index error if a different table has written a row with a higher rowid than exists in the first table. Here's a basic example:\r\n\r\n```py\r\nfrom sqlite_utils import Database\r\n\r\n\r\ndef test_pk_for_insert(fresh_db):\r\n user = {\"id\": \"abc\", \"name\": \"david\"}\r\n\r\n fresh_db[\"users\"].insert(user, pk=\"id\")\r\n\r\n fresh_db[\"comments\"].insert_all(\r\n [\r\n {\"id\": \"def\", \"text\": \"ok\"},\r\n {\"id\": \"ghi\", \"text\": \"great\"},\r\n ],\r\n )\r\n\r\n fresh_db[\"users\"].insert(\r\n user,\r\n ignore=True,\r\n # BUG: when specifying pk on the second insert call \r\n # db.py goes into a block it doesn't expect and we get the error\r\n pk=\"id\",\r\n )\r\n\r\n\r\nif __name__ == \"__main__\":\r\n db = Database(\"bug.db\")\r\n if db[\"users\"].exists():\r\n raise ValueError(\r\n \"bug only shows on a new database - remove bug.db before running the script\"\r\n )\r\n test_pk_for_insert(db)\r\n```\r\n\r\nThe error is:\r\n\r\n```py\r\n File \"/Users/david/projects/reddit-to-sqlite/.venv/lib/python3.11/site-packages/sqlite_utils/db.py\", line 2960, in insert_chunk\r\n row = list(self.rows_where(\"rowid = ?\", [self.last_rowid]))[0]\r\n ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^\r\nIndexError: list index out of range\r\n```\r\n\r\nThe issue is in this block: \r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/2747257a3334d55e890b40ec58fada57ae8cfbfd/sqlite_utils/db.py#L2954-L2958\r\n\r\nrelevant locals are:\r\n\r\n- `pk`: `'id'`\r\n- `result.lastrowid`: `2`\r\n\r\nWhat's most interesting is the comment `# self.last_rowid will be 0 if a \"INSERT OR IGNORE\" happened`, which doesn't seem to be the case here. ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/554/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1279144769, "node_id": "I_kwDOCGYnMM5MPjNB", "number": 448, "title": "Reading rows from a file => AttributeError: '_io.StringIO' object has no attribute 'readinto'", "user": {"value": 236907, "label": "mungewell"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2022-06-21T21:48:27Z", "updated_at": "2023-05-08T22:01:00Z", "closed_at": "2023-05-08T22:01:00Z", "author_association": "NONE", "pull_request": null, "body": "Attempting to run the example given here (without extra bracket ;-):\r\nhttps://sqlite-utils.datasette.io/en/stable/python-api.html#reading-rows-from-a-file\r\n```\r\nfrom sqlite_utils.utils import rows_from_file\r\nimport io\r\n\r\nrows, format = rows_from_file(io.StringIO(\"id,name\\n1,Cleo\"))\r\nprint(list(rows), format)\r\n# Outputs [{'id': '1', 'name': 'Cleo'}] Format.CSV\r\n```\r\n\r\nGives error\r\n```\r\n>\"c:\\Program Files\\Python37\\python.exe\" test2.py\r\nTraceback (most recent call last):\r\n File \"test2.py\", line 4, in \r\n rows, format = rows_from_file(io.StringIO(\"id,name\\n1,Cleo\"))\r\n File \"C:\\Users\\swood\\Downloads\\sqlite-utils-main-20220621\\sqlite-utils-main\\sqlite_utils\\utils.py\", line 300, in rows_from_file\r\n first_bytes = buffered.peek(2048).strip()\r\nAttributeError: '_io.StringIO' object has no attribute 'readinto'\r\n```\r\n\r\nI am running Python on Windows.\r\n```\r\n>\"c:\\Program Files\\Python37\\python.exe\"\r\nPython 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32\r\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/448/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1465194249, "node_id": "I_kwDOCGYnMM5XVRcJ", "number": 514, "title": "upsert of new row with check constraints fails", "user": {"value": 193185, "label": "cldellow"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2022-11-26T16:12:23Z", "updated_at": "2023-05-08T21:50:52Z", "closed_at": "2023-05-08T21:50:51Z", "author_association": "NONE", "pull_request": null, "body": "(I originally opened this in https://github.com/simonw/datasette-insert/issues/20, but I see that that library depends on sqlite-utils)\r\n\r\nIn the case of a new row, upsert first adds the row, specifying only its pkeys: https://github.com/simonw/sqlite-utils/blob/965ca0d5f5bffe06cc02cd7741344d1ddddf9d56/sqlite_utils/db.py#L2783-L2787\r\n\r\nThis means that a table with NON NULL (or other constraint) columns that aren't part of the pkey can't have new rows upserted.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/514/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1465194930, "node_id": "PR_kwDOCGYnMM5DvZxa", "number": 515, "title": "upsert new rows with constraints, fixes #514", "user": {"value": 193185, "label": "cldellow"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-11-26T16:15:21Z", "updated_at": "2023-05-08T21:27:11Z", "closed_at": "2023-05-08T21:27:10Z", "author_association": "NONE", "pull_request": "simonw/sqlite-utils/pulls/515", "body": "This fixes #514 by making the initial insert for upserts include all columns, so that new rows can be added to tables with non-pkey columns that have constraints.\r\n\r\n(aside: I'm not a python programmer. `pip`? `pipenv`? `venv`? These are mystical incantations to me. The process to set up this repo for local development and testing was _so easy_. Thank you for the excellent contributing documentation!)\r\n\r\n\r\n----\r\n:books: Documentation preview :books:: https://sqlite-utils--515.org.readthedocs.build/en/515/\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/515/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1432377191, "node_id": "I_kwDOCGYnMM5VYFdn", "number": 509, "title": "`sqlite-utils transform` breaks DEFAULT string values and STRFTIME()", "user": {"value": 2199875, "label": "kennysong"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-11-02T02:32:23Z", "updated_at": "2023-05-08T21:13:38Z", "closed_at": "2023-05-08T21:13:38Z", "author_association": "NONE", "pull_request": null, "body": "Very nice library! Our team found sqlite-utils through @simonw's [comment on the \"Simple declarative schema migration for SQLite\" article](https://news.ycombinator.com/item?id=31249823), and we were excited to use it, but unfortunately `sqlite-utils transform` seems to break our DB. \r\n\r\nRunning `sqlite-utils transform` to modify a column mangles their DEFAULT values:\r\n\r\n- Default string values are wrapped in extra single quotes\r\n- Function expressions such as [`STRFTIME()`](https://www.sqlite.org/lang_datefunc.html) are turned into strings!\r\n\r\n------\r\n\r\nHere are steps to reproduce:\r\n\r\n**Original database**\r\n\r\n```\r\n$ sqlite3 test.db << EOF\r\nCREATE TABLE mytable (\r\n col1 TEXT DEFAULT 'foo',\r\n col2 TEXT DEFAULT (STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW'))\r\n)\r\nEOF\r\n\r\n$ sqlite3 test.db \"SELECT sql FROM sqlite_master WHERE name = 'mytable';\"\r\nCREATE TABLE mytable (\r\n col1 TEXT DEFAULT 'foo',\r\n col2 TEXT DEFAULT (STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW'))\r\n)\r\n```\r\n\r\n**Modified database after sqlite-utils**\r\n\r\n```\r\n$ sqlite3 test.db \"INSERT INTO mytable DEFAULT VALUES; SELECT * FROM mytable;\"\r\nfoo|2022-11-02 02:26:58.038\r\n\r\n$ sqlite-utils transform test.db mytable --rename col1 renamedcol1\r\n\r\n$ sqlite3 test.db \"SELECT sql FROM sqlite_master WHERE name = 'mytable';\"\r\nCREATE TABLE \"mytable\" (\r\n [renamedcol1] TEXT DEFAULT '''foo''',\r\n [col2] TEXT DEFAULT 'STRFTIME(''%Y-%m-%d %H:%M:%f'', ''NOW'')'\r\n)\r\n\r\n$ sqlite3 test.db \"INSERT INTO mytable DEFAULT VALUES; SELECT * FROM mytable;\"\r\nfoo|2022-11-02 02:26:58.038\r\n'foo'|STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')\r\n```\r\n\r\n(Related: #336)", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/509/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1620254998, "node_id": "I_kwDOCGYnMM5gkyEW", "number": 532, "title": "Show more information when JSON can't be imported with sqlite-utils insert", "user": {"value": 83080728, "label": "voltagex"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-03-12T06:41:44Z", "updated_at": "2023-05-08T20:32:16Z", "closed_at": "2023-05-08T20:32:02Z", "author_association": "NONE", "pull_request": null, "body": "I am currently trying to import the [JSON export of my data from Discord](https://support.discord.com/hc/en-us/articles/360004027692-Requesting-a-Copy-of-your-Data), specifically `activity/reporting/events-*.json`\r\n\r\n```\r\nsqlite-utils.exe insert test.db reporting events-2023-00000-of-00001.json\r\n [###################################-] 99% 00:00:00\r\nError: Invalid JSON - use --csv for CSV or --tsv for TSV files\r\n```\r\n\r\nPlease show more information as to *why* this is invalid, if possible.\r\n\r\nI am using version 3.30 with Python 3.10 on Windows 11.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/532/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1695428235, "node_id": "I_kwDOCGYnMM5lDi6L", "number": 538, "title": "`table.upsert_all` fails to write rows when `not_null` is present", "user": {"value": 1231935, "label": "xavdid"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2023-05-04T07:30:38Z", "updated_at": "2023-05-08T20:06:35Z", "closed_at": "2023-05-08T19:27:02Z", "author_association": "NONE", "pull_request": null, "body": "I found an odd bug today, where calls to `table.upsert_all` don't write rows if you include the `not_null` kwarg.\r\n\r\n## Repro Example\r\n\r\n```py\r\nfrom sqlite_utils import Database\r\n\r\ndb = Database(\"upsert-test.db\")\r\n\r\ndb[\"comments\"].upsert_all(\r\n [{\"id\": 1, \"name\": \"david\"}],\r\n pk=\"id\",\r\n not_null=[\"name\"],\r\n)\r\n\r\nassert list(db[\"comments\"].rows) # err!\r\n```\r\n\r\nThe schema is correctly created:\r\n\r\n```sql\r\nCREATE TABLE [comments] (\r\n [id] INTEGER PRIMARY KEY,\r\n [name] TEXT NOT NULL\r\n)\r\n```\r\n\r\nBut no rows are created. Removing either the `not_null` kwargs works as expected, as does an `insert_all` call.\r\n\r\n## Version Info\r\n\r\n- Python: `3.11.0`\r\n- sqlite-utils: `3.30`\r\n- sqlite: `3.39.5 2022-10-14`", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/538/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1622640374, "node_id": "I_kwDOCGYnMM5gt4b2", "number": 534, "title": " ResourceWarning: unclosed file", "user": {"value": 1244826, "label": "djhenderson"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-03-14T03:02:18Z", "updated_at": "2023-05-08T19:56:29Z", "closed_at": "2023-05-08T19:56:29Z", "author_association": "NONE", "pull_request": null, "body": "Issuing either\r\n\r\n```\r\npy -Wdefault -m sqlite_utils insert dogs.db dogs dogs0.csv --csv\r\n [#############-----------------------] 36%\r\n [####################################] 100%C:\\Users\\Doug\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\sqlite_utils\\cli.py:1187: ResourceWarning: unclosed file <_io.TextIOWrapper name='dogs0.csv' encoding='utf-8-sig'>\r\n insert_upsert_implementation(\r\nResourceWarning: Enable tracemalloc to get the object allocation traceback\r\n```\r\nor\r\n```\r\nset pythonwarnings=default\r\nsqlite-utils insert dogs.db dogs dogs0.csv --csv\r\n [#############-----------------------] 36%\r\n [####################################] 100%C:\\Users\\Doug\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\sqlite_utils\\cli.py:1187: ResourceWarning: unclosed file <_io.TextIOWrapper name='dogs0.csv' encoding='utf-8-sig'>\r\n insert_upsert_implementation(\r\nResourceWarning: Enable tracemalloc to get the object allocation traceback\r\n```\r\n\r\nexhibits a ResourceWarning indicating that the CSV file being loaded is not closed.\r\n\r\nsqlite-utils --version\r\nsqlite-utils, version 3.30\r\npy --version\r\nPython 3.11.2\r\nWindows Version 10.0.19045 Build 19045\r\nSQLite version 3.41.0 2023-02-21 18:09:37\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/534/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1698865182, "node_id": "I_kwDOBm6k_c5lQqAe", "number": 2069, "title": "[BUG] Cannot insert new data to deployed instance", "user": {"value": 31861128, "label": "yqlbu"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-05-07T02:59:42Z", "updated_at": "2023-05-07T03:17:35Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "## Summary\r\n\r\nRecently, I deployed an instance of datasette to Vercel with the following plugins:\r\n\r\n- datasette-auth-tokens\r\n- datasette-insert\r\n\r\nWith the above plugins, I was able to insert new data to local sqlite db. However, when it comes to the deployment on Vercel, things behave differently. I observed some errors from the logs console on Vercel:\r\n\r\n```console\r\nFile \"/var/task/datasette/database.py\", line 179, in _execute_writes\r\nconn = self.connect(write=True)\r\nFile \"/var/task/datasette/database.py\", line 93, in connect\r\nassert not (write and not self.is_mutable)\r\nAssertionError\r\n``` \r\n\r\n\"image\"\r\n\r\nI think it is a potential bug.\r\n\r\n## Reproduce\r\n\r\n
metadata.json\r\n
\r\n\r\n```json\r\n{\r\n \"plugins\": {\r\n \"datasette-insert\": {\r\n \"allow\": {\r\n \"id\": \"*\"\r\n }\r\n },\r\n \"datasette-auth-tokens\": {\r\n \"tokens\": [\r\n {\r\n \"token\": {\r\n \"$env\": \"INSERT_TOKEN\"\r\n },\r\n \"actor\": {\r\n \"id\": \"repeater\"\r\n }\r\n }\r\n ],\r\n \"param\": \"_auth_token\"\r\n }\r\n }\r\n}\r\n```\r\n\r\n
\r\n\r\n
commands\r\n
\r\n\r\n```bash\r\n# deploy\r\ndatasette publish vercel remote.db \\\r\n --project=repeater-bot-sqlite \\\r\n --metadata metadata.json \\\r\n --install datasette-auth-tokens \\\r\n --install datasette-insert \\\r\n --vercel-json=vercel.json\r\n\r\n# test insert\r\ncat fixtures/dogs.json | curl --request POST -d @- -H \"Authorization: Bearer \" \\\r\n 'https://repeater-bot-sqlite.vercel.app/-/insert/remote/dogs?pk=id'\r\n```\r\n\r\n
\r\n\r\n
logs\r\n
\r\n\r\n```console\r\nTraceback (most recent call last):\r\nFile \"/var/task/datasette/app.py\", line 1354, in route_path\r\nresponse = await view(request, send)\r\nFile \"/var/task/datasette/app.py\", line 1500, in async_view_fn\r\nresponse = await async_call_with_supported_arguments(\r\nFile \"/var/task/datasette/utils/__init__.py\", line 1005, in async_call_with_supported_arguments\r\nreturn await fn(*call_with)\r\nFile \"/var/task/datasette_insert/__init__.py\", line 14, in insert_or_upsert\r\nresponse = await insert_or_upsert_implementation(request, datasette)\r\nFile \"/var/task/datasette_insert/__init__.py\", line 91, in insert_or_upsert_implementation\r\ntable_count = await db.execute_write_fn(write_in_thread, block=True)\r\nFile \"/var/task/datasette/database.py\", line 167, in execute_write_fn\r\nraise result\r\nFile \"/var/task/datasette/database.py\", line 179, in _execute_writes\r\nconn = self.connect(write=True)\r\nFile \"/var/task/datasette/database.py\", line 93, in connect\r\nassert not (write and not self.is_mutable)\r\nAssertionError\r\n```\r\n\r\n
", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2069/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1690765434, "node_id": "I_kwDOBm6k_c5kxwh6", "number": 2067, "title": "Litestream-restored db: errors on 3.11 and 3.10.8; but works on py3.10.7 and 3.10.6", "user": {"value": 39538958, "label": "justmars"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-05-01T12:42:28Z", "updated_at": "2023-05-03T00:16:03Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi! Wondering if this issue is limited to my local system or if it affects others as well. \r\n\r\nIt seems like 3.11 errors out on a \"litestream-restored\" database. On further investigation, it also appears to conk out on 3.10.8 but works on 3.10.7 and 3.10.6.\r\n\r\nTo demo issue I created a test database, replicated it to an aws s3 bucket, then restored the same under various .pyenv-versioned shells where I test whether I can read the database via the sqlite3 cli.\r\n\r\n```sh\r\n# create new shell with 3.11.3\r\nlitestream restore -o data/db.sqlite s3://mytestbucketxx/db\r\nsqlite3 data/db.sqlite \r\n# SQLite version 3.41.2 2023-03-22 11:56:21\r\n# Enter \".help\" for usage hints.\r\n# sqlite> .tables\r\n# _litestream_lock _litestream_seq movie \r\n# sqlite> \r\n```\r\n\r\nHowever this get me an `OperationalError` when reading via datasette:\r\n\r\n
\r\nError on 3.11.3 and 3.10.8\r\n\r\n```sh\r\ndatasette data/db.sqlite\r\n```\r\n\r\n```console\r\n/tester/.venv/lib/python3.11/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API\r\n warnings.warn(\"pkg_resources is deprecated as an API\", DeprecationWarning)\r\nTraceback (most recent call last):\r\n File \"/tester/.venv/bin/datasette\", line 8, in \r\n sys.exit(cli())\r\n ^^^^^\r\n File \"/tester/.venv/lib/python3.11/site-packages/click/core.py\", line 1130, in __call__\r\n return self.main(*args, **kwargs)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/tester/.venv/lib/python3.11/site-packages/click/core.py\", line 1055, in main\r\n rv = self.invoke(ctx)\r\n ^^^^^^^^^^^^^^^^\r\n File \"/tester/.venv/lib/python3.11/site-packages/click/core.py\", line 1657, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/tester/.venv/lib/python3.11/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/tester/.venv/lib/python3.11/site-packages/click/core.py\", line 760, in invoke\r\n return __callback(*args, **kwargs)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/tester/.venv/lib/python3.11/site-packages/datasette/cli.py\", line 143, in wrapped\r\n return fn(*args, **kwargs)\r\n ^^^^^^^^^^^^^^^^^^^\r\n File \"/tester/.venv/lib/python3.11/site-packages/datasette/cli.py\", line 615, in serve\r\n asyncio.get_event_loop().run_until_complete(check_databases(ds))\r\n File \"/Users/mv/.pyenv/versions/3.11.3/lib/python3.11/asyncio/base_events.py\", line 653, in run_until_complete\r\n return future.result()\r\n ^^^^^^^^^^^^^^^\r\n File \"/tester/.venv/lib/python3.11/site-packages/datasette/cli.py\", line 660, in check_databases\r\n await database.execute_fn(check_connection)\r\n File \"/tester/.venv/lib/python3.11/site-packages/datasette/database.py\", line 213, in execute_fn\r\n return await asyncio.get_event_loop().run_in_executor(\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/mv/.pyenv/versions/3.11.3/lib/python3.11/concurrent/futures/thread.py\", line 58, in run\r\n result = self.fn(*self.args, **self.kwargs)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/tester/.venv/lib/python3.11/site-packages/datasette/database.py\", line 211, in in_thread\r\n return fn(conn)\r\n ^^^^^^^^\r\n File \"/tester/.venv/lib/python3.11/site-packages/datasette/utils/__init__.py\", line 951, in check_connection\r\n for r in conn.execute(\r\n ^^^^^^^^^^^^^\r\nsqlite3.OperationalError: unable to open database file\r\n```\r\n\r\n
\r\n\r\n\r\n
\r\nWorks on 3.10.7, 3.10.6\r\n\r\n```sh\r\n# create new shell with 3.10.7 / 3.10.6\r\nlitestream restore -o data/db.sqlite s3://mytestbucketxx/db\r\ndatasette data/db.sqlite\r\n# ...\r\n# INFO: Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)\r\n```\r\n\r\n
\r\n\r\nIn both scenarios, the only dependencies were the pinned python version and the latest Datasette version 0.64.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2067/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1665053646, "node_id": "I_kwDOBm6k_c5jPrPO", "number": 2059, "title": "\"Deceptive site ahead\" alert on Heroku deployment", "user": {"value": 1186275, "label": "mtdukes"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-04-12T18:34:51Z", "updated_at": "2023-04-13T01:13:01Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I deployed a fairly basic instance of Datasette (`datasette-auth-passwords` is the only plugin) using Heroku. The deployed URL now gives a \"Deceptive site ahead\" warning to users.\r\n\r\nIs there way around this? Maybe a way to add ownership verification [through Google's search console](https://search.google.com/search-console/welcome)? ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2059/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1373210675, "node_id": "I_kwDODD6af85R2Ygz", "number": 13, "title": "fails before generating views. ERR: table sqlite_master may not be modified", "user": {"value": 116795, "label": "pax"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-09-14T15:41:50Z", "updated_at": "2023-04-11T03:46:17Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "generates checkins.db but seems to fail before generating views \r\n\r\nnote: it worked on an Ubuntu WSL but fails on macOS 12.5.1\r\n\r\nlater edit: I suspect this is a problem with my local set-up, `dogsheep-beta index` also throws the same error\r\n\r\nfull error:\r\n\r\n Importing 2591 checkins [###################################-] 98% 00:00:00\r\n Traceback (most recent call last):\r\n File \"/Users/pax/devbox/envAll/bin/swarm-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/swarm_to_sqlite/cli.py\", line 77, in cli\r\n ensure_foreign_keys(db)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/swarm_to_sqlite/utils.py\", line 145, in ensure_foreign_keys\r\n db[fk.table].add_foreign_key(fk.column, fk.other_table, fk.other_column)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/sqlite_utils/db.py\", line 2123, in add_foreign_key\r\n self.db.add_foreign_keys([(self.name, column, other_table, other_column)])\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/sqlite_utils/db.py\", line 1086, in add_foreign_keys\r\n cursor.execute(\r\n sqlite3.OperationalError: table sqlite_master may not be modified", "repo": {"value": 205429375, "label": "swarm-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1650981564, "node_id": "I_kwDOJHON9s5iZ_q8", "number": 12, "title": "Error running pytest", "user": {"value": 14314871, "label": "amlestin"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-04-02T15:02:36Z", "updated_at": "2023-04-02T15:07:10Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "`______________________________________________________ ERROR collecting tests/test_apple_notes_to_sqlite.py _______________________________________________________\r\nImportError while importing test module '/Users/lol/development/apple-notes-to-sqlite/tests/test_apple_notes_to_sqlite.py'.\r\nHint: make sure your test modules/packages have valid Python names.\r\nTraceback:\r\n/opt/homebrew/Cellar/python@3.9/3.9.16/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module\r\n return _bootstrap._gcd_import(name[level:], package, level)\r\ntests/test_apple_notes_to_sqlite.py:2: in \r\n from apple_notes_to_sqlite.cli import cli, COUNT_SCRIPT, FOLDERS_SCRIPT\r\nE ModuleNotFoundError: No module named 'apple_notes_to_sqlite'`\r\n\r\nSolution:\r\nThis is likely a PYTHONPATH issue due to having pytest installed both globally and in the venv. We can guarantee the tests run by adding the current directory to sys.path automatically using\r\n\r\n`python -m pytest`\r\n\r\nThe alternative is to activate the venv, install pytest, deactivate, then activate the venv again (https://stackoverflow.com/questions/35045038/how-do-i-use-pytest-with-virtualenv)", "repo": {"value": 611552758, "label": "apple-notes-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/12/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 907795562, "node_id": "MDU6SXNzdWU5MDc3OTU1NjI=", "number": 265, "title": "Using enable_fts before search term", "user": {"value": 36287, "label": "prabhur"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-06-01T01:43:34Z", "updated_at": "2023-04-01T17:27:18Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Many thanks for the sqlite-utils suite of utilities. Has made my life much much easier. \r\nI used this to create a table and enable FTS. All works fine. The datasette utility detects FTS and shows a text box. Searching for a term using that interface works well.\r\n\r\nHowever, when I start to use features by following https://www.sqlite.org/fts5.html section **\"3. Full-text Query Syntax\"** I seem to run into issues that I suspect is due to `escape_fts` wrapper function.\r\n\r\nAs an example, if i search for the term `\"^\u0b95\u0bc1\u0b95\u0bc8\" `on the text box in datasette it produces 140 results. However, when i tweak the query produced by datasette to not use \"escape_fts\" it produces 5 results.\r\n\r\nSimilarly, when I try to restrict the search to a single column in FTS using a spec like `{title : ^\u0b95\u0bc1\u0b95\u0bc8}` it returns no rows. The same thing pulls results when used without `escape_fts`. The text in the table is in Tamil language and the search term is a Tamil word.\r\n\r\n```\r\n ...\r\n where\r\n posts_fts match escape_fts(:search)\r\n```\r\nvs\r\n\r\n```\r\n ...\r\n where\r\n posts_fts match (:search)\r\n```\r\n\r\nAny ideas why? How can I get the benefits of both escaping as well as utilizing different facets of providing / controlling search terms? Thanks.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/265/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 702386948, "node_id": "MDU6SXNzdWU3MDIzODY5NDg=", "number": 159, "title": ".delete_where() does not auto-commit (unlike .insert() or .upsert())", "user": {"value": 11712349, "label": "spdkils"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2020-09-16T01:55:52Z", "updated_at": "2023-04-01T17:21:05Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "When you use the delete_where() function on a table, it never commits....\r\n\r\nIs that intentional?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/159/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1531991339, "node_id": "I_kwDOBm6k_c5bUFUr", "number": 1989, "title": "Suggestion: Hiding columns", "user": {"value": 116795, "label": "pax"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2023-01-13T09:33:32Z", "updated_at": "2023-03-31T06:18:05Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "As there's the possibility of [hiding tables](https://docs.datasette.io/en/stable/metadata.html#hiding-tables) - I've run into the **need of hiding specific columns** - data that's either not relevant for public or can't be shown due to privacy reasons. ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1989/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1515883470, "node_id": "I_kwDOC8tyDs5aWovO", "number": 24, "title": "DOC: xml.etree.ElementTree.ParseError due to healthkit version 12 ", "user": {"value": 6231413, "label": "mmngreco"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-01-01T23:00:38Z", "updated_at": "2023-03-30T10:17:31Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi @simonw \r\n\r\nI hope you find this issue ok, the idea is provide some documentation to other users like me about how to solve this problem and save some time.\r\n\r\nFollowing the instructions from the `README.md` I've faced this error:\r\n\r\n```bash\r\n(venv) mgreco@pop-os apple-health master* (23:44|0s)\r\n$ healthkit-to-sqlite apple_health_export/export.xml healthkit.db --xml\r\nImporting from HealthKit [------------------------------------] 0%\r\nTraceback (most recent call last):\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/bin/healthkit-to-sqlite\", line 33, in \r\n sys.exit(load_entry_point('healthkit-to-sqlite', 'console_scripts', 'healthkit-to-sqlite')())\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py\", line 1130, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py\", line 1055, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py\", line 760, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/home/mgreco/github/mmngreco/apple-health/.deps/healthkit-to-sqlite/healthkit_to_sqlite/cli.py\", line 57, in cli\r\n convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)\r\n File \"/home/mgreco/github/mmngreco/apple-health/.deps/healthkit-to-sqlite/healthkit_to_sqlite/utils.py\", line 25, in convert_xml_to_sqlite\r\n for tag, el in find_all_tags(\r\n File \"/home/mgreco/github/mmngreco/apple-health/.deps/healthkit-to-sqlite/healthkit_to_sqlite/utils.py\", line 12, in find_all_tags\r\n for event, el in parser.read_events():\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/xml/etree/ElementTree.py\", line 1324, in read_events\r\n raise event\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/xml/etree/ElementTree.py\", line 1296, in feed\r\n self._parser.feed(data)\r\nxml.etree.ElementTree.ParseError: syntax error: line 156, column 0\r\n```\r\n\r\nSo, after debugging and searching on internet I found this useful link: https://discussions.apple.com/thread/254202523 (etresoft, the real hero). Which basically says that the xml given by the health app (healthkit version 12) has some bugs but fortunately, they can be solved with a couple of commads:\r\n\r\n1. Uncompress the zip and move the new folder where `export.xml` is.\r\n1. Create a `patch.txt` with the following content\r\n\r\n ```diff\r\n --- export.xml\t2022-09-18 15:17:09.000000000 -0400\r\n +++ export-fixed.xml\t2022-09-18 16:37:08.000000000 -0400\r\n @@ -15,6 +15,7 @@\r\n HKCharacteristicTypeIdentifierBiologicalSex CDATA #REQUIRED\r\n HKCharacteristicTypeIdentifierBloodType CDATA #REQUIRED\r\n HKCharacteristicTypeIdentifierFitzpatrickSkinType CDATA #REQUIRED\r\n + HKCharacteristicTypeIdentifierCardioFitnessMedicationsUse CDATA #IMPLIED\r\n >\r\n \r\n \r\n -\r\n +\r\n \r\n -\r\n +\r\n \r\n \r\n \r\n \r\n \r\n - device CDATA #IMPLIED\r\n -\r\n -\r\n ->\r\n ]>\r\n \r\n \r\n ```\r\n1. Apply the path with the command: `patch < patch.txt`\r\n1. Fix endDates with the command `sed 's/startDate/endDate/2' export.xml > export-fixed.xml`\r\n1. Try again `healthkit-to-sqlite export-fixed.xml healthkit.db --xml`", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/24/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1646068413, "node_id": "I_kwDOBm6k_c5iHQK9", "number": 2048, "title": "Test failures encountered while packaging for GNU Guix", "user": {"value": 8332263, "label": "Apteryks"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-03-29T15:36:54Z", "updated_at": "2023-03-29T15:36:54Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hello,\r\n\r\nWhile reviewing a packaged submitted to Guix to add `datasette`, the test suite produces the following errors:\r\n```\r\n=================================== FAILURES ===================================\r\n_________________________ test_row_strange_table_name __________________________\r\n[gw21] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python\r\n\r\napp_client = \r\n\r\n def test_row_strange_table_name(app_client):\r\n response = app_client.get(\r\n \"/fixtures/table~2Fwith~2Fslashes~2Ecsv/3.json?_shape=objects\"\r\n )\r\n> assert response.status == 200\r\nE assert 400 == 200\r\nE + where 400 = .status\r\n\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/tests/test_api.py:701: AssertionError\r\n----------------------------- Captured stderr call -----------------------------\r\nERROR: conn=, sql = 'select rowid, * from [table%7E2Fwith%7E2Fslashes%7E2Ecsv] where \"rowid\"=:p0', params = {'p0': '3'}: no such table: table%7E2Fwith%7E2Fslashes%7E2Ecsv\r\n_______________ test_database_page_for_database_with_dot_in_name _______________\r\n[gw15] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python\r\n\r\napp_client_with_dot = \r\n\r\n def test_database_page_for_database_with_dot_in_name(app_client_with_dot):\r\n response = app_client_with_dot.get(\"/fixtures~2Edot.json\")\r\n> assert response.status == 200\r\nE assert 302 == 200\r\nE + where 302 = .status\r\n\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/tests/test_api.py:633: AssertionError\r\n___________________ test_tilde_encoded_database_names[fo%o] ____________________\r\n[gw6] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python\r\n\r\ndb_name = 'fo%o'\r\n\r\n @pytest.mark.asyncio\r\n @pytest.mark.parametrize(\"db_name\", (\"foo\", r\"fo%o\", \"f~/c.d\"))\r\n async def test_tilde_encoded_database_names(db_name):\r\n ds = Datasette()\r\n ds.add_memory_database(db_name)\r\n response = await ds.client.get(\"/.json\")\r\n assert db_name in response.json().keys()\r\n path = response.json()[db_name][\"path\"]\r\n # And the JSON for that database\r\n response2 = await ds.client.get(path + \".json\")\r\n> assert response2.status_code == 200\r\nE assert 302 == 200\r\nE + where 302 = .status_code\r\n\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/tests/test_api.py:983: AssertionError\r\n__________________ test_tilde_encoded_database_names[f~/c.d] ___________________\r\n[gw7] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python\r\n\r\ndb_name = 'f~/c.d'\r\n\r\n @pytest.mark.asyncio\r\n @pytest.mark.parametrize(\"db_name\", (\"foo\", r\"fo%o\", \"f~/c.d\"))\r\n async def test_tilde_encoded_database_names(db_name):\r\n ds = Datasette()\r\n ds.add_memory_database(db_name)\r\n response = await ds.client.get(\"/.json\")\r\n assert db_name in response.json().keys()\r\n path = response.json()[db_name][\"path\"]\r\n # And the JSON for that database\r\n response2 = await ds.client.get(path + \".json\")\r\n> assert response2.status_code == 200\r\nE assert 302 == 200\r\nE + where 302 = .status_code\r\n\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/tests/test_api.py:983: AssertionError\r\n______________ test_database_with_space_in_name[/searchable.json] ______________\r\n[gw21] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python\r\n\r\napp_client_two_attached_databases = \r\npath = '/searchable.json'\r\n\r\n @pytest.mark.parametrize(\r\n \"path\",\r\n (\r\n \"/\",\r\n \".json\",\r\n \"/searchable\",\r\n \"/searchable.json\",\r\n \"/searchable_view\",\r\n \"/searchable_view.json\",\r\n ),\r\n )\r\n def test_database_with_space_in_name(app_client_two_attached_databases, path):\r\n> response = app_client_two_attached_databases.get(\r\n \"/extra~20database\" + path, follow_redirects=True\r\n )\r\n\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/tests/test_api.py:920: \r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\n/gnu/store/mcclmphjgbrgpa0v037a4nlq336482g8-python-asgiref-3.4.1/lib/python3.9/site-packages/asgiref/sync.py:223: in __call__\r\n return call_result.result()\r\n/gnu/store/65i3nhcwmz0p8rqbg48gaavyky4g4hwk-python-3.9.9/lib/python3.9/concurrent/futures/_base.py:438: in result\r\n return self.__get_result()\r\n/gnu/store/65i3nhcwmz0p8rqbg48gaavyky4g4hwk-python-3.9.9/lib/python3.9/concurrent/futures/_base.py:390: in __get_result\r\n raise self._exception\r\n/gnu/store/mcclmphjgbrgpa0v037a4nlq336482g8-python-asgiref-3.4.1/lib/python3.9/site-packages/asgiref/sync.py:292: in main_wrap\r\n result = await self.awaitable(*args, **kwargs)\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/datasette/utils/testing.py:66: in get\r\n return await self._request(\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/datasette/utils/testing.py:156: in _request\r\n httpx_response = await self.ds.client.request(\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/datasette/app.py:1602: in request\r\n return await client.request(\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1527: in request\r\n return await self.send(request, auth=auth, follow_redirects=follow_redirects)\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1614: in send\r\n response = await self._send_handling_auth(\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1642: in _send_handling_auth\r\n response = await self._send_handling_redirects(\r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\n\r\nself = \r\nrequest = \r\nfollow_redirects = True\r\nhistory = [, , , , , , ...]\r\n\r\n async def _send_handling_redirects(\r\n self,\r\n request: Request,\r\n follow_redirects: bool,\r\n history: typing.List[Response],\r\n ) -> Response:\r\n while True:\r\n if len(history) > self.max_redirects:\r\n> raise TooManyRedirects(\r\n \"Exceeded maximum allowed redirects.\", request=request\r\n )\r\nE httpx.TooManyRedirects: Exceeded maximum allowed redirects.\r\n\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1672: TooManyRedirects\r\n___________________ test_database_with_space_in_name[.json] ____________________\r\n[gw19] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python\r\n\r\napp_client_two_attached_databases = \r\npath = '.json'\r\n\r\n @pytest.mark.parametrize(\r\n \"path\",\r\n (\r\n \"/\",\r\n \".json\",\r\n \"/searchable\",\r\n \"/searchable.json\",\r\n \"/searchable_view\",\r\n \"/searchable_view.json\",\r\n ),\r\n )\r\n def test_database_with_space_in_name(app_client_two_attached_databases, path):\r\n> response = app_client_two_attached_databases.get(\r\n \"/extra~20database\" + path, follow_redirects=True\r\n )\r\n\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/tests/test_api.py:920: \r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\n/gnu/store/mcclmphjgbrgpa0v037a4nlq336482g8-python-asgiref-3.4.1/lib/python3.9/site-packages/asgiref/sync.py:223: in __call__\r\n return call_result.result()\r\n/gnu/store/65i3nhcwmz0p8rqbg48gaavyky4g4hwk-python-3.9.9/lib/python3.9/concurrent/futures/_base.py:438: in result\r\n return self.__get_result()\r\n/gnu/store/65i3nhcwmz0p8rqbg48gaavyky4g4hwk-python-3.9.9/lib/python3.9/concurrent/futures/_base.py:390: in __get_result\r\n raise self._exception\r\n/gnu/store/mcclmphjgbrgpa0v037a4nlq336482g8-python-asgiref-3.4.1/lib/python3.9/site-packages/asgiref/sync.py:292: in main_wrap\r\n result = await self.awaitable(*args, **kwargs)\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/datasette/utils/testing.py:66: in get\r\n return await self._request(\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/datasette/utils/testing.py:156: in _request\r\n httpx_response = await self.ds.client.request(\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/datasette/app.py:1602: in request\r\n return await client.request(\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1527: in request\r\n return await self.send(request, auth=auth, follow_redirects=follow_redirects)\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1614: in send\r\n response = await self._send_handling_auth(\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1642: in _send_handling_auth\r\n response = await self._send_handling_redirects(\r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\n\r\nself = \r\nrequest = \r\nfollow_redirects = True\r\nhistory = [, , , , , , ...]\r\n\r\n async def _send_handling_redirects(\r\n self,\r\n request: Request,\r\n follow_redirects: bool,\r\n history: typing.List[Response],\r\n ) -> Response:\r\n while True:\r\n if len(history) > self.max_redirects:\r\n> raise TooManyRedirects(\r\n \"Exceeded maximum allowed redirects.\", request=request\r\n )\r\nE httpx.TooManyRedirects: Exceeded maximum allowed redirects.\r\n\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1672: TooManyRedirects\r\n______________ test_database_with_space_in_name[/searchable_view] ______________\r\n[gw22] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python\r\n\r\napp_client_two_attached_databases = \r\npath = '/searchable_view'\r\n\r\n @pytest.mark.parametrize(\r\n \"path\",\r\n (\r\n \"/\",\r\n \".json\",\r\n \"/searchable\",\r\n \"/searchable.json\",\r\n \"/searchable_view\",\r\n \"/searchable_view.json\",\r\n ),\r\n )\r\n def test_database_with_space_in_name(app_client_two_attached_databases, path):\r\n> response = app_client_two_attached_databases.get(\r\n \"/extra~20database\" + path, follow_redirects=True\r\n )\r\n\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/tests/test_api.py:920: \r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\n/gnu/store/mcclmphjgbrgpa0v037a4nlq336482g8-python-asgiref-3.4.1/lib/python3.9/site-packages/asgiref/sync.py:223: in __call__\r\n return call_result.result()\r\n/gnu/store/65i3nhcwmz0p8rqbg48gaavyky4g4hwk-python-3.9.9/lib/python3.9/concurrent/futures/_base.py:438: in result\r\n return self.__get_result()\r\n/gnu/store/65i3nhcwmz0p8rqbg48gaavyky4g4hwk-python-3.9.9/lib/python3.9/concurrent/futures/_base.py:390: in __get_result\r\n raise self._exception\r\n/gnu/store/mcclmphjgbrgpa0v037a4nlq336482g8-python-asgiref-3.4.1/lib/python3.9/site-packages/asgiref/sync.py:292: in main_wrap\r\n result = await self.awaitable(*args, **kwargs)\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/datasette/utils/testing.py:66: in get\r\n return await self._request(\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/datasette/utils/testing.py:156: in _request\r\n httpx_response = await self.ds.client.request(\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/datasette/app.py:1602: in request\r\n return await client.request(\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1527: in request\r\n return await self.send(request, auth=auth, follow_redirects=follow_redirects)\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1614: in send\r\n response = await self._send_handling_auth(\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1642: in _send_handling_auth\r\n response = await self._send_handling_redirects(\r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\n\r\nself = \r\nrequest = \r\nfollow_redirects = True\r\nhistory = [, , , , , , ...]\r\n\r\n async def _send_handling_redirects(\r\n self,\r\n request: Request,\r\n follow_redirects: bool,\r\n history: typing.List[Response],\r\n ) -> Response:\r\n while True:\r\n if len(history) > self.max_redirects:\r\n> raise TooManyRedirects(\r\n \"Exceeded maximum allowed redirects.\", request=request\r\n )\r\nE httpx.TooManyRedirects: Exceeded maximum allowed redirects.\r\n\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1672: TooManyRedirects\r\n_____________________ test_database_with_space_in_name[/] ______________________\r\n[gw18] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python\r\n\r\napp_client_two_attached_databases = \r\npath = '/'\r\n\r\n @pytest.mark.parametrize(\r\n \"path\",\r\n (\r\n \"/\",\r\n \".json\",\r\n \"/searchable\",\r\n \"/searchable.json\",\r\n \"/searchable_view\",\r\n \"/searchable_view.json\",\r\n ),\r\n )\r\n def test_database_with_space_in_name(app_client_two_attached_databases, path):\r\n> response = app_client_two_attached_databases.get(\r\n \"/extra~20database\" + path, follow_redirects=True\r\n )\r\n\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/tests/test_api.py:920: \r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\n/gnu/store/mcclmphjgbrgpa0v037a4nlq336482g8-python-asgiref-3.4.1/lib/python3.9/site-packages/asgiref/sync.py:223: in __call__\r\n return call_result.result()\r\n/gnu/store/65i3nhcwmz0p8rqbg48gaavyky4g4hwk-python-3.9.9/lib/python3.9/concurrent/futures/_base.py:438: in result\r\n return self.__get_result()\r\n/gnu/store/65i3nhcwmz0p8rqbg48gaavyky4g4hwk-python-3.9.9/lib/python3.9/concurrent/futures/_base.py:390: in __get_result\r\n raise self._exception\r\n/gnu/store/mcclmphjgbrgpa0v037a4nlq336482g8-python-asgiref-3.4.1/lib/python3.9/site-packages/asgiref/sync.py:292: in main_wrap\r\n result = await self.awaitable(*args, **kwargs)\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/datasette/utils/testing.py:66: in get\r\n return await self._request(\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/datasette/utils/testing.py:156: in _request\r\n httpx_response = await self.ds.client.request(\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/datasette/app.py:1602: in request\r\n return await client.request(\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1527: in request\r\n return await self.send(request, auth=auth, follow_redirects=follow_redirects)\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1614: in send\r\n response = await self._send_handling_auth(\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1642: in _send_handling_auth\r\n response = await self._send_handling_redirects(\r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\n\r\nself = \r\nrequest = \r\nfollow_redirects = True\r\nhistory = [, , , , , , ...]\r\n\r\n async def _send_handling_redirects(\r\n self,\r\n request: Request,\r\n follow_redirects: bool,\r\n history: typing.List[Response],\r\n ) -> Response:\r\n while True:\r\n if len(history) > self.max_redirects:\r\n> raise TooManyRedirects(\r\n \"Exceeded maximum allowed redirects.\", request=request\r\n )\r\nE httpx.TooManyRedirects: Exceeded maximum allowed redirects.\r\n\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1672: TooManyRedirects\r\n________________ test_database_with_space_in_name[/searchable] _________________\r\n[gw20] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python\r\n\r\napp_client_two_attached_databases = \r\npath = '/searchable'\r\n\r\n @pytest.mark.parametrize(\r\n \"path\",\r\n (\r\n \"/\",\r\n \".json\",\r\n \"/searchable\",\r\n \"/searchable.json\",\r\n \"/searchable_view\",\r\n \"/searchable_view.json\",\r\n ),\r\n )\r\n def test_database_with_space_in_name(app_client_two_attached_databases, path):\r\n> response = app_client_two_attached_databases.get(\r\n \"/extra~20database\" + path, follow_redirects=True\r\n )\r\n\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/tests/test_api.py:920: \r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\n/gnu/store/mcclmphjgbrgpa0v037a4nlq336482g8-python-asgiref-3.4.1/lib/python3.9/site-packages/asgiref/sync.py:223: in __call__\r\n return call_result.result()\r\n/gnu/store/65i3nhcwmz0p8rqbg48gaavyky4g4hwk-python-3.9.9/lib/python3.9/concurrent/futures/_base.py:438: in result\r\n return self.__get_result()\r\n/gnu/store/65i3nhcwmz0p8rqbg48gaavyky4g4hwk-python-3.9.9/lib/python3.9/concurrent/futures/_base.py:390: in __get_result\r\n raise self._exception\r\n/gnu/store/mcclmphjgbrgpa0v037a4nlq336482g8-python-asgiref-3.4.1/lib/python3.9/site-packages/asgiref/sync.py:292: in main_wrap\r\n result = await self.awaitable(*args, **kwargs)\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/datasette/utils/testing.py:66: in get\r\n return await self._request(\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/datasette/utils/testing.py:156: in _request\r\n httpx_response = await self.ds.client.request(\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/datasette/app.py:1602: in request\r\n return await client.request(\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1527: in request\r\n return await self.send(request, auth=auth, follow_redirects=follow_redirects)\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1614: in send\r\n response = await self._send_handling_auth(\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1642: in _send_handling_auth\r\n response = await self._send_handling_redirects(\r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\n\r\nself = \r\nrequest = \r\nfollow_redirects = True\r\nhistory = [, , , , , , ...]\r\n\r\n async def _send_handling_redirects(\r\n self,\r\n request: Request,\r\n follow_redirects: bool,\r\n history: typing.List[Response],\r\n ) -> Response:\r\n while True:\r\n if len(history) > self.max_redirects:\r\n> raise TooManyRedirects(\r\n \"Exceeded maximum allowed redirects.\", request=request\r\n )\r\nE httpx.TooManyRedirects: Exceeded maximum allowed redirects.\r\n\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1672: TooManyRedirects\r\n___________ test_database_with_space_in_name[/searchable_view.json] ____________\r\n[gw23] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python\r\n\r\napp_client_two_attached_databases = \r\npath = '/searchable_view.json'\r\n\r\n @pytest.mark.parametrize(\r\n \"path\",\r\n (\r\n \"/\",\r\n \".json\",\r\n \"/searchable\",\r\n \"/searchable.json\",\r\n \"/searchable_view\",\r\n \"/searchable_view.json\",\r\n ),\r\n )\r\n def test_database_with_space_in_name(app_client_two_attached_databases, path):\r\n> response = app_client_two_attached_databases.get(\r\n \"/extra~20database\" + path, follow_redirects=True\r\n )\r\n\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/tests/test_api.py:920: \r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\n/gnu/store/mcclmphjgbrgpa0v037a4nlq336482g8-python-asgiref-3.4.1/lib/python3.9/site-packages/asgiref/sync.py:223: in __call__\r\n return call_result.result()\r\n/gnu/store/65i3nhcwmz0p8rqbg48gaavyky4g4hwk-python-3.9.9/lib/python3.9/concurrent/futures/_base.py:438: in result\r\n return self.__get_result()\r\n/gnu/store/65i3nhcwmz0p8rqbg48gaavyky4g4hwk-python-3.9.9/lib/python3.9/concurrent/futures/_base.py:390: in __get_result\r\n raise self._exception\r\n/gnu/store/mcclmphjgbrgpa0v037a4nlq336482g8-python-asgiref-3.4.1/lib/python3.9/site-packages/asgiref/sync.py:292: in main_wrap\r\n result = await self.awaitable(*args, **kwargs)\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/datasette/utils/testing.py:66: in get\r\n return await self._request(\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/datasette/utils/testing.py:156: in _request\r\n httpx_response = await self.ds.client.request(\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/datasette/app.py:1602: in request\r\n return await client.request(\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1527: in request\r\n return await self.send(request, auth=auth, follow_redirects=follow_redirects)\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1614: in send\r\n response = await self._send_handling_auth(\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1642: in _send_handling_auth\r\n response = await self._send_handling_redirects(\r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\n\r\nself = \r\nrequest = \r\nfollow_redirects = True\r\nhistory = [, , , , , , ...]\r\n\r\n async def _send_handling_redirects(\r\n self,\r\n request: Request,\r\n follow_redirects: bool,\r\n history: typing.List[Response],\r\n ) -> Response:\r\n while True:\r\n if len(history) > self.max_redirects:\r\n> raise TooManyRedirects(\r\n \"Exceeded maximum allowed redirects.\", request=request\r\n )\r\nE httpx.TooManyRedirects: Exceeded maximum allowed redirects.\r\n\r\n/gnu/store/bj5lb299rfb4cbbq5kczq9imdk9a7y64-python-httpx-0.23.0/lib/python3.9/site-packages/httpx/_client.py:1672: TooManyRedirects\r\n________________ test_weird_database_names[database (1).sqlite] ________________\r\n[gw7] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python\r\n\r\ntmpdir = local('/tmp/guix-build-datasette-0.64.2.drv-0/pytest-of-nixbld/pytest-0/popen-gw7/test_weird_database_names_data0')\r\nfilename = 'database (1).sqlite'\r\n\r\n @pytest.mark.parametrize(\r\n \"filename\", [\"test-database (1).sqlite\", \"database (1).sqlite\"]\r\n )\r\n def test_weird_database_names(tmpdir, filename):\r\n # https://github.com/simonw/datasette/issues/1181\r\n runner = CliRunner()\r\n db_path = str(tmpdir / filename)\r\n sqlite3.connect(db_path).execute(\"vacuum\")\r\n result1 = runner.invoke(cli, [db_path, \"--get\", \"/\"])\r\n assert result1.exit_code == 0, result1.output\r\n filename_no_stem = filename.rsplit(\".\", 1)[0]\r\n expected_link = '{}'.format(\r\n tilde_encode(filename_no_stem), filename_no_stem\r\n )\r\n assert expected_link in result1.output\r\n # Now try hitting that database page\r\n result2 = runner.invoke(\r\n cli, [db_path, \"--get\", \"/{}\".format(tilde_encode(filename_no_stem))]\r\n )\r\n> assert result2.exit_code == 0, result2.output\r\nE AssertionError: \r\nE \r\nE assert 1 == 0\r\nE + where 1 = .exit_code\r\n\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/tests/test_cli.py:321: AssertionError\r\n_____________ test_weird_database_names[test-database (1).sqlite] ______________\r\n[gw6] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python\r\n\r\ntmpdir = local('/tmp/guix-build-datasette-0.64.2.drv-0/pytest-of-nixbld/pytest-0/popen-gw6/test_weird_database_names_test0')\r\nfilename = 'test-database (1).sqlite'\r\n\r\n @pytest.mark.parametrize(\r\n \"filename\", [\"test-database (1).sqlite\", \"database (1).sqlite\"]\r\n )\r\n def test_weird_database_names(tmpdir, filename):\r\n # https://github.com/simonw/datasette/issues/1181\r\n runner = CliRunner()\r\n db_path = str(tmpdir / filename)\r\n sqlite3.connect(db_path).execute(\"vacuum\")\r\n result1 = runner.invoke(cli, [db_path, \"--get\", \"/\"])\r\n assert result1.exit_code == 0, result1.output\r\n filename_no_stem = filename.rsplit(\".\", 1)[0]\r\n expected_link = '{}'.format(\r\n tilde_encode(filename_no_stem), filename_no_stem\r\n )\r\n assert expected_link in result1.output\r\n # Now try hitting that database page\r\n result2 = runner.invoke(\r\n cli, [db_path, \"--get\", \"/{}\".format(tilde_encode(filename_no_stem))]\r\n )\r\n> assert result2.exit_code == 0, result2.output\r\nE AssertionError: \r\nE \r\nE assert 1 == 0\r\nE + where 1 = .exit_code\r\n\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/tests/test_cli.py:321: AssertionError\r\n_ test_row_html_compound_primary_key[/fixtures/compound_primary_key/a~2Fb,~2Ec~2Dd-expected1] _\r\n[gw11] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python\r\n\r\napp_client = \r\npath = '/fixtures/compound_primary_key/a~2Fb,~2Ec~2Dd'\r\nexpected = [['a/b', '.c-d', 'c']]\r\n\r\n @pytest.mark.parametrize(\r\n \"path,expected\",\r\n (\r\n (\r\n \"/fixtures/compound_primary_key/a,b\",\r\n [\r\n [\r\n 'a',\r\n 'b',\r\n 'c',\r\n ]\r\n ],\r\n ),\r\n (\r\n \"/fixtures/compound_primary_key/a~2Fb,~2Ec~2Dd\",\r\n [\r\n [\r\n 'a/b',\r\n '.c-d',\r\n 'c',\r\n ]\r\n ],\r\n ),\r\n ),\r\n )\r\n def test_row_html_compound_primary_key(app_client, path, expected):\r\n response = app_client.get(path)\r\n> assert response.status == 200\r\nE assert 302 == 200\r\nE + where 302 = .status\r\n\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/tests/test_html.py:370: AssertionError\r\n_ test_css_classes_on_body[/fixtures/table~2Fwith~2Fslashes~2Ecsv-expected_classes5] _\r\n[gw3] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python\r\n\r\napp_client = \r\npath = '/fixtures/table~2Fwith~2Fslashes~2Ecsv'\r\nexpected_classes = ['table', 'db-fixtures', 'table-tablewithslashescsv-fa7563']\r\n\r\n @pytest.mark.parametrize(\r\n \"path,expected_classes\",\r\n [\r\n (\"/\", [\"index\"]),\r\n (\"/fixtures\", [\"db\", \"db-fixtures\"]),\r\n (\"/fixtures?sql=select+1\", [\"query\", \"db-fixtures\"]),\r\n (\r\n \"/fixtures/simple_primary_key\",\r\n [\"table\", \"db-fixtures\", \"table-simple_primary_key\"],\r\n ),\r\n (\r\n \"/fixtures/neighborhood_search\",\r\n [\"query\", \"db-fixtures\", \"query-neighborhood_search\"],\r\n ),\r\n (\r\n \"/fixtures/table~2Fwith~2Fslashes~2Ecsv\",\r\n [\"table\", \"db-fixtures\", \"table-tablewithslashescsv-fa7563\"],\r\n ),\r\n (\r\n \"/fixtures/simple_primary_key/1\",\r\n [\"row\", \"db-fixtures\", \"table-simple_primary_key\"],\r\n ),\r\n ],\r\n )\r\n def test_css_classes_on_body(app_client, path, expected_classes):\r\n response = app_client.get(path)\r\n> assert response.status == 200\r\nE assert 302 == 200\r\nE + where 302 = .status\r\n\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/tests/test_html.py:238: AssertionError\r\n_ test_templates_considered[/fixtures/table~2Fwith~2Fslashes~2Ecsv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html] _\r\n[gw3] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python\r\n\r\napp_client = \r\npath = '/fixtures/table~2Fwith~2Fslashes~2Ecsv'\r\nexpected_considered = 'table-fixtures-tablewithslashescsv-fa7563.html, *table.html'\r\n\r\n @pytest.mark.parametrize(\r\n \"path,expected_considered\",\r\n [\r\n (\"/\", \"*index.html\"),\r\n (\"/fixtures\", \"database-fixtures.html, *database.html\"),\r\n (\r\n \"/fixtures/simple_primary_key\",\r\n \"table-fixtures-simple_primary_key.html, *table.html\",\r\n ),\r\n (\r\n \"/fixtures/table~2Fwith~2Fslashes~2Ecsv\",\r\n \"table-fixtures-tablewithslashescsv-fa7563.html, *table.html\",\r\n ),\r\n (\r\n \"/fixtures/simple_primary_key/1\",\r\n \"row-fixtures-simple_primary_key.html, *row.html\",\r\n ),\r\n ],\r\n )\r\n def test_templates_considered(app_client, path, expected_considered):\r\n response = app_client.get(path)\r\n> assert response.status == 200\r\nE assert 302 == 200\r\nE + where 302 = .status\r\n\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/tests/test_html.py:264: AssertionError\r\n_ test_alternate_url_json[/fixtures/table~2Fwith~2Fslashes~2Ecsv-http://localhost/fixtures/table~2Fwith~2Fslashes~2Ecsv.json] _\r\n[gw21] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python\r\n\r\napp_client = \r\npath = '/fixtures/table~2Fwith~2Fslashes~2Ecsv'\r\nexpected = 'http://localhost/fixtures/table~2Fwith~2Fslashes~2Ecsv.json'\r\n\r\n @pytest.mark.parametrize(\r\n \"path,expected\",\r\n (\r\n # Instance index page\r\n (\"/\", \"http://localhost/.json\"),\r\n # Table page\r\n (\"/fixtures/facetable\", \"http://localhost/fixtures/facetable.json\"),\r\n (\r\n \"/fixtures/table~2Fwith~2Fslashes~2Ecsv\",\r\n \"http://localhost/fixtures/table~2Fwith~2Fslashes~2Ecsv.json\",\r\n ),\r\n # Row page\r\n (\r\n \"/fixtures/no_primary_key/1\",\r\n \"http://localhost/fixtures/no_primary_key/1.json\",\r\n ),\r\n # Database index page\r\n (\r\n \"/fixtures\",\r\n \"http://localhost/fixtures.json\",\r\n ),\r\n # Custom query page\r\n (\r\n \"/fixtures?sql=select+*+from+facetable\",\r\n \"http://localhost/fixtures.json?sql=select+*+from+facetable\",\r\n ),\r\n # Canned query page\r\n (\r\n \"/fixtures/neighborhood_search?text=town\",\r\n \"http://localhost/fixtures/neighborhood_search.json?text=town\",\r\n ),\r\n # /-/ pages\r\n (\r\n \"/-/plugins\",\r\n \"http://localhost/-/plugins.json\",\r\n ),\r\n ),\r\n )\r\n def test_alternate_url_json(app_client, path, expected):\r\n response = app_client.get(path)\r\n> assert response.status == 200\r\nE assert 302 == 200\r\nE + where 302 = .status\r\n\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/tests/test_html.py:948: AssertionError\r\n_ test_edit_sql_link_on_canned_queries[/fixtures/~F0~9D~90~9C~F0~9D~90~A2~F0~9D~90~AD~F0~9D~90~A2~F0~9D~90~9E~F0~9D~90~AC-/fixtures?sql=select+id%2C+name+from+facet_cities+order+by+id+limit+1%3B] _\r\n[gw18] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python\r\n\r\napp_client = \r\npath = '/fixtures/~F0~9D~90~9C~F0~9D~90~A2~F0~9D~90~AD~F0~9D~90~A2~F0~9D~90~9E~F0~9D~90~AC'\r\nexpected = '/fixtures?sql=select+id%2C+name+from+facet_cities+order+by+id+limit+1%3B'\r\n\r\n @pytest.mark.parametrize(\r\n \"path,expected\",\r\n [\r\n (\r\n \"/fixtures/neighborhood_search\",\r\n \"/fixtures?sql=%0Aselect+_neighborhood%2C+facet_cities.name%2C+state%0Afrom+facetable%0A++++join+facet_cities%0A++++++++on+facetable._city_id+%3D+facet_cities.id%0Awhere+_neighborhood+like+%27%25%27+%7C%7C+%3Atext+%7C%7C+%27%25%27%0Aorder+by+_neighborhood%3B%0A&text=\",\r\n ),\r\n (\r\n \"/fixtures/neighborhood_search?text=ber\",\r\n \"/fixtures?sql=%0Aselect+_neighborhood%2C+facet_cities.name%2C+state%0Afrom+facetable%0A++++join+facet_cities%0A++++++++on+facetable._city_id+%3D+facet_cities.id%0Awhere+_neighborhood+like+%27%25%27+%7C%7C+%3Atext+%7C%7C+%27%25%27%0Aorder+by+_neighborhood%3B%0A&text=ber\",\r\n ),\r\n (\"/fixtures/pragma_cache_size\", None),\r\n (\r\n # /fixtures/\ud835\udc1c\ud835\udc22\ud835\udc2d\ud835\udc22\ud835\udc1e\ud835\udc2c\r\n \"/fixtures/~F0~9D~90~9C~F0~9D~90~A2~F0~9D~90~AD~F0~9D~90~A2~F0~9D~90~9E~F0~9D~90~AC\",\r\n \"/fixtures?sql=select+id%2C+name+from+facet_cities+order+by+id+limit+1%3B\",\r\n ),\r\n (\"/fixtures/magic_parameters\", None),\r\n ],\r\n )\r\n def test_edit_sql_link_on_canned_queries(app_client, path, expected):\r\n response = app_client.get(path)\r\n> assert response.status == 200\r\nE assert 302 == 200\r\nE + where 302 = .status\r\n\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/tests/test_html.py:841: AssertionError\r\n_______________________ test_table_with_slashes_in_name ________________________\r\n[gw9] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python\r\n\r\napp_client = \r\n\r\n def test_table_with_slashes_in_name(app_client):\r\n response = app_client.get(\r\n \"/fixtures/table~2Fwith~2Fslashes~2Ecsv.json?_shape=objects\"\r\n )\r\n> assert response.status == 200\r\nE assert 302 == 200\r\nE + where 302 = .status\r\n\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/tests/test_table_api.py:141: AssertionError\r\n__________________ test_custom_query_with_unicode_characters ___________________\r\n[gw8] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python\r\n\r\napp_client = \r\n\r\n def test_custom_query_with_unicode_characters(app_client):\r\n # /fixtures/\ud835\udc1c\ud835\udc22\ud835\udc2d\ud835\udc22\ud835\udc1e\ud835\udc2c.json\r\n response = app_client.get(\r\n \"/fixtures/~F0~9D~90~9C~F0~9D~90~A2~F0~9D~90~AD~F0~9D~90~A2~F0~9D~90~9E~F0~9D~90~AC.json?_shape=array\"\r\n )\r\n> assert [{\"id\": 1, \"name\": \"San Francisco\"}] == response.json\r\n\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/tests/test_table_api.py:1042: \r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/datasette/utils/testing.py:40: in json\r\n return json.loads(self.text)\r\n/gnu/store/65i3nhcwmz0p8rqbg48gaavyky4g4hwk-python-3.9.9/lib/python3.9/json/__init__.py:346: in loads\r\n return _default_decoder.decode(s)\r\n/gnu/store/65i3nhcwmz0p8rqbg48gaavyky4g4hwk-python-3.9.9/lib/python3.9/json/decoder.py:337: in decode\r\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\n\r\nself = , s = '', idx = 0\r\n\r\n def raw_decode(self, s, idx=0):\r\n \"\"\"Decode a JSON document from ``s`` (a ``str`` beginning with\r\n a JSON document) and return a 2-tuple of the Python\r\n representation and the index in ``s`` where the document ended.\r\n \r\n This can be used to decode a JSON document from a string that may\r\n have extraneous data at the end.\r\n \r\n \"\"\"\r\n try:\r\n obj, end = self.scan_once(s, idx)\r\n except StopIteration as err:\r\n> raise JSONDecodeError(\"Expecting value\", s, err.value) from None\r\nE json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\r\n\r\n/gnu/store/65i3nhcwmz0p8rqbg48gaavyky4g4hwk-python-3.9.9/lib/python3.9/json/decoder.py:355: JSONDecodeError\r\n_ test_searchable[/fixtures/searchable.json?_search=te*+AND+do*&_searchmode=raw-expected_rows3] _\r\n[gw13] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python\r\n\r\napp_client = \r\npath = '/fixtures/searchable.json?_search=te*+AND+do*&_searchmode=raw'\r\nexpected_rows = [[1, 'barry cat', 'terry dog', 'panther'], [2, 'terry dog', 'sara weasel', 'puma']]\r\n\r\n @pytest.mark.parametrize(\r\n \"path,expected_rows\",\r\n [\r\n (\r\n \"/fixtures/searchable.json?_search=dog\",\r\n [\r\n [1, \"barry cat\", \"terry dog\", \"panther\"],\r\n [2, \"terry dog\", \"sara weasel\", \"puma\"],\r\n ],\r\n ),\r\n (\r\n # Special keyword shouldn't break FTS query\r\n \"/fixtures/searchable.json?_search=AND\",\r\n [],\r\n ),\r\n (\r\n # Without _searchmode=raw this should return no results\r\n \"/fixtures/searchable.json?_search=te*+AND+do*\",\r\n [],\r\n ),\r\n (\r\n # _searchmode=raw\r\n \"/fixtures/searchable.json?_search=te*+AND+do*&_searchmode=raw\",\r\n [\r\n [1, \"barry cat\", \"terry dog\", \"panther\"],\r\n [2, \"terry dog\", \"sara weasel\", \"puma\"],\r\n ],\r\n ),\r\n (\r\n # _searchmode=raw combined with _search_COLUMN\r\n \"/fixtures/searchable.json?_search_text2=te*&_searchmode=raw\",\r\n [\r\n [1, \"barry cat\", \"terry dog\", \"panther\"],\r\n ],\r\n ),\r\n (\r\n \"/fixtures/searchable.json?_search=weasel\",\r\n [[2, \"terry dog\", \"sara weasel\", \"puma\"]],\r\n ),\r\n (\r\n \"/fixtures/searchable.json?_search_text2=dog\",\r\n [[1, \"barry cat\", \"terry dog\", \"panther\"]],\r\n ),\r\n (\r\n \"/fixtures/searchable.json?_search_name%20with%20.%20and%20spaces=panther\",\r\n [[1, \"barry cat\", \"terry dog\", \"panther\"]],\r\n ),\r\n ],\r\n )\r\n def test_searchable(app_client, path, expected_rows):\r\n response = app_client.get(path)\r\n> assert expected_rows == response.json[\"rows\"]\r\nE AssertionError: assert [[1, 'barry cat', 'terry dog', 'panther'],\\n [2, 'terry dog', 'sara weasel', 'puma']] == []\r\nE Left contains 2 more items, first extra item: [1, 'barry cat', 'terry dog', 'panther']\r\nE Full diff:\r\nE [\r\nE - ,\r\nE + [1,\r\nE + 'barry cat',\r\nE + 'terry dog',\r\nE + 'panther'],\r\nE + [2,\r\nE + 'terry dog',\r\nE + 'sara weasel',\r\nE + 'puma'],\r\nE ]\r\n\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/tests/test_table_api.py:402: AssertionError\r\n_____ test_searchmode[table_metadata1-_search=te*+AND+do*-expected_rows1] ______\r\n[gw20] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python\r\n\r\ntable_metadata = {'searchmode': 'raw'}, querystring = '_search=te*+AND+do*'\r\nexpected_rows = [[1, 'barry cat', 'terry dog', 'panther'], [2, 'terry dog', 'sara weasel', 'puma']]\r\n\r\n @pytest.mark.parametrize(\r\n \"table_metadata,querystring,expected_rows\",\r\n [\r\n (\r\n {},\r\n \"_search=te*+AND+do*\",\r\n [],\r\n ),\r\n (\r\n {\"searchmode\": \"raw\"},\r\n \"_search=te*+AND+do*\",\r\n _SEARCHMODE_RAW_RESULTS,\r\n ),\r\n (\r\n {},\r\n \"_search=te*+AND+do*&_searchmode=raw\",\r\n _SEARCHMODE_RAW_RESULTS,\r\n ),\r\n # Can be over-ridden with _searchmode=escaped\r\n (\r\n {\"searchmode\": \"raw\"},\r\n \"_search=te*+AND+do*&_searchmode=escaped\",\r\n [],\r\n ),\r\n ],\r\n )\r\n def test_searchmode(table_metadata, querystring, expected_rows):\r\n with make_app_client(\r\n metadata={\"databases\": {\"fixtures\": {\"tables\": {\"searchable\": table_metadata}}}}\r\n ) as client:\r\n response = client.get(\"/fixtures/searchable.json?\" + querystring)\r\n> assert expected_rows == response.json[\"rows\"]\r\nE AssertionError: assert [[1, 'barry cat', 'terry dog', 'panther'],\\n [2, 'terry dog', 'sara weasel', 'puma']] == []\r\nE Left contains 2 more items, first extra item: [1, 'barry cat', 'terry dog', 'panther']\r\nE Full diff:\r\nE [\r\nE - ,\r\nE + [1,\r\nE + 'barry cat',\r\nE + 'terry dog',\r\nE + 'panther'],\r\nE + [2,\r\nE + 'terry dog',\r\nE + 'sara weasel',\r\nE + 'puma'],\r\nE ]\r\n\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/tests/test_table_api.py:442: AssertionError\r\n_ test_searchmode[table_metadata2-_search=te*+AND+do*&_searchmode=raw-expected_rows2] _\r\n[gw20] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python\r\n\r\ntable_metadata = {}, querystring = '_search=te*+AND+do*&_searchmode=raw'\r\nexpected_rows = [[1, 'barry cat', 'terry dog', 'panther'], [2, 'terry dog', 'sara weasel', 'puma']]\r\n\r\n @pytest.mark.parametrize(\r\n \"table_metadata,querystring,expected_rows\",\r\n [\r\n (\r\n {},\r\n \"_search=te*+AND+do*\",\r\n [],\r\n ),\r\n (\r\n {\"searchmode\": \"raw\"},\r\n \"_search=te*+AND+do*\",\r\n _SEARCHMODE_RAW_RESULTS,\r\n ),\r\n (\r\n {},\r\n \"_search=te*+AND+do*&_searchmode=raw\",\r\n _SEARCHMODE_RAW_RESULTS,\r\n ),\r\n # Can be over-ridden with _searchmode=escaped\r\n (\r\n {\"searchmode\": \"raw\"},\r\n \"_search=te*+AND+do*&_searchmode=escaped\",\r\n [],\r\n ),\r\n ],\r\n )\r\n def test_searchmode(table_metadata, querystring, expected_rows):\r\n with make_app_client(\r\n metadata={\"databases\": {\"fixtures\": {\"tables\": {\"searchable\": table_metadata}}}}\r\n ) as client:\r\n response = client.get(\"/fixtures/searchable.json?\" + querystring)\r\n> assert expected_rows == response.json[\"rows\"]\r\nE AssertionError: assert [[1, 'barry cat', 'terry dog', 'panther'],\\n [2, 'terry dog', 'sara weasel', 'puma']] == []\r\nE Left contains 2 more items, first extra item: [1, 'barry cat', 'terry dog', 'panther']\r\nE Full diff:\r\nE [\r\nE - ,\r\nE + [1,\r\nE + 'barry cat',\r\nE + 'terry dog',\r\nE + 'panther'],\r\nE + [2,\r\nE + 'terry dog',\r\nE + 'sara weasel',\r\nE + 'puma'],\r\nE ]\r\n\r\n/tmp/guix-build-datasette-0.64.2.drv-0/source/tests/test_table_api.py:442: AssertionError\r\n=========================== short test summary info ============================\r\nFAILED tests/test_api.py::test_row_strange_table_name - assert 400 == 200\r\nFAILED tests/test_api.py::test_database_page_for_database_with_dot_in_name - ...\r\nFAILED tests/test_api.py::test_tilde_encoded_database_names[fo%o] - assert 30...\r\nFAILED tests/test_api.py::test_tilde_encoded_database_names[f~/c.d] - assert ...\r\nFAILED tests/test_api.py::test_database_with_space_in_name[/searchable.json]\r\nFAILED tests/test_api.py::test_database_with_space_in_name[.json] - httpx.Too...\r\nFAILED tests/test_api.py::test_database_with_space_in_name[/searchable_view]\r\nFAILED tests/test_api.py::test_database_with_space_in_name[/] - httpx.TooMany...\r\nFAILED tests/test_api.py::test_database_with_space_in_name[/searchable] - htt...\r\nFAILED tests/test_api.py::test_database_with_space_in_name[/searchable_view.json]\r\nFAILED tests/test_cli.py::test_weird_database_names[database (1).sqlite] - As...\r\nFAILED tests/test_cli.py::test_weird_database_names[test-database (1).sqlite]\r\nFAILED tests/test_html.py::test_row_html_compound_primary_key[/fixtures/compound_primary_key/a~2Fb,~2Ec~2Dd-expected1]\r\nFAILED tests/test_html.py::test_css_classes_on_body[/fixtures/table~2Fwith~2Fslashes~2Ecsv-expected_classes5]\r\nFAILED tests/test_html.py::test_templates_considered[/fixtures/table~2Fwith~2Fslashes~2Ecsv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html]\r\nFAILED tests/test_html.py::test_alternate_url_json[/fixtures/table~2Fwith~2Fslashes~2Ecsv-http://localhost/fixtures/table~2Fwith~2Fslashes~2Ecsv.json]\r\nFAILED tests/test_html.py::test_edit_sql_link_on_canned_queries[/fixtures/~F0~9D~90~9C~F0~9D~90~A2~F0~9D~90~AD~F0~9D~90~A2~F0~9D~90~9E~F0~9D~90~AC-/fixtures?sql=select+id%2C+name+from+facet_cities+order+by+id+limit+1%3B]\r\nFAILED tests/test_table_api.py::test_table_with_slashes_in_name - assert 302 ...\r\nFAILED tests/test_table_api.py::test_custom_query_with_unicode_characters - j...\r\nFAILED tests/test_table_api.py::test_searchable[/fixtures/searchable.json?_search=te*+AND+do*&_searchmode=raw-expected_rows3]\r\nFAILED tests/test_table_api.py::test_searchmode[table_metadata1-_search=te*+AND+do*-expected_rows1]\r\nFAILED tests/test_table_api.py::test_searchmode[table_metadata2-_search=te*+AND+do*&_searchmode=raw-expected_rows2]\r\n=========== 22 failed, 1049 passed, 3 skipped in 1522.28s (0:25:22) ============\r\nerror: in phase 'check': uncaught exception:\r\n%exception #<&invoke-error program: \"/gnu/store/ziqwkzz6znb5d3c245xn0cq5ra2ly0w3-python-pytest-7.1.3/bin/pytest\" arguments: (\"-vv\" \"-n\" \"24\" \"-m\" \"not serial\") exit-status: 1 term-signal: #f stop-signal: #f> \r\nphase `check' failed after 1523.3 seconds\r\n```\r\nThe tests run in a private namespace without internet connectivity, and the Python dependencies are at:\r\n```\r\npython-aiofiles@0.6.0 python-asgi-csrf@0.9 python-asgiref@3.4.1\r\n+ python-beautifulsoup4@4.11.1 python-black@22.3.0 python-click-default-group@1.2.2 python-click@8.1.3\r\n+ python-cogapp@3.3.0 python-httpx@0.23.0 python-hupper@1.10.3 python-itsdangerous@2.0.1\r\n+ python-janus@1.0.0 python-jinja2@3.1.1 python-mergedeep@1.3.4 python-pint@0.20.1 python-pluggy@1.0.0\r\n+ python-pytest-asyncio@0.17.2 python-pytest-runner@5.2 python-pytest-timeout@2.0.2\r\n+ python-pytest-xdist@2.5.0 python-pytest@7.1.3 python-pyyaml@6.0 python-setuptools@64.0.3\r\n+ python-trustme@0.9.0 python-uvicorn@0.17.6\r\n```\r\nWith Python 3.9.9.\r\n\r\nThank you!", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2048/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1617823309, "node_id": "I_kwDOJHON9s5gbgZN", "number": 8, "title": "Increase performance using macnotesapp", "user": {"value": 41546558, "label": "RhetTbull"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-03-09T18:51:05Z", "updated_at": "2023-03-14T22:00:22Z", "closed_at": "2023-03-14T22:00:21Z", "author_association": "NONE", "pull_request": null, "body": "Neat project! You can probably increase performance using my python interface to Notes, [macnotesapp](https://github.com/RhetTbull/macnotesapp), which uses Scripting Bridge and bulk queries for much better performance than AppleScript.\r\n\r\nAnother related project is [PyXA](https://github.com/SKaplanOfficial/PyXA) which uses Scripting Bridge to access Notes (and many other apps) and can return all the notes at once as opposed to calling AppleScript for each note. \r\n\r\nmacnotesapp allows you to access multiple accounts and folders as well.\r\n\r\n```python\r\nfrom macnotesapp import NotesApp\r\n\r\n# NotesApp() provides interface to Notes.app\r\nnotesapp = NotesApp()\r\n\r\n# Get list of notes (Note objects for each note)\r\nnotes = notesapp.notes()\r\nnote = notes[0]\r\nprint(\r\n note.id,\r\n note.account,\r\n note.folder,\r\n note.name,\r\n note.body,\r\n note.plaintext,\r\n note.password_protected,\r\n)\r\n\r\nprint(note.asdict())\r\n\r\n```", "repo": {"value": 611552758, "label": "apple-notes-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/8/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1590183272, "node_id": "I_kwDOBm6k_c5eyEVo", "number": 2027, "title": "How to redirect from \"/\" to a specific db/table", "user": {"value": 1350673, "label": "dmick"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-02-18T03:14:01Z", "updated_at": "2023-03-08T04:42:22Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Using nginx to redirect public IP to the local uvicorn server as 'normal'. I can't figure out how to redirect such that '/' results in accessing the one db/table I want to serve; redirecting / to /db/table breaks some of the CSS; fooling with base_url doesn't seem to help. Can someone explain this, if it's possible?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2027/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1594383280, "node_id": "I_kwDOBm6k_c5fCFuw", "number": 2030, "title": "How to use Datasette with apache webserver on GCP?", "user": {"value": 19700859, "label": "gk7279"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-02-22T03:08:49Z", "updated_at": "2023-02-22T21:54:39Z", "closed_at": "2023-02-22T21:54:39Z", "author_association": "NONE", "pull_request": null, "body": "Hi Simon and Datasette team-\r\n\r\nI have installed apache2 webserver inside GCP VM using apt.\r\n\r\nI can see my \"Hello World\" index.html if I use the external IP of this GCP in a browser.\r\n\r\nHowever, when I try to run datasette with different combinations of -h and -p, I am still unable to access the webpage.\r\n\r\nI cannot invest Docker on this VM.\r\n\r\nAny pointers to use datasette with already existing apache2 webserver on GCP is appreciated.\r\n\r\nThanks.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2030/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 828858421, "node_id": "MDU6SXNzdWU4Mjg4NTg0MjE=", "number": 1258, "title": "Allow canned query params to specify default values", "user": {"value": 1385831, "label": "wdccdw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2021-03-11T07:19:02Z", "updated_at": "2023-02-20T23:39:58Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "If I call a canned query that includes named parameters, without passing any parameters, datasette runs the query anyway, resulting in an HTTP status code 400, and a visible error in the browser, with only a link back to home. This means that one of the default links on https://site/database/ will lead to a broken page with no apparent way out.\r\n\r\n![image](https://user-images.githubusercontent.com/1385831/110748683-13e72300-820e-11eb-855c-32e03dfef5bf.png)\r\n\r\nIs there any way to skip performing the query when parameters aren't supplied, but otherwise render the usual canned query page? Alternatively, can I supply default values for my parameters, either when defining my canned queries or when linking to the canned query page from the default database template.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1258/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1592327343, "node_id": "I_kwDOBm6k_c5e6Pyv", "number": 2029, "title": "Sorry Simon, didn't know how else to contact you", "user": {"value": 5804626, "label": "llchristopherson"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-02-20T19:02:53Z", "updated_at": "2023-02-20T19:02:53Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi Simon,\r\n\r\nWould you be willing to chat with me about Datasette? I have some questions. I am working on a project to evaluate data ingestion tools for a research organization and I ran across Datasette. I have looked through a lot of your documentation, but still have some questions, which are very specific. If you would be willing to write me back about this, my email is laura@renci.org.\r\n\r\nThanks,\r\nLaura", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2029/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1572766460, "node_id": "I_kwDOCGYnMM5dvoL8", "number": 524, "title": "Transformation type `--type DATETIME`", "user": {"value": 21095447, "label": "4l1fe"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 15, "created_at": "2023-02-06T15:18:42Z", "updated_at": "2023-02-15T12:10:54Z", "closed_at": "2023-02-15T12:10:54Z", "author_association": "NONE", "pull_request": null, "body": "Hey. Currently i do transformation with the type `--type TEXT`, but i noticed using the sqlalchemy based library [dataset](https://github.com/pudo/dataset) that is reading and writing differ depending on the column types `TEXT`, `DATETIME`.\r\n\r\nIs it possible to alter a column type to `DATETIME` somehow using Sqlite-Utils?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/524/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1323346408, "node_id": "I_kwDOBm6k_c5O4Kno", "number": 1775, "title": "i18n support", "user": {"value": 428820, "label": "johnfelipe"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2022-07-31T02:51:04Z", "updated_at": "2023-02-10T18:04:40Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I want contribute for translate UI to es, de, de and it if you share strings", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1775/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1579695809, "node_id": "I_kwDOBm6k_c5eKD7B", "number": 2023, "title": "Error: Invalid setting 'hash_urls' in settings.json in 0.64.1", "user": {"value": 80409402, "label": "mlaparie"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-02-10T13:35:01Z", "updated_at": "2023-02-10T15:40:00Z", "closed_at": "2023-02-10T15:39:59Z", "author_association": "NONE", "pull_request": null, "body": "On a Debian machine, using datasette 0.64.1 installed with `pip3`, I am getting a `datasette[114272]: Error: Invalid setting 'hash_urls' in settings.json` in `journalctl -xe`. The same settings work on 0.54.1 on another Debian server.\r\n\r\nThis is my `settings.json`:\r\n\r\n```json\r\n{\r\n \"default_page_size\": 200,\r\n \"max_returned_rows\": 8000,\r\n \"num_sql_threads\": 3,\r\n \"sql_time_limit_ms\": 1000,\r\n \"default_facet_size\": 30,\r\n \"facet_time_limit_ms\": 200,\r\n \"facet_suggest_time_limit_ms\": 50,\r\n \"hash_urls\": false,\r\n \"allow_facet\": true,\r\n \"allow_download\": true,\r\n \"suggest_facets\": true,\r\n \"default_cache_ttl\": 5,\r\n \"default_cache_ttl_hashed\": 31536000,\r\n \"cache_size_kb\": 0,\r\n \"allow_csv_stream\": true,\r\n \"max_csv_mb\": 100,\r\n \"truncate_cells_html\": 2048,\r\n \"force_https_urls\": false,\r\n \"template_debug\": false,\r\n \"base_url\": \"/pclim/db/\"\r\n}\r\n```\r\n\r\nThis looks ok to me. Would you have any ideas?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2023/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1578609658, "node_id": "I_kwDOBm6k_c5eF6v6", "number": 2022, "title": "Error 500 - not clear the cause", "user": {"value": 1667631, "label": "DavidPratten"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-02-09T20:57:17Z", "updated_at": "2023-02-09T21:13:50Z", "closed_at": "2023-02-09T21:13:50Z", "author_association": "NONE", "pull_request": null, "body": "On the database that I have sent via linkedIn, datasette works great, but the following URL gives a 500 error.\r\n\r\nhttp://127.0.0.1:8001/literature/authors_papers?authorId=100550354\r\n\r\nThe cause of the error is not apparent.\r\n\r\nIs this expected behaviour?\r\n\r\nDavid", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2022/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1577548579, "node_id": "I_kwDOBm6k_c5eB3sj", "number": 2021, "title": "Docker images for 1.0 alphas?", "user": {"value": 1563881, "label": "meowcat"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-02-09T09:35:52Z", "updated_at": "2023-02-09T09:35:52Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi,\r\nwould you consider putting 1.0alpha images on Dockerhub? \r\n\r\n(Also, how usable are the alphas?)", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2021/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1575880841, "node_id": "I_kwDOBm6k_c5d7giJ", "number": 2020, "title": "Documentation refers to \"off\" setting; doesn't seem to work, \"false\" does", "user": {"value": 1350673, "label": "dmick"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-02-08T10:38:10Z", "updated_at": "2023-02-08T10:38:10Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "https://docs.datasette.io/en/stable/settings.html#suggest-facets, among others, suggests using \"off\" to disable the setting; however, this doesn't appear to work in the JSON config files, where it apparently needs to be a \"JSON boolean\" and have the values \"true\" or \"false\". Perhaps the Python code is more flexible?...but either way, the documentation probably should mention it.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2020/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1571207083, "node_id": "I_kwDOBm6k_c5dprer", "number": 2016, "title": "Database metadata fields like description are not available in the index page template's context", "user": {"value": 9993, "label": "palewire"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 1, "created_at": "2023-02-05T02:25:53Z", "updated_at": "2023-02-05T22:56:43Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "When looping through `databases` in the index.html template, I'd like to print the description of each database alongside its name. But it appears that isn't passed in from the view, unless I'm missing it. It would be great to have that.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2016/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1557599877, "node_id": "I_kwDODFE5qs5c1xaF", "number": 12, "title": "location history changes", "user": {"value": 14809320, "label": "gerardrbentley"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-01-26T03:57:25Z", "updated_at": "2023-01-26T03:57:25Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "not sure if each download is unique, but I had to change some things to work with the takeout zip I made 2023-01-25\r\n\r\nfilename changed from \"Location History.json\" to \"Records.json\"\r\n\r\n`\"timestampMs\"` is not present, `\"timestamp\"` is roughly iso timestamp\r\n\r\n```py\r\ndef get_timestamp_ms(raw_timestamp):\r\n try:\r\n return datetime.datetime.strptime(raw_timestamp, \"%Y-%m-%dT%H:%M:%SZ\").timestamp()\r\n except ValueError:\r\n return datetime.datetime.strptime(raw_timestamp, \"%Y-%m-%dT%H:%M:%S.%fZ\").timestamp()\r\n\r\ndef save_location_history(db, zf):\r\n location_history = json.load(\r\n zf.open(\"Takeout/Location History/Records.json\")\r\n )\r\n db[\"location_history\"].upsert_all(\r\n (\r\n {\r\n \"id\": id_for_location_history(row),\r\n \"latitude\": row[\"latitudeE7\"] / 1e7,\r\n \"longitude\": row[\"longitudeE7\"] / 1e7,\r\n \"accuracy\": row[\"accuracy\"],\r\n \"timestampMs\": get_timestamp_ms(row[\"timestamp\"]),\r\n \"when\": row[\"timestamp\"],\r\n }\r\n for row in location_history[\"locations\"]\r\n ),\r\n pk=\"id\",\r\n )\r\n\r\n\r\ndef id_for_location_history(row):\r\n # We want an ID that is unique but can be sorted by in\r\n # date order - so we use the isoformat date + the first\r\n # 6 characters of a hash of the JSON\r\n first_six = hashlib.sha1(\r\n json.dumps(row, separators=(\",\", \":\"), sort_keys=True).encode(\"utf8\")\r\n ).hexdigest()[:6]\r\n return \"{}-{}\".format(\r\n row['timestamp'],\r\n first_six,\r\n )\r\n```\r\n\r\nexample locations from mine\r\n\r\n```json\r\n{\r\n \"latitudeE7\": 427220206,\r\n \"longitudeE7\": -923423972,\r\n \"accuracy\": 10,\r\n \"deviceTag\": -1312429967,\r\n \"deviceDesignation\": \"PRIMARY\",\r\n \"timestamp\": \"2019-01-08T23:31:50.867Z\"\r\n }\r\n```\r\n\r\n```json\r\n{\r\n \"latitudeE7\": 427011317,\r\n \"longitudeE7\": -923448300,\r\n \"accuracy\": 5,\r\n \"deviceTag\": -1312429967,\r\n \"deviceDesignation\": \"PRIMARY\",\r\n \"timestamp\": \"2019-01-08T23:33:53Z\"\r\n }, \r\n```", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/12/reactions\", \"total_count\": 2, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 2}", "draft": null, "state_reason": null} {"id": 1553615704, "node_id": "I_kwDOBm6k_c5cmktY", "number": 2001, "title": "Datasette is not compatible with SQLite's strict quoting compilation option", "user": {"value": 406380, "label": "gwk"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-01-23T19:10:07Z", "updated_at": "2023-01-25T04:59:58Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I have linked Python3.11 on macOS against recent SQLite that was compiled using `-DSQLITE_DQS=0`. This option disables interpretation of double-quoted identifiers as string literals, described in the SQLite docs as a \"MySQL 3.x misfeature\". See https://www.sqlite.org/quirks.html#dblquote for background.\r\n\r\nDatasette uses the double-quote syntax in a number of key places, and is thus completely broken in this environment.\r\n\r\nMy experience was to `pip install datasette`, then run `datasette serve -I my-data.db`. When I visit `http://127.0.0.1:8001` I get a 500 response.\r\n\r\nThe error: `sqlite3.OperationalError: no such column: geometry_columns`\r\n\r\nThe responsible SQL: `'select 1 from sqlite_master where tbl_name = \"geometry_columns\"'`\r\n\r\nI then installed datasette from GitHub master in development mode and changed the offending SQL to use correct quotes: `\"select 1 from sqlite_master where tbl_name = 'geometry_columns'\"`.\r\n\r\nWith this change, I get a little further, but have the same problem with the first table name in my database (in my case, \"Meta\"):\r\n```\r\nOperationalError: no such column: Meta\r\nTraceback (most recent call last):\r\n File \"/Users/gwk/external/datasette/datasette/app.py\", line 1522, in route_path\r\n response = await view(request, send)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/views/base.py\", line 151, in view\r\n return await self.dispatch_request(request)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/views/base.py\", line 105, in dispatch_request\r\n response = await handler(request)\r\n ^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/views/index.py\", line 70, in get\r\n \"fts_table\": await db.fts_table(table),\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/database.py\", line 363, in fts_table\r\n return await self.execute_fn(lambda conn: detect_fts(conn, table))\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/database.py\", line 213, in execute_fn\r\n return await asyncio.get_event_loop().run_in_executor(\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/py/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/thread.py\", line 58, in run\r\n result = self.fn(*self.args, **self.kwargs)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/database.py\", line 211, in in_thread\r\n return fn(conn)\r\n ^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/database.py\", line 363, in \r\n return await self.execute_fn(lambda conn: detect_fts(conn, table))\r\n ^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/gwk/external/datasette/datasette/utils/__init__.py\", line 588, in detect_fts\r\n rows = conn.execute(detect_fts_sql(table)).fetchall()\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\nsqlite3.OperationalError: no such column: Meta\r\nINFO: 127.0.0.1:50258 - \"GET / HTTP/1.1\" 500 Internal Server Error\r\n```\r\n\r\nI will try to continue playing with this, but I also hope that the datasette developers will enable this mode in a test environment as I am unlikely to be able to exercise all of the SQL in the codebase, or make a pull request very soon.\r\n\r\nNote that the DQS setting compile-time option can be overridden at runtime with calls to the C API:\r\n```\r\nsqlite3_db_config(db, SQLITE_DBCONFIG_DQS_DDL, 0, (void*)0);\r\nsqlite3_db_config(db, SQLITE_DBCONFIG_DQS_DML, 0, (void*)0);\r\n```\r\n\r\nAs far as I can tell, `sqlite3_db_config` is not exposed in Python, but perhaps we could figure out how to invoke it using `ctypes`.\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2001/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1536851861, "node_id": "I_kwDOBm6k_c5bmn-V", "number": 1994, "title": "Stuck on loading screen", "user": {"value": 10913053, "label": "jackhagley"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-01-17T18:33:49Z", "updated_at": "2023-01-23T08:21:08Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Can\u2019t actually open it!\r\n\r\nDownloaded today from the releases tab\r\n\r\nRunning macOS13.1 \r\n\r\n```\r\nbin/python3.9 --version\r\nPython 3.9.6\r\nTook 83ms\r\nbin/python3.9 --version\r\nPython 3.9.6\r\nTook 113ms\r\nbin/pip install datasette>=0.59 datasette-app-support>=0.11.6 datasette-vega>=0.6.2 datasette-cluster-map>=0.17.1 datasette-pretty-json>=0.2.1 datasette-edit-schema>=0.4 datasette-configure-fts>=1.1 datasette-leaflet>=0.2.2 --disable-pip-version-check\r\nRequirement already satisfied: datasette>=0.59 in lib/python3.9/site-packages (0.63)\r\nRequirement already satisfied: datasette-app-support>=0.11.6 in lib/python3.9/site-packages (0.11.6)\r\nRequirement already satisfied: datasette-vega>=0.6.2 in lib/python3.9/site-packages (0.6.2)\r\nRequirement already satisfied: datasette-cluster-map>=0.17.1 in lib/python3.9/site-packages (0.17.2)\r\nRequirement already satisfied: datasette-pretty-json>=0.2.1 in lib/python3.9/site-packages (0.2.2)\r\nRequirement already satisfied: datasette-edit-schema>=0.4 in lib/python3.9/site-packages (0.5.1)\r\nRequirement already satisfied: datasette-configure-fts>=1.1 in lib/python3.9/site-packages (1.1)\r\nRequirement already satisfied: datasette-leaflet>=0.2.2 in lib/python3.9/site-packages (0.2.2)\r\nRequirement already satisfied: click>=7.1.1 in lib/python3.9/site-packages (from datasette>=0.59) (8.1.3)\r\nRequirement already satisfied: hupper>=1.9 in lib/python3.9/site-packages (from datasette>=0.59) (1.10.3)\r\nRequirement already satisfied: pint>=0.9 in lib/python3.9/site-packages (from datasette>=0.59) (0.20.1)\r\nRequirement already satisfied: PyYAML>=5.3 in lib/python3.9/site-packages (from datasette>=0.59) (6.0)\r\nRequirement already satisfied: httpx>=0.20 in lib/python3.9/site-packages (from datasette>=0.59) (0.23.0)\r\nRequirement already satisfied: aiofiles>=0.4 in lib/python3.9/site-packages (from datasette>=0.59) (22.1.0)\r\nRequirement already satisfied: asgi-csrf>=0.9 in lib/python3.9/site-packages (from datasette>=0.59) (0.9)\r\nRequirement already satisfied: asgiref>=3.2.10 in lib/python3.9/site-packages (from datasette>=0.59) (3.5.2)\r\nRequirement already satisfied: uvicorn>=0.11 in lib/python3.9/site-packages (from datasette>=0.59) (0.19.0)\r\nRequirement already satisfied: itsdangerous>=1.1 in lib/python3.9/site-packages (from datasette>=0.59) (2.1.2)\r\nRequirement already satisfied: click-default-group-wheel>=1.2.2 in lib/python3.9/site-packages (from datasette>=0.59) (1.2.2)\r\nRequirement already satisfied: janus>=0.6.2 in lib/python3.9/site-packages (from datasette>=0.59) (1.0.0)\r\nRequirement already satisfied: pluggy>=1.0 in lib/python3.9/site-packages (from datasette>=0.59) (1.0.0)\r\nRequirement already satisfied: Jinja2>=2.10.3 in lib/python3.9/site-packages (from datasette>=0.59) (3.1.2)\r\nRequirement already satisfied: mergedeep>=1.1.1 in lib/python3.9/site-packages (from datasette>=0.59) (1.3.4)\r\nRequirement already satisfied: sqlite-utils in lib/python3.9/site-packages (from datasette-app-support>=0.11.6) (3.30)\r\nRequirement already satisfied: packaging in lib/python3.9/site-packages (from datasette-app-support>=0.11.6) (21.3)\r\nRequirement already satisfied: python-multipart in lib/python3.9/site-packages (from asgi-csrf>=0.9->datasette>=0.59) (0.0.5)\r\nRequirement already satisfied: httpcore<0.16.0,>=0.15.0 in lib/python3.9/site-packages (from httpx>=0.20->datasette>=0.59) (0.15.0)\r\nRequirement already satisfied: certifi in lib/python3.9/site-packages (from httpx>=0.20->datasette>=0.59) (2022.9.24)\r\nRequirement already satisfied: rfc3986[idna2008]<2,>=1.3 in lib/python3.9/site-packages (from httpx>=0.20->datasette>=0.59) (1.5.0)\r\nRequirement already satisfied: sniffio in lib/python3.9/site-packages (from httpx>=0.20->datasette>=0.59) (1.3.0)\r\nRequirement already satisfied: h11<0.13,>=0.11 in lib/python3.9/site-packages (from httpcore<0.16.0,>=0.15.0->httpx>=0.20->datasette>=0.59) (0.12.0)\r\nRequirement already satisfied: anyio==3.* in lib/python3.9/site-packages (from httpcore<0.16.0,>=0.15.0->httpx>=0.20->datasette>=0.59) (3.6.2)\r\nRequirement already satisfied: idna>=2.8 in lib/python3.9/site-packages (from anyio==3.*->httpcore<0.16.0,>=0.15.0->httpx>=0.20->datasette>=0.59) (3.4)\r\nRequirement already satisfied: typing-extensions>=3.7.4.3 in lib/python3.9/site-packages (from janus>=0.6.2->datasette>=0.59) (4.4.0)\r\nRequirement already satisfied: MarkupSafe>=2.0 in lib/python3.9/site-packages (from Jinja2>=2.10.3->datasette>=0.59) (2.1.1)\r\nRequirement already satisfied: tabulate in lib/python3.9/site-packages (from sqlite-utils->datasette-app-support>=0.11.6) (0.9.0)\r\nRequirement already satisfied: python-dateutil in lib/python3.9/site-packages (from sqlite-utils->datasette-app-support>=0.11.6) (2.8.2)\r\nRequirement already satisfied: sqlite-fts4 in lib/python3.9/site-packages (from sqlite-utils->datasette-app-support>=0.11.6) (1.0.3)\r\nRequirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in lib/python3.9/site-packages (from packaging->datasette-app-support>=0.11.6) (3.0.9)\r\nRequirement already satisfied: six>=1.5 in lib/python3.9/site-packages (from python-dateutil->sqlite-utils->datasette-app-support>=0.11.6) (1.16.0)\r\nTook 784ms\r\n```\r\nSTUCK", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1994/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1550536442, "node_id": "I_kwDOCGYnMM5ca076", "number": 521, "title": "Custom JSON encoder", "user": {"value": 31504, "label": "janrito"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-01-20T09:19:40Z", "updated_at": "2023-01-20T09:19:40Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "It would be nice if we could specify a custom encoder (and decoder) for types that will need extra deserialisation \u2013 e.g., sets, enums or sparse matrices \u2013 or even project-specific types", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/521/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1533673397, "node_id": "I_kwDOBm6k_c5baf-1", "number": 1991, "title": "fts5 tables are not auto-detected and hidden", "user": {"value": 83819, "label": "keturn"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-01-15T06:00:42Z", "updated_at": "2023-01-20T04:54:24Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I set up a [Datasette instance](https://huggingface.co/spaces/Sygil/INE-dataset-explorer/tree/main) and was following the docs on full-text search.\r\n\r\nWhen I used fts4, datasette automatically hid the FTS tables and added the FTS search box where appropriate, but when I changed to fts5 it no longer does either.\r\n\r\nIf I [manually set](https://huggingface.co/spaces/keturn/INED-datasette/blob/main/metadata.json#L9) `fts_table` for a view, then search does work as expected.\r\n\r\nMy table and view creation code looks like this:\r\n```py\r\nconnection.execute(\"\"\"CREATE TABLE IF NOT EXISTS\r\n captions(image_key text PRIMARY KEY, caption text NOT NULL)\r\n\"\"\")\r\n\u00a0\r\nconnection.execute(\"\"\"CREATE VIRTUAL TABLE\r\n captions_fts USING\r\n fts5(caption, image_key UNINDEXED, content=captions)\r\n\"\"\")\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1991/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1538197093, "node_id": "I_kwDOBm6k_c5brwZl", "number": 1995, "title": "foreign_keys error 500", "user": {"value": 137183, "label": "jonschoning"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-01-18T15:27:36Z", "updated_at": "2023-01-18T16:44:01Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "**Error 500 expected string or bytes-like object**\r\n\r\n[espial-new.sqlite3.zip](https://github.com/simonw/datasette/files/10447965/espial-new.sqlite3.zip)\r\n\r\nrun `datasette espial-new.sqlite3` & click on any table other than `User`\r\n\r\n```\r\n/home/jon/.local/lib/python3.10/site-packages/datasette/app.py:814 in \u2502\r\n\u2502 expand_foreign_keys \u2502\r\n\u2502 \u2502\r\n\u2502 811 \u2502 \u2502 \u2502 from {other_table} \u2502\r\n\u2502 812 \u2502 \u2502 \u2502 where {other_column} in ({placeholders}) \u2502\r\n\u2502 813 \u2502 \u2502 \"\"\".format( \u2502\r\n\u2502 \u2771 814 \u2502 \u2502 \u2502 other_column=escape_sqlite(fk[\"other_column\"]), \u2502\r\n\u2502 815 \u2502 \u2502 \u2502 label_column=escape_sqlite(label_column), \u2502\r\n\u2502 816 \u2502 \u2502 \u2502 other_table=escape_sqlite(fk[\"other_table\"]), \u2502\r\n\u2502 817 \u2502 \u2502 \u2502 placeholders=\", \".join([\"?\"] * len(set(values))), \u2502\r\n\u2502 \u2502\r\n\u2502 \u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 locals \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e \u2502\r\n\u2502 \u2502 column = 'user_id' \u2502 \u2502\r\n\u2502 \u2502 database = 'espial-new' \u2502 \u2502\r\n\u2502 \u2502 db = \u2502 \u2502\r\n\u2502 \u2502 fk = { \u2502 \u2502\r\n\u2502 \u2502 \u2502 'column': 'user_id', \u2502 \u2502\r\n\u2502 \u2502 \u2502 'other_table': 'user', \u2502 \u2502\r\n\u2502 \u2502 \u2502 'other_column': None \u2502 \u2502\r\n\u2502 \u2502 } \u2502 \u2502\r\n\u2502 \u2502 foreign_keys = [ \u2502 \u2502\r\n\u2502 \u2502 \u2502 { \u2502 \u2502\r\n\u2502 \u2502 \u2502 \u2502 'column': 'user_id', \u2502 \u2502\r\n\u2502 \u2502 \u2502 \u2502 'other_table': 'user', \u2502 \u2502\r\n\u2502 \u2502 \u2502 \u2502 'other_column': None \u2502 \u2502\r\n\u2502 \u2502 \u2502 } \u2502 \u2502\r\n\u2502 \u2502 ] \u2502 \u2502\r\n\u2502 \u2502 label_column = 'name' \u2502 \u2502\r\n\u2502 \u2502 labeled_fks = {} \u2502 \u2502\r\n\u2502 \u2502 self = \u2502 \u2502\r\n\u2502 \u2502 table = 'bookmark' \u2502 \u2502\r\n\u2502 \u2502 values = [] \u2502 \u2502\r\n\u2502 \u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f \u2502\r\n\u2502 \u2502\r\n\u2502 /home/jon/.local/lib/python3.10/site-packages/datasette/utils/__init__.py:346 \u2502\r\n\u2502 in escape_sqlite \u2502\r\n\u2502 \u2502\r\n\u2502 343 \u2502\r\n\u2502 344 \u2502\r\n\u2502 345 def escape_sqlite(s): \u2502\r\n\u2502 \u2771 346 \u2502 if _boring_keyword_re.match(s) and (s.lower() not in reserved_words) \u2502\r\n\u2502 347 \u2502 \u2502 return s \u2502\r\n\u2502 348 \u2502 else: \u2502\r\n\u2502 349 \u2502 \u2502 return f\"[{s}]\" \u2502\r\n\u2502 \u2502\r\n\u2502 \u256d\u2500 locals \u2500\u256e \u2502\r\n\u2502 \u2502 s = None \u2502 \u2502\r\n\u2502 \u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f \u2502\r\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\r\nTypeError: expected string or bytes-like object\r\nTraceback (most recent call last):\r\n File \"/home/jon/.local/lib/python3.10/site-packages/datasette/app.py\", line 1354, in route_path\r\n response = await view(request, send)\r\n File \"/home/jon/.local/lib/python3.10/site-packages/datasette/views/base.py\", line 134, in view\r\n return await self.dispatch_request(request)\r\n File \"/home/jon/.local/lib/python3.10/site-packages/datasette/views/base.py\", line 91, in dispatch_request\r\n return await handler(request)\r\n File \"/home/jon/.local/lib/python3.10/site-packages/datasette/views/base.py\", line 361, in get\r\n response_or_template_contexts = await self.data(request, **data_kwargs)\r\n File \"/home/jon/.local/lib/python3.10/site-packages/datasette/views/table.py\", line 158, in data\r\n return await self._data_traced(request, default_labels, _next, _size)\r\n File \"/home/jon/.local/lib/python3.10/site-packages/datasette/views/table.py\", line 603, in _data_traced\r\n await self.ds.expand_foreign_keys(\r\n File \"/home/jon/.local/lib/python3.10/site-packages/datasette/app.py\", line 814, in expand_foreign_keys\r\n other_column=escape_sqlite(fk[\"other_column\"]),\r\n File \"/home/jon/.local/lib/python3.10/site-packages/datasette/utils/__init__.py\", line 346, in escape_sqlite\r\n if _boring_keyword_re.match(s) and (s.lower() not in reserved_words):\r\nTypeError: expected string or bytes-like object\r\nINFO: 127.0.0.1:38574 - \"GET /espial-new/bookmark HTTP/1.1\" 500 Internal Server Error\r\nINFO: 127.0.0.1:38574 - \"GET /-/static/app.css?d59929 HTTP/1.1\" 200 OK\r\n```\r\n\r\nSchema:\r\n```\r\nCREATE TABLE IF NOT EXISTS \"user\"\r\n (\r\n \"id\" INTEGER PRIMARY KEY,\r\n \"name\" VARCHAR NOT NULL,\r\n \"password_hash\" VARCHAR NOT NULL,\r\n \"api_token\" VARCHAR NULL,\r\n \"private_default\" BOOLEAN NOT NULL,\r\n \"archive_default\" BOOLEAN NOT NULL,\r\n \"privacy_lock\" BOOLEAN NOT NULL,\r\n CONSTRAINT \"unique_user_name\" UNIQUE (\"name\")\r\n );\r\n\r\nCREATE TABLE IF NOT EXISTS \"bookmark\"\r\n (\r\n \"id\" INTEGER PRIMARY KEY,\r\n \"user_id\" INTEGER NOT NULL REFERENCES \"user\" ON DELETE RESTRICT ON UPDATE RESTRICT,\r\n \"slug\" VARCHAR NOT NULL DEFAULT (Lower(Hex(Randomblob(6)))),\r\n \"href\" VARCHAR NOT NULL,\r\n \"description\" VARCHAR NOT NULL,\r\n \"extended\" VARCHAR NOT NULL,\r\n \"time\" TIMESTAMP NOT NULL,\r\n \"shared\" BOOLEAN NOT NULL,\r\n \"to_read\" BOOLEAN NOT NULL,\r\n \"selected\" BOOLEAN NOT NULL,\r\n \"archive_href\" VARCHAR NULL,\r\n CONSTRAINT \"unique_user_href\" UNIQUE (\"user_id\", \"href\"),\r\n CONSTRAINT \"unique_user_slug\" UNIQUE (\"user_id\", \"slug\")\r\n );\r\n\r\nCREATE TABLE IF NOT EXISTS \"bookmark_tag\"\r\n (\r\n \"id\" INTEGER PRIMARY KEY,\r\n \"user_id\" INTEGER NOT NULL REFERENCES \"user\" ON DELETE RESTRICT ON UPDATE RESTRICT,\r\n \"tag\" VARCHAR NOT NULL,\r\n \"bookmark_id\" INTEGER NOT NULL REFERENCES \"bookmark\" ON DELETE RESTRICT ON UPDATE RESTRICT,\r\n \"seq\" INTEGER NOT NULL,\r\n CONSTRAINT \"unique_user_tag_bookmark_id\" UNIQUE (\"user_id\", \"tag\", \"bookmark_id\"),\r\n CONSTRAINT \"unique_user_bookmark_id_tag_seq\" UNIQUE (\"user_id\", \"bookmark_id\", \"tag\", \"seq\")\r\n );\r\n\r\nCREATE TABLE IF NOT EXISTS \"note\"\r\n (\r\n \"id\" INTEGER PRIMARY KEY,\r\n \"user_id\" INTEGER NOT NULL REFERENCES \"user\" ON DELETE RESTRICT ON UPDATE RESTRICT,\r\n \"slug\" VARCHAR NOT NULL DEFAULT (Lower(Hex(Randomblob(10)))),\r\n \"length\" INTEGER NOT NULL,\r\n \"title\" VARCHAR NOT NULL,\r\n \"text\" VARCHAR NOT NULL,\r\n \"is_markdown\" BOOLEAN NOT NULL,\r\n \"shared\" BOOLEAN NOT NULL DEFAULT false,\r\n \"created\" TIMESTAMP NOT NULL,\r\n \"updated\" TIMESTAMP NOT NULL\r\n );\r\nCREATE INDEX idx_bookmark_time ON bookmark (user_id, time DESC);\r\nCREATE INDEX idx_bookmark_tag_bookmark_id ON bookmark_tag (bookmark_id, id, tag, seq);\r\nCREATE INDEX idx_note_user_created ON note (user_id, created DESC); \r\n```\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1995/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1532000914, "node_id": "I_kwDOBm6k_c5bUHqS", "number": 1990, "title": "Suggestion: Highlight error messages ('These facets timed out')", "user": {"value": 116795, "label": "pax"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-01-13T09:40:58Z", "updated_at": "2023-01-13T09:40:58Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I had trouble figuring out why faceting didn't work in some instances, it took a while before I noticed the _These facets timed out_ notice. \r\n\r\nIt might help if that would be highlighted, or fading out highlight - if one might think it would be too visually disturbing.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1990/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1524431805, "node_id": "I_kwDODEm0Qs5a3Pu9", "number": 72, "title": "Import thread, including self- and others' replies", "user": {"value": 601708, "label": "mcint"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-01-08T09:51:06Z", "updated_at": "2023-01-08T09:51:06Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "statuses-lookup, home-timeline, mentions (only for auth'ed user) don't cover this.\r\n\r\n`twitter-to-sqlite fetch-thread tw-group1.db 1234123412341234`\r\n\r\ntwitter-to-sqlite focuses on archiving users, but does not easily support archiving conversations or community activity.\r\n\r\nFor reference, this is [implemented in twarc](https://sourcegraph.com/github.com/DocNow/twarc/-/blob/twarc/client.py?L708-766&subtree=true), using a search, optionally recursively.\r\n\r\nOther research suggests that this formerly, or currently, requires a [search query](https://stackoverflow.com/a/30480103/1020467), use of [undocumented `related_results` api](https://stackoverflow.com/a/9419346/1020467), or with requested inclusion of [newer conversation_id](https://stackoverflow.com/a/68115718/1020467) with subsequent query.\r\n\r\n", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/72/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1501900064, "node_id": "I_kwDOBm6k_c5ZhS0g", "number": 1966, "title": "Broken link to live demo in Getting started docs", "user": {"value": 7551922, "label": "lbellomo"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-12-18T13:17:00Z", "updated_at": "2022-12-31T19:15:19Z", "closed_at": "2022-12-31T19:15:10Z", "author_association": "NONE", "pull_request": null, "body": "The link in [Play with a live demo in Getting started](https://github.com/simonw/datasette/blob/main/docs/getting_started.rst#play-with-a-live-demo) to [https://fivethirtyeight.datasettes.com/fivethirtyeight](https://fivethirtyeight.datasettes.com/fivethirtyeight) is broken and the datasette is no longer working (maybe due to the end of the free tier).", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1966/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1000275035, "node_id": "PR_kwDOCGYnMM4r7n-9", "number": 327, "title": "Extract expand: Support JSON Arrays", "user": {"value": 101753, "label": "phaer"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-09-19T10:34:30Z", "updated_at": "2022-12-29T09:05:36Z", "closed_at": "2022-12-29T09:05:36Z", "author_association": "NONE", "pull_request": "simonw/sqlite-utils/pulls/327", "body": "Hi,\r\n\r\nI needed to extract data in JSON Arrays to normalize data imports. I've quickly hacked the following together based on #241 which refers to #239 where you, @simonw, wrote:\r\n\r\n> Could this handle lists of objects too? That would be pretty amazing - if the column has a [{...}, {...}] list in it could turn that into a many-to-many.\r\n\r\nThey way this works in my work is that many-to-many relationships are created for anything that maps to an dictionary in a list, and many-to-one relations for everything else (assumed to be scalar values). Not sure what the best approach here would be? Are many-to-one relationships are at all useful here?\r\n\r\nWhat do you think about this approach? I could try to add it to the cli interface and documentation if wanted.\r\n\r\nThanks for this awesome piece of software in any case! :sun_with_face: ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/327/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1505411725, "node_id": "I_kwDODFdgUs5ZusKN", "number": 78, "title": "self-hosted or corp github enterprise", "user": {"value": 549431, "label": "ebdavison"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-12-20T22:51:45Z", "updated_at": "2022-12-20T22:51:45Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "We use github enterprise at work and I would like to use this tool to pull info from that site rather than the public github.com instance. Is there an option for this? If not, can one be added for a custom repo URL?", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/78/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 664485022, "node_id": "MDU6SXNzdWU2NjQ0ODUwMjI=", "number": 46, "title": "Feature: pull request reviews and comments", "user": {"value": 1326704, "label": "bhrutledge"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-07-23T13:43:45Z", "updated_at": "2022-12-20T14:40:15Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi there! I saw your [presentation at Boston Python](https://www.meetup.com/bostonpython/events/271887195). I'm already a light user of Datasette (thank you!), but wasn't aware of this project.\r\n\r\nI've been working on a \"pull request dashboard\" to get a comprehensive view of the state of open PR's, esp. related to reviews (i.e., pending, approved, changes requested). Currently it's a CLI command, but I thought a Datasette UI might be fun.\r\n\r\nI see that PR's are available from the `issues` command, but I don't see reviews anywhere. From the [API docs](https://docs.github.com/en/rest/reference/pulls#reviews), it looks like there are separate endpoints for those (as well as pull requests in general). What do you think about adding that? Would you accept a PR? Any sense of the level of effort?", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1504352503, "node_id": "I_kwDOBm6k_c5Zqpj3", "number": 1968, "title": "Allow to hide some queries in metadata.yml", "user": {"value": 562352, "label": "CharlesNepote"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-12-20T10:45:41Z", "updated_at": "2022-12-20T10:45:41Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "By default all queries are displayed.\r\n\r\nBut there are many cases where it would be interesting to hide the queries by default:\r\n* the website is targeting non-tech people\r\n* the query is veeeeeery long ([eg.](https://mirabelle.openfoodfacts.org/products/energy_calculator))\r\n* reading the query is not important for the users, they only want to see the result\r\n\r\nOf course, the user still could have the option to see the query.\r\n\r\nIt could be an option in the metadata file:\r\n```yml\r\ndatabases:\r\n awesome_db:\r\n tables:\r\n products:\r\n hide_sql: true\r\n queries:\r\n great_query:\r\n hide_sql: true\r\n sql: select * from products where code = :barcode\r\n```\r\n\r\nThe priority could be:\r\n* no option in the metadata and nothing in the URL: query displayed\r\n* hide_sql in the metadata and nothing in the URL: query displayed as asked in the metadata\r\n* hide_sql in the metadata and &_hide_sql= in the URL: query as asked in the URL\r\n\r\nSee also: #1824\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1968/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1496652622, "node_id": "I_kwDOBm6k_c5ZNRtO", "number": 1955, "title": "invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things", "user": {"value": 32839123, "label": "Rik-de-Kort"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 36, "created_at": "2022-12-14T13:39:56Z", "updated_at": "2022-12-19T04:34:16Z", "closed_at": "2022-12-18T02:45:18Z", "author_association": "NONE", "pull_request": null, "body": "In the past (pre-september 14, #1809) I had a running deployment of Datasette on Azure WebApps by emulating the call in cli.py to Gunicorn: `gunicorn -w 2 -k uvicorn.workers.UvicornWorker app:app`.\r\n\r\nMy most recent deployment, however, fails loudly by shouting that `Datasette.invoke_startup()` was not called. It does not seem to be possible to call `invoke_startup` when running using a uvicorn command directly like this (I've reproduced this locally using `uvicorn`). Two candidates that I have tried:\r\n* Uvicorn has a `--factory` option, but the app factory has to be synchronous, so no `await invoke_startup` there\r\n* `asyncio.get_event_loop().run_until_complete` is also not an option because `uvicorn` already has the event loop running.\r\n\r\nOne additional option is:\r\n* Use Gunicorn's [server hooks](https://docs.gunicorn.org/en/stable/settings.html#server-hooks) to call `invoke_startup`. These are also synchronous, but I might be able to get ahead of the event loop starting here.\r\n\r\nIn my current deployment setup, it does not appear to be possible to use `datasette serve` directly, so I'm stuck either\r\n* Trying to rework my complete deployment setup, for instance, using Azure functions as described [here](https://github.com/simonw/azure-functions-datasette))\r\n* Or dig into the ASGI spec and write a wrapper for the sole purpose of launching Datasette using a direct Uvicorn invocation.\r\n\r\nQuestions for the maintainers:\r\n* Is this intended behaviour/will not support/etc.? If so, I'd be happy to add a PR with a couple lines in the documentation.\r\n* if this is not intended behaviour, what is a good way to fix it? I could have a go at the ASGI spec thing (I think the Azure Functions thing is related) and provide a PR with the wrapper here, but I'm all ears!\r\n\r\nAlmost forgot, minimal reproducer:\r\n```python\r\nfrom datasette import Datasette\r\n\r\nds = Datasette(files=['./global-power-plants.db'])]\r\napp = ds.app()\r\n```\r\n\r\nSave as app.py in the same folder as global-power-plants.db, and then try running\r\n`uvicorn app:app`.\r\n\r\nOpening the resulting Datasette instance in the browser will show the error message.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1955/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1306984363, "node_id": "I_kwDOBm6k_c5N5v-r", "number": 1771, "title": "minor a11y: