Custom SQL query returning 101 rows (hide)

621989740 MDU6SXNzdWU2MjE5ODk3NDA= 114 table.transform_table() method for advanced alter table 9599 open 0     10 2020-05-20T18:20:46Z 2020-07-08T22:16:54Z   OWNER   SQLite's `ALTER TABLE` can only do the following: * Rename a table * Rename a column * Add a column Notably, it cannot drop columns - so tricks like "add a float version of this text column, populate it, then drop the old one and rename" won't work. The docs here describe a way of implementing full alters safely within a transaction, but it's fiddly. 1. Create new table 2. Copy data 3. Drop old table 4. Rename new into old It would be great if `sqlite-utils` provided an abstraction to help make these kinds of changes safely. 140912432 issue  
653529088 MDU6SXNzdWU2NTM1MjkwODg= 891 Consider using enable_callback_tracebacks(True) 9599 open 0     0 2020-07-08T19:07:16Z 2020-07-08T19:07:16Z   OWNER   From > `sqlite3.``enable_callback_tracebacks`(*flag*)[¶]( "Permalink to this definition") > > By default you will not get any tracebacks in user-defined functions, aggregates, converters, authorizer callbacks etc. If you want to debug them, you can call this function with *flag* set to `True`. Afterwards, you will get tracebacks from callbacks on `sys.stderr`. Use [`False`]( "False") to disable the feature again. Maybe turn this on for all of Datasette? Are there any disadvantages to doing that? 107914493 issue  
652700770 MDU6SXNzdWU2NTI3MDA3NzA= 119 Ability to remove a foreign key 9599 open 0     1 2020-07-07T22:31:37Z 2020-07-08T18:10:18Z   OWNER   Useful if you add one but make a mistake and need to undo it without recreating the database from scratch. 140912432 issue  
652961907 MDU6SXNzdWU2NTI5NjE5MDc= 121 Better documented support for transactions 9599 open 0     2 2020-07-08T04:56:51Z 2020-07-08T18:08:11Z   OWNER   _Originally posted by @simonw in We should put some thought into how this library supports and encourages smart use of transactions. 140912432 issue  
651844316 MDExOlB1bGxSZXF1ZXN0NDQ1MDIzMzI2 118 Add insert --truncate option 79913 closed 0     9 2020-07-06T21:58:40Z 2020-07-08T17:26:21Z 2020-07-08T17:26:21Z CONTRIBUTOR simonw/sqlite-utils/pulls/118 Deletes all rows in the table (if it exists) before inserting new rows. SQLite doesn't implement a TRUNCATE TABLE statement but does optimize an unqualified DELETE FROM. This can be handy if you want to refresh the entire contents of a table but a) don't have a PK (so can't use --replace), b) don't want the table to disappear (even briefly) for other connections, and c) have to handle records that used to exist being deleted. Ideally the replacement of rows would appear instantaneous to other connections by putting the DELETE + INSERT in a transaction, but this is very difficult without breaking other code as the current transaction handling is inconsistent and non-systematic. There exists the possibility for the DELETE to succeed but the INSERT to fail, leaving an empty table. This is not much worse, however, than the current possibility of one chunked INSERT succeeding and being committed while the next chunked INSERT fails, leaving a partially complete operation. 140912432 pull  
652816158 MDExOlB1bGxSZXF1ZXN0NDQ1ODMzOTA4 120 Fix query command's support for DML 79913 closed 0     1 2020-07-08T01:36:34Z 2020-07-08T05:14:04Z 2020-07-08T05:14:04Z CONTRIBUTOR simonw/sqlite-utils/pulls/120 See commit messages for details. I ran into this while investigating another feature/issue. 140912432 pull  
628003707 MDU6SXNzdWU2MjgwMDM3MDc= 784 Ability to sign in to Datasette as a root account 9599 closed 0   5512395 5 2020-05-31T17:10:15Z 2020-07-06T19:31:53Z 2020-06-01T01:18:20Z OWNER   > I'm going to draw the line here: default Datasette supports authentication but only for a single user account ("admin"). Plugins can then add support for multiple user accounts, social auth, SSO etc. _Originally posted by @simonw in 107914493 issue  
651159727 MDU6SXNzdWU2NTExNTk3Mjc= 41 Demo is failing to deploy 9599 closed 0     7 2020-07-05T22:40:33Z 2020-07-06T01:07:03Z 2020-07-06T01:07:02Z MEMBER ``` Creating Revision.........................................................................................................................................failed Deployment failed ERROR: ( Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information. Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.8.3/x64/bin/datasette", line 8, in <module> sys.exit(cli()) File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/", line 829, in __call__ return self.main(*args, **kwargs) File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/", line 782, in main rv = self.invoke(ctx) File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/", line 610, in invoke return callback(*args, **kwargs) File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/publish/", line 138, in cloudrun check_call( File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/", line 364, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'gcloud run deploy --allow-unauthenticated --platform=managed --image github-to-sqlite' returned non-zero exit status 1. ##[error]Process completed with exit code 1. ``` 207052882 issue  
650305298 MDExOlB1bGxSZXF1ZXN0NDQzODIzMDQw 890 Load only python files from plugins-dir. 49260 closed 0     2 2020-07-03T02:47:32Z 2020-07-03T03:08:33Z 2020-07-03T03:08:33Z CONTRIBUTOR simonw/datasette/pulls/890 The current behavior for `--plugins-dir` is to load every file in that folder as a python module. This can result in errors if there are non-python files in the plugins dir (such as .mypy_cache). This PR restricts the module loading to only python files. 107914493 pull  
638270441 MDExOlB1bGxSZXF1ZXN0NDM0MDg1MjM1 848 Reload support for config_dir mode. 49260 closed 0     1 2020-06-14T02:34:46Z 2020-07-03T02:44:54Z 2020-07-03T02:44:53Z CONTRIBUTOR simonw/datasette/pulls/848 A reference implementation for adding support to reload when datasette is in the config_dir mode. This implementation is flawed since it is watching the entire directory and any changes to the database will reload the server and adding unrelated files to the directory will also reload the server. 107914493 pull  
573755726 MDU6SXNzdWU1NzM3NTU3MjY= 690 Mechanism for plugins to add UI to pages in specific locations 9599 open 0   5607421 5 2020-03-02T06:48:36Z 2020-07-02T17:11:25Z   OWNER   Now that we have support for plugins that can write I'm seeing all sorts of places where a plugin might need to add UI to the table page. Some examples: - `datasette-configure-fts` needs to add a "configure search for this table" link - a plugin that lets you render or delete tables needs to add a link or button somewhere - existing plugins like `datasette-vega` and `datasette-cluster-map` already do this with JavaScript The challenge here is that multiple plugins may want to do this, so simply overriding templates and populating names blocks doesn't entirely work as templates may override each other. 107914493 issue  
627794879 MDU6SXNzdWU2Mjc3OTQ4Nzk= 782 Redesign default JSON format in preparation for Datasette 1.0 9599 open 0   5607421 2 2020-05-30T18:47:07Z 2020-07-02T17:11:15Z   OWNER   The default JSON just isn't right. I find myself using `?_shape=array` for almost everything I build against the API. 107914493 issue  
649907676 MDU6SXNzdWU2NDk5MDc2NzY= 889 asgi_wrapper plugin hook is crashing at startup 49260 open 0     2 2020-07-02T12:53:13Z 2020-07-02T13:22:14Z   CONTRIBUTOR   Steps to reproduce: 1. Install datasette-media plugin `pip install datasette-media` 2. Launch datasette `datasette databasename.db` 3. Error ``` INFO: Started server process [927704] INFO: Waiting for application startup. ERROR: Exception in 'lifespan' protocol Traceback (most recent call last): File "/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/uvicorn/lifespan/", line 48, in main await app(scope, self.receive, self.send) File "/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/uvicorn/middleware/", line 45, in __call__ return await, receive, send) File "/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/datasette_media/", line 9, in wrapped_app path = scope["path"] KeyError: 'path' ERROR: Application startup failed. Exiting. ``` 107914493 issue  
649702801 MDU6SXNzdWU2NDk3MDI4MDE= 888 URLs in release notes point to 3243482 open 0     0 2020-07-02T07:28:04Z 2020-07-02T07:28:04Z   CONTRIBUTOR   Just a quick heads up: Release notes for 0.45 include urls that point to localhost. 107914493 issue  
649429772 MDU6SXNzdWU2NDk0Mjk3NzI= 886 Reconsider how _actor_X magic parameter deals with missing values 9599 open 0   5607421 2 2020-07-02T00:00:38Z 2020-07-02T01:52:02Z   OWNER   I had to build a custom `_actorornull` prefix for [datasette-saved-queries]( ```python def actorornull(key, request): if is None: return None return @hookimpl def register_magic_parameters(): return [ ("actorornull", actorornull), ] ``` Maybe the `actor` magic in Datasette core should do that out of the box? 107914493 issue  
649437530 MDU6SXNzdWU2NDk0Mzc1MzA= 887 Canned query page should show the name of the canned query 9599 closed 0   5607421 3 2020-07-02T00:10:39Z 2020-07-02T00:31:33Z 2020-07-02T00:23:45Z OWNER   This page here - the URL is but "all_tables" is not shown in the UI: <img width="784" alt="data__select_sqlite_master_name_as_table_name__table_info___from_sqlite_master_join_pragma_table_info_sqlite_master_name__as_table_info_order_by_sqlite_master_name__table_info_cid_and_data__insert_into_saved_queries__name__sql__author_id__v" src=""> 107914493 issue  
648749062 MDExOlB1bGxSZXF1ZXN0NDQyNTA1MDg4 883 Skip counting hidden tables 3243482 open 0     4 2020-07-01T07:38:08Z 2020-07-02T00:25:44Z   CONTRIBUTOR simonw/datasette/pulls/883 Potential fix for Disabling table counts for hidden tables speeds up database page quite a bit. In my setup it reduced load time by 2/3 (~300 -> ~90ms) 107914493 pull  
276718605 MDU6SXNzdWUyNzY3MTg2MDU= 151 Set up a pattern portfolio 9599 closed 0     2 2017-11-25T02:09:49Z 2020-07-02T00:13:24Z 2020-05-03T03:13:16Z OWNER This will be a single page that demonstrates all of the different CSS styles and classes available to Datasette. 107914493 issue  
647103735 MDU6SXNzdWU2NDcxMDM3MzU= 875 "Logged in as: XXX - logout" navigation item 9599 closed 0   5533512 3 2020-06-29T04:31:14Z 2020-07-02T00:13:24Z 2020-06-29T18:43:50Z OWNER   _Originally posted by @simonw in 107914493 issue  
646992096 MDU6SXNzdWU2NDY5OTIwOTY= 872 Release non-alpha plugins when 0.45 is out 9599 closed 0   5533512 0 2020-06-28T19:42:01Z 2020-07-01T23:48:51Z 2020-07-01T23:48:51Z OWNER   I have several plugins currently marked as alphas because they depend on `0.45a3`. When 0.45 is released I can ship new versions of these plugins that are full releases, not alphas - and switch them to depending on 0.45 (as opposed to the alpha): - [x] - [x] - [x] - [x] 107914493 issue  
649373451 MDU6SXNzdWU2NDkzNzM0NTE= 885 Blog entry about the release 9599 closed 0   5533512 1 2020-07-01T22:44:37Z 2020-07-01T22:44:48Z 2020-07-01T22:44:47Z OWNER     107914493 issue  
632724154 MDU6SXNzdWU2MzI3MjQxNTQ= 805 Writable canned queries live demo on Glitch 9599 open 0     11 2020-06-06T20:52:13Z 2020-07-01T22:44:01Z   OWNER   Needs to run somewhere with a mutable disk drive, so not Cloud Run or Heroku or Vercel. I think I'll put it on Glitch. 107914493 issue  
648673556 MDU6SXNzdWU2NDg2NzM1NTY= 882 Release notes for 0.45 9599 closed 0   5533512 2 2020-07-01T05:00:17Z 2020-07-01T21:48:08Z 2020-07-01T21:48:08Z OWNER   These are mostly done thanks to the alphas, but I went to have more paragraphs of prose and less bullet points. 107914493 issue  
649329013 MDU6SXNzdWU2NDkzMjkwMTM= 884 Only show "log out" button if user is authenticated using a ds_actor cookie 9599 closed 0   5533512 0 2020-07-01T21:21:28Z 2020-07-01T21:26:07Z 2020-07-01T21:26:06Z OWNER   Right now the "Log out" button in the navigation will show up even if the user was authenticated by a plugin using a mechanism other than the `ds_actor` cookie. It should only show if the logged-in user has that cookie. 107914493 issue  
648637666 MDU6SXNzdWU2NDg2Mzc2NjY= 880 POST to /db/canned-query.json should be supported 9599 open 0   5607421 2 2020-07-01T03:14:43Z 2020-07-01T21:06:21Z   OWNER   Now that CSRF is solved for API requests (#835) it would be good to support API requests to the `.json` extension. 107914493 issue  
648421105 MDU6SXNzdWU2NDg0MjExMDU= 877 Consider dropping explicit CSRF protection entirely? 9599 open 0     8 2020-06-30T19:00:55Z 2020-07-01T19:12:16Z   OWNER from Feb 2017 has background here. The `SameSite=lax` cookie property effectively eliminates CSRF in modern browsers. shows 92.13% global support for it. Datasette already uses `SameSite=lax` when it sets cookies by default: A few options then. I could ditch CSRF protection entirely. I could make it optional - turn it off by default, but let users who care about that remaining 7.87% of global users opt back into it. One catch: login CSRF: I don't see how `SameSite=lax` protects against that attack. 107914493 issue  
648659536 MDU6SXNzdWU2NDg2NTk1MzY= 881 Figure out why restore_working_directory is needed in some places 9599 open 0     0 2020-07-01T04:19:25Z 2020-07-01T04:19:25Z   OWNER   This is a frustrating workaround. I have a `restore_working_directory` fixture that I wrote to solve errors that look like this: ``` /Users/simon/Dropbox/Development/datasette/tests/ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ in __enter__ return next(self.gen) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <click.testing.CliRunner object at 0x1135ad110> @contextlib.contextmanager def isolated_filesystem(self): """A context manager that creates a temporary folder and changes the current working directory to it for isolated filesystem tests. """ > cwd = os.getcwd() E FileNotFoundError: [Errno 2] No such file or directory ``` Here's an example of it in use: removing the `restore_working_directory` argument from this function causes the failure. I'd like to not have to do this. 107914493 issue  
634112607 MDU6SXNzdWU2MzQxMTI2MDc= 812 Ability to customize what happens when a view permission fails 9599 closed 0   5533512 3 2020-06-08T04:26:14Z 2020-07-01T04:17:46Z 2020-07-01T04:17:45Z OWNER   Currently view permission failures raise a `Forbidden` error which is transformed into a 403. It would be good if this page could offer a way forward - maybe just by linking to (or redirecting to) a login screen. This behaviour will vary based on authentication plugins, so a new plugin hook is probably the best way to do this. 107914493 issue  
642572841 MDU6SXNzdWU2NDI1NzI4NDE= 859 Database page loads too slowly with many large tables (due to table counts) 3243482 open 0     17 2020-06-21T14:23:17Z 2020-07-01T03:10:21Z   CONTRIBUTOR   Hey, I have a database that I save in HTML from couple of web scrapers. There are around 200k+, 50+ rows in a couple of tables, with sqlite file weighing around 600MB. The app runs on a VPS with 2 core CPU, 4GB RAM and refreshing database page regularly takes more than 10 seconds. I was suspecting that counting tables was the culprit, but manually running `select count(*) from table_name` for the largest table finishes under a second. I've looked at the source code. There's a check for index page for mutable databases larger than 100MB but this check is not performed for database page. I've manually crippled `Database::table_counts` method ```py async def table_counts(self, limit=10): if not self.is_mutable and self.cached_table_counts is not None: return self.cached_table_counts # Try to get counts for each table, $limit timeout for each count counts = {} for table in await self.table_names(): try: # table_count = ( # await self.execute( # "select count(*) from [{}]".format(table), # custom_time_limit=limit, # ) # ).rows[0][0] counts[table] = 10 # table_count # In some cases I saw "SQL Logic Error" here in addition to # QueryInterrupted - so we catch that too: except (QueryInterrupted, sqlite3.OperationalError, sqlite3.DatabaseError): counts[table] = None if not self.is_mutable: self.cached_table_counts = counts return counts ``` now the page loads in <100ms. Is it possible to apply size check on database page too? <details> <summary> /-/versions output </summary> <pre> { "python": { "version": "3.8.0", "full": "3.8.0 (default, Oct 28 2019, 16:14:01) \n[GCC 8.3.0]" }, "datasette": { "version": "0.44" }, "asgi": "3.0", "uvicorn": "0.11.5", "sqlite": { "version": "3.22.0", "fts_versions": [ "FTS5", "FTS4", "FTS3" ], "extensions": { "json1": null }, "compile_options": [ "COMPILER=gcc-7.4.0", "ENABLE_COLUMN_METADATA", "ENABLE_DBSTAT_VTAB", "ENABLE_FTS3", "ENABLE_FTS3_PARENTHESIS", "ENABLE_FTS3_TOKENIZER", "ENABLE_FTS4", "ENABLE_FTS5", "ENABLE_JSON1", "ENABLE_LOAD_EXTENSION", "ENABLE_PREUPDATE_HOOK", "ENABLE_RTREE", "ENABLE_SESSION", "ENABLE_STMTVTAB", "ENABLE_UNLOCK_NOTIFY", "ENABLE_UPDATE_DELETE_LIMIT", "HAVE_ISNAN", "LIKE_DOESNT_MATCH_BLOBS", "MAX_SCHEMA_RETRY=25", "MAX_VARIABLE_NUMBER=250000", "OMIT_LOOKASIDE", "SECURE_DELETE", "SOUNDEX", "TEMP_STORE=1", "THREADSAFE=1" ] } } </pre> </details> 107914493 issue  
637363686 MDU6SXNzdWU2MzczNjM2ODY= 835 Mechanism for skipping CSRF checks on API posts 9599 closed 0   5533512 13 2020-06-11T22:41:10Z 2020-07-01T03:08:07Z 2020-07-01T03:08:07Z OWNER   While experimenting with I realized it's not currently possible to build API client programs that POST to Datasette because there's no mechanism for them to skip the CSRF checks added in #798. 107914493 issue  
647879783 MDU6SXNzdWU2NDc4Nzk3ODM= 876 Add log out link to the pattern portfolio 9599 closed 0   5533512 1 2020-06-30T05:42:15Z 2020-06-30T23:50:04Z 2020-06-30T23:47:31Z OWNER   Follows #875 107914493 issue  
648569227 MDU6SXNzdWU2NDg1NjkyMjc= 879 Database page documentation still talks about hashes in URLs 9599 closed 0   5533512 1 2020-06-30T23:43:17Z 2020-06-30T23:48:06Z 2020-06-30T23:45:42Z OWNER > Note that these URLs end in a 7 character hash. This hash is derived from the contents of the database, and ensures that each URL is immutable: the data returned from a URL containing the hash will always be the same, since if the contents of the database file changes by even a single byte a new hash will be generated. This isn't accurate any more - that's not default behaviour, and it may be removed entirely in #647. 107914493 issue  
636722501 MDU6SXNzdWU2MzY3MjI1MDE= 832 Having view-table permission but NOT view-database should still grant access to /db/table 9599 closed 0   5533512 12 2020-06-11T05:12:59Z 2020-06-30T23:42:11Z 2020-06-30T23:42:11Z OWNER   Stumbled into this while working on `datasette-permissions-sql`. I had granted table permissions, but the permission check wasn't even executed because the user failed the previous `view-database` check. 107914493 issue  
646737558 MDU6SXNzdWU2NDY3Mzc1NTg= 870 Refactor default views to use register_routes 9599 open 0   3268330 10 2020-06-27T18:53:12Z 2020-06-30T19:26:35Z   OWNER   It would be much cleaner if Datasette's default views were all registered using the new `register_routes()` plugin hook. Could dramatically reduce the code in `datasette/`. > The ideal fix here would be to rework my `BaseView` subclass mechanism to work with `register_routes()` so that those views don't have any special privileges above plugin-provided views. _Originally posted by @simonw in 107914493 issue  
648435885 MDU6SXNzdWU2NDg0MzU4ODU= 878 BaseView should be a documented API for plugins to use 9599 open 0   3268330 0 2020-06-30T19:26:13Z 2020-06-30T19:26:26Z   OWNER   Can be part of #870 - refactoring existing views to use `register_routes()`. > I'm going to put the new `check_permissions()` method on `BaseView` as well. If I want that method to be available to plugins I can do so by turning that `BaseView` class into a documented API that plugins are encouraged to use themselves. _Originally posted by @simonw in 107914493 issue  
648245071 MDU6SXNzdWU2NDgyNDUwNzE= 8 Error thrown: table photos has no column named hasSticker 18504 open 0     0 2020-06-30T14:54:37Z 2020-06-30T14:54:37Z   NONE   While running `swarm-to-sqlite` it throws an error: harper@:~/dogsheep/swarm$ swarm-to-sqlite checkins.db --save=checkins.json Please provide your Foursquare OAuth token: Importing 8127 checkins [#################-------------------] 49% 00:01:52 Traceback (most recent call last): File "/home/harper/.local/bin/swarm-to-sqlite", line 11, in <module> sys.exit(cli()) File "/home/harper/.local/lib/python3.6/site-packages/click/", line 829, in __call__ return self.main(*args, **kwargs) File "/home/harper/.local/lib/python3.6/site-packages/click/", line 782, in main rv = self.invoke(ctx) File "/home/harper/.local/lib/python3.6/site-packages/click/", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/harper/.local/lib/python3.6/site-packages/click/", line 610, in invoke return callback(*args, **kwargs) File "/home/harper/.local/lib/python3.6/site-packages/swarm_to_sqlite/", line 73, in cli save_checkin(checkin, db) File "/home/harper/.local/lib/python3.6/site-packages/swarm_to_sqlite/", line 94, in save_checkin photos_table.insert(photo, replace=True) File "/home/harper/.local/lib/python3.6/site-packages/sqlite_utils/", line 963, in insert alter = self.value_or_default("alter", alter) File "/home/harper/.local/lib/python3.6/site-packages/sqlite_utils/", line 1142, in insert_all def upsert_all( sqlite3.OperationalError: table photos has no column named hasSticker Where should i dig in? 205429375 issue  
646448486 MDExOlB1bGxSZXF1ZXN0NDQwNzM1ODE0 868 initial windows ci setup 702729 open 0     2 2020-06-26T18:49:13Z 2020-06-30T03:51:22Z   FIRST_TIME_CONTRIBUTOR simonw/datasette/pulls/868 Picking up the work done on #557 with a new PR. Seeing if I can get this working. 107914493 pull  
647095487 MDU6SXNzdWU2NDcwOTU0ODc= 873 "datasette -p 0 --root" gives the wrong URL 9599 open 0     12 2020-06-29T04:03:06Z 2020-06-29T15:44:54Z   OWNER   ``` $ datasette -p 0 --root ``` The port is incorrect. 107914493 issue  
637966833 MDU6SXNzdWU2Mzc5NjY4MzM= 840 Log out mechanism for clearing ds_actor cookie 9599 closed 0   5533512 4 2020-06-12T19:41:51Z 2020-06-29T04:31:43Z 2020-06-29T04:31:43Z OWNER   Need a cookie clearing mechanism and a way to show that you are logged in. `datasette-auth-github` had a solution for this that can be pulled into core. 107914493 issue  
647095808 MDU6SXNzdWU2NDcwOTU4MDg= 874 /favicon.ico 500 error 9599 closed 0   5533512 0 2020-06-29T04:04:22Z 2020-06-29T04:27:18Z 2020-06-29T04:27:18Z OWNER   ``` Traceback (most recent call last): File "...datasette/datasette/", line 969, in route_path response = await view(request, send) TypeError: favicon() missing 1 required positional argument: 'send' ``` 107914493 issue  
644309017 MDU6SXNzdWU2NDQzMDkwMTc= 864 datasette.add_message() doesn't work inside plugins 9599 closed 0   5533512 6 2020-06-24T04:30:06Z 2020-06-29T00:51:01Z 2020-06-29T00:51:01Z OWNER   Similar problem to #863 - calling `datasette.add_message()` in a view registered using the `register_routes()` plugin hook doesn't work, because the code that writes accumulated messages to the `ds_messages` signed cookie lives in the `BaseView` class here: 107914493 issue  
638259643 MDU6SXNzdWU2MzgyNTk2NDM= 847 Take advantage of .coverage being a SQLite database 9599 closed 0     4 2020-06-14T00:41:25Z 2020-06-28T20:50:21Z 2020-06-28T20:50:21Z OWNER   The `.coverage` file generated by running `pytest-cov` is now a SQLite database! I could do something interesting with this. Maybe after each test run for a new commit I could store that database file somewhere? Lots of interesting challenges here. I got a change into `coveragepy` last year which helps make the custom SQL functions available for doing fun things in Datasette: Bigger challenge: if I have a DB file for every commit, that's hundreds (potentially thousands) of DB files. Datasette isn't designed to handle thousands of files like that. So, do I figure out how to have Datasette open a file on-command for just a single request? Or, an easier option, do I copy data from those files into a single database with a modified schema to include the commit hash in each table row? (Following on from #841 and #844) 107914493 issue  
646840273 MDU6SXNzdWU2NDY4NDAyNzM= 871 Rename the _timestamp magic parameters to _now 9599 closed 0   5533512 1 2020-06-28T04:49:08Z 2020-06-28T19:49:49Z 2020-06-28T19:49:49Z OWNER   I like the shorter name better. Follows on from #842. 107914493 issue  
637342551 MDU6SXNzdWU2MzczNDI1NTE= 834 startup() plugin hook 9599 closed 0   5533512 6 2020-06-11T21:48:14Z 2020-06-28T19:38:50Z 2020-06-13T17:56:12Z OWNER   It might be useful to have an `startup` hook which gets passed the `datasette` object as soon as Datasette has finished initializing. My initial use-case for this is configuration verification - checking that the `"plugins"` configuration block for this plugin contains valid details. I imagine there are plenty of other potential uses for this as well. 107914493 issue  
638212085 MDU6SXNzdWU2MzgyMTIwODU= 842 Magic parameters for canned queries 9599 closed 0   5533512 18 2020-06-13T18:50:08Z 2020-06-28T03:30:31Z 2020-06-28T02:58:18Z OWNER   Now that writable canned queries (#698) have landed, it would be neat if they supported "magic" parameters - parameters that are automatically populated with: - the current actor ID / other actor properties - the current date and time - the user's IP or user-agent And maybe other things potentially added by plugins. 107914493 issue  
646734280 MDExOlB1bGxSZXF1ZXN0NDQwOTQ2ODE3 869 Magic parameters for canned queries 9599 closed 0   5533512 1 2020-06-27T18:37:21Z 2020-06-28T02:58:18Z 2020-06-28T02:58:17Z OWNER simonw/datasette/pulls/869 Implementation for #842 TODO: - [x] Add tests for built-in magic parameters - [x] Magic parameters should not show up as blank form fields on the query page - [x] Update documentation for new `_request_X` (now called `_header_X`) implementation where X is a key from the ASGI scope - [x] Make sure these only work for canned queries, not for arbitrary SQL queries (security issue) - [x] Add test for the `register_magic_parameters` plugin hook - [x] Add documentation for the `register_magic_parameters` plugin hook 107914493 pull  
645975649 MDU6SXNzdWU2NDU5NzU2NDk= 867 register_routes() should support non-async view functions too 9599 closed 0   5533512 1 2020-06-26T03:11:25Z 2020-06-27T18:30:41Z 2020-06-27T18:30:40Z OWNER   I was looking at this: ```python from datasette import hookimpl from datasette.utils.asgi import Response async def robots_txt(): return Response.text("User-agent: *\nDisallow: /") @hookimpl def register_routes(): return [ (r"^/robots\.txt$", robots_txt), ] ``` And I realized that if `register_routes()` could support non-async view functions it could be reduced to this: ```python @hookimpl def register_routes(): return [ (r"^/robots\.txt$", lambda: Response.text("User-agent: *\nDisallow: /")), ] ``` 107914493 issue  
639072811 MDU6SXNzdWU2MzkwNzI4MTE= 849 Rename master branch to main 9599 open 0   3268330 6 2020-06-15T19:05:54Z 2020-06-26T02:09:10Z   OWNER   I was waiting for consensus to form around this (and kind-of hoping for `trunk` since I like the tree metaphor) and it looks like `main` is it. I've seen convincing arguments against `trunk` too - it indicates that the branch has some special significance like in Subversion (where all branches come from trunk) when it doesn't. So `main` is better anyway. 107914493 issue  
644610729 MDExOlB1bGxSZXF1ZXN0NDM5MjAzODA4 866 Update pytest-asyncio requirement from <0.13,>=0.10 to >=0.10,<0.15 27856297 closed 0     1 2020-06-24T13:21:47Z 2020-06-24T18:50:57Z 2020-06-24T18:50:56Z CONTRIBUTOR simonw/datasette/pulls/866 Updates the requirements on [pytest-asyncio]( to permit the latest version. <details> <summary>Commits</summary> <ul> <li><a href=""><code>53f3da7</code></a> Prepare for release</li> <li><a href=""><code>e99569d</code></a> A line is added to the changelog.</li> <li><a href=""><code>4099b63</code></a> One import is not needed</li> <li><a href=""><code>68513b3</code></a> Clarify names and comments, according to yanlend comments 26 May</li> <li><a href=""><code>907e8f2</code></a> FIX new test_cases on python 3.5 &amp; 3.6</li> <li><a href=""><code>51d986c</code></a> To solve test cases that fail:</li> <li><a href=""><code>f97e900</code></a> 1) Test case (test_async_fixtures_with_finalizer) refactoring to pass on pyth...</li> <li><a href=""><code>c1131f8</code></a> 1) A new test case that fails with 0.12.0, and pass with this commit.</li> <li><a href=""><code>7a255bc</code></a> 0.13.0 open for business</li> <li><a href=""><code>b8e2a45</code></a> 0.12.0</li> <li>Additional commits viewable in <a href="">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme Additionally, you can set the following in your Dependabot [dashboard]( - Update frequency (including time of day and day of week) - Pull request limits (per update run and/or open at any time) - Out-of-range updates (receive only lockfile updates, if desired) - Security updates (receive only security updates, if desired) </details> 107914493 pull  
644582921 MDU6SXNzdWU2NDQ1ODI5MjE= 865 base_url doesn't seem to work when adding criteria and clicking "apply" 6739646 open 0     2 2020-06-24T12:39:57Z 2020-06-24T18:43:08Z   NONE   Over on Apache Tika, we're using datasette to allow users to make sense of the metadata for our file regression testing corpus. This could be user error in how I've set up the reverse proxy! I started datasette like so: `docker run -d -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette -p 8001 -h /mnt/corpora-metadata.db --config sql_time_limit_ms:60000 --config base_url:/datasette/` I then reverse proxied like so: ProxyPreserveHost On ProxyPass /datasette http://x.y.z.q:xxxx ProxyPassReverse /datasette http://x.y.z.q:xxx Regular sql works perfectly: However, adding criteria and clicking 'Apply' bounces back to: 107914493 issue  
642388564 MDU6SXNzdWU2NDIzODg1NjQ= 858 publish heroku does not work on Windows 10 870912 open 0     1 2020-06-20T14:40:28Z 2020-06-24T18:42:10Z   NONE   When executing "datasette publish heroku schools.db" on Windows 10, I get the following error ```shell File "c:\users\dell\.virtualenvs\sec-schools-jn-cwk8z\lib\site-packages\datasette\publish\", line 54, in heroku line.split()[0] for line in check_output(["heroku", "plugins"]).splitlines() File "c:\python38\lib\", line 411, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "c:\python38\lib\", line 489, in run with Popen(*popenargs, **kwargs) as process: File "c:\python38\lib\", line 854, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "c:\python38\lib\", line 1307, in _execute_child hp, ht, pid, tid = _winapi.CreateProcess(executable, args, FileNotFoundError: [WinError 2] The system cannot find the file specified ``` Changing to ```python line.split()[0] for line in check_output(["heroku", "plugins"], shell=True).splitlines() ``` as well as the other `check_output()` and `call()` within the same file leads me to another recursive error about temp files 107914493 issue  
637395097 MDU6SXNzdWU2MzczOTUwOTc= 838 Incorrect URLs when served behind a proxy with base_url set 79913 open 0     4 2020-06-11T23:58:55Z 2020-06-24T12:51:48Z   NONE   I'm running `datasette serve --config base_url:/foo/ …`, proxying to it with this Apache config: ProxyPass /foo/ http://localhost:8001/ ProxyPassReverse /foo/ http://localhost:8001/ and then accessing it via ``. Although many of the URLs in the pages are correct (presumably because they either use absolute paths which include `base_url` or relative paths), the faceting and pagination links still use fully-qualified URLs pointing at `http://localhost:8001`. I looked into this a little in the source code, and it seems to be an issue anywhere `request.url` or `request.path` is used, as these contain the values for the request between the frontend (Apache) and backend (Datasette) server. Those properties are primarily used via the `path_with_…` family of utility functions and the `Datasette.absolute_url` method. 107914493 issue  
640943441 MDU6SXNzdWU2NDA5NDM0NDE= 853 Ensure register_routes() works for POST 9599 closed 0   5533512 1 2020-06-18T06:24:55Z 2020-06-24T04:30:30Z 2020-06-18T16:22:02Z OWNER 107914493 issue  
644283211 MDU6SXNzdWU2NDQyODMyMTE= 863 {{ csrftoken() }} doesn't work with datasette.render_template() 9599 closed 0   5533512 0 2020-06-24T03:11:49Z 2020-06-24T04:30:30Z 2020-06-24T03:24:01Z OWNER   The documentation here suggests that it will work: But right now the `csrftoken` variable is set in BaseView.render, which means it's not visible to plugins that try to render templates using `datasette.render_template`: 107914493 issue  
644161221 MDU6SXNzdWU2NDQxNjEyMjE= 117 Support for compound (composite) foreign keys 9599 open 0     3 2020-06-23T21:33:42Z 2020-06-23T21:40:31Z   OWNER   It turns out SQLite supports composite foreign keys: Their example looks like this: ```sql CREATE TABLE album( albumartist TEXT, albumname TEXT, albumcover BINARY, PRIMARY KEY(albumartist, albumname) ); CREATE TABLE song( songid INTEGER, songartist TEXT, songalbum TEXT, songname TEXT, FOREIGN KEY(songartist, songalbum) REFERENCES album(albumartist, albumname) ); ``` Here's what that looks like in sqlite-utils: ``` In [1]: import sqlite_utils In [2]: import sqlite3 In [3]: conn = sqlite3.connect(":memory:") In [4]: conn Out[4]: <sqlite3.Connection at 0x1087186c0> In [5]: conn.executescript(""" ...: CREATE TABLE album( ...: albumartist TEXT, ...: albumname TEXT, ...: albumcover BINARY, ...: PRIMARY KEY(albumartist, albumname) ...: ); ...: ...: CREATE TABLE song( ...: songid INTEGER, ...: songartist TEXT, ...: songalbum TEXT, ...: songname TEXT, ...: FOREIGN KEY(songartist, songalbum) REFERENCES album(albumartist, albumname) ...: ); ...: """) Out[5]: <sqlite3.Cursor at 0x1088def10> In [6]: db = sqlite_utils.Database(conn) In [7]: db.tables Out[7]: [<Table album (albumartist, albumname, albumcover)>, <Table song (songid, songartist, songalbum, songname)>] In [8]: db.tables[0].foreign_keys Out[8]: [] In [9]: db.tables[1].foreign_keys Out[9]: [ForeignKey(table='song', column='songartist', other_table='album', other_column='albumartist'), ForeignKey(table='song', column='songalbum', other_table='album', other_column='albumname')] ``` The table appears to have two separate foreign keys, when actually it has a single compound composite foreign key. 140912432 issue  
644122661 MDU6SXNzdWU2NDQxMjI2NjE= 116 Documentation for table.pks introspection property 9599 closed 0     2 2020-06-23T20:27:24Z 2020-06-23T21:21:33Z 2020-06-23T21:03:14Z OWNER 140912432 issue  
576582604 MDU6SXNzdWU1NzY1ODI2MDQ= 694 datasette publish cloudrun --memory option 9599 closed 0     8 2020-03-05T22:59:57Z 2020-06-23T17:10:51Z 2020-03-05T23:49:41Z OWNER   Got this error deploying large (603MB) database with Cloud Run ``` X Deploying... Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revi sion might contain more information. X Creating Revision... Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information. . Routing traffic... ✓ Setting IAM Policy... Deployment failed ERROR: ( Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information. ``` 107914493 issue  
507454958 MDU6SXNzdWU1MDc0NTQ5NTg= 596 Handle really wide tables better 9599 open 0     4 2019-10-15T20:05:46Z 2020-06-23T03:59:52Z   OWNER   If a table has hundreds of columns the Datasette UI starts getting unwieldy. Addressing this would be neat. One option would be to only select the first 30 columns by default and provide a UI for selecting more. 107914493 issue  
643510821 MDU6SXNzdWU2NDM1MTA4MjE= 862 Set an upper limit on total facet suggestion time for a page 9599 open 0     1 2020-06-23T03:57:55Z 2020-06-23T03:58:48Z   OWNER   If a table has 100 columns the facet suggestion code will currently run 100 times, taking a max of `facet_suggest_time_limit_ms` which defaults to 50ms per column: So for 100 columns, that's 100 * 50ms = 5s total time that might be spent attempting to calculate facets on a large table! I should implement a hard upper limit on the total amount of time taken suggesting facets - probably of around 500ms. If it takes longer than that the remaining columns will not be considered. 107914493 issue  
642652808 MDU6SXNzdWU2NDI2NTI4MDg= 861 Script to generate larger SQLite test files 9599 closed 0     3 2020-06-21T22:30:58Z 2020-06-23T03:44:18Z 2020-06-23T03:44:18Z OWNER   > I'll write a little script which generates a 300MB SQLite file with a bunch of tables with lots of randomly generated rows in to help test this. > > Having a tool like that which can generate larger databases with different gnarly performance characteristics will be useful for other performance work too. _Originally posted by @simonw in 107914493 issue  
572896293 MDU6SXNzdWU1NzI4OTYyOTM= 687 Expand plugins documentation to multiple pages 9599 closed 0   5533512 11 2020-02-28T17:26:21Z 2020-06-22T03:55:20Z 2020-06-22T03:53:54Z OWNER   I think the plugins docs need to extend beyond a single page now. I want to add a whole section on writing tests for plugins, showing how `httpx` can be used as seen in and suchlike. 107914493 issue  
642127307 MDU6SXNzdWU2NDIxMjczMDc= 855 Add instructions for using cookiecutter plugin template to plugin docs 9599 closed 0   5533512 2 2020-06-19T17:33:25Z 2020-06-22T02:51:38Z 2020-06-22T02:51:38Z OWNER   Once I ship the `datasette-plugin` template: 107914493 issue  
642651572 MDU6SXNzdWU2NDI2NTE1NzI= 860 Plugin hook for database/table metadata 9599 open 0     1 2020-06-21T22:20:25Z 2020-06-21T22:25:27Z   OWNER   I'm not happy with how `metadata.(json|yaml)` keeps growing new features. Rather than having a single plugin hook for all of `metadata.json` I'm going to split out the feature that shows actual real metadata for tables and databases - `source`, `license` etc - into its own plugin-powered mechanism. _Originally posted by @simonw in 107914493 issue  
348043884 MDU6SXNzdWUzNDgwNDM4ODQ= 357 Plugin hook for loading metadata.json 9599 open 0     6 2018-08-06T19:00:01Z 2020-06-21T22:19:58Z   OWNER   For I wrote a script to convert YAML to JSON because YAML is a better format for embedding multi-line HTML descriptions and canned SQL statements. Example yaml metadata file: It would be useful if Datasette could be fed a YAML file directly: datasette -m metadata.yaml Question is... should this be a native feature (hence adding a YAML dependency) or should it be handled by a `datasette-metadata-yaml` plugin, using a new plugin hook for loading metadata? If so, what would other use-cases for that plugin hook be? 107914493 issue  
529429214 MDU6SXNzdWU1Mjk0MjkyMTQ= 642 Provide a cookiecutter template for creating new plugins 9599 closed 0   3268330 6 2019-11-27T15:46:36Z 2020-06-20T03:20:33Z 2020-06-20T03:20:25Z OWNER   See this conversation: 107914493 issue  
642297505 MDU6SXNzdWU2NDIyOTc1MDU= 857 Comprehensive documentation for variables made available to templates 9599 open 0   3268330 0 2020-06-20T03:19:43Z 2020-06-20T03:19:44Z   OWNER   Needed for the Datasette 1.0 release, so template authors can trust that Datasette is unlikely to break their templates. 107914493 issue  
642296989 MDU6SXNzdWU2NDIyOTY5ODk= 856 Consider pagination of canned queries 9599 open 0     0 2020-06-20T03:15:59Z 2020-06-20T03:15:59Z   OWNER   The new `canned_queries()` plugin hook from #852 combined with plugins like could mean that some installations end up with hundreds or even thousands of canned queries. I should consider pagination or some other way of ensuring that this doesn't cause performance problems for Datasette. 107914493 issue  
640917326 MDU6SXNzdWU2NDA5MTczMjY= 852 canned_queries() plugin hook 9599 closed 0   5533512 9 2020-06-18T05:24:35Z 2020-06-20T03:08:40Z 2020-06-20T03:08:40Z OWNER   Canned queries are currently baked into `metadata.json` which is read once on startup. Allowing users to interactively create new canned queries - even if just through a plugin - would make a lot of sense. Is this a new plugin hook or some other mechanism? Lots to think about here. 107914493 issue  
632843030 MDU6SXNzdWU2MzI4NDMwMzA= 807 Ability to ship alpha and beta releases 9599 closed 0   5533512 18 2020-06-07T00:12:55Z 2020-06-18T21:41:16Z 2020-06-18T21:41:16Z OWNER   I'd like to be able to ship alphas and betas to PyPI so in-development plugins can depend on them and help test unreleased plugin hooks. 107914493 issue  
641460179 MDU6SXNzdWU2NDE0NjAxNzk= 854 Respect default scope["actor"] if one exists 9599 closed 0   5533512 0 2020-06-18T18:25:08Z 2020-06-18T18:39:22Z 2020-06-18T18:39:22Z OWNER   ASGI wrapper plugins that themselves set the `actor` scope variable should be respected (though `actor_from_request` plugins should still execute and get the chance to replace that initial `actor` value). Relevant code: 107914493 issue  
635049296 MDU6SXNzdWU2MzUwNDkyOTY= 820 Idea: Plugin hook for registering canned queries 9599 closed 0     2 2020-06-09T01:58:21Z 2020-06-18T17:58:02Z 2020-06-18T17:58:02Z OWNER   Thought of this while thinking about possible permissions plugins (#818). Imagine an API key plugin which allows access for API keys. It could let users register new API keys by providing a writable canned query for writing to the `api_keys` table. To do this the plugin needs to register the query. At the moment queries have to be registered in `metadata.json` - a plugin hook for registering additional queries could help solve this. One challenge: how does the plugin know which named database the query should be registered for? It could default to the first attached database and allow users to optionally tell the plugin "actually use this named database instead" in plugin configuration. 107914493 issue  
639542974 MDU6SXNzdWU2Mzk1NDI5NzQ= 47 Fall back to FTS4 if FTS5 is not available 73579 open 0     3 2020-06-16T10:11:23Z 2020-06-17T20:13:48Z   NONE   got this with version 0.21.1 from pypi. twitter-to-sqlite auth worked but then "twitter-to-sqlite user-timeline USER.db" produced a tracekback ending in "no such module: FTS5". 206156866 issue  
640330278 MDU6SXNzdWU2NDAzMzAyNzg= 851 Having trouble getting writable canned queries to work 3243482 closed 0     1 2020-06-17T10:30:28Z 2020-06-17T10:33:25Z 2020-06-17T10:32:33Z CONTRIBUTOR   Hey, I'm trying to get canned inserts to work. I have an DB with following metadata: ```text sqlite> .mode line sqlite> select name, sql from sqlite_master where name like '%search%'; name = search sql = CREATE TABLE "search" ("id" INTEGER NOT NULL PRIMARY KEY, "name" VARCHAR(255) NOT NULL, "url" VARCHAR(255) NOT NULL) ``` ```yaml # ... queries: add_search: sql: insert into search(name, url) VALUES (:name, :url), write: true ``` which renders a form as expected, but when I submit the form I get `incomplete input` error. ![image]( but when submit post the form I've attached a debugger to see where the error comes from, because `incomplete input` string doesn't appear in datasette codebase. Inside `datasette.database.Database.execute_write_fn` ```py result = await reply_queue.async_q.get() ``` this line raises an exception. That led me to believe I had something wrong with my SQL. But running the command in `sqlite3` inserts the record just fine. ```text sqlite> insert into search (name, url) values ('my name', 'my url'); sqlite> SELECT last_insert_rowid(); last_insert_rowid() = 3 ``` So I'm a bit lost here. --- - datasette, version 0.44 - Python 3.8.1 107914493 issue  
639993467 MDU6SXNzdWU2Mzk5OTM0Njc= 850 Proof of concept for Datasette on AWS Lambda with EFS 9599 open 0     25 2020-06-16T21:48:31Z 2020-06-16T23:52:16Z   OWNER If Datasette can run on Lambda with access to EFS it could both read AND write large databases there. 107914493 issue  
317001500 MDU6SXNzdWUzMTcwMDE1MDA= 236 datasette publish lambda plugin 9599 open 0     4 2018-04-23T22:10:30Z 2020-06-16T23:50:59Z   OWNER   Refs #217 - create a publish plugin that can deploy to AWS Lambda. says lambda packages can be up to 50 MB, so this would only work with smaller databases (the command can check the filesize before attempting to package and deploy it). Lambdas do get a 512 MB `/tmp` directory too, so for larger databases the function could start and then download up to 512MB from an S3 bucket - so the plugin could take an optional S3 bucket to write to and know how to upload the `.db` file there and then have the lambda download it on startup. 107914493 issue  
638375985 MDExOlB1bGxSZXF1ZXN0NDM0MTYyMzE2 29 Fixed bug in SQL query for photo scores 41546558 open 0     0 2020-06-14T15:39:22Z 2020-06-14T15:39:22Z   FIRST_TIME_CONTRIBUTOR dogsheep/dogsheep-photos/pulls/29 The join on ZCOMPUTEDASSETATTRIBUTES used the wrong columns. In most of the Photos database tables, table.ZASSET joins with ZGENERICASSET.Z_PK 256834907 pull  
574021194 MDU6SXNzdWU1NzQwMjExOTQ= 691 --reload sould reload server if code in --plugins-dir changes 9599 open 0     1 2020-03-02T14:42:21Z 2020-06-14T02:35:17Z   OWNER     107914493 issue  
638241779 MDU6SXNzdWU2MzgyNDE3Nzk= 846 "Too many open files" error running tests 9599 closed 0     6 2020-06-13T22:11:40Z 2020-06-14T00:26:31Z 2020-06-14T00:26:31Z OWNER   I got this on my laptop: ```pytest ... /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.7/site-packages/jinja2/ in get_source f = open_if_exists(filename) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ filename = '/Users/simon/Dropbox/Development/datasette/datasette/templates/400.html', mode = 'rb' def open_if_exists(filename, mode='rb'): """Returns a file descriptor for the filename if that file exists, otherwise `None`. """ try: > return open(filename, mode) E OSError: [Errno 24] Too many open files: '/Users/simon/Dropbox/Development/datasette/datasette/templates/400.html' /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.7/site-packages/jinja2/ OSError ``` Based on the conversation in I'm worried that my tests are opening too many files without closing them. In particular... I call `sqlite3.connect(filepath)` a LOT - and I don't ever call `conn.close()` on those opened connections: Could this be resulting in my tests eventually opening too many unclosed file handles? How could I confirm this? 107914493 issue  
638238548 MDU6SXNzdWU2MzgyMzg1NDg= 845 Code coverage should ignore files in .coveragerc 9599 open 0     0 2020-06-13T21:45:42Z 2020-06-13T21:46:03Z   OWNER   I'm not sure why this is, but the code coverage I have running in a GitHub Action doesn't take my `.coveragerc` file into account. It should: Here's the bit that's ignored: As a result my coverage score is 84%, when it should be 92%: ``` 2020-06-13T21:41:18.4404252Z ----------- coverage: platform linux, python 3.8.3-final-0 ----------- 2020-06-13T21:41:18.4404570Z Name Stmts Miss Cover 2020-06-13T21:41:18.4404971Z -------------------------------------------------------- 2020-06-13T21:41:18.4405227Z datasette/ 3 0 100% 2020-06-13T21:41:18.4405441Z datasette/ 3 3 0% 2020-06-13T21:41:18.4405668Z datasette/ 279 279 0% 2020-06-13T21:41:18.4405921Z datasette/ 20 0 100% 2020-06-13T21:41:18.4406135Z datasette/ 499 27 95% 2020-06-13T21:41:18.4406343Z datasette/ 162 45 72% 2020-06-13T21:41:18.4406553Z datasette/ 236 17 93% 2020-06-13T21:41:18.4406761Z datasette/ 40 0 100% 2020-06-13T21:41:18.4406975Z datasette/ 210 24 89% 2020-06-13T21:41:18.4407186Z datasette/ 122 7 94% 2020-06-13T21:41:18.4407394Z datasette/ 34 0 100% 2020-06-13T21:41:18.4407600Z datasette/ 36 23 36% 2020-06-13T21:41:18.4407807Z datasette/ 34 6 82% 2020-06-13T21:41:18.4408014Z datasette/publish/ 0 0 100% 2020-06-13T21:41:18.4408240Z datasette/publish/ 57 2 96% 2020-06-13T21:41:18.4408786Z datasette/publish/ 19 1 95% 2020-06-13T21:41:18.4409029Z datasette/publish/ 97 13 87% 2020-06-13T21:41:18.4409243Z datasette/ 63 4 94% 2020-06-13T21:41:18.4409450Z datasette/ 5 0 100% 2020-06-13T21:41:18.4410480Z datasette/ 87 16 82% 2020-06-13T21:41:18.4410972Z datasette/utils/ 504 31 94% 2020-06-13T21:41:18.4411755Z datasette/utils/ 264 24 91% 2020-06-13T21:41:18.4412173Z datasette/utils/ 44 44 0% 2020-06-13T21:41:18.4412822Z datasette/ 4 0 100% 2020-06-13T21:41:18.4413562Z datasette/views/ 0 0 100% 2020-06-13T21:41:18.4414276Z datasette/views/ 288 19 93% 2020-06-13T21:41:18.4414579Z datasette/views/ 120 2 98% 2020-06-13T21:41:18.4414860Z datasette/views/ 57 2 96% 2020-06-13T21:41:18.4415379Z datasette/views/ 72 16 78% 2020-06-13T21:41:18.4418994Z datasette/views/ 418 18 96% 2020-06-13T21:41:18.4428811Z -------------------------------------------------------- 2020-06-13T21:41:18.4430394Z TOTAL 3777 623 84% ``` 107914493 issue  
638104520 MDU6SXNzdWU2MzgxMDQ1MjA= 841 Research feasibility of 100% test coverage 9599 closed 0     9 2020-06-13T06:07:01Z 2020-06-13T21:38:46Z 2020-06-13T21:38:46Z OWNER   Inspired by > Almost every library I’ve written in the last 2 years has had 100% coverage and that’s probably not going to change in the future. It’s not that hard to start at 100% and hold onto it and the workflow it enables is so much nicer. 107914493 issue  
638229448 MDU6SXNzdWU2MzgyMjk0NDg= 843 Configure 9599 closed 0     2 2020-06-13T20:45:00Z 2020-06-13T21:36:52Z 2020-06-13T21:36:52Z OWNER   _Originally posted by @simonw in 107914493 issue  
638230433 MDExOlB1bGxSZXF1ZXN0NDM0MDU1NzUy 844 Action to run tests and upload coverage report 9599 closed 0     1 2020-06-13T20:52:47Z 2020-06-13T21:36:52Z 2020-06-13T21:36:50Z OWNER simonw/datasette/pulls/844 Refs #843 107914493 pull  
637899539 MDU6SXNzdWU2Mzc4OTk1Mzk= 40 Demo deploy is broken 9599 closed 0     2 2020-06-12T17:20:17Z 2020-06-12T18:06:48Z 2020-06-12T18:06:48Z MEMBER ``` The following NEW packages will be installed: sqlite3 0 upgraded, 1 newly installed, 0 to remove and 11 not upgraded. Need to get 752 kB of archives. After this operation, 2482 kB of additional disk space will be used. Ign:1 bionic-updates/main amd64 sqlite3 amd64 3.22.0-1ubuntu0.3 Err:1 bionic-updates/main amd64 sqlite3 amd64 3.22.0-1ubuntu0.3 404 Not Found [IP: 80] E: Failed to fetch 404 Not Found [IP: 80] E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing? ##[error]Process completed with exit code 100. ``` 207052882 issue  
637889964 MDU6SXNzdWU2Mzc4ODk5NjQ= 115 Ability to execute insert/update statements with the CLI 9599 closed 0     1 2020-06-12T17:01:17Z 2020-06-12T17:51:11Z 2020-06-12T17:41:10Z OWNER   ``` $ sqlite-utils github.db "update stars set starred_at = ''" Traceback (most recent call last): File "/Users/simon/.local/bin/sqlite-utils", line 8, in <module> sys.exit(cli()) File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/", line 829, in __call__ return self.main(*args, **kwargs) File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/", line 782, in main rv = self.invoke(ctx) File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/", line 610, in invoke return callback(*args, **kwargs) File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/", line 673, in query headers = [c[0] for c in cursor.description] TypeError: 'NoneType' object is not iterable ``` 140912432 issue  
632753851 MDU6SXNzdWU2MzI3NTM4NTE= 806 Release Datasette 0.44 9599 closed 0   5512395 10 2020-06-06T21:49:52Z 2020-06-12T01:20:03Z 2020-06-12T01:20:03Z OWNER   See also [milestone]( This is a pretty big release: flash messaging, writable canned queries, authentication and permissions! I'll want to ship some plugin releases in conjunction with this - `datasette-auth-github` for example. 107914493 issue  
637409144 MDU6SXNzdWU2Mzc0MDkxNDQ= 839 {"$file": ...} mechanism is broken 9599 closed 0   5512395 0 2020-06-12T00:46:24Z 2020-06-12T00:48:26Z 2020-06-12T00:48:26Z OWNER ``` def test_plugin_config_file(app_client): open(TEMP_PLUGIN_SECRET_FILE, "w").write("FROM_FILE") > assert {"foo": "FROM_FILE"} == app_client.ds.plugin_config("file-plugin") E AssertionError: assert {'foo': 'FROM_FILE'} == {'foo': {'$fi...ugin-secret'}} E Differing items: E {'foo': 'FROM_FILE'} != {'foo': {'$file': '/tmp/plugin-secret'}} E Use -v to get the full diff ``` Broken in as part of #837 107914493 issue  
637370652 MDU6SXNzdWU2MzczNzA2NTI= 837 Plugin $env secrets mechanism doesn't work inside lists 9599 closed 0   5512395 0 2020-06-11T22:59:54Z 2020-06-12T00:25:20Z 2020-06-12T00:25:19Z OWNER   This didn't work: ```json { "plugins": { "datasette-auth-tokens": [ { "token": { "$env": "BOT_TOKEN" }, "actor": { "bot_id": "my-bot" } } ] } } ``` 107914493 issue  
635108074 MDU6SXNzdWU2MzUxMDgwNzQ= 824 Example authentication plugin 9599 closed 0   5512395 4 2020-06-09T04:49:53Z 2020-06-12T00:11:51Z 2020-06-12T00:11:50Z OWNER will work for this. 107914493 issue  
637365801 MDU6SXNzdWU2MzczNjU4MDE= 836 actor_matches_allow fails to consider all keys 9599 closed 0   5512395 0 2020-06-11T22:46:34Z 2020-06-11T22:47:25Z 2020-06-11T22:47:25Z OWNER   actor: `{"id": "root"}` allow block: `{"bot_id": "my-bot", "id": ["root"]}` This should pass, because the `id` matches - but it fails. 107914493 issue  
637253789 MDU6SXNzdWU2MzcyNTM3ODk= 833 /-/metadata and so on should respect view-instance permission 9599 closed 0   5512395 4 2020-06-11T19:07:21Z 2020-06-11T22:15:32Z 2020-06-11T22:14:59Z OWNER   The only URLs that should be available without authentication at all times are the `/-/static/` prefix, to allow for HTTP caching. 107914493 issue  
314847571 MDU6SXNzdWUzMTQ4NDc1NzE= 220 Investigate syntactic sugar for plugins 9599 closed 0     2 2018-04-16T23:01:39Z 2020-06-11T21:50:06Z 2020-06-11T21:49:55Z OWNER   Suggested by @andrewhayward on Twitter: > Have you considered a basic abstraction on top of that, for standard hook features? ``` @sql_function random_integer(a,b): return random.randint(a,b) @template_filter uppercase(str): return str.upper() ``` Maybe `from datasette.plugins import template_filter`? Would have to work out how to get this to play well with pluggy 107914493 issue  
631932926 MDU6SXNzdWU2MzE5MzI5MjY= 801 allow_by_query setting for configuring permissions with a SQL statement 9599 closed 0   3268330 6 2020-06-05T20:30:19Z 2020-06-11T18:58:56Z 2020-06-11T18:58:49Z OWNER   > Idea: an `"allow_sql"` key with a SQL query that gets passed the actor JSON as `:actor` and can extract the relevant keys from it and return 1 or 0. _Originally posted by @simonw in See also #800 107914493 issue  
614806683 MDExOlB1bGxSZXF1ZXN0NDE1Mjg2MTA1 763 Documentation + improvements for db.execute() and Results class 9599 closed 0     0 2020-05-08T15:16:02Z 2020-06-11T16:05:48Z 2020-05-08T16:05:46Z OWNER simonw/datasette/pulls/763 Refs #685 Still TODO: - [x] Implement `results.first()` - [x] Implement `results.single_value()` - [x] Unit tests for the above 107914493 pull  
632919570 MDExOlB1bGxSZXF1ZXN0NDI5NjEzODkz 809 Publish secrets 9599 closed 0   5512395 4 2020-06-07T02:00:31Z 2020-06-11T16:02:13Z 2020-06-11T16:02:03Z OWNER simonw/datasette/pulls/809 Refs #787. Will need quite a bit of manual testing since this involves code which runs against Heroku and Cloud Run. 107914493 pull  
628089318 MDU6SXNzdWU2MjgwODkzMTg= 787 "datasette publish" should bake in a random --secret 9599 closed 0   5512395 1 2020-06-01T01:15:26Z 2020-06-11T16:02:05Z 2020-06-11T16:02:05Z OWNER   To allow signed cookies etc to work reliably (see #785) all of the `datasette publish` commands should generate a random secret on publish and bake it into the configuration - probably by setting the `DATASETTE_SECRET` environment variable. - [ ] Cloud Run - [ ] Heroku - [ ] - [ ] 107914493 issue  
396212021 MDU6SXNzdWUzOTYyMTIwMjE= 394 base_url configuration setting 9599 closed 0   5234079 27 2019-01-05T23:48:48Z 2020-06-11T09:15:20Z 2020-03-25T00:18:45Z OWNER   I've identified a couple of use-cases for running Datasette in a way that over-rides the default way that internal URLs are generated. 1. Running behind a reverse proxy. I tried running Datasette behind a proxy and found that some of the generated internal links incorrectly referenced `` - when they should have been referencing `` - this is a problem both for links within the HTML interface but also for the `toggle_url` keys returned in the JSON as part of the facets datastructure. 2. I would like it to be possible to host a Datasette instance at e.g. `` - either through careful HTTP proxying or, once Datasette has been ported to ASGI, by mounting a Datasette ASGI instance deep within an existing set of URL routes. I'm going to add a `url_prefix` configuration option. This will default to `""`, which means Datasette will behave as it does at the moment - it will use `/` for most URL prefixes in the HTML version, and an absolute URL derived from the incoming `Host` header for URLs that are returned as part of the JSON output. If `url_prefix` is set to another value (either a full URL or a path) then this path will be appended to all generated URLs. 107914493 issue  
634917088 MDU6SXNzdWU2MzQ5MTcwODg= 818 Example permissions plugin 9599 closed 0   5512395 9 2020-06-08T20:35:56Z 2020-06-11T05:40:07Z 2020-06-11T05:40:07Z OWNER   To show how they work. Also useful to confirm how they interact with the default permissions. 107914493 issue  
636614868 MDU6SXNzdWU2MzY2MTQ4Njg= 831 It would be more intuitive if "allow": none meant "no-one can do this" 9599 closed 0   5512395 1 2020-06-10T23:43:56Z 2020-06-10T23:57:25Z 2020-06-10T23:50:55Z OWNER   Now that I'm starting to write alternative plugins to control permissions - see #818 - I think I need an easy way to tell Datasette "no-one has permission to do X unless a plugin says otherwise". One relatively intuitive way to do that could be like this: ```json { "databases": { "fixtures": { "allow": null } } } ``` Right now I think that opens up permissions to everyone, which isn't as obvious. 107914493 issue  
636511683 MDU6SXNzdWU2MzY1MTE2ODM= 830 Redesign register_facet_classes plugin hook 9599 open 0   3268330 0 2020-06-10T20:03:27Z 2020-06-10T20:03:27Z   OWNER   Nothing uses this plugin hook yet, so the design is not yet proven. I'm going to build a real plugin against it and use that process to inform any design changes that may need to be made. I'll add a warning about this to the documentation. 107914493 issue  
636426530 MDU6SXNzdWU2MzY0MjY1MzA= 829 Ability to set ds_actor cookie such that it expires 9599 closed 0   5512395 6 2020-06-10T17:31:40Z 2020-06-10T19:41:35Z 2020-06-10T19:40:05Z OWNER   I need this for `datasette-auth-github`: 107914493 issue  
635914822 MDU6SXNzdWU2MzU5MTQ4MjI= 828 Horizontal scrollbar on changelog page on mobile 9599 closed 0   5512395 3 2020-06-10T04:18:54Z 2020-06-10T04:28:17Z 2020-06-10T04:28:17Z OWNER   You can scroll sideways on that page and it looks bad: <img width="419" alt="Changelog_—_Datasette_documentation" src=""> The cause is these long links: <img width="452" alt="Changelog_—_Datasette_documentation" src=""> 107914493 issue  
Powered by Datasette · Query took 19.554ms · About: github-to-sqlite