# issue_comments

html_url: https://github.com/simonw/datasette/pull/672#issuecomment-586112662
issue_url: https://api.github.com/repos/simonw/datasette/issues/672
node_id: MDEyOklzc3VlQ29tbWVudDU4NjExMjY2Mg==
created_at: 2020-02-14T06:05:27Z
updated_at: 2020-02-14T06:05:27Z
author_association: OWNER
body:
  It think the fix is to use an old-fashioned `threading` module daemon thread directly. That should exit cleanly when the program exits.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/672#issuecomment-586441484
issue_url: https://api.github.com/repos/simonw/datasette/issues/672
node_id: MDEyOklzc3VlQ29tbWVudDU4NjQ0MTQ4NA==
created_at: 2020-02-14T19:34:25Z
updated_at: 2020-02-14T19:34:25Z
author_association: OWNER
body:
  I've figured out how to tell if a database is safe to open or not:
  ```sql
  select sql from sqlite_master where sql like 'CREATE VIRTUAL TABLE%';
  ```
  This returns the SQL definitions for virtual tables. The bit after `using` tells you what they need.

  Run this against a SpatiaLite database and you get the following:
  ```sql
  CREATE VIRTUAL TABLE SpatialIndex USING VirtualSpatialIndex()
  CREATE VIRTUAL TABLE ElementaryGeometries USING VirtualElementary()
  ```
  Run it against an Apple Photos `photos.db` file (found with `find ~/Library | grep photos.db`) and you get this (partial list):
  ```sql
  CREATE VIRTUAL TABLE RidList_VirtualReader using RidList_VirtualReaderModule
  CREATE VIRTUAL TABLE Array_VirtualReader using Array_VirtualReaderModule
  CREATE VIRTUAL TABLE LiGlobals_VirtualBufferReader using VirtualBufferReaderModule
  CREATE VIRTUAL TABLE RKPlace_RTree using rtree (modelId,minLongitude,maxLongitude,minLatitude,maxLatitude)
  ```
  For a database with FTS4 you get:
  ```sql
  CREATE VIRTUAL TABLE "docs_fts" USING FTS4 (
      [title], [content], content="docs"
  )
  ```
  FTS5:
  ```sql
  CREATE VIRTUAL TABLE [FARA_All_Registrants_fts] USING FTS5 (
                  [Name], [Address_1], [Address_2],
                  content=[FARA_All_Registrants]
              )
  ```
  So I can use this to figure out all of the `using` pieces and then compare them to a list of known support ones.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/673#issuecomment-586442978
issue_url: https://api.github.com/repos/simonw/datasette/issues/673
node_id: MDEyOklzc3VlQ29tbWVudDU4NjQ0Mjk3OA==
created_at: 2020-02-14T19:38:19Z
updated_at: 2020-02-14T19:38:19Z
author_association: OWNER
body:
  Amazingly, I get 0 search results on Google for `RidList_VirtualReaderModule`! I guess no-one has reverse engineered the Apple Photos SQLite database at that level yet.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/673#issuecomment-586443837
issue_url: https://api.github.com/repos/simonw/datasette/issues/673
node_id: MDEyOklzc3VlQ29tbWVudDU4NjQ0MzgzNw==
created_at: 2020-02-14T19:40:42Z
updated_at: 2020-02-14T19:41:56Z
author_association: OWNER
body:
  Here's how to test if the `rtree` virtual table is supported:
  ```
  >>> import sqlite3
  >>> c = sqlite3.connect(":memory:")
  >>> c.execute("create virtual table blah using rtree (a, b, c)")
  <sqlite3.Cursor object at 0x107bc4180>
  >>> c.execute("create virtual table blah2 using rtree2 (a, b, c)")
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  sqlite3.OperationalError: table blah already exists
  ```
  Also:
  ```
  >>> c.execute('''CREATE VIRTUAL TABLE SpatialIndex USING VirtualSpatialIndex()''')
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  sqlite3.OperationalError: no such module: VirtualSpatialIndex
  >>> c.enable_load_extension(
  ... True)
  >>> 
  >>> c.load_extension("/usr/local/lib/mod_spatialite.dylib")
  >>> c.execute('''CREATE VIRTUAL TABLE SpatialIndex USING VirtualSpatialIndex()''')
  <sqlite3.Cursor object at 0x107bc4c70>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/673#issuecomment-586444835
issue_url: https://api.github.com/repos/simonw/datasette/issues/673
node_id: MDEyOklzc3VlQ29tbWVudDU4NjQ0NDgzNQ==
created_at: 2020-02-14T19:43:27Z
updated_at: 2020-02-14T19:43:27Z
author_association: OWNER
body:
  I can extend this function (maybe also rename it):

  https://github.com/simonw/datasette/blob/52ba34701cdbf510236de87d35b0e6df330626d1/datasette/utils/__init__.py#L595-L610

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/673#issuecomment-586445210
issue_url: https://api.github.com/repos/simonw/datasette/issues/673
node_id: MDEyOklzc3VlQ29tbWVudDU4NjQ0NTIxMA==
created_at: 2020-02-14T19:44:27Z
updated_at: 2020-02-14T19:44:27Z
author_association: OWNER
body:
  For the unit tests I think I'm going to have to create minimal binary SQLite file examples and include them in the repo.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/673#issuecomment-586448292
issue_url: https://api.github.com/repos/simonw/datasette/issues/673
node_id: MDEyOklzc3VlQ29tbWVudDU4NjQ0ODI5Mg==
created_at: 2020-02-14T19:53:05Z
updated_at: 2020-02-14T19:53:05Z
author_association: OWNER
body:
  I may be re-inventing this code at the moment:
  https://github.com/simonw/datasette/blob/3ffb8f3b98252531d11897fd431711e9b8045ace/datasette/app.py#L219-L237

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/673#issuecomment-586449286
issue_url: https://api.github.com/repos/simonw/datasette/issues/673
node_id: MDEyOklzc3VlQ29tbWVudDU4NjQ0OTI4Ng==
created_at: 2020-02-14T19:56:00Z
updated_at: 2020-02-14T19:57:17Z
author_association: OWNER
body:
  I tried to make the smallest SpatiaLite database file I could (to use for the tests), but it ended up over 5MB!
  ```
  $ echo '{"type":"Feature","properties":{"name":"Hearst Castle"},"geometry":{"type":"Point","coordinates":[-121.1686,35.685]}}' | geojson-to-sqlite /tmp/hearst.db places - --spatialite
  $ ls -lah /tmp/hearst.db 
  -rw-r--r--  1 simonw  wheel   5.3M Feb 14 11:54 /tmp/hearst.db
  ```
  I imagine that's because of these tables:
  <img width="798" alt="tiny" src="https://user-images.githubusercontent.com/9599/74563213-24509f80-4f21-11ea-93ab-0ca118c2e39b.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/673#issuecomment-586450571
issue_url: https://api.github.com/repos/simonw/datasette/issues/673
node_id: MDEyOklzc3VlQ29tbWVudDU4NjQ1MDU3MQ==
created_at: 2020-02-14T19:59:41Z
updated_at: 2020-02-14T20:01:14Z
author_association: OWNER
body:
  This helped:
  ```
  $ sqlite3 /tmp/hearst.db 
  SQLite version 3.24.0 2018-06-04 14:10:15
  Enter ".help" for usage hints.
  sqlite> delete from spatial_ref_sys where srid != 4326;
  sqlitte> delete from spatial_ref_sys_aux where srid != 4326;
  sqlite> vacuum;
  sqlite> ^D
  $ ls -lah /tmp/hearst.db 
  -rw-r--r--  1 simonw  wheel   216K Feb 14 12:01 /tmp/hearst.db
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/673#issuecomment-586454371
issue_url: https://api.github.com/repos/simonw/datasette/issues/673
node_id: MDEyOklzc3VlQ29tbWVudDU4NjQ1NDM3MQ==
created_at: 2020-02-14T20:11:02Z
updated_at: 2020-02-14T20:11:02Z
author_association: OWNER
body:
  The technique from `run_sanity_checks` of running `PRAGMA table_info({})` for every table seems to work just fine. It failed for the Apple Photos database for example:
  ```
  sqlite> pragma table_info(RKSceneInVersion_VirtualBufferReader);
  Error: no such module: VirtualBufferReaderModule
  ```
  So I think the solution to this ticket is going to be moving that logic into a new utility function.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/417#issuecomment-586599424
issue_url: https://api.github.com/repos/simonw/datasette/issues/417
node_id: MDEyOklzc3VlQ29tbWVudDU4NjU5OTQyNA==
created_at: 2020-02-15T15:12:19Z
updated_at: 2020-02-15T15:12:33Z
author_association: CONTRIBUTOR
body:
  So could the polling support also allow you to call sqlite_utils to update a database with csv files? (Though I'm guessing you would only want to handle changed files? Do your scrapers check and cache csv datestamps/hashes?)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/674#issuecomment-586622194
issue_url: https://api.github.com/repos/simonw/datasette/issues/674
node_id: MDEyOklzc3VlQ29tbWVudDU4NjYyMjE5NA==
created_at: 2020-02-15T17:22:57Z
updated_at: 2020-02-15T17:22:57Z
author_association: OWNER
body:
  I did some research for this in #673 - I think the solution is to split this out into a separate utility function:

  https://github.com/simonw/datasette/blob/3ffb8f3b98252531d11897fd431711e9b8045ace/datasette/app.py#L219-L237

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/674#issuecomment-586622557
issue_url: https://api.github.com/repos/simonw/datasette/issues/674
node_id: MDEyOklzc3VlQ29tbWVudDU4NjYyMjU1Nw==
created_at: 2020-02-15T17:26:43Z
updated_at: 2020-02-15T17:38:56Z
author_association: OWNER
body:
  `datasette.utils.check_connection(conn)`

  I'd like it to be able to return an indicator of what the problem is, so that the spatialite check above can behave in a special way. So maybe it raises a `ConnectionProblem` exception if there is a problem? `SpatialiteConnectionProblem` can be a subclass of that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/674#issuecomment-586622677
issue_url: https://api.github.com/repos/simonw/datasette/issues/674
node_id: MDEyOklzc3VlQ29tbWVudDU4NjYyMjY3Nw==
created_at: 2020-02-15T17:28:03Z
updated_at: 2020-02-15T17:28:03Z
author_association: OWNER
body:
  I can unit test this with a minimal SpatiaLite database, created like this: https://github.com/simonw/datasette/issues/673#issuecomment-586449286

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/674#issuecomment-586623462
issue_url: https://api.github.com/repos/simonw/datasette/issues/674
node_id: MDEyOklzc3VlQ29tbWVudDU4NjYyMzQ2Mg==
created_at: 2020-02-15T17:36:53Z
updated_at: 2020-03-26T17:19:23Z
author_association: OWNER
body:
  ... actually we don't need to do that, we have a mechanism for that already:

  https://github.com/simonw/datasette/blob/f1442a8151f66ceef6517b6d3d045e2ec1d0f0ec/tests/build_small_spatialite_db.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/674#issuecomment-586625451
issue_url: https://api.github.com/repos/simonw/datasette/issues/674
node_id: MDEyOklzc3VlQ29tbWVudDU4NjYyNTQ1MQ==
created_at: 2020-02-15T17:57:43Z
updated_at: 2020-02-15T17:57:43Z
author_association: OWNER
body:
  I don't have a unit test for other databases that can't be opened, but I tested it manually against a copy of an Apple Photos `photos.db` file (found using `find ~/Library | grep photos.db`) and got this:
  ```
  $ datasette /tmp/photos.db 
  Serve! files=('/tmp/photos.db',) (immutables=()) on port 8001
  Usage: datasette serve [OPTIONS] [FILES]...

  Error: Connection to /tmp/photos.db failed check: no such module: RidList_VirtualReaderModule
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/87#issuecomment-586661250
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/87
node_id: MDEyOklzc3VlQ29tbWVudDU4NjY2MTI1MA==
created_at: 2020-02-16T02:19:33Z
updated_at: 2020-02-16T02:19:33Z
author_association: OWNER
body:
  Here's the code: 

  https://github.com/simonw/sqlite-utils/blob/5e0000609f9be6efafea1b96f610988eb18d6d89/sqlite_utils/utils.py#L18-L24

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/87#issuecomment-586661276
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/87
node_id: MDEyOklzc3VlQ29tbWVudDU4NjY2MTI3Ng==
created_at: 2020-02-16T02:20:14Z
updated_at: 2020-02-16T02:20:14Z
author_association: OWNER
body:
  `OrderedDict` is actually a subclass of `dict` - so a smart fix would be for this logic to check and see if the type `t` is a subclass of one of `list`, `tuple` or `dict`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586661934
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/86
node_id: MDEyOklzc3VlQ29tbWVudDU4NjY2MTkzNA==
created_at: 2020-02-16T02:33:07Z
updated_at: 2020-02-16T02:33:07Z
author_association: OWNER
body:
  Thanks for the example file - looks like it can be trimmed down to just these two lines to replicate the bug:
  ```csv
  "MTU (CET)","Day-ahead Price [EUR/MWh]"
  "01.01.2016 00:00 - 01.01.2016 01:00","23.86"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586662404
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/86
node_id: MDEyOklzc3VlQ29tbWVudDU4NjY2MjQwNA==
created_at: 2020-02-16T02:43:12Z
updated_at: 2020-02-16T02:43:12Z
author_association: OWNER
body:
  https://stackoverflow.com/a/22694438 looks like the answer:
  > When using square brackets, it is not possible to have these characters in the identifier.
  > 
  > When using double quotes, you can escape them in the name by doubling them:
  > 
  > `CREATE TABLE "hello ""world"""(key INTEGER PRIMARY KEY);`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586676640
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/86
node_id: MDEyOklzc3VlQ29tbWVudDU4NjY3NjY0MA==
created_at: 2020-02-16T07:16:31Z
updated_at: 2020-02-16T07:16:31Z
author_association: OWNER
body:
  There's something weird about this. I created a test database file like so:
  ```
  sqlite3 /tmp/demo.db <<EOF
      BEGIN TRANSACTION;
      CREATE TABLE "data" (
          "MTU (CET)" TEXT,
          "Day-ahead Price [EUR/MWh]" TEXT
      );
      INSERT INTO "data" VALUES('01.01.2016 00:00 - 01.01.2016 01:00','23.86');
      COMMIT;
  EOF
  ```
  Then confirmed that it works as expected in SQLite:
  ```
   $ sqlite3 /tmp/demo.db 
  SQLite version 3.24.0 2018-06-04 14:10:15
  Enter ".help" for usage hints.
  sqlite> .schema
  CREATE TABLE IF NOT EXISTS "data" (
          "MTU (CET)" TEXT,
          "Day-ahead Price [EUR/MWh]" TEXT
      );
  sqlite> .headers on
  sqlite> select * from data;
  MTU (CET)|Day-ahead Price [EUR/MWh]
  01.01.2016 00:00 - 01.01.2016 01:00|23.86
  sqlite> 
  ```
  BUT... if I open the same database in Python, something weird happens:
  ```
  In [1]: import sqlite3                                                                                                         
  In [2]: conn = sqlite3.connect("/tmp/demo.db")                                                                                 
  In [3]: cursor = conn.cursor()                                                                                                 
  In [4]: cursor.execute("select * from data")                                                                                   
  Out[4]: <sqlite3.Cursor at 0x10c70a0a0>
  In [5]: cursor.fetchall()                                                                                                      
  Out[5]: [('01.01.2016 00:00 - 01.01.2016 01:00', '23.86')]
  In [6]: cursor.description                                                                                                     
  Out[6]: 
  (('MTU (CET)', None, None, None, None, None, None),
   ('Day-ahead Price', None, None, None, None, None, None))
  In [7]: conn.row_factory = sqlite3.Row                                                                                         
  In [8]: cursor = conn.cursor()                                                                                                 
  In [9]: cursor.execute("select * from data")                                                                                   
  Out[9]: <sqlite3.Cursor at 0x10c7a8490>
  In [10]: row = cursor.fetchall()                                                                                                     
  In [12]: row                                                                                                                   
  Out[12]: <sqlite3.Row at 0x10c3fe670>
  In [15]: row.keys()                                                                                                            
  Out[15]: ['MTU (CET)', 'Day-ahead Price']
  ```
  Note that in `cursor.description` AND in `row.keys()` above the second column is displayed as `'Day-ahead Price'` - when we would expect it to be displayed as `Day-ahead Price [EUR/MWh]`

  So.... it looks like there may be a bug in Python's `sqlite3` module where columns with square braces in them have that portion of the name stripped out!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586676856
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/86
node_id: MDEyOklzc3VlQ29tbWVudDU4NjY3Njg1Ng==
created_at: 2020-02-16T07:20:34Z
updated_at: 2020-02-16T07:20:34Z
author_association: OWNER
body:
  I'm not sure what to do about this one.

  I can't fix it: this bug in Python's `sqlite3` module means that even if I write a database out with column names that include `[]` I won't be able to read them back again.

  So... I could do one of the following:

  - Throw an error if a column name includes those characters. That's my preferred option I think.
  - Automatically replace `[` in column names with `(` and `]` with `)`
  - Do the automatic replacement but show a user-visible warning when I do it
  - Throw an error, but give the user an option to run with e.g. `--fix-column-names` which applies that automatic fix.

  Since this is likely to be an incredibly rare edge-case I think I'd rather minimize the amount of code that deals with it, so my preferred option is to just throw that error and stop.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586683572
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/86
node_id: MDEyOklzc3VlQ29tbWVudDU4NjY4MzU3Mg==
created_at: 2020-02-16T09:03:54Z
updated_at: 2020-02-16T09:03:54Z
author_association: NONE
body:
  Probably the best option to just throw the error.
  Is there any active dev chan where we could post the issue to python sqlite3?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/678#issuecomment-589908060
issue_url: https://api.github.com/repos/simonw/datasette/issues/678
node_id: MDEyOklzc3VlQ29tbWVudDU4OTkwODA2MA==
created_at: 2020-02-22T02:28:51Z
updated_at: 2020-02-22T02:28:51Z
author_association: OWNER
body:
  Done. Updated documentation is here: https://datasette.readthedocs.io/en/latest/plugins.html#prepare-connection-conn-database-datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/676#issuecomment-589908293
issue_url: https://api.github.com/repos/simonw/datasette/issues/676
node_id: MDEyOklzc3VlQ29tbWVudDU4OTkwODI5Mw==
created_at: 2020-02-22T02:31:21Z
updated_at: 2020-02-22T02:31:21Z
author_association: OWNER
body:
  I'm afraid this is by design: supporting wildcards in that mode was more-or-less accidental and often resulted in unexpected user-facing syntax errors when users accidentally entered search text that included characters with special meaning to SQLite.

  You can still execute wildcard searches but you have to do so using custom SQL statements.

  I'm open to suggestions for more friendly ways to support this functionality. Would an alternative `_search_wildcard=` URL parameter be worth supporting I wonder? What's your specific use-case for wildcards?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/666#issuecomment-589908465
issue_url: https://api.github.com/repos/simonw/datasette/issues/666
node_id: MDEyOklzc3VlQ29tbWVudDU4OTkwODQ2NQ==
created_at: 2020-02-22T02:33:18Z
updated_at: 2020-02-22T02:33:18Z
author_association: OWNER
body:
  This is a smart optimization. I'd like to add a test for it, since the logic is a tiny bit more complex.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/675#issuecomment-589908912
issue_url: https://api.github.com/repos/simonw/datasette/issues/675
node_id: MDEyOklzc3VlQ29tbWVudDU4OTkwODkxMg==
created_at: 2020-02-22T02:38:21Z
updated_at: 2020-02-22T02:38:21Z
author_association: OWNER
body:
  Interesting feature suggestion.

  My initial instinct was that this would be better handled using the layered nature of Docker - so build a Docker image with `datasette package` and then have a separate custom script which takes that image, copies in the extra data and outputs a new image.

  But... `datasette package` is already meant to be more convenient than messing around with Docker by hand like this - so actually having a `--copy` option like you describe here feels like it's within scope of what `datasette package` is meant to do.

  So yeah - if you're happy to design this I think it would be worth us adding.

  Small design suggestion: allow `--copy` to be applied multiple times, so you can do something like this:

      datasette package \
          --copy ~/project/templates /templates \
          --copy ~/project/README.md /README.md \
          data.db

  Also since Click arguments can take multiple options I don't think you need to have the `:` in there - although if it better matches Docker's own UI it might be more consistent to have it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/679#issuecomment-589909401
issue_url: https://api.github.com/repos/simonw/datasette/issues/679
node_id: MDEyOklzc3VlQ29tbWVudDU4OTkwOTQwMQ==
created_at: 2020-02-22T02:43:42Z
updated_at: 2020-02-22T02:46:16Z
author_association: OWNER
body:
  Before shipping there are a few additions I can make to the https://datasette.readthedocs.io/en/stable/ecosystem.html page:

  * https://github.com/simonw/shapefile-to-sqlite
  * https://github.com/simonw/datasette-mask-columns (which will itself depend on 0.36 being out)
  * https://github.com/simonw/datasette-auth-existing-cookies
  * https://github.com/simonw/datasette-sentry

  I could also add links to associated blog entries directly to those descriptions on the ecosystem page:

  * https://simonwillison.net/2020/Jan/29/weeknotes-datasette-cookies-sentry/
  * https://simonwillison.net/2020/Feb/19/shapefile-to-sqlite/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/680#issuecomment-589914711
issue_url: https://api.github.com/repos/simonw/datasette/issues/680
node_id: MDEyOklzc3VlQ29tbWVudDU4OTkxNDcxMQ==
created_at: 2020-02-22T03:51:47Z
updated_at: 2020-02-22T03:51:47Z
author_association: OWNER
body:
  I considered just updating the GitHub release with a link to the release notes, but I'm building other things against the GitHub releases API such that it's nicer to have the release notes live there.

  I bet I could use pandoc to automate the conversion of the formats in one or other direction.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/676#issuecomment-589922016
issue_url: https://api.github.com/repos/simonw/datasette/issues/676
node_id: MDEyOklzc3VlQ29tbWVudDU4OTkyMjAxNg==
created_at: 2020-02-22T05:50:10Z
updated_at: 2020-02-22T05:50:10Z
author_association: NONE
body:
  Thanks Simon,
  My use case is using Datasette for full text search type ahead.  That was working pretty well. The _search_wildcard= option will be awesome.  Thanks

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/680#issuecomment-590030258
issue_url: https://api.github.com/repos/simonw/datasette/issues/680
node_id: MDEyOklzc3VlQ29tbWVudDU5MDAzMDI1OA==
created_at: 2020-02-23T05:51:04Z
updated_at: 2020-02-23T05:51:04Z
author_association: OWNER
body:
  Did a bunch of research in a Jupyter notebook and it looks like I can use `pypandoc` to achieve this conversion without too much trouble: https://gist.github.com/simonw/41d56712427e6a4178fc6495d664005f

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/680#issuecomment-590030425
issue_url: https://api.github.com/repos/simonw/datasette/issues/680
node_id: MDEyOklzc3VlQ29tbWVudDU5MDAzMDQyNQ==
created_at: 2020-02-23T05:53:48Z
updated_at: 2020-02-23T05:53:48Z
author_association: OWNER
body:
  For the actual implementation: I think this is a GitHub Action.

  It looks for new tags, and when a tag is created it builds the docs using Sphinx, extracts the latest HTML changelog entry using BeautifulSoup, cleans it up a bit (I think the only thing that needs doing is turning relative links into absolute ones), runs it through pypandoc to create `gfm`, then uses the GitHub releases API to post the changelog as a release.

  https://developer.github.com/v3/repos/releases/#create-a-release

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/680#issuecomment-590106416
issue_url: https://api.github.com/repos/simonw/datasette/issues/680
node_id: MDEyOklzc3VlQ29tbWVudDU5MDEwNjQxNg==
created_at: 2020-02-23T19:26:40Z
updated_at: 2020-02-23T19:26:40Z
author_association: OWNER
body:
  I think this should go a step further: I'd like some extra tests that run when a new build is being shipped and confirm that it has release notes in the right place.

  So this is a Travis CI responsibility.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/681#issuecomment-590107309
issue_url: https://api.github.com/repos/simonw/datasette/issues/681
node_id: MDEyOklzc3VlQ29tbWVudDU5MDEwNzMwOQ==
created_at: 2020-02-23T19:36:33Z
updated_at: 2020-02-23T19:36:33Z
author_association: OWNER
body:
  For adding extra HTTP headers I suggest writing a plugin. There's an example in the docs of one that adds a simple HTTP header here: https://datasette.readthedocs.io/en/stable/plugins.html#asgi-wrapper-datasette

  You don't need to bundle it up as a full Python package - you can create a `plugins/add_headers.py` file and run Datasette with `--plugins-dir=plugins/`: https://datasette.readthedocs.io/en/stable/plugins.html#writing-plugins

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/681#issuecomment-590107464
issue_url: https://api.github.com/repos/simonw/datasette/issues/681
node_id: MDEyOklzc3VlQ29tbWVudDU5MDEwNzQ2NA==
created_at: 2020-02-23T19:38:16Z
updated_at: 2020-02-23T19:38:16Z
author_association: OWNER
body:
  The `default_cache_ttl` config currently only takes effect for databases that are loaded in immutable mode using `-i database.db`: https://datasette.readthedocs.io/en/stable/performance.html#immutable-mode

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/681#issuecomment-590107843
issue_url: https://api.github.com/repos/simonw/datasette/issues/681
node_id: MDEyOklzc3VlQ29tbWVudDU5MDEwNzg0Mw==
created_at: 2020-02-23T19:42:15Z
updated_at: 2020-02-23T19:42:15Z
author_association: OWNER
body:
  Those security headers from your screenshot are definitely worth me thinking more about. Even if not as core features they could make a great officially supported plugin.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/676#issuecomment-590109450
issue_url: https://api.github.com/repos/simonw/datasette/issues/676
node_id: MDEyOklzc3VlQ29tbWVudDU5MDEwOTQ1MA==
created_at: 2020-02-23T19:59:11Z
updated_at: 2020-02-23T19:59:11Z
author_association: OWNER
body:
  I'm going to call it `?_search_raw=` because it will pass through the value as a raw (unprocessed) argument to SQLite FTS, which means it will expose features other than just wildcards: https://www.sqlite.org/fts5.html#full_text_query_syntax

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/676#issuecomment-590109778
issue_url: https://api.github.com/repos/simonw/datasette/issues/676
node_id: MDEyOklzc3VlQ29tbWVudDU5MDEwOTc3OA==
created_at: 2020-02-23T20:02:28Z
updated_at: 2020-02-23T20:02:28Z
author_association: OWNER
body:
  `?_search_raw=` doesn't work because it clashes with an existing feature - you can use `?_search_colname=` to search just within a specific column, and there could possibly be a column called `raw`.

  I could go with `?_searchraw=` - or maybe have an additional "mode" selection like `?_search=foo*&_searchmode=raw`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/676#issuecomment-590110086
issue_url: https://api.github.com/repos/simonw/datasette/issues/676
node_id: MDEyOklzc3VlQ29tbWVudDU5MDExMDA4Ng==
created_at: 2020-02-23T20:06:03Z
updated_at: 2020-02-23T20:06:52Z
author_association: OWNER
body:
  Nice thing about the "search mode" concept is that I can optionally reflect it in the HTML UI in some way - maybe with a checkbox for "raw" mode that only shows up if the user hits the page with `?_searchmode=raw` in the querystring.

  It also hints at potentially adding further search modes in the future. Not sure if I'd do that but it's an interesting potential avenue to explore.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/676#issuecomment-590113869
issue_url: https://api.github.com/repos/simonw/datasette/issues/676
node_id: MDEyOklzc3VlQ29tbWVudDU5MDExMzg2OQ==
created_at: 2020-02-23T20:43:59Z
updated_at: 2020-02-23T20:43:59Z
author_association: OWNER
body:
  You can try this right now like so:

      pip install https://github.com/simonw/datasette/archive/search-raw.zip

  Then use the following:

      ?_search=foo*&_searchmode=raw`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/682#issuecomment-590153892
issue_url: https://api.github.com/repos/simonw/datasette/issues/682
node_id: MDEyOklzc3VlQ29tbWVudDU5MDE1Mzg5Mg==
created_at: 2020-02-24T03:10:45Z
updated_at: 2020-02-24T03:13:03Z
author_association: OWNER
body:
  Some more detailed notes I made earlier:

  Datasette would run a single write thread per database. That thread gets an exclusive connection, and a queue. Plugins can add functions to the queue which will be called and given access to that connection.

  The write thread for that database is created the first time a write is attempted.

  Question: should that thread have its own asyncio loop so that async techniques like httpx can be used within the thread? I think not at first - only investigate this if it turns out to be necessary in the future.

  This thread will run as part of the Datasette process. This means there is always a risk that the thread will die in the middle of something because the server got restarted - so use transactions to limit risk of damage to database should that happen.

  I don’t want web responses blocking waiting for stuff to happen here - so every task put on that queue will have a task ID, and that ID will be returned such that client code can poll for its completion.

  Could the request block for up to 0.5s just in case the write is really fast, then return a polling token if it isn't finished yet? Looks possible - `Queue.get` can block with a timeout.

  There will be a `/-/writes` page which shows currently queued writes - so each one needs a human-readable description of some sort. (You can access a deque called `q.queue` to see what’s in there)

  Stretch goal: It would be cool if write operations could optionally handle their own progress reports. That way I can do some really nice UI around what’s going on with these things.

  This mechanism has a ton of potential. It may even be how we handle things like Twitter imports and suchlike - queued writing tasks.

  One catch with this approach: if a plugin is reading from APIs etc it shouldn't block writes to the database while it is doing so. So sticking a function in the queue that does additional time consuming stuff is actually an anti pattern. Instead, plugins should schedule their API access in the main event loop and occasionally write just the updates they need to make to that write queue.

  ### Implementation notes

  Maybe each item in the queue is a `(callable, uuid, reply_queue)` triple. You can do a blocking `.get()` on the `reply_queue` if you want to wait for the answer. The execution framework could look for the return value from `callable()` and automatically send it to `reply_queue`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/682#issuecomment-590154309
issue_url: https://api.github.com/repos/simonw/datasette/issues/682
node_id: MDEyOklzc3VlQ29tbWVudDU5MDE1NDMwOQ==
created_at: 2020-02-24T03:14:10Z
updated_at: 2020-02-24T03:14:10Z
author_association: OWNER
body:
  Some prior art: Charles Leifer implemented a `SqliteQueueDatabase` class that automatically queues writes for you: https://charlesleifer.com/blog/multi-threaded-sqlite-without-the-operationalerrors/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/676#issuecomment-590209074
issue_url: https://api.github.com/repos/simonw/datasette/issues/676
node_id: MDEyOklzc3VlQ29tbWVudDU5MDIwOTA3NA==
created_at: 2020-02-24T08:20:15Z
updated_at: 2020-02-24T08:20:15Z
author_association: NONE
body:
  Awesome, thank you so much. I’ll try it out and let you know.

  On Sun, Feb 23, 2020 at 1:44 PM Simon Willison <notifications@github.com>
  wrote:

  > You can try this right now like so:
  >
  > pip install https://github.com/simonw/datasette/archive/search-raw.zip
  >
  > Then use the following:
  >
  > ?_search=foo*&_searchmode=raw`
  >
  > —
  > You are receiving this because you authored the thread.
  > Reply to this email directly, view it on GitHub
  > <https://github.com/simonw/datasette/issues/676?email_source=notifications&email_token=AN3FXEFS6B22U2NOT6M5FULRELNY7A5CNFSM4KYIOIB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMWGYTI#issuecomment-590113869>,
  > or unsubscribe
  > <https://github.com/notifications/unsubscribe-auth/AN3FXEANDJ6AIHGU4ADK4D3RELNY7ANCNFSM4KYIOIBQ>
  > .
  >

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/682#issuecomment-590399600
issue_url: https://api.github.com/repos/simonw/datasette/issues/682
node_id: MDEyOklzc3VlQ29tbWVudDU5MDM5OTYwMA==
created_at: 2020-02-24T15:56:10Z
updated_at: 2020-02-24T15:56:23Z
author_association: OWNER
body:
  ## Implementation plan

  Method on Database class called `execute_write(sql)`

  Which calls `.execute_write_fn(fn)` - so you can instead create a function that applies a whole batch of writes and pass that instead if you need to

  Throws an error of database isn't mutable.

  Add `._writer_thread` thread property to Database - we start that thread the first time we need it. It blocks on `._writer_queue.get()`

  We write to that queue with `WriteTask(fn, uuid, reply_queue)` namedtuples - then time-out block awaiting reply for 0.5s

  Have a `.write_status(uuid)` method that checks if `uuid` has completed

  This should be enough to get it all working. MVP can skip the .5s timeout entirely

  But... what about that progress bar supporting stretch goal?

  For that let's have each write operation that's currently in progress have total and done integer properties. So I guess we can add those to the `WriteTask`.

  Should we have the ability to see what the currently executing write is? Seems useful.

  Hopefully I can integrate https://github.com/tqdm/tqdm such that it calculates ETAs without actually trying to print to the console.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/675#issuecomment-590405736
issue_url: https://api.github.com/repos/simonw/datasette/issues/675
node_id: MDEyOklzc3VlQ29tbWVudDU5MDQwNTczNg==
created_at: 2020-02-24T16:06:27Z
updated_at: 2020-02-24T16:06:27Z
author_association: NONE
body:
  > So yeah - if you're happy to design this I think it would be worth us adding.

  Great! I’ll give it a go.



  > Small design suggestion: allow `--copy` to be applied multiple times…

  Makes a ton of sense, will do.

  > Also since Click arguments can take multiple options I don't think you need to have the `:` in there - although if it better matches Docker's own UI it might be more consistent to have it.

  Great point. I double checked the docs for `docker cp` and in that context the colon is used to delimit a container and a path, while spaces are used to separate the source and target.

  The usage string is:

  ```text
  docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
  docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
  ```

  so in fact it’ll be more consistent to use a space to delimit the source and destination paths, like so:

  ```shell
  $ datasette package --copy /the/source/path /the/target/path data.db
  ```

  and I suppose the short-form version of the option should be `cp` like so:

  ```shell
  $ datasette package -cp /the/source/path /the/target/path data.db
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/682#issuecomment-590417366
issue_url: https://api.github.com/repos/simonw/datasette/issues/682
node_id: MDEyOklzc3VlQ29tbWVudDU5MDQxNzM2Ng==
created_at: 2020-02-24T16:27:10Z
updated_at: 2020-02-24T16:27:10Z
author_association: OWNER
body:
  I wonder if I even need the `reply_queue` mechanism? Are the replies from writes generally even interesting?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/682#issuecomment-590417619
issue_url: https://api.github.com/repos/simonw/datasette/issues/682
node_id: MDEyOklzc3VlQ29tbWVudDU5MDQxNzYxOQ==
created_at: 2020-02-24T16:27:36Z
updated_at: 2020-02-24T16:27:36Z
author_association: OWNER
body:
  Error handling could be tricky. Exceptions thrown in threads don't show up anywhere by default - I would need to explicitly catch them and decide what to do with them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/682#issuecomment-590430988
issue_url: https://api.github.com/repos/simonw/datasette/issues/682
node_id: MDEyOklzc3VlQ29tbWVudDU5MDQzMDk4OA==
created_at: 2020-02-24T16:50:48Z
updated_at: 2020-02-24T16:50:48Z
author_association: OWNER
body:
  I'm dropping the progress bar idea. This mechanism is supposed to guarantee exclusive access to the single write connection, which means it should be targeted by operations that are as short as possible. An operation running long enough to need a progress bar is too long!

  Any implementation of progress bars for long running write operations needs to happen elsewhere in the stack.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/682#issuecomment-590436368
issue_url: https://api.github.com/repos/simonw/datasette/issues/682
node_id: MDEyOklzc3VlQ29tbWVudDU5MDQzNjM2OA==
created_at: 2020-02-24T17:00:21Z
updated_at: 2020-02-24T17:00:21Z
author_association: OWNER
body:
  Interesting challenge: I would like to be able to "await" on `queue.get()` (with a timeout).

  Problem is: `queue.Queue()` is designed for threading and cannot be awaited. `asyncio.Queue` can be awaited but is not meant to be used with threads.

  https://stackoverflow.com/a/32894169 suggests using Janus, a thread-aware asyncio queue: https://github.com/aio-libs/janus

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/682#issuecomment-590511601
issue_url: https://api.github.com/repos/simonw/datasette/issues/682
node_id: MDEyOklzc3VlQ29tbWVudDU5MDUxMTYwMQ==
created_at: 2020-02-24T19:38:27Z
updated_at: 2020-02-24T19:38:27Z
author_association: OWNER
body:
  I tested this using the following code in a view (after `from sqlite_utils import Database`):
  ```python
      db = next(iter(self.ds.databases.values()))
      db.execute_write_fn(lambda conn: Database(conn)["counter"].insert({"id": 1, "count": 0}, pk="id", ignore=True))
      db.execute_write("update counter set count = count + 1 where id = 1")
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/682#issuecomment-590517338
issue_url: https://api.github.com/repos/simonw/datasette/issues/682
node_id: MDEyOklzc3VlQ29tbWVudDU5MDUxNzMzOA==
created_at: 2020-02-24T19:51:21Z
updated_at: 2020-02-24T19:51:21Z
author_association: OWNER
body:
  I filed a question / feature request with Janus about supporting timeouts for `.get()` against async queues here: https://github.com/aio-libs/janus/issues/240

  I'm going to move ahead without needing that ability though. I figure SQLite writes are _fast_, and plugins can be trusted to implement just fast writes. So I'm going to support either fire-and-forget writes (they get added to the queue and a task ID is returned) or have the option to block awaiting the completion of the write (using Janus) but let callers decide which version they want. I may add optional timeouts some time in the future.

  I am going to make both `execute_write()` and `execute_write_fn()` awaitable functions though, for consistency with `.execute()` and to give me flexibility to change how they work in the future.

  I'll also add a `block=True` option to both of them which causes the function to wait for the write to be successfully executed - defaults to `False` (fire-and-forget mode).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/683#issuecomment-590518182
issue_url: https://api.github.com/repos/simonw/datasette/issues/683
node_id: MDEyOklzc3VlQ29tbWVudDU5MDUxODE4Mg==
created_at: 2020-02-24T19:53:12Z
updated_at: 2020-02-24T19:53:12Z
author_association: OWNER
body:
  Next steps are from comment https://github.com/simonw/datasette/issues/682#issuecomment-590517338
  > I'm going to move ahead without needing that ability though. I figure SQLite writes are _fast_, and plugins can be trusted to implement just fast writes. So I'm going to support either fire-and-forget writes (they get added to the queue and a task ID is returned) or have the option to block awaiting the completion of the write (using Janus) but let callers decide which version they want. I may add optional timeouts some time in the future.
  > 
  > I am going to make both `execute_write()` and `execute_write_fn()` awaitable functions though, for consistency with `.execute()` and to give me flexibility to change how they work in the future.
  > 
  > I'll also add a `block=True` option to both of them which causes the function to wait for the write to be successfully executed - defaults to `False` (fire-and-forget mode).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/675#issuecomment-590539805
issue_url: https://api.github.com/repos/simonw/datasette/issues/675
node_id: MDEyOklzc3VlQ29tbWVudDU5MDUzOTgwNQ==
created_at: 2020-02-24T20:44:59Z
updated_at: 2020-02-24T20:45:08Z
author_association: OWNER
body:
  Design looks great to me.

  I'm not keen on two letter short versions (`-cp`) - I'd rather either have a single character or no short form at all. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/681#issuecomment-590543398
issue_url: https://api.github.com/repos/simonw/datasette/issues/681
node_id: MDEyOklzc3VlQ29tbWVudDU5MDU0MzM5OA==
created_at: 2020-02-24T20:53:56Z
updated_at: 2020-02-24T20:53:56Z
author_association: NONE
body:
  Excellent. I'll implement the simple plugin-solution now. And will have a go at a more mature plugin later. Thanks!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/683#issuecomment-590592581
issue_url: https://api.github.com/repos/simonw/datasette/issues/683
node_id: MDEyOklzc3VlQ29tbWVudDU5MDU5MjU4MQ==
created_at: 2020-02-24T23:00:44Z
updated_at: 2020-02-24T23:01:09Z
author_association: OWNER
body:
  I've been testing this out by running one-off demo plugins. I saved the following in a file called `write-plugins/log_asgi.py` (it's a hacked around copy of [asgi-log-to-sqlite](https://github.com/simonw/asgi-log-to-sqlite)) and then running `datasette data.db --plugins-dir=write-plugins/`:
  ```python
  from datasette import hookimpl
  import sqlite_utils
  import time


  class AsgiLogToSqliteViaWriteQueue:
      lookup_columns = (
          "path",
          "user_agent",
          "referer",
          "accept_language",
          "content_type",
          "query_string",
      )

      def __init__(self, app, db):
          self.app = app
          self.db = db
          self._tables_ensured = False

      async def ensure_tables(self):
          def _ensure_tables(conn):
              db = sqlite_utils.Database(conn)
              for column in self.lookup_columns:
                  table = "{}s".format(column)
                  if not db[table].exists():
                      db[table].create({"id": int, "name": str}, pk="id")
              if "requests" not in db.table_names():
                  db["requests"].create(
                      {
                          "start": float,
                          "method": str,
                          "path": int,
                          "query_string": int,
                          "user_agent": int,
                          "referer": int,
                          "accept_language": int,
                          "http_status": int,
                          "content_type": int,
                          "client_ip": str,
                          "duration": float,
                          "body_size": int,
                      },
                      foreign_keys=self.lookup_columns,
                  )
          await self.db.execute_write_fn(_ensure_tables)

      async def __call__(self, scope, receive, send):
          if not self._tables_ensured:
              self._tables_ensured = True
              await self.ensure_tables()

          response_headers = []
          body_size = 0
          http_status = None

          async def wrapped_send(message):
              nonlocal body_size, response_headers, http_status
              if message["type"] == "http.response.start":
                  response_headers = message["headers"]
                  http_status = message["status"]

              if message["type"] == "http.response.body":
                  body_size += len(message["body"])

              await send(message)

          start = time.time()
          await self.app(scope, receive, wrapped_send)
          end = time.time()

          path = str(scope["path"])
          query_string = None
          if scope.get("query_string"):
              query_string = "?{}".format(scope["query_string"].decode("utf8"))

          request_headers = dict(scope.get("headers") or [])

          referer = header(request_headers, "referer")
          user_agent = header(request_headers, "user-agent")
          accept_language = header(request_headers, "accept-language")

          content_type = header(dict(response_headers), "content-type")

          def _log_to_database(conn):
              db = sqlite_utils.Database(conn)
              db["requests"].insert(
                  {
                      "start": start,
                      "method": scope["method"],
                      "path": lookup(db, "paths", path),
                      "query_string": lookup(db, "query_strings", query_string),
                      "user_agent": lookup(db, "user_agents", user_agent),
                      "referer": lookup(db, "referers", referer),
                      "accept_language": lookup(db, "accept_languages", accept_language),
                      "http_status": http_status,
                      "content_type": lookup(db, "content_types", content_type),
                      "client_ip": scope.get("client", (None, None))[0],
                      "duration": end - start,
                      "body_size": body_size,
                  },
                  alter=True,
                  foreign_keys=self.lookup_columns,
              )

          await self.db.execute_write_fn(_log_to_database)


  def header(d, name):
      return d.get(name.encode("utf8"), b"").decode("utf8") or None


  def lookup(db, table, value):
      return db[table].lookup({"name": value}) if value else None


  @hookimpl
  def asgi_wrapper(datasette):
      def wrap_with_class(app):
          return AsgiLogToSqliteViaWriteQueue(
              app, next(iter(datasette.databases.values()))
          )

      return wrap_with_class
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/683#issuecomment-590593120
issue_url: https://api.github.com/repos/simonw/datasette/issues/683
node_id: MDEyOklzc3VlQ29tbWVudDU5MDU5MzEyMA==
created_at: 2020-02-24T23:02:30Z
updated_at: 2020-02-24T23:02:30Z
author_association: OWNER
body:
  I'm going to muck around with a couple more demo plugins - in particular one derived from [datasette-upload-csvs](https://github.com/simonw/datasette-upload-csvs) - to make sure I'm comfortable with this API - then add a couple of tests and merge it with documentation that warns "this is still an experimental feature and may change".

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/675#issuecomment-590593247
issue_url: https://api.github.com/repos/simonw/datasette/issues/675
node_id: MDEyOklzc3VlQ29tbWVudDU5MDU5MzI0Nw==
created_at: 2020-02-24T23:02:52Z
updated_at: 2020-02-24T23:02:52Z
author_association: NONE
body:
  > Design looks great to me.

  Excellent, thanks!

  > I'm not keen on two letter short versions (`-cp`) - I'd rather either have a single character or no short form at all.

  Hmm, well, anyone running `datasette package` is probably at least somewhat familiar with UNIX CLIs… so how about `--cp` as a middle ground?

  ```shell
  $ datasette package --cp /the/source/path /the/target/path data.db
  ```

  I think I like it. Easy to remember!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/683#issuecomment-590598248
issue_url: https://api.github.com/repos/simonw/datasette/issues/683
node_id: MDEyOklzc3VlQ29tbWVudDU5MDU5ODI0OA==
created_at: 2020-02-24T23:18:50Z
updated_at: 2020-02-24T23:18:50Z
author_association: OWNER
body:
  I'm not convinced by the return value of the `.execute_write_fn()` method:

  https://github.com/simonw/datasette/blob/ab2348280206bde1390b931ae89d372c2f74b87e/datasette/database.py#L79-L83

  Do I really need that `WriteResponse` class or can I do something nicer?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/683#issuecomment-590598689
issue_url: https://api.github.com/repos/simonw/datasette/issues/683
node_id: MDEyOklzc3VlQ29tbWVudDU5MDU5ODY4OQ==
created_at: 2020-02-24T23:20:11Z
updated_at: 2020-02-24T23:20:11Z
author_association: OWNER
body:
  I think `if block` it makes sense to return the return value of the function that was executed. Without it all I really need to do is return the `uuid` so something could theoretically poll for completion later on.

  But is it weird having a function that returns different types depending on if you passed `block=True` or not? Should they be differently named functions?

  I'm OK with the `block=True` pattern changing the return value I think.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/683#issuecomment-590599257
issue_url: https://api.github.com/repos/simonw/datasette/issues/683
node_id: MDEyOklzc3VlQ29tbWVudDU5MDU5OTI1Nw==
created_at: 2020-02-24T23:21:56Z
updated_at: 2020-02-24T23:22:35Z
author_association: OWNER
body:
  Also: are UUIDs really necessary here or could I use a simpler form of task identifier? Like an in-memory counter variable that starts at 0 and increments every time this instance of Datasette issues a new task ID?

  The neat thing about UUIDs is that I don't have to worry if there are multiple Datasette instances accepting writes behind a load balancer. That seems pretty unlikely (especially considering SQLite databases encourage only one process to be writing at a time)... but I am experimenting with PostgreSQL support in #670 so it's probably worth ensuring these task IDs really are globally unique.

  I'm going to stick with UUIDs. They're short-lived enough that their size doesn't really matter.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/683#issuecomment-590606825
issue_url: https://api.github.com/repos/simonw/datasette/issues/683
node_id: MDEyOklzc3VlQ29tbWVudDU5MDYwNjgyNQ==
created_at: 2020-02-24T23:47:38Z
updated_at: 2020-02-24T23:47:38Z
author_association: OWNER
body:
  Another demo plugin: `delete_table.py`
  ```python
  from datasette import hookimpl
  from datasette.utils import escape_sqlite
  from starlette.responses import HTMLResponse
  from starlette.endpoints import HTTPEndpoint


  class DeleteTableApp(HTTPEndpoint):
      def __init__(self, scope, receive, send, datasette):
          self.datasette = datasette
          super().__init__(scope, receive, send)

      async def post(self, request):
          formdata = await request.form()
          database = formdata["database"]
          db = self.datasette.databases[database]
          await db.execute_write("drop table {}".format(escape_sqlite(formdata["table"])))
          return HTMLResponse("Table has been deleted.")


  @hookimpl
  def asgi_wrapper(datasette):
      def wrap_with_asgi_auth(app):
          async def wrapped_app(scope, recieve, send):
              if scope["path"] == "/-/delete-table":
                  await DeleteTableApp(scope, recieve, send, datasette)
              else:
                  await app(scope, recieve, send)

          return wrapped_app

      return wrap_with_asgi_auth
  ```
  Then I saved this as `table.html` in the `write-templates/` directory:
  ```html+django
  {% extends "default:table.html" %}

  {% block content %}
  <form action="/-/delete-table" method="POST">
      <p>
          <input type="hidden" name="database" value="{{ database }}">
          <input type="hidden" name="table" value="{{ table }}">
          <input type="submit" value="Delete this table">
      </p>
  </form>
  {{ super() }}
  {% endblock %}
  ```
  (Needs CSRF protection added)

  I ran Datasette like this:

      $ datasette --plugins-dir=write-plugins/ data.db --template-dir=write-templates/

  Result: I can delete tables!

  <img width="596" alt="data__everything__30_132_rows_-_Mozilla_Firefox" src="https://user-images.githubusercontent.com/9599/75201302-f9cec580-571c-11ea-9c55-67a49e68ec0c.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/683#issuecomment-590607385
issue_url: https://api.github.com/repos/simonw/datasette/issues/683
node_id: MDEyOklzc3VlQ29tbWVudDU5MDYwNzM4NQ==
created_at: 2020-02-24T23:49:37Z
updated_at: 2020-02-24T23:49:37Z
author_association: OWNER
body:
  Here's the `upload_csv.py` plugin file I've been playing with:
  ```python
  from datasette import hookimpl
  from starlette.responses import PlainTextResponse, HTMLResponse
  from starlette.endpoints import HTTPEndpoint
  import csv as csv_std
  import codecs
  import sqlite_utils


  class UploadApp(HTTPEndpoint):
      def __init__(self, scope, receive, send, datasette):
          self.datasette = datasette
          super().__init__(scope, receive, send)

      def get_database(self):
          # For the moment just use the first one that's not immutable
          mutable = [db for db in self.datasette.databases.values() if db.is_mutable]
          return mutable[0]

      async def get(self, request):
          return HTMLResponse(
              await self.datasette.render_template(
                  "upload_csv.html", {"database_name": self.get_database().name}
              )
          )

      async def post(self, request):
          formdata = await request.form()
          csv = formdata["csv"]
          # csv.file is a SpooledTemporaryFile, I can read it directly
          filename = csv.filename
          # TODO: Support other encodings:
          reader = csv_std.reader(codecs.iterdecode(csv.file, "utf-8"))
          headers = next(reader)
          docs = (dict(zip(headers, row)) for row in reader)
          if filename.endswith(".csv"):
              filename = filename[:-4]
          # Import data into a table of that name using sqlite-utils
          db = self.get_database()

          def fn(conn):
              writable_conn = sqlite_utils.Database(db.path)
              writable_conn[filename].insert_all(docs, alter=True)
              return writable_conn[filename].count

          # Without block=True we may attempt 'select count(*) from ...'
          # before the table has been created by the write thread
          count = await db.execute_write_fn(fn, block=True)

          return HTMLResponse(
              await self.datasette.render_template(
                  "upload_csv_done.html",
                  {
                      "database": self.get_database().name,
                      "table": filename,
                      "num_docs": count,
                  },
              )
          )


  @hookimpl
  def asgi_wrapper(datasette):
      def wrap_with_asgi_auth(app):
          async def wrapped_app(scope, recieve, send):
              if scope["path"] == "/-/upload-csv":
                  await UploadApp(scope, recieve, send, datasette)
              else:
                  await app(scope, recieve, send)

          return wrapped_app

      return wrap_with_asgi_auth
  ```
  I also dropped copies of the two template files from https://github.com/simonw/datasette-upload-csvs/tree/699e6ca591f36264bfc8e590d877e6852f274beb/datasette_upload_csvs/templates into my `write-templates/` directory.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/683#issuecomment-590608228
issue_url: https://api.github.com/repos/simonw/datasette/issues/683
node_id: MDEyOklzc3VlQ29tbWVudDU5MDYwODIyOA==
created_at: 2020-02-24T23:52:35Z
updated_at: 2020-02-24T23:52:35Z
author_association: OWNER
body:
  I'm going to punt on the ability to introspect the write queue and poll for completion using a UUID for the moment. Can add those later.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/683#issuecomment-590610180
issue_url: https://api.github.com/repos/simonw/datasette/issues/683
node_id: MDEyOklzc3VlQ29tbWVudDU5MDYxMDE4MA==
created_at: 2020-02-25T00:00:07Z
updated_at: 2020-02-25T00:00:07Z
author_association: OWNER
body:
  Basic stuff to cover in unit tests:
  - Exercise `.execute_write(sql)` - both with block=True and block=False
  - Exercise `.execute_write_fn(fn)` in the same way
  - Throw 10 updates in the queue, block on just the last one, check it worked correctly

  I'm going to write these tests directly against a `Database()` object rather than booting up an entire Datasette instance.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/683#issuecomment-590614896
issue_url: https://api.github.com/repos/simonw/datasette/issues/683
node_id: MDEyOklzc3VlQ29tbWVudDU5MDYxNDg5Ng==
created_at: 2020-02-25T00:16:51Z
updated_at: 2020-02-25T00:16:51Z
author_association: OWNER
body:
  The other problem with the poll-for-UUID-completion idea: how long does this mean Datasette needs to keep holding onto the `WriteTask` objects?

  Maybe we say you only get to ask "is this UUID still in the queue" and if the answer is "no" then you assume the task has been completed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/683#issuecomment-590617822
issue_url: https://api.github.com/repos/simonw/datasette/issues/683
node_id: MDEyOklzc3VlQ29tbWVudDU5MDYxNzgyMg==
created_at: 2020-02-25T00:26:48Z
updated_at: 2020-02-25T00:26:48Z
author_association: OWNER
body:
  This failing test is a nasty one - the whole thing just hangs (so I imagine Travis will run for a while before hopefully giving up). Here's what happens if I add `--full-trace` and then hit Ctrl+C to cancel a test run:
  ```
  $ pytest -k test_execute_write_fn_block_true --full-trace
  =================================================================== test session starts ===================================================================
  platform darwin -- Python 3.7.5, pytest-5.2.4, py-1.8.1, pluggy-0.13.1
  rootdir: /Users/simonw/Dropbox/Development/datasette, inifile: pytest.ini
  plugins: asyncio-0.10.0
  collected 410 items / 409 deselected / 1 selected                                                                                                         

  tests/test_database.py ^C^C

  ================================================================= 409 deselected in 4.45s =================================================================
  Traceback (most recent call last):
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py", line 193, in wrap_session
      session.exitstatus = doit(config, session) or 0
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py", line 237, in _main
      config.hook.pytest_runtestloop(session=session)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
      return self._hookexec(self, self.get_hookimpls(), kwargs)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
      return self._inner_hookexec(hook, methods, kwargs)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
      firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
      return outcome.get_result()
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
      raise ex[1].with_traceback(ex[2])
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
      res = hook_impl.function(*args)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py", line 258, in pytest_runtestloop
      item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
      return self._hookexec(self, self.get_hookimpls(), kwargs)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
      return self._inner_hookexec(hook, methods, kwargs)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
      firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
      return outcome.get_result()
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
      raise ex[1].with_traceback(ex[2])
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
      res = hook_impl.function(*args)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 80, in pytest_runtest_protocol
      runtestprotocol(item, nextitem=nextitem)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 95, in runtestprotocol
      reports.append(call_and_report(item, "call", log))
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 176, in call_and_report
      call = call_runtest_hook(item, when, **kwds)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 201, in call_runtest_hook
      lambda: ihook(item=item, **kwds), when=when, reraise=reraise
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 229, in from_call
      result = func()
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 201, in <lambda>
      lambda: ihook(item=item, **kwds), when=when, reraise=reraise
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
      return self._hookexec(self, self.get_hookimpls(), kwargs)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
      return self._inner_hookexec(hook, methods, kwargs)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
      firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
      return outcome.get_result()
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
      raise ex[1].with_traceback(ex[2])
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
      res = hook_impl.function(*args)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 125, in pytest_runtest_call
      item.runtest()
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/python.py", line 1429, in runtest
      self.ihook.pytest_pyfunc_call(pyfuncitem=self)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
      return self._hookexec(self, self.get_hookimpls(), kwargs)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
      return self._inner_hookexec(hook, methods, kwargs)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
      firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
      return outcome.get_result()
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
      raise ex[1].with_traceback(ex[2])
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
      res = hook_impl.function(*args)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pytest_asyncio/plugin.py", line 158, in pytest_pyfunc_call
      pyfuncitem.obj(**testargs), loop=event_loop))
    File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 566, in run_until_complete
      self.run_forever()
    File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 534, in run_forever
      self._run_once()
    File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 1735, in _run_once
      event_list = self._selector.select(timeout)
    File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/selectors.py", line 558, in select
      kev_list = self._selector.control(None, max_ev, timeout)
  KeyboardInterrupt

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/bin/pytest", line 8, in <module>
      sys.exit(main())
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/config/__init__.py", line 90, in main
      return config.hook.pytest_cmdline_main(config=config)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
      return self._hookexec(self, self.get_hookimpls(), kwargs)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
      return self._inner_hookexec(hook, methods, kwargs)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
      firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
      return outcome.get_result()
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
      raise ex[1].with_traceback(ex[2])
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
      res = hook_impl.function(*args)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py", line 230, in pytest_cmdline_main
      return wrap_session(config, _main)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py", line 209, in wrap_session
      config.hook.pytest_keyboard_interrupt(excinfo=excinfo)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
      return self._hookexec(self, self.get_hookimpls(), kwargs)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
      return self._inner_hookexec(hook, methods, kwargs)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
      firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
      return outcome.get_result()
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
      raise ex[1].with_traceback(ex[2])
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
      res = hook_impl.function(*args)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/terminal.py", line 680, in pytest_keyboard_interrupt
      self._keyboardinterrupt_memo = excinfo.getrepr(funcargs=True)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py", line 598, in getrepr
      return fmt.repr_excinfo(self)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py", line 830, in repr_excinfo
      reprtraceback = self.repr_traceback(excinfo)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py", line 778, in repr_traceback
      reprentry = self.repr_traceback_entry(entry, einfo)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py", line 737, in repr_traceback_entry
      reprargs = self.repr_args(entry) if not short else None
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py", line 656, in repr_args
      args.append((argname, saferepr(argvalue)))
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_io/saferepr.py", line 67, in saferepr
      return SafeRepr(maxsize).repr(obj)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_io/saferepr.py", line 36, in repr
      s = super().repr(x)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/reprlib.py", line 52, in repr
      return self.repr1(x, self.maxlevel)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/reprlib.py", line 60, in repr1
      return getattr(self, 'repr_' + typename)(x, level)
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/reprlib.py", line 112, in repr_dict
      for key in islice(_possibly_sorted(x), self.maxdict):
  KeyboardInterrupt
  Task was destroyed but it is pending!
  task: <Task pending coro=<test_execute_write_fn_block_true() running at /Users/simonw/Dropbox/Development/datasette/tests/test_database.py:139> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x105580510>()]>>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/567#issuecomment-590667545
issue_url: https://api.github.com/repos/simonw/datasette/issues/567
node_id: MDEyOklzc3VlQ29tbWVudDU5MDY2NzU0NQ==
created_at: 2020-02-25T03:40:49Z
updated_at: 2020-02-25T03:40:49Z
author_association: OWNER
body:
  Closing this in favour of #682 - a mechanism for writing to databases from a clue which is being developed in pull request #683.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/684#issuecomment-590675727
issue_url: https://api.github.com/repos/simonw/datasette/issues/684
node_id: MDEyOklzc3VlQ29tbWVudDU5MDY3NTcyNw==
created_at: 2020-02-25T04:20:42Z
updated_at: 2020-02-26T00:03:19Z
author_association: OWNER
body:
  Here's what I have so far:
  ```
  db.name

  db.size

  db.mtime_ns

  await db.table_exists(table)

  await db.table_names()

  await db.table_columns(table)

  await db.primary_keys(table)

  Returns the names of the columns that are part of the primary key for this table.

  await db.fts_table(table)

  Returns the name of the FTS table associated with this table, if one exists.

  await db.label_column_for_table(table)

  Returns the label column that is associated with this table - either automatically detected or using the ``"label_column"`` key from :ref:`metadata`.

  await db.foreign_keys_for_table(table)

  await db.hidden_table_names()

  await db.view_names()

  await db.get_all_foreign_keys()

  await db.get_outbound_foreign_keys(table)

  await db.get_table_definition(table)

  await db.get_view_definition(view)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/683#issuecomment-590681676
issue_url: https://api.github.com/repos/simonw/datasette/issues/683
node_id: MDEyOklzc3VlQ29tbWVudDU5MDY4MTY3Ng==
created_at: 2020-02-25T04:48:29Z
updated_at: 2020-02-25T04:48:29Z
author_association: OWNER
body:
  Documentation: https://datasette.readthedocs.io/en/latest/internals.html#await-db-execute-write-sql-params-none-block-false

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/685#issuecomment-590682210
issue_url: https://api.github.com/repos/simonw/datasette/issues/685
node_id: MDEyOklzc3VlQ29tbWVudDU5MDY4MjIxMA==
created_at: 2020-02-25T04:50:53Z
updated_at: 2020-02-25T04:50:53Z
author_association: OWNER
body:
  Current implementations:

  https://github.com/simonw/datasette/blob/a093c5f79fa034a97d2ad8b606745dd3b80365af/datasette/database.py#L103-L168

  At the very least the method name `execute_against_connection_in_thread()` should be updated to something that's more similar to the new (and documented) `.execute_write_fn()` method.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/668#issuecomment-590687137
issue_url: https://api.github.com/repos/simonw/datasette/issues/668
node_id: MDEyOklzc3VlQ29tbWVudDU5MDY4NzEzNw==
created_at: 2020-02-25T05:11:35Z
updated_at: 2020-02-25T05:11:35Z
author_association: OWNER
body:
  I can imitate how `shapefile-to-SQLite` does it: https://github.com/simonw/shapefile-to-sqlite/blob/5e3fb00eb1eee6ae0103a9ce4167a5e5ec17cf9f/shapefile_to_sqlite/utils.py#L105

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/676#issuecomment-590697977
issue_url: https://api.github.com/repos/simonw/datasette/issues/676
node_id: MDEyOklzc3VlQ29tbWVudDU5MDY5Nzk3Nw==
created_at: 2020-02-25T05:57:24Z
updated_at: 2020-02-25T05:57:24Z
author_association: OWNER
body:
  OK, this is in master now. You can install master using:

      pip install https://github.com/simonw/datasette/archive/master.zip

  Documentation here: https://datasette.readthedocs.io/en/latest/json_api.html#special-table-arguments

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591768604
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/88
node_id: MDEyOklzc3VlQ29tbWVudDU5MTc2ODYwNA==
created_at: 2020-02-27T04:03:03Z
updated_at: 2020-02-27T04:03:03Z
author_association: OWNER
body:
  `drop table resources_fts` drops the FTS table and the other ones that it created (resources_fts_data, resources_fts_idx, resources_fts_docsize, resources_fts_config) - but keeps the triggers.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769046
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/88
node_id: MDEyOklzc3VlQ29tbWVudDU5MTc2OTA0Ng==
created_at: 2020-02-27T04:05:15Z
updated_at: 2020-02-27T04:05:15Z
author_association: OWNER
body:
  I can reliably get the list of triggers to delete from `select name from sqlite_master where type = 'trigger' and tbl_name = 'resources';`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769171
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/88
node_id: MDEyOklzc3VlQ29tbWVudDU5MTc2OTE3MQ==
created_at: 2020-02-27T04:05:58Z
updated_at: 2020-02-27T04:26:31Z
author_association: OWNER
body:
  Strange - https://www.sqlite.org/lang_droptrigger.html says "Note that triggers are automatically dropped when the associated table is dropped" but that doesn't seem to be true in my experimenting.

  UPDATE: no that makes sense - the triggers are on `resources` which still exists, it was `resources_fts` that was dropped.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769373
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/88
node_id: MDEyOklzc3VlQ29tbWVudDU5MTc2OTM3Mw==
created_at: 2020-02-27T04:06:47Z
updated_at: 2020-02-27T04:06:47Z
author_association: OWNER
body:
  Looks like safest option is to loop through those trigger names and run `DROP TRIGGER IF EXISTS foo` on each one.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769759
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/88
node_id: MDEyOklzc3VlQ29tbWVudDU5MTc2OTc1OQ==
created_at: 2020-02-27T04:08:29Z
updated_at: 2020-02-27T04:08:29Z
author_association: OWNER
body:
  I think the method should be called `table.disable_fts()` - the opposite of `table.enable_fts(...)`.

  There should be a `sqlite-utils disable-fts database.db tablename` command to match it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/86#issuecomment-591770623
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/86
node_id: MDEyOklzc3VlQ29tbWVudDU5MTc3MDYyMw==
created_at: 2020-02-27T04:12:39Z
updated_at: 2020-02-27T04:12:39Z
author_association: OWNER
body:
  I pushed a branch with my experiment in it, but I'm going to fix this by throwing an error on `[` or `]` in a column name instead - I won't implement the changes from that branch.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/580#issuecomment-592395768
issue_url: https://api.github.com/repos/simonw/datasette/issues/580
node_id: MDEyOklzc3VlQ29tbWVudDU5MjM5NTc2OA==
created_at: 2020-02-28T07:58:46Z
updated_at: 2020-02-28T07:58:46Z
author_association: OWNER
body:
  I don't think this is needed any more - I've found a really nice pattern for testing plugins using `httpx` instead, e.g. https://github.com/simonw/datasette-mask-columns/blob/f6f08ce21bdfc2731a02c42e1c9e25b4df769292/tests/test_mask_columns.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/581#issuecomment-592621235
issue_url: https://api.github.com/repos/simonw/datasette/issues/581
node_id: MDEyOklzc3VlQ29tbWVudDU5MjYyMTIzNQ==
created_at: 2020-02-28T17:24:06Z
updated_at: 2020-05-27T21:12:21Z
author_association: OWNER
body:
  Rather than pass a request object (hence promoting that object into part of the documented, stable API) I think I'll pass the ASGI scope - that's already a stable, documented standard.

  UPDATE: changed my mind since `request` is used by other plugins too, see #706.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/687#issuecomment-592626515
issue_url: https://api.github.com/repos/simonw/datasette/issues/687
node_id: MDEyOklzc3VlQ29tbWVudDU5MjYyNjUxNQ==
created_at: 2020-02-28T17:27:08Z
updated_at: 2020-02-28T17:27:46Z
author_association: OWNER
body:
  Suggested pages:

  - `plugins/tutorial.html`
  - `plugins/hooks.html`
  - `plugins/testing.html`
  - `plugins/packaging.html`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/649#issuecomment-592798079
issue_url: https://api.github.com/repos/simonw/datasette/issues/649
node_id: MDEyOklzc3VlQ29tbWVudDU5Mjc5ODA3OQ==
created_at: 2020-02-29T01:00:16Z
updated_at: 2020-02-29T01:00:16Z
author_association: OWNER
body:
  I tried this out in https://github.com/simonw/datasette/commit/34bc16aa591a2f02b2bb33277d33e5bccfb95569 with an upper limit of 100MB and it worked an absolute treat against my personal Dogsheep.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/687#issuecomment-592814114
issue_url: https://api.github.com/repos/simonw/datasette/issues/687
node_id: MDEyOklzc3VlQ29tbWVudDU5MjgxNDExNA==
created_at: 2020-02-29T02:07:03Z
updated_at: 2020-02-29T02:07:03Z
author_association: OWNER
body:
  The tutorial should start by taking about development environments - how to get a new fresh copy of Datasette that you can start hacking plugins for, in a fresh virtual environment.

  It should then explain the `--plugins-dir` mechanism, and show how to use it to create a simple plugin that adds a custom SQL function.

  Does `--reload` work against plugins in the `plugins/` directory? It should.

  Then it should demonstrate a more complex plugin that uses the `datasette` argument to render a template.

  It should promote the next page which lists the plugin hooks.

  The hooks page should include links next to each hook to examples of plugins that use them.

  The packaging page can explain `setup.py` for distributing packages (let's not bother explaining single modules) and also needs to show how to bundle package data such as templates.

  It should also show how to use `pip install -e .[test]` for development, and mention that this works with `--reload`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/687#issuecomment-592814610
issue_url: https://api.github.com/repos/simonw/datasette/issues/687
node_id: MDEyOklzc3VlQ29tbWVudDU5MjgxNDYxMA==
created_at: 2020-02-29T02:08:49Z
updated_at: 2020-02-29T02:08:49Z
author_association: OWNER
body:
  Eventually I can also document what to do if your plugin needs to include its own CLI tool. I'm goong it show how to ship a CLI tool with the same name as the plugin using Click and `setup.py`. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/46#issuecomment-592999503
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/46
node_id: MDEyOklzc3VlQ29tbWVudDU5Mjk5OTUwMw==
created_at: 2020-02-29T22:08:20Z
updated_at: 2020-02-29T22:08:20Z
author_association: NONE
body:
  @simonw any thoughts on allow extracts to specify the lookup column name? If I'm understanding the documentation right, `.lookup()` allows you to define the "value" column (the documentation uses name), but when you use `extracts` keyword as part of `.insert()`, `.upsert()` etc. the lookup must be done against a column named "value". I have an existing lookup table that I've populated with columns "id" and "name" as opposed to "id" and "value", and seems I can't use `extracts=`, unless I'm missing something...

  Initial thought on how to do this would be to allow the dictionary value to be a tuple of table name column pair... so:
  ```
  table = db.table("trees", extracts={"species_id": ("Species", "name"})
  ```

  I haven't dug too much into the existing code yet, but does this make sense? Worth doing?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/573#issuecomment-593026413
issue_url: https://api.github.com/repos/simonw/datasette/issues/573
node_id: MDEyOklzc3VlQ29tbWVudDU5MzAyNjQxMw==
created_at: 2020-03-01T01:24:45Z
updated_at: 2020-03-01T01:24:45Z
author_association: CONTRIBUTOR
body:
  Did you manage to find an answer to this? I've got a notebook to help people generate datasets on the fly from an API, so it would be cool if they flick it to Datasette for initial exploration.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/89#issuecomment-593122605
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/89
node_id: MDEyOklzc3VlQ29tbWVudDU5MzEyMjYwNQ==
created_at: 2020-03-01T17:33:11Z
updated_at: 2020-03-01T17:33:11Z
author_association: NONE
body:
  If you're happy with the proposed implementation, I have code & tests written that I'll get ready for a PR.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/690#issuecomment-593247614
issue_url: https://api.github.com/repos/simonw/datasette/issues/690
node_id: MDEyOklzc3VlQ29tbWVudDU5MzI0NzYxNA==
created_at: 2020-03-02T06:49:12Z
updated_at: 2020-03-02T06:49:12Z
author_association: OWNER
body:
  The answer may lie in a plugin hook which returns a list of HTML fragments to be added to the page in a particular place.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/692#issuecomment-593446034
issue_url: https://api.github.com/repos/simonw/datasette/issues/692
node_id: MDEyOklzc3VlQ29tbWVudDU5MzQ0NjAzNA==
created_at: 2020-03-02T15:03:48Z
updated_at: 2020-03-02T15:03:48Z
author_association: OWNER
body:
  Workaround I used:
  ```python
  @hookimpl
  def extra_template_vars(datasette, database, table):
      async def is_hidden_table():
          if database and table:
              try:
                  db = datasette.databases[database]
              except KeyError:
                  return {"is_hidden_table": False}
              return {"is_hidden_table": table in (await db.hidden_table_names())}
          else:
              return {"is_hidden_table": False}

      return is_hidden_table
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/693#issuecomment-593452830
issue_url: https://api.github.com/repos/simonw/datasette/issues/693
node_id: MDEyOklzc3VlQ29tbWVudDU5MzQ1MjgzMA==
created_at: 2020-03-02T15:16:20Z
updated_at: 2020-03-02T15:16:20Z
author_association: OWNER
body:
  Here's why:
  https://github.com/simonw/datasette/blob/613f6fad7249da547c06ebc46616aba0f74259a4/datasette/views/base.py#L84-L96
  It's bypassing the `.ds.render_template()` call - and that's where the plugin hook is executed:
  https://github.com/simonw/datasette/blob/613f6fad7249da547c06ebc46616aba0f74259a4/datasette/app.py#L520-L564

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-594151327
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12
node_id: MDEyOklzc3VlQ29tbWVudDU5NDE1MTMyNw==
created_at: 2020-03-03T20:26:15Z
updated_at: 2020-03-03T20:32:23Z
author_association: MEMBER
body:
  Better version (since this also includes JSON array of repository topics):
  ```sql
  CREATE VIEW recent_releases AS select
    repos.rowid as rowid,
    json_object("label", repos.full_name, "href", repos.html_url) as repo,
    json_object(
      "href",
      releases.html_url,
      "label",
      releases.name
    ) as release,
    substr(releases.published_at, 0, 11) as date,
    releases.body as body_markdown,
    releases.published_at,
    coalesce(repos.topics, '[]') as topics
  from
    releases
    join repos on repos.id = releases.repo
  order by
    releases.published_at desc
  ```
  That `repos.rowid as rowid` bit is necessary because otherwise clicking on a link in facet-by-topic doesn't return any results.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-594155249
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12
node_id: MDEyOklzc3VlQ29tbWVudDU5NDE1NTI0OQ==
created_at: 2020-03-03T20:35:17Z
updated_at: 2020-03-03T20:35:17Z
author_association: MEMBER
body:
  `swarm-to-sqlite` has an example of adding views here: https://github.com/dogsheep/swarm-to-sqlite/blob/f2c89dd613fb8a7f14e5267ccc2145463b996190/swarm_to_sqlite/utils.py#L141

  I think that approach can be approved by first checking if the view exists, then dropping it, then recreating it. Could even try to see if the view exists and matches what we were going to set it to and do nothing if that is the case.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/694#issuecomment-595489222
issue_url: https://api.github.com/repos/simonw/datasette/issues/694
node_id: MDEyOklzc3VlQ29tbWVudDU5NTQ4OTIyMg==
created_at: 2020-03-05T23:00:33Z
updated_at: 2020-03-05T23:00:33Z
author_association: OWNER
body:
  The initial `datasette publish cloudrun` failed, now I can replicate that error by running:
  ```
  gcloud run deploy --allow-unauthenticated --platform=managed \
    --image gcr.io/datasette-222320/datasette non-profit-ethics
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/694#issuecomment-595489514
issue_url: https://api.github.com/repos/simonw/datasette/issues/694
node_id: MDEyOklzc3VlQ29tbWVudDU5NTQ4OTUxNA==
created_at: 2020-03-05T23:01:35Z
updated_at: 2020-03-05T23:01:35Z
author_association: OWNER
body:
  Aha! The logs said "Memory limit of 244M exceeded with 247M used. Consider increasing the memory limit, see https://cloud.google.com/run/docs/configuring/memory-limits"

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/694#issuecomment-595490889
issue_url: https://api.github.com/repos/simonw/datasette/issues/694
node_id: MDEyOklzc3VlQ29tbWVudDU5NTQ5MDg4OQ==
created_at: 2020-03-05T23:06:30Z
updated_at: 2020-03-05T23:06:30Z
author_association: OWNER
body:
  This fixed it (I tried 1Gi first but that gave the same error):
  ```
  gcloud run deploy --allow-unauthenticated --platform=managed \
    --image gcr.io/datasette-222320/datasette non-profit-ethics --memory=2Gi
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/694#issuecomment-595491182
issue_url: https://api.github.com/repos/simonw/datasette/issues/694
node_id: MDEyOklzc3VlQ29tbWVudDU5NTQ5MTE4Mg==
created_at: 2020-03-05T23:07:33Z
updated_at: 2020-03-05T23:45:38Z
author_association: OWNER
body:
  So two things I need to do for this:

  * Add a `--memory` option to `datasette publish cloudrun`
  * Maybe capture this error and output a helpful suggestion that you increase the memory with that option? Not sure how feasible that is.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/694#issuecomment-595492478
issue_url: https://api.github.com/repos/simonw/datasette/issues/694
node_id: MDEyOklzc3VlQ29tbWVudDU5NTQ5MjQ3OA==
created_at: 2020-03-05T23:12:25Z
updated_at: 2020-03-05T23:12:25Z
author_association: OWNER
body:
  I wonder if there's some weird reason that we churn through too much RAM on initial datasette startup here? I wouldn't expect startup to use a huge spike of RAM. Maybe need to profile a bit.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/694#issuecomment-595498926
issue_url: https://api.github.com/repos/simonw/datasette/issues/694
node_id: MDEyOklzc3VlQ29tbWVudDU5NTQ5ODkyNg==
created_at: 2020-03-05T23:35:32Z
updated_at: 2020-03-05T23:35:32Z
author_association: OWNER
body:
  Tested that with:
  ```
  datasette publish cloudrun fixtures.db --memory 512Mi --service fixtures-memory-512mi
  ```
  Here's the result in the Google Cloud web console:

  <img width="557" alt="fixtures-m…ory-512mi_–_Service_details_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform_-_Mozilla_Firefox" src="https://user-images.githubusercontent.com/9599/76035699-b1c35600-5f07-11ea-8eee-7a1e159e50f7.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/695#issuecomment-595610265
issue_url: https://api.github.com/repos/simonw/datasette/issues/695
node_id: MDEyOklzc3VlQ29tbWVudDU5NTYxMDI2NQ==
created_at: 2020-03-06T05:43:36Z
updated_at: 2020-03-06T05:43:36Z
author_association: OWNER
body:
  From https://www.sqlite.org/download.html the download URL is https://www.sqlite.org/2020/sqlite-autoconf-3310100.tar.gz

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/695#issuecomment-595610855
issue_url: https://api.github.com/repos/simonw/datasette/issues/695
node_id: MDEyOklzc3VlQ29tbWVudDU5NTYxMDg1NQ==
created_at: 2020-03-06T05:46:11Z
updated_at: 2020-03-06T05:46:11Z
author_association: OWNER
body:
  On my laptop I can build and test like so:

      cd datasette
      docker build -t datasette-sqlite .
      docker run -p 8001:8001 -v `pwd`:/mnt \
          datasette-sqlite \
          datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/695#issuecomment-595611438
issue_url: https://api.github.com/repos/simonw/datasette/issues/695
node_id: MDEyOklzc3VlQ29tbWVudDU5NTYxMTQzOA==
created_at: 2020-03-06T05:48:44Z
updated_at: 2020-03-06T05:48:44Z
author_association: OWNER
body:
  Need to test with SpatiaLite here too.
  ```
  docker run -p 8001:8001 -v `pwd`:/mnt \
      datasette-sqlite \
      datasette -p 8001 -h 0.0.0.0 /mnt/spatial.db \
      --load-extension=/usr/local/lib/mod_spatialite.so
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/695#issuecomment-595614469
issue_url: https://api.github.com/repos/simonw/datasette/issues/695
node_id: MDEyOklzc3VlQ29tbWVudDU5NTYxNDQ2OQ==
created_at: 2020-03-06T06:01:16Z
updated_at: 2020-03-06T06:01:16Z
author_association: OWNER
body:
  Here's how to run the unit test suite inside the freshly built Docker container (built with a new SQLite):
  ```
  docker run -it -v `pwd`:/mnt datasette-sqlite /bin/bash
  root@0e1928cfdf79:/# cd /mnt
  root@0e1928cfdf79:/mnt# pip install -e .[test]
  root@0e1928cfdf79:/mnt# pytest
  ```
  I ran that against SQLite 3.31.1 and got one failure!

  It was for `test_searchable[/fixtures/searchable.json?_search=te*+AND+do*&_searchmode=raw-expected_rows3]`
  ```
      def test_searchable(app_client, path, expected_rows):
          response = app_client.get(path)
  >       assert expected_rows == response.json["rows"]
  E       AssertionError: assert [[1, 'barry c...sel', 'puma']] == []
  E         Left contains 2 more items, first extra item: [1, 'barry cat', 'terry dog', 'panther']
  E         Full diff:
  E         + []
  E         - [[1, 'barry cat', 'terry dog', 'panther'],
  E         -  [2, 'terry dog', 'sara weasel', 'puma']]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/695#issuecomment-595617072
issue_url: https://api.github.com/repos/simonw/datasette/issues/695
node_id: MDEyOklzc3VlQ29tbWVudDU5NTYxNzA3Mg==
created_at: 2020-03-06T06:12:07Z
updated_at: 2020-03-06T06:12:07Z
author_association: OWNER
body:
  Well that's odd. If I run the tests against SQLite 3.26.0 inside the container I get the same failure there as well:

  ```
  docker run -it -v `pwd`:/mnt datasetteproject/datasette:0.37.1 /bin/bash
  root@710eed6278b2:/# cd /mnt
  root@710eed6278b2:/mnt# pip install -e .[test] && pytest
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/695#issuecomment-595617762
issue_url: https://api.github.com/repos/simonw/datasette/issues/695
node_id: MDEyOklzc3VlQ29tbWVudDU5NTYxNzc2Mg==
created_at: 2020-03-06T06:14:53Z
updated_at: 2020-03-06T06:14:53Z
author_association: OWNER
body:
  SpatiaLite testing works fine. I'm going to ship this even though there's a single failing test, since that's not a regression from the previous Docker release. I'm going to file a separate issue about that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/697#issuecomment-596261835
issue_url: https://api.github.com/repos/simonw/datasette/issues/697
node_id: MDEyOklzc3VlQ29tbWVudDU5NjI2MTgzNQ==
created_at: 2020-03-08T22:45:03Z
updated_at: 2020-03-08T22:45:03Z
author_association: OWNER
body:
  I think the bug is in this code:
  https://github.com/simonw/datasette/blob/7508477a96492caa1893c52411405656c50d30f9/datasette/app.py#L612-L623
  I added some debug logging to it and found that the bit that returns the `plugin["templates_path"]` list is producing this:
  ```
  plugin_template_paths: [
    '/Users/simonw/Dropbox/Development/datasette/datasette/templates',
    '/Users/simonw/Dropbox/Development/datasette/datasette/templates',
    '/Users/simonw/Dropbox/Development/datasette-search-all/datasette_search_all/templates'
  ]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/697#issuecomment-596261973
issue_url: https://api.github.com/repos/simonw/datasette/issues/697
node_id: MDEyOklzc3VlQ29tbWVudDU5NjI2MTk3Mw==
created_at: 2020-03-08T22:46:47Z
updated_at: 2020-03-08T22:46:47Z
author_association: OWNER
body:
  More debugging code shows this:
  ```
  [
      (
          {
              "name": "datasette.facets",
              "static_path": "/Users/simonw/Dropbox/Development/datasette/datasette/static",
              "templates_path": "/Users/simonw/Dropbox/Development/datasette/datasette/templates",
          },
          "/Users/simonw/Dropbox/Development/datasette/datasette/templates",
      ),
      (
          {
              "name": "datasette.sql_functions",
              "static_path": "/Users/simonw/Dropbox/Development/datasette/datasette/static",
              "templates_path": "/Users/simonw/Dropbox/Development/datasette/datasette/templates",
          },
          "/Users/simonw/Dropbox/Development/datasette/datasette/templates",
      ),
      (
          {
              "name": "datasette-search-all",
              "static_path": None,
              "templates_path": "/Users/simonw/Dropbox/Development/datasette-search-all/datasette_search_all/templates",
              "version": "0.1",
          },
          "/Users/simonw/Dropbox/Development/datasette-search-all/datasette_search_all/templates",
      ),
  ]
  ```
  So the problem is that the default plugins that ship with Datasette are adding a template path to the lookup list!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/697#issuecomment-596262118
issue_url: https://api.github.com/repos/simonw/datasette/issues/697
node_id: MDEyOklzc3VlQ29tbWVudDU5NjI2MjExOA==
created_at: 2020-03-08T22:48:28Z
updated_at: 2020-03-08T22:48:28Z
author_association: OWNER
body:
  I think the bug is in this code:
  https://github.com/simonw/datasette/blob/7508477a96492caa1893c52411405656c50d30f9/datasette/utils/__init__.py#L619-L645

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/697#issuecomment-596262197
issue_url: https://api.github.com/repos/simonw/datasette/issues/697
node_id: MDEyOklzc3VlQ29tbWVudDU5NjI2MjE5Nw==
created_at: 2020-03-08T22:49:15Z
updated_at: 2020-03-08T22:49:15Z
author_association: OWNER
body:
  I need that code to set `static_path` and `templates_path` to `None` for plugins that ship as part of Datasette.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/697#issuecomment-596264781
issue_url: https://api.github.com/repos/simonw/datasette/issues/697
node_id: MDEyOklzc3VlQ29tbWVudDU5NjI2NDc4MQ==
created_at: 2020-03-08T23:17:01Z
updated_at: 2020-03-08T23:17:01Z
author_association: OWNER
body:
  I'm going to release this as 0.37.2.

  This is the first time I've released a version NOT from master (since I don't want to ship the `--memory` feature for Cloud Run until 0.38 - #694). So I've cut a `v0.37.2` release branch instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/697#issuecomment-596264937
issue_url: https://api.github.com/repos/simonw/datasette/issues/697
node_id: MDEyOklzc3VlQ29tbWVudDU5NjI2NDkzNw==
created_at: 2020-03-08T23:18:48Z
updated_at: 2020-03-08T23:18:48Z
author_association: OWNER
body:
  Cancel that plan: I'm pretty sure the Travis configuration that publishes a demo to Zeit Now and builds a Docker image isn't designed to handle releases that don't correspond to current master. I guess I'll release 0.38 instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/690#issuecomment-596275489
issue_url: https://api.github.com/repos/simonw/datasette/issues/690
node_id: MDEyOklzc3VlQ29tbWVudDU5NjI3NTQ4OQ==
created_at: 2020-03-09T00:55:29Z
updated_at: 2020-03-09T00:55:29Z
author_association: OWNER
body:
  I just hacked around this again in `datasette-search-all`, which abuses the `description_source_license` block to add a search for to the top of the `index.html` template: https://github.com/simonw/datasette-search-all/blob/f9b2a9c3328f35021a4744205d862c108f8f08eb/datasette_search_all/templates/index.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/690#issuecomment-596296745
issue_url: https://api.github.com/repos/simonw/datasette/issues/690
node_id: MDEyOklzc3VlQ29tbWVudDU5NjI5Njc0NQ==
created_at: 2020-03-09T01:48:04Z
updated_at: 2020-03-09T01:48:53Z
author_association: OWNER
body:
  Maybe there are a few things here:
  * A mechanism for adding **actions** - essentially menu items - to tables and rows and queries and the index page
  * A way of defining blocks on the page where a plugin can add an entire block of custom HTML - above and below the table, or at the top of the index page for example.

  Anything fancier than this can be achieved using JavaScript - provided there are documented blocks with IDs that JavaScript code can then manipulate in sensible ways.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/17#issuecomment-597358364
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17
node_id: MDEyOklzc3VlQ29tbWVudDU5NzM1ODM2NA==
created_at: 2020-03-10T22:50:20Z
updated_at: 2020-03-11T01:18:36Z
author_association: MEMBER
body:
  By default it will stop when it sees a commit that has already been stored. You will be able to over-ride that behaviour using `--all`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599125455
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/92
node_id: MDEyOklzc3VlQ29tbWVudDU5OTEyNTQ1NQ==
created_at: 2020-03-14T19:34:35Z
updated_at: 2020-03-14T19:34:35Z
author_association: OWNER
body:
  From https://www.sqlite.org/datatype3.html it looks like `FLOAT` is a supported keyword for creating tables but `REAL` is the correct keyword.

  So actually `sqlite-utils` gets this wrong, because when we create a table we turn Python `float` values into a `FLOAT` column. Looks like the correct behaviour would be to turn them into a `REAL` column.

  https://github.com/simonw/sqlite-utils/blob/43f1c6ab4e3a6b76531fb6f5447adb83d26f3971/sqlite_utils/db.py#L28-L48

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599125557
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/92
node_id: MDEyOklzc3VlQ29tbWVudDU5OTEyNTU1Nw==
created_at: 2020-03-14T19:35:29Z
updated_at: 2020-03-14T19:35:29Z
author_association: OWNER
body:
  Fixing that would technically constitute a breaking change for library consumers, so it should be a major version release.

  I'm not inclined to release `3.0` just for this one issue, so I'm going to hold back on fixing that and address the smaller issue in this bug as a dot release instead for the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599126831
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/92
node_id: MDEyOklzc3VlQ29tbWVudDU5OTEyNjgzMQ==
created_at: 2020-03-14T19:45:28Z
updated_at: 2020-03-14T19:45:28Z
author_association: OWNER
body:
  Turns out there are a TON of valid column definitions that aren't being considered yet - https://www.sqlite.org/datatype3.html#affinity_name_examples - stuff like `VARYING CHARACTER(255)` and `DECIMAL(10,5)`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599127197
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/92
node_id: MDEyOklzc3VlQ29tbWVudDU5OTEyNzE5Nw==
created_at: 2020-03-14T19:48:06Z
updated_at: 2020-03-14T19:48:06Z
author_association: OWNER
body:
  Actually it looks like I should implement the exact rules described in https://www.sqlite.org/datatype3.html#determination_of_column_affinity

  > The affinity of a column is determined by the declared type of the column, according to the following rules in the order shown:
  > 
  > 1. If the declared type contains the string "INT" then it is assigned INTEGER affinity.
  > 2. If the declared type of the column contains any of the strings "CHAR", "CLOB", or "TEXT" then that column has TEXT affinity. Notice that the type VARCHAR contains the string "CHAR" and is thus assigned TEXT affinity.
  > 3. If the declared type for a column contains the string "BLOB" or if no type is specified then the column has affinity BLOB.
  > 4. If the declared type for a column contains any of the strings "REAL", "FLOA", or "DOUB" then the column has REAL affinity.
  > 5. Otherwise, the affinity is NUMERIC.
  > 
  > Note that the order of the rules for determining column affinity is important. A column whose declared type is "CHARINT" will match both rules 1 and 2 but the first rule takes precedence and so the column affinity will be INTEGER.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599127453
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/92
node_id: MDEyOklzc3VlQ29tbWVudDU5OTEyNzQ1Mw==
created_at: 2020-03-14T19:50:08Z
updated_at: 2020-03-14T19:50:08Z
author_association: OWNER
body:
  > If the declared type for a column contains the string "BLOB" or if no type is specified then the column has affinity BLOB

  I currently treat those as `str` - it sounds like I should treat them as `bytes`:

  https://github.com/simonw/sqlite-utils/blob/43f1c6ab4e3a6b76531fb6f5447adb83d26f3971/sqlite_utils/db.py#L68-L69

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/698#issuecomment-599696286
issue_url: https://api.github.com/repos/simonw/datasette/issues/698
node_id: MDEyOklzc3VlQ29tbWVudDU5OTY5NjI4Ng==
created_at: 2020-03-16T18:32:59Z
updated_at: 2020-03-16T18:33:33Z
author_association: OWNER
body:
  ```json
  {
      "databases": {
         "my-database": {
             "queries": {
                 "add_twitter_handle": {
                     "sql": "insert into twitter_handles (username) values (:username)",
                     "write": true
                 }
             }
         }
      }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/698#issuecomment-599697164
issue_url: https://api.github.com/repos/simonw/datasette/issues/698
node_id: MDEyOklzc3VlQ29tbWVudDU5OTY5NzE2NA==
created_at: 2020-03-16T18:35:04Z
updated_at: 2020-03-16T18:35:27Z
author_association: OWNER
body:
  By default this will extract the `:params` using the existing regular expression - which can occasionally break if there is a rogue `:` in the rest of the query.

  To address this: allow an extra optional `"params": ["username"]` field in metadata which, if available, is used instead of the regular expression extraction.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/120#issuecomment-599702870
issue_url: https://api.github.com/repos/simonw/datasette/issues/120
node_id: MDEyOklzc3VlQ29tbWVudDU5OTcwMjg3MA==
created_at: 2020-03-16T18:48:05Z
updated_at: 2020-03-16T18:48:05Z
author_association: OWNER
body:
  I've built two of these so far: https://github.com/simonw/datasette-auth-github and https://github.com/simonw/datasette-auth-existing-cookies

  Closing this ticket in favour of #699

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/698#issuecomment-599704264
issue_url: https://api.github.com/repos/simonw/datasette/issues/698
node_id: MDEyOklzc3VlQ29tbWVudDU5OTcwNDI2NA==
created_at: 2020-03-16T18:51:27Z
updated_at: 2020-03-17T02:44:44Z
author_association: OWNER
body:
  This is also going to need me to handle POST form submissions which means I need to be able to parse the form body. I guess that will go in [datasette/utils/asgi.py](https://github.com/simonw/datasette/blob/master/datasette/utils/asgi.py).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/698#issuecomment-599704655
issue_url: https://api.github.com/repos/simonw/datasette/issues/698
node_id: MDEyOklzc3VlQ29tbWVudDU5OTcwNDY1NQ==
created_at: 2020-03-16T18:52:23Z
updated_at: 2020-03-16T18:52:23Z
author_association: OWNER
body:
  I have code for form parsing in `asgi-csrf`: https://github.com/simonw/asgi-csrf/blob/3fc164a9258ebfa616d4e724fcb102b117277c6e/asgi_csrf.py#L91-L109

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/698#issuecomment-599706260
issue_url: https://api.github.com/repos/simonw/datasette/issues/698
node_id: MDEyOklzc3VlQ29tbWVudDU5OTcwNjI2MA==
created_at: 2020-03-16T18:56:00Z
updated_at: 2020-03-16T18:56:00Z
author_association: OWNER
body:
  It would be useful if this feature supported different types of input - at the most basic level `<input type="text">` v.s. `<textarea>`, but numbers, `<select>` etc would be useful too. This can be added later as additional `metadata.json` settings.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/700#issuecomment-599849497
issue_url: https://api.github.com/repos/simonw/datasette/issues/700
node_id: MDEyOklzc3VlQ29tbWVudDU5OTg0OTQ5Nw==
created_at: 2020-03-17T02:46:59Z
updated_at: 2020-03-17T02:46:59Z
author_association: OWNER
body:
  Right now the `Request` class is created by passing it an ASGI scope:

  https://github.com/simonw/datasette/blob/dc80e779a2e708b2685fc641df99e6aae9ad6f97/datasette/utils/asgi.py#L174

  I can extend this to `Request(scope, receive)` - then I can add a `await request.post_vars()` method.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/698#issuecomment-599856081
issue_url: https://api.github.com/repos/simonw/datasette/issues/698
node_id: MDEyOklzc3VlQ29tbWVudDU5OTg1NjA4MQ==
created_at: 2020-03-17T03:14:20Z
updated_at: 2020-03-17T03:14:20Z
author_association: OWNER
body:
  This is going to require a pretty full rewrite of the existing canned query logic, which currently shares a codepath with the code that lets you submit a SQL query:

  https://github.com/simonw/datasette/blob/298a899e792ebd0cd82a5f01b613c31f19082e51/datasette/views/base.py#L464-L563

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/34#issuecomment-601861908
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/34
node_id: MDEyOklzc3VlQ29tbWVudDYwMTg2MTkwOA==
created_at: 2020-03-20T18:56:44Z
updated_at: 2020-03-20T18:56:44Z
author_association: MEMBER
body:
  Could this be a bug in `sqlite-utils`? This table has a primary key, so why is it running a query on `rowid = ?`?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/pull/33#issuecomment-601871930
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/33
node_id: MDEyOklzc3VlQ29tbWVudDYwMTg3MTkzMA==
created_at: 2020-03-20T19:21:41Z
updated_at: 2020-03-20T19:21:41Z
author_association: MEMBER
body:
  Made obsolete by https://github.com/dogsheep/twitter-to-sqlite/commit/d3887d1535c3edd59d3ebf6aea4495bacd82e59b

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601875405
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35
node_id: MDEyOklzc3VlQ29tbWVudDYwMTg3NTQwNQ==
created_at: 2020-03-20T19:30:09Z
updated_at: 2020-03-20T19:32:33Z
author_association: MEMBER
body:
  I'm going to redesign the command a little. Right now it defaults to fetching the authenticated user's timeline and allows `--screen_name` to be used to fetch for one other user instead.

  I think the design of `users-lookup` is much better. I'm going to copy that:

  ```
  twitter-to-sqlite user-timeline users.db simonw cleopaws
  twitter-to-sqlite user-timeline users.db 12497 3166449535 --ids
  ```
  It will also accept `--since` and `--sql` and `--alter` and `--since_id`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601876051
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35
node_id: MDEyOklzc3VlQ29tbWVudDYwMTg3NjA1MQ==
created_at: 2020-03-20T19:31:43Z
updated_at: 2020-03-20T19:31:43Z
author_association: MEMBER
body:
  I'm going to keep `--screen_name` working for backwards compatibility. Using `hidden=True` on it should hide it from the help output.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/38#issuecomment-601939656
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/38
node_id: MDEyOklzc3VlQ29tbWVudDYwMTkzOTY1Ng==
created_at: 2020-03-20T22:37:04Z
updated_at: 2020-03-20T22:37:04Z
author_association: MEMBER
body:
  New formatting:
  ```
  @Alaska_DHSS      [####################################]  3205
  @BWhippleKS       [####################################]  1
  @CAPublicHealth   [####################################]  2
  @CDPHE            [####################################]  67
  @CHFSKy           [####################################]  3216
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/702#issuecomment-601963535
issue_url: https://api.github.com/repos/simonw/datasette/issues/702
node_id: MDEyOklzc3VlQ29tbWVudDYwMTk2MzUzNQ==
created_at: 2020-03-21T00:20:50Z
updated_at: 2020-03-21T00:20:50Z
author_association: OWNER
body:
  I can solve #677 (The first time you click sort by ID it should show you results in reverse order) at the same time.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-601979709
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDYwMTk3OTcwOQ==
created_at: 2020-03-21T02:20:08Z
updated_at: 2020-03-21T02:20:08Z
author_association: MEMBER
body:
  I'm pretty sure this means I need to upgrade the underlying `sqlite-utils` dependency. I saw the same bug in `twitter-to-sqlite`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-601980957
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18
node_id: MDEyOklzc3VlQ29tbWVudDYwMTk4MDk1Nw==
created_at: 2020-03-21T02:30:03Z
updated_at: 2020-03-21T02:30:03Z
author_association: MEMBER
body:
  https://github.community/t5/GitHub-API-Development-and/Request-for-commits-quot-author-quot-null-and-quot-committer/m-p/35842/highlight/true#M3372
  > Commits aren't always associated with a GitHub user. For example, perhaps a friend of mine and I were working on a project together. I have a GitHub account and my friend doesn't. If we both add commits to the repository using our own email addresses and names and then I push the repository to GitHub, my commits will be associated with my GitHub user account but my friends' commits will show up with `author` and `committer` as `null`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/702#issuecomment-602122492
issue_url: https://api.github.com/repos/simonw/datasette/issues/702
node_id: MDEyOklzc3VlQ29tbWVudDYwMjEyMjQ5Mg==
created_at: 2020-03-22T00:01:56Z
updated_at: 2020-03-22T00:01:56Z
author_association: OWNER
body:
  I'm going to mirror the URL syntax here, so the options in `metadata.json` will be:

  * `"sort": "created"` to default sort by created
  * `"sort_desc": "created"` to default sort by created descending

  It will throw an error if you try to define both.

  If I add the ability to sort by multiple columns in the future (#197) I can allow a list of columns here - though I'm not sure how I would represent "sort by created asc, title desc" in that case.

  If you want to disable user-specified sorting you can do so using the existing `"sortable_columns": []` mechanism.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-602136481
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDYwMjEzNjQ4MQ==
created_at: 2020-03-22T02:08:57Z
updated_at: 2020-03-22T02:08:57Z
author_association: NONE
body:
  I'd love to be using your library as a better cached gh layer for a new library I have built, replacing large parts of the very ugly https://github.com/jayvdb/pypidb/blob/master/pypidb/_github.py , and then probably being able to rebuild the setuppy chunk as a feature here at a later stage.

  I would also need tokenless and netrc support, but I would be happy to add those bits.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/702#issuecomment-602138528
issue_url: https://api.github.com/repos/simonw/datasette/issues/702
node_id: MDEyOklzc3VlQ29tbWVudDYwMjEzODUyOA==
created_at: 2020-03-22T02:29:23Z
updated_at: 2020-03-22T02:29:23Z
author_association: OWNER
body:
  Documentation: https://datasette.readthedocs.io/en/latest/metadata.html#setting-a-default-sort-order

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/702#issuecomment-602139031
issue_url: https://api.github.com/repos/simonw/datasette/issues/702
node_id: MDEyOklzc3VlQ29tbWVudDYwMjEzOTAzMQ==
created_at: 2020-03-22T02:36:09Z
updated_at: 2020-03-22T02:36:09Z
author_association: OWNER
body:
  Found a bug: if you click on a column with a default sort order applied to it you get: "Cannot use _sort and _sort_desc at the same time"

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/698#issuecomment-602140071
issue_url: https://api.github.com/repos/simonw/datasette/issues/698
node_id: MDEyOklzc3VlQ29tbWVudDYwMjE0MDA3MQ==
created_at: 2020-03-22T02:53:48Z
updated_at: 2020-03-22T02:53:48Z
author_association: OWNER
body:
  This feature should include the ability to set a custom redirect URL for after the query has been executed - that way it can be used to build things like "delete this row" which redirect back to the correct table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/650#issuecomment-602150641
issue_url: https://api.github.com/repos/simonw/datasette/issues/650
node_id: MDEyOklzc3VlQ29tbWVudDYwMjE1MDY0MQ==
created_at: 2020-03-22T05:31:13Z
updated_at: 2020-03-22T05:31:13Z
author_association: OWNER
body:
  Ansible has a good example of a glossary: https://docs.ansible.com/ansible/latest/reference_appendices/glossary.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/19#issuecomment-602304531
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19
node_id: MDEyOklzc3VlQ29tbWVudDYwMjMwNDUzMQ==
created_at: 2020-03-23T00:20:28Z
updated_at: 2020-03-23T00:20:28Z
author_association: MEMBER
body:
  Current FTS code:
  https://github.com/dogsheep/github-to-sqlite/blob/e35eec4343aa560c58c1634cc228d0d46c442304/github_to_sqlite/utils.py#L136-L143

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602309825
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15
node_id: MDEyOklzc3VlQ29tbWVudDYwMjMwOTgyNQ==
created_at: 2020-03-23T00:52:38Z
updated_at: 2020-03-23T00:52:38Z
author_association: MEMBER
body:
  Splitting assets out into a separate table totally makes sense to me. They can still be fetched as part of the `releases` command.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602311701
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15
node_id: MDEyOklzc3VlQ29tbWVudDYwMjMxMTcwMQ==
created_at: 2020-03-23T01:05:15Z
updated_at: 2020-03-23T01:06:14Z
author_association: MEMBER
body:
  None of my own releases use assets (they are all pushed to PyPI instead) but I spotted that your project here uses assets, so I'll test against that: https://github.com/instrumenta/conftest/releases/tag/v0.18.0

  ```
  github-to-sqlite releases releases.db instrumenta/conftest
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602312054
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15
node_id: MDEyOklzc3VlQ29tbWVudDYwMjMxMjA1NA==
created_at: 2020-03-23T01:07:10Z
updated_at: 2020-03-23T01:07:10Z
author_association: MEMBER
body:
  Each asset looks like this:
  ```json
      {
          "url": "https://api.github.com/repos/instrumenta/conftest/releases/assets/11811946",
          "id": 11811946,
          "node_id": "MDEyOlJlbGVhc2VBc3NldDExODExOTQ2",
          "name": "checksums.txt",
          "label": "",
          "uploader": {
              "login": "garethr",
              "id": 2029,
              "node_id": "MDQ6VXNlcjIwMjk=",
              "avatar_url": "https://avatars2.githubusercontent.com/u/2029?v=4",
              "gravatar_id": "",
              "url": "https://api.github.com/users/garethr",
              "html_url": "https://github.com/garethr",
              "followers_url": "https://api.github.com/users/garethr/followers",
              "following_url": "https://api.github.com/users/garethr/following{/other_user}",
              "gists_url": "https://api.github.com/users/garethr/gists{/gist_id}",
              "starred_url": "https://api.github.com/users/garethr/starred{/owner}{/repo}",
              "subscriptions_url": "https://api.github.com/users/garethr/subscriptions",
              "organizations_url": "https://api.github.com/users/garethr/orgs",
              "repos_url": "https://api.github.com/users/garethr/repos",
              "events_url": "https://api.github.com/users/garethr/events{/privacy}",
              "received_events_url": "https://api.github.com/users/garethr/received_events",
              "type": "User",
              "site_admin": false
          },
          "content_type": "text/plain; charset=utf-8",
          "state": "uploaded",
          "size": 600,
          "download_count": 2,
          "created_at": "2019-03-30T16:56:44Z",
          "updated_at": "2019-03-30T16:56:44Z",
          "browser_download_url": "https://github.com/instrumenta/conftest/releases/download/v0.1.0/checksums.txt"
      }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/19#issuecomment-602782437
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19
node_id: MDEyOklzc3VlQ29tbWVudDYwMjc4MjQzNw==
created_at: 2020-03-23T18:36:12Z
updated_at: 2020-03-23T18:51:23Z
author_association: MEMBER
body:
  * `commits`: `message`
  * `issue_comments`: `body`
  * `issues`: `title`, `body`
  * `labels`: `name`, `description`
  * `licenses`: `name`
  * `milestones`: `title`, `description`
  * `releases`: `name`, `body`
  * `repos`: `name`, `description`
  * `users`: `login`, `name` (maybe `company`, `bio`)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602811785
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18
node_id: MDEyOklzc3VlQ29tbWVudDYwMjgxMTc4NQ==
created_at: 2020-03-23T19:33:56Z
updated_at: 2020-03-23T19:34:06Z
author_association: MEMBER
body:
  ```json
  [
    {
      "sha": "a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0",
      "node_id": "MDY6Q29tbWl0OTMyMDk6YThkYzkxNDA4OWQzOTlkOWI1MjJlYmI1MWI2N2Y5YWMyZThhYTZiMA==",
      "commit": {
        "author": {
          "name": "Simon Willison",
          "email": "simon@...",
          "date": "2008-12-18T23:17:12Z"
        },
        "committer": {
          "name": "Simon Willison",
          "email": "simon@...",
          "date": "2008-12-18T23:17:12Z"
        },
        "message": "First commit",
        "tree": {
          "sha": "ac2dfb75e2592c59165c2880f3f7a16dafd452a1",
          "url": "https://api.github.com/repos/simonw/simonw.github.com/git/trees/ac2dfb75e2592c59165c2880f3f7a16dafd452a1"
        },
        "url": "https://api.github.com/repos/simonw/simonw.github.com/git/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0",
        "comment_count": 0,
        "verification": {
          "verified": false,
          "reason": "unsigned",
          "signature": null,
          "payload": null
        }
      },
      "url": "https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0",
      "html_url": "https://github.com/simonw/simonw.github.com/commit/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0",
      "comments_url": "https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0/comments",
      "author": null,
      "committer": null,
      "parents": []
    }
  ]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602813731
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18
node_id: MDEyOklzc3VlQ29tbWVudDYwMjgxMzczMQ==
created_at: 2020-03-23T19:38:03Z
updated_at: 2020-03-23T20:44:49Z
author_association: MEMBER
body:
  So it turns out `"author"` and `"committer"` on the commit are `null` if the email address in the nested `"commit"` doesn't match an existing GitHub user.

  Maybe I should be storing the nested data somewhere as well?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602815120
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18
node_id: MDEyOklzc3VlQ29tbWVudDYwMjgxNTEyMA==
created_at: 2020-03-23T19:40:55Z
updated_at: 2020-03-23T19:43:19Z
author_association: MEMBER
body:
  I could pull a pk-hashed version of the name/email into separate `raw_author` and `raw_committer` columns perhaps - against a `commit_authors` table. Could be interesting.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602862236
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13
node_id: MDEyOklzc3VlQ29tbWVudDYwMjg2MjIzNg==
created_at: 2020-03-23T21:20:26Z
updated_at: 2020-03-23T21:20:26Z
author_association: MEMBER
body:
  I'll run the `commits` and `issues` and `issue-comments` commands in addition to the `releases` command.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602895896
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13
node_id: MDEyOklzc3VlQ29tbWVudDYwMjg5NTg5Ng==
created_at: 2020-03-23T22:42:25Z
updated_at: 2020-03-23T22:42:25Z
author_association: MEMBER
body:
  Urgh this is such a mess! I should have done this on a branch / pull request to avoid polluting my main master history, but never mind.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/21#issuecomment-602896434
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21
node_id: MDEyOklzc3VlQ29tbWVudDYwMjg5NjQzNA==
created_at: 2020-03-23T22:43:37Z
updated_at: 2020-03-23T22:43:37Z
author_association: MEMBER
body:
  I'm going to do this now to help figure out the latest error in #13:
  ```
  Traceback (most recent call last):
    File "/opt/hostedtoolcache/Python/3.8.2/x64/bin/github-to-sqlite", line 11, in <module>
      load_entry_point('github-to-sqlite', 'console_scripts', 'github-to-sqlite')()
    File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py", line 237, in commits
      utils.save_commits(db, commits, repo_full["id"])
    File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 345, in save_commits
      for commit in commits:
    File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 207, in fetch_commits
      if stop_when(commit):
    File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py", line 224, in stop_when
      db["commits"].get(commit["sha"])
  TypeError: string indices must be integers
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/394#issuecomment-602904184
issue_url: https://api.github.com/repos/simonw/datasette/issues/394
node_id: MDEyOklzc3VlQ29tbWVudDYwMjkwNDE4NA==
created_at: 2020-03-23T23:03:42Z
updated_at: 2020-03-23T23:03:42Z
author_association: NONE
body:
  On mybinder.org we allow access to arbitrary processes listening on a port inside the container via a [reverse proxy](https://github.com/jupyterhub/jupyter-server-proxy).

  This means we need support for a proxy prefix as the proxy ends up running at a URL like `/something/random/proxy/datasette/...`

  An example that shows the problem is https://github.com/psychemedia/jupyterserverproxy-datasette-demo. Launch directly into a datasette instance on mybinder.org with https://mybinder.org/v2/gh/psychemedia/jupyterserverproxy-datasette-demo/master?urlpath=datasette then try to follow links inside the UI.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/394#issuecomment-602907207
issue_url: https://api.github.com/repos/simonw/datasette/issues/394
node_id: MDEyOklzc3VlQ29tbWVudDYwMjkwNzIwNw==
created_at: 2020-03-23T23:12:18Z
updated_at: 2020-03-23T23:12:18Z
author_association: CONTRIBUTOR
body:
  This would also be useful for running Datasette in Jupyter notebooks on [Binder](https://mybinder.org/). While you can use [Jupyter-server-proxy](https://github.com/jupyterhub/jupyter-server-proxy) to access Datasette on Binder, the links are broken.

  Why run Datasette on Binder? I'm developing a [range of Jupyter notebooks](https://glam-workbench.github.io/) that are aimed at getting humanities researchers to explore data from libraries, archives, and museums. Many of them are aimed at researchers with limited digital skills, so being able to run examples in Binder without them installing anything is fantastic.

  For example, there are a [series of notebooks](https://glam-workbench.github.io/trove-harvester/) that help researchers harvest digitised historical newspaper articles from Trove. The metadata from this harvest is saved as a CSV file that users can download. I've also provided some extra notebooks that use Pandas etc to demonstrate ways of analysing and visualising the harvested data.

  But it would be really nice if, after completing a harvest, the user could spin up Datasette for some initial exploration of their harvested data without ever leaving their browser.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/394#issuecomment-602911133
issue_url: https://api.github.com/repos/simonw/datasette/issues/394
node_id: MDEyOklzc3VlQ29tbWVudDYwMjkxMTEzMw==
created_at: 2020-03-23T23:22:10Z
updated_at: 2020-03-23T23:22:10Z
author_association: NONE
body:
  I just updated #652 to remove a merge conflict.  I think it's an easy way to add this functionality. I don't have time to do more though, sorry!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/394#issuecomment-602916580
issue_url: https://api.github.com/repos/simonw/datasette/issues/394
node_id: MDEyOklzc3VlQ29tbWVudDYwMjkxNjU4MA==
created_at: 2020-03-23T23:37:06Z
updated_at: 2020-03-23T23:37:06Z
author_association: NONE
body:
  @simonw You're welcome - I was just trying it out back in December as I thought it should work. Now there's a pandemic to work on though.... so no time at all for more at the moment. BTW, I have datasette running on several protein and full (virus) genome databases I build, and it's great - thank you!  Hi and best regards to you & Nat :-)

reactions:
  {"total_count": 1, "+1": 0, "-1": 0, "laugh": 1, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602917713
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13
node_id: MDEyOklzc3VlQ29tbWVudDYwMjkxNzcxMw==
created_at: 2020-03-23T23:40:29Z
updated_at: 2020-03-23T23:40:29Z
author_association: MEMBER
body:
  Most recently updated issues across all Dogsheep repos, with faceting: https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/github/issues?_facet=repo&_facet=user&_facet=state&_facet=author_association&_facet=type&_sort_desc=updated_at

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602918689
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13
node_id: MDEyOklzc3VlQ29tbWVudDYwMjkxODY4OQ==
created_at: 2020-03-23T23:43:39Z
updated_at: 2020-03-23T23:47:50Z
author_association: MEMBER
body:
  I pointed https://github-to-sqlite.dogsheep.net/ at it. May take a few minutes for the new certificate to provision though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602919058
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13
node_id: MDEyOklzc3VlQ29tbWVudDYwMjkxOTA1OA==
created_at: 2020-03-23T23:44:48Z
updated_at: 2020-03-23T23:44:48Z
author_association: MEMBER
body:
  Next step: use a `metadata.json` file to add some extras. And add the `datasette-render-markdown` plugin as soon as I ship https://github.com/simonw/datasette-render-markdown/issues/2 (GFM support).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/21#issuecomment-602920163
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21
node_id: MDEyOklzc3VlQ29tbWVudDYwMjkyMDE2Mw==
created_at: 2020-03-23T23:48:22Z
updated_at: 2020-03-23T23:48:22Z
author_association: MEMBER
body:
  I'm happy with this pattern:
  https://github.com/dogsheep/github-to-sqlite/blob/f78c4e9baaf0970ffab266ba780df7240aae9f32/github_to_sqlite/utils.py#L4-L18

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/394#issuecomment-602955699
issue_url: https://api.github.com/repos/simonw/datasette/issues/394
node_id: MDEyOklzc3VlQ29tbWVudDYwMjk1NTY5OQ==
created_at: 2020-03-24T01:34:06Z
updated_at: 2020-03-24T01:34:15Z
author_association: OWNER
body:
  I don't think I'll go with the `<base>` solution purely because it doesn't work with JSON APIs - and there are quite a few places where Datasette APIs return URLs (for things like toggling facets - e.g. `suggested_facets` on https://latest.datasette.io/fixtures/facetable.json?_labels=on&_size=0 )

  The good news is that if you look at the templates almost all of the URLs have been generated in Python code: https://github.com/simonw/datasette/blob/a498d0fe6590f9bdbc4faf9e0dd5faeb3b06002c/datasette/templates/table.html - so it shouldn't be too hard to fix in Python. Ideally I'd like to fix this with as few template changes as possible.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/394#issuecomment-603501719
issue_url: https://api.github.com/repos/simonw/datasette/issues/394
node_id: MDEyOklzc3VlQ29tbWVudDYwMzUwMTcxOQ==
created_at: 2020-03-24T20:59:28Z
updated_at: 2020-03-24T20:59:28Z
author_association: OWNER
body:
  Here's the line I'm stuck on now:

  https://github.com/simonw/datasette/blob/298a899e792ebd0cd82a5f01b613c31f19082e51/datasette/views/base.py#L417

  Tricky question: do I continue to rebuild URLs based on the incoming `request` (on the assumption that it has been modified to the new thing) or do I expect that I may still see un-prefixed incoming requests and need to change them?

  If the incoming URL paths contain the prefix, at what point do I drop that so I can run the regular URL matching code?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/394#issuecomment-603508356
issue_url: https://api.github.com/repos/simonw/datasette/issues/394
node_id: MDEyOklzc3VlQ29tbWVudDYwMzUwODM1Ng==
created_at: 2020-03-24T21:14:31Z
updated_at: 2020-03-24T21:14:31Z
author_association: OWNER
body:
  I'm going to assume that whatever is proxying to Datasette leaves the full incoming URL path intact, so I'm going to need to teach the URL routing code to strip off the prefix before processing the incoming request.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/394#issuecomment-603508785
issue_url: https://api.github.com/repos/simonw/datasette/issues/394
node_id: MDEyOklzc3VlQ29tbWVudDYwMzUwODc4NQ==
created_at: 2020-03-24T21:15:28Z
updated_at: 2020-03-24T21:15:28Z
author_association: OWNER
body:
  That means I should teach `AsgiRouter` how to handle an optional prefix:
  https://github.com/simonw/datasette/blob/298a899e792ebd0cd82a5f01b613c31f19082e51/datasette/utils/asgi.py#L81-L93

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/394#issuecomment-603509266
issue_url: https://api.github.com/repos/simonw/datasette/issues/394
node_id: MDEyOklzc3VlQ29tbWVudDYwMzUwOTI2Ng==
created_at: 2020-03-24T21:16:34Z
updated_at: 2020-03-24T21:16:34Z
author_association: OWNER
body:
  Actually I'll teach `DatasetteRouter` since that subclasses `AsgiRouter` but has access to a `datasette` instance (which it can read configuration values from):
  https://github.com/simonw/datasette/blob/298a899e792ebd0cd82a5f01b613c31f19082e51/datasette/app.py#L750-L753

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/394#issuecomment-603525339
issue_url: https://api.github.com/repos/simonw/datasette/issues/394
node_id: MDEyOklzc3VlQ29tbWVudDYwMzUyNTMzOQ==
created_at: 2020-03-24T21:55:46Z
updated_at: 2020-03-24T22:07:40Z
author_association: OWNER
body:
  OK, I have an implementation of this over in the `base-url` branch (see pull request #708) which is passing all of the unit tests.

  Anyone willing to give it a quick test and see if it works for your particular use-case? You can install it with:

      pip install https://github.com/simonw/datasette/archive/base-url.zip

  Then you can run Datasette like this:

      datasette fixtures.db --config base_url:/new-base/path/here/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/708#issuecomment-603525872
issue_url: https://api.github.com/repos/simonw/datasette/issues/708
node_id: MDEyOklzc3VlQ29tbWVudDYwMzUyNTg3Mg==
created_at: 2020-03-24T21:57:02Z
updated_at: 2020-03-24T21:57:02Z
author_association: OWNER
body:
  Spotted a bug: `<link rel="stylesheet" href="/-/static/app.css?ffa51a">` in the templates. I need to ensure the static assets are served from the new location as well (and any plugin assets too).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/708#issuecomment-603531775
issue_url: https://api.github.com/repos/simonw/datasette/issues/708
node_id: MDEyOklzc3VlQ29tbWVudDYwMzUzMTc3NQ==
created_at: 2020-03-24T22:11:57Z
updated_at: 2020-03-24T22:11:57Z
author_association: OWNER
body:
  This is now ready for people to try out. Quoting https://github.com/simonw/datasette/issues/394#issuecomment-603525339

  > Anyone willing to give it a quick test and see if it works for your particular use-case? You can install it with:
  > 
  > ```
  > pip install https://github.com/simonw/datasette/archive/base-url.zip
  > ```
  > 
  > Then you can run Datasette like this:
  > 
  > ```
  > datasette fixtures.db --config base_url:/new-base/path/here/
  > ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/227#issuecomment-603534725
issue_url: https://api.github.com/repos/simonw/datasette/issues/227
node_id: MDEyOklzc3VlQ29tbWVudDYwMzUzNDcyNQ==
created_at: 2020-03-24T22:19:54Z
updated_at: 2020-03-24T22:19:54Z
author_association: OWNER
body:
  I think the [extra_template_vars()](https://datasette.readthedocs.io/en/stable/plugins.html#extra-template-vars-template-database-table-view-name-request-datasette) hook covers this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/394#issuecomment-603539349
issue_url: https://api.github.com/repos/simonw/datasette/issues/394
node_id: MDEyOklzc3VlQ29tbWVudDYwMzUzOTM0OQ==
created_at: 2020-03-24T22:33:23Z
updated_at: 2020-03-24T22:33:23Z
author_association: NONE
body:
  Hi Simon - I'm just (trying, at least) to follow along in the above.  I can't try it out now, but I will if no one else gets to it.  Sorry I didn't write any tests in the original bit of code I pushed - I was just trying to see if it could work & whether you'd want to maybe head in that direction. Anyway, thank you, I will certainly use this.  Comment back here if no one tried it out & I'll make time.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/701#issuecomment-603547065
issue_url: https://api.github.com/repos/simonw/datasette/issues/701
node_id: MDEyOklzc3VlQ29tbWVudDYwMzU0NzA2NQ==
created_at: 2020-03-24T22:56:47Z
updated_at: 2020-03-24T22:56:47Z
author_association: OWNER
body:
  From the comments on that css-tricks article, this appears to work:
  ```css
  input[type=search] {
      -webkit-appearance: textfield;
  }
  input[type="search"]::-webkit-search-decoration,
  input[type="search"]::-webkit-search-cancel-button,
  input[type="search"]::-webkit-search-results-button,
  input[type="search"]::-webkit-search-results-decoration {
      display: none;
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/705#issuecomment-603560898
issue_url: https://api.github.com/repos/simonw/datasette/issues/705
node_id: MDEyOklzc3VlQ29tbWVudDYwMzU2MDg5OA==
created_at: 2020-03-24T23:41:32Z
updated_at: 2020-03-24T23:41:32Z
author_association: OWNER
body:
  I can switch over to deploying that using Cloud Run. Unfortunately if I move away from Zeit Now v1 (since it's no longer supported and might stop working) I don't think I'll be able to deploy a permanent URL for every commit hash that I push any more, which is a real shame.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/705#issuecomment-603561207
issue_url: https://api.github.com/repos/simonw/datasette/issues/705
node_id: MDEyOklzc3VlQ29tbWVudDYwMzU2MTIwNw==
created_at: 2020-03-24T23:42:42Z
updated_at: 2020-03-24T23:42:42Z
author_association: OWNER
body:
  I'd like to still provide https://v0-35.datasette.io/ URLs for every named release, but I'm not sure how to best automate the provisioning of those subdomains.

  For the moment I'm going to focus on getting `latest.datasette.io` working again, using Cloud Run.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/394#issuecomment-603570972
issue_url: https://api.github.com/repos/simonw/datasette/issues/394
node_id: MDEyOklzc3VlQ29tbWVudDYwMzU3MDk3Mg==
created_at: 2020-03-25T00:17:24Z
updated_at: 2020-03-25T00:17:24Z
author_association: OWNER
body:
  I got this working as a proxied instance inside Binder, building on @psychemedia's work: https://github.com/simonw/jupyterserverproxy-datasette-demo/issues/1

  Now that I've seen it working there I'm going to land the pull request.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/705#issuecomment-603582122
issue_url: https://api.github.com/repos/simonw/datasette/issues/705
node_id: MDEyOklzc3VlQ29tbWVudDYwMzU4MjEyMg==
created_at: 2020-03-25T01:00:30Z
updated_at: 2020-03-25T01:00:30Z
author_association: OWNER
body:
  I'm going to do this using GitHub Actions and the recipe from https://simonwillison.net/2020/Jan/21/github-actions-cloud-run/

  First step: deploying manually using this:
  ```
  datasette publish cloudrun fixtures.db \
      -m fixtures.json \
      --branch=7656fd64d8b6a32ebc34d89c1b8711cc5ea240f7 \
      --version-note=7656fd64d8b6a32ebc34d89c1b8711cc5ea240f7 \
      --extra-options="--config template_debug:1" \
      --service=datasette-latest
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/705#issuecomment-603599464
issue_url: https://api.github.com/repos/simonw/datasette/issues/705
node_id: MDEyOklzc3VlQ29tbWVudDYwMzU5OTQ2NA==
created_at: 2020-03-25T02:07:16Z
updated_at: 2020-03-25T02:07:16Z
author_association: OWNER
body:
  I need to verify `datasette.io` with Google. These instructions look good: https://cloud.google.com/endpoints/docs/openapi/verify-domain-name

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/705#issuecomment-603599677
issue_url: https://api.github.com/repos/simonw/datasette/issues/705
node_id: MDEyOklzc3VlQ29tbWVudDYwMzU5OTY3Nw==
created_at: 2020-03-25T02:08:11Z
updated_at: 2020-03-25T02:08:11Z
author_association: OWNER
body:
  <img width="1197" alt="Webmaster_Central" src="https://user-images.githubusercontent.com/9599/77494367-cb6b0580-6e02-11ea-8cb3-8de1353d7d3e.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/705#issuecomment-603599974
issue_url: https://api.github.com/repos/simonw/datasette/issues/705
node_id: MDEyOklzc3VlQ29tbWVudDYwMzU5OTk3NA==
created_at: 2020-03-25T02:09:34Z
updated_at: 2020-03-25T02:09:34Z
author_association: OWNER
body:
  I tried running this:

      $ now dns add datasette.io '@' TXT google-site-verification=wHw15S1fkNOMzk42PEVK6XLnSG9qEfyq6paO8_G4IVU

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/705#issuecomment-603600064
issue_url: https://api.github.com/repos/simonw/datasette/issues/705
node_id: MDEyOklzc3VlQ29tbWVudDYwMzYwMDA2NA==
created_at: 2020-03-25T02:09:56Z
updated_at: 2020-03-25T02:09:56Z
author_association: OWNER
body:
  That worked:
  <img width="737" alt="Webmaster_Central" src="https://user-images.githubusercontent.com/9599/77494472-0b31ed00-6e03-11ea-9444-849b21bd91c8.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/705#issuecomment-603600372
issue_url: https://api.github.com/repos/simonw/datasette/issues/705
node_id: MDEyOklzc3VlQ29tbWVudDYwMzYwMDM3Mg==
created_at: 2020-03-25T02:11:11Z
updated_at: 2020-03-25T02:11:11Z
author_association: OWNER
body:
  Now add the service mapping in Cloud Run: https://console.cloud.google.com/run/domains?project=datasette-222320

  <img width="601" alt="Domain_mappings_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform" src="https://user-images.githubusercontent.com/9599/77494538-387e9b00-6e03-11ea-8502-110f6d25526d.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/705#issuecomment-603600553
issue_url: https://api.github.com/repos/simonw/datasette/issues/705
node_id: MDEyOklzc3VlQ29tbWVudDYwMzYwMDU1Mw==
created_at: 2020-03-25T02:11:56Z
updated_at: 2020-03-25T02:12:09Z
author_association: OWNER
body:
  ... and set up the CNAME record:

      $ now dns add datasette.io latest CNAME ghs.googlehosted.com.

  Got an error:

      > Error!  A conflicting record exists "rec_e4c36ae94cf0a2b7b1781329".

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/705#issuecomment-603601073
issue_url: https://api.github.com/repos/simonw/datasette/issues/705
node_id: MDEyOklzc3VlQ29tbWVudDYwMzYwMTA3Mw==
created_at: 2020-03-25T02:13:44Z
updated_at: 2020-03-25T02:13:44Z
author_association: OWNER
body:
  Removed the conflicting record with:

       $ now dns rm rec_e4c36ae94cf0a2b7b1781329

  Then:

      $ now dns add datasette.io latest CNAME ghs.googlehosted.com.
      > Success! DNS record for domain datasette.io (rec_5b8855344da0be1861c4d55b) created under simonw [10s]

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/705#issuecomment-603601247
issue_url: https://api.github.com/repos/simonw/datasette/issues/705
node_id: MDEyOklzc3VlQ29tbWVudDYwMzYwMTI0Nw==
created_at: 2020-03-25T02:14:25Z
updated_at: 2020-03-25T02:14:25Z
author_association: OWNER
body:
  Could take a while for the certificate to be issued:

  <img width="471" alt="Domain_mappings_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform" src="https://user-images.githubusercontent.com/9599/77494719-ab881180-6e03-11ea-8953-07cd4ce5fdc3.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/705#issuecomment-603602781
issue_url: https://api.github.com/repos/simonw/datasette/issues/705
node_id: MDEyOklzc3VlQ29tbWVudDYwMzYwMjc4MQ==
created_at: 2020-03-25T02:20:09Z
updated_at: 2020-03-25T02:20:09Z
author_association: OWNER
body:
  Now following steps in https://simonwillison.net/2020/Jan/21/github-actions-cloud-run/ - I've added the `GCP_SA_EMAIL` and `GCP_SA_KEY` secrets to this repository: https://github.com/simonw/datasette/settings/secrets

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/705#issuecomment-603605246
issue_url: https://api.github.com/repos/simonw/datasette/issues/705
node_id: MDEyOklzc3VlQ29tbWVudDYwMzYwNTI0Ng==
created_at: 2020-03-25T02:30:24Z
updated_at: 2020-03-25T02:30:24Z
author_association: OWNER
body:
  It worked! After that push https://latest.datasette.io/-/versions updated to show `90015b26895fd2bd29676bc375f096102b0587ed` as the version note.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/394#issuecomment-604166918
issue_url: https://api.github.com/repos/simonw/datasette/issues/394
node_id: MDEyOklzc3VlQ29tbWVudDYwNDE2NjkxOA==
created_at: 2020-03-26T00:56:30Z
updated_at: 2020-03-26T00:56:30Z
author_association: CONTRIBUTOR
body:
  Thanks! I'm trying to launch Datasette from *within* a notebook using the jupyter-server-proxy and the new `base_url` parameter. While the assets load ok, and the breadcrumb navigation works, the facet links don't seem to use the `base_url`. Or have I missed something?

  My test repository is here: https://github.com/wragge/datasette-test

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/712#issuecomment-604193362
issue_url: https://api.github.com/repos/simonw/datasette/issues/712
node_id: MDEyOklzc3VlQ29tbWVudDYwNDE5MzM2Mg==
created_at: 2020-03-26T02:32:57Z
updated_at: 2020-03-26T02:32:57Z
author_association: OWNER
body:
  Weird... the unit test I just added passes. So suggested facets look like they're OK.

  And yet the demo in https://mybinder.org/v2/gh/wragge/datasette-test/master?urlpath=lab doesn't work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/712#issuecomment-604194598
issue_url: https://api.github.com/repos/simonw/datasette/issues/712
node_id: MDEyOklzc3VlQ29tbWVudDYwNDE5NDU5OA==
created_at: 2020-03-26T02:37:46Z
updated_at: 2020-03-26T02:37:46Z
author_association: OWNER
body:
  Here's the truncatted HTML for https://hub.gke.mybinder.org/user/wragge-datasette-test-czqms2oy/proxy/8001/results/articles
  ```html
  <!DOCTYPE html>
  <html>
  <head>
      <title>results: articles: 400 rows
      </title>
      <link rel="stylesheet" href="/user/wragge-datasette-test-czqms2oy/proxy/8001/-/static/app.css?4434ab">
      <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
  /head>
  <body class="table db-results table-articles">

  <nav class="hd">
      <p class="crumbs">
          <a href="/user/wragge-datasette-test-czqms2oy/proxy/8001/">home</a> /
          <a href="/user/wragge-datasette-test-czqms2oy/proxy/8001/results">results</a>
      </p>
      
  </nav>

  <div class="bd">

  <h1 style="padding-left: 10px; border-left: 10px solid #ff0000">articles</h1>

  <h3>400 rows</h3>


  <form class="filters" action="/user/wragge-datasette-test-czqms2oy/proxy/8001/results/articles" method="get">
      ...    
      <input type="submit" value="Apply">
  </form>

  <p><a class="not-underlined" title="select rowid, article_id, title, newspaper_id, newspaper_title, page, date, category, words, illustrated, corrections, url, page_url from articles order by rowid limit 101" href="/user/wragge-datasette-test-czqms2oy/proxy/8001/results?sql=select+rowid%2C+article_id%2C+title%2C+newspaper_id%2C+newspaper_title%2C+page%2C+date%2C+category%2C+words%2C+illustrated%2C+corrections%2C+url%2C+page_url+from+articles+order+by+rowid+limit+101">&#x270e; <span class="underlined">View and edit SQL</span></a></p>

  <p class="export-links">This data as <a href="/results/articles.json">json</a>, <a href="/results/articles.csv?_size=max">CSV</a> (<a href="#export">advanced</a>)</p>
      <p class="suggested-facets">
          Suggested facets: <a href="/results/articles?_facet=newspaper_id#facet-newspaper_id">newspaper_id</a>, <a href="/results/articles?_facet=newspaper_title#facet-newspaper_title">newspaper_title</a>, <a href="/results/articles?_facet=page#facet-page">page</a>, <a href="/results/articles?_facet=category#facet-category">category</a>, <a href="/results/articles?_facet=illustrated#facet-illustrated">illustrated</a>, <a href="/results/articles?_facet=corrections#facet-corrections">corrections</a>, <a href="/results/articles?_facet_date=date#facet-date">date</a> (date)
      </p>
      <table class="rows-and-columns">
          <thead>
              <tr>
                      <th class="col-Link" scope="col">
                              Link
                      </th>
                      <th class="col-rowid" scope="col">
                                  <a href="/results/articles?_sort_desc=rowid" rel="nofollow">rowid&nbsp;▼</a>
                      </th>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/712#issuecomment-604194872
issue_url: https://api.github.com/repos/simonw/datasette/issues/712
node_id: MDEyOklzc3VlQ29tbWVudDYwNDE5NDg3Mg==
created_at: 2020-03-26T02:38:48Z
updated_at: 2020-03-26T04:04:04Z
author_association: OWNER
body:
  I have a hunch about this: maybe the URLs that work are the ones that use `{{ base_url }}` directly in the template, while the ones that break are the ones that reconstruct the URL based on the incoming path?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/712#issuecomment-604195244
issue_url: https://api.github.com/repos/simonw/datasette/issues/712
node_id: MDEyOklzc3VlQ29tbWVudDYwNDE5NTI0NA==
created_at: 2020-03-26T02:40:23Z
updated_at: 2020-03-26T04:17:57Z
author_association: OWNER
body:
  Here's the output of the `/-/asgi-scope` plugin:
  ```
  {'client': ('10.12.3.15', 0),
   'headers': [(b'cookie',
                b''),
               (b'upgrade-insecure-requests', b'1'),
               (b'dnt', b'1'),
               (b'accept-encoding', b'gzip'),
               (b'accept-language', b'en-US,en;q=0.5'),
               (b'accept',
                b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'
                b'webp,*/*;q=0.8'),
               (b'user-agent',
                b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:74.0) Gecko'
                b'/20100101 Firefox/74.0'),
               (b'x-scheme', b'https'),
               (b'x-original-uri',
                b'/user/wragge-datasette-test-czqms2oy/proxy/8001/-/asgi-scope'),
               (b'x-forwarded-proto', b'https,http'),
               (b'x-forwarded-port', b'443,80'),
               (b'x-forwarded-host', b'hub.gke.mybinder.org'),
               (b'x-forwarded-for', b'148.64.98.14,10.12.3.15'),
               (b'x-real-ip', b'148.64.98.14'),
               (b'connection', b'close'),
               (b'host', b'hub.gke.mybinder.org'),
               (b'x-forwarded-context',
                b'/user/wragge-datasette-test-czqms2oy/proxy/8001'),
               (b'x-proxycontextpath',
                b'/user/wragge-datasette-test-czqms2oy/proxy/8001')],
   'http_version': '1.1',
   'method': 'GET',
   'path': '/-/asgi-scope',
   'query_string': b'',
   'raw_path': b'/-/asgi-scope',
   'root_path': '',
   'scheme': 'https,http',
   'server': ('127.0.0.1', 8001),
   'type': 'http'}
  ```
  From this output... it looks like the `path` being passed through to Datasette is still `/-/asgi-scope` here when I though it would be `/user/wragge-datasette-test-czqms2oy/proxy/8001/-/asgi-scope`

  The original URL path (the one I want) is passed in the `x-original-uri` HTTP header.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/712#issuecomment-604195577
issue_url: https://api.github.com/repos/simonw/datasette/issues/712
node_id: MDEyOklzc3VlQ29tbWVudDYwNDE5NTU3Nw==
created_at: 2020-03-26T02:41:40Z
updated_at: 2020-03-26T02:41:40Z
author_association: OWNER
body:
  Aha! It turns out my demo at https://github.com/simonw/jupyterserverproxy-datasette-demo suffers from the same bug.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/712#issuecomment-604196158
issue_url: https://api.github.com/repos/simonw/datasette/issues/712
node_id: MDEyOklzc3VlQ29tbWVudDYwNDE5NjE1OA==
created_at: 2020-03-26T02:43:47Z
updated_at: 2020-03-26T04:20:15Z
author_association: OWNER
body:
  I think the solution here may be to let the incoming `x-original-uri` header over-ride the `raw_path` used by Datasette. This may require a new Datasette setting - maybe this:

      --config path_from_header:x-original-uri

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/712#issuecomment-604218449
issue_url: https://api.github.com/repos/simonw/datasette/issues/712
node_id: MDEyOklzc3VlQ29tbWVudDYwNDIxODQ0OQ==
created_at: 2020-03-26T04:12:59Z
updated_at: 2020-03-26T04:15:18Z
author_association: OWNER
body:
  I'm running this latest release like so:

  Launch this binder: https://mybinder.org/v2/gh/wragge/datasette-test/master?urlpath=lab

  Open the notebook.

  In the notebook run this:

      !pip install datasette-debug-asgi https://github.com/simonw/datasette/archive/7b205af0b53c92d17b69f8c95d70a24a0c06ebce.zip

  And use this block to run Datasette:

  ```
  # Get current running servers
  servers = list_running_servers()

  # Get the current base url
  base_url = next(servers)['base_url']

  # Create a base url for Datasette suing the proxy path
  proxy_url = f'{base_url}proxy/8001/'

  # Display a link to Datasette
  display(HTML(f'<p><a href="{proxy_url}">View Datasette</a> (Click on the stop button to close the Datasette server)</p>'))

  # Launch Datasette
  !datasette -- results.db --port 8001 --config base_url:$proxy_url --config path_from_header:x-original-uri --config template_debug:1
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/712#issuecomment-604219753
issue_url: https://api.github.com/repos/simonw/datasette/issues/712
node_id: MDEyOklzc3VlQ29tbWVudDYwNDIxOTc1Mw==
created_at: 2020-03-26T04:18:13Z
updated_at: 2020-03-26T04:18:13Z
author_association: OWNER
body:
  Woohoo that worked!
  <img width="1386" alt="results__articles__400_rows" src="https://user-images.githubusercontent.com/9599/77609678-db0b4c80-6edd-11ea-947b-1f7b1f14f56e.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/712#issuecomment-604225034
issue_url: https://api.github.com/repos/simonw/datasette/issues/712
node_id: MDEyOklzc3VlQ29tbWVudDYwNDIyNTAzNA==
created_at: 2020-03-26T04:40:08Z
updated_at: 2020-03-26T04:40:08Z
author_association: CONTRIBUTOR
body:
  Great! Yes, can confirm that this works on Binder. However, when I try to run the same code locally, I get an Internal Server Error when I try to access Datasette.

  ```
  ERROR:    Exception in ASGI application
  Traceback (most recent call last):
    File "/Volumes/Workspace/mycode/datasette-test/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 385, in run_asgi
      result = await app(self.scope, self.receive, self.send)
    File "/Volumes/Workspace/mycode/datasette-test/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
      return await self.app(scope, receive, send)
    File "/Volumes/Workspace/mycode/datasette-test/lib/python3.7/site-packages/datasette_debug_asgi.py", line 24, in wrapped_app
      await app(scope, recieve, send)
    File "/Volumes/Workspace/mycode/datasette-test/lib/python3.7/site-packages/datasette/utils/asgi.py", line 174, in __call__
      await self.app(scope, receive, send)
    File "/Volumes/Workspace/mycode/datasette-test/lib/python3.7/site-packages/datasette/tracer.py", line 75, in __call__
      await self.app(scope, receive, send)
    File "/Volumes/Workspace/mycode/datasette-test/lib/python3.7/site-packages/datasette/app.py", line 746, in __call__
      raw_path = dict(scope["headers"])[path_from_header.encode("utf8")].split(b"?")[0]
  KeyError: b'x-original-uri'
  INFO:     127.0.0.1:49320 - "GET / HTTP/1.1" 500 Internal Server Error
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/712#issuecomment-604249402
issue_url: https://api.github.com/repos/simonw/datasette/issues/712
node_id: MDEyOklzc3VlQ29tbWVudDYwNDI0OTQwMg==
created_at: 2020-03-26T06:11:44Z
updated_at: 2020-03-26T06:11:44Z
author_association: CONTRIBUTOR
body:
  Following on from @betatim's suggestion on Twitter, I've changed the proxy url to include 'absolute'.

  ``` python
  proxy_url = f'{base_url}proxy/absolute/8001/'
  ```
  This works both on Binder and locally, without using the `path_from_header` option. I've updated the demo repository. Sorry @simonw if I've led you down the wrong path!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/712#issuecomment-604486651
issue_url: https://api.github.com/repos/simonw/datasette/issues/712
node_id: MDEyOklzc3VlQ29tbWVudDYwNDQ4NjY1MQ==
created_at: 2020-03-26T15:11:49Z
updated_at: 2020-03-26T15:11:49Z
author_association: OWNER
body:
  And for https://github.com/simonw/jupyterserverproxy-datasette-demo setting `"absolute_url": True` appears to fix the bug as well (it previously caused an infinite redirect loop but that seems not to happen any more with the latest `base_url` setting): https://github.com/simonw/jupyterserverproxy-datasette-demo/commit/880f9d50566357160b793557f145c99e74ad759a

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/672#issuecomment-604561639
issue_url: https://api.github.com/repos/simonw/datasette/issues/672
node_id: MDEyOklzc3VlQ29tbWVudDYwNDU2MTYzOQ==
created_at: 2020-03-26T17:22:07Z
updated_at: 2020-03-26T17:22:07Z
author_association: OWNER
body:
  Here's the new utility function I should be using to verify database files that I find: 

  https://github.com/simonw/datasette/blob/6aa516d82dea9885cb4db8d56ec2ccfd4cd9b840/datasette/utils/__init__.py#L773-L787

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/672#issuecomment-604569063
issue_url: https://api.github.com/repos/simonw/datasette/issues/672
node_id: MDEyOklzc3VlQ29tbWVudDYwNDU2OTA2Mw==
created_at: 2020-03-26T17:32:06Z
updated_at: 2020-03-26T17:32:06Z
author_association: OWNER
body:
  While running it against a nested directory with a TON of databases I kept seeing errors like this:
  ```
  Traceback (most recent call last):
    File "/Users/simonw/Dropbox/Development/datasette/datasette/utils/asgi.py", line 121, in route_path
      return await view(new_scope, receive, send)
    File "/Users/simonw/Dropbox/Development/datasette/datasette/utils/asgi.py", line 193, in view
      request, **scope["url_route"]["kwargs"]
    File "/Users/simonw/Dropbox/Development/datasette/datasette/views/index.py", line 58, in get
      tables[table]["num_relationships_for_sorting"] = count
  KeyError: 'primary-candidates-2018/rep_candidates'
  ```
  And
  ```
  Traceback (most recent call last):
    File "/Users/simonw/Dropbox/Development/datasette/datasette/utils/asgi.py", line 121, in route_path
      return await view(new_scope, receive, send)
    File "/Users/simonw/Dropbox/Development/datasette/datasette/utils/asgi.py", line 193, in view
      request, **scope["url_route"]["kwargs"]
    File "/Users/simonw/Dropbox/Development/datasette/datasette/views/index.py", line 58, in get
      tables[table]["num_relationships_for_sorting"] = count
  KeyError: 'space_used'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/672#issuecomment-604665229
issue_url: https://api.github.com/repos/simonw/datasette/issues/672
node_id: MDEyOklzc3VlQ29tbWVudDYwNDY2NTIyOQ==
created_at: 2020-03-26T20:22:48Z
updated_at: 2020-03-26T20:22:48Z
author_association: OWNER
body:
  I also eventually get this error:
  ```
  Traceback (most recent call last):
    File "/Users/simonw/Dropbox/Development/datasette/datasette/utils/asgi.py", line 121, in route_path
      return await view(new_scope, receive, send)
    File "/Users/simonw/Dropbox/Development/datasette/datasette/utils/asgi.py", line 336, in inner_static
      await asgi_send_file(send, full_path, chunk_size=chunk_size)
    File "/Users/simonw/Dropbox/Development/datasette/datasette/utils/asgi.py", line 303, in asgi_send_file
      async with aiofiles.open(str(filepath), mode="rb") as fp:
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/aiofiles/base.py", line 78, in __aenter__
    File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/aiofiles/threadpool/__init__.py", line 35, in _open
    File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/thread.py", line 57, in run
  OSError: [Errno 24] Too many open files: '/Users/simonw/Dropbox/Development/datasette/datasette/static/app.css'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/672#issuecomment-604667029
issue_url: https://api.github.com/repos/simonw/datasette/issues/672
node_id: MDEyOklzc3VlQ29tbWVudDYwNDY2NzAyOQ==
created_at: 2020-03-26T20:26:46Z
updated_at: 2020-03-26T20:26:46Z
author_association: OWNER
body:
  I think I can tell what the current file limit is like so:
  ```
  In [1]: import resource                                                                                                                                                                   

  In [2]: resource.getrlimit(resource.RLIMIT_NOFILE)                                                                                                                                        
  Out[2]: (256, 9223372036854775807)
  ```
  So maybe I should have Datasette refuse to open more database files than that number minus 5 (to give me some spare room for opening CSS files etc).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-605316146
issue_url: https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDYwNTMxNjE0Ng==
created_at: 2020-03-27T21:09:15Z
updated_at: 2020-03-27T21:09:22Z
author_association: MEMBER
body:
  For a progress bar I need to know how many total items there are.

  I found an undocumented API for this! `/v3/stats` which returns:
  ```json
  {
      "count_list": 7394,
      "count_read": 1016,
      "count_unread": 6378,
      "status": 1
  }
  ```
  I guessed this based on the documented v2 API: https://getpocket.com/api/v2_docs/#stats

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-605325897
issue_url: https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDYwNTMyNTg5Nw==
created_at: 2020-03-27T21:37:26Z
updated_at: 2020-03-27T21:38:37Z
author_association: MEMBER
body:
  I keep getting 503 errors even though I appear to be staying within the rate limit:
  ```
  {'Date': 'Fri, 27 Mar 2020 21:35:57 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding':
  'chunked', 'Connection': 'keep-alive', 'Server': 'Apache/2.4.25 (Debian)', 'Content-Location':
  'get.php', 'Vary': 'negotiate', 'TCN': 'choice', 'Set-Cookie': '...;  httponly',
  'X-Frame-Options': 'SAMEORIGIN', 'Status': '200 OK', 'X-Limit-Key-Limit': '10000',
  'X-Limit-Key-Remaining': '9960', 'X-Limit-Key-Reset': '282', 'X-Source': 'Pocket',
  'P3P': 'policyref="/w3c/p3p.xml", CP="ALL CURa ADMa DEVa OUR IND UNI COM NAV INT STA PRE"'}

    [##----------------------------------]    6%  06:49:27

  {'Date': 'Fri, 27 Mar 2020 21:36:06 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Content-Length':
  '23', 'Connection': 'keep-alive', 'Server': 'Apache/2.4.25 (Debian)', 'Content-Location': 'get.php',
  'Vary': 'negotiate', 'TCN': 'choice', 'Set-Cookie': '...', 'X-Frame-Options': 'SAMEORIGIN',
  'X-Error': 'Pocket is currently under heavy load. Please wait a moment and try again.',
  'X-Error-Code': '199', 'Status': '503 Service Unavailable', 'X-Source': 'Pocket',
  'P3P': 'policyref="/w3c/p3p.xml", CP="ALL CURa ADMa DEVa OUR IND UNI COM NAV INT STA PRE"'}
  ```
  I'm going to try doing a few automatic retries any time I see a 503 error.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-605327655
issue_url: https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDYwNTMyNzY1NQ==
created_at: 2020-03-27T21:42:49Z
updated_at: 2020-03-27T21:42:49Z
author_association: MEMBER
body:
  Or maybe it was because of the current Google Cloud outage? https://news.ycombinator.com/item?id=22706677

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/pocket-to-sqlite/issues/2#issuecomment-605337941
issue_url: https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/2
node_id: MDEyOklzc3VlQ29tbWVudDYwNTMzNzk0MQ==
created_at: 2020-03-27T22:16:32Z
updated_at: 2020-03-27T22:16:32Z
author_association: MEMBER
body:
  Need to test this. I have 7,394 items in my database right now. I'm going to save a new thing.

  Then I ran this:
  ```
  pocket-to-sqlite fetch pocket-simon.db
  ```
  And it worked!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606304837
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39
node_id: MDEyOklzc3VlQ29tbWVudDYwNjMwNDgzNw==
created_at: 2020-03-30T23:27:50Z
updated_at: 2020-03-30T23:29:31Z
author_association: MEMBER
body:
  One option would be something like this:
  ```sql
  select max(id) from tweets
  where user = ?
  and not exists (select id from tweets where retweeted_status = id)
  and not exists (select id from tweets where quoted_status = id)
  and not exists (select id from tweets where in_reply_to_status_id = id)
  ```
  Might be a good idea to index those columns (after confirming that doing so would indeed speed up the query).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606305701
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39
node_id: MDEyOklzc3VlQ29tbWVudDYwNjMwNTcwMQ==
created_at: 2020-03-30T23:30:27Z
updated_at: 2020-03-30T23:30:27Z
author_association: MEMBER
body:
  A better alternative would be to maintain a separate table with the last seen since value for when we ran `user-timeline` for any specific user.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-606307019
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40
node_id: MDEyOklzc3VlQ29tbWVudDYwNjMwNzAxOQ==
created_at: 2020-03-30T23:34:27Z
updated_at: 2020-03-30T23:34:27Z
author_association: MEMBER
body:
  The count properties available for a user are:

  * followers_count
  * friends_count
  * listed_count
  * favourites_count
  * statuses_count

  May as well track history for all of them? Should be pretty cheap to store.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-606307376
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40
node_id: MDEyOklzc3VlQ29tbWVudDYwNjMwNzM3Ng==
created_at: 2020-03-30T23:35:40Z
updated_at: 2020-03-30T23:39:15Z
author_association: MEMBER
body:
  I think five separate tables:

  * followers_count_history
  * friends_count_history
  * listed_count_history
  * favourites_count_history
  * statuses_count_history

  Each with the following structure:

  * datetime (ISO UTC)
  * user (ID, foreign key to users)
  * count (integer)

  I'm tempted to have a compound primary key here - user, datetime

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/96#issuecomment-606394619
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/96
node_id: MDEyOklzc3VlQ29tbWVudDYwNjM5NDYxOQ==
created_at: 2020-03-31T04:38:17Z
updated_at: 2020-03-31T04:40:23Z
author_association: OWNER
body:
  I wonder if there are any other Pandas conversions we should be doing? https://pandas.pydata.org/pandas-docs/stable/getting_started/basics.html#dtypes 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606824992
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39
node_id: MDEyOklzc3VlQ29tbWVudDYwNjgyNDk5Mg==
created_at: 2020-03-31T19:24:23Z
updated_at: 2020-03-31T19:24:23Z
author_association: MEMBER
body:
  The `--since` option is actually used by four commands:

  * `user-timeline`
  * `home-timeline`
  * `mentions-timeline`
  * `search`

  All of them use the same `fetch_timeline()` utility function under the hood. I should move the logic that looks up the last `since_id` into that shared function.

  Question: should I have a table for each of those four methods or a single table that is used by them all? I'm leaning towards four separate tables.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606843224
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39
node_id: MDEyOklzc3VlQ29tbWVudDYwNjg0MzIyNA==
created_at: 2020-03-31T19:59:11Z
updated_at: 2020-03-31T20:06:32Z
author_association: MEMBER
body:
  Or... have a single `since_ids` table to track since values, and have its primary key be a string that looks something like this:

  `user:123145`
  `home:23441`
  `mentions:23425`
  `search:99ff9cefff5cbfd804f7cd43e2b27ced8addbe8d`

  That last example would use the hash generated here:

  https://github.com/dogsheep/twitter-to-sqlite/blob/810cb2af5a175837204389fd7f4b5721f8b325ab/twitter_to_sqlite/cli.py#L792-L808

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606850008
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39
node_id: MDEyOklzc3VlQ29tbWVudDYwNjg1MDAwOA==
created_at: 2020-03-31T20:13:59Z
updated_at: 2020-04-01T00:23:00Z
author_association: MEMBER
body:
  Table design for `since_ids` table:

  type | key | since_id
  --- | --- | ---
  1 | 124324 | 2347239847293
  2 | 99ff9cefff5cbfd804f7cd43e2b27ced8addbe8d | 2125947927344

  Primary compound key on `(category, key)`

  `type` is also a foreign key to a `since_id_types` table with `id` and `name` columns (probably created using https://sqlite-utils.readthedocs.io/en/stable/python-api.html#working-with-lookup-tables )

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606998669
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39
node_id: MDEyOklzc3VlQ29tbWVudDYwNjk5ODY2OQ==
created_at: 2020-04-01T02:57:36Z
updated_at: 2020-04-01T02:57:36Z
author_association: MEMBER
body:
  The tricky thing here is thinking about the interaction between the recorded since_id and a desire to run the initial import.

  The first time you run `twitter-to-sqlite user-timeline db.db username` we want to fetch as many tweets from that user as possible - probably around 3,200 before the API limitations cut us off.

  We need to record the maximum ID from those as the `since_id` - which we will see on the very first page we paginate through. That way next time we run the command with `--since` we will only fetch new tweets.

  But what happens if our initial import is cancelled after only a few tweets? We risk never pulling in the rest of the tweets.

  Not sure if I need to solve this at all or if I should instead trust users to run the command a second time without `--since` if they think they didn't retrieve anything the first time.

  I had considered letting `--stop_after=` over-ride `--since` but that doesn't actually make sense - if you send a since_id to the Twitter API you'll never get back more tweets than exist after that ID, so the `--stop_after` would not make a meaningful difference.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607011421
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40
node_id: MDEyOklzc3VlQ29tbWVudDYwNzAxMTQyMQ==
created_at: 2020-04-01T03:47:37Z
updated_at: 2020-04-01T03:55:08Z
author_association: MEMBER
body:
  Actually a single table with a `type` integer ID referencing a `count_history_types` table would better match the way I implemented the `since_ids` table:

  https://github.com/dogsheep/twitter-to-sqlite/blob/4b6c8d8c1cc6fefdb566ec8506157133f47c569a/twitter_to_sqlite/utils.py#L331-L341

  In which case the compound primary key would be `type`, `user`, `datetime`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607011972
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40
node_id: MDEyOklzc3VlQ29tbWVudDYwNzAxMTk3Mg==
created_at: 2020-04-01T03:49:02Z
updated_at: 2020-04-01T03:50:01Z
author_association: MEMBER
body:
  I want the datetime value to look like `2020-04-01T03:34:58+00:00` (the format returned by the Twitter API which I am storing in other tables at the moment).
  ```
  >>> datetime.utcnow().isoformat().split('.')[0] + '+00:00'
  '2020-04-01T03:49:52+00:00'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607019151
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40
node_id: MDEyOklzc3VlQ29tbWVudDYwNzAxOTE1MQ==
created_at: 2020-04-01T04:11:10Z
updated_at: 2020-04-01T04:11:10Z
author_association: MEMBER
body:
  In testing this collects a LOT of data. I'm going to skip tracking favourites_count and statuses_count and just track followers, friends and listed instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/713#issuecomment-608020485
issue_url: https://api.github.com/repos/simonw/datasette/issues/713
node_id: MDEyOklzc3VlQ29tbWVudDYwODAyMDQ4NQ==
created_at: 2020-04-02T18:11:53Z
updated_at: 2020-04-02T18:11:53Z
author_association: OWNER
body:
  I can add https://pypi.org/project/PyYAML/ as a dependency for this.

  One open question: how best to tell the difference between a JSON and a YAML file? I'd rather not do it based on a file extension.

  Since I'm going to read the whole file into memory anyway (rather than try to stream it) and I only load it once at startup, maybe I try to parse with one and, if there's a parsing error, try the other one before giving up?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/713#issuecomment-608021760
issue_url: https://api.github.com/repos/simonw/datasette/issues/713
node_id: MDEyOklzc3VlQ29tbWVudDYwODAyMTc2MA==
created_at: 2020-04-02T18:14:20Z
updated_at: 2020-04-02T18:14:58Z
author_association: OWNER
body:
  I'm going to depend on ~=5.3 https://github.com/yaml/pyyaml/blob/master/CHANGES - though 
  I could probably depend on a much wider set of versions (or maybe even no pinned minimum version at all).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/713#issuecomment-608023877
issue_url: https://api.github.com/repos/simonw/datasette/issues/713
node_id: MDEyOklzc3VlQ29tbWVudDYwODAyMzg3Nw==
created_at: 2020-04-02T18:18:29Z
updated_at: 2020-04-02T18:19:43Z
author_association: OWNER
body:
  Don't forget to fix the code in publish that loads and modifies metadata, e.g.:

  https://github.com/simonw/datasette/blob/2aaad72789c427875426673c1a43e67c86fc970e/datasette/utils/__init__.py#L361-L367

  And:

  https://github.com/simonw/datasette/blob/2aaad72789c427875426673c1a43e67c86fc970e/datasette/publish/heroku.py#L156-L168

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/714#issuecomment-608033652
issue_url: https://api.github.com/repos/simonw/datasette/issues/714
node_id: MDEyOklzc3VlQ29tbWVudDYwODAzMzY1Mg==
created_at: 2020-04-02T18:37:08Z
updated_at: 2020-04-02T18:37:08Z
author_association: OWNER
body:
  Writing unit tests for this is tricky as the YAML loading happens in the CLI code, but most of my existing unit tests skip that step and run directly against a Datasette instance.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/713#issuecomment-608062703
issue_url: https://api.github.com/repos/simonw/datasette/issues/713
node_id: MDEyOklzc3VlQ29tbWVudDYwODA2MjcwMw==
created_at: 2020-04-02T19:35:53Z
updated_at: 2020-04-02T19:35:53Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/6717c719dd36dc2adc0f9da38a8c8e08129e96b4/datasette/utils/__init__.py#L795-L803

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/698#issuecomment-608125928
issue_url: https://api.github.com/repos/simonw/datasette/issues/698
node_id: MDEyOklzc3VlQ29tbWVudDYwODEyNTkyOA==
created_at: 2020-04-02T22:32:41Z
updated_at: 2020-04-02T22:33:10Z
author_association: OWNER
body:
  I really want the option to use a `<textarea>` for a specific value.

  Idea: metadata syntax like this:

  ```json
  {
      "databases": {
         "my-database": {
             "queries": {
                 "add_twitter_handle": {
                     "sql": "insert into twitter_handles (username) values (:username)",
                     "write": true,
                     "params": {
                         "username": {
                              "widget": "textarea"
                         }
                     }
                 }
             }
         }
      }
  }
  ```
  I can ship with some default widgets and provide a plugin hook for registering extra widgets.

  This opens up some really exciting possibilities for things like map widgets that let you draw polygons.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/236#issuecomment-608716819
issue_url: https://api.github.com/repos/simonw/datasette/issues/236
node_id: MDEyOklzc3VlQ29tbWVudDYwODcxNjgxOQ==
created_at: 2020-04-03T22:19:00Z
updated_at: 2020-04-03T22:19:00Z
author_association: CONTRIBUTOR
body:
  Hi Simon,

  I'm thinking of attempting this. Can you clarify some questions I have?

  1) I assume the goal is to have a CORS-friendly HTTPS endpoint that hosts the datasette service + user's db.

  2) If that's the goal, I think Lambda alone is insufficient. Lambda provides the compute fabric, but not the HTTP routing. You'd also need to add Application Load Balancer or API Gateway to provide an HTTP endpoint that routes to the lambda function.

  Do you have a preference between ALB or API GW? ALB has better economics at scale, but has a minimum monthly cost. API GW has worse per-request economics, but scales to zero when no requests are happening.

  3) Does Datasette have any native components, or is it all pure python? If it has native bits, they'll likely need to be recompiled to work on Amazon Linux 2.

  4) There are a few disparate services that need to be wired together to expose a Python service securely to the web. If I was doing this outside of the datasette publish system, I'd use an AWS CloudFormation template. Even within datasette, I think it still makes sense to use a CloudFormation template and just have the publish plugin invoke it (via the standard `aws` cli) with user-specified parameters. Does that sound reasonable to you?

  Thanks for your help!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/710#issuecomment-608973849
issue_url: https://api.github.com/repos/simonw/datasette/issues/710
node_id: MDEyOklzc3VlQ29tbWVudDYwODk3Mzg0OQ==
created_at: 2020-04-04T04:52:22Z
updated_at: 2020-04-04T04:52:44Z
author_association: OWNER
body:
  Need to remove this entirely - it's going away in August (and new deployments will stop working in May).

  From an email they just sent out:

  ![3FF51C75-E8B8-4B84-B2A7-A439CEBF595C](https://user-images.githubusercontent.com/9599/78418903-6202a800-75f5-11ea-807d-65e0646669f5.jpeg)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/716#issuecomment-609106252
issue_url: https://api.github.com/repos/simonw/datasette/issues/716
node_id: MDEyOklzc3VlQ29tbWVudDYwOTEwNjI1Mg==
created_at: 2020-04-04T23:58:12Z
updated_at: 2020-04-04T23:58:12Z
author_association: OWNER
body:
  I think I'll use `git bisect run` for this one, mainly to practice using it. https://git-scm.com/docs/git-bisect#_bisect_run

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/716#issuecomment-609107679
issue_url: https://api.github.com/repos/simonw/datasette/issues/716
node_id: MDEyOklzc3VlQ29tbWVudDYwOTEwNzY3OQ==
created_at: 2020-04-05T00:10:37Z
updated_at: 2020-04-05T00:11:06Z
author_association: OWNER
body:
  Once I fix this bug I should update https://github.com/simonw/museums to deploy using the latest Datasette release as opposed to being anchored to 286ed28.

  https://github.com/simonw/museums/blob/1bbed542617757e9e276a5098193d6288b7f421d/.github/workflows/push.yml#L61

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/716#issuecomment-609109354
issue_url: https://api.github.com/repos/simonw/datasette/issues/716
node_id: MDEyOklzc3VlQ29tbWVudDYwOTEwOTM1NA==
created_at: 2020-04-05T00:30:30Z
updated_at: 2020-04-05T00:31:55Z
author_association: OWNER
body:
  In order to use `git bisect run` I need to write a standalone script that can tell if the bug is present or not.

  I'm going to use something I can execute with `pytest`, inspired loosely by this code:

  https://github.com/simonw/datasette/blob/7656fd64d8b6a32ebc34d89c1b8711cc5ea240f7/tests/test_plugins.py#L205-L227

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/717#issuecomment-609111516
issue_url: https://api.github.com/repos/simonw/datasette/issues/717
node_id: MDEyOklzc3VlQ29tbWVudDYwOTExMTUxNg==
created_at: 2020-04-05T00:57:29Z
updated_at: 2020-04-05T00:57:29Z
author_association: OWNER
body:
  If I can get this working I think I'll release it as a plugin, similar to [datasette-publish-fly](https://github.com/simonw/datasette-publish-fly).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/717#issuecomment-609306846
issue_url: https://api.github.com/repos/simonw/datasette/issues/717
node_id: MDEyOklzc3VlQ29tbWVudDYwOTMwNjg0Ng==
created_at: 2020-04-05T04:18:48Z
updated_at: 2020-04-05T04:18:48Z
author_association: OWNER
body:
  I need to route all paths to the same function. This should help: https://twitter.com/aboodman/status/1246605658067066882

  ![image](https://user-images.githubusercontent.com/9599/78466750-d4d55700-76b9-11ea-8fac-cfa674b9785b.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/716#issuecomment-609453886
issue_url: https://api.github.com/repos/simonw/datasette/issues/716
node_id: MDEyOklzc3VlQ29tbWVudDYwOTQ1Mzg4Ng==
created_at: 2020-04-05T17:38:38Z
updated_at: 2020-04-05T17:42:14Z
author_association: OWNER
body:
  OK, here's the test harness. Three files:

  `check_view_name.py`
  ```python
  import asyncio
  import pathlib
  from datasette.app import Datasette
  import httpx

  root = pathlib.Path(__file__).parent


  async def run_check():
      ds = Datasette(
          [], template_dir=str(root / "templates"), plugins_dir=str(root / "plugins")
      )
      async with httpx.AsyncClient(app=ds.app()) as client:
          response = await client.get("http://localhost/")
          assert 200 == response.status_code
          assert b"view_name:index" == response.content, response.content


  if __name__ == "__main__":
      loop = asyncio.get_event_loop()
      loop.run_until_complete(run_check())
  ```

  `templates/index.html`
  ```
  view_name:{{ view_name }}
  ```
  `plugins/extra_vars.py`
  ```python
  from datasette import hookimpl


  @hookimpl
  def extra_template_vars(view_name):
      return {"view_name": view_name}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/716#issuecomment-609454072
issue_url: https://api.github.com/repos/simonw/datasette/issues/716
node_id: MDEyOklzc3VlQ29tbWVudDYwOTQ1NDA3Mg==
created_at: 2020-04-05T17:39:47Z
updated_at: 2020-04-05T17:39:47Z
author_association: OWNER
body:
  `python check_view_name.py` against 286ed28 exits cleanly.

  `python check_view_name.py` against current master (e0e7a0fa) fails:

  ```
  $ python check_view_name.py 
  Traceback (most recent call last):
    File "check_view_name.py", line 16, in <module>
      loop.run_until_complete(run_check())
    File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
      return future.result()
    File "check_view_name.py", line 11, in run_check
      assert b"view_name:index" == response.content, response.content
  AssertionError: b'view_name:None'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/716#issuecomment-609455243
issue_url: https://api.github.com/repos/simonw/datasette/issues/716
node_id: MDEyOklzc3VlQ29tbWVudDYwOTQ1NTI0Mw==
created_at: 2020-04-05T17:47:33Z
updated_at: 2020-04-05T17:47:33Z
author_association: OWNER
body:
  You start `git bisect` by giving it a known bad commit and a known good one:
  ```
  git bisect start master 286ed28 
  ```
  Then you tell it to start running your script:
  ```
  git bisect run python ../datasette-issue-716/check_view_name.py
  ```
  Here's what I got:
  ```
  (datasette) ~/Dropbox/Development/datasette $ git bisect start master 286ed28
  Bisecting: 30 revisions left to test after this (roughly 5 steps)
  [dc80e779a2e708b2685fc641df99e6aae9ad6f97] Handle scope path if it is a string
  (datasette) ~/Dropbox/Development/datasette $ git bisect run python ../datasette-issue-716/check_view_name.py
  running python ../datasette-issue-716/check_view_name.py
  Traceback (most recent call last):
  ...
  Bisecting: 15 revisions left to test after this (roughly 4 steps)
  [7c6a9c35299f251f9abfb03fd8e85143e4361709] Better tests for prepare_connection() plugin hook, refs #678
  running python ../datasette-issue-716/check_view_name.py
  Traceback (most recent call last):
  ...
  Bisecting: 7 revisions left to test after this (roughly 3 steps)
  [0091dfe3e5a3db94af8881038d3f1b8312bb857d] More reliable tie-break ordering for facet results
  running python ../datasette-issue-716/check_view_name.py
  Traceback (most recent call last):
  ...
  Bisecting: 3 revisions left to test after this (roughly 2 steps)
  [ce12244037b60ba0202c814871218c1dab38d729] Release notes for 0.35
  running python ../datasette-issue-716/check_view_name.py
  Traceback (most recent call last):
  ...
  Bisecting: 0 revisions left to test after this (roughly 1 step)
  [4d7dae9eb75e5430c3ee3c369bb5cd9ba0a148bc] Added a bunch more plugins to the Ecosystem page
  running python ../datasette-issue-716/check_view_name.py
  Traceback (most recent call last):
  ...
  70b915fb4bc214f9d064179f87671f8a378aa127 is the first bad commit
  commit 70b915fb4bc214f9d064179f87671f8a378aa127
  Author: Simon Willison <swillison@gmail.com>
  Date:   Tue Feb 4 12:26:17 2020 -0800

      Datasette.render_template() method, closes #577
      
      Pull request #664.

  :040000 040000 def9e31252e056845609de36c66d4320dd0c47f8 da19b7f8c26d50a4c05e5a7f05220b968429725c M	datasette
  bisect run success
  ```
  So 70b915fb4bc214f9d064179f87671f8a378aa127 introduced the bug!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/716#issuecomment-609456030
issue_url: https://api.github.com/repos/simonw/datasette/issues/716
node_id: MDEyOklzc3VlQ29tbWVudDYwOTQ1NjAzMA==
created_at: 2020-04-05T17:52:39Z
updated_at: 2020-04-05T20:02:13Z
author_association: OWNER
body:
  Found it. Prior to that change I passed `view_name` to the callbacks like this:

  https://github.com/simonw/datasette/blob/286ed286b68793532c2a38436a08343b45cfbc91/datasette/views/base.py#L114-L132

  I switched over to doing this:

  https://github.com/simonw/datasette/blob/07e208cc6d9e901b87552c1be2854c220b3f9b6d/datasette/views/base.py#L95-L97

  But forgot to pass in the optional `view_name=` argument to that `render_template()` method:

  https://github.com/simonw/datasette/blob/2aaad72789c427875426673c1a43e67c86fc970e/datasette/app.py#L554-L556

  Next step: write a failing test, then fix it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/716#issuecomment-609461331
issue_url: https://api.github.com/repos/simonw/datasette/issues/716
node_id: MDEyOklzc3VlQ29tbWVudDYwOTQ2MTMzMQ==
created_at: 2020-04-05T18:31:32Z
updated_at: 2020-04-05T20:04:08Z
author_association: OWNER
body:
  The test ended up being a bit fiddly - here it is:

  https://github.com/simonw/datasette/blob/09253817dea3c131553494f9b2eb9c03f94ae761/tests/test_plugins.py#L266-L317

  I used `tmp_path_factory` here because the `tmpdir` fixture I usually use isn't compatible with `scope="session"`, and I wanted to only create those temporary plugins and templates directories once rather than create them for each run of the parametrized test function.

  In writing this I realized that the `name` on the `QueryView` class wasn't being used, because that class is currently just used for its `.data()` method:

  https://github.com/simonw/datasette/blob/09253817dea3c131553494f9b2eb9c03f94ae761/datasette/views/database.py#L27-L31

  https://github.com/simonw/datasette/blob/07e208cc6d9e901b87552c1be2854c220b3f9b6d/datasette/views/table.py#L224-L227

  So I removed the `name = "query"` line since it was misleading.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/689#issuecomment-609467523
issue_url: https://api.github.com/repos/simonw/datasette/issues/689
node_id: MDEyOklzc3VlQ29tbWVudDYwOTQ2NzUyMw==
created_at: 2020-04-05T19:16:13Z
updated_at: 2020-04-05T19:16:13Z
author_association: OWNER
body:
  I'm going to debug this using `git bisect run` - the same technique I used in #716 - https://github.com/simonw/datasette/issues/716#issuecomment-609453886

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/689#issuecomment-609467876
issue_url: https://api.github.com/repos/simonw/datasette/issues/689
node_id: MDEyOklzc3VlQ29tbWVudDYwOTQ2Nzg3Ng==
created_at: 2020-04-05T19:18:36Z
updated_at: 2020-04-05T19:18:36Z
author_association: OWNER
body:
  Just need the one checking script to run with bisect this time:

  `check_templates_considered.py`

  ```python
  import asyncio
  import pathlib
  from datasette.app import Datasette
  import httpx


  async def run_check():
      ds = Datasette([])
      async with httpx.AsyncClient(app=ds.app()) as client:
          response = await client.get("http://localhost/")
          assert 200 == response.status_code
          assert "Templates considered" in response.text


  if __name__ == "__main__":
      loop = asyncio.get_event_loop()
      loop.run_until_complete(run_check())
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/689#issuecomment-609468485
issue_url: https://api.github.com/repos/simonw/datasette/issues/689
node_id: MDEyOklzc3VlQ29tbWVudDYwOTQ2ODQ4NQ==
created_at: 2020-04-05T19:22:31Z
updated_at: 2020-04-05T19:22:31Z
author_association: OWNER
body:
  ```
  $ git bisect start master 0.34
  Bisecting: 32 revisions left to test after this (roughly 5 steps)
  [dc80e779a2e708b2685fc641df99e6aae9ad6f97] Handle scope path if it is a string
  $ git bisect run python check_templates_considered.py
  running python check_templates_considered.py
  Traceback (most recent call last):
  ...
  AssertionError
  Bisecting: 15 revisions left to test after this (roughly 4 steps)
  [7c6a9c35299f251f9abfb03fd8e85143e4361709] Better tests for prepare_connection() plugin hook, refs #678
  running python check_templates_considered.py
  Traceback (most recent call last):
  ...
  AssertionError
  Bisecting: 7 revisions left to test after this (roughly 3 steps)
  [0091dfe3e5a3db94af8881038d3f1b8312bb857d] More reliable tie-break ordering for facet results
  running python check_templates_considered.py
  Traceback (most recent call last):
  ...
  AssertionError
  Bisecting: 3 revisions left to test after this (roughly 2 steps)
  [ce12244037b60ba0202c814871218c1dab38d729] Release notes for 0.35
  running python check_templates_considered.py
  Traceback (most recent call last):
  ...
  AssertionError
  Bisecting: 1 revision left to test after this (roughly 1 step)
  [70b915fb4bc214f9d064179f87671f8a378aa127] Datasette.render_template() method, closes #577
  running python check_templates_considered.py
  Traceback (most recent call last):
  ...
  AssertionError
  Bisecting: 0 revisions left to test after this (roughly 0 steps)
  [286ed286b68793532c2a38436a08343b45cfbc91] geojson-to-sqlite
  running python check_templates_considered.py
  70b915fb4bc214f9d064179f87671f8a378aa127 is the first bad commit
  commit 70b915fb4bc214f9d064179f87671f8a378aa127
  Author: Simon Willison <swillison@gmail.com>
  Date:   Tue Feb 4 12:26:17 2020 -0800

      Datasette.render_template() method, closes #577
      
      Pull request #664.

  :040000 040000 def9e31252e056845609de36c66d4320dd0c47f8 da19b7f8c26d50a4c05e5a7f05220b968429725c M	datasette
  bisect run success
  ```
  It was 70b915fb4bc214f9d064179f87671f8a378aa127 - the same bad commit that caused #716!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/689#issuecomment-609469318
issue_url: https://api.github.com/repos/simonw/datasette/issues/689
node_id: MDEyOklzc3VlQ29tbWVudDYwOTQ2OTMxOA==
created_at: 2020-04-05T19:28:14Z
updated_at: 2020-04-05T19:28:14Z
author_association: OWNER
body:
  Here's why: the `BaseView.render()` method is running `jinja_env.select_template()` now here:

  https://github.com/simonw/datasette/blob/e89b0ef2f9ae89eb3bde83b694f21452ea4858da/datasette/views/base.py#L75-L88

  Which means this logic is always called with a template, not a list of strings:

  https://github.com/simonw/datasette/blob/e89b0ef2f9ae89eb3bde83b694f21452ea4858da/datasette/app.py#L555-L571

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/689#issuecomment-609469440
issue_url: https://api.github.com/repos/simonw/datasette/issues/689
node_id: MDEyOklzc3VlQ29tbWVudDYwOTQ2OTQ0MA==
created_at: 2020-04-05T19:28:59Z
updated_at: 2020-04-05T19:28:59Z
author_association: OWNER
body:
  So I think the fix is to move the `"select_templates": select_templates` context setting bit to here instead:

  https://github.com/simonw/datasette/blob/e89b0ef2f9ae89eb3bde83b694f21452ea4858da/datasette/views/base.py#L75-L83

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/717#issuecomment-609883545
issue_url: https://api.github.com/repos/simonw/datasette/issues/717
node_id: MDEyOklzc3VlQ29tbWVudDYwOTg4MzU0NQ==
created_at: 2020-04-06T16:00:41Z
updated_at: 2020-04-06T16:03:21Z
author_association: OWNER
body:
  https://now-2-datasette-e4l4whc9f.now.sh/ - first proof of concept!

  It's just four files right now, all in the same directory and deployed by running `now`.

  `now.json`
  ```json
  {
      "version": 2,
      "builds": [
          {
              "src": "index.py",
              "use": "@now/python"
          }
      ],
      "routes": [
          {
              "src": "(.*)",
              "dest": "index.py"
          }
      ]
  }
  ```
  `requirements.txt`:
  ```
  datasette
  datasette-debug-asgi
  ```
  `index.py`:
  ```python
  from datasette.app import Datasette

  app = Datasette(["fixtures.db"]).app()
  ```
  Plus a copy of the standard `fixtures.db` file.

  It almost entirely works! Just one bug so far:

  https://now-2-datasette-e4l4whc9f.now.sh/fixtures/table%2Fwith%2Fslashes.csv returns a 404
  https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv does not.

  Plugins work too - here's the output from `datasette-debug-asgi`: https://now-2-datasette-e4l4whc9f.now.sh/-/asgi-scope

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/717#issuecomment-609887635
issue_url: https://api.github.com/repos/simonw/datasette/issues/717
node_id: MDEyOklzc3VlQ29tbWVudDYwOTg4NzYzNQ==
created_at: 2020-04-06T16:07:51Z
updated_at: 2020-04-06T16:08:47Z
author_association: OWNER
body:
  Built myself a quick ASGI scope debugging tool: https://now-2-asgi-scope.now.sh/

  Same `now.json` as above, but `index.py` is this:
  ```python
  from pprint import pformat


  async def app(scope, receive, send):
      await send({
          'type': 'http.response.start',
          'status': 200,
          'headers': [
              [b'content-type', b'text/plain'],
          ],
      })
      await send({
          'type': 'http.response.body',
          'body': pformat(scope).encode('utf8'),
      })
  ```
  https://now-2-asgi-scope.now.sh/fixtures/table%2Fwith%2Fslashes.csv shows what's going on - those `%2F` have been decoded to `/` before they get to the ASGI app - probably by the Now routing layer:
  ```
  ...
   'http_version': '1.1',
   'method': 'GET',
   'path': '/fixtures/table/with/slashes.csv',
   'query_string': b'',
   'raw_path': b'/fixtures/table/with/slashes.csv',
  ...
  ```
  That `raw_path` there needs to be `b'/fixtures/table%2Fwith%2Fslashes.csv'` in order for Datasette to fully work here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/717#issuecomment-609889528
issue_url: https://api.github.com/repos/simonw/datasette/issues/717
node_id: MDEyOklzc3VlQ29tbWVudDYwOTg4OTUyOA==
created_at: 2020-04-06T16:11:07Z
updated_at: 2020-04-06T16:11:07Z
author_association: OWNER
body:
  https://now-2-datasette-e4l4whc9f.now.sh/-/versions
  ```json
  {
      "python": {
          "version": "3.6.10",
          "full": "3.6.10 (default, Mar 10 2020, 22:54:43) \n[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)]"
      },
      "datasette": {
          "version": "0.39"
      },
      "asgi": "3.0",
      "uvicorn": "0.11.3",
      "sqlite": {
          "version": "3.7.17",
          "fts_versions": [
              "FTS4",
              "FTS3"
          ],
          "extensions": {},
          "compile_options": [
              "DISABLE_DIRSYNC",
              "ENABLE_COLUMN_METADATA",
              "ENABLE_FTS3",
              "ENABLE_RTREE",
              "ENABLE_UNLOCK_NOTIFY",
              "SECURE_DELETE",
              "TEMP_STORE=1",
              "THREADSAFE=1"
          ]
      }
  }
  ```
  SQLite 3.7.17 is from May 2013: http://www.sqlite.org/releaselog/3_7_17.html - and there's no FTS5 (but we do get FTS4).

  So the Now v2 version will be a bit limited (no support for table names containing `/` and an ancient SQLite version without FTS5) - but it's definitely enough for a `datasette-publish-now` plugin.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/717#issuecomment-609905147
issue_url: https://api.github.com/repos/simonw/datasette/issues/717
node_id: MDEyOklzc3VlQ29tbWVudDYwOTkwNTE0Nw==
created_at: 2020-04-06T16:39:50Z
updated_at: 2020-04-06T16:39:50Z
author_association: OWNER
body:
  I tried deploying the whole of www.niche-museums.com with mixed results.

  https://datasette-jm25oxd7s.now.sh/browse/museums works
  https://datasette-jm25oxd7s.now.sh/ shows "disk I/O error"

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/717#issuecomment-609905777
issue_url: https://api.github.com/repos/simonw/datasette/issues/717
node_id: MDEyOklzc3VlQ29tbWVudDYwOTkwNTc3Nw==
created_at: 2020-04-06T16:41:04Z
updated_at: 2020-04-06T16:43:05Z
author_association: OWNER
body:
  Full traceback from Zeit Now logs:
  ```
  ERROR: conn=<sqlite3.Connection object at 0x7efc6fb5e2d0>, sql = "select name from sqlite_master where type='table'", params = None: disk I/O error
  Traceback (most recent call last):
  File "/var/task/datasette/utils/asgi.py", line 121, in route_path
  return await view(new_scope, receive, send)
  File "/var/task/datasette/utils/asgi.py", line 193, in view
  request, **scope["url_route"]["kwargs"]
  File "/var/task/datasette/views/base.py", line 61, in head
  response = await self.get(*args, **kwargs)
  File "/var/task/datasette/views/index.py", line 27, in get
  table_names = await db.table_names()
  File "/var/task/datasette/database.py", line 221, in table_names
  "select name from sqlite_master where type='table'"
  File "/var/task/datasette/database.py", line 167, in execute
  sql_operation_in_thread
  File "/var/task/datasette/database.py", line 114, in execute_against_connection_in_thread
  self.ds.executor, in_thread
  File "/var/lang/lib/python3.6/concurrent/futures/thread.py", line 56, in run
  result = self.fn(*self.args, **self.kwargs)
  File "/var/task/datasette/database.py", line 111, in in_thread
  return fn(conn)
  File "/var/task/datasette/database.py", line 137, in sql_operation_in_thread
  cursor.execute(sql, params or {})sqlite3.OperationalError: disk I/O error
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/717#issuecomment-610066089
issue_url: https://api.github.com/repos/simonw/datasette/issues/717
node_id: MDEyOklzc3VlQ29tbWVudDYxMDA2NjA4OQ==
created_at: 2020-04-06T22:17:32Z
updated_at: 2020-04-06T22:17:53Z
author_association: OWNER
body:
  I think that error might be because Niche Museums uses two database files that are 0 bytes (about.db and map.db) - I just tried purely publishing the empty about.db file and got the same error: https://datasette-ks55wp251.now.sh/ - https://datasette-ks55wp251.now.sh/-/databases

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/717#issuecomment-610067097
issue_url: https://api.github.com/repos/simonw/datasette/issues/717
node_id: MDEyOklzc3VlQ29tbWVudDYxMDA2NzA5Nw==
created_at: 2020-04-06T22:20:11Z
updated_at: 2020-04-06T22:20:11Z
author_association: OWNER
body:
  Yes I think that's it - I created a table in `about.db`, then dropped that table and ran `vacuum` and the result was an empty database file that wasn't zero bytes. I deployed it and it worked: https://datasette-koo7abu7j.now.sh/ 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/717#issuecomment-610076073
issue_url: https://api.github.com/repos/simonw/datasette/issues/717
node_id: MDEyOklzc3VlQ29tbWVudDYxMDA3NjA3Mw==
created_at: 2020-04-06T22:47:21Z
updated_at: 2020-04-06T22:47:21Z
author_association: OWNER
body:
  I'm confident it's possible to create a plugin that deploys to Now v2 now. I'll do the rest of the work in a separate repo: https://github.com/simonw/datasette-publish-now

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/514#issuecomment-610186532
issue_url: https://api.github.com/repos/simonw/datasette/issues/514
node_id: MDEyOklzc3VlQ29tbWVudDYxMDE4NjUzMg==
created_at: 2020-04-07T05:43:45Z
updated_at: 2020-04-07T05:43:45Z
author_association: OWNER
body:
  I've run Datasette in production directly on Ubuntu with systemd in a few places now. I think I'm ready to turn this into real documentation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-611879821
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYxMTg3OTgyMQ==
created_at: 2020-04-10T05:06:56Z
updated_at: 2020-04-10T05:06:56Z
author_association: OWNER
body:
  Another problem this would solve: if you want multiple authentication mechanisms - GitHub auth for users, `Authorization: bearer xxx` auth for API keys - the order in which they run might end up mattering.

  I dealt with this a bit in https://github.com/simonw/datasette-auth-github/issues/59

  But having an authentication plugin hook - where playing get to decide if a user should be authenticated based on the incoming ASGI scopes - would be neater.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-611880250
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYxMTg4MDI1MA==
created_at: 2020-04-10T05:08:54Z
updated_at: 2020-04-10T05:08:54Z
author_association: OWNER
body:
  So maybe this is all handled by plugin hooks?

  `auth_from_scope(datasette, scope)` would be the main one - for deciding if a user should be authenticated based on data from the scope.

  How would a permissions hook work though?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/724#issuecomment-612154749
issue_url: https://api.github.com/repos/simonw/datasette/issues/724
node_id: MDEyOklzc3VlQ29tbWVudDYxMjE1NDc0OQ==
created_at: 2020-04-10T18:19:31Z
updated_at: 2020-04-10T18:19:31Z
author_association: OWNER
body:
  Actually that code isn't the problem, since `extra_metadata` has not yet been read from `metadata.json` at that point.

  The bug is here: https://github.com/simonw/datasette/blob/d55fe8cdfc2ce7bc6960bf2507766c1fcd1d31a7/datasette/utils/__init__.py#L362-L368

  I need to merge dictionaries in a smarter way.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612173156
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/98
node_id: MDEyOklzc3VlQ29tbWVudDYxMjE3MzE1Ng==
created_at: 2020-04-10T19:03:32Z
updated_at: 2020-04-10T23:08:28Z
author_association: OWNER
body:
  Investigate this traceback:
  ```
  Traceback (most recent call last):
    File "fetch_projects.py", line 60, in <module>
      fetch_projects(db, token)
    File "fetch_projects.py", line 41, in fetch_projects
      db["projects"].upsert(project, pk="id")
    File "/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py", line 1139, in upsert
      conversions=conversions,
    File "/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py", line 1168, in upsert_all
      upsert=True,
    File "/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py", line 1107, in insert_all
      row = list(self.rows_where("rowid = ?", [self.last_rowid]))[0]
  IndexError: list index out of range
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/236#issuecomment-612216820
issue_url: https://api.github.com/repos/simonw/datasette/issues/236
node_id: MDEyOklzc3VlQ29tbWVudDYxMjIxNjgyMA==
created_at: 2020-04-10T21:03:38Z
updated_at: 2020-04-10T21:03:38Z
author_association: CONTRIBUTOR
body:
  I made a repo at https://github.com/code402/datasette-lambda to demonstrate the idea, and scratch my personal itch for this.

  The demo relies on some central authority having already published a public, reusable Lambda layer with Datasette & its dependencies. I think that differs from the other publish plugins which seem to mainly publish Dockerfiles that the host will interpret to install deps from a requirements.txt file.

  I chose that approach because `uvloop` appears to be a dependency with native code that needs to be compiled for the target runtime environment. In this case, that's Amazon Linux 2. I'm not 100% clear on whether that's still required, because:

  - maybe `uvloop` is only needed for `uvicorn`, which the demo doesn't actually use since HTTP routing is handled by API Gateway
  - it seems like `uvloop` may be an optional, drop-in optimization for `asyncio` in any case (but I may be misreading this; I'm very much a Python noob)

  If it's the case that `uvloop` is truly optional, then I think the publish plugin could do the packaging on the user's machine, regardless of what flavour of operating system they're on. That'd be a bit slower for the user, but would provide the most long-term flexibility in terms of supporting plugins.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612707293
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/98
node_id: MDEyOklzc3VlQ29tbWVudDYxMjcwNzI5Mw==
created_at: 2020-04-13T01:21:22Z
updated_at: 2020-04-13T01:21:22Z
author_association: OWNER
body:
  I have a hunch that the root of the problem here is that accessing `result.lastrowid` during my version of an `.upsert()` doesn't actually make sense:

  https://github.com/simonw/sqlite-utils/blob/6161ebf4de44411b3f33feeacaf4501e803d1116/sqlite_utils/db.py#L1102-L1113

  In the bug I'm seeing (which I still haven't reduced to a reproducible test) the debugger shows me this at that point:

  ```
  (Pdb) query
  'UPDATE [files] SET [createdAt] = ?, [ext] = ?, [updatedAt] = ?, [uri] = ?, [uriType] = ? WHERE [project] = ? AND [name] = ?'
  (Pdb) params
  ['2020-03-04T04:04:40.152000+00:00', 'csv', '2020-03-04T04:04:40.152000+00:00', 'https://storage.googleapis.com/bln_prod/...', 'download', 'UHJvamVjdDo4MTgyMjU2Ny01ZjI0LTQxM2ItYWZmNi05NTlmNGY3MjExMjI=', 'loans_to_documentation.csv']
  (Pdb) result.lastrowid
  100
  ```
  But here's the weird thing... there's no row in the table with a rowid of 100!
  ```
  (Pdb) [r['rowid'] for r in self.db.execute_returning_dicts('select rowid, * from files')]
  [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,
  28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50,
  51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73,
  74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99]
  ```
  So what the heck is going on?

  The last SQL statement I executed here was an `UPDATE`. The `lastrowid` docs say: https://kite.com/python/docs/sqlite3.Cursor.lastrowid

  > This read-only attribute provides the rowid of the last modified row. It is only set if you issued a INSERT statement using the execute() method. For operations other than INSERT or when executemany() is called, lastrowid is set to None.

  So where did that `100` come from? It should be `None`!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612707828
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/98
node_id: MDEyOklzc3VlQ29tbWVudDYxMjcwNzgyOA==
created_at: 2020-04-13T01:24:05Z
updated_at: 2020-04-13T01:24:16Z
author_association: OWNER
body:
  Why do I even care about `lastrowid` here?

  I'm trying to ensure that after you insert or upsert a row you can use `table.last_pk` to start doing things like building additional foreign key relationships.

  So maybe it doesn't make sense to make `.last_pk` available _at all_ for cases where you called `.upsert_all()` or `.insert_all()` - it should just be populated for `.upsert()` and `.insert()`.

  The documentation doesn't say it should work for `.upsert_all()` - it's only documented for the single actions.

  https://github.com/simonw/sqlite-utils/blob/6161ebf4de44411b3f33feeacaf4501e803d1116/sqlite_utils/db.py#L1113-L1124

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612708274
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/98
node_id: MDEyOklzc3VlQ29tbWVudDYxMjcwODI3NA==
created_at: 2020-04-13T01:25:59Z
updated_at: 2020-04-13T01:26:11Z
author_association: OWNER
body:
  In mucking around with `sqlite3` it looks like `result.lastrowid` is indeed populated for `UPDATE` - in this case with the last inserted rowid in the table. This differs from the documented behaviour I linked to above.

  ```
  In [1]: import sqlite3                                                          

  In [2]: c = sqlite3.connect(":memory:")                                         

  In [3]: c                                                                       
  Out[3]: <sqlite3.Connection at 0x103c4d490>

  In [4]: c.execute('create table foo (bar integer);')                            
  Out[4]: <sqlite3.Cursor at 0x103f760a0>

  In [5]: c.execute('insert into foo (bar) values (1)')                           
  Out[5]: <sqlite3.Cursor at 0x103f76650>

  In [6]: c.execute('select * from foo').fetchall()                               
  Out[6]: [(1,)]

  In [7]: c.execute('insert into foo (bar) values (1)')                           
  Out[7]: <sqlite3.Cursor at 0x103f766c0>

  In [8]: c.execute('select * from foo').fetchall()                               
  Out[8]: [(1,), (1,)]

  In [9]: c.execute('insert into foo (bar) values (1)').lastrowid                 
  Out[9]: 3

  In [10]: c.execute('select * from foo').fetchall()                              
  Out[10]: [(1,), (1,), (1,)]

  In [11]: c.execute('select rowid, bar from foo').fetchall()                     
  Out[11]: [(1, 1), (2, 1), (3, 1)]

  In [12]: c.execute('insert into foo (bar) values (1)').lastrowid                
  Out[12]: 4

  In [13]: c.execute('select rowid, bar from foo').fetchall()                     
  Out[13]: [(1, 1), (2, 1), (3, 1), (4, 1)]

  In [14]: r = c.execute('update foo set bar =2 where rowid = 1')                 

  In [15]: r.lastrowid                                                            
  Out[15]: 4

  In [16]: c.execute('select rowid, bar from foo').fetchall()                     
  Out[16]: [(1, 2), (2, 1), (3, 1), (4, 1)]

  In [17]: r = c.execute('select rowid, bar from foo')                            

  In [18]: r.fetchall()                                                           
  Out[18]: [(1, 2), (2, 1), (3, 1), (4, 1)]

  In [19]: r.lastrowid                                                            
  Out[19]: 4
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/99#issuecomment-612727400
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/99
node_id: MDEyOklzc3VlQ29tbWVudDYxMjcyNzQwMA==
created_at: 2020-04-13T03:03:09Z
updated_at: 2020-04-13T03:03:09Z
author_association: OWNER
body:
  I think I'm going to leave this as intended behaviour. Or maybe passing multiple dictionaries to `.upsert_all()` with different numbers of keys should raise an error?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/99#issuecomment-612727814
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/99
node_id: MDEyOklzc3VlQ29tbWVudDYxMjcyNzgxNA==
created_at: 2020-04-13T03:05:04Z
updated_at: 2020-04-13T03:05:04Z
author_association: OWNER
body:
  Bit trick from an implementation point of view this, since we want to be able to handle input that is a generator - so we can't scan through the input to validate that every dictionary has the same exact keys without consuming the entire iterator.

  The alternative would be to raise an error the first time we spot a dictionary with keys that differ... but that's weird because we commit changes in batches, so we may end up only applying half of the changes before exiting with the error.

  On that basis, I'm going to leave this as-is and mark this as wontfix.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612728047
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/98
node_id: MDEyOklzc3VlQ29tbWVudDYxMjcyODA0Nw==
created_at: 2020-04-13T03:06:10Z
updated_at: 2020-04-13T03:06:10Z
author_association: OWNER
body:
  Implementation plan: `.insert_all()` and `.upsert_all()` should only set `.last_rowid` and `last_pk` if they were called with a single item.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/97#issuecomment-612732129
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/97
node_id: MDEyOklzc3VlQ29tbWVudDYxMjczMjEyOQ==
created_at: 2020-04-13T03:25:29Z
updated_at: 2020-04-13T03:25:29Z
author_association: OWNER
body:
  Interesting thought. I've run into this myself a lot - many of my scripts intend to create the database from scratch, so I end up running `!rm /tmp/blah.db` in Jupyter and occasionally getting errors if the file doesn't exist.

  I think adding `recreate=True` could make sense. It could throw an error if you attempt to use it after passing in something other than a path to a file on disk.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/97#issuecomment-612732453
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/97
node_id: MDEyOklzc3VlQ29tbWVudDYxMjczMjQ1Mw==
created_at: 2020-04-13T03:26:46Z
updated_at: 2020-04-13T03:26:46Z
author_association: OWNER
body:
  I wonder if it should delete an recreate the file or if it would be safer to drop every table instead?

  Dropping tables gets messy: then you need to drop triggers and views, and you need to run `vacuum` to clean up the space.

  My worry with deleting and recreating the file is that it could trigger errors in other processes that are currently attached to that database file. But... if you know that's going to be likely, maybe you shouldn't use the `recreate=True` feature?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/97#issuecomment-612738311
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/97
node_id: MDEyOklzc3VlQ29tbWVudDYxMjczODMxMQ==
created_at: 2020-04-13T03:55:11Z
updated_at: 2020-04-13T03:55:11Z
author_association: OWNER
body:
  Shipped in 2.5 - documentation is here: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#connecting-to-or-creating-a-database

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-613611455
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDYxMzYxMTQ1NQ==
created_at: 2020-04-14T18:37:21Z
updated_at: 2020-04-14T18:37:21Z
author_association: MEMBER
body:
  This should have been fixed by #20 and #23 

  @jayvdb I'm definitely interested in this tool working as a library - it's purely designed as a CLI tool at the moment, but cleaning it up to work better as a dependency is totally in-scope for the project. https://sqlite-utils.readthedocs.io/ is an example of a tool I've built that works for both.

  Feel free to open a new issue here with some notes on what you would need for this to work as a library for your project!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-613641947
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDYxMzY0MTk0Nw==
created_at: 2020-04-14T19:38:24Z
updated_at: 2020-04-14T19:38:34Z
author_association: MEMBER
body:
  Since events include payloads with full object representations in them (for issues, repos and more) running this command every few minutes may be all it takes to keep a constant copy of everything updated in a very rate-limit friendly manner (thanks to the ETags).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/97#issuecomment-614073859
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/97
node_id: MDEyOklzc3VlQ29tbWVudDYxNDA3Mzg1OQ==
created_at: 2020-04-15T14:29:30Z
updated_at: 2020-04-15T14:29:30Z
author_association: NONE
body:
  Woah! Thanks a lot. Next time I will add a more obvious/explicit "if you like this idea let me know I'd love to work on it to get my feet wet here" :D

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/727#issuecomment-614276522
issue_url: https://api.github.com/repos/simonw/datasette/issues/727
node_id: MDEyOklzc3VlQ29tbWVudDYxNDI3NjUyMg==
created_at: 2020-04-15T20:58:14Z
updated_at: 2020-04-15T20:58:45Z
author_association: OWNER
body:
  Compare with this table page https://latest.datasette.io/fixtures/complex_foreign_keys which has:
  ```html
  <body class="table db-fixtures table-complex_foreign_keys">
  ```
  So in the above example the extra class should be `query-neighborhood_search`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/727#issuecomment-614276988
issue_url: https://api.github.com/repos/simonw/datasette/issues/727
node_id: MDEyOklzc3VlQ29tbWVudDYxNDI3Njk4OA==
created_at: 2020-04-15T20:59:17Z
updated_at: 2020-04-15T20:59:45Z
author_association: OWNER
body:
  Relevant code:

  https://github.com/simonw/datasette/blob/d349d57cdf3d577afb62bdf784af342a4d5be660/datasette/templates/query.html#L19

  Compare with:

  https://github.com/simonw/datasette/blob/d349d57cdf3d577afb62bdf784af342a4d5be660/datasette/templates/table.html#L17

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/727#issuecomment-614278078
issue_url: https://api.github.com/repos/simonw/datasette/issues/727
node_id: MDEyOklzc3VlQ29tbWVudDYxNDI3ODA3OA==
created_at: 2020-04-15T21:01:44Z
updated_at: 2020-04-15T21:01:44Z
author_association: OWNER
body:
  And here's the test: https://github.com/simonw/datasette/blob/d349d57cdf3d577afb62bdf784af342a4d5be660/tests/test_html.py#L433-L456

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/727#issuecomment-614282980
issue_url: https://api.github.com/repos/simonw/datasette/issues/727
node_id: MDEyOklzc3VlQ29tbWVudDYxNDI4Mjk4MA==
created_at: 2020-04-15T21:13:00Z
updated_at: 2020-04-15T21:13:00Z
author_association: OWNER
body:
  https://latest.datasette.io/fixtures/neighborhood_search now has the following:
  ```html
  <body class="query db-fixtures query-neighborhood_search">
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/727#issuecomment-614283842
issue_url: https://api.github.com/repos/simonw/datasette/issues/727
node_id: MDEyOklzc3VlQ29tbWVudDYxNDI4Mzg0Mg==
created_at: 2020-04-15T21:14:57Z
updated_at: 2020-04-15T21:14:57Z
author_association: OWNER
body:
  https://datasette.readthedocs.io/en/latest/custom_templates.html#css-classes-on-the-body has the updated documentation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/76#issuecomment-614354219
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/76
node_id: MDEyOklzc3VlQ29tbWVudDYxNDM1NDIxOQ==
created_at: 2020-04-16T01:01:34Z
updated_at: 2020-04-16T01:01:34Z
author_association: OWNER
body:
  I think a neat way to do this would be with an optional argument for `.rows_where()`:

  ```python
  rows = db["table"].rows_where("age > 10", order_by="age desc")
  ```

  If you want everything you can use this:

  ```python
  rows = db["table"].rows_where(order_by="age desc")
  ```

  It's a tiny bit weird calling `.rows_where()` without a where clause, but I think it makes sense here - especially since `.rows` is a property that can't take any arguments - though under the hood it actually does this:

  https://github.com/simonw/sqlite-utils/blob/ad6ac19470a67867b96cb4c086450b8e4e46bf02/sqlite_utils/db.py#L436-L443

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/26#issuecomment-614794739
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26
node_id: MDEyOklzc3VlQ29tbWVudDYxNDc5NDczOQ==
created_at: 2020-04-16T17:38:28Z
updated_at: 2020-04-16T17:38:28Z
author_association: MEMBER
body:
  I'm already doing this here: https://github.com/dogsheep/github-to-sqlite/blob/c4aaa50e167cfa9021c7c94260bc3e89e10947bf/github_to_sqlite/utils.py#L246-L250

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614831842
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27
node_id: MDEyOklzc3VlQ29tbWVudDYxNDgzMTg0Mg==
created_at: 2020-04-16T18:48:18Z
updated_at: 2020-04-16T18:48:18Z
author_association: MEMBER
body:
  I'm going to make `organization` another foreign key to the `users` table just in case it IS possible (maybe with GitHub Enterprise or similar?)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614843406
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27
node_id: MDEyOklzc3VlQ29tbWVudDYxNDg0MzQwNg==
created_at: 2020-04-16T19:11:53Z
updated_at: 2020-04-16T19:20:23Z
author_association: MEMBER
body:
  This didn't quite work: the column type is incorrect, so the foreign key relationship isn't sticking:

  https://github-to-sqlite.dogsheep.net/github/repos?organization=53015001

  `[organization] TEXT REFERENCES [users]([id])` - should be `INTEGER`.

  The problem is that if the first repo inserted has no organization it's set to `null`, which `sqlite-utils` derives as a `TEXT` column.

  One solution would be to create the column explicitly with a type, but this could get messy.

  I think I want a new sqlite-utils feature for this instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/100#issuecomment-614845657
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/100
node_id: MDEyOklzc3VlQ29tbWVudDYxNDg0NTY1Nw==
created_at: 2020-04-16T19:17:05Z
updated_at: 2020-04-16T19:17:05Z
author_association: OWNER
body:
  The syntax for explicitly creating a table looks like this:
  ```python
  db["cats"].create({
      "id": int,
      "name": str,
      "weight": float,
  }, pk="id")
  ```
  So using a dictionary of column name -> Python type here would be good.

  I think it becomes a new optional argument to `.insert_all()` and `.insert()` and `.upsert_all()` and `.upsert()`. Question is, what should that argument be called?

  Options:

  - `columns`
  - `types`
  - `type_overrides`
  - `extra_columns`

  I like the first two best. I'm leaning towards `columns` - it would serve two purposes: you can use it to ensure extra columns are created, and you can use it to specify the type on extra OR deteced columns.

  Maybe something like this:

  ```python
  db["authors"].insert_all(
      [{"id": 1, "name": "Sally", "score": "2"}],
      pk="id",
      not_null={"name", "score"},
      defaults={"score": 1},
      columns={"score": int, "weight": float}
  )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/100#issuecomment-614846059
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/100
node_id: MDEyOklzc3VlQ29tbWVudDYxNDg0NjA1OQ==
created_at: 2020-04-16T19:17:56Z
updated_at: 2020-04-16T19:17:56Z
author_association: OWNER
body:
  Reminder: the current list of arguments for those methods is:
  https://github.com/simonw/sqlite-utils/blob/13528faa817d79bc3900d3af7473300686b145d7/sqlite_utils/db.py#L951-L967

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615509578
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37
node_id: MDEyOklzc3VlQ29tbWVudDYxNTUwOTU3OA==
created_at: 2020-04-17T23:36:00Z
updated_at: 2020-04-17T23:36:00Z
author_association: MEMBER
body:
  ```
  $ twitter-to-sqlite user-timeline doggo.db doggoenthuonetuh
  Traceback (most recent call last):
  ...
    File "/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py", line 272, in transform_user
      user["created_at"] = parser.parse(user["created_at"])
  KeyError: 'created_at'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615509803
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37
node_id: MDEyOklzc3VlQ29tbWVudDYxNTUwOTgwMw==
created_at: 2020-04-17T23:36:40Z
updated_at: 2020-04-17T23:36:40Z
author_association: MEMBER
body:
  I'm going to print a warning to stderr, skip and continue - because if you have 100 screen names and only one of them is invalid you should still execute for the other 99.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615510361
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37
node_id: MDEyOklzc3VlQ29tbWVudDYxNTUxMDM2MQ==
created_at: 2020-04-17T23:38:27Z
updated_at: 2020-04-17T23:38:27Z
author_association: MEMBER
body:
  That's a bit tricky since I'd have to rewrite the internals of a bunch of other commands. For the moment I'll exit the script with an error but at least it will be a decent error!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-615519409
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27
node_id: MDEyOklzc3VlQ29tbWVudDYxNTUxOTQwOQ==
created_at: 2020-04-18T00:19:16Z
updated_at: 2020-04-18T00:19:16Z
author_association: MEMBER
body:
  ```
  $ github-to-sqlite repos b.db dogsheep
  $ sqlite3 b.db '.schema repos'
  CREATE TABLE [repos] (
     [id] INTEGER PRIMARY KEY,
  ...
     [permissions] TEXT,
     [organization] INTEGER REFERENCES [users]([id]),
     FOREIGN KEY(license) REFERENCES licenses(key)
  );
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615883040
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28
node_id: MDEyOklzc3VlQ29tbWVudDYxNTg4MzA0MA==
created_at: 2020-04-18T14:45:38Z
updated_at: 2020-04-18T14:45:38Z
author_association: MEMBER
body:
  ```
    File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py", line 219, in contributors
      utils.save_contributors(db, contributors, repo_full["id"])
    File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 354, in save_contributors
      for contributor in contributors:
    File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 228, in fetch_contributors
      for contributors in paginate(url, headers):
    File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 286, in paginate
      data = response.json()
    File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/requests/models.py", line 898, in json
      return complexjson.loads(self.text, **kwargs)
    File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/__init__.py", line 357, in loads
      return _default_decoder.decode(s)
    File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/decoder.py", line 337, in decode
      obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/decoder.py", line 355, in raw_decode
      raise JSONDecodeError("Expecting value", s, err.value) from None
  json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615883687
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28
node_id: MDEyOklzc3VlQ29tbWVudDYxNTg4MzY4Nw==
created_at: 2020-04-18T14:49:58Z
updated_at: 2020-04-18T14:49:58Z
author_association: MEMBER
body:
  That happened trying to pull contributors for `dogsheep/beta` - an empty repository. Turns out it was returning a `204 no content`:
  ```
  ~ $ curl -i 'https://api.github.com/repos/dogsheep/beta/contributors'
  HTTP/1.1 204 No Content
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615886206
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28
node_id: MDEyOklzc3VlQ29tbWVudDYxNTg4NjIwNg==
created_at: 2020-04-18T15:04:59Z
updated_at: 2020-04-18T15:04:59Z
author_association: MEMBER
body:
  Demo: https://github-to-sqlite.dogsheep.net/github/contributors

  Documentation: https://github.com/dogsheep/github-to-sqlite/blob/13f8868fb5efa01c263b24f6dd91c617e6e938e1/README.md#fetching-contributors-to-a-repository 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615932007
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDYxNTkzMjAwNw==
created_at: 2020-04-18T19:27:55Z
updated_at: 2020-04-18T19:27:55Z
author_association: MEMBER
body:
  Research thread: https://twitter.com/simonw/status/1249049694984011776
  > I want to build some software that lets people store their own data in their own S3 bucket, but if possible I'd like not to have to teach people the incantations needed to get their bucket setup and minimum-permission credentials figures out

  https://testdriven.io/blog/storing-django-static-and-media-files-on-amazon-s3/ looks useful

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615933273
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDYxNTkzMzI3Mw==
created_at: 2020-04-18T19:37:33Z
updated_at: 2020-04-18T19:37:33Z
author_association: MEMBER
body:
  https://console.aws.amazon.com/s3/bucket/create?region=us-west-1

  ![S3_Management_Console](https://user-images.githubusercontent.com/9599/79669552-33e2a380-8171-11ea-9ab5-5785d34f652a.png)

  I created it with no public read-write access. I plan to use signed URLs via a transforming proxy to access images for display on the web.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615935577
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDYxNTkzNTU3Nw==
created_at: 2020-04-18T19:54:59Z
updated_at: 2020-04-18T19:55:30Z
author_association: MEMBER
body:
  Creating IAM groups called `dogsheep-photos-simon-read-write` and `dogsheep-photos-simon-read`: https://console.aws.amazon.com/iam/home#/groups - I created them with no attached policies.

  Now I can attach an "inline policy" to each one. For the read-write group I go here:

  https://console.aws.amazon.com/iam/home#/groups/dogsheep-photos-simon-read-write

  ![IAM_Management_Console](https://user-images.githubusercontent.com/9599/79669703-2d086080-8172-11ea-9597-83e0b155193e.png)

  Example policies are here: https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html

  For the read-write one I went with:
  ```json
  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Action": "s3:*",
              "Resource": [
                  "arn:aws:s3:::dogsheep-photos-simon/*"
              ]
          }
      ]
  }
  ```
  For the read-only policy I'm going to guess that this is appropriate:

  ```json
  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Action": [
                  "s3:GetObject*",
                  "s3:ListBucket"
              ],
              "Resource": [
                  "arn:aws:s3:::dogsheep-photos-simon/*"
              ]
          }
      ]
  }
  ```
  I tried the policy simulator to test this out: https://policysim.aws.amazon.com/home/index.jsp?#groups/dogsheep-photos-simon-read - this worked:

  ![IAM_Policy_Simulator](https://user-images.githubusercontent.com/9599/79669893-cd12b980-8173-11ea-8dfb-5660ce3652da.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615936880
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDYxNTkzNjg4MA==
created_at: 2020-04-18T20:04:31Z
updated_at: 2020-04-18T20:04:31Z
author_association: MEMBER
body:
  Next step: create two IAM users, one for each of those groups.

  https://console.aws.amazon.com/iam/home#/users$new?step=details

  ![IAM_Management_Console](https://user-images.githubusercontent.com/9599/79669931-1bc05380-8174-11ea-9657-0e0c6a692d42.png)

  ![IAM_Management_Console](https://user-images.githubusercontent.com/9599/79669941-27137f00-8174-11ea-8ce7-249f0d4f96f6.png)

  I copied the keys into a secure note in 1password.

  Couldn't get into Transmit with them though! https://library.panic.com/transmit/transmit5/iam-roles/ may help.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615941746
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDYxNTk0MTc0Ng==
created_at: 2020-04-18T20:29:36Z
updated_at: 2020-04-18T20:29:36Z
author_association: MEMBER
body:
  I'm going to create another user just for Transmit, with full S3 access.

  name: `dogsheep-photos-simon-s3-all-access`

  Rather than creating a group for that user, I'm trying the "Attach existing policies directly" option:

  ![IAM_Management_Console](https://user-images.githubusercontent.com/9599/79670182-03513880-8176-11ea-811a-c80aefb4538a.png)

  That user DID work with Transmit. I uploaded a test HEIC image. I used Transmit to copy a signed URL for it.

  ```
  ~ $ curl -i 'https://dogsheep-photos-simon.s3.us-west-1.amazonaws.com/IMG_7195.HEIC?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAWXFXAI...' | head -n 100
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0HTTP/1.1 200 OK
  x-amz-id-2: gBOCYqZfbNAnv0R/uJ++qm2NbW5SgD4TapgF9RQjzzeDIThcCz/BkKU+YoxlG4NJHlcmMgAHyh4=
  x-amz-request-id: C2FE7FCC3BD53A84
  Date: Sat, 18 Apr 2020 20:28:54 GMT
  Last-Modified: Sat, 18 Apr 2020 20:13:49 GMT
  ETag: "fe3e081239a123ef745517878c53b854"
  Accept-Ranges: bytes
  Content-Type: image/heic
  Content-Length: 1913097
  Server: AmazonS3
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615942116
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDYxNTk0MjExNg==
created_at: 2020-04-18T20:30:56Z
updated_at: 2020-04-18T20:30:56Z
author_association: MEMBER
body:
  Next step: attempt a programmatic upload using the `dogsheep-photos-simon-read-write` credentials from a Jupyter notebook.

  Also attempt a programmatic bucket listing and read using `dogsheep-photos-simon-read` credentials.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615944806
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDYxNTk0NDgwNg==
created_at: 2020-04-18T20:41:39Z
updated_at: 2020-04-18T20:41:39Z
author_association: MEMBER
body:
  This worked!

  ![Dogsheep_Photos_S3_access](https://user-images.githubusercontent.com/9599/79670712-d868e380-8179-11ea-82a5-5dfd17356113.png)

  And this worked:

  ![Dogsheep_Photos_S3_access](https://user-images.githubusercontent.com/9599/79670777-50370e00-817a-11ea-83cd-18ebf5702878.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615945056
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDYxNTk0NTA1Ng==
created_at: 2020-04-18T20:42:41Z
updated_at: 2020-04-18T20:42:41Z
author_association: MEMBER
body:
  But... `list_objects` failed for both of my keys (read and write):

  ![Dogsheep_Photos_S3_access](https://user-images.githubusercontent.com/9599/79670798-75c41780-817a-11ea-9907-2cbc4a2e497c.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615946537
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDYxNTk0NjUzNw==
created_at: 2020-04-18T20:48:13Z
updated_at: 2020-04-18T20:48:13Z
author_association: MEMBER
body:
  How about generating a signed URL?
  ```python
  read_client.generate_presigned_url(
      "get_object",
      Params={
          "Bucket": "dogsheep-photos-simon",
          "Key": "this_is_fine.jpg",
      },
      ExpiresIn=600
  )
  ```
  Gave me https://dogsheep-photos-simon.s3.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398

  Which does this:

  ```
  ~ $ curl -i 'https://dogsheep-photos-simon.s3.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398'
  HTTP/1.1 307 Temporary Redirect
  x-amz-bucket-region: us-west-1
  x-amz-request-id: E78CD859AEE21D33
  x-amz-id-2: 648mx+1+YSGga7NDOU7Q6isfsKnEPWOLC+DI4+x2o9FCc6pSCdIaoHJUbFMI8Vsuh1ADtx46ymU=
  Location: https://dogsheep-photos-simon.s3-us-west-1.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398
  Content-Type: application/xml
  Transfer-Encoding: chunked
  Date: Sat, 18 Apr 2020 20:47:21 GMT
  Server: AmazonS3

  <?xml version="1.0" encoding="UTF-8"?>
  <Error><Code>TemporaryRedirect</Code><Message>Please re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests.</Message><Endpoint>dogsheep-photos-simon.s3-us-west-1.amazonaws.com</Endpoint><Bucket>dogsheep-photos-simon</Bucket><RequestId>E78CD859AEE21D33</RequestId><HostId>648mx+1+YSGga7NDOU7Q6isfsKnEPWOLC+DI4+x2o9FCc6pSCdIaoHJUbFMI8Vsuh1ADtx46ymU=</HostId></Error>~ $ 
  ```
  So it redirects to another URL... which returns this:
  ```
  ~ $ curl -i 'https://dogsheep-photos-simon.s3-us-west-1.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398'
  HTTP/1.1 200 OK
  x-amz-id-2: XafOl6mswj3yz0GJC9+Ptot1ll5sROVwqsMc10CUUfgpaUANTdIx2GhnONb5d1GVFJ6wlS2j3UY=
  x-amz-request-id: 258387C180411AFE
  Date: Sat, 18 Apr 2020 20:47:52 GMT
  Last-Modified: Sat, 18 Apr 2020 20:37:35 GMT
  ETag: "ee04081c3182a44a1c6944e94012e977"
  Accept-Ranges: bytes
  Content-Type: binary/octet-stream
  Content-Length: 53072
  Server: AmazonS3

  ????JFIF??C
  ```
  So that worked! It did come back with `Content-Type: binary/octet-stream` though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615947229
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDYxNTk0NzIyOQ==
created_at: 2020-04-18T20:51:26Z
updated_at: 2020-04-18T20:51:26Z
author_association: MEMBER
body:
  Running the upload again like this resulted in the correct content-type:
  ```python
  client.upload_file(
      "/Users/simonw/Desktop/this_is_fine.jpg",
      "dogsheep-photos-simon",
      "this_is_fine.jpg",
      ExtraArgs={
          "ContentType": "image/jpeg"
      }
  )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615947370
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDYxNTk0NzM3MA==
created_at: 2020-04-18T20:52:13Z
updated_at: 2020-04-18T20:52:13Z
author_association: MEMBER
body:
  This is great! I now have a key that can upload photos, and a separate key that can download photos OR generate signed URLs to access those photos.

  Next step: a script that starts uploading my photos.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615948102
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDYxNTk0ODEwMg==
created_at: 2020-04-18T20:56:59Z
updated_at: 2020-04-18T20:56:59Z
author_association: MEMBER
body:
  I'm going to start with this:

  `photos-to-sqlite upload photos.db ~/path/to/directory`

  This will scan the provided directory (and all sub-directories) for image files. It will then:

  * Calculate a sha256 of the contents of that file
  * Upload the file to a key that's `sha256.jpg` or `.heic`
  * Upload a `sha256.json` file with the original path to the image
  * Add that image to a `uploads` table in `photos.db`

  Stretch goal: grab the EXIF data and include that in the `.json` upload AND the `uploads` database table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/5#issuecomment-615949574
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDYxNTk0OTU3NA==
created_at: 2020-04-18T21:06:07Z
updated_at: 2020-04-18T21:06:07Z
author_association: MEMBER
body:
  ```
  $ photos-to-sqlite s3-auth
  Create S3 credentials and paste them here:

  Access key ID: xxx
  Secret access key: yyy
  $ cat auth.json 
  {
      "access_key_id": "xxx",
      "secret_access_key": "yyy"
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/6#issuecomment-615979923
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/6
node_id: MDEyOklzc3VlQ29tbWVudDYxNTk3OTkyMw==
created_at: 2020-04-18T23:36:02Z
updated_at: 2020-04-18T23:36:02Z
author_association: MEMBER
body:
  I'll use a Click progress bar. To do this I need to first calculate the sum number of bytes in the photos that are going to be uploaded, then run the upload.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/6#issuecomment-615983393
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/6
node_id: MDEyOklzc3VlQ29tbWVudDYxNTk4MzM5Mw==
created_at: 2020-04-18T23:53:10Z
updated_at: 2020-04-18T23:53:10Z
author_association: MEMBER
body:
  ```
  $ photos-to-sqlite upload photos3.db ~/Pictures/Photos\ Library.photoslibrary/Masters/2020
  Uploading 2.09 GB  [##----------------------------------]    6%  00:36:37
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/45#issuecomment-616029262
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/45
node_id: MDEyOklzc3VlQ29tbWVudDYxNjAyOTI2Mg==
created_at: 2020-04-19T04:39:21Z
updated_at: 2020-04-19T04:39:21Z
author_association: MEMBER
body:
  ![44714E00-8CC5-46CD-9E48-1F4DD148FCC8](https://user-images.githubusercontent.com/9599/79679696-09b6d300-81bd-11ea-80e4-0653d92e4f58.jpeg)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/30#issuecomment-616883726
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30
node_id: MDEyOklzc3VlQ29tbWVudDYxNjg4MzcyNg==
created_at: 2020-04-21T00:45:23Z
updated_at: 2020-04-21T00:45:23Z
author_association: MEMBER
body:
  Demo of fix: https://github-to-sqlite.dogsheep.net/github/issues?assignee__notblank=1&milestone__notblank=1

  ![github__issues__4_rows_where_where_assignee_is_not_blank_and_milestone_is_not_blank_sorted_by_updated_at_descending](https://user-images.githubusercontent.com/9599/79812758-b49ace80-832e-11ea-81db-bdf993b872cc.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-616884647
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31
node_id: MDEyOklzc3VlQ29tbWVudDYxNjg4NDY0Nw==
created_at: 2020-04-21T00:49:16Z
updated_at: 2020-04-21T00:50:20Z
author_association: MEMBER
body:
  The API just gives us the `repository_url`: https://api.github.com/repos/simonw/datasette/issues

  ![Mozilla_Firefox_and_Topic__Week_2__Discussion__Submit_your_six_story_points_here](https://user-images.githubusercontent.com/9599/79812950-283cdb80-832f-11ea-8759-9633087d1e7e.png)

  We currently turn that into a `simonw/datasette` string here:
  https://github.com/dogsheep/github-to-sqlite/blob/e0e8d8caa9657b04bfb8a2cf16c9b580f38b1805/github_to_sqlite/utils.py#L43-L46

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-617348174
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31
node_id: MDEyOklzc3VlQ29tbWVudDYxNzM0ODE3NA==
created_at: 2020-04-21T18:50:29Z
updated_at: 2020-04-21T18:50:29Z
author_association: MEMBER
body:
  Since this represents a breaking schema change for anyone running SQL queries against these tables, I'm going to do a major version bump to 2.0 when I release this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617364956
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32
node_id: MDEyOklzc3VlQ29tbWVudDYxNzM2NDk1Ng==
created_at: 2020-04-21T19:24:45Z
updated_at: 2020-04-21T19:24:45Z
author_association: MEMBER
body:
  That's because I just broke this code:

  https://github.com/dogsheep/github-to-sqlite/blob/2cf75a0a036719eb7e57fdc7c5c2ea0f4c26978a/github_to_sqlite/utils.py#L131-L139

  It expects the `repo` column to be `simonw/datasette` but it's now an ID instead.

  I should add a test for this as part of the fix.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617490914
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32
node_id: MDEyOklzc3VlQ29tbWVudDYxNzQ5MDkxNA==
created_at: 2020-04-22T01:17:44Z
updated_at: 2020-04-22T01:17:44Z
author_association: MEMBER
body:
  https://github-to-sqlite.dogsheep.net/github?sql=select+html_url%2C+id%2C+issue+from+issue_comments+order+by+updated_at+desc+limit+101 now shows issues.

  And https://github-to-sqlite.dogsheep.net/github/issue_comments links to them:

  <img width="395" alt="Screen Shot 2020-04-21 at 6 17 33 PM" src="https://user-images.githubusercontent.com/9599/79930066-6227e380-83fc-11ea-8416-8b9d55660cb5.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-617491607
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31
node_id: MDEyOklzc3VlQ29tbWVudDYxNzQ5MTYwNw==
created_at: 2020-04-22T01:20:19Z
updated_at: 2020-04-22T01:20:19Z
author_association: MEMBER
body:
  https://github-to-sqlite.dogsheep.net/github/milestones now link to repo:

  <img width="1095" alt="Screen Shot 2020-04-21 at 6 19 03 PM" src="https://user-images.githubusercontent.com/9599/79930145-956a7280-83fc-11ea-9b47-b0c5674a0d4f.png">

  And so do issues: https://github-to-sqlite.dogsheep.net/github/issues

  <img width="1095" alt="Screen Shot 2020-04-21 at 6 19 53 PM" src="https://user-images.githubusercontent.com/9599/79930200-bd59d600-83fc-11ea-8fb8-b40772140409.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/731#issuecomment-618070541
issue_url: https://api.github.com/repos/simonw/datasette/issues/731
node_id: MDEyOklzc3VlQ29tbWVudDYxODA3MDU0MQ==
created_at: 2020-04-22T22:20:38Z
updated_at: 2020-04-22T22:20:38Z
author_association: OWNER
body:
  I started thinking about this while building https://github.com/simonw/datasette-publish-now

  The Vercel hosting platform is built on ASGI - so I need a way to provide an `app` ASGI object for it to use rather than Datasette's normal pattern of starting up the server using the `datasette serve` command.

  The plugin currently does that by generating an `index.py` file that looks like this:

  ```python
  from datasette.app import Datasette
  import json
  metadata = dict()
  try:
      metadata = json.load(open("metadata.json"))
  except Exception:
      pass
  app = Datasette([], ["fixtures.db"], metadata=metadata).app()
  ```
  But that means I have to replicate the different CLI options - `--plugins-dir` etc - and figure out how to pass them to the `Datasette()` constructor.

  This would be a whole lot easier if I could use `app = Datasette(configure_directory=".")` and provide a magically configured instance based on the presence of files in the specified directory.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/8#issuecomment-618100434
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/8
node_id: MDEyOklzc3VlQ29tbWVudDYxODEwMDQzNA==
created_at: 2020-04-23T00:02:53Z
updated_at: 2020-04-23T00:02:53Z
author_association: MEMBER
body:
  I don't think it matters one way or the other - I'm storing the sha256 in the filename, so the fact that I could read the MD5 back from the list bucket operation doesn't give me any benefits.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/8#issuecomment-618100658
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/8
node_id: MDEyOklzc3VlQ29tbWVudDYxODEwMDY1OA==
created_at: 2020-04-23T00:03:35Z
updated_at: 2020-04-23T00:03:35Z
author_association: MEMBER
body:
  Also MD5 isn't guaranteed for the ETag:
  > If an object is created by either the Multipart Upload or Part Copy operation, the ETag is not an MD5 digest, regardless of the method of encryption.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/731#issuecomment-618126449
issue_url: https://api.github.com/repos/simonw/datasette/issues/731
node_id: MDEyOklzc3VlQ29tbWVudDYxODEyNjQ0OQ==
created_at: 2020-04-23T01:38:55Z
updated_at: 2020-04-23T01:38:55Z
author_association: CONTRIBUTOR
body:
  I've almost suggested this same thing a couple times. I tend to have Makefile (because I'm doing other `make` stuff anyway to get data prepped), and I end up putting all those CLI options in something like `make run`. But it would be way easier to just have all those typical options -- plugins, templates, metadata -- be defaults.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/731#issuecomment-618155472
issue_url: https://api.github.com/repos/simonw/datasette/issues/731
node_id: MDEyOklzc3VlQ29tbWVudDYxODE1NTQ3Mg==
created_at: 2020-04-23T03:28:42Z
updated_at: 2020-04-23T03:28:56Z
author_association: OWNER
body:
  As an alternative to `--static` this could work by letting you create the following:

  - `static/css/`
  - `static/js/`

  Which would be automatically mounted at `/js/...` and `/css/...`

  Or maybe just mount `static/` at `/static/` instead? 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/9#issuecomment-618724149
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/9
node_id: MDEyOklzc3VlQ29tbWVudDYxODcyNDE0OQ==
created_at: 2020-04-23T23:35:29Z
updated_at: 2020-04-23T23:35:29Z
author_association: MEMBER
body:
  ```
  % photos-to-sqlite upload photos.db ~/Pictures/Photos\ Library.photoslibrary/originals
  Fetching existing keys from S3...
  Got 22,446 existing keys
  Calculating hashes  [####--------------------------------]   13%  00:04:14
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/9#issuecomment-618725155
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/9
node_id: MDEyOklzc3VlQ29tbWVudDYxODcyNTE1NQ==
created_at: 2020-04-23T23:39:14Z
updated_at: 2020-04-23T23:39:14Z
author_association: MEMBER
body:
  A few minutes later...
  ```
  Fetching existing keys from S3...
  Got 22,446 existing keys
  Calculating hashes  [####################################]  100%          
  22,441 hashed files, 610 are not yet in S3
  Uploading 0.99 GB
  Uploading 610 photos  [------------------------------------]  1/610  03:10:35
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/731#issuecomment-618758326
issue_url: https://api.github.com/repos/simonw/datasette/issues/731
node_id: MDEyOklzc3VlQ29tbWVudDYxODc1ODMyNg==
created_at: 2020-04-24T01:55:00Z
updated_at: 2020-04-24T01:55:00Z
author_association: CONTRIBUTOR
body:
  Mounting `./static` at `/static` seems the simplest way. Saves you the trouble of deciding what else (`img` for example) gets special treatment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/731#issuecomment-618772317
issue_url: https://api.github.com/repos/simonw/datasette/issues/731
node_id: MDEyOklzc3VlQ29tbWVudDYxODc3MjMxNw==
created_at: 2020-04-24T02:49:46Z
updated_at: 2020-04-24T02:49:46Z
author_association: OWNER
body:
  Yeah I'll do that. My original thinking on allowing the `--static` option to mount other things to the root is that it would make it easy for people to add HTML "pages" to their site - like https://www.niche-museums.com/map for example.

  But I've proposed a better solution for that here: #648

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/648#issuecomment-618775631
issue_url: https://api.github.com/repos/simonw/datasette/issues/648
node_id: MDEyOklzc3VlQ29tbWVudDYxODc3NTYzMQ==
created_at: 2020-04-24T03:03:35Z
updated_at: 2020-04-24T03:03:35Z
author_association: OWNER
body:
  > Stretch goal: it would be neat if these pages could return custom HTTP headers (eg content-type) and maybe even status codes (eg for redirects) somehow.

  I think I could do that with a custom template function - if that function is called during the render then we follow those instructions instead of returning the rendered HTML.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/12#issuecomment-618796564
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/12
node_id: MDEyOklzc3VlQ29tbWVudDYxODc5NjU2NA==
created_at: 2020-04-24T04:35:25Z
updated_at: 2020-04-24T04:35:25Z
author_association: MEMBER
body:
  Code: https://github.com/dogsheep/photos-to-sqlite/blob/a388cf1f1b6b67752d669466cda8b171b6582171/photos_to_sqlite/cli.py#L109-L114

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/648#issuecomment-619259859
issue_url: https://api.github.com/repos/simonw/datasette/issues/648
node_id: MDEyOklzc3VlQ29tbWVudDYxOTI1OTg1OQ==
created_at: 2020-04-24T22:10:34Z
updated_at: 2020-04-24T22:10:34Z
author_association: OWNER
body:
  The trickiest part here is the 404 logic. It's spread out through a couple of places right now. It's in `datasette.utils.asgi.AsgiRouter`:

  https://github.com/simonw/datasette/blob/227bb3e91fe34811a9374150798643a5af98ed79/datasette/utils/asgi.py#L98-L134

  Then also in the `DatasetteRouter subclass of that:

  https://github.com/simonw/datasette/blob/227bb3e91fe34811a9374150798643a5af98ed79/datasette/app.py#L736-L757

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/648#issuecomment-619263600
issue_url: https://api.github.com/repos/simonw/datasette/issues/648
node_id: MDEyOklzc3VlQ29tbWVudDYxOTI2MzYwMA==
created_at: 2020-04-24T22:23:13Z
updated_at: 2020-04-24T22:23:13Z
author_association: OWNER
body:
  Idea: `handle_404` can hand certain not found errors off to `handle_500` so it can render them correctly;

  https://github.com/simonw/datasette/blob/227bb3e91fe34811a9374150798643a5af98ed79/datasette/app.py#L748-L765

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/725#issuecomment-619489720
issue_url: https://api.github.com/repos/simonw/datasette/issues/725
node_id: MDEyOklzc3VlQ29tbWVudDYxOTQ4OTcyMA==
created_at: 2020-04-26T06:09:59Z
updated_at: 2020-04-26T06:10:13Z
author_association: NONE
body:
  as a complementary remark: the versioning of datasette dependancies will become a problem when the new pip "dependancy resolver" will be activated. for now, it's just warnings via pip checks, later it will be a "no":

  ````
  datasette 0.40 has requirement aiofiles~=0.4.0, but you have aiofiles 0.5.0.
  datasette 0.40 has requirement Jinja2~=2.10.3, but you have jinja2 2.11.2.
  ````

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/648#issuecomment-619591380
issue_url: https://api.github.com/repos/simonw/datasette/issues/648
node_id: MDEyOklzc3VlQ29tbWVudDYxOTU5MTM4MA==
created_at: 2020-04-26T17:33:04Z
updated_at: 2020-04-26T17:33:04Z
author_association: OWNER
body:
  > > Stretch goal: it would be neat if these pages could return custom HTTP headers (eg content-type) and maybe even status codes (eg for redirects) somehow.
  > 
  > I think I could do that with a custom template function - if that function is called during the render then we follow those instructions instead of returning the rendered HTML.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/737#issuecomment-619591533
issue_url: https://api.github.com/repos/simonw/datasette/issues/737
node_id: MDEyOklzc3VlQ29tbWVudDYxOTU5MTUzMw==
created_at: 2020-04-26T17:33:48Z
updated_at: 2020-04-26T17:33:48Z
author_association: OWNER
body:
  > > Stretch goal: it would be neat if these pages could return custom HTTP headers (eg content-type) and maybe even status codes (eg for redirects) somehow.
  > 
  > I think I could do that with a custom template function - if that function is called during the render then we follow those instructions instead of returning the rendered HTML.
  https://github.com/simonw/datasette/issues/648#issuecomment-619591380

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/737#issuecomment-619593811
issue_url: https://api.github.com/repos/simonw/datasette/issues/737
node_id: MDEyOklzc3VlQ29tbWVudDYxOTU5MzgxMQ==
created_at: 2020-04-26T17:48:31Z
updated_at: 2020-04-26T17:48:31Z
author_association: OWNER
body:
  Design challenge: the function that can be called in the custom template to return a custom status code and/or headers.

  Proposed design:
  ```
  {{ custom_headers({"x-foo": "bar"}) }}
  {{ custom_status_code(404) }}
  ```
  Or should I combine them into one function?
  ```
  {{ custom_headers({"x-foo": "bar"}, status=404) }}
  ```
  A common use-case here would be a redirect. How should that look?
  ```
  {{ custom_redirect("/foo") }} - defaults to 301
  {{ custom_redirect("/foo", status=302) }}
  ```
  Maybe avoid dictionary syntax entirely:
  ```
  {{ custom_header("x-foo", "bar") }}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/737#issuecomment-619593899
issue_url: https://api.github.com/repos/simonw/datasette/issues/737
node_id: MDEyOklzc3VlQ29tbWVudDYxOTU5Mzg5OQ==
created_at: 2020-04-26T17:49:06Z
updated_at: 2020-04-26T17:49:06Z
author_association: OWNER
body:
  I'm going with `custom_header(name, value)`, `custom_status(numeric_code)`, `custom_redirect("URL")`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/738#issuecomment-619605811
issue_url: https://api.github.com/repos/simonw/datasette/issues/738
node_id: MDEyOklzc3VlQ29tbWVudDYxOTYwNTgxMQ==
created_at: 2020-04-26T18:58:56Z
updated_at: 2020-04-26T18:58:56Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/304e7b1d9fd904ae1c35600bc03662eb90eeeae0/datasette/app.py#L781-L789

  `request` is an optional argument here:

  https://github.com/simonw/datasette/blob/304e7b1d9fd904ae1c35600bc03662eb90eeeae0/datasette/app.py#L556-L558

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/731#issuecomment-619611417
issue_url: https://api.github.com/repos/simonw/datasette/issues/731
node_id: MDEyOklzc3VlQ29tbWVudDYxOTYxMTQxNw==
created_at: 2020-04-26T19:28:12Z
updated_at: 2020-04-26T19:28:12Z
author_association: OWNER
body:
  I've shipped #648 now, so no reason not to go with the `static/` folder as the thing that gets detected and mounted as `/static`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/648#issuecomment-619612675
issue_url: https://api.github.com/repos/simonw/datasette/issues/648
node_id: MDEyOklzc3VlQ29tbWVudDYxOTYxMjY3NQ==
created_at: 2020-04-26T19:35:09Z
updated_at: 2020-04-26T19:35:09Z
author_association: OWNER
body:
  https://www.niche-museums.com/about now uses this mechanism. It used to be an empty `about.db` database but you can see from https://www.niche-museums.com/-/databases that it's now using the new mechanism.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/739#issuecomment-619622311
issue_url: https://api.github.com/repos/simonw/datasette/issues/739
node_id: MDEyOklzc3VlQ29tbWVudDYxOTYyMjMxMQ==
created_at: 2020-04-26T20:45:07Z
updated_at: 2020-04-26T20:45:07Z
author_association: OWNER
body:
  I think ALL explicit options should over-ride options that were detected in the directory structure. This means my current implementation isn't quite right - using a class method makes it hard to merge the details with the explicit options passed to the CLI function.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/739#issuecomment-619678614
issue_url: https://api.github.com/repos/simonw/datasette/issues/739
node_id: MDEyOklzc3VlQ29tbWVudDYxOTY3ODYxNA==
created_at: 2020-04-27T02:44:21Z
updated_at: 2020-04-27T02:44:21Z
author_association: OWNER
body:
  I'm going to add a `Datasette(config_dir=".")` parameter which initializes from the given directory, allowing other parameters to over-ride it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/739#issuecomment-619700113
issue_url: https://api.github.com/repos/simonw/datasette/issues/739
node_id: MDEyOklzc3VlQ29tbWVudDYxOTcwMDExMw==
created_at: 2020-04-27T03:58:09Z
updated_at: 2020-04-27T03:58:09Z
author_association: OWNER
body:
  I think documentation goes on https://datasette.readthedocs.io/en/stable/config.html - but I'm going to need to re-arrange that page to fit in a section about configuration directories.

  I'll move the existing list of settings into a "configration options" section.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/741#issuecomment-619708745
issue_url: https://api.github.com/repos/simonw/datasette/issues/741
node_id: MDEyOklzc3VlQ29tbWVudDYxOTcwODc0NQ==
created_at: 2020-04-27T04:29:26Z
updated_at: 2020-04-27T04:29:26Z
author_association: OWNER
body:
  I can show deprecation warnings for `--extra-options` and remove the option entirely in Datasette 1.0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/741#issuecomment-619708914
issue_url: https://api.github.com/repos/simonw/datasette/issues/741
node_id: MDEyOklzc3VlQ29tbWVudDYxOTcwODkxNA==
created_at: 2020-04-27T04:30:02Z
updated_at: 2020-04-27T04:30:24Z
author_association: OWNER
body:
  This can generate a `config.json` file that's included the published deployment, thanks to the work in #731 (pull request #739).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/742#issuecomment-620153909
issue_url: https://api.github.com/repos/simonw/datasette/issues/742
node_id: MDEyOklzc3VlQ29tbWVudDYyMDE1MzkwOQ==
created_at: 2020-04-27T18:24:53Z
updated_at: 2020-04-27T18:24:53Z
author_association: OWNER
body:
  I'm already using it extensively:

  https://github.com/simonw/datasette/blob/25014ca25eb70b4c1217558ebd14db2845973bfb/tests/fixtures.py#L165-L189

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/743#issuecomment-620166959
issue_url: https://api.github.com/repos/simonw/datasette/issues/743
node_id: MDEyOklzc3VlQ29tbWVudDYyMDE2Njk1OQ==
created_at: 2020-04-27T18:50:30Z
updated_at: 2020-04-27T18:50:30Z
author_association: OWNER
body:
  Here's the `escape_fts()` function:
  https://github.com/simonw/datasette/blob/89c4ddd4828623888e91a1d2cb396cba12d4e7b4/datasette/utils/__init__.py#L742-L753

  https://latest.datasette.io/fixtures?sql=select+escape_fts%28%27bar%2A%27%29

  <img width="466" alt="fixtures__select_escape_fts__bar____and_Launch_Meeting_-_Zoom" src="https://user-images.githubusercontent.com/9599/80409172-3df55800-887d-11ea-9c71-2245f2f942e3.png">

  So apparently wrapping a SQLite FTS word like `"bar*"` doesn't prevent SQLite from expanding the wildcard.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/743#issuecomment-620170826
issue_url: https://api.github.com/repos/simonw/datasette/issues/743
node_id: MDEyOklzc3VlQ29tbWVudDYyMDE3MDgyNg==
created_at: 2020-04-27T18:58:04Z
updated_at: 2020-04-27T18:58:04Z
author_association: OWNER
body:
  Maybe this is moot because you can't store a `*` character in a FTS table anyway, so it would never make sense to search for one? In which case maybe `escape_fts()` should just strip out `*` entirely?

  Best source of information I could find was this tiny thread from 2014 about FTS4:

  http://sqlite.1065341.n5.nabble.com/Escaping-conventions-for-FTS4-virtual-table-queries-td74589.html

  >  Dave Baggett wrote:
  > > What if I want docids of documents containing the exact literal token "any*"?
  > 
  > You would have to use one of the Unicode tokenizers, and configure it to
  > interpret * as a token character.
  > 
  > > how do I escape the asterisk so that it's not interpreted as a wildcard?
  > 
  > There are no escapes.  When * is a token character, you lose the ability
  > to do prefix searches. 

  I could investigate further by learning to use the fts5vocab virtual table debugging tool to see what's actually stored in those FTS5 indexes and check if `*` is indeed stripped by them.

  https://www.sqlite.org/fts5.html#the_fts5vocab_virtual_table_module

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/736#issuecomment-620171434
issue_url: https://api.github.com/repos/simonw/datasette/issues/736
node_id: MDEyOklzc3VlQ29tbWVudDYyMDE3MTQzNA==
created_at: 2020-04-27T18:59:16Z
updated_at: 2020-04-27T18:59:16Z
author_association: OWNER
body:
  I'm suspicious that this might be a bug with `datasette-publish-now` not with core Datasette. See https://github.com/simonw/datasette-publish-now/issues/20 - I think that might be the same issue.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/736#issuecomment-620171785
issue_url: https://api.github.com/repos/simonw/datasette/issues/736
node_id: MDEyOklzc3VlQ29tbWVudDYyMDE3MTc4NQ==
created_at: 2020-04-27T18:59:54Z
updated_at: 2020-04-27T18:59:54Z
author_association: OWNER
body:
  Would you mind trying publishing your database using one of the other options - Heroku, Cloud Run or https://fly.io/ - and see if you have the same bug there?

  If you DO get the same bug with another option, please re-open this issue.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/735#issuecomment-620172085
issue_url: https://api.github.com/repos/simonw/datasette/issues/735
node_id: MDEyOklzc3VlQ29tbWVudDYyMDE3MjA4NQ==
created_at: 2020-04-27T19:00:30Z
updated_at: 2020-04-27T19:00:30Z
author_association: OWNER
body:
  I think this may be a `datasette-publish-now` bug, as in https://github.com/simonw/datasette-publish-now/issues/20 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/743#issuecomment-620174977
issue_url: https://api.github.com/repos/simonw/datasette/issues/743
node_id: MDEyOklzc3VlQ29tbWVudDYyMDE3NDk3Nw==
created_at: 2020-04-27T19:05:56Z
updated_at: 2020-04-27T19:05:56Z
author_association: OWNER
body:
  The other option would be to leave this as-is, and let people wildcard search all they want. I'm leaning in that direction.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/743#issuecomment-620177365
issue_url: https://api.github.com/repos/simonw/datasette/issues/743
node_id: MDEyOklzc3VlQ29tbWVudDYyMDE3NzM2NQ==
created_at: 2020-04-27T19:11:01Z
updated_at: 2020-04-27T19:11:30Z
author_association: OWNER
body:
  Huh... turns out the documentation already claims that wildcards work! Closing this as wontfix:

  https://datasette.readthedocs.io/en/stable/full_text_search.html#the-table-view-api

  <img width="751" alt="Full-text_search_—_Datasette_documentation" src="https://user-images.githubusercontent.com/9599/80410937-34212400-8880-11ea-84ba-1099c0f26563.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/726#issuecomment-620260658
issue_url: https://api.github.com/repos/simonw/datasette/issues/726
node_id: MDEyOklzc3VlQ29tbWVudDYyMDI2MDY1OA==
created_at: 2020-04-27T22:05:46Z
updated_at: 2020-04-27T22:05:46Z
author_association: OWNER
body:
  Aah - yes I've seen this a few times before in my own projects. The problem is that the column types don't match up - your `oeuvre_id` column here is an integer, but if you look at the schema for `prelib_oeuvre` (at the bottom of this page: http://crbc-dataset.huma-num.fr/prelib/prelib_oeuvre) you'll see that it's defined as text:

  ```sql
  CREATE TABLE "prelib_oeuvre" (
  "id" TEXT,
    "titre" TEXT,
    "descriptif" TEXT,
    "wikidata" TEXT,
    "ark_bnf" TEXT,
    "notes" TEXT
    ,PRIMARY KEY ([id])
  );
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/13#issuecomment-620273692
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/13
node_id: MDEyOklzc3VlQ29tbWVudDYyMDI3MzY5Mg==
created_at: 2020-04-27T22:42:50Z
updated_at: 2020-04-27T22:42:50Z
author_association: MEMBER
body:
  ```
  >>> def ext_counts(directory):
  ...     counts = {}
  ...     for path in pathlib.Path(directory).glob("**/*"):
  ...         ext = path.suffix
  ...         counts[ext] = counts.get(ext, 0) + 1
  ...     return counts
  ... 
  >>> 
  >>> ext_counts("/Users/simon/Pictures/Photos Library.photoslibrary/originals")
  {'': 16, '.heic': 15478, '.jpeg': 21691, '.mov': 946, '.png': 2262, '.gif': 38, '.mp4': 116, '.aae': 2}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/13#issuecomment-620309185
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/13
node_id: MDEyOklzc3VlQ29tbWVudDYyMDMwOTE4NQ==
created_at: 2020-04-28T00:39:45Z
updated_at: 2020-04-28T00:39:45Z
author_association: MEMBER
body:
  I'm going to leave this until I have the mechanism for associating a live photo video with the photo.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/736#issuecomment-620401172
issue_url: https://api.github.com/repos/simonw/datasette/issues/736
node_id: MDEyOklzc3VlQ29tbWVudDYyMDQwMTE3Mg==
created_at: 2020-04-28T06:09:28Z
updated_at: 2020-04-28T06:09:28Z
author_association: NONE
body:
  > Would you mind trying publishing your database using one of the other options - Heroku, Cloud Run or https://fly.io/ - and see if you have the same bug there?

  It works in heroku, than might be a bug with datasette-publish-now.

  Thank you

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620771067
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDYyMDc3MTA2Nw==
created_at: 2020-04-28T18:12:34Z
updated_at: 2020-04-28T18:15:38Z
author_association: MEMBER
body:
  Python library docs: https://googleapis.dev/python/vision/latest/index.html

  I'm creating a new project for this called simonwillison-photos: https://console.cloud.google.com/projectcreate

  https://console.cloud.google.com/home/dashboard?project=simonwillison-photos

  Then I enabled the Vision API. The direct link to https://console.cloud.google.com/flows/enableapi?apiid=vision-json.googleapis.com which they provided in the docs didn't work - it gave me a "You don't have sufficient permissions to use the requested API" error - but starting at the "Enable APIs" page and searching for it worked fine.

  I created a new service account as an "owner" of that project: https://console.cloud.google.com/apis/credentials/serviceaccountkey (and complained about it on Twitter and through their feedback form)

  `pip install google-cloud-vision`

  ```python
  from google.cloud import vision
  client = vision.ImageAnnotatorClient.from_service_account_file("simonwillison-photos-18c570b301fe.json")
  # Photo of a lemur
  response = client.annotate_image(
      {
          "image": {
              "source": {
                  "image_uri": "https://photos.simonwillison.net/i/1b3414ee9ade67ce04ade9042e6d4b433d1e523c9a16af17f490e2c0a619755b.jpeg"
              }
          },
          "features": [
              {"type": vision.enums.Feature.Type.IMAGE_PROPERTIES},
              {"type": vision.enums.Feature.Type.OBJECT_LOCALIZATION},
              {"type": vision.enums.Feature.Type.LABEL_DETECTION},
          ],
      }
  )
  response
  ```
  Output is:
  ```
  label_annotations {
    mid: "/m/09686"
    description: "Vertebrate"
    score: 0.9851104021072388
    topicality: 0.9851104021072388
  }
  label_annotations {
    mid: "/m/04rky"
    description: "Mammal"
    score: 0.975814163684845
    topicality: 0.975814163684845
  }
  label_annotations {
    mid: "/m/01280g"
    description: "Wildlife"
    score: 0.8973650336265564
    topicality: 0.8973650336265564
  }
  label_annotations {
    mid: "/m/02f9pk"
    description: "Lemur"
    score: 0.8270352482795715
    topicality: 0.8270352482795715
  }
  label_annotations {
    mid: "/m/0fbf1m"
    description: "Terrestrial animal"
    score: 0.7443860769271851
    topicality: 0.7443860769271851
  }
  label_annotations {
    mid: "/m/06z_nw"
    description: "Tail"
    score: 0.6934166550636292
    topicality: 0.6934166550636292
  }
  label_annotations {
    mid: "/m/0b5gs"
    description: "Branch"
    score: 0.6203985214233398
    topicality: 0.6203985214233398
  }
  label_annotations {
    mid: "/m/05s2s"
    description: "Plant"
    score: 0.585474967956543
    topicality: 0.585474967956543
  }
  label_annotations {
    mid: "/m/089v3"
    description: "Zoo"
    score: 0.5488107800483704
    topicality: 0.5488107800483704
  }
  label_annotations {
    mid: "/m/02tcwp"
    description: "Trunk"
    score: 0.5200017690658569
    topicality: 0.5200017690658569
  }
  image_properties_annotation {
    dominant_colors {
      colors {
        color {
          red: 172.0
          green: 146.0
          blue: 116.0
        }
        score: 0.24523821473121643
        pixel_fraction: 0.027533333748579025
      }
      colors {
        color {
          red: 54.0
          green: 50.0
          blue: 42.0
        }
        score: 0.10449723154306412
        pixel_fraction: 0.12893334031105042
      }
      colors {
        color {
          red: 141.0
          green: 121.0
          blue: 97.0
        }
        score: 0.1391485631465912
        pixel_fraction: 0.039133332669734955
      }
      colors {
        color {
          red: 28.0
          green: 25.0
          blue: 20.0
        }
        score: 0.08589499443769455
        pixel_fraction: 0.11506666988134384
      }
      colors {
        color {
          red: 87.0
          green: 82.0
          blue: 74.0
        }
        score: 0.0845794677734375
        pixel_fraction: 0.16113333404064178
      }
      colors {
        color {
          red: 121.0
          green: 117.0
          blue: 108.0
        }
        score: 0.05901569500565529
        pixel_fraction: 0.13379999995231628
      }
      colors {
        color {
          red: 94.0
          green: 83.0
          blue: 66.0
        }
        score: 0.049011144787073135
        pixel_fraction: 0.03946666792035103
      }
      colors {
        color {
          red: 155.0
          green: 117.0
          blue: 90.0
        }
        score: 0.04164913296699524
        pixel_fraction: 0.0023333332501351833
      }
      colors {
        color {
          red: 178.0
          green: 143.0
          blue: 102.0
        }
        score: 0.02993861958384514
        pixel_fraction: 0.0012666666880249977
      }
      colors {
        color {
          red: 61.0
          green: 51.0
          blue: 35.0
        }
        score: 0.027391711249947548
        pixel_fraction: 0.01953333243727684
      }
    }
  }
  crop_hints_annotation {
    crop_hints {
      bounding_poly {
        vertices {
          x: 2073
        }
        vertices {
          x: 4008
        }
        vertices {
          x: 4008
          y: 3455
        }
        vertices {
          x: 2073
          y: 3455
        }
      }
      confidence: 0.65625
      importance_fraction: 0.746666669845581
    }
  }
  localized_object_annotations {
    mid: "/m/0jbk"
    name: "Animal"
    score: 0.7008256912231445
    bounding_poly {
      normalized_vertices {
        x: 0.0390297956764698
        y: 0.26235100626945496
      }
      normalized_vertices {
        x: 0.8466796875
        y: 0.26235100626945496
      }
      normalized_vertices {
        x: 0.8466796875
        y: 0.9386426210403442
      }
      normalized_vertices {
        x: 0.0390297956764698
        y: 0.9386426210403442
      }
    }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620771698
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDYyMDc3MTY5OA==
created_at: 2020-04-28T18:13:48Z
updated_at: 2020-04-28T18:13:48Z
author_association: MEMBER
body:
  For face detection:
  ```
      {"type": vision.enums.Feature.Type.Type.FACE_DETECTION}
  ```
  For OCR:
  ```
      {"type": vision.enums.Feature.Type.DOCUMENT_TEXT_DETECTION}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620772190
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDYyMDc3MjE5MA==
created_at: 2020-04-28T18:14:43Z
updated_at: 2020-04-28T18:14:43Z
author_association: MEMBER
body:
  Database schema for this will require some thought. Just dumping the output into a JSON column isn't going to be flexible enough - I want to be able to FTS against labels and OCR text, and potentially query against other characteristics too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620774507
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDYyMDc3NDUwNw==
created_at: 2020-04-28T18:19:06Z
updated_at: 2020-04-28T18:19:06Z
author_association: MEMBER
body:
  The default timeout is a bit aggressive and sometimes failed for me if my resizing proxy took too long to fetch and resize the image.

  `client.annotate_image(..., timeout=3.0)` may be worth trying.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/633#issuecomment-620841496
issue_url: https://api.github.com/repos/simonw/datasette/issues/633
node_id: MDEyOklzc3VlQ29tbWVudDYyMDg0MTQ5Ng==
created_at: 2020-04-28T20:37:50Z
updated_at: 2020-04-28T20:37:50Z
author_association: NONE
body:
  Using the Heroku web interface, you can set the WEB_CONCURRENCY = 1

  ![image](https://user-images.githubusercontent.com/46165/80535319-352c8100-8966-11ea-9d4f-df2622ec8bff.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/744#issuecomment-620969639
issue_url: https://api.github.com/repos/simonw/datasette/issues/744
node_id: MDEyOklzc3VlQ29tbWVudDYyMDk2OTYzOQ==
created_at: 2020-04-29T03:23:31Z
updated_at: 2020-04-29T03:23:31Z
author_association: OWNER
body:
  Is it possible that your `/tmp` directory is on a different volume from the `template` folder? That could cause a problem with the symlinks.

  Here's the code in question:

  https://github.com/simonw/datasette/blob/89c4ddd4828623888e91a1d2cb396cba12d4e7b4/datasette/utils/__init__.py#L595-L609

  It looks to me like we were expecting an `OSError` but in your environment you got a `shutil.Error` instead. So a fix could be to capture that error as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/744#issuecomment-620970159
issue_url: https://api.github.com/repos/simonw/datasette/issues/744
node_id: MDEyOklzc3VlQ29tbWVudDYyMDk3MDE1OQ==
created_at: 2020-04-29T03:25:45Z
updated_at: 2020-04-29T03:25:45Z
author_association: OWNER
body:
  https://docs.python.org/3/library/shutil.html#shutil.copytree says that `shutil.Error` should be raised here, so I think this is a bug in the `link_or_copy_directory` function.

  I don't have an environment that can replicate this bug. @aborruso I'm going to ship a fix in a branch that you can test against.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/744#issuecomment-620971526
issue_url: https://api.github.com/repos/simonw/datasette/issues/744
node_id: MDEyOklzc3VlQ29tbWVudDYyMDk3MTUyNg==
created_at: 2020-04-29T03:32:14Z
updated_at: 2020-04-29T03:32:14Z
author_association: OWNER
body:
  @aborruso I think I have a branch with a fix - could you try it out?

  Install the new branch like this:

  ```
  pip install https://github.com/simonw/datasette/archive/issue-744.zip
  ```
  Then try running `datasette publish heroku` again.

  If this fixes your bug I'll merge that pull request!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/744#issuecomment-621008152
issue_url: https://api.github.com/repos/simonw/datasette/issues/744
node_id: MDEyOklzc3VlQ29tbWVudDYyMTAwODE1Mg==
created_at: 2020-04-29T06:05:02Z
updated_at: 2020-04-29T06:05:02Z
author_association: NONE
body:
  Hi @simonw , I have installed it and I have the below errors.

  > Is it possible that your /tmp directory is on a different volume from the template folder? That could cause a problem with the symlinks.

  No, /tmp folder is in the same volume. 

  Thank you

  ```
  Traceback (most recent call last):
    File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/utils/__init__.py", line 607, in link_or_copy_directory
      shutil.copytree(src, dst, copy_function=os.link)
    File "/usr/lib/python3.7/shutil.py", line 365, in copytree
      raise Error(errors)
  shutil.Error: [('/var/youtubeComunePalermo/processing/./template/base.html', '/tmp/tmpcqv_1i5d/templates/base.html', "[Errno 18] Invalid cross-device link: '/var/youtubeComunePalermo/processing/./template/base.html' -> '/tmp/tmpcqv_1i5d/templates/base.html'"), ('/var/youtubeComunePalermo/processing/./template/index.html', '/tmp/tmpcqv_1i5d/templates/index.html', "[Errno 18] Invalid cross-device link: '/var/youtubeComunePalermo/processing/./template/index.html' -> '/tmp/tmpcqv_1i5d/templates/index.html'")]

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/aborruso/.local/bin/datasette", line 8, in <module>
      sys.exit(cli())
    File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 610, in invoke    return callback(*args, **kwargs)
    File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py", line 103, in heroku
      extra_metadata,
    File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
      return next(self.gen)
    File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py", line 191, in temporary_heroku_directory
      os.path.join(tmp.name, "templates"),
    File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/utils/__init__.py", line 609, in link_or_copy_directory
      shutil.copytree(src, dst)
    File "/usr/lib/python3.7/shutil.py", line 321, in copytree
      os.makedirs(dst)
    File "/usr/lib/python3.7/os.py", line 221, in makedirs
      mkdir(name, mode)
  FileExistsError: [Errno 17] File exists: '/tmp/tmpcqv_1i5d/templates'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/744#issuecomment-621011554
issue_url: https://api.github.com/repos/simonw/datasette/issues/744
node_id: MDEyOklzc3VlQ29tbWVudDYyMTAxMTU1NA==
created_at: 2020-04-29T06:17:26Z
updated_at: 2020-04-29T06:17:26Z
author_association: NONE
body:
  A stupid note: I have no `tmpcqv_1i5d` folder in in `/tmp`.

  It seems to me that it does not create any `/tmp/tmpcqv_1i5d/templates` folder (or other name folder, inside /tmp)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/746#issuecomment-621027374
issue_url: https://api.github.com/repos/simonw/datasette/issues/746
node_id: MDEyOklzc3VlQ29tbWVudDYyMTAyNzM3NA==
created_at: 2020-04-29T07:07:23Z
updated_at: 2020-04-29T07:07:23Z
author_association: OWNER
body:
  It looks like this isn't the right fix: https://github.com/simonw/datasette/issues/744#issuecomment-621008152

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/744#issuecomment-621030783
issue_url: https://api.github.com/repos/simonw/datasette/issues/744
node_id: MDEyOklzc3VlQ29tbWVudDYyMTAzMDc4Mw==
created_at: 2020-04-29T07:16:27Z
updated_at: 2020-04-29T07:16:27Z
author_association: NONE
body:
  Hi @simonw it's debian as Windows Subsystem for Linux 

  ```
  PRETTY_NAME="Pengwin"
  NAME="Pengwin"
  VERSION_ID="10"
  VERSION="10 (buster)"
  ID=debian
  ID_LIKE=debian
  HOME_URL="https://github.com/whitewaterfoundry/Pengwin"
  SUPPORT_URL="https://github.com/whitewaterfoundry/Pengwin"
  BUG_REPORT_URL="https://github.com/whitewaterfoundry/Pengwin"
  VERSION_CODENAME=buster
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/698#issuecomment-621036032
issue_url: https://api.github.com/repos/simonw/datasette/issues/698
node_id: MDEyOklzc3VlQ29tbWVudDYyMTAzNjAzMg==
created_at: 2020-04-29T07:29:52Z
updated_at: 2020-04-29T07:29:52Z
author_association: OWNER
body:
  What should happen when a query has been successfully executed?

  That depends on the query. Some queries may wish to redirect to another page. Other queries might want to show a custom message.

  There should at least be a default message saying the query has been executed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/698#issuecomment-621037724
issue_url: https://api.github.com/repos/simonw/datasette/issues/698
node_id: MDEyOklzc3VlQ29tbWVudDYyMTAzNzcyNA==
created_at: 2020-04-29T07:34:02Z
updated_at: 2020-04-29T07:34:02Z
author_association: OWNER
body:
  Concept for displaying a success message:

  <img width="662" alt="fixtures__compound_three_primary_keys__1_001_rows" src="https://user-images.githubusercontent.com/9599/80571201-0a1b4f00-89b1-11ea-8688-2e3e4304e475.png">
  CSS:

  ```css
  .success {
      padding: 1em;
      border: 1px solid green;
      background-color: #c7fbc7;
  }

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/698#issuecomment-621041812
issue_url: https://api.github.com/repos/simonw/datasette/issues/698
node_id: MDEyOklzc3VlQ29tbWVudDYyMTA0MTgxMg==
created_at: 2020-04-29T07:42:48Z
updated_at: 2020-04-29T07:42:48Z
author_association: OWNER
body:
  Need to figure out what the `.json` mode for this looks like - and if there's a `.csv` mode (I think not).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/747#issuecomment-621948898
issue_url: https://api.github.com/repos/simonw/datasette/issues/747
node_id: MDEyOklzc3VlQ29tbWVudDYyMTk0ODg5OA==
created_at: 2020-04-30T16:05:50Z
updated_at: 2020-04-30T16:05:50Z
author_association: OWNER
body:
  Relevant code:

  https://github.com/simonw/datasette/blob/e37f4077c0f1cd09d4102213d4e2a512af471b8d/datasette/app.py#L206-L208

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/747#issuecomment-622003376
issue_url: https://api.github.com/repos/simonw/datasette/issues/747
node_id: MDEyOklzc3VlQ29tbWVudDYyMjAwMzM3Ng==
created_at: 2020-04-30T17:45:32Z
updated_at: 2020-04-30T17:45:32Z
author_association: OWNER
body:
  I can use this function to load the JSON-or-YAML:
  https://github.com/simonw/datasette/blob/d349d57cdf3d577afb62bdf784af342a4d5be660/datasette/utils/__init__.py#L798-L806

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133298
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34
node_id: MDEyOklzc3VlQ29tbWVudDYyMjEzMzI5OA==
created_at: 2020-04-30T21:48:24Z
updated_at: 2020-04-30T21:48:24Z
author_association: MEMBER
body:
  Unfortunately it's not available through any GitHub API - I managed to figure out how to get dependencies, but I need dependents. https://github.com/simonw/til/blob/master/github/dependencies-graphql-api.md

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133422
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34
node_id: MDEyOklzc3VlQ29tbWVudDYyMjEzMzQyMg==
created_at: 2020-04-30T21:48:39Z
updated_at: 2020-04-30T21:48:39Z
author_association: MEMBER
body:
  It looks like the only option is to scrape them. I'll do that and then replace with an API as soon as one becomes available.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133775
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34
node_id: MDEyOklzc3VlQ29tbWVudDYyMjEzMzc3NQ==
created_at: 2020-04-30T21:49:27Z
updated_at: 2020-04-30T21:49:27Z
author_association: MEMBER
body:
  Proposed command:

      github-to-sqlite scrape-dependents github.db simonw/datasette

  I'll pull full details of the scraped repos from the regular API. I'll also record when they were "first seen" by the command.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622135654
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34
node_id: MDEyOklzc3VlQ29tbWVudDYyMjEzNTY1NA==
created_at: 2020-04-30T21:53:44Z
updated_at: 2020-04-30T21:56:06Z
author_association: MEMBER
body:
  I think this is the neatest scraping pattern:
  ```python
  [a["href"].lstrip("/") for a in soup.select("a[data-hovercard-type=repository]")]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622136585
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34
node_id: MDEyOklzc3VlQ29tbWVudDYyMjEzNjU4NQ==
created_at: 2020-04-30T21:55:51Z
updated_at: 2020-04-30T21:55:51Z
author_association: MEMBER
body:
  And to find the "Next" pagination link:
  ```python
  soup.select(".paginate-container")[0].find("a", text="Next")
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622162835
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34
node_id: MDEyOklzc3VlQ29tbWVudDYyMjE2MjgzNQ==
created_at: 2020-04-30T22:59:26Z
updated_at: 2020-04-30T22:59:26Z
author_association: MEMBER
body:
  Documentation: https://github.com/dogsheep/github-to-sqlite/blob/c9f48404481882e8b3af06f35e4801a80ac79ed6/README.md#scraping-dependents-for-a-repository

reactions:
  {"total_count": 2, "+1": 0, "-1": 0, "laugh": 0, "hooray": 2, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622171097
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33
node_id: MDEyOklzc3VlQ29tbWVudDYyMjE3MTA5Nw==
created_at: 2020-04-30T23:22:45Z
updated_at: 2020-04-30T23:23:57Z
author_association: MEMBER
body:
  The `auth.json` mechanism this uses is standard across all of the other Dogsheep tools - it's actually designed so you can have one `auth.json` with a bunch of different credentials for different tools:

  ```json
  {
      "goodreads_personal_token": "...",
      "goodreads_user_id": "...",
      "github_personal_token": "...",
      "pocket_consumer_key": "...",
      "pocket_username": "...",
      "pocket_access_token": "..."
  }
  ```
  But... `github-to-sqlite` does feel like it deserves a special case here, since it's such a good fit for running inside of GitHub Actions - which even provide a `GITHUB_TOKEN` for you to use!

  So I don't think it will harm the family of tools too much if this has an environment variable alternative to the `-a` file.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622213950
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35
node_id: MDEyOklzc3VlQ29tbWVudDYyMjIxMzk1MA==
created_at: 2020-05-01T02:09:04Z
updated_at: 2020-05-01T02:09:04Z
author_association: MEMBER
body:
  It sped up this query a lot - 2.5s down to 300ms:
  ```sql
  select
    repos.full_name,
    json_object(
      'href', 'https://github.com/' || repos.full_name || '/issues/' || issues.number,
      'label', '#' || issues.number
    ) as issue,
    issues.title,
    users.login,
    users.id,
    issues.state,
    issues.locked,
    issues.assignee,
    issues.milestone,
    issues.comments,
    issues.created_at,
    issues.updated_at,
    issues.closed_at,
    issues.author_association,
    issues.pull_request,
    issues.repo,
    issues.type
  from
    issues
    join repos on repos.id = issues.repo
    join users on issues.user = users.id
  where
    issues.state = 'open'
    and issues.user not in (9599, 27856297)
    and not exists (
      select
        id
      from
        issue_comments
      where
        issue_comments.user = 9599
        and issues.id = issue_comments.issue
    )
  order by
    issues.updated_at desc;
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622214262
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35
node_id: MDEyOklzc3VlQ29tbWVudDYyMjIxNDI2Mg==
created_at: 2020-05-01T02:10:32Z
updated_at: 2020-05-01T02:11:19Z
author_association: MEMBER
body:
  This sped that query up even more - down to 4ms.
  ```sql
  create index issue_comments_issue on issue_comments(issue);
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622279374
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33
node_id: MDEyOklzc3VlQ29tbWVudDYyMjI3OTM3NA==
created_at: 2020-05-01T07:12:47Z
updated_at: 2020-05-01T07:12:47Z
author_association: NONE
body:
  I also go it working with:

  ```yaml
  run: echo ${{ secrets.github_token }} | github-to-sqlite auth
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/749#issuecomment-622450636
issue_url: https://api.github.com/repos/simonw/datasette/issues/749
node_id: MDEyOklzc3VlQ29tbWVudDYyMjQ1MDYzNg==
created_at: 2020-05-01T16:08:46Z
updated_at: 2020-05-01T16:08:46Z
author_association: OWNER
body:
  Proposed solution: on Cloud Run don't show the "download database" link if the database file is larger than 32MB.

  I can do this with a new config setting, `max_db_mb`, which is automatically set by the `publish cloudrun` command.

  This is consistent with the existing `max_csv_mb` setting: https://datasette.readthedocs.io/en/stable/config.html#max-csv-mb

  I should set `max_csv_mb` to 32MB on Cloud Run deploys as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622461537
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37
node_id: MDEyOklzc3VlQ29tbWVudDYyMjQ2MTUzNw==
created_at: 2020-05-01T16:35:40Z
updated_at: 2020-05-01T16:35:40Z
author_association: MEMBER
body:
  This will check if the view exists and has the exact same matching definition as the one we want. If it doesn't, we will drop it (if it exists) and recreate it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622561585
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/103
node_id: MDEyOklzc3VlQ29tbWVudDYyMjU2MTU4NQ==
created_at: 2020-05-01T20:46:50Z
updated_at: 2020-05-01T20:46:50Z
author_association: OWNER
body:
  The varying number of columns thing is interesting - I don't think the tests cover that case much if at all.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622561944
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/103
node_id: MDEyOklzc3VlQ29tbWVudDYyMjU2MTk0NA==
created_at: 2020-05-01T20:47:51Z
updated_at: 2020-05-01T20:47:51Z
author_association: OWNER
body:
  Yup we only take the number of columns in the first record into account at the moment:
  https://github.com/simonw/sqlite-utils/blob/d56029549acae0b0ea94c5a0f783e3b3895d9218/sqlite_utils/db.py#L1007-L1016

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622563059
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/103
node_id: MDEyOklzc3VlQ29tbWVudDYyMjU2MzA1OQ==
created_at: 2020-05-01T20:51:01Z
updated_at: 2020-05-01T20:51:01Z
author_association: OWNER
body:
  I'm not sure what to do about this.

  I was thinking the solution would be to look at ALL of the rows in a batch before deciding on the maximum number of columns, but that doesn't work because we calculate batch size based on the number of columns!

  I think my recommendation here is to manually pass a `batch_size=` argument to `.insert_all()` if you run into this error.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622563188
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/103
node_id: MDEyOklzc3VlQ29tbWVudDYyMjU2MzE4OA==
created_at: 2020-05-01T20:51:24Z
updated_at: 2020-05-01T20:51:29Z
author_association: OWNER
body:
  Hopefully anyone who runs into this problem in the future will search for and find this issue thread!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622565276
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/103
node_id: MDEyOklzc3VlQ29tbWVudDYyMjU2NTI3Ng==
created_at: 2020-05-01T20:57:16Z
updated_at: 2020-05-01T20:57:16Z
author_association: OWNER
body:
  I'm reconsidering this: I think this is going to happen ANY time someone has at least one row that is wider than the first row. So at the very least I should show a more understandable error message.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622584433
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/103
node_id: MDEyOklzc3VlQ29tbWVudDYyMjU4NDQzMw==
created_at: 2020-05-01T21:57:52Z
updated_at: 2020-05-01T21:57:52Z
author_association: OWNER
body:
  @b0b5h4rp13 I'm having trouble creating a test that triggers this bug. Could you share a chunk of code that replicates what you're seeing here?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622587177
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/103
node_id: MDEyOklzc3VlQ29tbWVudDYyMjU4NzE3Nw==
created_at: 2020-05-01T22:07:51Z
updated_at: 2020-05-01T22:07:51Z
author_association: OWNER
body:
  This is my failed attempt to recreate the bug (plus some extra debugging output):
  ```diff
   % git diff
  diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
  index dd49d5c..ea42aea 100644
  --- a/sqlite_utils/db.py
  +++ b/sqlite_utils/db.py
  @@ -1013,7 +1013,11 @@ class Table(Queryable):
           assert (
               num_columns <= SQLITE_MAX_VARS
           ), "Rows can have a maximum of {} columns".format(SQLITE_MAX_VARS)
  +        print("default batch_size = ", batch_size)
           batch_size = max(1, min(batch_size, SQLITE_MAX_VARS // num_columns))
  +        print("new batch_size = {},num_columns = {}, MAX_VARS // num_columns = {}".format(
  +            batch_size, num_columns, SQLITE_MAX_VARS // num_columns
  +        ))
           self.last_rowid = None
           self.last_pk = None
           for chunk in chunks(itertools.chain([first_record], records), batch_size):
  @@ -1124,6 +1128,9 @@ class Table(Queryable):
                   )
                   flat_values = list(itertools.chain(*values))
                   queries_and_params = [(sql, flat_values)]
  +                print(sql.count("?"), len(flat_values))
  +
  +            # print(json.dumps(queries_and_params, indent=4))
   
               with self.db.conn:
                   for query, params in queries_and_params:
  diff --git a/tests/test_create.py b/tests/test_create.py
  index 5290cd8..52940df 100644
  --- a/tests/test_create.py
  +++ b/tests/test_create.py
  @@ -853,3 +853,33 @@ def test_create_with_nested_bytes(fresh_db):
       record = {"id": 1, "data": {"foo": b"bytes"}}
       fresh_db["t"].insert(record)
       assert [{"id": 1, "data": '{"foo": "b\'bytes\'"}'}] == list(fresh_db["t"].rows)
  +
  +
  +def test_create_throws_useful_error_with_increasing_number_of_columns(fresh_db):
  +    # https://github.com/simonw/sqlite-utils/issues/103
  +    def rows():
  +        yield {"name": 0}
  +        for i in range(1, 1001):
  +            yield {
  +                "name": i,
  +                "age": i,
  +                "size": i,
  +                "name2": i,
  +                "age2": i,
  +                "size2": i,
  +                "name3": i,
  +                "age3": i,
  +                "size3": i,
  +                "name4": i,
  +                "age4": i,
  +                "size4": i,
  +                "name5": i,
  +                "age5": i,
  +                "size5": i,
  +                "name6": i,
  +                "age6": i,
  +                "size6": i,
  +            }
  +
  +    fresh_db["t"].insert_all(rows())
  +    assert 1001 == fresh_db["t"].count
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622599528
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/103
node_id: MDEyOklzc3VlQ29tbWVudDYyMjU5OTUyOA==
created_at: 2020-05-01T22:49:12Z
updated_at: 2020-05-02T11:15:44Z
author_association: CONTRIBUTOR
body:
  With  SQLITE_MAX_VARS = 999, or even 899, This hits the problem with the batch rows causing a  overflow (works fine if SQLITE_MAX_VARS = 799).

  p.s. I have tried a few list of dicts to sqlite modules and this was the easiest to use/understand

  ------------- file begins ------------------
  import sqlite_utils as su


  data = [
  {'tickerId': 913324382, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'CONSTELLATION B', 'symbol': 'STZ B', 'disSymbol': 'STZ-B', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'status': 'D', 'close': '163.13', 'change': '6.46', 'changeRatio': '0.0412', 'marketValue': '31180699895.63', 'volume': '417', 'turnoverRate': '0.0000'},
  {'tickerId': 913323791, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Molina Health', 'symbol': 'MOH', 'disSymbol': 'MOH', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '173.25', 'change': '9.28', 'changeRatio': '0.0566', 'pPrice': '173.25', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '10520341695.50', 'volume': '1281557', 'turnoverRate': '0.0202'},
  {'tickerId': 913257501, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Seattle Genetics', 'symbol': 'SGEN', 'disSymbol': 'SGEN', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '145.64', 'change': '8.41', 'changeRatio': '0.0613', 'pPrice': '146.45', 'pChange': '0.8100', 'pChRatio': '0.0056', 'marketValue': '25117961347.60', 'volume': '2791411', 'turnoverRate': '0.0162'},
  {'tickerId': 925381971, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Bandwidth', 'symbol': 'BAND', 'disSymbol': 'BAND', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '89.22', 'change': '7.66', 'changeRatio': '0.0939', 'pPrice': '89.00', 'pChange': '-0.2200', 'pChRatio': '-0.0025', 'marketValue': '2100025474.98', 'volume': '1508629', 'turnoverRate': '0.0641'},
  {'tickerId': 913323935, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Magellan Health', 'symbol': 'MGLN', 'disSymbol': 'MGLN', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '68.00', 'change': '7.27', 'changeRatio': '0.1197', 'pPrice': '68.00', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '1697894040.00', 'volume': '448919', 'turnoverRate': '0.0180'},
  {'tickerId': 913254854, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'On Assignment', 'symbol': 'ASGN', 'disSymbol': 'ASGN', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '53.04', 'change': '6.59', 'changeRatio': '0.1419', 'pPrice': '53.04', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '2811120000.00', 'volume': '1339771', 'turnoverRate': '0.0253'},
  {'tickerId': 913255732, 'exchangeId': 95, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Arcturus', 'symbol': 'ARCT', 'disSymbol': 'ARCT', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NMS', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '40.86', 'change': '6.36', 'changeRatio': '0.1843', 'pPrice': '42.60', 'pChange': '1.740', 'pChRatio': '0.0426', 'marketValue': '812021444.46', 'volume': '1577508', 'turnoverRate': '0.0794'},
  {'tickerId': 913256616, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'DexCom', 'symbol': 'DXCM', 'disSymbol': 'DXCM', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '341.52', 'change': '6.32', 'changeRatio': '0.0189', 'pPrice': '340.00', 'pChange': '-1.5200', 'pChRatio': '-0.0045', 'marketValue': '31522296000.00', 'volume': '1008849', 'turnoverRate': '0.0109'},
  {'tickerId': 913255108, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Clorox', 'symbol': 'CLX', 'disSymbol': 'CLX', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '192.71', 'change': '6.27', 'changeRatio': '0.0336', 'pPrice': '192.95', 'pChange': '0.2400', 'pChRatio': '0.0012', 'marketValue': '24185773318.28', 'volume': '4996414', 'turnoverRate': '0.0398'},
  {'tickerId': 925314627, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'FRANCO NEVADA', 'symbol': 'FNV', 'disSymbol': 'FNV', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '137.85', 'change': '5.64', 'changeRatio': '0.0427', 'pPrice': '138.50', 'pChange': '0.6500', 'pChRatio': '0.0047', 'marketValue': '26110405326.30', 'volume': '1047688', 'turnoverRate': '0.0055'},
  {'tickerId': 913254955, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Aon Plc', 'symbol': 'AON', 'disSymbol': 'AON', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '178.21', 'change': '5.54', 'changeRatio': '0.0321', 'pPrice': '178.21', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '41181209117.22', 'volume': '2026234', 'turnoverRate': '0.0088'},
  {'tickerId': 913324105, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Willis Towers', 'symbol': 'WLTW', 'disSymbol': 'WLTW', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '183.34', 'change': '5.05', 'changeRatio': '0.0283', 'pPrice': '183.34', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '23597461124.96', 'volume': '968943', 'turnoverRate': '0.0075'},
  {'tickerId': 913254759, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'TELADOC HEALTH', 'symbol': 'TDOC', 'disSymbol': 'TDOC', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '169.43', 'change': '4.84', 'changeRatio': '0.0294', 'pPrice': '168.88', 'pChange': '-0.5500', 'pChRatio': '-0.0032', 'marketValue': '12614616858.38', 'volume': '2628946', 'turnoverRate': '0.0353'},
  {'tickerId': 913255222, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Emergent Bio', 'symbol': 'EBS', 'disSymbol': 'EBS', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '78.70', 'change': '4.75', 'changeRatio': '0.0642', 'pPrice': '78.40', 'pChange': '-0.3000', 'pChRatio': '-0.0038', 'marketValue': '4113368277.10', 'volume': '783804', 'turnoverRate': '0.0150'},
  {'tickerId': 913323443, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Pool', 'symbol': 'POOL', 'disSymbol': 'POOL', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '216.02', 'change': '4.36', 'changeRatio': '0.0206', 'pPrice': '216.02', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '8696077573.82', 'volume': '310837', 'turnoverRate': '0.0077'},
  {'tickerId': 913257075, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Masimo', 'symbol': 'MASI', 'disSymbol': 'MASI', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '218.00', 'change': '4.09', 'changeRatio': '0.0191', 'pPrice': '217.00', 'pChange': '-1.0000', 'pChRatio': '-0.0046', 'marketValue': '11797070000.00', 'volume': '542131', 'turnoverRate': '0.0100'},
  {'tickerId': 913253761, 'exchangeId': 10, 'type': 2, 'secType': [62], 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Pope Resources', 'symbol': 'POPE', 'disSymbol': 'POPE', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NAS', 'listStatus': 1, 'template': 'stock', 'status': 'D', 'close': '101.05', 'change': '3.95', 'changeRatio': '0.0407', 'pPrice': '99.90', 'pChange': '2.800', 'pChRatio': '0.0288', 'marketValue': '447370075.75', 'volume': '33138', 'turnoverRate': '0.0075'},
  {'tickerId': 913323560, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Seneca Foods', 'symbol': 'SENEB', 'disSymbol': 'SENEB', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'status': 'D', 'close': '40.04', 'change': '3.84', 'changeRatio': '0.1061', 'marketValue': '347950039.71', 'volume': '501'},
  {'tickerId': 913324274, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Resmed', 'symbol': 'RMD', 'disSymbol': 'RMD', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '159.07', 'change': '3.75', 'changeRatio': '0.0241', 'pPrice': '159.07', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '23004217759.29', 'volume': '1267075', 'turnoverRate': '0.0088'},
  {'tickerId': 913323736, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Vertex Pharms', 'symbol': 'VRTX', 'disSymbol': 'VRTX', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '254.90', 'change': '3.70', 'changeRatio': '0.0147', 'pPrice': '255.00', 'pChange': '0.1000', 'pChRatio': '0.0004', 'marketValue': '66062980780.10', 'volume': '1939843', 'turnoverRate': '0.0075'},
  {'tickerId': 913323767, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'MCCORMICK VTG', 'symbol': 'MKC V', 'disSymbol': 'MKC-V', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'status': 'D', 'close': '159.99', 'change': '3.42', 'changeRatio': '0.0218', 'marketValue': '21262671000.00', 'volume': '432', 'turnoverRate': '0.0000'},
  {'tickerId': 950118595, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'ZOOM VIDEO', 'symbol': 'ZM', 'disSymbol': 'ZM', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '138.56', 'change': '3.39', 'changeRatio': '0.0251', 'pPrice': '138.99', 'pChange': '0.4300', 'pChRatio': '0.0031', 'marketValue': '38620532420.16', 'volume': '13786017', 'turnoverRate': '0.0495'},
  {'tickerId': 916040738, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'WHEATON PRECIOUS', 'symbol': 'WPM', 'disSymbol': 'WPM', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '41.10', 'change': '3.34', 'changeRatio': '0.0885', 'pPrice': '41.09', 'pChange': '-0.0100', 'pChRatio': '-0.0002', 'marketValue': '18404536146.30', 'volume': '5019137', 'turnoverRate': '0.0112'},
  {'tickerId': 913257174, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Royal Gold', 'symbol': 'RGLD', 'disSymbol': 'RGLD', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '125.86', 'change': '3.33', 'changeRatio': '0.0272', 'pPrice': '125.86', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '8253015011.08', 'volume': '853473', 'turnoverRate': '0.0130'},
  {'tickerId': 913254394, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Fortune Brand', 'symbol': 'FBHS', 'disSymbol': 'FBHS', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '51.50', 'change': '3.30', 'changeRatio': '0.0685', 'pPrice': '51.50', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '7194870278.50', 'volume': '3004021', 'turnoverRate': '0.0214'},
  {'tickerId': 913323312, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Liberty Global', 'symbol': 'LBTYK', 'disSymbol': 'LBTYK', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '21.49', 'change': '3.18', 'changeRatio': '0.1737', 'pPrice': '21.48', 'pChange': '-0.0100', 'pChRatio': '-0.0005', 'marketValue': '13594662302.41', 'volume': '19980228', 'turnoverRate': '0.0315'},
  {'tickerId': 913323882, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Preformed Line', 'symbol': 'PLPC', 'disSymbol': 'PLPC', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'status': 'D', 'close': '52.82', 'change': '3.14', 'changeRatio': '0.0632', 'pPrice': '52.10', 'pChange': '-0.7200', 'pChRatio': '-0.0136', 'marketValue': '264979981.20', 'volume': '9305', 'turnoverRate': '0.0018'},
  {'tickerId': 913323248, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Discovery', 'symbol': 'DISCB', 'disSymbol': 'DISCB', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'status': 'A', 'close': '57.95', 'change': '23.63', 'changeRatio': '0.6884', 'pPrice': '54.26', 'pChange': '-3.6900', 'pChRatio': '-0.0637', 'marketValue': '29362894177.95', 'volume': '218305', 'turnoverRate': '0.0004'},
  {'tickerId': 913323930, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'MercadoLibre', 'symbol': 'MELI', 'disSymbol': 'MELI', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '605.52', 'change': '22.01', 'changeRatio': '0.0377', 'pPrice': '603.69', 'pChange': '-1.8300', 'pChRatio': '-0.0030', 'marketValue': '30226598045.28', 'volume': '699008', 'turnoverRate': '0.0140'},
  {'tickerId': 913257170, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Liberty Global', 'symbol': 'LBTYA', 'disSymbol': 'LBTYA', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '22.28', 'change': '2.86', 'changeRatio': '0.1473', 'pPrice': '22.29', 'pChange': '0.0100', 'pChRatio': '0.0004', 'marketValue': '14094419548.52', 'volume': '10534672', 'turnoverRate': '0.0167'},
  {'tickerId': 913303991, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Liberty Brodband', 'symbol': 'LBRDK', 'disSymbol': 'LBRDK', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '125.44', 'change': '2.76', 'changeRatio': '0.0225', 'pPrice': '125.44', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '22817900904.96', 'volume': '926177', 'turnoverRate': '0.0042'},
  {'tickerId': 913257082, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Helen of Troy', 'symbol': 'HELE', 'disSymbol': 'HELE', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '167.04', 'change': '2.76', 'changeRatio': '0.0168', 'pPrice': '167.04', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '4216707982.08', 'volume': '341465', 'turnoverRate': '0.0135'},
  {'tickerId': 913256458, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Forrester', 'symbol': 'FORR', 'disSymbol': 'FORR', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '33.88', 'change': '2.58', 'changeRatio': '0.0824', 'marketValue': '635419400.00', 'volume': '85115', 'turnoverRate': '0.0045'},
  {'tickerId': 950158952, 'exchangeId': 95, 'type': 2, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'LYRA THERAPEUTICS, INC.', 'symbol': 'LYRA', 'disSymbol': 'LYRA', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NMS', 'listStatus': 1, 'template': 'ipo', 'status': 'A', 'close': '18.56', 'change': '2.56', 'changeRatio': '0.1600', 'pPrice': '18.96', 'pChange': '0.4000', 'pChRatio': '0.0216', 'marketValue': '229705575.68', 'volume': '1738472', 'turnoverRate': '0.1405'},
  {'tickerId': 913257570, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Bio-Techne', 'symbol': 'TECH', 'disSymbol': 'TECH', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '227.54', 'change': '2.54', 'changeRatio': '0.0113', 'pPrice': '227.54', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '8726538309.18', 'volume': '497006', 'turnoverRate': '0.0130'},
  {'tickerId': 913323246, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Bel Fuse', 'symbol': 'BELFB', 'disSymbol': 'BELFB', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '9.99', 'change': '2.53', 'changeRatio': '0.3391', 'pPrice': '9.75', 'pChange': '-0.2400', 'pChRatio': '-0.0240', 'marketValue': '122562454.86', 'volume': '177634', 'turnoverRate': '0.0145'},
  {'tickerId': 916040647, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Agnico Eagle', 'symbol': 'AEM', 'disSymbol': 'AEM', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '61.20', 'change': '2.52', 'changeRatio': '0.0429', 'pPrice': '61.10', 'pChange': '-0.1000', 'pChRatio': '-0.0016', 'marketValue': '14739911553.60', 'volume': '2820765', 'turnoverRate': '0.0117'},
  {'tickerId': 913303768, 'exchangeId': 12, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'CHASE CORP', 'symbol': 'CCF', 'disSymbol': 'CCF', 'disExchangeCode': 'AMEX', 'exchangeCode': 'ASE', 'listStatus': 1, 'template': 'stock', 'status': 'D', 'close': '96.71', 'change': '2.45', 'changeRatio': '0.0260', 'marketValue': '916799598.60', 'volume': '29229', 'turnoverRate': '0.0031'},
  {'tickerId': 913324557, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Allergan', 'symbol': 'AGN', 'disSymbol': 'AGN', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '189.74', 'change': '2.40', 'changeRatio': '0.0128', 'pPrice': '189.76', 'pChange': '0.0200', 'pChRatio': '0.0001', 'marketValue': '62424842326.10', 'volume': '5787032', 'turnoverRate': '0.0176'},
  {'tickerId': 913324566, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'West Pharm Svc', 'symbol': 'WST', 'disSymbol': 'WST', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '191.64', 'change': '2.38', 'changeRatio': '0.0126', 'pPrice': '191.64', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '14078267117.08', 'volume': '352460', 'turnoverRate': '0.0042'}
  ]

  db = su.Database(f"overnight hold.db" )
  db['active'].insert_all(data)

  --------------- file ends ----------------------

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-622980203
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10
node_id: MDEyOklzc3VlQ29tbWVudDYyMjk4MDIwMw==
created_at: 2020-05-02T16:34:29Z
updated_at: 2020-05-02T16:34:29Z
author_association: MEMBER
body:
  Fixed definition:
  ```sql
  select
    stars.starred_at,
    starring_user.login as starred_by,
    repos.*
  from
    repos
    join stars on repos.id = stars.repo
    join users as starring_user on stars.user = starring_user.id
    join users on repos.owner = users.id
  order by
    starred_at desc;
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622982667
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35
node_id: MDEyOklzc3VlQ29tbWVudDYyMjk4MjY2Nw==
created_at: 2020-05-02T16:52:53Z
updated_at: 2020-05-02T16:52:53Z
author_association: MEMBER
body:
  Easiest option: use `db.index_foreign_keys()`: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#adding-indexes-for-all-foreign-keys

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622989874
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12
node_id: MDEyOklzc3VlQ29tbWVudDYyMjk4OTg3NA==
created_at: 2020-05-02T17:46:14Z
updated_at: 2020-05-02T17:46:14Z
author_association: MEMBER
body:
  Without the rowid column facet by topics breaks: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet=repo&_facet_array=topics&topics__arraycontains=datasette-io

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-622990947
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDYyMjk5MDk0Nw==
created_at: 2020-05-02T17:54:16Z
updated_at: 2020-05-02T17:54:16Z
author_association: MEMBER
body:
  I could add that window function query as a view, but only if the detected version of SQLite supports window functions.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622997410
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33
node_id: MDEyOklzc3VlQ29tbWVudDYyMjk5NzQxMA==
created_at: 2020-05-02T18:46:10Z
updated_at: 2020-05-02T18:46:10Z
author_association: MEMBER
body:
  Documented here: https://github.com/dogsheep/github-to-sqlite/blob/10fb34de41aaa35681f08b5991540d65bfcf2e2e/README.md#authentication

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/750#issuecomment-622998529
issue_url: https://api.github.com/repos/simonw/datasette/issues/750
node_id: MDEyOklzc3VlQ29tbWVudDYyMjk5ODUyOQ==
created_at: 2020-05-02T18:56:00Z
updated_at: 2020-05-02T18:56:00Z
author_association: OWNER
body:
  So that I can do this: https://github-to-sqlite.dogsheep.net/github/dependent_repos?_where=dependent%20not%20like%20%27%simonw/%%27

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622998813
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12
node_id: MDEyOklzc3VlQ29tbWVudDYyMjk5ODgxMw==
created_at: 2020-05-02T18:58:17Z
updated_at: 2020-05-02T18:58:17Z
author_association: MEMBER
body:
  Faceting works now: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet_array=topics&topics__arraycontains=datasette-io&topics__arraycontains=sqlite&_facet=repo#facet-repo

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/750#issuecomment-622999623
issue_url: https://api.github.com/repos/simonw/datasette/issues/750
node_id: MDEyOklzc3VlQ29tbWVudDYyMjk5OTYyMw==
created_at: 2020-05-02T19:05:07Z
updated_at: 2020-05-02T19:05:07Z
author_association: OWNER
body:
  <img width="672" alt="data__names__5_rows_where_where_name_not_like__Sim___sorted_by_rowid" src="https://user-images.githubusercontent.com/9599/80873394-116e8100-8c6d-11ea-9814-5ae66d3ba869.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/750#issuecomment-623000252
issue_url: https://api.github.com/repos/simonw/datasette/issues/750
node_id: MDEyOklzc3VlQ29tbWVudDYyMzAwMDI1Mg==
created_at: 2020-05-02T19:10:44Z
updated_at: 2020-05-02T19:10:44Z
author_association: OWNER
body:
  Demo: https://latest.datasette.io/fixtures/roadside_attractions?name__notlike=%25museum%25

  Docs: https://datasette.readthedocs.io/en/latest/json_api.html#column-filter-arguments

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623000814
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDYyMzAwMDgxNA==
created_at: 2020-05-02T19:15:23Z
updated_at: 2020-05-02T19:15:23Z
author_association: MEMBER
body:
  I'm not going to do the `--sql` bit just yet. I have patterns for working around this for other commands which are working fine:

  https://github.com/dogsheep/github-to-sqlite/blob/d00a53061556dc403c166b443d141c4e1adbd64a/.github/workflows/deploy-demo.yml#L53-L70

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623004836
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDYyMzAwNDgzNg==
created_at: 2020-05-02T19:49:51Z
updated_at: 2020-05-02T19:49:51Z
author_association: MEMBER
body:
  Alternative pattern:
  ```
  sqlite-utils releases.db 'select full_name from repos' --csv --no-headers \
    | tr -d '\r' \
    | xargs github-to-sqlite stargazers stars.db
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623007441
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDYyMzAwNzQ0MQ==
created_at: 2020-05-02T20:13:37Z
updated_at: 2020-05-02T20:13:37Z
author_association: MEMBER
body:
  Datasette cumulative stars over time: https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++yyyymmdd%2C%0D%0A++sum%28n%29+over+%28%0D%0A++++order+by%0D%0A++++++yyyymmdd+rows+unbounded+preceding%0D%0A++%29+as+cumulative_count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++%29

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623010272
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDYyMzAxMDI3Mg==
created_at: 2020-05-02T20:39:14Z
updated_at: 2020-05-02T20:39:14Z
author_association: MEMBER
body:
  Graph of cumulative stars for Datasette over time: https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++yyyymmdd%2C%0D%0A++sum%28n%29+over+%28%0D%0A++++order+by%0D%0A++++++yyyymmdd+rows+unbounded+preceding%0D%0A++%29+as+cumulative_count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++%29#g.mark=line&g.x_column=yyyymmdd&g.x_type=temporal&g.y_column=cumulative_count&g.y_type=quantitative

  <img width="291" alt="github__select_yyyymmdd__sum_n__over___order_by_yyyymmdd_rows_unbounded_preceding___as_cumulative_count_from___select_substr_starred_at__0__11__as_yyyymmdd__count____as_n_from_stars_where_repo___107914493_group_by_yyyymmdd__" src="https://user-images.githubusercontent.com/9599/80891642-31587180-8c7a-11ea-9169-60eb3cb6d3e0.png">

  Stars per day (as a label bar chart, so very wide):

  https://github-to-sqlite.dogsheep.net/github?sql=%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++#g.mark=bar&g.x_column=yyyymmdd&g.x_type=ordinal&g.y_column=n&g.y_type=quantitative

  <img width="1424" alt="github__select_substr_starred_at__0__11__as_yyyymmdd__count____as_n_from_stars_where_repo___107914493_group_by_yyyymmdd" src="https://user-images.githubusercontent.com/9599/80891656-4e8d4000-8c7a-11ea-9ff2-02c9cf4cb5d5.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623027889
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38
node_id: MDEyOklzc3VlQ29tbWVudDYyMzAyNzg4OQ==
created_at: 2020-05-02T23:15:11Z
updated_at: 2020-05-02T23:15:11Z
author_association: MEMBER
body:
  This is one of the use-cases for the `repos_starred` view: it allows you to easily run this kid of query without having to construct the SQL by hand. Here's a demo:

  https://github-to-sqlite.dogsheep.net/github/repos_starred?name__contains=twitter

  My philosophy here is to keep the raw tables (like `stars`) as normalized as possible, then use SQL views which expose the data in a form that's easier to query.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623038148
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38
node_id: MDEyOklzc3VlQ29tbWVudDYyMzAzODE0OA==
created_at: 2020-05-03T01:18:57Z
updated_at: 2020-05-03T01:18:57Z
author_association: NONE
body:
  Thanks, @simonw! 

  I feel a little foolish in hindsight, but I'm on the same page now and am glad to have discovered first-hand a motivation for this `repos_starred` use case.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623038378
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38
node_id: MDEyOklzc3VlQ29tbWVudDYyMzAzODM3OA==
created_at: 2020-05-03T01:21:13Z
updated_at: 2020-05-03T01:21:13Z
author_association: MEMBER
body:
  No this is really useful feedback! I'm so close to this project that I miss what's not obvious to people dropping in for the first time.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623044643
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38
node_id: MDEyOklzc3VlQ29tbWVudDYyMzA0NDY0Mw==
created_at: 2020-05-03T02:34:32Z
updated_at: 2020-05-03T02:34:32Z
author_association: NONE
body:
  1. More than glad to share feedback from the sidelines as a [starrer](https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++starred_at%2C%0D%0A++starred_by%2C%0D%0A++full_name+as+repo_name%0D%0Afrom%0D%0A++repos_starred%0D%0Awhere%0D%0A++starred_by+%3D+%22zzeleznick%22%0D%0Aorder+by%0D%0A++starred_at+desc). 

  ```
  -- Motivation:
  --  Datasette is a data hammer and I'm looking for nails
  --  e.g. Find which repos a user has starred => trigger a TBD downstream action
  select
    starred_at,
    starred_by,
    full_name as repo_name
  from
    repos_starred
  where
    starred_by = "zzeleznick"
  order by
    starred_at desc
  ``` 

  | starred_at | starred_by | repo_name |
  | --- | --- | --- |
  | 2020-02-11T01:08:59Z | zzeleznick | dogsheep/twitter-to-sqlite |
  | 2020-01-11T21:57:34Z | zzeleznick | simonw/datasette |

  2. In my day job, I use [airflow](https://github.com/apache/airflow), and that's the mental model I'm bringing to [datasette](https://github.com/simonw/datasette). 

  3. I see your project like [twitter-to-sqlite](https://github.com/dogsheep/twitter-to-sqlite) akin to [Operators](https://airflow.apache.org/docs/stable/_api/index.html#pythonapi-operators) in Airflow world.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/27#issuecomment-623048530
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/27
node_id: MDEyOklzc3VlQ29tbWVudDYyMzA0ODUzMA==
created_at: 2020-05-03T03:30:31Z
updated_at: 2020-05-03T03:30:31Z
author_association: OWNER
body:
  Copy the design for `--not-null` and `--default` from the `insert` command:
  ```
  $ sqlite-utils insert dogs.db dogs_with_scores dogs-with-scores.json \
      --not-null=age \
      --not-null=name \
      --default age 2 \
      --default score 5
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/27#issuecomment-623049505
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/27
node_id: MDEyOklzc3VlQ29tbWVudDYyMzA0OTUwNQ==
created_at: 2020-05-03T03:45:32Z
updated_at: 2020-05-03T03:45:32Z
author_association: OWNER
body:
  Could take `--ignore` to ignore if table already exists, and `--replace` to drop and replace it if it exists.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/108#issuecomment-623050375
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/108
node_id: MDEyOklzc3VlQ29tbWVudDYyMzA1MDM3NQ==
created_at: 2020-05-03T03:59:16Z
updated_at: 2020-05-03T03:59:27Z
author_association: OWNER
body:
  ```
  import re
  r = re.compile(r'\$ sqlite-utils (\S+) ')
  rst = open("docs/cli.rst").read()
  {command for command in r.findall(rst) if "." not in command and ":" not in command}
  ```
  Outputs:
  ```
  {'add-column',
   'add-foreign-key',
   'create-index',
   'disable-fts',
   'enable-fts',
   'index-foreign-keys',
   'insert',
   'optimize',
   'populate-fts',
   'query',
   'rows',
   'tables',
   'vacuum',
   'views'}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/108#issuecomment-623050458
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/108
node_id: MDEyOklzc3VlQ29tbWVudDYyMzA1MDQ1OA==
created_at: 2020-05-03T04:00:32Z
updated_at: 2020-05-03T04:00:32Z
author_association: OWNER
body:
  ```
  from sqlite_utils import cli
  cli.cli.commands.keys()          
  ```
  Outputs:
  ```                                                                                                                                                 
  dict_keys(['tables', 'views', 'vacuum', 'optimize', 'add-column', 'add-foreign-key', 'index-foreign-keys', 'create-index', 'enable-fts', 'populate-fts', 'disable-fts', 'insert', 'upsert', 'create-table', 'query', 'rows'])
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/27#issuecomment-623051392
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/27
node_id: MDEyOklzc3VlQ29tbWVudDYyMzA1MTM5Mg==
created_at: 2020-05-03T04:15:05Z
updated_at: 2020-05-03T04:15:05Z
author_association: OWNER
body:
  I need tests for what happens if table already exists, or if the user provides invalid column types.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/27#issuecomment-623051550
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/27
node_id: MDEyOklzc3VlQ29tbWVudDYyMzA1MTU1MA==
created_at: 2020-05-03T04:17:18Z
updated_at: 2020-05-03T04:17:18Z
author_association: OWNER
body:
  Be nice if you could do `--fk colname` and have it guess the rest, but I'm not sure how to do that with a CLI option - they need a fixed number of arguments so that they don't consume the next batch of options.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/27#issuecomment-623126943
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/27
node_id: MDEyOklzc3VlQ29tbWVudDYyMzEyNjk0Mw==
created_at: 2020-05-03T15:25:55Z
updated_at: 2020-05-03T15:25:55Z
author_association: OWNER
body:
  Documentation: https://github.com/simonw/sqlite-utils/blob/78264b738cd72ffad6e5c32ede3f074f8aad0ca4/docs/cli.rst#creating-tables

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/107#issuecomment-623127140
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/107
node_id: MDEyOklzc3VlQ29tbWVudDYyMzEyNzE0MA==
created_at: 2020-05-03T15:27:22Z
updated_at: 2020-05-03T15:27:22Z
author_association: OWNER
body:
  Syntax:

      $ sqlite-utils create-view my.db myview "select sqlite_version()"

  Also accepts `--replace` and `--ignore` options, same as `create-table`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/107#issuecomment-623128528
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/107
node_id: MDEyOklzc3VlQ29tbWVudDYyMzEyODUyOA==
created_at: 2020-05-03T15:36:58Z
updated_at: 2020-05-03T15:36:58Z
author_association: OWNER
body:
  Documentation: https://github.com/simonw/sqlite-utils/blob/d16097231c5e51ea857b58c700f97a17b68dc583/docs/cli.rst#creating-views

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623193947
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDYyMzE5Mzk0Nw==
created_at: 2020-05-03T22:36:17Z
updated_at: 2020-05-03T22:36:17Z
author_association: MEMBER
body:
  I'm going to use [osxphotos](https://github.com/RhetTbull/osxphotos) for this.

  Since I've already got code to upload photos and insert them into a table based on their `sha256` hash, my first go at this will be to import data using the tool and foreign-key it to the `sha256` hash in the existing table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623195197
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDYyMzE5NTE5Nw==
created_at: 2020-05-03T22:44:33Z
updated_at: 2020-05-03T22:44:33Z
author_association: MEMBER
body:
  Command will be this:

      $ photos-to-sqlite apple-photos photos.db

  This will populate a `apple_photos` table with the data imported by the `osxphotos` library, plus the calculated sha256.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623198653
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDYyMzE5ODY1Mw==
created_at: 2020-05-03T23:09:57Z
updated_at: 2020-05-03T23:09:57Z
author_association: MEMBER
body:
  For locations: I'll add `place_x` columns for all of these:
  ```
  (Pdb) photo.place.address._asdict()
  {'street': None, 'sub_locality': None, 'city': 'Loreto', 'sub_administrative_area': 'Loreto', 'state_province': 'BCS', 'postal_code': None, 'country': 'Mexico', 'iso_country_code': 'MX'}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623198986
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDYyMzE5ODk4Ng==
created_at: 2020-05-03T23:12:31Z
updated_at: 2020-05-03T23:12:46Z
author_association: MEMBER
body:
  To get the taken date in UTC:
  ```
  from datetime import timezone
  (Pdb) photo.date.astimezone(timezone.utc).isoformat()
  '2018-02-13T20:21:31.620000+00:00'
  (Pdb) photo.date.astimezone(timezone.utc).isoformat().split(".")
  ['2018-02-13T20:21:31', '620000+00:00']
  (Pdb) photo.date.astimezone(timezone.utc).isoformat().split(".")[0]
  '2018-02-13T20:21:31'
  (Pdb) photo.date.astimezone(timezone.utc).isoformat().split(".")[0] + "+00:00"
  '2018-02-13T20:21:31+00:00'

  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199214
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDYyMzE5OTIxNA==
created_at: 2020-05-03T23:14:08Z
updated_at: 2020-05-03T23:14:08Z
author_association: MEMBER
body:
  Albums have UUIDs:
  ```
  (Pdb) photo.album_info[0].__dict__
  {'_uuid': '17816791-ABF3-447B-942C-9FA8065EEBBA', '_db': osxphotos.PhotosDB(dbfile='/Users/simon/Pictures/Photos Library.photoslibrary/database/photos.db'), '_title': 'Geotaggable Photos geotagged'}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199701
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDYyMzE5OTcwMQ==
created_at: 2020-05-03T23:17:38Z
updated_at: 2020-05-03T23:17:38Z
author_association: MEMBER
body:
  Record burst_uuid as a column:
  ```
  (Pdb) with_bursts[0]._info["burstUUID"]
  '703FAA23-57BF-40B4-8A33-D9CEB143391B'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199750
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDYyMzE5OTc1MA==
created_at: 2020-05-03T23:17:58Z
updated_at: 2020-05-03T23:17:58Z
author_association: MEMBER
body:
  Reading this source code is really useful for figuring out how to store a photo in a DB table: https://github.com/RhetTbull/osxphotos/blob/7444b6d173918a3ad2a07aefce5ecf054786c787/osxphotos/photoinfo.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/751#issuecomment-623207541
issue_url: https://api.github.com/repos/simonw/datasette/issues/751
node_id: MDEyOklzc3VlQ29tbWVudDYyMzIwNzU0MQ==
created_at: 2020-05-04T00:13:09Z
updated_at: 2020-05-04T00:13:09Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/datasette/blob/07e208cc6d9e901b87552c1be2854c220b3f9b6d/datasette/views/table.py#L519-L520

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/751#issuecomment-623207636
issue_url: https://api.github.com/repos/simonw/datasette/issues/751
node_id: MDEyOklzc3VlQ29tbWVudDYyMzIwNzYzNg==
created_at: 2020-05-04T00:13:46Z
updated_at: 2020-05-04T00:13:46Z
author_association: OWNER
body:
  Syntax suggestion:
  ```json
  {
      "databases": {
          "mydatabase": {
              "tables": {
                  "example_table": {
                      "sort": "created",
                      "size": 10
                  }
              }
          }
      }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/751#issuecomment-623207741
issue_url: https://api.github.com/repos/simonw/datasette/issues/751
node_id: MDEyOklzc3VlQ29tbWVudDYyMzIwNzc0MQ==
created_at: 2020-05-04T00:14:35Z
updated_at: 2020-05-04T00:14:35Z
author_association: OWNER
body:
  I'm calling it `"size"` for consistency with the querystring parameter `_size=`, even though `"page_size"` might be a more accurate name.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/754#issuecomment-623560217
issue_url: https://api.github.com/repos/simonw/datasette/issues/754
node_id: MDEyOklzc3VlQ29tbWVudDYyMzU2MDIxNw==
created_at: 2020-05-04T16:15:28Z
updated_at: 2020-05-04T16:15:28Z
author_association: OWNER
body:
  I think those will be fixed by an upgrade to aiofiles 0.5.0:

  https://github.com/Tinche/aiofiles/compare/v0.4.0...v0.5.0

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/754#issuecomment-623561480
issue_url: https://api.github.com/repos/simonw/datasette/issues/754
node_id: MDEyOklzc3VlQ29tbWVudDYyMzU2MTQ4MA==
created_at: 2020-05-04T16:17:42Z
updated_at: 2020-05-04T16:17:42Z
author_association: OWNER
body:
  Sure enough this run https://travis-ci.org/github/simonw/datasette/jobs/674415040 from that PR no longer shows the warnings.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/752#issuecomment-623598224
issue_url: https://api.github.com/repos/simonw/datasette/issues/752
node_id: MDEyOklzc3VlQ29tbWVudDYyMzU5ODIyNA==
created_at: 2020-05-04T17:28:11Z
updated_at: 2020-05-04T17:28:20Z
author_association: OWNER
body:
  Good catch: it looks like we're not specifying the charset for those pages:
  ```
  $ curl -i 'https://latest.datasette.io/404'
  HTTP/1.1 404 Not Found
  access-control-allow-origin: *
  content-type: text/html
  Date: Mon, 04 May 2020 16:23:25 GMT
  Server: Google Frontend
  Content-Length: 605

  <!DOCTYPE html>
  <html>
  <head>
      <title>Error 404</title>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/755#issuecomment-623635525
issue_url: https://api.github.com/repos/simonw/datasette/issues/755
node_id: MDEyOklzc3VlQ29tbWVudDYyMzYzNTUyNQ==
created_at: 2020-05-04T18:40:15Z
updated_at: 2020-05-04T18:40:15Z
author_association: OWNER
body:
  ```
  $ pytest tests/test_database.py -v
  ================================================================================ test session starts =================================================================================
  platform darwin -- Python 3.7.7, pytest-5.2.4, py-1.8.1, pluggy-0.13.1 -- /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/python
  cachedir: .pytest_cache
  rootdir: /Users/simon/Dropbox/Development/datasette, inifile: pytest.ini
  plugins: asyncio-0.10.0
  collected 9 items                                                                                                                                                                    

  tests/test_database.py::test_table_exists[tables0-True] PASSED                                                                                                                 [ 11%]
  tests/test_database.py::test_table_exists[tables1-False] PASSED                                                                                                                [ 22%]
  tests/test_database.py::test_get_all_foreign_keys PASSED                                                                                                                       [ 33%]
  tests/test_database.py::test_table_names PASSED                                                                                                                                [ 44%]
  tests/test_database.py::test_execute_write_block_true PASSED                                                                                                                   [ 55%]
  tests/test_database.py::test_execute_write_block_false PASSED                                                                                                                  [ 66%]
  tests/test_database.py::test_execute_write_fn_block_false no such column: id
  PASSED                                                                                                               [ 77%]
  tests/test_database.py::test_execute_write_fn_block_true PASSED                                                                                                                [ 88%]
  tests/test_database.py::test_execute_write_fn_exception PASSED                                                                                                                 [100%]

  ================================================================================= 9 passed in 1.90s ==================================================================================
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/756#issuecomment-623650324
issue_url: https://api.github.com/repos/simonw/datasette/issues/756
node_id: MDEyOklzc3VlQ29tbWVudDYyMzY1MDMyNA==
created_at: 2020-05-04T19:10:23Z
updated_at: 2020-05-04T19:10:23Z
author_association: OWNER
body:
  https://github.com/pipxproject/pipx/issues/79 showed me how to upgrade plugins within the pipx virtual environment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/756#issuecomment-623654436
issue_url: https://api.github.com/repos/simonw/datasette/issues/756
node_id: MDEyOklzc3VlQ29tbWVudDYyMzY1NDQzNg==
created_at: 2020-05-04T19:19:06Z
updated_at: 2020-05-04T19:19:06Z
author_association: OWNER
body:
  New documentation is here: https://datasette.readthedocs.io/en/latest/installation.html#install-using-pipx

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623723026
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15
node_id: MDEyOklzc3VlQ29tbWVudDYyMzcyMzAyNg==
created_at: 2020-05-04T21:41:30Z
updated_at: 2020-05-04T21:41:30Z
author_association: MEMBER
body:
  I'm going to put these in a table called `apple_photos_scores` - I'll also pull in the following columns from the `ZGENERICASSET` table:

  * `ZOVERALLAESTHETICSCORE`
  * `ZCURATIONSCORE`
  * `ZHIGHLIGHTVISIBILITYSCORE`
  * `ZPROMOTIONSCORE`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623723687
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15
node_id: MDEyOklzc3VlQ29tbWVudDYyMzcyMzY4Nw==
created_at: 2020-05-04T21:43:06Z
updated_at: 2020-05-04T21:43:06Z
author_association: MEMBER
body:
  It looks like I can map the photos I'm importing to these tables using the `ZUUID` column on `ZGENERICASSET` to get a `Z_PK` which then maps to the rows in `ZGENERICASSET`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623730934
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15
node_id: MDEyOklzc3VlQ29tbWVudDYyMzczMDkzNA==
created_at: 2020-05-04T22:00:38Z
updated_at: 2020-05-04T22:00:48Z
author_association: MEMBER
body:
  Here's the query to create the new table:
  ```sql
  create table apple_photos_scores as select
      ZGENERICASSET.ZUUID,
      ZGENERICASSET.ZOVERALLAESTHETICSCORE,
      ZGENERICASSET.ZCURATIONSCORE,
      ZGENERICASSET.ZPROMOTIONSCORE,
      ZGENERICASSET.ZHIGHLIGHTVISIBILITYSCORE,
      ZCOMPUTEDASSETATTRIBUTES.ZBEHAVIORALSCORE,
      ZCOMPUTEDASSETATTRIBUTES.ZFAILURESCORE,
      ZCOMPUTEDASSETATTRIBUTES.ZHARMONIOUSCOLORSCORE,
      ZCOMPUTEDASSETATTRIBUTES.ZIMMERSIVENESSSCORE,
      ZCOMPUTEDASSETATTRIBUTES.ZINTERACTIONSCORE,
      ZCOMPUTEDASSETATTRIBUTES.ZINTERESTINGSUBJECTSCORE,
      ZCOMPUTEDASSETATTRIBUTES.ZINTRUSIVEOBJECTPRESENCESCORE,
      ZCOMPUTEDASSETATTRIBUTES.ZLIVELYCOLORSCORE,
      ZCOMPUTEDASSETATTRIBUTES.ZLOWLIGHT,
      ZCOMPUTEDASSETATTRIBUTES.ZNOISESCORE,
      ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTCAMERATILTSCORE,
      ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTCOMPOSITIONSCORE,
      ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTLIGHTINGSCORE,
      ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTPATTERNSCORE,
      ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTPERSPECTIVESCORE,
      ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTPOSTPROCESSINGSCORE,
      ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTREFLECTIONSSCORE,
      ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTSYMMETRYSCORE,
      ZCOMPUTEDASSETATTRIBUTES.ZSHARPLYFOCUSEDSUBJECTSCORE,
      ZCOMPUTEDASSETATTRIBUTES.ZTASTEFULLYBLURREDSCORE,
      ZCOMPUTEDASSETATTRIBUTES.ZWELLCHOSENSUBJECTSCORE,
      ZCOMPUTEDASSETATTRIBUTES.ZWELLFRAMEDSUBJECTSCORE,
      ZCOMPUTEDASSETATTRIBUTES.ZWELLTIMEDSHOTSCORE
  from
      attached.ZGENERICASSET
        join attached.ZCOMPUTEDASSETATTRIBUTES on
            attached.ZGENERICASSET.Z_PK = attached.ZCOMPUTEDASSETATTRIBUTES.Z_PK;
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623739934
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15
node_id: MDEyOklzc3VlQ29tbWVudDYyMzczOTkzNA==
created_at: 2020-05-04T22:24:26Z
updated_at: 2020-05-04T22:24:26Z
author_association: MEMBER
body:
  Twitter thread with some examples of photos that are coming up from queries against these scores: https://twitter.com/simonw/status/1257434670750408705

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623805823
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDYyMzgwNTgyMw==
created_at: 2020-05-05T02:45:56Z
updated_at: 2020-05-05T02:45:56Z
author_association: MEMBER
body:
  I filed an issue with `osxphotos` about this here: https://github.com/RhetTbull/osxphotos/issues/121

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623806085
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDYyMzgwNjA4NQ==
created_at: 2020-05-05T02:47:18Z
updated_at: 2020-05-05T02:47:18Z
author_association: MEMBER
body:
  In https://github.com/RhetTbull/osxphotos/issues/121#issuecomment-623249263 Rhet Turnbull spotted a table called `ZSCENEIDENTIFIER` which looked like it might have the right data, but the columns in it aren't particularly helpful:
  ```
  Z_PK,Z_ENT,Z_OPT,ZSCENEIDENTIFIER,ZASSETATTRIBUTES,ZCONFIDENCE
  8,49,1,731,5,0.11834716796875
  9,49,1,684,6,0.0233648251742125
  10,49,1,1702,1,0.026153564453125
  ```
  I love the look of those confidence scores, but what do the numbers mean?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623806533
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDYyMzgwNjUzMw==
created_at: 2020-05-05T02:50:16Z
updated_at: 2020-05-05T02:50:16Z
author_association: MEMBER
body:
  I figured there must be a separate database that Photos uses to store the text of the identified labels.

  I used "Open Files and Ports" in Activity Monitor against the Photos app to try and spot candidates... and found `/Users/simon/Pictures/Photos Library.photoslibrary/database/search/psi.sqlite` - a 53MB SQLite database file.

  <img width="1365" alt="Item-0_and_Item-0_and_Item-0_and_Item-0" src="https://user-images.githubusercontent.com/9599/81031213-61ea0800-8e40-11ea-8237-cfce4a5128e0.png">

  Here's the schema of that file:
  ```
  $ sqlite3 psi.sqlite .schema
  CREATE TABLE word_embedding(word TEXT, extended_word TEXT, score DOUBLE);
  CREATE INDEX word_embedding_index ON word_embedding(word);
  CREATE VIRTUAL TABLE word_embedding_prefix USING fts5(extended_word)
  /* word_embedding_prefix(extended_word) */;
  CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_data'(id INTEGER PRIMARY KEY, block BLOB);
  CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
  CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_content'(id INTEGER PRIMARY KEY, c0);
  CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
  CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_config'(k PRIMARY KEY, v) WITHOUT ROWID;
  CREATE TABLE groups(category INT2, owning_groupid INT, content_string TEXT, normalized_string TEXT, lookup_identifier TEXT, token_ranges_0 INT8, token_ranges_1 INT8, UNIQUE(category, owning_groupid, content_string, lookup_identifier, token_ranges_0, token_ranges_1));
  CREATE TABLE assets(uuid_0 INT, uuid_1 INT, creationDate INT, UNIQUE(uuid_0, uuid_1));
  CREATE TABLE ga(groupid INT, assetid INT, PRIMARY KEY(groupid, assetid));
  CREATE TABLE collections(uuid_0 INT, uuid_1 INT, startDate INT, endDate INT, title TEXT, subtitle TEXT, keyAssetUUID_0 INT, keyAssetUUID_1 INT, typeAndNumberOfAssets INT32, sortDate DOUBLE, UNIQUE(uuid_0, uuid_1));
  CREATE TABLE gc(groupid INT, collectionid INT, PRIMARY KEY(groupid, collectionid));
  CREATE VIRTUAL TABLE prefix USING fts5(content='groups', normalized_string, category UNINDEXED, tokenize = 'PSITokenizer');
  CREATE TABLE IF NOT EXISTS 'prefix_data'(id INTEGER PRIMARY KEY, block BLOB);
  CREATE TABLE IF NOT EXISTS 'prefix_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
  CREATE TABLE IF NOT EXISTS 'prefix_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
  CREATE TABLE IF NOT EXISTS 'prefix_config'(k PRIMARY KEY, v) WITHOUT ROWID;
  CREATE TABLE lookup(identifier TEXT PRIMARY KEY, category INT2);
  CREATE TRIGGER trigger_groups_insert AFTER INSERT ON groups BEGIN INSERT INTO prefix(rowid, normalized_string, category) VALUES (new.rowid, new.normalized_string, new.category); END;
  CREATE TRIGGER trigger_groups_delete AFTER DELETE ON groups BEGIN INSERT INTO prefix(prefix, rowid, normalized_string, category) VALUES('delete', old.rowid, old.normalized_string, old.category); END;
  CREATE INDEX group_pk ON groups(category, content_string, normalized_string, lookup_identifier);
  CREATE INDEX asset_pk ON assets(uuid_0, uuid_1);
  CREATE INDEX ga_assetid ON ga(assetid, groupid);
  CREATE INDEX collection_pk ON collections(uuid_0, uuid_1);
  CREATE INDEX gc_collectionid ON gc(collectionid);
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623806687
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDYyMzgwNjY4Nw==
created_at: 2020-05-05T02:51:16Z
updated_at: 2020-05-05T02:51:16Z
author_association: MEMBER
body:
  Running datasette against it directly doesn't work:
  ```
  simon@Simons-MacBook-Pro search % datasette psi.sqlite
  Serve! files=('psi.sqlite',) (immutables=()) on port 8001
  Usage: datasette serve [OPTIONS] [FILES]...

  Error: Connection to psi.sqlite failed check: no such tokenizer: PSITokenizer
  ```
  Instead, I created a new SQLite database with a copy of some of the key tables, like this:
  ```
  sqlite-utils rows psi.sqlite groups | sqlite-utils insert /tmp/search.db groups -
  sqlite-utils rows psi.sqlite assets | sqlite-utils insert /tmp/search.db assets -
  sqlite-utils rows psi.sqlite ga | sqlite-utils insert /tmp/search.db ga -
  sqlite-utils rows psi.sqlite collections | sqlite-utils insert /tmp/search.db collections -
  sqlite-utils rows psi.sqlite gc | sqlite-utils insert /tmp/search.db gc -
  sqlite-utils rows psi.sqlite lookup | sqlite-utils insert /tmp/search.db lookup -
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623807568
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDYyMzgwNzU2OA==
created_at: 2020-05-05T02:56:06Z
updated_at: 2020-05-05T02:56:06Z
author_association: MEMBER
body:
  I'm pretty sure this is what I'm after. The `groups` table has what looks like identified labels in the rows with category = 2025:

  <img width="1122" alt="words__groups__2_528_rows_where_where_category___2025" src="https://user-images.githubusercontent.com/9599/81031361-e0df4080-8e40-11ea-9060-6d850aa52140.png">

  Then there's a `ga` table that maps groups to assets:

  <img width="304" alt="words__ga__633_653_rows" src="https://user-images.githubusercontent.com/9599/81031387-f48aa700-8e40-11ea-9a3d-da23903be928.png">

  And an `assets` table which looks like it has one row for every one of my photos:

  <img width="645" alt="words__assets__40_419_rows" src="https://user-images.githubusercontent.com/9599/81031402-04a28680-8e41-11ea-8047-e9199d068563.png">

  One major challenge: these UUIDs are split into two integer numbers, `uuid_0` and `uuid_1` - but the main photos database uses regular UUIDs like this:

  ![image](https://user-images.githubusercontent.com/9599/81031481-39164280-8e41-11ea-983b-005ced641a18.png)

  I need to figure out how to match up these two different UUID representations. I asked on Twitter if anyone has any ideas: https://twitter.com/simonw/status/1257500689019703296

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623811131
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDYyMzgxMTEzMQ==
created_at: 2020-05-05T03:16:18Z
updated_at: 2020-05-05T03:16:18Z
author_association: MEMBER
body:
  Here's how to convert two integers unto a UUID using Java. Not sure if it's the solution I need though (or how to do the same thing in Python):

  https://repl.it/repls/EuphoricSomberClasslibrary

  <img width="1494" alt="Repl_it_-_EuphoricSomberClasslibrary" src="https://user-images.githubusercontent.com/9599/81032267-0d488c00-8e44-11ea-9be7-680eaccd1611.png">

  ```java
  import java.util.UUID;

  class Main {
    public static void main(String[] args) {
      java.util.UUID uuid = new java.util.UUID(
        2544182952487526660L,
        -3640314103732024685L
      );
      System.out.println(
        uuid
      );
    }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623845014
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDYyMzg0NTAxNA==
created_at: 2020-05-05T03:55:14Z
updated_at: 2020-05-05T03:56:24Z
author_association: CONTRIBUTOR
body:
  I'm traveling w/o access to my Mac so can't help with any code right now.  I suspected ZSCENEIDENTIFIER was a foreign key into one of these psi.sqlite tables.  But looks like you're on to something connecting groups to assets.  As for the UUID, I think there's two ints because each is 64-bits but UUIDs are 128-bits.  Thus they need to be combined to get the 128 bit UUID.  You might be able to use Apple's [NSUUID](https://developer.apple.com/documentation/foundation/nsuuid?language=objc), for example, by wrapping with pyObjC.  Here's one [example](https://github.com/ronaldoussoren/pyobjc/blob/881c82a7ba90f193934b52b44143360c80dce5e5/pyobjc-framework-Cocoa/PyObjCTest/test_nsuuid.py) of using this in PyObjC's test suite. Interesting it's stored this way instead of a UUIDString as in Photos.sqlite.  Perhaps it for faster indexing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623846880
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDYyMzg0Njg4MA==
created_at: 2020-05-05T04:06:08Z
updated_at: 2020-05-05T04:06:08Z
author_association: MEMBER
body:
  This function seems to convert them into UUIDs that match my photos:
  ```python
  def to_uuid(uuid_0, uuid_1):
      b = uuid_0.to_bytes(8, 'little', signed=True) + uuid_1.to_bytes(8, 'little', signed=True)
      return str(uuid.UUID(bytes=b)).upper()
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623855841
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDYyMzg1NTg0MQ==
created_at: 2020-05-05T04:54:28Z
updated_at: 2020-05-05T04:54:28Z
author_association: MEMBER
body:
  Things were not matching up for me correctly:

  <img width="1143" alt="search__select_json_object__img_src____https___photos_simonwillison_net_i______photos_sha256___________photos_ext______w_400___as_photo__groups_content_string__assets_uuid_0__assets_uuid_1__to_uuid_assets_uuid_0__assets_uuid_1__as_uuid__pho" src="https://user-images.githubusercontent.com/9599/81035923-ca8db080-8e51-11ea-95a7-6ee60bae7502.png">

  I think that's because my import script didn't correctly import the existing `rowid` values.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623855885
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDYyMzg1NTg4NQ==
created_at: 2020-05-05T04:54:39Z
updated_at: 2020-05-05T04:54:53Z
author_association: MEMBER
body:
  Trying this import mechanism instead:
  `sqlite3 /Users/simon/Pictures/Photos\ Library.photoslibrary/database/search/psi.sqlite .dump | grep -v 'CREATE INDEX' | grep -v 'CREATE TRIGGER' | grep -v 'CREATE VIRTUAL TABLE' | sqlite3 search.db`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623857417
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDYyMzg1NzQxNw==
created_at: 2020-05-05T05:01:47Z
updated_at: 2020-05-05T05:01:47Z
author_association: MEMBER
body:
  Even that didn't work - it didn't copy across the rowid values. I'm pretty sure that's what's wrong here:
  ```
  sqlite3 /Users/simon/Pictures/Photos\ Library.photoslibrary/database/search/psi.sqlite 'select rowid, uuid_0, uuid_1 from assets limit 10'                                              
  1619605|-9205353363298198838|4814875488794983828
  1641378|-9205348195631362269|390804289838822030
  1634974|-9205331524553603243|-3834026796261633148
  1619083|-9205326176986145401|7563404215614709654
  22131|-9205315724827218763|8370531509591906734
  1645633|-9205247376092758131|-1311540150497601346
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623863902
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDYyMzg2MzkwMg==
created_at: 2020-05-05T05:31:53Z
updated_at: 2020-05-05T05:31:53Z
author_association: MEMBER
body:
  Yes! Turning those `rowid` values into `id` with this script did the job:
  ```python
  import sqlite3
  import sqlite_utils

  conn = sqlite3.connect(
      "/Users/simon/Pictures/Photos Library.photoslibrary/database/search/psi.sqlite"
  )


  def all_rows(table):
      result = conn.execute("select rowid as id, * from {}".format(table))
      cols = [c[0] for c in result.description]
      for row in result.fetchall():
          yield dict(zip(cols, row))


  if __name__ == "__main__":
      db = sqlite_utils.Database("psi_copy.db")
      for table in ("assets", "collections", "ga", "gc", "groups"):
          db[table].upsert_all(all_rows(table), pk="id", alter=True)
  ```
  Then I ran this query:
  ```sql
  select 
    json_object('img_src', 'https://photos.simonwillison.net/i/' || photos.sha256 || '.' || photos.ext || '?w=400') as photo,
     group_concat(strip_null_chars(groups.content_string), ' ') as words, assets.uuid_0, assets.uuid_1, to_uuid(assets.uuid_0, assets.uuid_1) as uuid
  from assets join ga on assets.id = ga.assetid
  join groups on ga.groupid = groups.id
  join photos on photos.uuid = to_uuid(assets.uuid_0, assets.uuid_1)
  where groups.category = 2024
  group by assets.id
  order by random() limit 10
  ```
  And got these results!
  <img width="1054" alt="psi_copy__select_json_object__img_src____https___photos_simonwillison_net_i______photos_sha256___________photos_ext______w_400___as_photo__group_concat_strip_null_chars_groups_content_string________as_words__assets_uuid_0__assets_uuid_1__to" src="https://user-images.githubusercontent.com/9599/81037264-f1021a80-8e56-11ea-9924-6f9f55a0fb4b.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


