html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app https://github.com/simonw/datasette/issues/939#issuecomment-674547788,https://api.github.com/repos/simonw/datasette/issues/939,674547788,MDEyOklzc3VlQ29tbWVudDY3NDU0Nzc4OA==,9599,2020-08-16T16:31:08Z,2020-08-16T16:31:08Z,OWNER,Also: they should all be able to return `async def` inner functions. At the moment only `extra_template_vars` supports that pattern.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679779797, https://github.com/simonw/datasette/issues/939#issuecomment-674545058,https://api.github.com/repos/simonw/datasette/issues/939,674545058,MDEyOklzc3VlQ29tbWVudDY3NDU0NTA1OA==,9599,2020-08-16T16:07:20Z,2020-08-16T16:07:20Z,OWNER,"I'm going to implement this first as a single commit, then implement `columns` from #938 as a separate change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679779797, https://github.com/simonw/datasette/issues/939#issuecomment-674544973,https://api.github.com/repos/simonw/datasette/issues/939,674544973,MDEyOklzc3VlQ29tbWVudDY3NDU0NDk3Mw==,9599,2020-08-16T16:06:34Z,2020-08-16T16:06:34Z,OWNER,"They should also be next to each other in the documentation - right now they're a bit scattered in terms of page order: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679779797, https://github.com/simonw/datasette/issues/939#issuecomment-674544875,https://api.github.com/repos/simonw/datasette/issues/939,674544875,MDEyOklzc3VlQ29tbWVudDY3NDU0NDg3NQ==,9599,2020-08-16T16:05:36Z,2020-08-16T16:05:36Z,OWNER,"They should all take: template, database, table, view_name, request, datasette, columns `columns` is new, see #938.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679779797, https://github.com/simonw/datasette/issues/938#issuecomment-674544691,https://api.github.com/repos/simonw/datasette/issues/938,674544691,MDEyOklzc3VlQ29tbWVudDY3NDU0NDY5MQ==,9599,2020-08-16T16:03:52Z,2020-08-16T16:03:52Z,OWNER,"Four plugin hooks need this extra `columns` argument: - `extra_css_urls(template, database, table, datasette)` - `extra_js_urls(template, database, table, datasette)` - `extra_body_script(template, database, table, view_name, datasette)` - `extra_template_vars(template, database, table, view_name, request, datasette)` It's weird that these take different arguments. I should fix that too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679700269, https://github.com/simonw/datasette/pull/936#issuecomment-674453318,https://api.github.com/repos/simonw/datasette/issues/936,674453318,MDEyOklzc3VlQ29tbWVudDY3NDQ1MzMxOA==,9599,2020-08-15T22:29:15Z,2020-08-15T22:32:58Z,OWNER,"Test failure: ``` tests/test_canned_queries.py F >>> captured stdout >>> __enter__ >>> traceback >>> canned_write_client = def test_insert(canned_write_client): response = canned_write_client.post( ""/data/add_name"", {""name"": ""Hello""}, allow_redirects=False, csrftoken_from=True, cookies={""foo"": ""bar""}, ) assert 302 == response.status > assert ""/data/add_name?success"" == response.headers[""Location""] E AssertionError: assert '/data/add_name?success' == '/data/add_name' E - /data/add_name E + /data/add_name?success E ? ++++++++ /Users/simon/Dropbox/Development/datasette/tests/test_canned_queries.py:66: AssertionError ``` No idea why this change would affect that test.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679650632, https://github.com/simonw/datasette/issues/935#issuecomment-674451012,https://api.github.com/repos/simonw/datasette/issues/935,674451012,MDEyOklzc3VlQ29tbWVudDY3NDQ1MTAxMg==,9599,2020-08-15T21:56:13Z,2020-08-15T21:56:13Z,OWNER,"This implementation seems to fix it, need to work out how to test though. ```diff diff --git a/datasette/database.py b/datasette/database.py index ffa7a79..7ba1456 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -89,14 +89,22 @@ class Database: def _execute_writes(self): # Infinite looping thread that protects the single write connection # to this database - conn = self.connect(write=True) + conn_exception = None + conn = None + try: + conn = self.connect(write=True) + except Exception as e: + conn_exception = e while True: task = self._write_queue.get() - try: - result = task.fn(conn) - except Exception as e: - print(e) - result = e + if conn_exception is not None: + result = conn_exception + else: + try: + result = task.fn(conn) + except Exception as e: + print(e) + result = e task.reply_queue.sync_q.put(result) async def execute_fn(self, fn): ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679646710, https://github.com/simonw/datasette/issues/935#issuecomment-674450652,https://api.github.com/repos/simonw/datasette/issues/935,674450652,MDEyOklzc3VlQ29tbWVudDY3NDQ1MDY1Mg==,9599,2020-08-15T21:51:22Z,2020-08-15T21:51:22Z,OWNER,"The easiest way to recreate this is to attempt a write against an immutable database, which triggers this assertion error: https://github.com/simonw/datasette/blob/13b3b51087964d5e1a8c1cdd2495e07bdbe176b8/datasette/database.py#L47-L55","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679646710, https://github.com/simonw/datasette/issues/935#issuecomment-674450607,https://api.github.com/repos/simonw/datasette/issues/935,674450607,MDEyOklzc3VlQ29tbWVudDY3NDQ1MDYwNw==,9599,2020-08-15T21:50:41Z,2020-08-15T21:50:41Z,OWNER,"The bug is here: https://github.com/simonw/datasette/blob/13b3b51087964d5e1a8c1cdd2495e07bdbe176b8/datasette/database.py#L89-L100 If `conn = self.connect(write=True)` raises an exception the entire server hangs, like this: ``` % datasette -i fixtures.db --get / Exception in thread Thread-1: Traceback (most recent call last): File ""/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py"", line 932, in _bootstrap_inner self.run() File ""/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py"", line 870, in run self._target(*self._args, **self._kwargs) File ""/Users/simon/.local/share/virtualenvs/latest-datasette-with-all-plugins-PJL_Xy9e/lib/python3.8/site-packages/datasette/database.py"", line 92, in _execute_writes conn = self.connect(write=True) File ""/Users/simon/.local/share/virtualenvs/latest-datasette-with-all-plugins-PJL_Xy9e/lib/python3.8/site-packages/datasette/database.py"", line 55, in connect assert not (write and not self.is_mutable) AssertionError ... server hangs here ... ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679646710, https://github.com/simonw/datasette/issues/932#issuecomment-674144798,https://api.github.com/repos/simonw/datasette/issues/932,674144798,MDEyOklzc3VlQ29tbWVudDY3NDE0NDc5OA==,9599,2020-08-14T16:02:24Z,2020-08-14T16:02:24Z,OWNER,"Things to go in here: - What is Datasette? - A *database* contains *tables* full of *records*. A table has *rows* and *columns*. - Understanding faceting - How to use the filter interface - How to export data - How to link to data - How to run SQL","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",678760988, https://github.com/simonw/datasette/issues/932#issuecomment-673735299,https://api.github.com/repos/simonw/datasette/issues/932,673735299,MDEyOklzc3VlQ29tbWVudDY3MzczNTI5OQ==,9599,2020-08-13T22:10:40Z,2020-08-13T22:11:06Z,OWNER,"Idea: plugins can provide their own user-facing documentation. Datasette can like to eg `datasette.io/help?plugins=datasette-vega,datasette-cluster-map` to get the user manual with extra sections for those plugins. Or... link to `?url=datasette-url` and the documentation site can hit `/-/plugins.json` to figure out what extra manual sections to display!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",678760988, https://github.com/simonw/datasette/issues/932#issuecomment-673734387,https://api.github.com/repos/simonw/datasette/issues/932,673734387,MDEyOklzc3VlQ29tbWVudDY3MzczNDM4Nw==,9599,2020-08-13T22:08:06Z,2020-08-13T22:08:06Z,OWNER,One challenge: how does this interact with plugins?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",678760988, https://github.com/simonw/datasette/issues/932#issuecomment-673733904,https://api.github.com/repos/simonw/datasette/issues/932,673733904,MDEyOklzc3VlQ29tbWVudDY3MzczMzkwNA==,9599,2020-08-13T22:06:50Z,2020-08-13T22:06:50Z,OWNER,Title: **Using Datasette**. `using.rst`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",678760988, https://github.com/simonw/datasette/issues/931#issuecomment-673123213,https://api.github.com/repos/simonw/datasette/issues/931,673123213,MDEyOklzc3VlQ29tbWVudDY3MzEyMzIxMw==,9599,2020-08-12T21:36:20Z,2020-08-12T21:36:20Z,OWNER,That worked: https://hub.docker.com/r/datasetteproject/datasette/tags,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677926613, https://github.com/simonw/datasette/issues/931#issuecomment-673104851,https://api.github.com/repos/simonw/datasette/issues/931,673104851,MDEyOklzc3VlQ29tbWVudDY3MzEwNDg1MQ==,9599,2020-08-12T20:52:08Z,2020-08-12T20:52:08Z,OWNER,"``` docker run -p 8001:8001 -v `pwd`:/mnt \ datasette-updated-spatialite \ datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db --load-extension=/usr/local/lib/mod_spatialite.so ``` This seems to work `/-/versions` reports the SpatiaLite versions as expected: ``` ""extensions"": { ""json1"": null, ""spatialite"": ""4.4.0-RC0"" }, ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677926613, https://github.com/simonw/datasette/issues/931#issuecomment-673088110,https://api.github.com/repos/simonw/datasette/issues/931,673088110,MDEyOklzc3VlQ29tbWVudDY3MzA4ODExMA==,9599,2020-08-12T20:15:28Z,2020-08-12T20:15:28Z,OWNER,"I changed the Dockerfile and built it on my laptop using: docker build -f Dockerfile -t datasette-updated-spatialite . ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677926613, https://github.com/simonw/datasette/issues/931#issuecomment-673074297,https://api.github.com/repos/simonw/datasette/issues/931,673074297,MDEyOklzc3VlQ29tbWVudDY3MzA3NDI5Nw==,9599,2020-08-12T19:46:29Z,2020-08-12T19:46:29Z,OWNER,"Looks like the old files have moved to: - http://www.gaia-gis.it/gaia-sins/spatialite-tools-sources/ - http://www.gaia-gis.it/gaia-sins/libspatialite-sources/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677926613, https://github.com/simonw/datasette/issues/931#issuecomment-673070308,https://api.github.com/repos/simonw/datasette/issues/931,673070308,MDEyOklzc3VlQ29tbWVudDY3MzA3MDMwOA==,9599,2020-08-12T19:37:55Z,2020-08-12T19:37:55Z,OWNER,"Project news here: https://groups.google.com/g/spatialite-users/c/8AG3hQzXDmo > FreeXL > ============================== > the current stable version is now 1.0.6 > (supporting few minor bug fixes and refreshed > build scripts) > DONE: no further development is expected in > the near future. > > http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz > http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.zip > > > ReadOSM > =============================== > the current stable version is now 1.1.0a > (supporting few minor bug fixes and refreshed > build scripts) > DONE: no further development is expected in > the near future. > > http://www.gaia-gis.it/gaia-sins/readosm-1.1.0a.tar.gz > http://www.gaia-gis.it/gaia-sins/readosm-1.1.0a.zip > > > VirtualPG > =============================== > the current stanle version is now 2.0.0 > (supporting few minor bug fixes and refreshed > build scripts) > DONE: no further development is expected in > the near future. > > http://www.gaia-gis.it/gaia-sins/virtualpg-2.0.0.tar.gz > http://www.gaia-gis.it/gaia-sins/virtualpg-2.0.0.zip > > > libspatialite > =============================== > 5.0.0 Release Candidate 1 (RC1) is now ready > to become a stable release. > I'll simply wait for more or less a week so to > allow for a reasonable community testing; if > no critical issue will be reported in the meanwhile > I'll go ASAP to release 5.0.0 ""stable"" > > http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.0-RC1.tar.gz > http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.0-RC1.zip > > > librasterlite2 > ================================ > still under active development. > a very relevant milestone has been achieved, > now the DB layout required for fully integrating > libspatialte and librasterlie2 is definetely > consolidated, no further changes are expected. > > http://www.gaia-gis.it/gaia-sins/librasterlite2-sources/librasterlite2-1.1.0-beta1.tar.gz > http://www.gaia-gis.it/gaia-sins/librasterlite2-sources/librasterlite2-1.1.0-beta1.zip > > more development activity is still required for > fully implementing SQL-driven graphic rendering > of both Vector and Raster Coverages based on > standard SLD/SE styles. > > a reasonable estimate is two man-month (and > not necessarily one development month equals > one calendar month) > > > spatialite_gui > ================================= > the GUI tool is expected to take full profit from > the advanced features of RasterLite2 so to > become a complete self-contained GIS viewer > with map editing capabilities. > > consequently it wiil surely be the last > member of the SpatiaLite family to be > released in a final stable form. > several transient development versions > will be possibly released so to closely > follow the evolution of RasterLite2. > > http://www.gaia-gis.it/gaia-sins/spatialite-gui-sources/spatialite_gui-2.1.0-beta1.tar.gz > http://www.gaia-gis.it/gaia-sins/spatialite-gui-sources/spatialite_gui-2.1.0-beta1.zip","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677926613, https://github.com/simonw/datasette/issues/931#issuecomment-673068919,https://api.github.com/repos/simonw/datasette/issues/931,673068919,MDEyOklzc3VlQ29tbWVudDY3MzA2ODkxOQ==,9599,2020-08-12T19:34:57Z,2020-08-12T19:34:57Z,OWNER,"Looks like SpatiaLite released new versions: https://www.gaia-gis.it/fossil/freexl/index says ""Sources: current version is 1.0.6 (released on 2020-08-02)"" and links to http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677926613, https://github.com/simonw/datasette/issues/931#issuecomment-673068327,https://api.github.com/repos/simonw/datasette/issues/931,673068327,MDEyOklzc3VlQ29tbWVudDY3MzA2ODMyNw==,9599,2020-08-12T19:33:42Z,2020-08-12T19:33:42Z,OWNER,"https://hub.docker.com/r/datasetteproject/datasette/tags shows this: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677926613, https://github.com/simonw/sqlite-utils/issues/133#issuecomment-672997703,https://api.github.com/repos/simonw/sqlite-utils/issues/133,672997703,MDEyOklzc3VlQ29tbWVudDY3Mjk5NzcwMw==,9599,2020-08-12T17:05:06Z,2020-08-12T17:05:06Z,OWNER,Released.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677839979, https://github.com/simonw/datasette/issues/930#issuecomment-672550662,https://api.github.com/repos/simonw/datasette/issues/930,672550662,MDEyOklzc3VlQ29tbWVudDY3MjU1MDY2Mg==,9599,2020-08-12T03:30:59Z,2020-08-12T03:30:59Z,OWNER,https://datasette.readthedocs.io/en/stable/changelog.html#v0-47-1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677326155, https://github.com/simonw/datasette/issues/930#issuecomment-672519787,https://api.github.com/repos/simonw/datasette/issues/930,672519787,MDEyOklzc3VlQ29tbWVudDY3MjUxOTc4Nw==,9599,2020-08-12T02:52:46Z,2020-08-12T02:52:46Z,OWNER,"Homebrew install is now fixed, using a temporary URL while waiting for Travis to ship the new release to PyPI: https://github.com/simonw/homebrew-datasette/issues/6","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677326155, https://github.com/simonw/datasette/issues/930#issuecomment-672488293,https://api.github.com/repos/simonw/datasette/issues/930,672488293,MDEyOklzc3VlQ29tbWVudDY3MjQ4ODI5Mw==,9599,2020-08-12T02:35:34Z,2020-08-12T02:35:34Z,OWNER,"OK, this has fixed it - I used `sdist` to create the `.tar.gz` and then `pip installed` it into a new environment, and everything worked just fine. Going to ship this as Datasette 0.47.1.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677326155, https://github.com/simonw/datasette/issues/930#issuecomment-672480811,https://api.github.com/repos/simonw/datasette/issues/930,672480811,MDEyOklzc3VlQ29tbWVudDY3MjQ4MDgxMQ==,9599,2020-08-12T02:31:38Z,2020-08-12T02:31:38Z,OWNER,The root cause appears to be that the `templates/` are missing from the `sdist` `.tar.gz`: https://github.com/simonw/homebrew-datasette/issues/6#issuecomment-672477352,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677326155, https://github.com/simonw/datasette/issues/930#issuecomment-672472518,https://api.github.com/repos/simonw/datasette/issues/930,672472518,MDEyOklzc3VlQ29tbWVudDY3MjQ3MjUxOA==,9599,2020-08-12T02:24:49Z,2020-08-12T02:24:49Z,OWNER,I checked and `datasette publish` deploys a 0.47 version that works fine too: https://datasette-0-47-j7hipcg4aq-uc.a.run.app/-/versions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677326155, https://github.com/simonw/datasette/issues/930#issuecomment-672471431,https://api.github.com/repos/simonw/datasette/issues/930,672471431,MDEyOklzc3VlQ29tbWVudDY3MjQ3MTQzMQ==,9599,2020-08-12T02:21:19Z,2020-08-12T02:21:19Z,OWNER,"Correction: pip installed Datasette works fine, it's just the Homebrew release that is broken.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677326155, https://github.com/simonw/datasette/issues/926#issuecomment-672393737,https://api.github.com/repos/simonw/datasette/issues/926,672393737,MDEyOklzc3VlQ29tbWVudDY3MjM5MzczNw==,9599,2020-08-12T00:26:17Z,2020-08-12T00:26:17Z,OWNER,"``` $ datasette --get '/:memory:.json?sql=select+sqlite_version()' | jq . { ""database"": "":memory:"", ""query_name"": null, ""rows"": [ [ ""3.32.3"" ] ], ""truncated"": false, ""columns"": [ ""sqlite_version()"" ], ""query"": { ""sql"": ""select sqlite_version()"", ""params"": {} }, ""private"": false, ""allow_execute_sql"": true, ""query_ms"": 1.165628433227539 } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677250834, https://github.com/simonw/datasette/pull/927#issuecomment-672391299,https://api.github.com/repos/simonw/datasette/issues/927,672391299,MDEyOklzc3VlQ29tbWVudDY3MjM5MTI5OQ==,9599,2020-08-12T00:19:20Z,2020-08-12T00:19:20Z,OWNER,Docs: https://github.com/simonw/datasette/blob/2111da01a03cfc62303b6a4b59ea9f96d22c0f78/docs/getting_started.rst#datasette---get,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677265716, https://github.com/simonw/datasette/pull/927#issuecomment-672382108,https://api.github.com/repos/simonw/datasette/issues/927,672382108,MDEyOklzc3VlQ29tbWVudDY3MjM4MjEwOA==,9599,2020-08-12T00:09:18Z,2020-08-12T00:09:18Z,OWNER,Documentation can go here: https://datasette.readthedocs.io/en/latest/getting_started.html#datasette-serve-options,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677265716, https://github.com/simonw/datasette/issues/928#issuecomment-672373061,https://api.github.com/repos/simonw/datasette/issues/928,672373061,MDEyOklzc3VlQ29tbWVudDY3MjM3MzA2MQ==,9599,2020-08-11T23:56:19Z,2020-08-11T23:56:19Z,OWNER,"New implementation of the `install` command: https://github.com/simonw/datasette/blob/afdeda8216d4d3027f87583ccdbef17ad85022ef/datasette/cli.py#L235-L240","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677272618, https://github.com/simonw/datasette/issues/928#issuecomment-672372465,https://api.github.com/repos/simonw/datasette/issues/928,672372465,MDEyOklzc3VlQ29tbWVudDY3MjM3MjQ2NQ==,9599,2020-08-11T23:54:28Z,2020-08-11T23:54:28Z,OWNER,"While debugging this I found a useful clue in https://github.com/pypa/pip/blob/e060970d51c5946beac8447eb95585d83019582d/src/pip/_internal/cli/main.py#L23-L47 ``` # Do not import and use main() directly! Using it directly is actively # discouraged by pip's maintainers. The name, location and behavior of # this function is subject to change, so calling it directly is not # portable across different pip versions. # In addition, running pip in-process is unsupported and unsafe. This is # elaborated in detail at # https://pip.pypa.io/en/stable/user_guide/#using-pip-from-your-program. # That document also provides suggestions that should work for nearly # all users that are considering importing and using main() directly. # However, we know that certain users will still want to invoke pip # in-process. If you understand and accept the implications of using pip # in an unsupported manner, the best approach is to use runpy to avoid # depending on the exact location of this entry point. # The following example shows how to use runpy to invoke pip in that # case: # # sys.argv = [""pip"", your, args, here] # runpy.run_module(""pip"", run_name=""__main__"") # # Note that this will exit the process after running, unlike a direct # call to main. As it is not safe to do any processing after calling # main, this should not be an issue in practice. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677272618, https://github.com/simonw/datasette/issues/928#issuecomment-672372197,https://api.github.com/repos/simonw/datasette/issues/928,672372197,MDEyOklzc3VlQ29tbWVudDY3MjM3MjE5Nw==,9599,2020-08-11T23:53:38Z,2020-08-11T23:53:38Z,OWNER,Caused by the tests for #925,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677272618, https://github.com/simonw/datasette/pull/927#issuecomment-672357176,https://api.github.com/repos/simonw/datasette/issues/927,672357176,MDEyOklzc3VlQ29tbWVudDY3MjM1NzE3Ng==,9599,2020-08-11T23:32:08Z,2020-08-11T23:33:09Z,OWNER,Needs documentation and tests.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677265716, https://github.com/simonw/datasette/pull/927#issuecomment-672357902,https://api.github.com/repos/simonw/datasette/issues/927,672357902,MDEyOklzc3VlQ29tbWVudDY3MjM1NzkwMg==,9599,2020-08-11T23:32:39Z,2020-08-11T23:32:39Z,OWNER,"It works: ``` $ datasette --get '/:memory:.json?sql=select * from sqlite_master' | jq . { ""database"": "":memory:"", ""query_name"": null, ""rows"": [], ""truncated"": false, ""columns"": [ ""type"", ""name"", ""tbl_name"", ""rootpage"", ""sql"" ], ""query"": { ""sql"": ""select * from sqlite_master"", ""params"": {} }, ""private"": false, ""allow_execute_sql"": true, ""query_ms"": 0.8032321929931641 } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677265716, https://github.com/simonw/datasette/issues/926#issuecomment-672338113,https://api.github.com/repos/simonw/datasette/issues/926,672338113,MDEyOklzc3VlQ29tbWVudDY3MjMzODExMw==,9599,2020-08-11T22:57:28Z,2020-08-11T22:57:28Z,OWNER,"I partly want this so I can easily implement a better `test` method for the Homebrew package. The test I have right now looks like this: https://github.com/simonw/homebrew-datasette/blob/8aa30aa183158051a987a7e3f50e7e3ee05d8ee9/Formula/datasette.rb#L125-L127 ``` test do system bin/""datasette"", ""--help"" end ``` The Homebrew docs at https://docs.brew.sh/Formula-Cookbook#add-a-test-to-the-formula say: > We want tests that don't require any user input and test the basic functionality of the application. For example `foo build-foo input.foo` is a good test and (despite their widespread use) `foo --version` and `foo --help` are bad tests. However, a bad test is better than no test at all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677250834, https://github.com/simonw/datasette/issues/923#issuecomment-672336720,https://api.github.com/repos/simonw/datasette/issues/923,672336720,MDEyOklzc3VlQ29tbWVudDY3MjMzNjcyMA==,9599,2020-08-11T22:53:07Z,2020-08-11T22:53:07Z,OWNER,https://github.com/simonw/datasette/blob/5126ecb1267ed3850bf3b0ab270accd031a02e79/docs/installation.rst#using-homebrew,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677037043, https://github.com/simonw/datasette/issues/923#issuecomment-672329101,https://api.github.com/repos/simonw/datasette/issues/923,672329101,MDEyOklzc3VlQ29tbWVudDY3MjMyOTEwMQ==,9599,2020-08-11T22:35:13Z,2020-08-11T22:35:13Z,OWNER,I added the `datasette install name-of-plugin` command in #925 mainly to simplify the process of installing plugins if Datasette itself was installed using homebrew.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677037043, https://github.com/simonw/datasette/issues/925#issuecomment-672328807,https://api.github.com/repos/simonw/datasette/issues/925,672328807,MDEyOklzc3VlQ29tbWVudDY3MjMyODgwNw==,9599,2020-08-11T22:34:37Z,2020-08-11T22:34:37Z,OWNER,"This will simplify the instructions for installing plugins with Datasette install via homebrew, refs #923","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677227912, https://github.com/simonw/datasette/issues/925#issuecomment-672328436,https://api.github.com/repos/simonw/datasette/issues/925,672328436,MDEyOklzc3VlQ29tbWVudDY3MjMyODQzNg==,9599,2020-08-11T22:33:32Z,2020-08-11T22:33:42Z,OWNER,"``` $ datasette install --help Usage: datasette install [OPTIONS] PACKAGES... Install Python packages - e.g. Datasette plugins - into the same environment as Datasette Options: --help Show this message and exit. $ datasette uninstall --help Usage: datasette uninstall [OPTIONS] PACKAGES... Uninstall Python packages (e.g. plugins) from the Datasette environment Options: -y, --yes Don't ask for confirmation --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677227912, https://github.com/simonw/datasette/issues/925#issuecomment-672304650,https://api.github.com/repos/simonw/datasette/issues/925,672304650,MDEyOklzc3VlQ29tbWVudDY3MjMwNDY1MA==,9599,2020-08-11T22:04:48Z,2020-08-11T22:04:48Z,OWNER,Prototyped in this thread: https://github.com/simonw/datasette/issues/335#issuecomment-671005731,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677227912, https://github.com/simonw/datasette/issues/923#issuecomment-672288845,https://api.github.com/repos/simonw/datasette/issues/923,672288845,MDEyOklzc3VlQ29tbWVudDY3MjI4ODg0NQ==,9599,2020-08-11T21:28:17Z,2020-08-11T21:28:17Z,OWNER,"Here's a pattern for installing plugins: ``` $ datasette plugins [] $ /usr/local/opt/datasette/libexec/bin/pip install datasette-vega Collecting datasette-vega Using cached datasette_vega-0.6.2-py3-none-any.whl (1.8 MB) Requirement already satisfied: datasette in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette-vega) (0.46) Requirement already satisfied: click~=7.1.1 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (7.1.2) Requirement already satisfied: click-default-group~=1.2.2 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (1.2.2) Requirement already satisfied: Jinja2<2.12.0,>=2.10.3 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (2.11.2) Requirement already satisfied: hupper~=1.9 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (1.10.2) Requirement already satisfied: pint~=0.9 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.14) Requirement already satisfied: pluggy~=0.13.0 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.13.1) Requirement already satisfied: uvicorn~=0.11 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.11.8) Requirement already satisfied: aiofiles<0.6,>=0.4 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.5.0) Requirement already satisfied: janus<0.6,>=0.4 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.5.0) Requirement already satisfied: asgi-csrf>=0.6 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.6.1) Requirement already satisfied: PyYAML~=5.3 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (5.3.1) Requirement already satisfied: mergedeep<1.4.0,>=1.1.1 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (1.3.0) Requirement already satisfied: itsdangerous~=1.1 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (1.1.0) Requirement already satisfied: python-baseconv==1.2.2 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (1.2.2) Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from Jinja2<2.12.0,>=2.10.3->datasette->datasette-vega) (1.1.1) Requirement already satisfied: setuptools in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from pint~=0.9->datasette->datasette-vega) (49.3.1) Requirement already satisfied: packaging in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from pint~=0.9->datasette->datasette-vega) (20.4) Requirement already satisfied: h11<0.10,>=0.8 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from uvicorn~=0.11->datasette->datasette-vega) (0.9.0) Requirement already satisfied: websockets==8.* in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from uvicorn~=0.11->datasette->datasette-vega) (8.1) Requirement already satisfied: httptools==0.1.* in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from uvicorn~=0.11->datasette->datasette-vega) (0.1.1) Requirement already satisfied: uvloop>=0.14.0 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from uvicorn~=0.11->datasette->datasette-vega) (0.14.0) Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from packaging->pint~=0.9->datasette->datasette-vega) (2.4.7) Requirement already satisfied: six in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from packaging->pint~=0.9->datasette->datasette-vega) (1.15.0) Installing collected packages: datasette-vega Successfully installed datasette-vega-0.6.2 $ datasette plugins [ { ""name"": ""datasette-vega"", ""static"": true, ""templates"": false, ""version"": ""0.6.2"", ""hooks"": [ ""extra_css_urls"", ""extra_js_urls"" ] } ] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677037043, https://github.com/simonw/datasette/issues/923#issuecomment-672287754,https://api.github.com/repos/simonw/datasette/issues/923,672287754,MDEyOklzc3VlQ29tbWVudDY3MjI4Nzc1NA==,9599,2020-08-11T21:25:33Z,2020-08-11T21:25:33Z,OWNER,.. and confirm if `brew tap ...` is even needed if you run `brew install simonw/datasette/datasette`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677037043, https://github.com/simonw/datasette/issues/923#issuecomment-672089281,https://api.github.com/repos/simonw/datasette/issues/923,672089281,MDEyOklzc3VlQ29tbWVudDY3MjA4OTI4MQ==,9599,2020-08-11T16:54:50Z,2020-08-11T16:54:50Z,OWNER,Also need to talk about how you install plugins.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677037043, https://github.com/simonw/datasette/issues/335#issuecomment-672088880,https://api.github.com/repos/simonw/datasette/issues/335,672088880,MDEyOklzc3VlQ29tbWVudDY3MjA4ODg4MA==,9599,2020-08-11T16:54:06Z,2020-08-11T16:54:06Z,OWNER,"It works! ``` $ brew tap simonw/datasette $ brew install simonw/datasette/datasette $ datasette --version datasette, version 0.46 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204, https://github.com/simonw/datasette/issues/335#issuecomment-671733187,https://api.github.com/repos/simonw/datasette/issues/335,671733187,MDEyOklzc3VlQ29tbWVudDY3MTczMzE4Nw==,9599,2020-08-11T05:25:23Z,2020-08-11T05:25:23Z,OWNER,I got this almost working in `simonw/homebrew-datasette` - see https://github.com/simonw/homebrew-datasette/issues/2 for the last missing detail.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204, https://github.com/simonw/sqlite-utils/issues/132#issuecomment-671151461,https://api.github.com/repos/simonw/sqlite-utils/issues/132,671151461,MDEyOklzc3VlQ29tbWVudDY3MTE1MTQ2MQ==,9599,2020-08-10T03:54:06Z,2020-08-10T03:54:06Z,OWNER,For the moment I'll build it without the `--retry` mode.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675839512, https://github.com/simonw/sqlite-utils/issues/132#issuecomment-671151170,https://api.github.com/repos/simonw/sqlite-utils/issues/132,671151170,MDEyOklzc3VlQ29tbWVudDY3MTE1MTE3MA==,9599,2020-08-10T03:52:02Z,2020-08-10T03:52:02Z,OWNER,"I'm having trouble figuring out how to write a test that locks a SQLite database (so I can test that `--retry` actually works). I tried this recipe but it didn't seem to prevent another process from running `pragma journal_mode='wal';` against that database: ```python import time import sys import sqlite3 filename = sys.argv[-1] db = sqlite3.connect(filename) with db: db.execute(""create table if not exists counter(id integer primary key, counter text)"") time.sleep(100) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675839512, https://github.com/simonw/sqlite-utils/issues/132#issuecomment-671147344,https://api.github.com/repos/simonw/sqlite-utils/issues/132,671147344,MDEyOklzc3VlQ29tbWVudDY3MTE0NzM0NA==,9599,2020-08-10T03:29:00Z,2020-08-10T03:29:00Z,OWNER,"The CLI options should take multiple database files: $ sqlite-utils enable-wal *.db It's possible for this to fail if the DB is locked. How about a `--retry` option that causes it to retry a bunch of times if that happens?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675839512, https://github.com/simonw/sqlite-utils/issues/132#issuecomment-671147148,https://api.github.com/repos/simonw/sqlite-utils/issues/132,671147148,MDEyOklzc3VlQ29tbWVudDY3MTE0NzE0OA==,9599,2020-08-10T03:27:50Z,2020-08-10T03:27:50Z,OWNER,"https://www.sqlite.org/pragma.html#pragma_journal_mode lists six modes: DELETE | TRUNCATE | PERSIST | MEMORY | WAL | OFF I'm only going to implement utilities for DELETE (wal-off) and WAL (wal-on) - the other modes look like they're for specialist purposes that I don't need to support. If it turns out I do need them I can add those to `sqlite-utils` later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675839512, https://github.com/simonw/sqlite-utils/issues/132#issuecomment-671146948,https://api.github.com/repos/simonw/sqlite-utils/issues/132,671146948,MDEyOklzc3VlQ29tbWVudDY3MTE0Njk0OA==,9599,2020-08-10T03:26:51Z,2020-08-10T03:26:51Z,OWNER,"For the CLI: $ sqlite-utils enable-wal github.db $ sqlite-utils disable-wal github.db For the Python library: ```python import sqlite_utils db = sqlite_utils.Database(""github.db"") db.enable_wal() db.disable_wal() mode = db.journal_mode # ""wal"" or ""delete"" or others ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675839512, https://github.com/simonw/sqlite-utils/issues/131#issuecomment-671088832,https://api.github.com/repos/simonw/sqlite-utils/issues/131,671088832,MDEyOklzc3VlQ29tbWVudDY3MTA4ODgzMg==,9599,2020-08-09T19:00:41Z,2020-08-09T19:00:41Z,OWNER,"Should be consistent with the `create-table` command as much as possible: ``` $ sqlite-utils create-table mydb.db mytable \ id integer \ name text \ age integer \ is_good integer \ --not-null name \ --not-null age \ --default is_good 1 \ --pk=id ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675753042, https://github.com/simonw/datasette/issues/335#issuecomment-671077168,https://api.github.com/repos/simonw/datasette/issues/335,671077168,MDEyOklzc3VlQ29tbWVudDY3MTA3NzE2OA==,9599,2020-08-09T17:10:15Z,2020-08-09T18:13:39Z,OWNER,"Here's the issue that explains that warning: https://github.com/pypa/pip/issues/5599 This should fix it (risky): from pip._internal.cli.main import main","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204, https://github.com/simonw/datasette/issues/335#issuecomment-671076975,https://api.github.com/repos/simonw/datasette/issues/335,671076975,MDEyOklzc3VlQ29tbWVudDY3MTA3Njk3NQ==,9599,2020-08-09T17:08:34Z,2020-08-09T17:09:21Z,OWNER,"Quick prototype of `datasette install`: ```diff diff --git a/datasette/cli.py b/datasette/cli.py index 287195a..95b6eb7 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -231,6 +231,18 @@ def package( call(args) +@cli.command() +@click.argument(""packages"", nargs=-1, required=True) +def install(packages): + ""Install Python packages - e.g. Datasette plugins - into the same environment as Datasett"" + import pip + + try: + pip.main([""install""] + list(packages)) + except SystemExit as e: + pass + + @cli.command() @click.argument(""files"", type=click.Path(exists=True), nargs=-1) @click.option( ``` ``` $ datasette install Usage: datasette install [OPTIONS] PACKAGES... Try 'datasette install --help' for help. Error: Missing argument 'PACKAGES...'. $ datasette install datasette-vega WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip. Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue. To avoid this problem you can invoke Python with '-m pip' instead of running pip directly. Collecting datasette-vega Using cached datasette_vega-0.6.2-py3-none-any.whl (1.8 MB) ... ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204, https://github.com/simonw/datasette/issues/335#issuecomment-671005731,https://api.github.com/repos/simonw/datasette/issues/335,671005731,MDEyOklzc3VlQ29tbWVudDY3MTAwNTczMQ==,9599,2020-08-09T04:44:13Z,2020-08-09T17:04:21Z,OWNER,"Telling people how to figure out that `pip` location is going to be pretty unpleasant. How about instead providing a `datasette plugins --install=datasette-graphql` command? Or `datasette install datasette-vega` It would run `pip install` in the same virtualenv as Datasette itself. http://jelly.codes/articles/python-pip-module/ shows how to do this: ```python import pip try: pip.main([""install"", ""plumbum""]) except SystemExit as e: pass ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204, https://github.com/simonw/datasette/issues/918#issuecomment-671075764,https://api.github.com/repos/simonw/datasette/issues/918,671075764,MDEyOklzc3VlQ29tbWVudDY3MTA3NTc2NA==,9599,2020-08-09T16:56:48Z,2020-08-09T16:56:48Z,OWNER,GitHub security advisory: https://github.com/simonw/datasette/security/advisories/GHSA-q6j3-c4wc-63vw,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675724951, https://github.com/simonw/datasette/issues/915#issuecomment-671073223,https://api.github.com/repos/simonw/datasette/issues/915,671073223,MDEyOklzc3VlQ29tbWVudDY3MTA3MzIyMw==,9599,2020-08-09T16:35:20Z,2020-08-09T16:36:10Z,OWNER,"`datasette-graphql` uses the logic from `TableView` right now. It wasn't too unpleasant, but I do worry about the two of them being coupled together in this way. https://github.com/simonw/datasette-graphql/blob/cc65ec294b0bf8e26213fc68bb5487066de9caab/datasette_graphql/utils.py#L412-L417 ```python request = Request.fake(path_with_query_string) view = TableView(DatasetteSpecialConfig(datasette)) data, _, _ = await view.data( request, database=database_name, hash=None, table=table_name, _next=after ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",671763164, https://github.com/simonw/datasette/issues/919#issuecomment-671072223,https://api.github.com/repos/simonw/datasette/issues/919,671072223,MDEyOklzc3VlQ29tbWVudDY3MTA3MjIyMw==,9599,2020-08-09T16:26:17Z,2020-08-09T16:26:17Z,OWNER,Should be released in a couple of minutes: https://travis-ci.org/github/simonw/datasette/builds/716328883,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",675727366, https://github.com/simonw/datasette/issues/918#issuecomment-671071710,https://api.github.com/repos/simonw/datasette/issues/918,671071710,MDEyOklzc3VlQ29tbWVudDY3MTA3MTcxMA==,9599,2020-08-09T16:21:41Z,2020-08-09T16:21:41Z,OWNER,Submitting the form on https://latest.datasette.io/fixtures/neighborhood_search demonstrates that this is fixed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675724951, https://github.com/simonw/datasette/issues/919#issuecomment-671071461,https://api.github.com/repos/simonw/datasette/issues/919,671071461,MDEyOklzc3VlQ29tbWVudDY3MTA3MTQ2MQ==,9599,2020-08-09T16:19:37Z,2020-08-09T16:19:37Z,OWNER,That appears to have worked.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675727366, https://github.com/simonw/datasette/issues/918#issuecomment-671070528,https://api.github.com/repos/simonw/datasette/issues/918,671070528,MDEyOklzc3VlQ29tbWVudDY3MTA3MDUyOA==,9599,2020-08-09T16:12:16Z,2020-08-09T16:12:16Z,OWNER,"It's worth noting that in order to exploit this issue the following would all need to be true: - A user is running a copy of Datasette protected by a cookie-based authentication plugin AND configured with at least one writable canned query - An attacker is in control of a URL that could concievably be returned on a page that is displayed as the result of submitting a read-only canned query - An authenticated user of that Datasette instance, who is running a browser that doesn't support the `SameSite=lax` cookie parameter (which is [widely supported](https://caniuse.com/#feat=same-site-cookie-attribute) by modern browsers), submits the read-only canned query form and then clicks a link to the attacker's off-site page, exposing their CSRFToken in the attacker's HTTP referer logs - The attacker then tricks that user into visiting their own malicious web page which includes a POST form that auto-submits against the writable canned query that the attacker wishes to exploit, including the CSRF token as a hidden field The attacker would need full knowledge of the URL and form layout of the Datasette instance that they are exploiting. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675724951, https://github.com/simonw/datasette/issues/918#issuecomment-671070486,https://api.github.com/repos/simonw/datasette/issues/918,671070486,MDEyOklzc3VlQ29tbWVudDY3MTA3MDQ4Ng==,9599,2020-08-09T16:11:59Z,2020-08-09T16:11:59Z,OWNER,Fix has been released in Datasette 0.46: https://datasette.readthedocs.io/en/latest/changelog.html#v0-46,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675724951, https://github.com/simonw/datasette/issues/335#issuecomment-671001457,https://api.github.com/repos/simonw/datasette/issues/335,671001457,MDEyOklzc3VlQ29tbWVudDY3MTAwMTQ1Nw==,9599,2020-08-09T03:37:39Z,2020-08-09T03:37:39Z,OWNER,"Here's what happened when I installed `homebrew-vd`: https://gist.github.com/simonw/7bfd971a62743d7ca248e6b5e696c240 It worked! And from digging around, it has a virtual environment at `/usr/local/Cellar/visidata/1.5.2/libexec/` Which means `/usr/local/Cellar/visidata/1.5.2/libexec/bin/pip` is a working `pip` And I tried running these commands and confirmed that I get a `datasette` with an additional plugin: ``` /usr/local/Cellar/visidata/1.5.2/libexec/bin/pip install datasette /usr/local/Cellar/visidata/1.5.2/libexec/bin/pip install datasette-graphql /usr/local/Cellar/visidata/1.5.2/libexec/bin/datasette plugins [ { ""name"": ""datasette-graphql"", ""static"": false, ""templates"": true, ""version"": ""0.11"", ""hooks"": [ ""register_routes"", ""startup"" ] } ] ``` So I can package Datasette as a homebrew package AND I can give people instructions for installing plugins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204, https://github.com/simonw/datasette/issues/335#issuecomment-670999860,https://api.github.com/repos/simonw/datasette/issues/335,670999860,MDEyOklzc3VlQ29tbWVudDY3MDk5OTg2MA==,9599,2020-08-09T03:12:44Z,2020-08-09T03:12:44Z,OWNER,How would plugin installation work if Datasette was installed via homebrew?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204, https://github.com/simonw/datasette/issues/335#issuecomment-670999832,https://api.github.com/repos/simonw/datasette/issues/335,670999832,MDEyOklzc3VlQ29tbWVudDY3MDk5OTgzMg==,9599,2020-08-09T03:12:14Z,2020-08-09T03:12:14Z,OWNER,Another useful example: https://github.com/Homebrew/homebrew-core/blob/master/Formula/trailscraper.rb,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204, https://github.com/simonw/sqlite-utils/issues/130#issuecomment-667585598,https://api.github.com/repos/simonw/sqlite-utils/issues/130,667585598,MDEyOklzc3VlQ29tbWVudDY2NzU4NTU5OA==,9599,2020-08-01T20:51:28Z,2020-08-01T20:51:28Z,OWNER,CLI documentation: https://github.com/simonw/sqlite-utils/commit/57e4eb8e5564af5d97f892b3be8342451ee177a2?short_path=7240b7c#diff-7240b7c71b1a8194da0c001c64fc8d40,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",671130371, https://github.com/simonw/sqlite-utils/issues/130#issuecomment-667585561,https://api.github.com/repos/simonw/sqlite-utils/issues/130,667585561,MDEyOklzc3VlQ29tbWVudDY2NzU4NTU2MQ==,9599,2020-08-01T20:50:59Z,2020-08-01T20:50:59Z,OWNER,Turns out it works for FTS4 as well: https://www.sqlite.org/fts3.html#tokenizer,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",671130371, https://github.com/simonw/sqlite-utils/issues/130#issuecomment-667584567,https://api.github.com/repos/simonw/sqlite-utils/issues/130,667584567,MDEyOklzc3VlQ29tbWVudDY2NzU4NDU2Nw==,9599,2020-08-01T20:41:09Z,2020-08-01T20:41:09Z,OWNER,API documentation here: https://github.com/simonw/sqlite-utils/commit/617e6f070c85be66ea04c80b78dafd08c875f8c8?short_path=e156262#diff-e1562629b8def6da772d9b0903faf703,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",671130371, https://github.com/simonw/datasette/issues/900#issuecomment-667431123,https://api.github.com/repos/simonw/datasette/issues/900,667431123,MDEyOklzc3VlQ29tbWVudDY2NzQzMTEyMw==,9599,2020-07-31T23:56:33Z,2020-07-31T23:56:33Z,OWNER,I think this is the same issue as #865. I'll look at these together!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",661605489, https://github.com/simonw/datasette/issues/899#issuecomment-667430790,https://api.github.com/repos/simonw/datasette/issues/899,667430790,MDEyOklzc3VlQ29tbWVudDY2NzQzMDc5MA==,9599,2020-07-31T23:54:40Z,2020-07-31T23:54:40Z,OWNER,"There's no mechanism that can do this at the moment. You could absolutely support this with a plugin, probably using the `asgi_wrapper` plugin hook. There's an existing package at https://pypi.org/project/asgi-ratelimit/ which may be usable for this - it may even be possible to configure that using https://github.com/simonw/datasette-configure-asgi rather than using it to write a custom plugin. Using a separate revers proxy would also be a good way to solve this. It depends which option would work best in your environment. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",660827546, https://github.com/simonw/datasette/issues/913#issuecomment-667430352,https://api.github.com/repos/simonw/datasette/issues/913,667430352,MDEyOklzc3VlQ29tbWVudDY2NzQzMDM1Mg==,9599,2020-07-31T23:52:10Z,2020-07-31T23:52:10Z,OWNER,The bigger question here is when this mechanism should be used in place of `metadata.json` or `metadata.yml`. Especially since I'm already considering renaming or reworking that mechanism since plugin configuration has nothing to do with database metadata: #493,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",670209331, https://github.com/simonw/datasette/issues/913#issuecomment-667429616,https://api.github.com/repos/simonw/datasette/issues/913,667429616,MDEyOklzc3VlQ29tbWVudDY2NzQyOTYxNg==,9599,2020-07-31T23:48:25Z,2020-07-31T23:49:59Z,OWNER,"I could let plugins add additional options to `datasette serve` - but what if two plugins both try to register an option with the same name? A better solution could be to use the existing `--config` option - and allow plugins to register their own, namespaced config options. So you could do things like: datasette my.db --config datasette-insert:unsafe:1 Maybe even drop the `datasette-` prefix? datasette my.db --config insert:unsafe:1 I think I prefer keeping the prefix to be honest - it makes it more obvious that this is a setting which comes from a specific named plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",670209331, https://github.com/simonw/datasette/issues/913#issuecomment-667429690,https://api.github.com/repos/simonw/datasette/issues/913,667429690,MDEyOklzc3VlQ29tbWVudDY2NzQyOTY5MA==,9599,2020-07-31T23:48:48Z,2020-07-31T23:48:48Z,OWNER,"Here's the code in Datasette that parses `--config` options at the moment: https://github.com/simonw/datasette/blob/7ca8c0521ac1ea48a3cd8d0fe9275d1316e54b43/datasette/cli.py#L25-L40","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",670209331, https://github.com/simonw/datasette/issues/849#issuecomment-667424128,https://api.github.com/repos/simonw/datasette/issues/849,667424128,MDEyOklzc3VlQ29tbWVudDY2NzQyNDEyOA==,9599,2020-07-31T23:21:56Z,2020-07-31T23:23:24Z,OWNER,"I'm going to change the default branch on the GitHub repository. If something breaks I can always change it back again. Done that! Default is now `main`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639072811, https://github.com/simonw/datasette/issues/849#issuecomment-667424020,https://api.github.com/repos/simonw/datasette/issues/849,667424020,MDEyOklzc3VlQ29tbWVudDY2NzQyNDAyMA==,9599,2020-07-31T23:21:30Z,2020-07-31T23:21:30Z,OWNER,https://github.com/simonw/datasette/tree/main branch now exists and will automatically mirror master (and vice-versa).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639072811, https://github.com/simonw/datasette/issues/849#issuecomment-667295759,https://api.github.com/repos/simonw/datasette/issues/849,667295759,MDEyOklzc3VlQ29tbWVudDY2NzI5NTc1OQ==,9599,2020-07-31T18:45:35Z,2020-07-31T18:45:35Z,OWNER,"Watch out for places in the documentation that might link to `master` - e.g. here: https://github.com/simonw/datasette/blob/2d7fa8b9058dfbf9c7c371cdeec115d32a177dc9/docs/custom_templates.rst#L247","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639072811, https://github.com/simonw/sqlite-utils/issues/124#issuecomment-664105302,https://api.github.com/repos/simonw/sqlite-utils/issues/124,664105302,MDEyOklzc3VlQ29tbWVudDY2NDEwNTMwMg==,9599,2020-07-27T03:54:24Z,2020-07-30T22:57:51Z,OWNER,"Documentation: https://github.com/simonw/sqlite-utils/commit/814d4a7f90991be865d38aac45ff12e36df1c67d?short_path=7240b7c#diff-7240b7c71b1a8194da0c001c64fc8d40 > You can pass named parameters to the query using -p: > > $ sqlite-utils query dogs.db ""select :num * :num2"" -p num 5 -p num2 6 > [{"":num * :num2"": 30}] ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665802405, https://github.com/simonw/sqlite-utils/issues/129#issuecomment-666752039,https://api.github.com/repos/simonw/sqlite-utils/issues/129,666752039,MDEyOklzc3VlQ29tbWVudDY2Njc1MjAzOQ==,9599,2020-07-30T22:40:55Z,2020-07-30T22:40:55Z,OWNER,"This should be a separate command from `insert-files`. SQLite Archives should use a table with this schema: ```sql CREATE TABLE sqlar( name TEXT PRIMARY KEY, -- name of the file mode INT, -- access permissions mtime INT, -- last modification time sz INT, -- original file size data BLOB -- compressed content ); ``` `insert-files` currently treats the table name as a required argument - but it's not necessary for this table. Also there shouldn't be any support for the `--column` option. So if I write this command it should be this instead: sqlite-utils sqlar files.db file.txt file2.txt But at that point, why bother? Users can use `sqlite3 files.db -Ac *.txt` instead. So I'm not going to bother implementing this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",668308777, https://github.com/simonw/sqlite-utils/issues/127#issuecomment-666063689,https://api.github.com/repos/simonw/sqlite-utils/issues/127,666063689,MDEyOklzc3VlQ29tbWVudDY2NjA2MzY4OQ==,9599,2020-07-30T03:08:51Z,2020-07-30T03:08:51Z,OWNER,Documentation at the bottom of this section: https://github.com/simonw/sqlite-utils/blob/8fe1e6d1be021aeeb8f08b0f77f03b75a83b6f75/docs/cli.rst#inserting-binary-data-from-files,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",666040390, https://github.com/simonw/sqlite-utils/issues/127#issuecomment-666047928,https://api.github.com/repos/simonw/sqlite-utils/issues/127,666047928,MDEyOklzc3VlQ29tbWVudDY2NjA0NzkyOA==,9599,2020-07-30T02:31:05Z,2020-07-30T02:31:05Z,OWNER,"Maybe could do this using an improved version of this lambda? Could teach it to look for `-` and read from `sys.stdin` if it sees it. https://github.com/simonw/sqlite-utils/blob/710454d72aed5094573e642344fd075a0ef5372c/sqlite_utils/cli.py#L839","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",666040390, https://github.com/simonw/sqlite-utils/issues/129#issuecomment-666046819,https://api.github.com/repos/simonw/sqlite-utils/issues/129,666046819,MDEyOklzc3VlQ29tbWVudDY2NjA0NjgxOQ==,9599,2020-07-30T02:28:34Z,2020-07-30T02:28:34Z,OWNER,This code looks useful as inspiration: https://github.com/j4mie/sqlsite/blob/f2dadb8db5ed7880f8872b6591d8cb1487f777ea/sqlsite/sqlar.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",668308777, https://github.com/simonw/datasette/issues/909#issuecomment-666010395,https://api.github.com/repos/simonw/datasette/issues/909,666010395,MDEyOklzc3VlQ29tbWVudDY2NjAxMDM5NQ==,9599,2020-07-30T00:56:17Z,2020-07-30T00:56:17Z,OWNER,"``` $ curl -I https://latest.datasette.io/fixtures.db HTTP/1.1 200 OK content-disposition: attachment; filename=""fixtures.db"" content-type: application/octet-stream Date: Thu, 30 Jul 2020 00:56:05 GMT Server: Google Frontend Transfer-Encoding: chunked ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",667467128, https://github.com/simonw/datasette/issues/909#issuecomment-665854704,https://api.github.com/repos/simonw/datasette/issues/909,665854704,MDEyOklzc3VlQ29tbWVudDY2NTg1NDcwNA==,9599,2020-07-29T19:22:31Z,2020-07-29T19:22:31Z,OWNER,"I think this results in a bug where the ""download database"" link doesn't include the correct filename: https://github.com/simonw/datasette/blob/549b1c2063db48c4622ee5c7b478a1e3cbc1ac07/datasette/views/database.py#L110-L131","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",667467128, https://github.com/simonw/sqlite-utils/issues/128#issuecomment-664683608,https://api.github.com/repos/simonw/sqlite-utils/issues/128,664683608,MDEyOklzc3VlQ29tbWVudDY2NDY4MzYwOA==,9599,2020-07-27T23:09:22Z,2020-07-27T23:09:22Z,OWNER,"This seems to work, but needs more tests: ```diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index d6b9ecf..ee26433 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -7,6 +7,7 @@ import itertools import json import os import pathlib +import uuid SQLITE_MAX_VARS = 999 @@ -40,11 +41,13 @@ COLUMN_TYPE_MAPPING = { str: ""TEXT"", bytes.__class__: ""BLOB"", bytes: ""BLOB"", + memoryview: ""BLOB"", datetime.datetime: ""TEXT"", datetime.date: ""TEXT"", datetime.time: ""TEXT"", decimal.Decimal: ""FLOAT"", None.__class__: ""TEXT"", + uuid.UUID: ""TEXT"", # SQLite explicit types ""TEXT"": ""TEXT"", ""INTEGER"": ""INTEGER"", @@ -1336,6 +1339,8 @@ def jsonify_if_needed(value): return json.dumps(value, default=repr) elif isinstance(value, (datetime.time, datetime.date, datetime.datetime)): return value.isoformat() + elif isinstance(value, uuid.UUID): + return str(value) else: return value ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",666639051, https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664163524,https://api.github.com/repos/simonw/sqlite-utils/issues/122,664163524,MDEyOklzc3VlQ29tbWVudDY2NDE2MzUyNA==,9599,2020-07-27T07:10:41Z,2020-07-27T07:10:41Z,OWNER,Docs: https://github.com/simonw/sqlite-utils/blob/ebc802f7ff0e640b6ae11ea525290fea0115228c/docs/cli.rst#inserting-binary-data-from-files,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495, https://github.com/simonw/sqlite-utils/issues/127#issuecomment-664163206,https://api.github.com/repos/simonw/sqlite-utils/issues/127,664163206,MDEyOklzc3VlQ29tbWVudDY2NDE2MzIwNg==,9599,2020-07-27T07:10:05Z,2020-07-27T07:10:05Z,OWNER,I tried to get this working but it was a bit tricky because `-` doesn't behave like a regular `pathlib.Path` - needs a bit more thought on how the implementation would work.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",666040390, https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664128071,https://api.github.com/repos/simonw/sqlite-utils/issues/122,664128071,MDEyOklzc3VlQ29tbWVudDY2NDEyODA3MQ==,9599,2020-07-27T05:30:54Z,2020-07-27T05:30:54Z,OWNER,"Inserting files by piping them in should work - but since a filename cannot be derived this will need a `--name blah.gif` option. cat blah.gif | sqlite-utils insert-files files.db files - --name=blah.gif","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495, https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664127741,https://api.github.com/repos/simonw/sqlite-utils/issues/122,664127741,MDEyOklzc3VlQ29tbWVudDY2NDEyNzc0MQ==,9599,2020-07-27T05:29:48Z,2020-07-27T05:29:48Z,OWNER,"Test command: ``` sqlite-utils insert-files gifs.db *.gif \ -c filename:filename \ -c filepath:filepath \ -c absolutepath:absolutepath \ -c sha256:sha256 \ -c md5:md5 \ -c content:content \ -c mtime:mtime \ -c ctime:ctime \ -c mtime_iso:mtime_iso \ -c ctime_iso:ctime_iso \ -c size:size \ --pk absolutepath ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495, https://github.com/simonw/sqlite-utils/issues/122#issuecomment-663931279,https://api.github.com/repos/simonw/sqlite-utils/issues/122,663931279,MDEyOklzc3VlQ29tbWVudDY2MzkzMTI3OQ==,9599,2020-07-26T03:33:23Z,2020-07-27T04:30:49Z,OWNER,"One idea: `sqlite-utils insert-files` It could work something like this: sqlite-utils insert-files files.db /tmp/blah.jpg /tmp/foo.gif \ --table files \ -c key:filename -c hash:sha256 -c body:content \ --pk key This would insert those two image files into the database in a table called `files` with a schema that looks something like this: ```sql CREATE TABLE files ( key text primary key, hash text, body blob ); ``` The `-c key:filename` options here are the most interesting: they let you create the table with a specific layout. The bit before the `:` is the column name. The bit after the `:` can be a range of different things: - `filename` - just the filename - `filepath` - the full filepath (provided on the command-line) - `absolutepath` - the filepath expanded to start with `/home/...` or whatever - `sha256` - the SHA256 of the contents - `md5` - the MD5 - `content` - the binary content itself - `mtime` - the mtime (floating point timestamp) - `ctime` - the ctime (floating point timestamp) - `mtime_iso` - the mtime as an ISO datetime - `ctime_iso` - the mtime as an ISO datetime - `size` - the size of the file in bytes","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-664106621,https://api.github.com/repos/simonw/sqlite-utils/issues/114,664106621,MDEyOklzc3VlQ29tbWVudDY2NDEwNjYyMQ==,9599,2020-07-27T04:01:13Z,2020-07-27T04:01:13Z,OWNER,Work in progress in `transform` branch here: https://github.com/simonw/sqlite-utils/tree/transform,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740, https://github.com/simonw/sqlite-utils/issues/126#issuecomment-664106405,https://api.github.com/repos/simonw/sqlite-utils/issues/126,664106405,MDEyOklzc3VlQ29tbWVudDY2NDEwNjQwNQ==,9599,2020-07-27T04:00:08Z,2020-07-27T04:00:33Z,OWNER,"``` $ echo '[ { ""name"": ""transparent.gif"", ""content"": { ""$base64"": true, ""encoded"": ""R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7"" } } ]' | sqlite-utils insert trans.db files - --pk=name ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665819048, https://github.com/simonw/sqlite-utils/issues/126#issuecomment-664065597,https://api.github.com/repos/simonw/sqlite-utils/issues/126,664065597,MDEyOklzc3VlQ29tbWVudDY2NDA2NTU5Nw==,9599,2020-07-27T00:51:11Z,2020-07-27T00:51:11Z,OWNER,"I'm going to implement this as the reverse of #125 - binary columns in JSON are now output like this: ```json { ""name"": ""lorem.txt"", ""mode"": 33188, ""mtime"": 1595805965, ""sz"": 16984, ""data"": { ""$base64"": true, ""encoded"": ""eJzt0c1xAyEMBeC7q1ABHleR3HxNAQrIjmb4M0gelx+RTY7p4N2WBYT0vmufUknH8kq5lz5pqRFXsTOl3pYkE/NJnHXoStruJEVjc0mOCyTqq/ZMJnXEZW1Js2ZvRm5U+DPKk9hRWqjyvTFx0YfzhT6MpGmN2lR1fzxjyfVMD9dFrS+bnkleMpMam/ZGXgrX1I/K+5Au3S/9lNQRh0k4Gq/RUz8GiKfsQm+7JLsJ6fTo5JhVG00ZU76kZZkxePx49uIjnpNoJyYlWUsoaSl/CcVATje/Kxu13RANnrHweaH3V5Jh4jvGyKCnxJLiXPKhmW3fiCnG7Jql7RR3UvFo8jJ4z039dtOkTFmWzL1be9lt8A5II471m6vXy+l0BR/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984PuP7xubBoN9"" } } ] ``` So the `sqlite-utils insert` command should learn to spot `{""$base64"": true...}` values and base64 decode them before inserting them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665819048, https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664065341,https://api.github.com/repos/simonw/sqlite-utils/issues/125,664065341,MDEyOklzc3VlQ29tbWVudDY2NDA2NTM0MQ==,9599,2020-07-27T00:49:41Z,2020-07-27T00:49:41Z,OWNER,Documentation: https://github.com/simonw/sqlite-utils/commit/20e543e9a492f2e764caae73c38e87f18eaec444?short_path=7240b7c#diff-7240b7c71b1a8194da0c001c64fc8d40,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665817570, https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664062546,https://api.github.com/repos/simonw/sqlite-utils/issues/125,664062546,MDEyOklzc3VlQ29tbWVudDY2NDA2MjU0Ng==,9599,2020-07-27T00:33:03Z,2020-07-27T00:33:03Z,OWNER,"I'm going to imitate how Datasette solves this problem: ```json [ { ""name"": ""lorem.txt"", ""mode"": 33188, ""mtime"": 1595805965, ""sz"": 16984, ""data"": { ""$base64"": true, ""encoded"": ""eJzt0c1xAyEMBeC7q1ABHleR3HxNAQrIjmb4M0gelx+RTY7p4N2WBYT0vmufUknH8kq5lz5pqRFXsTOl3pYkE/NJnHXoStruJEVjc0mOCyTqq/ZMJnXEZW1Js2ZvRm5U+DPKk9hRWqjyvTFx0YfzhT6MpGmN2lR1fzxjyfVMD9dFrS+bnkleMpMam/ZGXgrX1I/K+5Au3S/9lNQRh0k4Gq/RUz8GiKfsQm+7JLsJ6fTo5JhVG00ZU76kZZkxePx49uIjnpNoJyYlWUsoaSl/CcVATje/Kxu13RANnrHweaH3V5Jh4jvGyKCnxJLiXPKhmW3fiCnG7Jql7RR3UvFo8jJ4z039dtOkTFmWzL1be9lt8A5II471m6vXy+l0BR/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984PuP7xubBoN9"" } } ] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665817570, https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664048720,https://api.github.com/repos/simonw/sqlite-utils/issues/122,664048720,MDEyOklzc3VlQ29tbWVudDY2NDA0ODcyMA==,9599,2020-07-26T22:32:50Z,2020-07-26T22:33:20Z,OWNER,"This seems to work in creating a SQLite archive containing all `.gif` files in the current directory: /usr/local/Cellar/sqlite/3.32.1/bin/sqlite3 archive.db -A -c *.gif Then listing files like this: ``` $ /usr/local/Cellar/sqlite/3.32.1/bin/sqlite3 archive.db -A -t copyable.gif debug-allow.gif flash.gif table-md.gif ``` Here's the schema: ``` $ sqlite3 archive.db .schema CREATE TABLE sqlar( name TEXT PRIMARY KEY, -- name of the file mode INT, -- access permissions mtime INT, -- last modification time sz INT, -- original file size data BLOB -- compressed content ); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495, https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664048432,https://api.github.com/repos/simonw/sqlite-utils/issues/122,664048432,MDEyOklzc3VlQ29tbWVudDY2NDA0ODQzMg==,9599,2020-07-26T22:29:31Z,2020-07-26T22:29:31Z,OWNER,"I'm trying to play with `sqlite3 -A` on my Mac. `sqlite3 -A` tells me that it's an unknown option - but I used `brew info sqlite` to find my homebrew installed version and it turns out this works: ``` % /usr/local/Cellar/sqlite/3.32.1/bin/sqlite3 -A Wrong number of arguments. Usage: .archive ... Manage SQL archives Each command must have exactly one of the following options: -c, --create Create a new archive -u, --update Add or update files with changed mtime -i, --insert Like -u but always add even if unchanged -t, --list List contents of archive -x, --extract Extract files from archive Optional arguments: -v, --verbose Print each filename as it is processed -f FILE, --file FILE Use archive FILE (default is current db) -a FILE, --append FILE Open FILE using the apndvfs VFS -C DIR, --directory DIR Read/extract files from directory DIR -n, --dryrun Show the SQL that would have occurred Examples: .ar -cf ARCHIVE foo bar # Create ARCHIVE from files foo and bar .ar -tf ARCHIVE # List members of ARCHIVE .ar -xvf ARCHIVE # Verbosely extract files from ARCHIVE See also: http://sqlite.org/cli.html#sqlar_archive_support ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495, https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664013338,https://api.github.com/repos/simonw/sqlite-utils/issues/122,664013338,MDEyOklzc3VlQ29tbWVudDY2NDAxMzMzOA==,9599,2020-07-26T16:57:35Z,2020-07-26T16:57:35Z,OWNER,"I should consider easy compatibility with https://www.sqlite.org/sqlar.html > An SQLite Archive is an ordinary SQLite database file that contains the following table as part of its schema: > ``` > CREATE TABLE sqlar( > name TEXT PRIMARY KEY, -- name of the file > mode INT, -- access permissions > mtime INT, -- last modification time > sz INT, -- original file size > data BLOB -- compressed content > ); > ``` > Each row of the SQLAR table holds the content of a single file. The filename (the full pathname relative to the root of the archive) is in the ""name"" field. The ""mode"" field is an integer which is the unix-style access permissions for the file. ""mtime"" is the modification time of the file in seconds since 1970. ""sz"" is the original uncompressed size of the file. The ""data"" field contains the file content. The content is usually compressed using [Deflate](http://zlib.net/), though not always. If the ""sz"" field is equal to the size of the ""data"" field, then the content is stored uncompressed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495, https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664012247,https://api.github.com/repos/simonw/sqlite-utils/issues/125,664012247,MDEyOklzc3VlQ29tbWVudDY2NDAxMjI0Nw==,9599,2020-07-26T16:48:46Z,2020-07-26T16:48:46Z,OWNER,"I could solve round tripping (at least a bit) by allowing insert to be run with a flag that says ""these columns are base64 encoded, store the decoded data in a BLOB"". That would solve inserting binary data using JSON too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665817570, https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664012148,https://api.github.com/repos/simonw/sqlite-utils/issues/125,664012148,MDEyOklzc3VlQ29tbWVudDY2NDAxMjE0OA==,9599,2020-07-26T16:47:51Z,2020-07-26T16:47:51Z,OWNER,Best solution I can think of is to return the data as base64. It's a bit nasty since it means you can't round trip it back again.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665817570, https://github.com/simonw/sqlite-utils/issues/122#issuecomment-663931426,https://api.github.com/repos/simonw/sqlite-utils/issues/122,663931426,MDEyOklzc3VlQ29tbWVudDY2MzkzMTQyNg==,9599,2020-07-26T03:35:58Z,2020-07-26T16:44:33Z,OWNER,Related: #123 (`--raw` option),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495, https://github.com/simonw/sqlite-utils/issues/122#issuecomment-663931662,https://api.github.com/repos/simonw/sqlite-utils/issues/122,663931662,MDEyOklzc3VlQ29tbWVudDY2MzkzMTY2Mg==,9599,2020-07-26T03:40:29Z,2020-07-26T03:40:29Z,OWNER,Maybe support `--replace` for replacing images with an existing primary key.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495,