{"html_url": "https://github.com/simonw/datasette/issues/983#issuecomment-752729035", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/983", "id": 752729035, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjcyOTAzNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-30T19:15:56Z", "updated_at": "2020-12-30T19:16:44Z", "author_association": "OWNER", "body": "The `column_actions` hook is the obvious first place to try this out. What are some demo plugins I could build for it?\r\n\r\n- Word cloud for this column\r\n- Count values (essentially linking to the SQL query for that column, as an extended version of the facet counts) - would be great if this could include pagination somehow, via #856.\r\n- Extract this column into a separate table\r\n- Add an index to this column", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 712260429, "label": "JavaScript plugin hooks mechanism similar to pluggy"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/983#issuecomment-752722863", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/983", "id": 752722863, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjcyMjg2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-30T18:52:39Z", "updated_at": "2020-12-30T18:52:39Z", "author_association": "OWNER", "body": "Then to call the plugins:\r\n```javascript\r\ndatasette.plugins.call('column_actions', {database: 'database', table: 'table'})\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 712260429, "label": "JavaScript plugin hooks mechanism similar to pluggy"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/983#issuecomment-752721840", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/983", "id": 752721840, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjcyMTg0MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-30T18:48:53Z", "updated_at": "2020-12-30T18:51:51Z", "author_association": "OWNER", "body": "Potential design:\r\n```javascript\r\ndatasette.plugins.register('column_actions', function(database, table, column, actor) {\r\n /* ... *l\r\n})\r\n```\r\nOr if you want to be explicit to survive minification:\r\n```javascript\r\ndatasette.plugins.register('column_actions', function(database, table, column, actor) {\r\n /* ... *l\r\n}, ['database', 'table', 'column', 'actor'])\r\n```\r\nI'm making that list of parameter names an optional third argument to the `register()` function. If that argument isn't passed, introspection will be used to figure out the parameter names.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 712260429, "label": "JavaScript plugin hooks mechanism similar to pluggy"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/983#issuecomment-752721069", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/983", "id": 752721069, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjcyMTA2OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-30T18:46:10Z", "updated_at": "2020-12-30T18:46:10Z", "author_association": "OWNER", "body": "Pluggy does dependency injection by introspecting the named arguments to the Python function, which I really like.\r\n\r\nThat's tricker in JavaScript. It looks like the only way to introspect a function is to look at the `.toString()` representation of it and parse the `(parameter, list)` using a regular expression.\r\n\r\nEven more challenging: JavaScript developers love minifying their code, and minification can shorten the function parameter names.\r\n\r\nFrom https://code-maven.com/dependency-injection-in-angularjs it looks like Angular.js does dependency injection and solves this by letting you optionally provide a separate list of the arguments your function uses:\r\n\r\n```javascript\r\n angular.module('DemoApp', [])\r\n .controller('DemoController', ['$scope', '$log', function($scope, $log) {\r\n $scope.message = \"Hello World\";\r\n $log.debug('logging hello');\r\n }]);\r\n```\r\n\r\nI can copy that approach: I'll introspect by default, but provide a documented mechanism for explicitly listing your parameter names so that if you know your plugin code will be minified you can use that instead.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 712260429, "label": "JavaScript plugin hooks mechanism similar to pluggy"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/983#issuecomment-752715412", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/983", "id": 752715412, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjcxNTQxMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-30T18:25:31Z", "updated_at": "2020-12-30T18:25:31Z", "author_association": "OWNER", "body": "I'm going to introduce a global `datasette` object which holds all the documented JavaScript API for plugin authors.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 712260429, "label": "JavaScript plugin hooks mechanism similar to pluggy"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/983#issuecomment-752715236", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/983", "id": 752715236, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjcxNTIzNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-30T18:24:54Z", "updated_at": "2020-12-30T18:24:54Z", "author_association": "OWNER", "body": "I think I'm going to try building a very lightweight clone of the core API design of Pluggy - not the advanced features, just the idea that plugins can register and a call to `plugin.nameOfHook()` will return the concatenated results of all of the registered hooks.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 712260429, "label": "JavaScript plugin hooks mechanism similar to pluggy"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/987#issuecomment-752714747", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/987", "id": 752714747, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjcxNDc0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-30T18:23:08Z", "updated_at": "2020-12-30T18:23:20Z", "author_association": "OWNER", "body": "In terms of \"places to put your plugin content\", the simplest solution I can think of is something like this:\r\n```html\r\n
\r\n```\r\nAlternative designs:\r\n\r\n- A documented JavaScript function that returns the CSS selector where plugins should put their content\r\n- A documented JavaScript function that returns a DOM node where plugins should put their content. This would allow the JavaScript to create the element if it does not already exist (though it wouldn't be obvious WHERE that element should be created)\r\n- Documented JavaScript functions for things like \"append this node/HTML to the place-where-plugins-go\"\r\n\r\nI think the original option - an empty `
` with a known `id` attribute - is the right one to go with here. It's the simplest, it's very easy for custom template authors to understand and it acknowledges that plugins may have all kinds of extra crazy stuff they want to do - like checking in that div to see if another plugin has written to it already, for example.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 712984738, "label": "Documented HTML hooks for JavaScript plugin authors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/987#issuecomment-752697279", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/987", "id": 752697279, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjY5NzI3OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-30T17:23:27Z", "updated_at": "2020-12-30T17:23:32Z", "author_association": "OWNER", "body": "Related problem: right now if you're writing custom template it's not at all obvious how to write them such that visualization plugins like `datasette-cluster-map` will continue to work.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 712984738, "label": "Documented HTML hooks for JavaScript plugin authors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/987#issuecomment-752696499", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/987", "id": 752696499, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjY5NjQ5OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-30T17:21:08Z", "updated_at": "2020-12-30T17:21:08Z", "author_association": "OWNER", "body": "More generally, I need to document certain areas of the page that JavaScript plugins are invited to append their content to - such that plugin authors can use them and feel confident that future changes to the Datasette templates won't break their plugins.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 712984738, "label": "Documented HTML hooks for JavaScript plugin authors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1160#issuecomment-752275611", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1160", "id": 752275611, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjI3NTYxMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T23:32:04Z", "updated_at": "2020-12-29T23:32:04Z", "author_association": "OWNER", "body": "If I can get this working for CSV, TSV, JSON and JSON-NL that should be enough to exercise the API design pretty well across both streaming and non-streaming formats.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 775666296, "label": "\"datasette insert\" command and plugin hook"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1160#issuecomment-752274509", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1160", "id": 752274509, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjI3NDUwOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T23:26:02Z", "updated_at": "2020-12-29T23:26:02Z", "author_association": "OWNER", "body": "The documentation for this plugin hook is going to be pretty detailed, since it involves writing custom classes.\r\n\r\nI'll stick it all on the existing hooks page for the moment, but I should think about breaking up the plugin hook documentation into a page-per-hook in the future.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 775666296, "label": "\"datasette insert\" command and plugin hook"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1163#issuecomment-752274340", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1163", "id": 752274340, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjI3NDM0MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T23:25:02Z", "updated_at": "2020-12-29T23:25:02Z", "author_association": "OWNER", "body": "This will be built on top of `httpx` since that's already a dependency.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 776128565, "label": "\"datasette insert data.db url-to-csv\""}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1160#issuecomment-752274078", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1160", "id": 752274078, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjI3NDA3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T23:23:39Z", "updated_at": "2020-12-29T23:23:39Z", "author_association": "OWNER", "body": "If I design this right I can ship a full version of the command-line `datasette insert` command in a release without doing any work at all on the Web UI version of it - that UI can then come later, without needing any changes to be made to the plugin hook.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 775666296, "label": "\"datasette insert\" command and plugin hook"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1160#issuecomment-752273873", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1160", "id": 752273873, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjI3Mzg3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T23:22:30Z", "updated_at": "2020-12-29T23:22:30Z", "author_association": "OWNER", "body": "How much of this should I get done in a branch before merging into `main`?\r\n\r\nThe challenge here is the plugin hook design: ideally I don't want an incomplete plugin hook design in `main` since that could be a blocker for a release.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 775666296, "label": "\"datasette insert\" command and plugin hook"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1160#issuecomment-752273400", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1160", "id": 752273400, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjI3MzQwMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T23:19:46Z", "updated_at": "2020-12-29T23:19:46Z", "author_association": "OWNER", "body": "I'm going to break out some separate tickets.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 775666296, "label": "\"datasette insert\" command and plugin hook"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1160#issuecomment-752273306", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1160", "id": 752273306, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjI3MzMwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T23:19:15Z", "updated_at": "2020-12-29T23:19:15Z", "author_association": "OWNER", "body": "It would be nice if this abstraction could support progress bars as well. These won't necessarily work for every format - or they might work for things loaded from files but not things loaded over URLs (if the `content-length` HTTP header is missing) - but if they ARE possible it would be good to provide them - both for the CLI interface and the web insert UI.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 775666296, "label": "\"datasette insert\" command and plugin hook"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1160#issuecomment-752267905", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1160", "id": 752267905, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjI2NzkwNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T22:52:09Z", "updated_at": "2020-12-29T22:52:09Z", "author_association": "OWNER", "body": "What's the simplest thing that could possible work? I think it's `datasette insert blah.db data.csv` - no URL handling, no other formats.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 775666296, "label": "\"datasette insert\" command and plugin hook"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1160#issuecomment-752266076", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1160", "id": 752266076, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjI2NjA3Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T22:42:23Z", "updated_at": "2020-12-29T22:42:59Z", "author_association": "OWNER", "body": "Aside: maybe `datasette insert` works against simple files, but a later mechanism called `datasette import` allows plugins to register sub-commands, like `datasette import github ...` or `datasette import jira ...` or whatever.\r\n\r\nThis would be useful for import mechanisms that are likely to need their own custom set of command-line options unique to that source.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 775666296, "label": "\"datasette insert\" command and plugin hook"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1160#issuecomment-752265600", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1160", "id": 752265600, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjI2NTYwMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T22:39:56Z", "updated_at": "2020-12-29T22:39:56Z", "author_association": "OWNER", "body": "Does it definitely make sense to break this operation up into the code that turns the incoming format into a iterator of dictionaries, then the code that inserts those into the database using `sqlite-utils`?\r\n\r\nThat seems right for simple imports, where the incoming file represents a sequence of records in a single table. But what about more complex formats? What if a format needs to be represented as multiple tables?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 775666296, "label": "\"datasette insert\" command and plugin hook"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1160#issuecomment-752259345", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1160", "id": 752259345, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjI1OTM0NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T22:11:54Z", "updated_at": "2020-12-29T22:11:54Z", "author_association": "OWNER", "body": "Important detail from https://docs.python.org/3/library/csv.html#csv.reader\r\n\r\n> If *csvfile* is a file object, it should be opened with `newline=''`. [1]\r\n>\r\n> [...]\r\n>\r\n> If `newline=''` is not specified, newlines embedded inside quoted fields will not be interpreted correctly, and on platforms that use `\\r\\n` linendings on write an extra `\\r` will be added. It should always be safe to specify `newline=''`, since the csv module does its own ([universal](https://docs.python.org/3/glossary.html#term-universal-newlines)) newline handling.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 775666296, "label": "\"datasette insert\" command and plugin hook"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1160#issuecomment-752257666", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1160", "id": 752257666, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjI1NzY2Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T22:09:18Z", "updated_at": "2020-12-29T22:09:18Z", "author_association": "OWNER", "body": "### Figuring out the API design\r\n\r\nI want to be able to support different formats, and be able to parse them into tables either streaming or in one go depending on if the format supports that.\r\n\r\nIdeally I want to be able to pull the first 1,024 bytes for the purpose of detecting the format, then replay those bytes again later. I'm considering this a stretch goal though.\r\n\r\nCSV is easy to parse as a stream - here\u2019s [how sqlite-utils does it](https://github.com/simonw/sqlite-utils/blob/f1277f638f3a54a821db6e03cb980adad2f2fa35/sqlite_utils/cli.py#L630):\r\n\r\n dialect = \"excel-tab\" if tsv else \"excel\"\r\n with file_progress(json_file, silent=silent) as json_file:\r\n reader = csv_std.reader(json_file, dialect=dialect)\r\n headers = next(reader)\r\n docs = (dict(zip(headers, row)) for row in reader)\r\n\r\nProblem: using `db.insert_all()` could block for a long time on a big set of rows. Probably easiest to batch the records before calling `insert_all()` and then run a batch at a time using a `db.execute_write_fn()` call.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 775666296, "label": "\"datasette insert\" command and plugin hook"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1161#issuecomment-752253095", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1161", "id": 752253095, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjI1MzA5NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T21:49:57Z", "updated_at": "2020-12-29T21:49:57Z", "author_association": "OWNER", "body": "https://github.com/Homebrew/homebrew-core/pull/67983", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 776101101, "label": "Update a whole bunch of links to datasette.io instead of datasette.readthedocs.io"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1160#issuecomment-752236520", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1160", "id": 752236520, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjIzNjUyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T20:48:51Z", "updated_at": "2020-12-29T20:48:51Z", "author_association": "OWNER", "body": "It would be neat if `datasette insert` could accept a `--plugins-dir` option which allowed one-off format plugins to be registered. Bit tricky to implement since the `--format` Click option will already be populated by that plugin hook call.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 775666296, "label": "\"datasette insert\" command and plugin hook"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1160#issuecomment-751925934", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1160", "id": 751925934, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MTkyNTkzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T02:40:13Z", "updated_at": "2020-12-29T20:25:57Z", "author_association": "OWNER", "body": "Basic command design:\r\n\r\n datasette insert data.db blah.csv\r\n\r\nThe options can include:\r\n\r\n- `--format` to specify the exact format - without this it will be guessed based on the filename\r\n- `--table` to specify the table (otherwise the filename is used)\r\n- `--pk` to specify one or more primary key columns\r\n- `--replace` to specify that existing rows with a matching primary key should be replaced\r\n- `--upsert` to specify that existing matching rows should be upserted\r\n- `--ignore` to ignore matching rows\r\n- `--alter` to alter the table to add missing columns\r\n- `--type column type` to specify the type of a column - useful when working with CSV or TSV files", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 775666296, "label": "\"datasette insert\" command and plugin hook"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1160#issuecomment-752208036", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1160", "id": 752208036, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjIwODAzNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T19:06:35Z", "updated_at": "2020-12-29T19:06:35Z", "author_association": "OWNER", "body": "If I'm going to execute 1000s of writes in an `async def` operation it may make sense to break that up into smaller chunks, so as not to block the event loop for too long.\r\n\r\nhttps://stackoverflow.com/a/36648102 and https://github.com/python/asyncio/issues/284 confirm that `await asyncio.sleep(0)` is the recommended way of doing this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 775666296, "label": "\"datasette insert\" command and plugin hook"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1160#issuecomment-752203909", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1160", "id": 752203909, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjIwMzkwOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T18:54:19Z", "updated_at": "2020-12-29T18:54:19Z", "author_association": "OWNER", "body": "More thoughts on this: the key mechanism that populates the tables needs to be an `aysnc def` method of some sort so that it can run as part of the async loop in core Datasette - for importing from web uploads.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 775666296, "label": "\"datasette insert\" command and plugin hook"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/417#issuecomment-752098906", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/417", "id": 752098906, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MjA5ODkwNg==", "user": {"value": 82988, "label": "psychemedia"}, "created_at": "2020-12-29T14:34:30Z", "updated_at": "2020-12-29T14:34:50Z", "author_association": "CONTRIBUTOR", "body": "FWIW, I had a look at `watchdog` for a `datasette` powered Jupyter notebook search tool: https://github.com/ouseful-testing/nbsearch/blob/main/nbsearch/nbwatchdog.py\r\n\r\nNot a production thing, just an experiment trying to explore what might be possible...", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421546944, "label": "Datasette Library"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1160#issuecomment-751947991", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1160", "id": 751947991, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MTk0Nzk5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T05:06:50Z", "updated_at": "2020-12-29T05:07:03Z", "author_association": "OWNER", "body": "Given the URL option could it be possible for plugins to \"subscribe\" to URLs that keep on streaming?\r\n\r\n datasette insert db.db https://example.con/streaming-api \\\r\n --format api-stream", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 775666296, "label": "\"datasette insert\" command and plugin hook"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1160#issuecomment-751946262", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1160", "id": 751946262, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MTk0NjI2Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T04:56:12Z", "updated_at": "2020-12-29T04:56:32Z", "author_association": "OWNER", "body": "Potential design for this: a `datasette memory` command which takes most of the same arguments as `datasette serve` but starts an in-memory database and treats the command arguments as things that should be inserted into that in-memory database.\r\n\r\n tail -f access.log | datasette memory - \\\r\n --format clf -p 8002 -o", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 775666296, "label": "\"datasette insert\" command and plugin hook"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1160#issuecomment-751945094", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1160", "id": 751945094, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MTk0NTA5NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T04:48:11Z", "updated_at": "2020-12-29T04:48:11Z", "author_association": "OWNER", "body": "It would be pretty cool if you could launch Datasette directly against an insert-compatible file or URL without first having to load it into a SQLite database file.\r\n\r\nOr imagine being able to tail a log file and like that directly into a new Datasette process, which then runs a web server with the UI while simultaneously continuing to load new entries from that log into the in-memory SQLite database that it is serving...\r\n\r\nNot quite sure what that CLI interface would look like. Maybe treat that as a future stretch goal for the moment.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 775666296, "label": "\"datasette insert\" command and plugin hook"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1160#issuecomment-751943837", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1160", "id": 751943837, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MTk0MzgzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T04:40:30Z", "updated_at": "2020-12-29T04:40:30Z", "author_association": "OWNER", "body": "The `insert` command should also accept URLs - anything starting with `http://` or `https://`.\r\n\r\nIt should accept more than one file name at a time for bulk inserts.\r\n\r\nif using a URL that URL will be passed to the method that decides if a plugin implementation can handle the import or not. This will allow plugins to register themselves for specific websites.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 775666296, "label": "\"datasette insert\" command and plugin hook"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1160#issuecomment-751926437", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1160", "id": 751926437, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MTkyNjQzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T02:43:21Z", "updated_at": "2020-12-29T02:43:37Z", "author_association": "OWNER", "body": "Default formats to support:\r\n\r\n- CSV\r\n- TSV\r\n- JSON and newline-delimited JSON\r\n- YAML\r\n\r\nEach of these will be implemented as a default plugin.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 775666296, "label": "\"datasette insert\" command and plugin hook"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1160#issuecomment-751926218", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1160", "id": 751926218, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MTkyNjIxOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T02:41:57Z", "updated_at": "2020-12-29T02:41:57Z", "author_association": "OWNER", "body": "Other names I considered:\r\n\r\n- `datasette load`\r\n- `datasette import` - I decided to keep this name available for any future work that might involve plugins that help import data from APIs as opposed to inserting it from files", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 775666296, "label": "\"datasette insert\" command and plugin hook"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1160#issuecomment-751926095", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1160", "id": 751926095, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MTkyNjA5NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-29T02:41:15Z", "updated_at": "2020-12-29T02:41:15Z", "author_association": "OWNER", "body": "The UI can live at `/-/insert` and be available by default to the `root` user only. It can offer the following:\r\n\r\n- Upload a file and have the import type detected (equivalent to `datasette insert data.db thatfile.csv`)\r\n- Copy and paste the data to be inserted into a textarea\r\n- API equivalents of these", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 775666296, "label": "\"datasette insert\" command and plugin hook"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/675#issuecomment-751826749", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/675", "id": 751826749, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MTgyNjc0OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-28T18:49:21Z", "updated_at": "2020-12-28T18:49:21Z", "author_association": "OWNER", "body": "That `--exec` could help solve all sorts of other problems too, like needing to `apt-get install` extra packages or download files from somewhere using `wget`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 567902704, "label": "--cp option for datasette publish and datasette package for shipping additional files and directories"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/675#issuecomment-751826621", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/675", "id": 751826621, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MTgyNjYyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-28T18:48:51Z", "updated_at": "2020-12-28T18:48:51Z", "author_association": "OWNER", "body": "I could make `--include` work if I also had a mechanism for running some shell commands inside the container at the end of the build - which users could then use to move files into the correct place.\r\n\r\n datasette publish cloudrun my.db --include src/ --exec 'mv /app/src/config.yml /etc/conf/config.yml'", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 567902704, "label": "--cp option for datasette publish and datasette package for shipping additional files and directories"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/417#issuecomment-751504136", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/417", "id": 751504136, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MTUwNDEzNg==", "user": {"value": 212369, "label": "drewda"}, "created_at": "2020-12-27T19:02:06Z", "updated_at": "2020-12-27T19:02:06Z", "author_association": "NONE", "body": "Very much looking forward to seeing this functionality come together. This is probably out-of-scope for an initial release, but in the future it could be useful to also think of how to run this is a container'ized context. For example, an immutable datasette container that points to an S3 bucket of SQLite DBs or CSVs. Or an immutable datasette container pointing to a NFS volume elsewhere on a Kubernetes cluster.", "reactions": "{\"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421546944, "label": "Datasette Library"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-751476406", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 751476406, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MTQ3NjQwNg==", "user": {"value": 18221871, "label": "noklam"}, "created_at": "2020-12-27T14:51:39Z", "updated_at": "2020-12-27T14:51:39Z", "author_association": "NONE", "body": "I like the idea of _internal, it's a nice way to get a data catalog quickly. I wonder if this trick applies to db other than SQLite.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/pull/59#issuecomment-751375487", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/59", "id": 751375487, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MTM3NTQ4Nw==", "user": {"value": 631242, "label": "frosencrantz"}, "created_at": "2020-12-26T17:08:44Z", "updated_at": "2020-12-26T17:08:44Z", "author_association": "CONTRIBUTOR", "body": "Hi @simonw, do I need to do anything else for this PR to be considered to be included? I've tried using this project and it is quite nice to be able to explore a repository, but noticed that a couple commands don't allow you to use authorization from the environment variable.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771872303, "label": "Remove unneeded exists=True for -a/--auth flag."}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/417#issuecomment-751127485", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/417", "id": 751127485, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MTEyNzQ4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-24T22:58:05Z", "updated_at": "2020-12-24T22:58:05Z", "author_association": "OWNER", "body": "That's a great idea. I'd ruled that out because working with the different operating system versions of those is tricky, but if `watchdog` can handle those differences for me this could be a really good option.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421546944, "label": "Datasette Library"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/417#issuecomment-751127384", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/417", "id": 751127384, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MTEyNzM4NA==", "user": {"value": 1279360, "label": "dyllan-to-you"}, "created_at": "2020-12-24T22:56:48Z", "updated_at": "2020-12-24T22:56:48Z", "author_association": "NONE", "body": "Instead of scanning the directory every 10s, have you considered listening for the native system events to notify you of updates?\r\n\r\nI think python has a nice module to do this for you called [watchdog](https://pypi.org/project/watchdog/)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421546944, "label": "Datasette Library"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/28#issuecomment-751125270", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/28", "id": 751125270, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MTEyNTI3MA==", "user": {"value": 129786, "label": "jmelloy"}, "created_at": "2020-12-24T22:26:22Z", "updated_at": "2020-12-24T22:26:22Z", "author_association": "NONE", "body": "This comes around if you\u2019ve run the photo export without running an s3 upload. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 624490929, "label": "Invalid SQL no such table: main.uploads"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1159#issuecomment-750849460", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1159", "id": 750849460, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MDg0OTQ2MA==", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2020-12-24T11:07:35Z", "updated_at": "2020-12-24T11:29:21Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1159?src=pr&el=h1) Report\n> Merging [#1159](https://codecov.io/gh/simonw/datasette/pull/1159?src=pr&el=desc) (c820abd) into [main](https://codecov.io/gh/simonw/datasette/commit/a882d679626438ba0d809944f06f239bcba8ee96?el=desc) (a882d67) will **not change** coverage.\n> The diff coverage is `n/a`.\n\n[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1159/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1159?src=pr&el=tree)\n\n```diff\n@@ Coverage Diff @@\n## main #1159 +/- ##\n=======================================\n Coverage 91.55% 91.55% \n=======================================\n Files 32 32 \n Lines 3930 3930 \n=======================================\n Hits 3598 3598 \n Misses 332 332 \n```\n\n\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1159?src=pr&el=continue).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1159?src=pr&el=footer). Last update [a882d67...c820abd](https://codecov.io/gh/simonw/datasette/pull/1159?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 774332247, "label": "Improve the display of facets information"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1158#issuecomment-750390741", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1158", "id": 750390741, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MDM5MDc0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-23T17:05:32Z", "updated_at": "2020-12-23T17:05:32Z", "author_association": "OWNER", "body": "Thanks for this!\r\n\r\nI'm fine keeping the `os.path` stuff as is.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 773913793, "label": "Modernize code to Python 3.6+"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1158#issuecomment-750389683", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1158", "id": 750389683, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MDM4OTY4Mw==", "user": {"value": 6774676, "label": "eumiro"}, "created_at": "2020-12-23T17:02:50Z", "updated_at": "2020-12-23T17:02:50Z", "author_association": "CONTRIBUTOR", "body": "The dict/set suggestion comes from `pyupgrade --py36-plus`, but then had to `black` the change.\r\n\r\nThe rest comes from PyCharm's Inspect code function. I reviewed all the suggestions and fixed a thing or two, such as leading/trailing spaces in the docstrings or turned around the chained conditions.\r\n\r\nThen I tried to convert all `os.path/glob/open` to `Path`, but there were some local test issues, so I'll have to start over in smaller chunks if you want to have that too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 773913793, "label": "Modernize code to Python 3.6+"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1158#issuecomment-750373603", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1158", "id": 750373603, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MDM3MzYwMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-23T16:26:21Z", "updated_at": "2020-12-23T16:26:21Z", "author_association": "OWNER", "body": "Did you use a tool to find these or are they all from manual code inspection?\r\n\r\nThey look good - I particularly like the set/dict literals.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 773913793, "label": "Modernize code to Python 3.6+"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1158#issuecomment-750373496", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1158", "id": 750373496, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MDM3MzQ5Ng==", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2020-12-23T16:26:06Z", "updated_at": "2020-12-23T16:26:06Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1158?src=pr&el=h1) Report\n> Merging [#1158](https://codecov.io/gh/simonw/datasette/pull/1158?src=pr&el=desc) (37ce72f) into [main](https://codecov.io/gh/simonw/datasette/commit/90eba4c3ca569c57e96bce314e7ac8caf67d884e?el=desc) (90eba4c) will **not change** coverage.\n> The diff coverage is `87.50%`.\n\n[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1158/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1158?src=pr&el=tree)\n\n```diff\n@@ Coverage Diff @@\n## main #1158 +/- ##\n=======================================\n Coverage 91.55% 91.55% \n=======================================\n Files 32 32 \n Lines 3930 3930 \n=======================================\n Hits 3598 3598 \n Misses 332 332 \n```\n\n\n| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1158?src=pr&el=tree) | Coverage \u0394 | |\n|---|---|---|\n| [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `77.41% <\u00f8> (\u00f8)` | |\n| [datasette/facets.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2ZhY2V0cy5weQ==) | `89.04% <\u00f8> (\u00f8)` | |\n| [datasette/filters.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2ZpbHRlcnMucHk=) | `94.35% <\u00f8> (\u00f8)` | |\n| [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <\u00f8> (\u00f8)` | |\n| [datasette/inspect.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2luc3BlY3QucHk=) | `36.11% <\u00f8> (\u00f8)` | |\n| [datasette/renderer.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3JlbmRlcmVyLnB5) | `94.02% <\u00f8> (\u00f8)` | |\n| [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `95.01% <50.00%> (\u00f8)` | |\n| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.85% <100.00%> (\u00f8)` | |\n| [datasette/utils/\\_\\_init\\_\\_.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.11% <100.00%> (\u00f8)` | |\n| [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `92.13% <100.00%> (\u00f8)` | |\n| ... and [1 more](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree-more) | |\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1158?src=pr&el=continue).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1158?src=pr&el=footer). Last update [90eba4c...37ce72f](https://codecov.io/gh/simonw/datasette/pull/1158?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 773913793, "label": "Modernize code to Python 3.6+"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1099#issuecomment-749845797", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1099", "id": 749845797, "node_id": "MDEyOklzc3VlQ29tbWVudDc0OTg0NTc5Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-23T00:13:29Z", "updated_at": "2020-12-23T00:14:25Z", "author_association": "OWNER", "body": "Also need to solve displaying these links in the opposite direction:\r\n\r\nhttps://latest.datasette.io/_internal/tables/fixtures,facet_cities\r\n\r\n\"_internal__tables\"\r\n\r\nThat page should link to lists of records in columns, foreign_keys and indexes - like this example: https://latest.datasette.io/fixtures/roadside_attractions/1\r\n\r\n\"fixtures__roadside_attractions\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 743371103, "label": "Support linking to compound foreign keys"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1152#issuecomment-748206874", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1152", "id": 748206874, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODIwNjg3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T17:03:00Z", "updated_at": "2020-12-22T23:58:04Z", "author_association": "OWNER", "body": "Another permissions thought: what if ALL Datasette permissions were default-deny, and plugins could only grant permission to things, not block permission?\r\n\r\nRight now a plugin can reply `False` to block, `True` to allow or `None` for \"I have no opinion on this, ask someone else\" - but even I'm confused by the interactions between block and allow and I implemented the system!\r\n\r\nIf everything in Datasette was default-deny then the user could use `--public-view` as an option when starting the server to default-allow view actions.\r\n\r\nMore importantly: plugins could return SQL statements that select a list of databases/tables the user is allowed access to. These could then be combined with `UNION` to create a full list of available resources.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770598024, "label": "Efficiently calculate list of databases/tables a user can view"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1152#issuecomment-747920515", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1152", "id": 747920515, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NzkyMDUxNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T07:29:21Z", "updated_at": "2020-12-22T23:57:29Z", "author_association": "OWNER", "body": "Could I solve this using a configured canned query against the `_internal` tables with the actor's properties as inputs?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770598024, "label": "Efficiently calculate list of databases/tables a user can view"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1099#issuecomment-749771231", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1099", "id": 749771231, "node_id": "MDEyOklzc3VlQ29tbWVudDc0OTc3MTIzMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-22T20:54:25Z", "updated_at": "2020-12-22T20:54:25Z", "author_association": "OWNER", "body": "https://latest.datasette.io/_internal/foreign_keys (use https://latest.datasette.io/login-as-root first) is now a compound foreign key table:\r\n```sql\r\nCREATE TABLE foreign_keys (\r\n \"database_name\" TEXT,\r\n \"table_name\" TEXT,\r\n \"id\" INTEGER,\r\n \"seq\" INTEGER,\r\n \"table\" TEXT,\r\n \"from\" TEXT,\r\n \"to\" TEXT,\r\n \"on_update\" TEXT,\r\n \"on_delete\" TEXT,\r\n \"match\" TEXT,\r\n PRIMARY KEY (database_name, table_name, id, seq),\r\n FOREIGN KEY (database_name) REFERENCES databases(database_name),\r\n FOREIGN KEY (database_name, table_name) REFERENCES tables(database_name, table_name)\r\n );\r\n```\r\nCurrently the `database_name` column becomes a link (because it's a single foreign key) but the `table_name` one remains a non-link:\r\n\r\n\"_internal__foreign_keys__24_rows\"\r\n\r\nMy original idea for compound foreign keys was to turn both of those columns into links, but that doesn't fit here because `database_name` is already part of a different foreign key.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 743371103, "label": "Support linking to compound foreign keys"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1152#issuecomment-749750995", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1152", "id": 749750995, "node_id": "MDEyOklzc3VlQ29tbWVudDc0OTc1MDk5NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-22T20:05:30Z", "updated_at": "2020-12-22T20:05:30Z", "author_association": "OWNER", "body": "#1150 is landed now, which means there's a new, hidden `_internal` SQLite in-memory database containing all of the tables and databases.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770598024, "label": "Efficiently calculate list of databases/tables a user can view"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/509#issuecomment-749749948", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/509", "id": 749749948, "node_id": "MDEyOklzc3VlQ29tbWVudDc0OTc0OTk0OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-22T20:03:10Z", "updated_at": "2020-12-22T20:03:10Z", "author_association": "OWNER", "body": "If you open multiple files with the same filename, e.g. like this:\r\n\r\n datasette fixtures.db templates/fixtures.db plugins/fixtures.db\r\n\r\nYou'll now get this:\r\n\r\n\"Datasette__fixtures__fixtures_2__fixtures_3\"\r\n", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 456568880, "label": "Support opening multiple databases with the same stem"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/509#issuecomment-749738241", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/509", "id": 749738241, "node_id": "MDEyOklzc3VlQ29tbWVudDc0OTczODI0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-22T19:38:14Z", "updated_at": "2020-12-22T19:38:14Z", "author_association": "OWNER", "body": "I'm fixing this in #1155.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 456568880, "label": "Support opening multiple databases with the same stem"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1155#issuecomment-749723557", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1155", "id": 749723557, "node_id": "MDEyOklzc3VlQ29tbWVudDc0OTcyMzU1Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-22T19:08:27Z", "updated_at": "2020-12-22T19:08:27Z", "author_association": "OWNER", "body": "I'm going to have the `.add_database()` method select the name used in the path, de-duping against any existing names. It will then set database.name to that so that the database has access to its own name.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771216293, "label": "Better internal database_name for _internal database"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1155#issuecomment-748356492", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1155", "id": 748356492, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODM1NjQ5Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T22:49:32Z", "updated_at": "2020-12-22T01:13:05Z", "author_association": "OWNER", "body": "There's some messy code that needs fixing here. The `datasette.databases` dictionary right now has a key that corresponds to the `/_internal` URL in the path, and a value that's a `Database()` object. BUT... the `Database()` object doesn't know what its key is.\r\n\r\nWhile fixing this I should fix the issue where Datasette gets confused by multiple databases with the same stem: https://github.com/simonw/datasette/issues/509", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771216293, "label": "Better internal database_name for _internal database"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1157#issuecomment-749179460", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1157", "id": 749179460, "node_id": "MDEyOklzc3VlQ29tbWVudDc0OTE3OTQ2MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-21T20:24:19Z", "updated_at": "2020-12-21T20:24:19Z", "author_association": "OWNER", "body": "Three places to fix:\r\n\r\nhttps://github.com/simonw/datasette/blob/dcdfb2c301341d45b66683e3e3be72f9c7585b2f/datasette/tracer.py#L40-L42\r\n\r\nhttps://github.com/simonw/datasette/blob/dcdfb2c301341d45b66683e3e3be72f9c7585b2f/datasette/utils/__init__.py#L139-L152\r\n\r\nhttps://github.com/simonw/datasette/blob/dcdfb2c301341d45b66683e3e3be72f9c7585b2f/datasette/views/base.py#L460-L461", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 772438273, "label": "Use time.perf_counter() instead of time.time() to measure performance"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1155#issuecomment-749176936", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1155", "id": 749176936, "node_id": "MDEyOklzc3VlQ29tbWVudDc0OTE3NjkzNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-21T20:18:15Z", "updated_at": "2020-12-21T20:18:15Z", "author_association": "OWNER", "body": "Fun query:\r\n```sql\r\nselect table_name, group_concat(name, ', ') from columns group by database_name, table_name\r\n```\r\nhttps://latest.datasette.io/_internal?sql=select+table_name%2C+group_concat%28name%2C+%27%2C+%27%29+from+columns+group+by+database_name%2C+table_name", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771216293, "label": "Better internal database_name for _internal database"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1155#issuecomment-749170608", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1155", "id": 749170608, "node_id": "MDEyOklzc3VlQ29tbWVudDc0OTE3MDYwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-21T20:01:47Z", "updated_at": "2020-12-21T20:01:47Z", "author_association": "OWNER", "body": "I removed that `MEMORY` object() in dcdfb2c301341d45b66683e3e3be72f9c7585b2f", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771216293, "label": "Better internal database_name for _internal database"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1156#issuecomment-749158111", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1156", "id": 749158111, "node_id": "MDEyOklzc3VlQ29tbWVudDc0OTE1ODExMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-21T19:33:45Z", "updated_at": "2020-12-21T19:33:45Z", "author_association": "OWNER", "body": "One reason for this change: it means I can use that database for more stuff. I've been thinking about moving metadata storage there for example, which fits a database called `_internal` but not one called `_schemas`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 772408750, "label": "Rename _schemas to _internal"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-748562330", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31", "id": 748562330, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODU2MjMzMA==", "user": {"value": 41546558, "label": "RhetTbull"}, "created_at": "2020-12-20T04:45:08Z", "updated_at": "2020-12-20T04:45:08Z", "author_association": "CONTRIBUTOR", "body": "Fixes the issue mentioned here: https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-748436115", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 1, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771511344, "label": "Update for Big Sur"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-748562288", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15", "id": 748562288, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODU2MjI4OA==", "user": {"value": 41546558, "label": "RhetTbull"}, "created_at": "2020-12-20T04:44:22Z", "updated_at": "2020-12-20T04:44:22Z", "author_association": "CONTRIBUTOR", "body": "@nickvazz @simonw I opened a [PR](https://github.com/dogsheep/dogsheep-photos/pull/31) that replaces the SQL for `ZCOMPUTEDASSETATTRIBUTES` to use osxphotos which now exposes all this data and has been updated for Big Sur. I did regression tests to confirm the extracted data is identical, with one exception which should not affect operation: the old code pulled data from `ZCOMPUTEDASSETATTRIBUTES` for missing photos while the main loop ignores missing photos and does not add them to `apple_photos`. The new code does not add rows to the `apple_photos_scores` table for missing photos.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 612151767, "label": "Expose scores from ZCOMPUTEDASSETATTRIBUTES"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-748436779", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15", "id": 748436779, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODQzNjc3OQ==", "user": {"value": 41546558, "label": "RhetTbull"}, "created_at": "2020-12-19T07:49:00Z", "updated_at": "2020-12-19T07:49:00Z", "author_association": "CONTRIBUTOR", "body": "@nickvazz ZGENERICASSET changed to ZASSET in Big Sur. Here's a list of other changes to the schema in Big Sur: https://github.com/RhetTbull/osxphotos/wiki/Changes-in-Photos-6---Big-Sur", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 612151767, "label": "Expose scores from ZCOMPUTEDASSETATTRIBUTES"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-748436115", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15", "id": 748436115, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODQzNjExNQ==", "user": {"value": 8573886, "label": "nickvazz"}, "created_at": "2020-12-19T07:43:38Z", "updated_at": "2020-12-19T07:47:36Z", "author_association": "NONE", "body": "Hey Simon! I really enjoy datasette so far, just started trying it out today following your iPhone photos [example](https://simonwillison.net/2020/May/21/dogsheep-photos/). \r\n\r\nI am not sure if you had run into this or not, but it seems like they might have changed one of the column names from\r\n`ZGENERICASSET` to `ZASSET`. Should I open a PR? \r\n\r\nWould change:\r\n- [here](https://github.com/dogsheep/dogsheep-photos/blob/master/dogsheep_photos/cli.py#L209-L213)\r\n- [here](https://github.com/dogsheep/dogsheep-photos/blob/master/dogsheep_photos/cli.py#L238)\r\n- [here](https://github.com/dogsheep/dogsheep-photos/blob/master/dogsheep_photos/cli.py#L240)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 612151767, "label": "Expose scores from ZCOMPUTEDASSETATTRIBUTES"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/53#issuecomment-748436453", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/53", "id": 748436453, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODQzNjQ1Mw==", "user": {"value": 27, "label": "anotherjesse"}, "created_at": "2020-12-19T07:47:01Z", "updated_at": "2020-12-19T07:47:01Z", "author_association": "NONE", "body": "I think this should probably be closed as won't fix.\r\n\r\nAttempting to make a patch for this I realized that the since_id would limit to tweets posted since that since_id, not when it was favorited. So favoriting something in the older would be missed if you used `--since` with a cron script\r\n\r\nBetter to just use `--stop_after` set to a couple hundred", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771324837, "label": "--since support for favorites"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-748436195", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21", "id": 748436195, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODQzNjE5NQ==", "user": {"value": 8573886, "label": "nickvazz"}, "created_at": "2020-12-19T07:44:32Z", "updated_at": "2020-12-19T07:44:49Z", "author_association": "NONE", "body": "I have also run into this a bit, would it be possible to post your `requirements.txt` so I can try and reproduce your [blog post](https://simonwillison.net/2020/May/21/dogsheep-photos/)?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 615474990, "label": "bpylist.archiver.CircularReference: archive has a cycle with uid(13)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426877", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31", "id": 748426877, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODQyNjg3Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-19T06:16:11Z", "updated_at": "2020-12-19T06:16:11Z", "author_association": "MEMBER", "body": "Here's why:\r\n\r\n if \"fts5\" in str(e):\r\n\r\nBut the error being raised here is:\r\n\r\n sqlite3.OperationalError: no such column: to\r\n\r\nI'm going to attempt the escaped on on every error.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771316301, "label": "Searching for \"github-to-sqlite\" throws an error"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426663", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31", "id": 748426663, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODQyNjY2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-19T06:14:06Z", "updated_at": "2020-12-19T06:14:06Z", "author_association": "MEMBER", "body": "Looks like I already do that here: https://github.com/dogsheep/dogsheep-beta/blob/9ba4401017ac24ffa3bc1db38e0910ea49de7616/dogsheep_beta/__init__.py#L141-L146", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771316301, "label": "Searching for \"github-to-sqlite\" throws an error"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426581", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31", "id": 748426581, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODQyNjU4MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-19T06:13:17Z", "updated_at": "2020-12-19T06:13:17Z", "author_association": "MEMBER", "body": "One fix for this could be to try running the raw query, but if it throws an error run it again with the query escaped.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771316301, "label": "Searching for \"github-to-sqlite\" throws an error"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426501", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31", "id": 748426501, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODQyNjUwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-19T06:12:22Z", "updated_at": "2020-12-19T06:12:22Z", "author_association": "MEMBER", "body": "I deliberately added support for advanced FTS in https://github.com/dogsheep/dogsheep-beta/commit/cbb2491b85d7ff416d6d429b60109e6c2d6d50b9 for #13 but that's the cause of this bug.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771316301, "label": "Searching for \"github-to-sqlite\" throws an error"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1155#issuecomment-748397998", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1155", "id": 748397998, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODM5Nzk5OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-19T01:28:18Z", "updated_at": "2020-12-19T01:28:18Z", "author_association": "OWNER", "body": "`datasette-graphql` returns an error due to this issue:\r\n\r\n\"GraphiQL_and_fixtures__compound_three_primary_keys__1_001_rows\"\r\n\r\nOn the console:\r\n```\r\nINFO: 127.0.0.1:63116 - \"POST /graphql/_internal HTTP/1.1\" 500 Internal Server Error\r\nTraceback (most recent call last):\r\n File \"/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/promise/promise.py\", line 844, in handle_future_result\r\n resolve(future.result())\r\n File \"/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/datasette_graphql/utils.py\", line 603, in resolve_table\r\n data, _, _ = await view.data(\r\n File \"/Users/simon/Dropbox/Development/datasette/datasette/views/table.py\", line 304, in data\r\n db = self.ds.databases[database]\r\ngraphql.error.located_error.GraphQLLocatedError: ':memory:c6dd5abe1a757a7de00d99b699175bd33d9a575f05b5751bf856b8656fb07edd'\r\n```\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771216293, "label": "Better internal database_name for _internal database"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1155#issuecomment-748368660", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1155", "id": 748368660, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODM2ODY2MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T23:18:04Z", "updated_at": "2020-12-19T01:12:00Z", "author_association": "OWNER", "body": "A `Database` should have a `.name` which is unique across the Datasette instance and is used in the URL. The `path` should be optional, only set for file databases. A new `.memory_name` property can be used for shared memory databases.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771216293, "label": "Better internal database_name for _internal database"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1155#issuecomment-748368938", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1155", "id": 748368938, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODM2ODkzOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T23:19:04Z", "updated_at": "2020-12-18T23:19:04Z", "author_association": "OWNER", "body": "`Database` internal class is documented here: https://docs.datasette.io/en/latest/internals.html#database-class", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771216293, "label": "Better internal database_name for _internal database"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1155#issuecomment-748368384", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1155", "id": 748368384, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODM2ODM4NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T23:17:00Z", "updated_at": "2020-12-18T23:17:00Z", "author_association": "OWNER", "body": "Here's the commit where I added it. https://github.com/simonw/datasette/commit/9743e1d91b5f0a2b3c1c0bd6ffce8739341f43c4 - I didn't yet have the `.add_database()` mechanism. Today the `MEMORY` object bit is no longer needed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771216293, "label": "Better internal database_name for _internal database"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1155#issuecomment-748367922", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1155", "id": 748367922, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODM2NzkyMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T23:15:24Z", "updated_at": "2020-12-18T23:15:24Z", "author_association": "OWNER", "body": "The code for building up that `.databases` dictionary is a bit convoluted. Here's the code that adds a `:memory:` database if the user specified `--memory` OR if there are no files to be attached:\r\n\r\nhttps://github.com/simonw/datasette/blob/ebc7aa287c99fe6114b79aeab8efb8d4489a6182/datasette/app.py#L221-L241\r\n\r\nI'm not sure why I wrote it this way, instead of just calling `.add_database(\":memory:\", Database(..., is_memory=True)`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771216293, "label": "Better internal database_name for _internal database"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/509#issuecomment-748356637", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/509", "id": 748356637, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODM1NjYzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T22:50:03Z", "updated_at": "2020-12-18T22:50:03Z", "author_association": "OWNER", "body": "Related problem caused by the new `_schemas` database - if a user attempts to open their own `_schemas.db` file it will fail. I'd like to open and mount that as `/_schemas_` instead.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 456568880, "label": "Support opening multiple databases with the same stem"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-748354841", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 748354841, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODM1NDg0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T22:43:49Z", "updated_at": "2020-12-18T22:43:49Z", "author_association": "OWNER", "body": "For a demo, visit https://latest.datasette.io/login-as-root and then hit https://latest.datasette.io/_schemas", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-748352106", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 748352106, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODM1MjEwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T22:34:40Z", "updated_at": "2020-12-18T22:34:40Z", "author_association": "OWNER", "body": "Needs documentation, but I can wait to write that until I've tested out the feature a bit more.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-748351350", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 748351350, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODM1MTM1MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T22:32:13Z", "updated_at": "2020-12-18T22:32:13Z", "author_association": "OWNER", "body": "Getting all the tests to pass is tricky because this adds a whole extra database to Datasette - and there's various code that loops through `ds.databases` as part of the tests.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/493#issuecomment-748305976", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/493", "id": 748305976, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODMwNTk3Ng==", "user": {"value": 50527, "label": "jefftriplett"}, "created_at": "2020-12-18T20:34:39Z", "updated_at": "2020-12-18T20:34:39Z", "author_association": "CONTRIBUTOR", "body": "I can't keep up with the renaming contexts, but I like having the ability to run datasette+ datasette-ripgrep against different configs: \r\n\r\n```shell\r\ndatasette serve --metadata=./metadata.json\r\n```\r\n\r\nI have one for all of my code and one per client who has lots of code. So as long as I can point to datasette to something, it's easy to work with. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 449886319, "label": "Rename metadata.json to config.json"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-748260875", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 748260875, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODI2MDg3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T18:55:12Z", "updated_at": "2020-12-18T18:55:12Z", "author_association": "OWNER", "body": "I'm going to move the code into a `utils/schemas.py` module, to avoid further extending the `Datasette` class definition and to make it more easily testable.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-748260118", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 748260118, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODI2MDExOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T18:54:12Z", "updated_at": "2020-12-18T18:54:12Z", "author_association": "OWNER", "body": "I'm going to tidy this up and land it. A couple of additional decisions:\r\n\r\n- The database will be called `/_schemas`\r\n- By default it will only be visible to `root` - thus avoiding having to solve the permissions problem with regards to users seeing schemas for tables that are otherwise invisible to them.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/493#issuecomment-747966232", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/493", "id": 747966232, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzk2NjIzMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T09:19:41Z", "updated_at": "2020-12-18T09:19:41Z", "author_association": "OWNER", "body": "Is there any reason to keep `--setting` rather than moving those items into a `configure.json` file with all the configuration options that currently live in `metadata.json`?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 449886319, "label": "Rename metadata.json to config.json"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1152#issuecomment-747921195", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1152", "id": 747921195, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NzkyMTE5NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T07:31:25Z", "updated_at": "2020-12-18T07:31:25Z", "author_association": "OWNER", "body": "It's also a really good fit for the new mechanism that's coming together in #1150.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770598024, "label": "Efficiently calculate list of databases/tables a user can view"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1152#issuecomment-747920852", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1152", "id": 747920852, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NzkyMDg1Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T07:30:22Z", "updated_at": "2020-12-18T07:30:22Z", "author_association": "OWNER", "body": "Redefining all Datasette permissions in terms of SQL queries that return the set of databases and tables that the user is allowed to interact with does feel VERY Datasette-y.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770598024, "label": "Efficiently calculate list of databases/tables a user can view"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1152#issuecomment-747920087", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1152", "id": 747920087, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NzkyMDA4Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T07:27:58Z", "updated_at": "2020-12-18T07:28:30Z", "author_association": "OWNER", "body": "I want to keep the existing `metadata.json` \"allow\" blocks mechanism working. Note that if you have 1,000 tables and a permissions policy you won't be using \"allow\" blocks, you'll be using a more sophisticated permissions plugin instead.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770598024, "label": "Efficiently calculate list of databases/tables a user can view"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1152#issuecomment-747919782", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1152", "id": 747919782, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NzkxOTc4Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T07:27:01Z", "updated_at": "2020-12-18T07:27:01Z", "author_association": "OWNER", "body": "Perhaps this can be solved by keeping the existing plugin hooks and adding new, optional ones for bulk lookups.\r\n\r\nIf your plugin doesn't implement the bulk lookup hooks Datasette will do an inefficient loop through everything checking permissions on each one.\r\n\r\nIf you DO implement it you can speed things up dramatically.\r\n\r\nNot sure if this would solve the homepage problem though, where you might need to run 1,000 table permission checks. That's more a case where you want to think in terms of a SQL where clause.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770598024, "label": "Efficiently calculate list of databases/tables a user can view"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747893704", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747893704, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzg5MzcwNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T06:19:13Z", "updated_at": "2020-12-18T06:19:13Z", "author_association": "OWNER", "body": "I'm not going to block this issue on permissions - I will tackle the efficient bulk permissions problem in #1152.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1152#issuecomment-747893423", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1152", "id": 747893423, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzg5MzQyMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T06:18:24Z", "updated_at": "2020-12-18T06:18:24Z", "author_association": "OWNER", "body": "What would Datasette's permission hooks look like if they all dealt with sets of items rather than individual items? So plugins could return a set of items that the user has permission to access, or even a WHERE clause?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770598024, "label": "Efficiently calculate list of databases/tables a user can view"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1152#issuecomment-747892731", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1152", "id": 747892731, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzg5MjczMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T06:16:29Z", "updated_at": "2020-12-18T06:16:29Z", "author_association": "OWNER", "body": "One enormous advantage I have is that after #1150 I will have a database table full of databases and tables that I can execute queries against.\r\n\r\nThis means I could calculate visible tables using SQL where clauses, which should be easily fast enough even against ten thousand plus tables.\r\n\r\nThe catch is the permissions hooks. Since I haven't hit Datasette 1.0 yet maybe I should redesign those hooks to work against the new in-memory database schema stuff?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770598024, "label": "Efficiently calculate list of databases/tables a user can view"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1152#issuecomment-747891854", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1152", "id": 747891854, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzg5MTg1NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T06:14:09Z", "updated_at": "2020-12-18T06:14:15Z", "author_association": "OWNER", "body": "This is a classic challenge in permissions systems. If I want Datasette to be able to handle thousands of tables I need a reasonable solution for it.\r\n\r\nTwitter conversation: https://twitter.com/simonw/status/1339791768842248192", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770598024, "label": "Efficiently calculate list of databases/tables a user can view"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747864831", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747864831, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzg2NDgzMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T04:46:18Z", "updated_at": "2020-12-18T04:46:18Z", "author_association": "OWNER", "body": "The homepage currently performs a massive flurry of permission checks - one for each, database, table and view: https://github.com/simonw/datasette/blob/0.53/datasette/views/index.py#L21-L75\r\n\r\nA paginated version of this is a little daunting as the permission checks would have to be carried out in every single table just to calculate the count that will be paginated.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747864080", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747864080, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzg2NDA4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T04:43:29Z", "updated_at": "2020-12-18T04:43:29Z", "author_association": "OWNER", "body": "I may be overthinking that problem. Many queries are fast in SQLite. If a Datasette instance has 1,000 connected tables will even that be a performance problem for permission checks? I should benchmark to find out.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747862001", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747862001, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzg2MjAwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T04:35:34Z", "updated_at": "2020-12-18T04:35:34Z", "author_association": "OWNER", "body": "I do need to solve the permissions problem properly though, because one of the goals of this system is to provide a paginated, searchable list of databases and tables for the homepage of the instance - #991.\r\n\r\nAs such, the homepage will need to be able to display only the tables and databases that the user has permission to view.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747861556", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747861556, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzg2MTU1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T04:33:45Z", "updated_at": "2020-12-18T04:33:45Z", "author_association": "OWNER", "body": "One solution on permissions: if Datasette had an efficient way of saying \"list the tables that this user has access to\" I could use that as a filter any time the user views the schema information. The implementation could be tricky though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747861357", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747861357, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzg2MTM1Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T04:32:52Z", "updated_at": "2020-12-18T04:32:52Z", "author_association": "OWNER", "body": "I need to figure out how this will interact with Datasette permissions.\r\n\r\nIf some tables are private, but others are public, should users be able to see the private tables listed in the schema metadata?\r\n\r\nIf not, how can that mechanism work?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747833639", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747833639, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NzgzMzYzOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T02:49:40Z", "updated_at": "2020-12-18T03:52:12Z", "author_association": "OWNER", "body": "I'm going to use five tables to start off with:\r\n\r\n- `databases` - a list of databases. Each one has a `name`, `path` (if it's on disk), `is_memory`, `schema_version`\r\n- `tables` - a list of tables. Each row is `database_name`, `table_name`, `sql` (the create table statement) - may add more tables in the future, in particular maybe a `last_row_count` to cache results of counting the rows.\r\n- `columns` - a list of columns. It's the output of `pragma_table_xinfo` with the `database_name` and `table_name` columns added at the beginning.\r\n- `foreign_keys` - a list of foreign keys - `pragma_foreign_key_list` output plus `database_name` and `table_name`.\r\n- `indexes` - a list of indexes - `pragma_table_xinfo` output plus `database_name` and `table_name`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747847405", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747847405, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzg0NzQwNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T03:36:04Z", "updated_at": "2020-12-18T03:36:04Z", "author_association": "OWNER", "body": "I could have another table that stores the combined rows from `sqlite_m\u00e1ster` on every connected database so I have a copy of the schema SQL.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747847180", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747847180, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzg0NzE4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T03:35:15Z", "updated_at": "2020-12-18T03:35:15Z", "author_association": "OWNER", "body": "Simpler implementation idea: a Datasette method `.refresh_schemas()` which loops through all known databases, checks their schema version and updates the in-memory schemas database if they have changed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747834762", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747834762, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NzgzNDc2Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T02:53:22Z", "updated_at": "2020-12-18T02:53:22Z", "author_association": "OWNER", "body": "I think I'm going to have to build this without using the `pragma_x()` SQL functions as they were only added in 3.16 in 2017-01-02 and I've seen plenty of Datasette instances running on older versions of SQLite.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}