home / github

Menu
  • Search all tables
  • GraphQL API

issue_comments

Table actions
  • GraphQL API for issue_comments

15 rows where user = 2670795 sorted by updated_at descending

✖
✖

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: issue_url, reactions, created_at (date), updated_at (date)

issue 7

  • Plugin hook for dynamic metadata 6
  • Database page loads too slowly with many large tables (due to table counts) 3
  • DRAFT: A new plugin hook for dynamic metadata 2
  • Mechanism for storing metadata in _metadata tables 1
  • Ability for plugins to collaborate when adding extra HTML to blocks in default templates 1
  • Allow canned query params to specify default values 1
  • Datasette feature for publishing snapshots of query results 1

user 1

  • brandonrobertz · 15 ✖

author_association 1

  • CONTRIBUTOR 15
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
1437671409 https://github.com/simonw/datasette/issues/1258#issuecomment-1437671409 https://api.github.com/repos/simonw/datasette/issues/1258 IC_kwDOBm6k_c5VsR_x brandonrobertz 2670795 2023-02-20T23:39:58Z 2023-02-20T23:39:58Z CONTRIBUTOR

This is pretty annoying for FTS because sqlite throws an error instead of just doing something like returning all or no results. This makes users who are unfamiliar with SQL and Datasette think the canned query page is broken and is a frequent source of confusion.

To anyone dealing with this: My solution is to modify the canned query so that it returns no results which cues people to fill in the blank parameters.

So instead of emails_fts match escape_fts(:search))

My canned queries now look like this:

emails_fts match escape_fts(iif(:search=="", "*", :search))

There are no asterisks in my data so the result is always blank.

Ultimately it would be nice to be able to handle this in the metadata. Either making some named parameters required or setting some default values.

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Allow canned query params to specify default values 828858421  
1200732975 https://github.com/simonw/datasette/issues/1191#issuecomment-1200732975 https://api.github.com/repos/simonw/datasette/issues/1191 IC_kwDOBm6k_c5Hkbsv brandonrobertz 2670795 2022-08-01T05:39:27Z 2022-08-01T05:39:27Z CONTRIBUTOR

I've got a URL shortening plugin that I would like to embed on the query page but I'd like avoid capturing the entire query.html template. A feature like this would solve it. Where's this at and how can I help?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability for plugins to collaborate when adding extra HTML to blocks in default templates 787098345  
1173358747 https://github.com/simonw/datasette/issues/1713#issuecomment-1173358747 https://api.github.com/repos/simonw/datasette/issues/1713 IC_kwDOBm6k_c5F8Aib brandonrobertz 2670795 2022-07-04T05:16:35Z 2022-07-04T05:16:35Z CONTRIBUTOR

This feature is pretty important and would be nice if it would be all within Datasette (no separate CLI/deploy required). My workflow now is to basically just copy the result and paste into a Google Sheet, which works, but then it's not discoverable to other journalists browsing the Datasette instance. I started building a plugin similar to datasette-saved-queries but one that maintains its own DB (required if you're working with all immutable DBs), but got bogged down in details.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Datasette feature for publishing snapshots of query results 1203943272  
1066222323 https://github.com/simonw/datasette/issues/1384#issuecomment-1066222323 https://api.github.com/repos/simonw/datasette/issues/1384 IC_kwDOBm6k_c4_jULz brandonrobertz 2670795 2022-03-14T00:36:42Z 2022-03-14T00:36:42Z CONTRIBUTOR

Ah, sorry, I didn't get what you were saying you the first time. Using _metadata_local in that way makes total sense -- I agree, refreshing metadata each cell was seeming quite excessive. Now I'm on the same page! :)

All good. Report back any issues you find with this stuff. Metadata/dynamic config hasn't been tested widely outside of what I've done AFAIK. If you find a strong use case for async meta, it's going to be better to know sooner rather than later!

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Plugin hook for dynamic metadata 930807135  
1066169718 https://github.com/simonw/datasette/issues/1384#issuecomment-1066169718 https://api.github.com/repos/simonw/datasette/issues/1384 IC_kwDOBm6k_c4_jHV2 brandonrobertz 2670795 2022-03-13T19:48:49Z 2022-03-13T19:48:49Z CONTRIBUTOR

For my reference, did you include a render_cell plugin calling get_metadata in those tests?

You shouldn't need to do this, as I mentioned previously. The code inside render_cell hook already has access to the most recently sync'd metadata via datasette._metadata_local. Refreshing the metadata for every cell seems ... excessive.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Plugin hook for dynamic metadata 930807135  
1066006292 https://github.com/simonw/datasette/issues/1384#issuecomment-1066006292 https://api.github.com/repos/simonw/datasette/issues/1384 IC_kwDOBm6k_c4_ifcU brandonrobertz 2670795 2022-03-13T02:09:44Z 2022-03-13T02:09:44Z CONTRIBUTOR

If I'm understanding your plugin code correctly, you query the db using the sync handle every time get_metdata is called, right? Won't this become a pretty big bottleneck if a hook into render_cell is trying to read metadata / plugin config?

Reading from sqlite DBs is pretty quick and I didn't notice significant performance issues when I was benchmarking. I tested on very large Datasette deployments (hundreds of DBs, millions of rows). See "Many small queries are efficient in sqlite" for more information on the rationale here. Also note that in the datasette-live-config reference plugin, the DB connection is cached, so that eliminated most of the performance worries we had.

If you need to ensure fresh metadata is being read inside of a render_cell hook specifically, you don't need to do anything further! get_metadata gets called before render_cell every request, so it already has access to the synced meta. There shouldn't be a need to call get_metadata(...) or metadata(...) inside render_cell, you can just use datasette._metadata_local if you're really worried about performance.

The plugin is close, but looks like it only grabs remote metadata, is that right? Instead what I'm wanting is to grab metadata embedded in the attached databases.

Yes correct, the datadette-remote-metadata plugin doesn't do that. But the datasette-live-config plugin does. It supports a __metadata table that, when it exists on an attached DB, gets pulled into the Datasette internal _metadata and is also accessible via get_metadata. Updating is instantaneous so there's no gotchas for users or security issues for users relying on the metadata-based permissions. Simon talked about eventually making something like this a standard feature of Datasette, but I'm not sure what the status is on that!

Good luck!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Plugin hook for dynamic metadata 930807135  
1065940779 https://github.com/simonw/datasette/issues/1384#issuecomment-1065940779 https://api.github.com/repos/simonw/datasette/issues/1384 IC_kwDOBm6k_c4_iPcr brandonrobertz 2670795 2022-03-12T18:49:29Z 2022-03-12T18:50:07Z CONTRIBUTOR

Hello! Just wanted to chime in and note that there's a plugin to have Datasette watch for updates to an external metadata.yaml/json and update the internal settings accordingly, so I think the cache/poll use case is already covered. @khusmann If you don't need truly dynamic metadata then what you've come up with or the plugin ought to work fine.

Making the get_metadata async won't improve the situation by itself as only some of the code paths accessing metadata use that hook. The other paths use the internal metadata dict. Trying to force all paths through a async hook would have performance ramifications and making everything use the internal meta will cause problems for users that need changes to take effect immediately. This is why I came to the non-async solution as it was the path of least change within Datasette. As always, open to new ideas, etc!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Plugin hook for dynamic metadata 930807135  
905904540 https://github.com/simonw/datasette/issues/859#issuecomment-905904540 https://api.github.com/repos/simonw/datasette/issues/859 IC_kwDOBm6k_c41_wGc brandonrobertz 2670795 2021-08-25T21:59:14Z 2021-08-25T21:59:55Z CONTRIBUTOR

I did two tests: one with 1000 5-30mb DBs and a second with 20 multi gig DBs. For the second, I created them like so: for i in {1..20}; do sqlite-generate db$i.db --tables ${i}00 --rows 100,2000 --columns 5,100 --pks 0 --fks 0; done

This was for deciding whether to use lots of small DBs or to group things into a smaller number of bigger DBs. The second strategy wins.

By simply persisting the _internal DB to disk, I was able to avoid most of the performance issues I was experiencing previously. (To do this, I changed the datasette/internal_db.py:init_internal_db creates to if not exists, and changed the _internal DB instantiation in datasette/app.py:Datasette.__init__ to a path with is_mutable=True.) Super rough, but the pages now load so I can continue testing ideas.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841  
905899177 https://github.com/simonw/datasette/issues/859#issuecomment-905899177 https://api.github.com/repos/simonw/datasette/issues/859 IC_kwDOBm6k_c41_uyp brandonrobertz 2670795 2021-08-25T21:48:00Z 2021-08-25T21:48:00Z CONTRIBUTOR

Upon first stab, there's two issues here: - DB/table/row counts (as discussed above). This isn't too bad if the DBs are actually above the MAX limit check. - Populating the internal DB. On first load of a giant set of DBs, it can take 10-20 mins to populate. By altering datasette and persisting the internal DB to disk, this problem is vastly improved, but I'm sure this will cause problems elsewhere.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841  
904982056 https://github.com/simonw/datasette/issues/859#issuecomment-904982056 https://api.github.com/repos/simonw/datasette/issues/859 IC_kwDOBm6k_c418O4o brandonrobertz 2670795 2021-08-24T21:15:04Z 2021-08-24T21:15:30Z CONTRIBUTOR

I'm running into issues with this as well. All other pages seem to work with lots of DBs except the home page, which absolutely tanks. Would be willing to put some work into this, if there's been any kind of progress on concepts on how this ought to work.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841  
869076254 https://github.com/simonw/datasette/issues/1168#issuecomment-869076254 https://api.github.com/repos/simonw/datasette/issues/1168 MDEyOklzc3VlQ29tbWVudDg2OTA3NjI1NA== brandonrobertz 2670795 2021-06-27T00:03:16Z 2021-06-27T00:05:51Z CONTRIBUTOR

Related: Here's an implementation of a get_metadata() plugin hook by @brandonrobertz next-LI@3fd8ce9

Here's a plugin that implements metadata-within-DBs: next-LI/datasette-live-config

How it works: If a database has a __metadata table, then it gets parsed and included in the global metadata. It also implements a database-action hook with a UI for managing config.

More context: https://github.com/next-LI/datasette-live-config/blob/72e335e887f1c69c54c6c2441e07148955b0fc9f/datasette_live_config/init.py#L109-L140

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Mechanism for storing metadata in _metadata tables 777333388  
869074701 https://github.com/simonw/datasette/issues/1384#issuecomment-869074701 https://api.github.com/repos/simonw/datasette/issues/1384 MDEyOklzc3VlQ29tbWVudDg2OTA3NDcwMQ== brandonrobertz 2670795 2021-06-26T23:45:18Z 2021-06-26T23:45:37Z CONTRIBUTOR

Here's where the plugin hook is called, demonstrating the fallback= argument:

https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/datasette/app.py#L426-L472

I'm not convinced of the use-case for passing fallback= to the hook here - is there a reason a plugin might care whether fallback is True or False, seeing as the metadata() method already respects that fallback logic on line 459?

I think you're right. I can't think of a reason why the plugin would care about the fallback parameter since plugins are currently mandated to return a full, global metadata dict.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Plugin hook for dynamic metadata 930807135  
869074182 https://github.com/simonw/datasette/issues/1384#issuecomment-869074182 https://api.github.com/repos/simonw/datasette/issues/1384 MDEyOklzc3VlQ29tbWVudDg2OTA3NDE4Mg== brandonrobertz 2670795 2021-06-26T23:37:42Z 2021-06-26T23:37:42Z CONTRIBUTOR

Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries. @brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?

Answering my own question: here's how Brandon implements it in his datasette-live-config plugin: https://github.com/next-LI/datasette-live-config/blob/72e335e887f1c69c54c6c2441e07148955b0fc9f/datasette_live_config/init.py#L50-L160

That's using a completely separate SQLite connection (actually wrapped in sqlite-utils) and making blocking synchronous calls to it.

This is a pragmatic solution, which works - and likely performs just fine, because SQL queries like this against a small database are so fast that not running them asynchronously isn't actually a problem.

But... it's weird. Everywhere else in Datasette land uses await db.execute(...) - but here's an example where users are encouraged to use blocking calls instead.

Ideally this hook would be asynchronous, but when I started down that path I quickly realized how large of a change this would be, since metadata gets used synchronously across the entire Datasette codebase. (And calling async code from sync is non-trivial.)

In my live-configuration implementation I use synchronous reads using a persistent sqlite connection. This works pretty well in practice, but I agree it's limiting. My thinking around this was to go with the path of least change as Datasette.metadata() is a critical core function.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Plugin hook for dynamic metadata 930807135  
865204472 https://github.com/simonw/datasette/pull/1368#issuecomment-865204472 https://api.github.com/repos/simonw/datasette/issues/1368 MDEyOklzc3VlQ29tbWVudDg2NTIwNDQ3Mg== brandonrobertz 2670795 2021-06-21T17:11:37Z 2021-06-21T17:11:37Z CONTRIBUTOR

If this is a concept ACK then I will move onto fixing the tests (adding new ones) and updating the documentation for the new plugin hook.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
DRAFT: A new plugin hook for dynamic metadata 913865304  
856182547 https://github.com/simonw/datasette/pull/1368#issuecomment-856182547 https://api.github.com/repos/simonw/datasette/issues/1368 MDEyOklzc3VlQ29tbWVudDg1NjE4MjU0Nw== brandonrobertz 2670795 2021-06-07T18:59:47Z 2021-06-07T23:04:25Z CONTRIBUTOR

Note that if we went with a "update_metadata" hook, the hook signature would look something like this (it would return nothing):

update_metadata( datasette=self, metadata=metadata, key=key, database=database, table=table, fallback=fallback )

The Datasette function _metadata_recursive_update(self, orig, updated) would disappear into the plugins. Doing this, though, we'd lose the easy ability to make the local metadata.yaml immutable (since we'd no longer have the recursive update).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
DRAFT: A new plugin hook for dynamic metadata 913865304  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);
Powered by Datasette · Queries took 20.137ms · About: github-to-sqlite
  • Sort ascending
  • Sort descending
  • Facet by this
  • Hide this column
  • Show all columns
  • Show not-blank rows