issue_comments

A good basic starting point for this would be to ignore the ability to add custom SQL fragments and instead focus on being able to show and hide specific columns. This will play particularly well with #613.

Proposed syntax for that:

/db/table?_col=id&_col=name - just show the id and name columns /db/table?_nocol=extras&_nocol=age - show all columns except for extras and age

I don't think it makes sense to allow both ?_col= and ?_nocol= arguments in the same request, so if you provide both I think we throw a 400 error.

Consider https://latest.datasette.io/fixtures/facetable

With ?_join=city_id the underlying query would become this:

select facetable.pk as 'facetable.pk', facetable.created as 'facetable.created', facetable.planet_int as 'facetable.planet_int', facetable.on_earth as 'facetable.on_earth', facetable.state as 'facetable.state', facetable.city_id as 'facetable.city_id', facetable.neighborhood as 'facetable.neighborhood', facetable.tags as 'facetable.tags', facetable.complex_array as 'facetable.complex_array', facet_cities.id as 'facet_cities.id', facet_cities.name as 'facet_cities.name' from facetable join facet_cities on facetable.city_id = facet_cities.id https://latest.datasette.io/fixtures?sql=select%0D%0A++facetable.pk+as+%27facetable.pk%27%2C%0D%0A++facetable.created+as+%27facetable.created%27%2C%0D%0A++facetable.planet_int+as+%27facetable.planet_int%27%2C%0D%0A++facetable.on_earth+as+%27facetable.on_earth%27%2C%0D%0A++facetable.state+as+%27facetable.state%27%2C%0D%0A++facetable.city_id+as+%27facetable.city_id%27%2C%0D%0A++facetable.neighborhood+as+%27facetable.neighborhood%27%2C%0D%0A++facetable.tags+as+%27facetable.tags%27%2C%0D%0A++facetable.complex_array+as+%27facetable.complex_array%27%2C%0D%0A++facet_cities.id+as+%27facet_cities.id%27%2C%0D%0A++facet_cities.name+as+%27facet_cities.name%27%0D%0Afrom%0D%0A++facetable%0D%0A++join+facet_cities+on+facetable.city_id+%3D+facet_cities.id

I'm using select foo.bar as 'foo.bar' here to ensure that every column has an unambiguous name, to avoid any weirdness from joining against a table where both tables have a 'name' column.

I think a .render_template() method on the Datasette class would be acceptable, since the purpose of that class will mainly be to provide a documented API for plugins to use: #576

It would be neat if this could support --since, with that argument automatically finding the maximum tweet ID from a previous search that used the same exact arguments (using the search_runs table).

It looks like Twitter really want you to subscribe to a premium API for this kind of thing and consume retweets via webhooks: https://developer.twitter.com/en/docs/accounts-and-users/subscribe-account-activity/api-reference

I'm going to give up on this for the moment.

I imagine it won't, since the data I would be recording and then passing to since_id would be the highest ID of my own tweets that have been retweeted at least once. So it won't be able to spot if I should check for fresh retweets of a given tweet.

Hmm... one thing that could be useful is that retweets_of_me can support a --since parameter - so if run frequently it should hopefully let us know which tweets we would need to run statuses/retweets/:id.json against.

I'm not sure if the --since parameter would show me a tweet that was previously retweeted but has now been retweeted again. I'll have a bit of a test and see.

Wow, that retweets_of_me endpoint is almost completely useless: $ twitter-to-sqlite fetch https://api.twitter.com/1.1/statuses/retweets_of_me.json It returns my own tweets that have been retweeted, but with no indication at all of who retweeted them.

It looks like this needs to be combined with this API - https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/get-statuses-retweets-id - to fetch the details of up to 100 recent users who actually DID retweet an individual status. But that has a one-every-12-seconds rate limit on it.

Might not need an incomplete boolean - may be possible to handle this with alter=True and then by filtering for users with null values in certain columns.

Also need to take #5 into account - if this command creates incomplete user records, how do we repair them?

And make sure that if we run this command first any future commands that populate users don't break (probably just a case of using alter=True in a few places).