home / github

Menu
  • Search all tables
  • GraphQL API

issue_comments

Table actions
  • GraphQL API for issue_comments

11 rows where author_association = "OWNER", issue = 1160182768 and user = 9599 sorted by updated_at descending

✖
✖
✖
✖

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

user 1

  • simonw · 11 ✖

issue 1

  • Optional Pandas integration · 11 ✖

author_association 1

  • OWNER · 11 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
1155364367 https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1155364367 https://api.github.com/repos/simonw/sqlite-utils/issues/412 IC_kwDOCGYnMM5E3XYP simonw 9599 2022-06-14T15:36:28Z 2022-06-14T15:36:28Z OWNER

Here's as far as I got with my initial prototype, in sqlite_utils/pandas.py:

```python from .db import Database as _Database, Table as _Table, View as _View import pandas as pd from typing import ( Iterable, Union, Optional, )

class Database(_Database): def query( self, sql: str, params: Optional[Union[Iterable, dict]] = None ) -> pd.DataFrame: return pd.DataFrame(super().query(sql, params))

def table(self, table_name: str, **kwargs) -> Union["Table", "View"]:
    "Return a table object, optionally configured with default options."
    klass = View if table_name in self.view_names() else Table
    return klass(self, table_name, **kwargs)

class PandasQueryable: def rows_where( self, where: str = None, where_args: Optional[Union[Iterable, dict]] = None, order_by: str = None, select: str = "*", limit: int = None, offset: int = None, ) -> pd.DataFrame: return pd.DataFrame( super().rows_where( where, where_args, order_by=order_by, select=select, limit=limit, offset=offset, ) )

class Table(PandasQueryable, _Table): pass

class View(PandasQueryable, _View): pass ```

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optional Pandas integration 1160182768  
1059652538 https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059652538 https://api.github.com/repos/simonw/sqlite-utils/issues/412 IC_kwDOCGYnMM4_KQO6 simonw 9599 2022-03-05T02:13:17Z 2022-03-05T02:13:17Z OWNER

It looks like the existing pd.read_sql_query() method has an optional dependency on SQLAlchemy:

``` ... import pandas as pd pd.read_sql_query(db.conn, "select * from articles")

ImportError: Using URI string without sqlalchemy installed.

Hah, no I was wrong about this: SQLAlchemy is not needed for SQLite to work, I just had the arguments the wrong way round:python pd.read_sql_query("select * from articles", db.conn)

Shows a DateFrame

```

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optional Pandas integration 1160182768  
1059651306 https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059651306 https://api.github.com/repos/simonw/sqlite-utils/issues/412 IC_kwDOCGYnMM4_KP7q simonw 9599 2022-03-05T02:10:49Z 2022-03-05T02:10:49Z OWNER

I could teach .insert_all() and .upsert_all() to optionally accept a DataFrame. A challenge there is mypy - if Pandas is an optional dependency, is it possibly to declare types that accept a Union that includes DataFrame?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optional Pandas integration 1160182768  
1059651056 https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059651056 https://api.github.com/repos/simonw/sqlite-utils/issues/412 IC_kwDOCGYnMM4_KP3w simonw 9599 2022-03-05T02:09:38Z 2022-03-05T02:09:38Z OWNER

OK, so reading results from existing sqlite-utils into a Pandas DataFrame turns out to be trivial.

How about writing a DataFrame to a database table?

That feels like it could a lot more useful.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optional Pandas integration 1160182768  
1059650190 https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059650190 https://api.github.com/repos/simonw/sqlite-utils/issues/412 IC_kwDOCGYnMM4_KPqO simonw 9599 2022-03-05T02:04:43Z 2022-03-05T02:04:54Z OWNER

To be honest, I'm having second thoughts about this now mainly because the idiom for turning a generator of dicts into a DataFrame is SO simple:

python df = pd.DataFrame(db.query("select * from articles")) Given it's that simple, I'm questioning if there's any value to adding this to sqlite-utils at all. This likely becomes a documentation thing instead!

{
    "total_count": 2,
    "+1": 2,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optional Pandas integration 1160182768  
1059649803 https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059649803 https://api.github.com/repos/simonw/sqlite-utils/issues/412 IC_kwDOCGYnMM4_KPkL simonw 9599 2022-03-05T02:02:41Z 2022-03-05T02:02:41Z OWNER

It looks like the existing pd.read_sql_query() method has an optional dependency on SQLAlchemy:

``` ... import pandas as pd pd.read_sql_query(db.conn, "select * from articles")

ImportError: Using URI string without sqlalchemy installed.

```

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optional Pandas integration 1160182768  
1059649213 https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059649213 https://api.github.com/repos/simonw/sqlite-utils/issues/412 IC_kwDOCGYnMM4_KPa9 simonw 9599 2022-03-05T02:00:10Z 2022-03-05T02:00:10Z OWNER

Requested feedback on Twitter here :https://twitter.com/simonw/status/1499927075930578948

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optional Pandas integration 1160182768  
1059649193 https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059649193 https://api.github.com/repos/simonw/sqlite-utils/issues/412 IC_kwDOCGYnMM4_KPap simonw 9599 2022-03-05T02:00:02Z 2022-03-05T02:00:02Z OWNER

Yeah, I imagine there are plenty of ways to do this with Pandas already - I'm opportunistically looking for a way to provide better integration with the rest of the Pandas situation from the work I've done in sqlite-utils already.

Might be that this isn't worth doing at all.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optional Pandas integration 1160182768  
1059646645 https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059646645 https://api.github.com/repos/simonw/sqlite-utils/issues/412 IC_kwDOCGYnMM4_KOy1 simonw 9599 2022-03-05T01:53:10Z 2022-03-05T01:53:10Z OWNER

I'm not an experienced enough Pandas user to know if this design is right or not. I'm going to leave this open for a while and solicit some feedback.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optional Pandas integration 1160182768  
1059646543 https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059646543 https://api.github.com/repos/simonw/sqlite-utils/issues/412 IC_kwDOCGYnMM4_KOxP simonw 9599 2022-03-05T01:52:47Z 2022-03-05T01:52:47Z OWNER

I built a prototype of that second option and it looks pretty good:

Here's the pandas.py prototype:

```python from .db import Database as _Database, Table as _Table, View as _View import pandas as pd from typing import ( Iterable, Union, Optional, )

class Database(_Database): def query( self, sql: str, params: Optional[Union[Iterable, dict]] = None ) -> pd.DataFrame: return pd.DataFrame(super().query(sql, params))

def table(self, table_name: str, **kwargs) -> Union["Table", "View"]:
    "Return a table object, optionally configured with default options."
    klass = View if table_name in self.view_names() else Table
    return klass(self, table_name, **kwargs)

class PandasQueryable: def rows_where( self, where: str = None, where_args: Optional[Union[Iterable, dict]] = None, order_by: str = None, select: str = "*", limit: int = None, offset: int = None, ) -> pd.DataFrame: return pd.DataFrame( super().rows_where( where, where_args, order_by=order_by, select=select, limit=limit, offset=offset, ) )

class Table(PandasQueryable, _Table): pass

class View(PandasQueryable, _View): pass ```

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optional Pandas integration 1160182768  
1059646247 https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059646247 https://api.github.com/repos/simonw/sqlite-utils/issues/412 IC_kwDOCGYnMM4_KOsn simonw 9599 2022-03-05T01:51:03Z 2022-03-05T01:51:03Z OWNER

I considered two ways of doing this.

First, have methods such as db.query_df() and table.rows_df which do the same as .query() and table.rows but return a DataFrame instead of a generator of dictionaries.

Second, have a compatibility class that is imported separately such as: python from sqlite_utils.pandas import Database Then have the .query() and .rows and other similar methods return dataframes.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optional Pandas integration 1160182768  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);
Powered by Datasette · Queries took 601.464ms · About: github-to-sqlite
  • Sort ascending
  • Sort descending
  • Facet by this
  • Hide this column
  • Show all columns
  • Show not-blank rows