Data Type Conversion Option for RJDBC

[joshbode]

Hi Simon,

I've added some code to a fork of RJDBC enabling the conversion of data returned from queries to specific R data-types, based on the Java SQL data type constants (see [http://docs.oracle.com/javase/7/docs/api/constant-values.html#java.sql]).

Not sure if this is something you want to incorporate, but my motivation for this is that I work with a lot of time-series data and just about every one of my queries has an as.POSIXlt or as.Date immediately following it. I much prefer RJDBC over other options for two reasons - 1) It's much more portable (I work on a Windows machine where I don't have admin access :( ), and 2) I've found some of the implicit type conversion in RODBC to be too enthusiastic at times (e.g. a string column containing only numbers converted to numeric that I'd rather keep as character).

To reduce impact on existing users, I've made it the default to perform no conversions, as per the existing code.

Happy to discuss and make changes if this is of any use.

[s-u]

Thanks, this is a good idea. There are a few minor things (e.g., you don't want to use as.POSIXlt on any real data), but I like the approach.

[s-u]

moved into data-type-conversion branch

[joshbode]

Great :)

Do you think the default conversion should be as.POSIXct or as.Date? To do it right with POSIXct probably requires some understanding of the timezone setting on the database and client?

Is there anything else you think should be altered?

[yasminlucero]

Can I ask about the status of this pull request? This looks like it will allow us to bring BIGINT over as character, (and then later apply bit64 int64 type). If so, we could really use this.

Do you need help with testing?

[joshbode]

I'd forgotten about this work! Based on Simon's comments, I've made some additional changes to fix the date conversion (DATE -> as.Date, DATETIME -> as.POSIXct) and brought my code up to date with master.

With the changes in this branch you should be able to achieve the conversions you described by setting up a higher priority type-conversion via the options - i.e. something like:

type.map = c('bigint'=c(types$BIGINT), default.type.map)
conversion.map = c('bigint'=bit64::as.integer64, default.conversion.map)
RJDBC.options(convert.types=TRUE, type.map=type.map, conversion.map=conversion.map)

[hhoeflin]

This code is very interesting for an issue I have with 64bit integers. However, I was wondering why over 2 years after the pull-request, it hasn't been merged. Is there some conflict anyone is aware of?

[joshbode]

Not to my knowledge - I'd completely forgotten about this work again.
Any thoughts @s-u?

[maxmoro]

Any news on this update. I'm really looking for this feature.
thanks

[joshbode]

Hi @maxmoro - I'm not using R much any more, so I had let this sit.

I haven't heard back from @s-u on this, but I've just brought the fork/PR up to date with master.
If you wanted to test/use it, let me know and I'll see how you could use it.

[maxmoro]

Hi Josh, thank you very much. This is very helpful. I'm out for the week. As soon I'm back, I'll use your version. Thanks Max

…

On Fri, Jun 23, 2017, 11:04 PM Josh Bode ***@***.***> wrote: Hi @maxmoro <https://github.com/maxmoro> - I'm not using R much any more, so I had let this sit. I haven't heard back from @s-u <https://github.com/s-u> on this, but I've just brought the fork/PR up to date with master. If you wanted to test/use it, let me know and I'll see how you could use it (probably using devtools install_github("joshbode/RJDBC") directly off my fork works) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIMgVnRnCC3r5fwP26aFGYF87cym_d1wks5sHKcGgaJpZM4B508Y> .

[joshbode]

Thinking about it, there's a build step in there (mkdist) that compiles the Java code and I don't believe there is a way to automate that with install_githib.

So, you may be better doing something like cloning the repo, running ./mkdist and installing the resultant package directly (e.g. install.packages("RJDBC_0.2-6.tar.gz", repos=NULL)).

If you have any issues (e.g. missing JDK, etc) let me know and I help or just compile it for you.

[joshbode]

OK - by adding src/Makefile (and a few other files), the package is installable directly from github via:

require(devtools)
install_github("joshbode/RJDBC")

assuming you are on Linux/macOS (or possibly Rtools if on Windows), and have make and the JDK (javac and jar) in your path.

[joshbode]

Hi @s-u - do you have any plans to merge this? Is there anything else you would like to see here?

Apart from the type conversions work, this also streamlines the build of the library so it works directly with R CMD INSTALL and devtools via install_github (by including a Makefile)

$ R CMD build .
* checking for file ‘./DESCRIPTION’ ... OK
* preparing ‘RJDBC’:
* checking DESCRIPTION meta-information ... OK
* cleaning src
* checking for LF line-endings in source and make files and shell scripts
* checking for empty or unneeded directories
Removed empty directory ‘RJDBC/inst’
* building ‘RJDBC_0.2-7.tar.gz’

If not, I'll close the PR.

[joshbode]

Closing - after all this time, I'm not using this package any more - if anyone wants to pick up this PR, feel free to fork my branch :)

[s-u]

Thanks, I have added a slightly different implementation in 1c6399c with similar goals. The main difference is that I have opted for single-point entry where the fetching form Java stays as-is and only the post-processing is done in R on per-column basis.